First, VIRTUAL CORE HOURS is determined daily for each cluster with the following procedure:
A script that runs daily, loops over all the condor SCHEDDs’ in a cluster. For each SCHEDD condor_history is queried and RemoteWallClocktime for the unique combination of a job tag and a user is determined.
The LDG accounting script script could be obtained at this repo.
Some clusters have enabled hyper threading. To compare a hyper-threaded cluster and a non hyper-threaded cluster, we covert VIRTUAL CORE HOURS to CPU CORE HOURS by the following procedure:
Each cluster has a maximum possible CPU core hours, let’s call it CPU_Hmax. We max out the CPU CORE HOURS at CPU_Hmax.
i.e.
if VIRTUAL CORE HOURS < CPU-Hmax, then CPU CORE HOURS = VIRTUAL CORE HOURS,
else, CPU CORE HOURS (for each job) = VIRTUAL CORE HOURS/CPU_Hmax.
The following table lists the CPU_Hmax for different clusters.
Cluster
CPU_Hmax (hours)
LIGO-CIT (O1)
100896
ATLAS-AEI (O1)
315120
LIGO-LLO (O1)
57984
LIGO-LHO (O1)
57984
NEMO-UWM (O1)
81408
SUGAR-SU (O1)
103968
VIRGO.CNAF (O1)
177792
IUCAA (O1)
60480
ARCCA-CDF (O1)
50400
LIGO-CIT (O2)
200256
ATLAS-AEI (O2)
511344
LIGO-LLO (O2)
69120
LIGO-LHO (O2)
69120
NEMO-UWM (O2)
35328
SUGAR-SU (O2)
96048
VIRGO.CNAF (O2)
177792
IUCAA (O2)
60480
ARCCA-CDF (O2)
50400
LIGO-CIT (Starting May 1, 2018)
276288
LIGO-LLO (Starting May 1, 2018)
78336
LIGO-LHO (Starting May 1, 2018)
78336
VULCAN
46224
NEMO-UWM scalable (Starting Oct 1, 2018)
54720
ARCCA-CDF (O3)
69408
OSG does not get scaled for hyperthreading. For OSG, VIRTUAL CORE HOURS = CPU CORE HOURS.
SU HOURS:
Clusters have different specs with heterogenous processors. Hence to do an apples to apples comparison, we scale (by multiplying a factor, SU_f) the CPU CORE HOURS to Service Unit (SU) Hours. 1 SU hour is 1 Intel E52670 core hour to perform a SPECfp computation.
The following table lists the SU_f for different clusters
Cluster
SU_f
LIGO-CIT (O1)
0.43
ATLAS-AEI (O1)
0.88
LIGO-LLO (O1)
0.48
LIGO-LHO (O1)
0.48
NEMO-UWM (O1)
0.77
SUGAR-SU (O1)
0.67
IUCAA (O1)
1.11
ARCCA-CDF (O1)
0.97
LIGO-CIT (O2)
1.39
ATLAS-AEI (O2)
1.11
LIGO-LLO (O2)
1.43
LIGO-LHO (O2)
1.43
NEMO-UWM (O2)
0.88
SUGAR-SU (O2)
0.69
IUCAA (O2)
1.11
ARCCA-CDF (O2)
0.97
OSG (O1/O2)
0.71
VIRGO.CNAF (O1/O2)
0.6125
LIGO-CIT (Starting May 1, 2018)
1.46
LIGO-LLO (Starting May 1, 2018)
1.46
LIGO-LHO (Starting May 1, 2018)
1.46
VULCAN
1.1685
NEMO-UWM scalable (Starting Oct 1, 2018)
1.67
NEMO-UWM non-scalable (Starting Oct 1, 2018)
0.57
ARCCA-CDF (O3)
1.3
OSG publishes a net SU conversion factor. We use that instead of determining conversion the factor for each OSG site, for the sake of simplicity.
Then we convert SU hours to a dollar amount with the rate of 0.02 $ per one SU hour.
Worksheet for calculating CPU_Hmax and SU_f
-----------------------------------------
ATLAS-AEI has:
1)1770 4-cores E3-1220v3 processors which has SPECfp rate per socket = 140
2)260 4-cores E3-1231v3 processors which has SPECfp rate per socket = 151
3)51 6-cores E5-1650v2 processors which has SPECfp rate per socket = 225
4)920 14-cores E5-2658v4 processors which has SPECfp rate per socket = 446
Total number of physical CPU cores at ATLAS-AEI is: 21306
Maximum daily physical CPU core hours, CPU_Hmax, for ATLAS-AEI: 511344
Average SPECfp rate per core for ATLAS-AEI: 33
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for ATLAS-AEI:
SU_f = Average SPECfp rate per core for ATLAS-AEI / SPECfp rate per core for E52670
= 1.10972652461
-----------------------------------------
LIGO-CIT has:
1)96 8-cores E5-2670 processors which has SPECfp rate per socket = 240
2)198 4-cores E3-1241v3 processors which has SPECfp rate per socket = 153
3)1120 4-cores E3-1240v5 processors which has SPECfp rate per socket = 192
4)192 12-cores E5-2650-v4 processors which has SPECfp rate per socket = 410
5)264 12-cores Xeon Gold 6136 processors which has SPECfp rate per socket = 600
Total number of physical CPU cores at LIGO-CIT is: 11512
Maximum daily physical CPU core hours, CPU_Hmax, for LIGO-CIT: 276288
Average SPECfp rate per core for LIGO-CIT: 43
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for LIGO-CIT:
SU_f = Average SPECfp rate per core for LIGO-CIT / SPECfp rate per core for E52670
= 1.46367268937
-----------------------------------------
LIGO-LLO has:
1)40 8-cores E5-2630v3 processors which has SPECfp rate per socket = 290
2)448 4-cores E3-1240v5 processors which has SPECfp rate per socket = 192
3)64 12-cores E5-2650-v4 processors which has SPECfp rate per socket = 410
4)32 12-cores Xeon Gold 6136 processors which has SPECfp rate per socket = 600
Total number of physical CPU cores at LIGO-LLO is: 3264
Maximum daily physical CPU core hours, CPU_Hmax, for LIGO-LLO: 78336
Average SPECfp rate per core for LIGO-LLO: 43
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for LIGO-LLO:
SU_f = Average SPECfp rate per core for LIGO-LLO / SPECfp rate per core for E52670
= 1.46094771242
-----------------------------------------
LIGO-LHO has:
1)40 8-cores E5-2630v3 processors which has SPECfp rate per socket = 290
2)448 4-cores E3-1240v5 processors which has SPECfp rate per socket = 192
3)64 12-cores E5-2650-v4 processors which has SPECfp rate per socket = 410
4)32 12-cores Xeon Gold 6136 processors which has SPECfp rate per socket = 600
Total number of physical CPU cores at LIGO-LHO is: 3264
Maximum daily physical CPU core hours, CPU_Hmax, for LIGO-LHO: 78336
Average SPECfp rate per core for LIGO-LHO: 43
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for LIGO-LHO:
SU_f = Average SPECfp rate per core for LIGO-LHO / SPECfp rate per core for E52670
= 1.46094771242
-----------------------------------------
NEMO-UWM scalable has:
1)95 24-cores Xeon Gold 6136 processors which has SPECfp rate per socket = 1200
Total number of physical CPU cores at NEMO-UWM scalable is: 2280
Maximum daily physical CPU core hours, CPU_Hmax, for NEMO-UWM scalable: 54720
Average SPECfp rate per core for NEMO-UWM scalable: 50
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for NEMO-UWM scalable:
SU_f = Average SPECfp rate per core for NEMO-UWM scalable / SPECfp rate per core for E52670
= 1.66666666667
-----------------------------------------
NEMO-UWM non-scalable has:
1)92 16-cores E5-2650 processors which has SPECfp rate per socket = 410
Total number of physical CPU cores at NEMO-UWM non-scalable is: 3392
Maximum daily physical CPU core hours, CPU_Hmax, for NEMO-UWM non-scalable: 81408
Average SPECfp rate per core for NEMO-UWM non-scalable: 17
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for NEMO-UWM non-scalable:
SU_f = Average SPECfp rate per core for NEMO-UWM non-scalable / SPECfp rate per core for E52670
= 0.567216981132
-----------------------------------------
VIRGO.CNAF has:
1)926 8-cores 6320 processors which has SPECfp rate per socket = 147
Total number of physical CPU cores at VIRGO.CNAF is: 7408
Maximum daily physical CPU core hours, CPU_Hmax, for VIRGO.CNAF: 177792
Average SPECfp rate per core for VIRGO.CNAF: 18
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for VIRGO.CNAF:
SU_f = Average SPECfp rate per core for VIRGO.CNAF / SPECfp rate per core for E52670
= 0.6125
-----------------------------------------
SUGAR-SU has:
1)667 6-cores X5650 processors
Total number of physical CPU cores at SUGAR-SU is: 4002
Maximum daily physical CPU core hours, CPU_Hmax, for SUGAR-SU: 96048
Average SPECfp rate per core for SUGAR-SU: 20
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for SUGAR-SU:
SU_f = Average SPECfp rate per core for SUGAR-SU / SPECfp rate per core for E52670
= 0.694444444444
-----------------------------------------
IUCAA has:
1)210 12-cores E5-2670 processors which has SPECfp rate per socket = 400
Total number of physical CPU cores at IUCAA is: 2520
Maximum daily physical CPU core hours, CPU_Hmax, for IUCAA: 60480
Average SPECfp rate per core for IUCAA: 33
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for IUCAA:
SU_f = Average SPECfp rate per core for IUCAA / SPECfp rate per core for E52670
= 1.11111111111
-----------------------------------------
ARCCA-CDF has:
1)72 20-cores Gold 6148 processors which has SPECfp rate per socket = 940
2)121 12-cores E5-2680 processors which has SPECfp rate per socket = 379
Total number of physical CPU cores at ARCCA-CDF is: 2892
Maximum daily physical CPU core hours, CPU_Hmax, for ARCCA-CDF: 69408
Average SPECfp rate per core for ARCCA-CDF: 39
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for ARCCA-CDF:
SU_f = Average SPECfp rate per core for ARCCA-CDF / SPECfp rate per core for E52670
= 1.3086560627
-----------------------------------------
Vulcan has:
1)471 4-cores E3-1220v3 processors which has SPECfp rate per socket = 140
2)7 6-cores E5-1650v2 processors which has SPECfp rate per socket = 225
Total number of physical CPU cores at Vulcan is: 1926
Maximum daily physical CPU core hours, CPU_Hmax, for Vulcan: 46224
Average SPECfp rate per core for Vulcan: 35
SPECfp rate per core for E52670 = 240/8 = 30
CPU core hours to SU conversion factor, SU_f, for Vulcan:
SU_f = Average SPECfp rate per core for Vulcan / SPECfp rate per core for E52670
= 1.16848390447
-----------------------------------------
