The HPC2N Seth log
| System: | A Linux cluster located in Sweden |
| Duration: | July 2002 thru January 2006 |
| Jobs: | 527,371 |
This log contains three and a half years worth of accounting records from the
High-Performance Computing Center North (HPC2N) in Sweden.
HPC2N is a joint operation with several universities and facilities.
For more information about this installation, see URL
http://www.hpc2n.umu.se/.
This log is from a 120-node Linux cluster named
Seth.
The HPC2N workload log was graciously provided by Ake Sandgren,
who also helped with background information and interpretation.
If you use this log in your work, please use a similar acknowledgment.
Michael Jack is credited with getting it to the archive.
Downloads:
(May need to click with right mouse button to save to disk)
|
|
System Environment
Seth is a 120-node Linux cluster.
Each node contains two 240 AMD Athlon MP2000+ processors, running at
1.667 GHz.
The total system peak performance is 800 Gflops.
Nodes have 1 GB of RAM each, shared by the two processors.
They are connected by a 3D SCI interconnect organized as a 4x5x6 grid,
and by a fast Ethernet.
Scheduling is performed with the Maui scheduler.
Log Format
The original log is available as HPC2N-2002-0.
This file contains one line per completed job and conforms to the
Maui
workload trace format.
Conversion Notes
The converted log is available as HPC2N-2002-2.swf.
The conversion from the original format to SWF was done subject to the following.
-
The log includes a "dispatch" field that was ignored; the job's start
time was taken from the "start" field.
-
The number of processors used is calculated as "allocated tasks"
multiplied by "dedicated processors per task".
This is supposed to capture the resources used.
However, the logical parallelism in the jobs may be better
approximated by just using "allocated tasks".
-
The average CPU time used was computed as the number of
processors seconds actually utilized by the job
divided by the number of tasks (not allocated processors).
-
Used memory per processor is computed as the "dedicated memory per
task" divided by the "dedicated processors per task", and multiplied
by 1024 to obtain a result in KB.
-
Requested processors was computed as "tasks requested" multiplied by
"dedicated processors per task".
-
Requested time is the wallclock limit, not a precise estimate.
-
Requested memory is the "requested node memory", and not
divided by 2.
The reason is that when it was divided by 2 (to obtain memory per
processor), 88% of the jobs were found to have used more memory than
requested.
When the value wasn't divided by 2, less than half percent of the jobs
were problematic this way.
It was therefore assumed in the conversion that the required node
memory in the log was per processor, and not per node.
-
The status field mapping used was
| Completed | 1 |
| Removed | 5 |
| Not Run | 0 |
-
The conversion loses the following data, that cannot be represented in
the SWF:
- Distinction between dispatch time and start time (but usually
they are actually the same).
- Job attributes (e.g. BACKFILL, RESTARTABLE).
- Distinctions between tasks (processes) and nodes:
while 332,275 requested 2 tasks per node (with is normal given two
processors per node), 164,739 requested only one task per node.
- Requirements for dedicated resources.
- List of specific nodes that were used for each job.
-
The following anomalies were identified in the conversion:
- 1,048 jobs had a status of "Running", which is undocumented.
These jobs were submitted from 2002 to 2004, so they are not jobs that
were left running when the log was collected (unless maybe it was
collected in parts).
Moreover, most of them have a reasonable looking end time.
The status of these jobs was set to -1.
- 77 jobs had undefined end times.
Their runtimes were approximated using the CPU time.
- 5 jobs had negative run times; they were changed to 0.
In 3 cases negative run times were larger than 1 minute.
- 72 jobs had negative wait times; they were changed to 0.
In 11 cases these were larger than 1 minute.
In 1 case it was larger than 1 hr 5 minutes, and changed to -1.
- 767 jobs got more processors than they requested.
- 30,357 jobs requested 0 tasks per node.
- 63,732 jobs were recorded as requesting 0 memory; this was changed to -1.
- 5,671 jobs were recorded as using 0 memory; this was changed to -1.
Of the jobs using 0 memory, 5646 had "success" status.
- 2,548 jobs used more memory than they requested.
- 15,827 jobs got more runtime than they requested.
In 6,784 cases the extra runtime was larger than 1 minute.
- 138,621 jobs had an average CPU time higher than their runtime.
In 60608 cases the extra CPU time was larger than 1 minute.
- 73,512 jobs were recorded as using 0 CPU time; this was changed to -1.
Of these, 73,483 had "success" status.
The differences between conversion 1 (reflected in HPC2N-2002-1.swf)
and conversion 2 (reflected in HPC2N-2002-2.swf) are
-
In conversion 1 the number of processors used was calculated as twice
the number of nodes, where the number of nodes is extracted using the
cardinality of the list of allocated nodes.
However, this is always an even number (because there are two
processors per node).
To correctly account for cases where the job actually used an odd
number, it was compared with the number of requested tasks multiplied by
the number of dedicated processors per task.
If the latter was odd and smaller by 1 from the former, it was used
instead.
In the new conversion, the number of allocated tasks multiplied
by the number of dedicated processors per task is used instead.
This is significant because 164739 jobs requested only one task per
node, presumably because of memory considerations.
-
In conversion 1, CPU time was calculated as total CPU time divided by
allocated processors.
In conversion 2 it is divided by the number of tasks.
Again, this is significant for all those jobs that requested only one
task per node.
The result is uniformly very close to the running time, lending
credence to this approach.
-
In conversion 2, CPU time was used to approximate runtime in 77 jobs.
The conversion was done by
a log-specific parser
in conjunction with a more general
converter module (version 2).
Usage Notes
The log contains abnormally high activity by several individual users.
This includes unparalleled activity by user 2, who submitted 305,178
jobs, which constitutes 57.8% of the whole log, and two smaller flurries.
These were removed in the cleaned version, and it is recommended that
this version be used.
For example, this removed the anomalous pattern of activity in
different days of the week.
In addition, the first 10 jobs were removed.
The cleaned log is available as HPC2N-2002-2.2-cln.swf.
A flurry is a burst of very high activity by a single
user.
The filters used to remove the activity of user 2 and the flurries that were identified are
user=2 (305178 jobs)
user=60 and job>254199 and job<271543 (12333 jobs)
user=67 and job>289097 and job<305173 (6984 jobs)
Removing remaining jobs up to job 10 was added in the second cleaned
version, as they seem to represent activity from long before the
actual logging started.
In total, 324500 jobs were removed.
Note that the filters were applied to the original log, and unfiltered
jobs remain untouched.
As a result, in the filtered log job numbering is not consecutive.
Further information on flurries and the justification for removing
them can be found in:
The Log in Graphics
File HPC2N-2002-2.swf
File HPC2N-2002-2.2-cln.swf
Parallel Workloads Archive - Logs