Parallel Workloads Archive: SDSC SP2

The San Diego Supercomputer Center (SDSC) SP2 log

System: 128-node IBM SP2
Duration: May 1998 thru April 2000
Jobs: 73,496

This extensive log contains information on the user, account, and application, requested and used nodes and time, CPU time, submit, wait and run times.

Note that in the first third of the log the utilization is about 10% lower than in the later two thirds. this could indicate that the system's configuration was different during this period.

The original log, with all the available information, was available directly from the NPACI JOBLOG repository, in a special format described on that site. The version in the Standard Workload Format loses some data as specified below.

The workload log from the SDSC SP2 was graciously provided by Victor Hazlewood, who also helped with background information and interpretation. If you use this log in your work, please use a similar acknowledgment.

This log is subject to the NPACI JOBLOG Repository Data Usage Agreement:
This Job Trace Repository is brought to you by the HPC Systems group of the San Diego Supercomputer Center (SDSC), which is the leading-edge site of the National Partnership for Advanced Computational Infrastructure (NPACI).
The JOBLOG data is Copyright 2000 The Regents of the University of California All Rights Reserved.
Permission to use, copy, modify and distribute any part of the JOBLOG data for educational, research and non-profit purposes, without fee, and without a written agreement is hereby granted, provided that this copyright notice is preserved in all copies and all works based on use or analysis of this data is properly referenced in any written or electronic publication.

Downloads:

SDSC-SP2-1998-4.swf 1.4 MB gz converted log
SDSC-SP2-1998-4.2-cln.swf 1.2 MB gz cleaned log -- RECOMMENDED, see usage notes
SDSC-SP2-1998-1.swf 1.3 MB gz OLD VERSION of converted log (replaced 8 Jan 2004)
SDSC-SP2-1998-2.swf 1.4 MB gz OLD VERSION of converted log (replaced 1 Aug 2006)
SDSC-SP2-1998-2.1-cln.swf 1.2 MB gz OLD VERSION of cleaned log (replaced 1 Aug 2006)
SDSC-SP2-1998-3.swf 1.4 MB gz OLD VERSION of converted log (replaced 30 Nov 2011)
SDSC-SP2-1998-3.1-cln.swf 1.2 MB gz OLD VERSION of cleaned log (replaced 30 Nov 2011)
SDSC-SP2-1998-4.1-cln.swf 1.2 MB gz OLD VERSION of cleaned log (replaced 27 Jan 2015)
(May need to click with right mouse button to save to disk)

Papers Using this Log:

This log was used in the following papers: [cirne00] [mualem01] [feitelson01] [cirne01b] [streit02] [krevat02] [srinivasan02] [lawson02] [sabin03] [shmueli03] [ernemann03] [feitelson03a] [song04] [streit04] [aridor04] [england04] [feitelson04b] [feitelson05b] [feitelson05c] [feitelson05d] [talby05] [tsafrir05b] [dutot05] [sabin05] [shmueli05] [zilber05] [yeo05] [yeo06] [brevik06] [feitelson06a] [tsafrir06a] [tsafrir06b] [shmueli06] [franke06] [ranjan06] [tsafrir07a] [feitelson07a] [tsafrir07b] [talby07] [shmueli07] [liy07] [esbaugh07] [ranjan08] [iosup08] [feitelson08] [shmueli09] [feitelson09] [buyya09] [tsafrir10] [yeo10] [sodan11] [vandenbossche11] [lindsay12] [liux12] [utrera12] [kurowski12] [krakov12] [kumar12] [zakay12] [neves12] [klusacek12] [etinski12] [deng13] [shih13] [huang13c] [zakay13] [krakov13] [rajbhandary13] [cao14] [kumar14] [zakay14] [zakay14b] [feitelson14] [liu15] [carastans17] [soysal19]

System Environment

This is a 128-node IBM SP2 system.

Log Format

The original log was available from the NPACI JOBLOG repository, which also included a description of its format. Luckily we have a cached copy of the format specification.

Conversion Notes

The converted log is available as SDSC-SP2-1998-4.swf. The conversion from the original format to SWF was done subject to the following. The conversion was done by a log-specific parser in conjunction with a more general converter module.

The differences between conversion 4 (reflected in SDSC-SP2-1998-4.swf) and conversion 3 (SDSC-SP2-1998-3.swf) are

The differences between conversion 3 (reflected in SDSC-SP2-1998-3.swf) and conversion 2 (SDSC-SP2-1998-2.swf) are The differences between conversion 2 (reflected in SDSC-SP2-1998-2.swf) and conversion 1 (SDSC-SP2-1998-1.swf) are

Usage Notes

The original log contains several flurries of very high activity by individual users, which may not be representative of normal usage. These were removed in the cleaned version, and it is recommended that this version be used. In addition, the first 10 jobs were removed.
The cleaned log is available as SDSC-SP2-1998-4.2-cln.swf.

A flurry is a burst of very high activity by a single user. The filters used to remove the four flurries that were identified are

user=21 and job>13716 and job<16208 (944 jobs)
user=374 and job>14968 and job<28553 (11740 jobs)
user=197 and job>31766 and job<33203 (635 jobs)
user=328 and job>66552 and job<68107 (452 jobs)
Removing the first 10 jobs was added in the second cleaned version, as they seem to represent activity from long before the actual logging started. Note that the filters were applied to the original log, and unfiltered jobs remain untouched. As a result, in the filtered logs job numbering is not consecutive.

Further information on flurries and the justification for removing them can be found in:

The Log in Graphics

File SDSC-SP2-1998-4.swf

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance

File SDSC-SP2-1998-4.2-cln.swf (cleaned)

weekly cycle daily cycle burstiness and active users job size and runtime histograms job size vs. runtime scatterplot utilization offered load performance


Parallel Workloads Archive - Logs