The SHARCNET log

System Environment

Jobs are submitted to each cluster using a set of queues. These queues have different limits on job sizes and run times. The configuration of queues for each cluster is different.

Note that all clusters use quad-nodes (except one that uses dual nodes). However, processors are allocated individually, so several different jobs may be running on the same node.

When started, jobs normally run to completion. However, the clusters support a "test" queue which has priority: jobs in this queue may cause running jobs to be suspended, and run in their place.

Log Format

This file contains one line per completed job with the following comma separated fields:

• job ID
• system (which cluster it ran on)
• user (sanitized)
• state (k=killed, d=done, f=failed)
• job type (1=serial, 2=parallel mpi, 3=threaded)
• submit date and time (seconds from epoch)
• start date and time (seconds from epoch)
• end date and time (seconds from epoch)
• utime (total user CPU time on all nodes)
• stime (total system CPU time on all nodes)
• atime (total allocated time of all nodes, excluding suspended time)
• ncpus (number of processors used)
• nnodes (number of nodes used; >1 only for mpi jobs)
• exit status
• memory in KB
• page faults
• command (sanitized)
• flags (serial, test, mpi, normal, default, threaded, ...)
• suspended time (wallclock for whole job, not per process.)
• suspension history (list of intervals that job was suspended, in seconds from beginning of job)
• array index (index of repeated job executions, but not necessarily consecutive)

Conversion Notes

• The cluster on which the job ran is noted as the partition.
• The original log contains both start/end information, and allocated time information. In general, runtime was computed as end - start. However, if the start time was missing (which happened for 89,579 jobs) or the end time was missing (which happened for 12,389 jobs), the total time (field atime) was used instead if it was available.
• If the suspended field was positive, its value was subtracted from the runtime, provided it had been calculated as end-start. This was done for 8,539 jobs. If the allocated time was used, this is supposed to already take suspension into account.
• The job status is based on the state field. The "done" code is actually based on the exit status being 0, so it is not necessarily completely reliable.
• The conversion loses the following data, that cannot be represented in the SWF:
  • The exact exit status of the job (unless 0)
  • The job flags (serial, mpi, ...). However, serial jobs can be identified as using only one processor.
  • The distinction between user time and system time.
  • The number of nodes used by the job.
  • The number of page faults.
  • The number of times a job was suspended, when these suspensions occured, and for how long. (Actually, this can be done using status codes 2, 3, and 4, but was not done.)
• The following anomalies were identified in the conversion:
  • 12,389 jobs had undefined end times.
  • 89,579 jobs had undefined start times, but only 26 had "success" status. In 42,711 cases the runtime was given, so start time was not needed; the rest had "failed" status.
  • Small negative wait times occured 5 times; they were changed to 0.
  • 2,440 jobs had an average CPU time higher than their runtime. In 1,237 cases the extra CPU time was larger than 1 minute.
  • 1,021 jobs were recorded as using 0 CPU time; this was changed to -1. 78 of these jobs had "success" status.
  • 1,058 jobs were recorded as using 0 processors; this was changed to -1. All these jobs had a "failed" status.
  • 58,459 jobs were recorded as using 0 memory; this was changed to -1 Of these jobs, 16,231 had "success" status
  • In 8477 jobs the given runtime (field atime) did not match the interval from the start time to the end time. The differences were often substantial (thousands or tens of thousands of seconds). In these cases the interval from start to end was prefered, because atime was negative in 817 cases.
  • In 80 jobs the suspend time was either negative or using it led to a negative runtime. In these cases the suspend time was ignored.

• In conversion 1 the status fields were inadvertantly reversed: failed jobs were marked 1 and successful jobs were 0. This was corrected in the new conversion.
• The logic for handling missing data was modified. In particular, when the start time is missing, the old conversion assigned the whole duration from submit to end to runtime. In the new conversion, if the job has "failed" status, this interval is instead assigned to wait time, and runtime is left as -1.

The conversion was done by a log-specific parser in conjunction with a more general converter module.

The Log in Graphics

File SHARCNET-Whale-2006-2.swf