Seaborg Accounts and Charging

When a job runs on Seaborg charges accrue against its repository's allocation. A parallel job is charged for exclusive use of each node it uses. The "SP Hours" charge for a given job is calculated by multiplying the job's "wall-clock time" by a factor of 16 (CPUs/node), and a "job priority" factor. (SP Hours are a factor of 2.5 smaller than the MPP Hours that we used in Fiscal Year 2003.)

Account information is available via the NIM web interface.

How accounts are charged and monitored

On the SP usage can be accrued in two ways:

• wallclock (or "connect") time for parallel jobs run on the compute nodes
• CPU time for serial jobs run on the login nodes (this includes user programs as well as system commands such as ls, vi, etc.)

Charges for jobs run on compute nodes

(wallclock hours used) 
	* (number of nodes) 
	* 16 (processors per node) 
	* (Class Charge Priority Factor) 

For example, if you run an 8-node regular priority job that begins at 12:00:00 and ends 2 hours later at 14:00:00, your SP charge is:

	2 hours * 8 nodes * 16 processors  * 1.0 = 256 SP Hours 

It doesn't matter if you use fewer than 16 processors per node, you are charged for all 16 (the default is to use 16). It doesn't matter if some or all of your tasks spent 1:55 hours waiting at a barrier and used only 5 minutes of CPU time. You are charged for all the time your parallel job is resident on multiple nodes.

Jobs submitted to loadleveler with both job_type=serial and class=regular are charged for use of 1 processor per node.

Charges for jobs run on the login nodes

(CPU hours used) 

Class Charge Priority Factor

Interactive and Debug Jobs

Interactive and debug jobs are charged at the same rate at the Regular rate described below.

Batch Priority Scheduling Classes

Three classes of batch scheduling are available:

• Premium
• Regular
• Low

A premium job is scheduled for execution before an otherwise equivalent regular job. A low job has a lower priority for scheduling. These priority classes affect how quickly a job is scheduled for execution in the "wait queues"; it does not effect the UNIX priority at which the job executes.

Charging for Priority Batch Scheduling

Priority scheduling classes have different charge rates. It is intended that most users, over the year, will run most of their jobs in the regular class.

Allocation awards are based on the assumption that about 10 percent of jobs will be run in the premium class, 10 percent in the low class, and 80 percent in the regular class. Projects that exhaust their allocation by spending more than 10% of their time in the premium class should not expect to receive additional mid-year funding to compensate them for having used the higher charge rate.

Rates:

• Premium scheduling at an elevated charge rate (2.0)
• Regular scheduling at the standard charge rate (1.0)
• Low scheduling at a reduced charge rate (0.5)

Exceptions:

• Regular charge priority jobs requesting 32 or more nodes are charged at the reduced charge rate (0.5)

The NERSC priority classes correspond to the LoadLeveler classes of the same name. Jobs in the SP debug and interactive LoadLeveler classes are charged at the "regular" rate. If no priority class is specified, a rate of "regular" is used.

See The IBM SP Batch System.

Determining your Priority Ratio

Although our accounting system does not currently report the time spent in each priority class, the NERSC Information Management sytem (NIM) shows your average charge factor (Avg CF) in its Account Summary display whenever you login (or select My Account Usage from the My Stuff pull-down menu).

To abide with the priority usage guidelines this ratio should be less than 1.1.

Charging to a different repo

Your charges will accrue to your default repo.

For interactive serial jobs you cannot specify a repo to charge; the SP operating system does not provide project accounting hooks for interactive jobs. Interactive jobs will be charged to your default repo, unless that repo is out of time (see below).

The LoadLeveler software does provide support for project accounting: to charge a batch job to a specific repo use the LoadLeveler keyword:

   #@ account_no = repo_name

If you don't specify a repo, your LoadLeveler job will be charged to your default repo (unless that repo is out of time, see below).

Interactive jobs may be charged to a non-default repository by using the poe command in conjunction with a LoadLeveler batch script. You must create a script with appropriate LoadLeveler keywords, for example

#@account_no = repo_name
#@job_type   = parallel
#@class      = interactive
#@node       = number_of_nodes 
#@total_tasks = number_of_tasks 
#@ network.MPI = csss,not_shared,US
#@queue

Then use the -llfile option to poe, e.g.:

% poe -llfile script_file_name

You will be prompted for the executable name and flags.

Running Out of Time

When a user exhausts his repository allocation, his account is placed in a restricted state: he can no longer run parallel jobs.

Accounting records are updated once a day at about 4:00 AM Pacific Time. Repo balances are adjusted shortly afterwards.

The following checks are made to determine whether the user has any valid repo to charge to and if so which repo to charge:

1. At LoadLeveler job submission time a filter runs to determine if the user is allowed to run batch jobs and if so what repo to charge:
   • If all the user's repos are out of time the submission fails.
   • If the user's specified repo is out of time then the submission fails.
   • If the user didn't specify a repo and the user's default repo is out of time and the user still has valid repos to charge to, then one of those valid repos will be charged. Note that the base default repo itself does not change. You can use NIM to see and change your default repo.
   • The repo chosen will be indicated in a message written to stdout.
2. At LoadLeveler and POE job run time a check is made to see if the job is still allowed to run:
   • for LoadLeveler jobs: does the repo determined in step 1 above still have time? If yes, the job can start running. If no, the job cannot.
   • for POE jobs: does any repo have time? If yes, the job can start running; if no, the job cannot.
3. Once a job has finished running the accounting process will make a final check on which repo to charge:
   • A LoadLeveler job is charged to the repo determined in step 1 above.
   • An interactive (including POE) job is charged to your default repo unless that repo is out if time. In the latter case, if the user has no other repos to charge to the job gets charged to the (negative) default repo; if the user has other valid repos to charge to, then the repo that gets charged is the first one with available time in the list returned by the getrepo utility. You can use NIM to change your default repo.