NGS accounting
The UK National Grid Service http://www.ngs.ac.uk/ implement a grid accounting system based on
OGF-UR XML accounting records. Grid-SAFE has special support for parsing and generating OGF-UR records.
The web-portal is not required to generate OGF-UR records this process can be run completely from the command line. A mysql databaseis required however.
Build the gridsafe executable jarfile using the ngs configuration parameters.
ant -Dservice.id=ngs app-jar
This includes additional configuration parameters to add support for globus and voms accounting logfiles.
Batch accounting files are loaded into either the PBSRecord, LSFRecord or SGERecord tables depending on your type of batch system.
java -jar gridsafe-apps.jar -p config.service.properties UploadUsageRecord -t PBSRecord -f 20090703 MachineName=mymachine.ngs.example.com Inserted lines: 935 Replaced lines: 0 Duplicate lines: 0 Partial lines: 0 Skipped lines: 1888 18: Not parsing records of type D 935: Not parsing records of type Q 935: Not parsing records of type S First record (completed): Fri Jul 03 00:01:08 BST 2009 Last record (completed): Fri Jul 03 16:13:03 BST 2009
Grid-SAFE will automatically recognise and ignore data that has already been loaded so it is safe to insert data multiple times. This reduces the need to keep track of which data files have previously been loaded.
Once the batch data is loaded the globus jobmanager logs can be added to the database. Grid-SAFE will only process a job-manager record where it finds a matching batch record so the batch data needs to be loaded first. As the various log files are written at points in the job lifecycle it can be more robust to regularly load the set of recent files rather than only load the newest file. This ensures the globus entries have additional chances to be processed if the batch job queues for a long time and the corresponding batch record is not generated.
java -jar gridsafe-apps.jar -p config.service.properties UploadUsageRecord -t Globus -f accounting.log.1 Inserted lines: 1708 Replaced lines: 0 Duplicate lines: 0 Partial lines: 5131 Skipped lines: 0
Finally if you are running LCAS/LCMAPS you can load the voms_accounting file to get VO information for the job.
java -jar gridsafe-apps.jar -p config.service.properties UploadUsageRecord -t VomsAccounting -f voms_accounting.log.1 Inserted lines: 1713 Replaced lines: 0 Duplicate lines: 0 Partial lines: 0 Skipped lines: 18287 18287: No link record found
Once the data is loaded into the database you can create XML records for upload to the NGS. The following command will generate the first four records from 2009 and put them in: output/record.0.xml output/record.1.xml output.record.2.xml output.record.3.xml which can then be uploaded using the ngs rusbulkupload utility.
java -jar gridsafe-apps.jar -p config.service.properties XMLGenerator -s 01/01/2009 -e 01/01/2010 -n 4 -f output/record
The XML records are generated by a from a template format.xml. This is acutally a standard Grid-SAFE report template so you can use the usual syntax to control the behaviour of the template.
