ARCGEN Processing

After the EDR file has been transferred and pre-processed with CDCOPY, it is ready for main processing with ARCGEN. This is the most important processing step in production as it creates the two primary files used in all subsequent processing and data analysis: ULAyyddd.RAT and ULAyyddd.PHA. This step is referred to as "running ARCGEN." ARCGEN, which stands for ARChive GENerator, is a fortran program. ARCGEN decodes, retrieves timing, and computes scientific data as described in the HISCALE Record Definitions, which can be found in dkb200:[ulysses.ulysses.landef.record_defs].

The command procedure GO_ARCGEN.COM, in dkb200:[ulysses.ulysses.autosys.go_arcgen], runs ARCGEN.EXE to create the ULAyyddd.RAT and ULAyyddd.PHA files. The following procedure shows an example of running GO_ARCGEN. This session shows two EDR files being processed. GO_ARCGEN can process any number of EDR files in the same session. The only limit to the number of files that can be processed is available disk space. Approximately 300,000 blocks of disk space are required on dkb200: to process one EDR file. GO_ARCGEN will evaluate the disk space and will not run if there is not enough space to complete an entire EDR file. As an example, if only 400,000 blocks of disk space are available on dkb200: when this particular session is started, GO_ARCGEN will process the first file, EDR92253_02.00 but will stop before attempting to process the second file, EDR92260_02.00. A message alerting the user of inadequate disk space is written to the file dkb200:[ulysses]WARNING.TXT. At this point, disk space must be cleared from dkb200:, the file EDR92253_02.00 must be deleted from the disk so it is not processed a second time, and GO_ARCGEN must be submitted again to process the remaining EDR file. As dkb200: usually remains above 1.6 million free blocks, this is not usually a problem.

Two log files are created by GO_ARCGEN in dkb200:[ulysses.ulysses.autosys.log] that can help locate the cause of problems if the job crashes or files are not created as expected. ARCGENyyddd_02.LOG shows the entire GO_ARCGEN session. As in the above example of inadequate disk space, this log file can be read to determine why the second EDR file was not processed. The second file, EDRyyddd.LOG, shows the details of ARCGEN.EXE processing. Also, any error or warnings will be written to WARNING.TXT in dkb200:[ulysses].

Depending on what other jobs are running on EPAM, GO_ARCGEN takes about 20 minutes for each EDR file (or one week of data). The following is a detailed list of procedures for running ARCGEN.

1. Log onto EPAM and run GO_ARCGEN (GO_ARCGEN is defined in the Ulysses LOGIN.COM as @autosys:[go_arcgen]GO_ARCGEN.COM).

$go_arcgen

---------------GO_ARCGEN---------------

Enter EDR directory (default is KUEPAM$dkb200:[ULYSSES.SECONDARY.WRK0.EDR]) <cr>
EDR92246_02.00 was last processed on 10-NOV-1992 00:51:02.34

Directory KUEPAM$DKB200:[ULYSSES.SECONDARY.WRK0.EDR]

EDR92253_02.00;1
EDR92260_02.00;1

Total of 2 files.

Are these the EDR files that you want to process (y/n)>y
When do you want to run ARCGEN
   default:now, (5)5:00 pm, or (m)midnight)>m
Enter number of high resolution plots to be created
  default:53 channels, or (1)1 page of selected channels><cr>
Job GO-ARCGEN92253_02 (queue MEDIUM, entry 799 holding until 17-NOV-1992 00:05
$

Response Options to GO_ARCGEN

Normally this will just be a return (<cr>) as dkb200:[ulysses.secondary.wrk0.edr] has been designated as the default directory for EDRyyddd_02.00 files.  However, this option allows another directory to be used for EDR file input.

Answer y for yes if all EDR files listed are to be processed.  If the EDR files listed are not all to be processed, respond with n.  A no response will abort the procedure and give a message directing the user to temporarily rename the unneeded files to another directory.

Return (<cr>) will start the procedure immediately, 5 will start the procedure at the next 1700 hour, and m will start the procedure at the next midnight.  As processing takes about one-eighth the time on EPAM that it did on KUSPAC, it is rarely necessary to put off the processing to midnight, when CPU usage is low.  The medium batch queue on EPAM is used for any of these options.

This option refers to an unused procedure of ARCGEN, RSPLOT.EXE.  This procedure produces high resolution plots that can be used as data quality control and for science purposes.  The procedure is obsolete as well as unexecutable, as the software that creates the plots does not work.

Files Created

Several files are created while GO_ARCGEN is running.  Most of the files created in dkb200:[ulysses] and its subdirectories are for information or trouble-shooting purposes and are automatically deleted in 30 days.  Some of the files are temporary, intermediate files used by the production software and are deleted before the job is completed.  Following is a list of files created by the GO_ARCGEN process.

dkb200:[ulysses]
    WARNING.TXT
    AUTONOTICE.TXT
    (These files are appended rather than created with each run of GO_ARCGEN.)
dkb200:[ulysses.ulysses.autosys.log]
    ARCGENyyddd_02.LOG
    EDRyyddd_02.LOG
dkb200:[ulysses.ulysses.autosys.go_arcgen]
    ARCGENyyddd_02.DAT
    DPIyyddd.DAT
    ERRyyddd_02.LOG
    FOR008.DAT
    RUN.COM
    RUN_ARCGEN.COM
    TIME.DAT
    ULA_DIR.LIS
dkb200:[ulysses.primary.wrk0.ula]
    IULAyyddd.RAT
    IULAyyddd.PHA
    ULAyyddd.RAT
    ULAyyddd.PHA

2. After ARCGEN has completed processing of the EDR files submitted, the quality of the files must be checked before further processing can be done.  First, the WARNING.TXT file must be examined for any new warnings incurred by ARCGEN processing.  In addition, the AUTONOTICE.TXT file can be examined to ensure that ARCGEN completed successfully.  A dir/size must also be performed on dkb200:[ulysses.primary.wrk0.ula] to determine the size of each ULA file created by ARCGEN.  Acceptable sizes are 18,000 to 25,000 blocks for .RAT files and 1500 to 3000 blocks for .PHA files. If the file sizes are not acceptable, the JPL quality control (QQC) reports must be examined (see the section entitled Miscellaneous Files and Procedures).  If there are no corresponding gaps in coverage reported in the QQC report, the EDRyyddd_02.LOG file must be checked.   This is the log file created when the EDRyyddd_02.00 file is originally scanned.   A gap appearing in the EDRyyddd.LOG file without a corresponding gap reported in the QQC could indicate a faulty, incomplete EDR file from JPL or problems with the HISCALE instrument. At this point, JPL must be contacted with this information.  If the QQC and the EDRyyddd_02.LOG files show no gaps, the problem could be in the production software ARCGEN.

In addition, email from other team members is sometimes sent announcing planned instrument calibrations or spacecraft anomalies that can affect the data.  These messages should be watched for.

3.  If the data quality appears normal after examining the various data and LOG files, the ULAyyddd.RAT and ULAyyddd.PHA files are renamed from dkb200:[ulysses.primary.wrk0.ula] to dkb200:[ulysses.primary.lan.ula].  This directory is designated as the input directory for further processing and is the designated directory available to other team members to copy the ULA files across the network.

IMPORTANT:  Once the .RAT and .PHA files have been renamed to dkb200:[ulysses.primary.lan.ula], the auto processing procedures will use them for input at the next midnight hour.

4.  Once the ULAyyddd.RAT and .PHA files have been renamed to dkb200:[ulysses.primary.lan.ula], the only files that should remain are IULAyyddd.RAT and .PHA files.  ARCGEN requires the last IULA files for processing the next batch of EDR data.  Be careful not to delete the latest IULA files.  For more information, see Notes about ARCGEN IULA Files.

5.  After about a month of data has accumulated in dkb200:[ulysses.secondary.wrk0.edr], or after the CD containing the data has been shipped, the EDRyydd.LAN files should be deleted, as each takes about 23,000 blocks of disk space.

6.  The ULAyyddd.RAT and ULAyyddd.PHA files are now ready for further processing.   See the section entitled Auto-Processing.

Running ARCGEN Interactively

It may sometimes be necessary to run ARCGEN interactively or with a short batch job so that it may be more closely monitored and controlled.This situation mainly arises when it is necessary to process an EDR file that is out of sequence with the normal processing schedule, or for testing ARCGEN or the EDR data.  ARCGEN requires the following input files to run:  IDF.DAT, EDRyyddd_02.00, and NAVyy001.DAT.  If magnetic field data is available for the time period being processed, it too can be used as input.  (This is rarely the case, however, as the magnetic field data for a given time is available some months after the EDR data.)

Three logical names must be pre-defined before running ARCGEN:  maginput, navinput, and lanoutput.  Maginput is the magnetic field file (UMAGyyddd.DAT) directory; navinput is the navigation file (NAVyy001.DAT) directory; and lanoutput is the ULA output file (ULAyyddd.RAT and ULAyyddd.PHA) directory.  ARCGEN will not run if navinput or the directory for LAN input (dka200:[ulysses.secondary.wrk0.edr]) is defined incorrectly.  If lanoutput is defined incorrectly, ARCGEN either will not run or will write the output files to the wrong directory. If maginput is assigned incorrectly, or the required UMAGyyddd.DAT file does not exist, ARCGEN will produce ULA files with no magnetic field data.  For information on magnetic field data files and navigation data files, see the section entitled Miscellaneous Files and Procedures.  Following is the procedure for interactive running of ARCGEN.

1.  The current ARCGEN.EXE and IDF.DAT files are located in dkb200:[ulysses.ulysses.autosys.go_arcgen], so set default to this directory.

2.  Assign directories for navinput, maginput (if any), and lanoutput.  Run ARCGEN program.

$assign dkb200:[ulysses.secondary.wrk0.nav]navinput
$assign dkb200:[ulysses.secondary.lan.umag]maginput
$assign dkb200:[ulysses.primary.wrk0.ula]lanoutput
$run/nodebug arcgen
EDR file name>dkb200:[ulysses.secondary.wrk0.edr]edr92253_02.00

Although lanoutput is defined to be the normal output directory for ULA files (dkb200:[ulysses.primary.wrk0.ula]) in the above example, it may be advisable to create a temporary directory somewhere under dkb200:[ulysses.primary.wrk0] in which to put these ULA files.  This will avoid errors and confusion between files created by normal processing and those created by a manual execution of ARCGEN.

3.  ARCGEN will create 3 files in the directory where it is being run:

DPIyyddd.DAT
ERRyyddd_02.LOG
FOR008.DAT

These files can be deleted if the job ran without error.  The ULA files can then be examined as outlined above.

4.  Alternatively, ARCGEN can be executed by a short command procedure.  The output of this procedure can be recorded and examined later.  An example command procedure to accomplish this:

$set verify
$set def dkb200:[ulysses.ulysses.autosys.go_arcgen]
$assign dkb200:[ulysses.secondary.wrk0.nav]navinput
$assign dkb200:[ulysses.secondary.lan.umag]maginput
$assign dkb200:[ulysses.primary.wrk0.ula]lanoutput
$run/nodebug arcgen
dkb200:[ulysses.secondary.wrk0.edr]edr92253_02.00
$exit

Notes about ARCGEN IULA Files

ARCGEN looks for incomplete (IULA).RAT and .PHA files in dkb200:[ulysses.primary.wrk0.ula] before attempting to create new .RAT and .PHA files using the specified EDRyyddd_02.00 files as input.  As in the above example using EDR92253_02.00 as input, ARCGEN looks for IULA92253.RAT and IULA92253.PHA, so the remainder of the day can be appended to these files to make a complete day's worth of data.  If the IULA files are not found, a warning message is generated in dkb200:[ulysses]WARNING.TXT stating that there is a data gap, and then ARCGEN proceeds to create the incomplete ULA92253.RAT and .PHA files.  ARCGEN looks for incomplete files for the five days prior to the starting records of the EDR input file. ARCGEN creates the IULA files and then automatically renames them to ULA files when the files contain one day's worth of data.

Next: Auto Processing