Auto Processing

Auto processing is a group of command procedures and programs that runs automatically every day to handle various routine data processing functions.  There are a number of command procedures and called programs that complete a variety of tasks, but currently only 2 general tasks are performed:  1) running the various versions of the LANAVG software which creates the averaged data files, and 2) creating hourly average plots from some of these averaged data files.  ULAyyddd.RAT and ULAyyddd.PHA files are required for all auto procedures that are executed.  Once newly created .RAT and .PHA files available, they must be renamed to the directory dkb200:[ulysses.primary.lan.ula] before the auto processing procedures can use them as input.

All of the auto processing command procedures, the called executable programs, and the related auxiliary files are located in dkb200:[ulysses.ulysses.autosys] and its subdirectories.  (The source code is in various subdirectories in dkb200:[ulysses.ulysses.production].)  The entire process is controlled by a command procedure called AUTO_WAKE_UP.COM that is automatically submitted every day at five minutes past midnight.  Submission at 12:05 am was chosen because CPU usage is low at this hour and auto processing can take anywhere from 30 minutes to 10 hours, depending on the data available for processing.  AUTO_WAKE_UP.COM was originally designed to restart itself if the system goes down so it should never have to be manually submitted.

AUTO_WAKE_UP.COM's primary function is to submit the main command procedure, AUTO_SCHEDULE.COM, which submits various other command procedures that in turn handle the two categories of processing outlined above.  Each procedure submitted by AUTO_SCHEDULE creates a daily log file in the subdirectory dkb200:[ulysses.ulysses.autosys.log] that can be used to trace possible problems.   The individual log file names correspond to the names of the command procedures.   For example, the log file AUTO_F_LANAVG.LOG is the history files of the command procedure AUTO_F_LANAVG.COM.  The log files are automatically purged by AUTO_SCHEDULE.COM after 30 days.  This time period allows old log files to be examined if a problem is discovered several days after the problem occurred.

The two files WARNING.TXT and AUTONOTICE.TXT in dkb200:[ulysses] are also appended whenever new activity has occurred as a result of any auto processing procedures.   The file AUTONOTICE.TXT alerts the user to new files created from any of the command procedures or programs.  The file WARNING.TXT alerts the user to any possible problems that may have occurred as a result of processing.

Since AUTO_SCHEDULE.COM breaks each task into its own separate command procedure or module, it can be easily modified to add new procedures or delete obsolete procedures.   (The current version of AUTO_SCHEDULE.COM has some procedures commented out because they are obsolete or have not been completely debugged.)  The following is an overview of the different procedures submitted by AUTO_SCHEDULE:

AUTO_LANAVG

AUTO_LANAVG runs LANAVG.EXE which creates the hourly average archive data files named UAVyyddd.LAN in dkb200:[ulysses.secondary.lan.uav].  The UAVyyddd.LAN files contain a time range of 32 days of data.  An IUAVyyddd.LAN file is created and appended as more data (ULAyyddd.RAT files) are made available for processing by LANAVG.  Once the IUAVyyddd.LAN file contains 32 days of data, it is automatically renamed UAVyyddd.LAN.

AUTO_F_LANAVG

AUTO_F_LANAVG runs F_LANAVG.EXE which creates the filtered (QCA, or Quality Controlled Averaging), hourly archive data files named UAFyyddd.LAN in dkb200:[ulysses.secondary.lan.uaf].  The functionality of AUTO_F_LANAVG is exactly the same as the above AUTO_LANAVG except that it runs a different executable.

AUTO_CUAF

AUTO_CUAF runs CMPAVG.EXE which compresses the UAFyyddd.LAN files.  An ICUAFyy001.LAN file is created and appended as more data (UAFyyddd.LAN files) are available.  Once the ICUAFyy001.LAN file contains one year of data, it is automatically renamed CUAFyy001.LAN.

Plotting Procedures

Currently, the only plotting procedure that is executed is AUTO_01HPLOT.COM.   (There are multiple versions of this procedure located in various directories.   The one that is executed is located in dkb200:[ulysses.ulysses.autosys.auto_f_lanavg.plot] and thus plots data resulting from the execution of AUTO_F_LANAVG.COM.)  The software (VEGII) used to create the .PS plot files is not located on EPAM, and therefore the plots must be created on KUSPA1.   AUTO_01HPLOT.COM automatically copies three sets of files to the directory kuspa1$dka100:[wolff.ulysses]:

1.  (I)UAFyyddd##.LIS files that contain the data to be plotted, where ## represents the number of the page of the plot, 01 through 10.  If the files are IUAFyyddd##.LIS files, it is not necessary to plot  the data, as it will result in an incomplete plot.  All files copied to kuspa1$dka100:[wolff.ulysses] can then be deleted until UAFyyddd##.LIS files are copied.  These contain all 32 days of data.

2.  A LANVGI.COM procedure for each of the ten pages of plots.  This procedure is executed through VEGII and extracts the data from the UAFyyddd##.LIS files; and 

3.  A PLOT.COM procedure for each of the ten pages which invokes VEGII, calls LANVGI.COM, and renames the output plot file to the appropriate name, UAFyyddd##.PS.

The procedure to run these PLOT.COM files is the following:

1.  Log onto KUSPA1 with the ulysses account.  Set default to dka100:[wolff.ulysses] and check to see whether all needed files are there.

2.  Run NAME.COM to rename all PLOT.COMs to separate file names.

3.  Use the EVE editor to modify RUN.COM to insert the appropriate yyddd in the UAFyyddd##.PS field.  Run RUN.COM.  This will execute the PLOT.COM procedures and print out all ten pages of the postscript plots.

4.  Although not necessary, it may be a good idea to copy all the .PS plot files to the directory kuepam$dkb200:[ulysses.secondary.lan.uaf] to save them for a month.   They will be deleted automatically by auto processing.

Next: Program and Procedure Library