Chapter 3: HISCALE Ground Data Processing and Software

Processing Data into ULAYYDDD.RAT, ULAYYDDD.PHA, and UAFYYDDD.LAN Files

1. Type “dir” at the prompt to get a current directory listing. An example is listed below.

    

   $dir

    

   Directory KUEPAM$DKB200:[ULYSSES.SECONDARY.WRK0.EDR
   AAA_README.TXT;1	ALPHA_COMPILE.COM;3	CDCOPY.EXE;5	CDCOPY.FOR;11
   CD_TO_VAX.COM;1	EDR06001.LAN;1	EDR06008.LAN;1	EDR06015.LAN;1
   EDR06022.LAN;1	EDR06029.LAN;1	EDR06036.LAN;1	EDR06043.LAN;1
   EDR06050.LAN;1	EDR06057.LAN;1	EDR06064.LAN;1	EDR06071.LAN;1
   EDR06078.LAN;1	EDR06085.LAN;1	EDR06092.LAN;1	EDR06099.LAN;1
   EDR06106.LAN;1	EDR06113.LAN;1	EDR06120.LAN;1	EDR06127.LAN;1
   EDR06127_02.00;1	EDR06134.LAN;1	ERR94001_02.LOG;2	ERR94001_02.LOG;1
   FTP_LAN.COM;13	FTP_LAN.COM;12	FTP_LAN.COM;11	FTP_LAN.COM;10
   LECP.TXT;1	OCTOBER-RUN.LOG:1	README.TXT;2	SEND.COM;1
   SETHOST.LOG;2	SETHOST.LOG;1	TEMP.DIR;1

    

   Total of 39 files.

  

2. Next, find and delete the existing EDRYYDDD_02.00;* file. The command is “delete,” or “del,” as shown in the following example.
   

   $del edr06127_02.00;*
   %DELETE-I-FILDEL,
   KUEPAM$DKB200:[ULYSSES.SECONDARY.WRK0.EDR]EDR06127_02.00;1 deleted (18865
   blocks)

    

3. After the file is deleted, run cdcopy.exe without the debugger on. Use the highlighted command in the following example, which contains an abbreviated version of the readout the program will produce.

    

   $run/nodebug cdcopy
   input filename: edr06134.lan
   1 2006 134 00:03:49.2 1024bps
   2 2006 134 00:08: 5.2 1024bps
   3 2006 134 00:12:21.2 1024bps
   4 2006 134 00:16:37.2 1024bps
   5 2006 134 00:20:53.2 1024bps
   6 2006 134 00:25: 9.2 1024bps…

   1232 2006 140 23:40:25.6 1024bps
   1233 2006 140 23:44:41.6 1024bps
   1234 2006 140 23:48:57.6 1024bps
   1235 2006 140 23:53:13.7 1024bps
   1236 2006 140 23:57:29.7 1024bps
   First time: 2006 134 00:03:49.2
   Last time : 2006 140 23:57:29.7
   

    
   

4. Cdcopy.exe will produce an EDRYYDDD.RAW file. Do a directory listing as follows to ensure that the file now exists.

    

   $dir

    

   AAA_README.TXT;1	ALPHA_COMPILE.COM;3	CDCOPY.EXE;5	CDCOPY.FOR;11
   CD_TO_VAX.COM;1	EDR06001.LAN;1	EDR06008.LAN;1	EDR06015.LAN;1
   EDR06022.LAN;1	EDR06029.LAN;1	EDR06036.LAN;1	EDR06043.LAN;1
   EDR06050.LAN;1	EDR06057.LAN;1	EDR06064.LAN;1	EDR06071.LAN;1
   EDR06078.LAN;1	EDR06085.LAN;1	EDR06092.LAN;1	EDR06099.LAN;1
   EDR06106.LAN;1	EDR06113.LAN;1	EDR06120.LAN;1	EDR06127.LAN;1
   EDR06134.LAN;1	EDR06134.RAW;1	ERR94001_02.LOG;2	ERR94001_02.LOG;1
   FTP_LAN.COM;13	FTP_LAN.COM;12	FTP_LAN.COM;11	FTP_LAN.COM;10
   LECP.TXT;1	OCTOBER-RUN.LOG;1	README.TXT;2	SEND.COM;1
   SETHOST.LOG;2	SETHOST.LOG;1	TEMP.DIR;1

    

   Total of 39 files..

 

5. Next, the EDRYYDDD.RAW must be renamed to EDRYYDDD_02.00 so it can be processed by ARCGEN.EXE. Rename the file as follows, and then do a directory listing to ensure that the file is renamed correctly.

    

   $rename edr06134.raw edr06134_02.00
   $dir

    

   AAA_README.TXT;1	ALPHA_COMPILE.COM;3	CDCOPY.EXE;5	CDCOPY.FOR;11
   CD_TO_VAX.COM;1	EDR06001.LAN;1	EDR06008.LAN;1	EDR06015.LAN;1
   EDR06022.LAN;1	EDR06029.LAN;1	EDR06036.LAN;1	EDR06043.LAN;1
   EDR06050.LAN;1	EDR06057.LAN;1	EDR06064.LAN;1	EDR06071.LAN;1
   EDR06078.LAN;1	EDR06085.LAN;1	EDR06092.LAN;1	EDR06099.LAN;1
   EDR06106.LAN;1	EDR06113.LAN;1	EDR06120.LAN;1	EDR06127.LAN;1
   EDR06134.LAN;1	EDR06134_02.00;1	ERR94001_02.LOG;2	ERR94001_02.LOG;1
   FTP_LAN.COM;13	FTP_LAN.COM;12	FTP_LAN.COM;11	FTP_LAN.COM;10
   LECP.TXT;1	OCTOBER-RUN.LOG;1	README.TXT;2	SEND.COM;1
   SETHOST.LOG;2	SETHOST.LOG;1	TEMP.DIR;1

    

   Total of 39 files.

 

6. Now the data file is prepared to be processed by ARCGEN.EXE, which will generate seven days worth of ULAYYDDD.RAT and ULAYYDDD.PHA from the data provided in that file. The output directory for these files is KUEPAM$DKB200:[ULYSSES.PRIMARY.WRK0.ULA]. To ensure that processing will proceed correctly, make sure the IULAYYDDD.RAT and IULAYYDDD.PHA files exist for the latest data (last day) of the preceding EDRYYDDD.LAN file. ARCGEN.EXE will automatically add to these files, making them into complete ULA*.* files. Below is an example run-through of ARCGEN.EXE.

    

   $go_arcgen

    

   ------------------------------ GO_ARCGEN --------------------------------

    

   Enter EDR directory (default is KUEPAM$DKB200:[ULYSSES.SECONDARY.WRK0.EDR] ) ><cr>>

    

   EDR06127_02.00 was last processed on 18-MAY-2006 07:36:07.55

    

   Directory KUEPAM$DKB200:[ULYSSES.SECONDARY.WRK0.EDR]

    

   EDR06134_02.00;1

    

   Total of 1 file.

    

   re these the EDR files that you want to process (y/n) >> y
   When do you want to run ARCGENN
   (default:now, (5)5:00 pm, or (m)midnight) >> <cr>>
   Enter number of high resolution plots to be createdd
   default:53 channels, or (1) 1 page of selected channels >> <cr>>
   F = "ARCGEN06134_02""
   Job ARCGEN06134_02 (queue MEDIUM, entry 521) started on MEDIUMM

 

7. Set your default directory to [ULYSSES.PRIMARY.WRK0.ULA]. Do a directory listing to ensure that new ULAYYDDD.RAT and ULAYYDDD.PHA files are being created. They will first appear as IULA*.* files, and, as each file becomes a ULA*.* file, new IULA*.* files will be created in daily sequence. Files for seven days should be created. Samples of a beginning and an ending directory listing are given below.

    

   $set def [ulysses.primary.wrk0.ula]
   $dir

    

   Directory KUEPAM$DKB200:[ULYSSES.PRIMARY.WRK0.ULA]

    

   IULA06135.PHA;1	IULA06135.RAT;1	SEND.COM;1	TEMP.DIR;1
   ULA06112.PHA;1	ULA06112.RAT;1	ULA06113.PHA;1	ULA06113.RAT;1
   ULA06114.PHA;1	ULA06114.RAT;1	ULA06115.PHA;1	ULA06115.RAT;1
   ULA06116.PHA;1	ULA06116.RAT;1	ULA06117.PHA;1	ULA06117.RAT;1
   ULA06118.PHA;1	ULA06118.RAT;1	ULA06119.PHA;1	ULA06119.RAT;1
   ULA06120.PHA;1	ULA06120.RAT;1	ULA06121.PHA;1	ULA06121.RAT;1
   ULA06122.PHA;1	ULA06122.RAT;1	ULA06123.PHA;1	ULA06123.RAT;1
   ULA06124.PHA;1	ULA06124.RAT;1	ULA06125.PHA;1	ULA06125.RAT;1
   ULA06126.PHA;1	ULA06126.RAT;1	ULA06127.PHA;1	ULA06127.RAT;1
   ULA06128.PHA;1	ULA06128.RAT;1	ULA06129.PHA;1	ULA06129.RAT;1
   ULA06130.PHA;1	ULA06130.RAT;1	ULA06131.PHA;1	ULA06131.RAT;1
   ULA06132.PHA;1	ULA06132.RAT;1	ULA06133.PHA;1	ULA06133.RAT;1
   ULA06134.PHA;1	ULA06134.RAT;1

    

   Total of 50 files

    

   $dir

    

   Directory KUEPAM$DKB200:[ULYSSES.PRIMARY.WRK0.ULA]

    

   IULA02159.PHA;1	IULA02159.RAT;1	ULA02131.PHA;1	ULA02131.RAT;1
   ULA02132.PHA;1	ULA02132.RAT;1	ULA02133.PHA;1	ULA02133.RAT;1
   ULA02134.PHA;1	ULA02134.RAT;1	ULA02135.PHA;1	ULA02135.RAT;1
   ULA02136.PHA;1	ULA02136.RAT;1	ULA02137.PHA;1	ULA02137.RAT;1
   ULA02138.PHA;1	ULA02138.RAT;1	ULA02139.PHA;1	ULA02139.RAT;1
   ULA02140.PHA;1	ULA02140.RAT;1	ULA02141.PHA;1	ULA02141.RAT;1
   ULA02142.PHA;1	ULA02142.RAT;1	ULA02143.PHA;1	ULA02143.RAT;1
   ULA02144.PHA;1	ULA02144.RAT;1	ULA02145.PHA;1	ULA02145.RAT;1
   ULA02146.PHA;1	ULA02146.RAT;1	ULA02147.PHA;1	ULA02147.RAT;1
   ULA02148.PHA;1	ULA02148.RAT;1	ULA02149.PHA;1	ULA02149.RAT;1
   ULA02150.PHA;1	ULA02150.RAT;1	ULA02151.PHA;1	ULA02151.RAT;1
   ULA02152.PHA;1	ULA02152.RAT;1	ULA02153.PHA;1	ULA02153.RAT;1
   ULA02154.PHA;1	ULA02154.RAT;1	ULA02155.PHA;1	ULA02155.RAT;1
   ULA02156.PHA;1	ULA02156.RAT;1	ULA02157.PHA;1	ULA02157.RAT;1
   ULA02158.PHA;1	ULA02158.RAT;1

   Total of 58 files.

 

8. Once ARCGEN.EXE is complete, a message will appear as follows:

    

   Job ARCGEN06134_02 (queue MEDIUM, entry 261) completed

 

9. After ARCGEN.EXE has finished running, do a directory listing of the files created the day you are processing (first of the *.RAT files, then the *.PHA files), and check the file sizes to ensure that they are within established parameters. Average file sizes range from 18-25K blocks for the *.RAT files (although they may get as high as 30K blocks or so during significant solar flare events), and 1.5-3K blocks for the *.PHA files. Examples of the directory listings, and the commands used to make the listings, are presented below.

    

   $dir/size/since=today *.rat

    

   Directory KUEPAM$DKB200:[ULYSSES.PRIMARY.WRK0.ULA]

    

   IULA06140.RAT;1	21028
   ULA06134.RAT;1	21028
   ULA06135.RAT;1	11127
   ULA06136.RAT;1	18724
   ULA06137.RAT;1	22793
   ULA06138.RAT;1	4510 *unusually small file size
   ULA06139.RAT;1	17891

    

   Total of 7 files, 117101 blocks.

    

   $ dir/size/since=today *.pha

    

   Directory KUEPAM$DKB200:[ULYSSES.PRIMARY.WRK0.ULA]

    

   IULA06140.PHA;1	1874
   ULA06134.PHA;1	1922
   ULA06135.PHA;1	1033
   ULA06136.PHA;1	1730
   ULA06137.PHA;1	2066
   ULA06138.PHA;1	385
   ULA06139.PHA;1	1658

    

   Total of 7 files, 10668 blocks.

 

10. The entire downloading and processing procedures must be repeated to retrieve successive EDRYYDDD.LAN files and process them into daily ULA files until 32 days, counting from the day after the last day of the last shipment. For example, if the last day of data on the last shipment is 06105 (06 being the two digit year, and 105 being the three digit day), then the next shipment should include days 06106-06137.

 

11. Once the required 32 days worth of ULA data is present, rename the ULA*.* files to the [ULYSSES.PRIMARY.LAN.ULA] directory, because this is the input directory for Auto Processing, which is the set of command procedures that will produce the UAFYYDDD.LAN file for this shipment. Leave the IULA*.* files alone, so they are present for future processing. Note: Complete ULA*.* files must be present for at least one day after the last day of the shipment now in processing to ensure the completion of the UAFYYDDD.LAN file. Use the following command to rename the files to the aforementioned directory:

     

    $rename/log ula*.* [ulysses.primary.lan.ula]

     

    After the files are moved (renamed), set default to that directory and check to make sure all of them are present.

     

    $set def [ulysses.primary.lan.ula]

     

    $dir

 

12. Once all the files are present and accounted for, run auto_wake_up.com to begin the auto processing protocol. To do this, utilize the following command:

     

    $@[Ulysses.ulysses.autosys]auto_wake_up

     

    The main bulk of the auto processing will occur overnight. Thus, the data should be ready the next day.

 

13. The next day, set default to [ulysses.secondary.lan.uaf] and check the size of the most recent UAFYYDDD.LAN file created. The file will share the same date as the first file of the current processing run. Its size should be around 37K.

     

    $dir/size *.lan

     

    Directory KUEPAM$DKB200:[ULYSSES.SECONDARY.LAN.UAF]

     

    IUAF06138.LAN;1	1324
    IUAF1D97353.LAN;1	10931
    UAF06010.LAN;1	37105
    UAF06042.LAN;1	37644
    UAF06074.LAN;1	37644
    UAF06106.LAN;1	36027

     

    Total of 6 files, 160675 blocks.
     

14. The data is now ready for shipment. You will need the latest copy of the ICUAFYYDDD.LAN for the current year, located in [ulysses.temp_cuaf], the latest UAFYYDDD.LAN file [ulysses.secondary.lan.uaf], and the 32 days worth of *.RAT and *.PHA files [ULYSSES.PRIMARY.LAN.ULA] needed for this processing run.

 

15. Utilizing WS_FTP, copy the needed files to Purple (C:\data shipments\ulysses) and create a CD hard disk image utilizing Easy CD Creator. The format needed for the image is ISO9660, and must be set to accept ‘long file names (30 characters max)’ (see settings under your cd burning software for both of these options).

 

16. After the hard disk image is saved, burn it to a CD, which will become your master copy. Check this CD on the Linux computer to ensure that the files can be accessed, and the file sizes are normal. (Note: this step is important, as the research teams utilizing this data will be processing it on Linux computers).

 

17. If the disk checks out, go back to your CD burning program and burn seven more CDs using the hard disk image you created. Repeat testing procedure.

 

18. It is now time to label the CDs. Open up the template for the HISCALE CD file in the jewel case creator program. Change the file names and production dates to match your current production run (note: production date reflects the date on which the master cd was produced).

  

19. Package as instructed and ship to the following people:
     

    Olga Malandraki ESTEC/ESA European Space Agency Research and Scientific Support Dept. 2200 AG Noordwijk THE NETHERLANDS global priority mail	Roger Williams Space Sciences Laboratory Univ. of California-Berkeley Berkeley, CA 94720 priority mail
    George Simnett Univ. of Birmingham School of Physics and Astronomy Edgbaston, Birmingham B15 2TT ENGLAND global priority mail	L. J. Lanzerotti Lucent Technologies 600 Mountain Ave. Room 1E-439 Murray Hill, NJ 07974 priority mail
    Monique Pick DASOP Observatoire de Meudon 92195 Meudon FRANCE global priority mail	Ollie Lamm SRS/Space Department Johns Hopkins University Applied Physics Laboratory 11100 Johns Hopkins Rd. Laurel, MD 20723-6099 priority mail
    E. T. Sarris University of Thrace Xanthi 67100 GREECE 10 x 13 manila envelope

     

    Be sure to place CDs in padded mailer before packaging them in the above mentioned envelopes. Include an updated copy of ‘note to HISCALE team.doc’ (located under my computer through H:Plum\Homes\Jbrockman\hiscale shipments). Label envelopes using provided address labels. Place clear packing tape over labels. Secure envelope closures with the same packing tape. Give to Jeanette for shipping.

 

20. Send email to the HISCALE Team Members to let them know that their data is forthcoming. Login into KUEPAM$DKB200:[ULYSSES] and edit the LAN_SHIP.TXT file to reflect the current data being shipped. Follow the example given to edit and send the email.

     

    $ show default

     

    KUEPAM$DKB200:[ULYSSES]

     

    $ edit lan_ship.txt

     

    Attention Hiscale Team Members-

     

    A Ulysses Hiscale/ACE EPAM shipment went out in today's mail. This shipment contains the following CDs:

     

    1. Ulysses Hiscale Data for 2006, day 106, to 2006, day 137
       including an updated copy of icuaf*.lan file for 2006
    2. ACE EPAM data: Disc 100 (2006, day 123, thru 2006, day 154)

    If you have any questions, don't hesitate to call [(785)-840-0800] or email me here at Fundamental Technologies.

    Jeni Eggers
    Fundamental Technologies, LLC

    [End of file]

    Press ctrl Z to exit editor.

    Buffer: LAN_SHIP.TXT                         | Write | Insert | Forward

    15 lines written to file KUEPAM$DKB200:[ULYSSES]LAN_SHIP.TXT;52

    $ mail

    You have 45 new messages.

    MAIL> send lan_ship.txt
    To: @hiteam.dis
    Subj: data shipment

    MAIL> exit

 

21. The monthly data processing and shipment is now complete.

Next: ARCGEN Processing

Return to Chapter 3 Table of Contents

Return to Ulysses HISCALE Data Analysis Handbook Table of Contents