Ulysses HISCALE Data Analysis Handbook
Appendix 1. HISCALE I/O Library
(Also see HSIO Version 3.11 (S. J. Tappin, March 1999) for additional updates.)
Document Reference: S. J. Tappin, 22 April 1998
A1.1 Introduction
Welcome to the long-awaited version 3 of the HISCALE i/o system HSIO. The primary purpose of HSIO version 3 is to support the reading of data directly from CD on Unix systems; to do this it has been necessary to introduce an automatic detection of the number-type in the file and build translation into the reading routines.
In addition there is also support for reading gzipped files directly, using the zlib compressed i/o library (compressed files are not supported and the library does not have LZ read or write functions); this is potentially a big space saver for phayyddd.lan files which will compress by a factor of 50-200 depending on activity levels. This ability can be disabled at compile-time if desired. I've also added support for Linux in the library and utilities.
Other minor enhancements include:
- The 128 character length restriction on filenames has been removed.
- The command line for the utilities has been improved---in particular allowing a list of files to be processed.
- The generation of the shareable library for IDL has been moved from the IDL directories to the HSIO directory.
- Some dangerous assumptions about initialization have been removed.
- The source code has been reorganized.
A1.2 Building
and Installing
A1.2.1 Unix & Linux
1. Obtain and unpack the file hsio3.tar.gz (or hsio3.tar-gz if you copy it from Fundamental Technologies). The locations are:
- ftp://ftp.sr.bham.ac.uk/pub/lan/hsio3.tar.gz
- KUEPAM$DKB100:[UKB_LAN]HSIO3.TAR-GZ
2. Edit the Makefile:
(a) Select the options needed for your machine type.
(b) Decide if you wish to incorporate zlib support. If so you may need to build and install Zlib. (zlib-1.0.4 is obtainable from most ftp archive sites, inter alia:
- ftp://ftp.warwick.ac.uk/pub/x11
- ftp://sunsite.unc.edu/pub/packages/TeX/support/ghostscript
On Linux it should come with the distribution.
3. Set the installation directories.
(a) Set any extra compile or link options that are needed. I do not recommend using the -DDEBUG option for an operational system as it can easily produce a screenful of output for each and every record processed.
4. make
This should compile the source and generate the static library and the utilities. If it fails go back to 2 and try again. If there is a real problem with your system, let me know.
5. make install
Will copy the library, the user include files and the utilities to the specified directories.
6. Check the Mk.shr makefile. This is more of a black art---all I know for sure is that:
(a) It compiled and built a supposed shareable object on:.
i. Solaris 2.5 with gcc.
ii. SunOS 4.1.3 with gcc.
iii. Digital Unix with Digital C
iv. Linux (RH 4.2) with gcc
(b) The shareable works with Solaris.
If you are using a different C compiler from the one assumed by RSI in their examples code, you will almost certainly have to change compile and/or link options.
7. make share
This will compile the source, including the interface stubs. Note that because of the different compiler options needed, the object files are deleted before the compile step, so there will always be a compilation.
8. make install.shr
Copies the sharable to the location specified. If the previous step has not been done, this will fail and it won't compile it for you.
After doing these steps you should have a working HSIO system.
A1.2.2 VMS
There are two possibilities for VMS:
Binary Installation.
The save-set KUEPAM$DKB100:[UKB_LAN]HSIO3.BCK contains the binary files needed for HSIO3, namely:
- HS.OLB - the HSIO library
- HSIOSHR.EXE - the HSIO shareable for IDL
- LIBZ.OLB - the zlib library
- HSTRANS.EXE, LANMAP.EXE, HSFIX.EXE, CDFMAP.EXE, CDFMAGMAP.EXE - the various utility programs.
- CSHR.OPT - a linker options file needed to use the shareable C libraries, this will make your PHAGEN, etc., executables much smaller (~100 blocks smaller).
- The HSIO and ZLIB header files needed for building your own C programs using HSIO.
These should be copied to suitable locations. The HSIOSHR.EXE has to go in the [.lib] subdirectory of the idl_hs package, otherwise it's pretty much up to you.
Source Installation.
“Make” software, although available for VMS, is not widely distributed so the tools for building from source are just a basic set of command files to do the compilations and links. You will need tar and gzip to build from source. The executables for these are in my home directory on KUEPAM (KUEPAM$DKB100:[UKB_LAN]).
- Get and unpack hsio3.tar-gz (and zlib-1.0.4.tar.gz).
- (optional) Build zlib, then copy libz.olb, zlib.h and zconf.h to the hsio directory [or remove the HAVE_ZLIB define from all the .com files].
- @make_all
This will compile all the routines and make the library and utilities. You will need C, Fortran and assembler for this.
- Copy the library and executables where you want them.
- Edit set.com to reflect the location of the utilities. Advise users to incorporate it into their login.com files.
- @linkshr
Builds the shareable for IDL. This gave a warning about relocation problems with part of zlib when I did it so I hope it's OK.
- Copy hsioshr.exe to the [.lib] subdirectory of your IDL_HS software.
A1.2.3 Windoze, MacOS, etc.
Forget it.
A1.3 The Utilities
The HSIO utilities are a group of tools to handle HSIO files.
A1.3.1 Hstrans
Hstrans converts an HSIO file into “native” representation. In version 3 it isn't strictly necessary as the read routine can translate on the fly, but it is more efficient to do it once.
Usage:
hstrans [options] file [further files and options]
The available options are:
-1 Process files “in place”, i.e., the output file will replace the input file. This is the default behaviour; the flag is needed to reverse an earlier -2 setting.
-2 Copy mode; filenames are taken in pairs with the first being the input and the second the output.
-c Output the file compressed.
-f Uncompressed output. Default, reverses an earlier -c setting.
-q Quiet operation. No information messages are displayed.
-n Normal level. Brief summary of the actions on each file is given. Default.
-v Verbose. A message is output for each record processed.
Options apply to files after the option on the command line, or until countermanded by an alternative option.
Note that there is no longer an input file specifying the format of the records. This has been built in to the code as it now seems unlikely that there will be any new HSIO record formats.
In “in place” mode if the compression state of the file is changed, then (a) any .gz or -gz suffixes are handled sensibly. (b) If the suffix is changed, then the original file is not removed.
A1.3.2 Lanmap
Lanmap generates a map file which allows display programs to skip to the appropriate place in the file (or at least near to it) without reading all through the file.
Usage:
lanmap list-of-files-and-options
The available options are:
-q Quiet operation. No information messages are displayed.
-n Normal level. Brief summary of the actions on each file is given. Default.
-v Verbose. A message is output for each record processed.
The output map file is the name of the original file with a _map suffix. It is not possible to make a map file for compressed data files as there is no seek possible in a compressed file.
A1.3.3 cdfmap, cdfmagmap
These make map files for CDF data and CDF mag field data respectively. The latter doesn't use HSIO, but has enough in common to be distributed therewith. The usage is exactly the same as lanmap.
A1.3.4 hsfix
This VMS-only routine converts HSIO files to a suitable fixed record length file format. It's really just an automated interface to FIXATR.
Usage:
HSFIX list of files
A1.4 Subroutines
In the descriptions that follow, the call is expressed in a “common” form, e.g.:
open_lanfile(file, mode, index, lrecl[, namelen])
To call from fortran: use call and omit arguments in square brackets, i.e.
CALL OPEN_LANFILE(FILE, MODE, INDEX, LRECL)
From IDL: separate the arguments from the routine name with a comma and omit bracketed args, i.e.
open_lanfile, file, mode, index, lrecl
From C: add an underscore to the routine name, remember that the main arguments are all pointers, the bracketed arguments must be given, however they should be zero, i.e.
open_lanfile_(file, &mode, &index, &lrecl, 0);
(All routines taking string arguments are coded such that a passed length of zero is interpreted as: use the actual length of a null-terminated string. )
A1.4.1 open_lanfile
Opens a file for HSIO operations. For reasons of initialization as well as logic, the first hsio call encountered in a program must be open_lanfile.
Usage:
open_lanfile(file, mode, index, lrecl[, namelen])
Arguments:
file: String, input: The name of the file to be opened.
mode: 4-byte int, input: the opening mode for the file. Allowed values are:
0: Open for reading
10: Open for writing
15: Open for writing compressed file. [This is only
available if zlib support has been compiled into hsio.]
20: Open for appending. N.B. This is only possible for a native, uncompressed file.
index: 4-byte int, output: The file index by which the file will be known for subsequent operations. If there is an error, then this value will be negative.
lrecl: 4-byte int, output: The length of the logical (pseudo-)record of the file. This is the size that should be requested in subsequent reads.
For read and append modes, the byte ordering and compression state is determined by the first 4 bytes of the file. Having done that the platform is determined from the day-of-year field.
A1.4.2 Close_lanfile
Close a file that has previously been opened by open_lanfile.
Usage:
close_lanfile(index, err)
Arguments:
index 4-byte int, input: The file index to close.
err 4-byte int, output: Zero on success or a negative error code.
A1.4.3 Read_lanfile
Read a record from a file opened for hsio use.
Usage:
read_lanfile(index, buffer, nreq, ngot, ierr)
Arguments:
index 4-byte int, input: The index of the file from which to read.
buffer any array, output: The buffer into which to read the data, must be big enough otherwise the results may be unpredictable.
nreq 4-byte int, input: The number of bytes requested (should usually be equal to the lrecl returned by open_lanfile).
ngot 4-byte int, output: How many bytes were actually read, zero if the read failed.
ierr 4-byte int, output: A negative error code on failure, lrecl+8 on success.
A1.4.4 write_lanfile
Write data to a file opened for HSIO.
Usage:
write_lanfile(index, buffer, size, nput, ierr)
Arguments:
index 4-byte int, input: The index of the file to which to write.
buffer any array, input: The buffer from which to write the data.
size 4-byte int, input: The amount of data to write from the buffer.
nput 4-byte int, output: The amount of data actually written.
ierr 4-byte int, output: On failure, a negative error code; on success lrecl+8.
A1.4.5 err_lanfile
Print the last HSIO error message.
Usage:
err_lanfile(identifier[, length])
Argument:
identifier string, input: A string to identify the the calling point, this is prepended to the error message.
A1.4.6 map_lanfile
Associate a map file with a file opened for HSIO usage.
Usage:
map_lanfile(index, name, ierr[, namelen])
Arguments:
index 4-byte int, input: The index of the file to map.
name string, input: The name of the map file. If a null string or a single space is passed, then the default mapfilename of <filename>_map is used. This is highly recommended, indeed in HSIO3, it is the only form that lanmap can generate.
ierr 4-byte int, output: zero or a negative error code.
A1.4.7 seek_lanfile
Set the pointer in a file opened for HSIO that has an associated map file.
Usage:
seek_lanfile(index, year, day, hour, position)
Arguments:
index 4-byte int, input: The index of the file to be positioned.
year,day,hour 4-byte ints, input: The time to be located. If the time is before the first record of the file then the pointer is left at the start of the file; if the time is after any data in the file, then the pointer is left BEFORE the last record of the file.
position 4-byte int, output: The position to which the pointer was set, or a negative error code.
Rename a file. This doesn't really use the HSIO system, it is just provided to give a nearly language & platform independent way of renaming files.
Usage:
rename_lanfile(oldname, newname, ierr[, oldlen, newlen])
Arguments:
oldname string, input: The current name of the file.
newname string, input: The new name.
ierr 4-byte int, output: Unchanged on success, -1 on failure.
A1.4.9 vers_lanfile
Return the HSIO version number.
Usage:
vers_lanfile(version)
Argument:
version float, output: The version number of the HSIO library.
A1.5 Odd Notes
Please note that although HSIO should work on any system using IEEE or VAX number formats, it has only been tested on a limited range of systems. The interface to IDL and fortran has been tested on even less, so if you don't have a Sun running Solaris or a VAX running VMS you may need to fiddle with the settings to get the right lengths and directions for numbers etc.
Return to Ulysses HISCALE Data Analysis Handbook Table of Contents
Updated 8/8/19, Cameron Crane
QUICK FACTS
Mission End Date: June 30, 2009
Destination: The inner heliosphere of the sun away from the ecliptic plane
Orbit: Elliptical orbit transversing the polar regions of the sun outside of the ecliptic plane