AIRS/AMSU/HSB PROCESSING PACKAGE FOR DIRECT BROADCAST ----------------------------------------------------- Version 4.0 22 September 2005 Liam.Gumley@ssec.wisc.edu 1.0 INTRODUCTION ---------------- This package contains the software needed to process Aqua direct broadcast AIRS, AMSU-A, and HSB(*) data from raw packets in PDS (Production Data Set) format into Level 1B calibrated radiances and Level 2 geophysical products. It uses v4.0.9 of the NASA/JPL AIRS Science Processing System (SPS). Information on the AIRS instrument is available at http://www-airs.jpl.nasa.gov/ and information on AIRS Level 1 and Level 2 products is available at http://disc.gsfc.nasa.gov/AIRS/ Platforms supported in this release are - Red Hat Linux (x86) versions 7, 8, 9, Enterprise - Sun Solaris (SPARC) versions 7, 8, 9 Approximately 25 Gigabytes of disk space are required for the software and associated ancillary data files. This is a binary-only package (no source code). Future releases will include source code, making it possible to port the package to other platforms. (*) HSB ceased operating on 5 February 2003 at 21:50 UTC. 2.0 PACKAGE DETAILS ------------------- This package produces Level 1B and Level 2 output files in HDF-EOS swath format, identical to the GSFC DAAC format. While Aqua passes of arbitrary length may be used as input to the package, output files are always broken into 6 minute granules (the AIRS default). Granules at the start or end of a direct broadcast pass will typically contain some fill values (i.e., missing data). For information on decoding and interpreting AIRS product files, see http://disc.gsfc.nasa.gov/AIRS/documentation/v4_docs/v4_docs_list.shtml The disclaimer at http://disc.gsfc.nasa.gov/AIRS/documentation/v4_docs/V4.0_Data_Disclaimer.pdf describes the liens against this release. 3.0 PACKAGE CONTENTS -------------------- The software package can be downloaded by anonymous FTP from ftp://ftp.ssec.wisc.edu/pub/IMAPP/AIRS/v4.0 Name Size Description ---- ---- ----------- AIRS_linux_v4.tar.gz 27 MB Linux distribution AIRS_sun_v4.tar.gz 31 MB Sun distribution AIRS_store_v4.tar.gz 80 MB Coefficient files AIRS_anc_avhrr.tar 927 MB AVHRR ancillary data AIRS_anc_mod11.tar 1000 MB MODIS ancillary data DEM30ARC.tar 684 MB DEM ancillary data STD30ARC.tar 227 MB DEM support data AIRS_data_input.tar.gz 79 MB Sample input data AIRS_data_output_linux_v4.tar.gz 84 MB Sample output data (Linux) AIRS_data_output_sun_v4.tar.gz 84 MB Sample output data (Sun) No other additional ancillary data or toolkits are required to run the package. Please remember to use binary mode when downloading the files. 4.0 INSTALLATION ---------------- This package requires the following utilities to be installed: - Korn shell (ksh) - Python 1.5 or higher - Perl 5.0 or higher To check if these utilities are installed on your system: $ ls -l /bin/ksh $ python -V $ perl -v (1) Go to the directory where you want the AIRS software to be installed, and then unpack the software and coefficient files, e.g., $ cd $HOME $ gzip -dc AIRS_linux_v4.tar.gz | tar xvf - $ gzip -dc AIRS_store_v4.tar.gz | tar xvf - A new directory named 'AIRS' containing the software package will be created. (2) The AVHRR and MODIS ancillary files must be stored in the AIRS/STORE/ancillary directory, e.g., $ cd AIRS/STORE/ancillary $ tar xvf $HOME/AIRS_anc_avhrr.tar $ gzip -vd AVHRR*.gz $ tar xvf $HOME/AIRS_anc_mod11.tar $ gzip -vd MOD*.gz NOTE: About 16 GB of disk space is required for the uncompressed AVHRR and MODIS files. If you store the AVHRR files in a different location, please create soft links from the AVHRR files to the AIRS/STORE/ancillary directory. (3) Go to the directory where you want the DEM data to be stored, and then unpack the DEM files, e.g., $ cd $HOME/AIRS $ tar xvf $HOME/DEM30ARC.tar $ tar xvf $HOME/STD30ARC.tar $ cd DEM $ gzip -vd *.gz A new directory named 'DEM' containing the DEM files will be created. NOTE: About 10 GB of disk space is required for the uncompressed DEM files. (4) Go to the directory where you want the sample input data to be stored, and then unpack the sample data files, e.g., $ cd $HOME/AIRS $ gzip -dc $HOME/AIRS_input.tar.gz | tar xvf - A new directory named 'data' containing the sample input files will be created. (5) Make a directory to hold runtime log files, e.g., $ cd $HOME/AIRS $ mkdir run (6) Download updated versions of the ancillary data files leapsec.dat and utcpole.dat from ftp://oceans.gsfc.nasa.gov/COMMON/ and copy them to AIRS/toolkit/database/common/TD/leapsec.dat AIRS/toolkit/database/$OS/TD/leapsec.dat AIRS/toolkit/database/common/CSC/utcpole.dat AIRS/toolkit/database/$OS/CSC/utcpole.dat where $OS is your platform name ('linux' or 'sun5') NOTE: The package will crash if the leapsec.dat and utcpole.dat files are out of date. We recommend that you update the leapsec.dat and utcpole.dat files once per week. The files won't change every week, however, this way you will be sure to catch the new versions if and when they are updated. 5.0 INPUT FILES --------------- The package requires AIRS, AMSU, HSB, and GBAD Level 0 PDS files for input, where each file contains time ordered packets for a single Application Process ID (APID). The APIDs required from each instrument are as follows: AMSU => APIDs 261, 262, 290 HSB => APIDs 342 AIRS => APIDs 404, 405, 406, 407, 414, 415 GBAD => APID 957 Therefore eleven separate Level 0 input files are required if all instruments are processed. The HSB file may be omitted if desired. The package expects the input files to have filenames formatted as follows: PsssppppAAAAAAAAAAAAAAyyjjjhhmmss001.PDS where sss is the spacecraft id (154 = Aqua) pppp is the 4-digit APID yy is the last two digits of the year jjj is the day of year hhmmss is the UTC time (hours, minutes, seconds) at the start of the pass NOTE: Only the spacecraft and APID fields are actually used. The other fields must be present, but their values are ignored. For example, the following list of file names is valid for input to the package: P1540261AAAAAAAAAAAAAA00000000000001.PDS P1540262AAAAAAAAAAAAAA00000000000001.PDS P1540290AAAAAAAAAAAAAA00000000000001.PDS P1540342AAAAAAAAAAAAAA00000000000001.PDS P1540404AAAAAAAAAAAAAA00000000000001.PDS P1540405AAAAAAAAAAAAAA00000000000001.PDS P1540406AAAAAAAAAAAAAA00000000000001.PDS P1540407AAAAAAAAAAAAAA00000000000001.PDS P1540414AAAAAAAAAAAAAA00000000000001.PDS P1540415AAAAAAAAAAAAAA00000000000001.PDS P1540957AAAAAAAAAAAAAA00000000000001.PDS If your Level-0 processing creates filenames in a different format, you can simply create soft links to the filenames shown above. The package also requires a construction record file and a metadata file for each input Level 0 PDS file, with names formatted as shown below: PsssppppAAAAAAAAAAAAAAyyjjjhhmmss000.PDS PsssppppAAAAAAAAAAAAAAyyjjjhhmmss001.PDS.met If the construction record and metadata files are not found, they will be created automatically by the AIRS package when needed. This option is recommended even if your system creates the construction record file or metadata files, because it ensures that the construction record and metadata files are created in the format expected by the AIRS package. For near real-time applications, the package uses the GSFC GBAD processor to convert Level 0 files containing APID 957 to Aqua attitude and ephemeris files. The GBAD processor also requires a two-line elements (TLE) file to be available. The latest TLE file is available at the following sites: http://www.celestrak.com/NORAD/elements/resource.txt ftp://modis-dbc.gsfc.nasa.gov/ephemeris/tle/ The default TLE file name expected by the GBAD processor is 'noradfile'. Note that only the orbit number is extracted from the TLE file, so it is only necessary to update the file once per week. The TLE file is *not* used to determine the position or attitude of the Aqua spacecraft. If you wish to use post-processed attitude and ephemeris files (typically these are delayed by 12 hours or more), simply move them to the same directory as the Level 0 PDS input files. Recent attitude and ephemeris files for Aqua are available at ftp://aqua.ssec.wisc.edu/pub/aqua/ephemeris/ and a deeper archive is available at ftp://g0dps01u.ecs.nasa.gov/Ancillary_Other/ 6.0 RUNNING THE SOFTWARE ------------------------ The package is invoked by the following Python script: AIRS/script/AirsDb.py Before this script is called, environment variables must be set to tell the script where the input files, binary exectuables, and ancillary datasets are located. For this purpose, a sample run script is included named Run.csh.linux or Run.csh.sun The Linux version of the script is shown below. The script creates Level 1A, Level 1B, and Level 2 products for one Aqua direct broadcast overpass. #---start of script--- #!/bin/csh -f # Set AIRS DB environment variables setenv AIRSROOT /modisnfs2/MODIS/gumley/AIRS setenv AIRSDB $AIRSROOT setenv AIRSDB_HOME $AIRSROOT setenv ARCH linux setenv ARCH_TOOLKIT linux set path = ($path $AIRSROOT/$ARCH/bin $AIRSDB/script) # Set location of input files setenv AIRSDB_PACKETS $AIRSROOT/data setenv GBAD_DATA $AIRSROOT/data # Set location of runtime files setenv RUN_DIR $AIRSROOT/run # Set location of DEM files setenv DEM_DATA /modisnfs2/MODIS/gumley/AIRS/DEM # Set SDP toolkit environment variables setenv PGSHOME $AIRSROOT/toolkit setenv HDFHOME /dev/null setenv HDFEOS_HOME /dev/null # Run the AIRS processing script cd $RUN_DIR $AIRSDB/script/AirsDb.py -u -l 2 if ($status != 0) then echo "Error: AIRS processing script failed" exit(-1) endif #---end of script--- An explanation of each environment variable follows: AIRSROOT Directory where the AIRS package is installed ARCH Architecture identifier for scripts (linux or SUN) ARCH_TOOLKIT Architecture identifier for toolkit (linux or sun5) AIRSDB_PACKETS Directory for AIRS, AMSU, HSB and GBAD Level-0 PDS files GBAD_DATA Directory for NORAD TLE file ('noradfile') RUN_DIR Directory where the package will create runtime logfiles DEM_DATA Directory containing DEM files Edit the script to set the environment variables appropriately for your system. To run the sample script, simply type $ ./Run.csh.linux or $ ./Run.csh.sun depending on your operating system. By default, all input files found in the AIRSDB_PACKETS directory will be processed. If only a subset of the input set is to be processed, the following command line pptions for the AirsDb.py script may be used: -D YYYY-MM-DD : specify the processed date -G XXX : specify the starting granule (between 1 and 240) -N X : specify the number of granules to be processed. To see the list of options available, use $ $AIRSROOT/script/AirsDb.py -h Note: If you only wish to process the data up to Level 1B, remove the '-l 2' command line option. 7.0 OUTPUT FILES ---------------- Level 1B and Level 2 output files are created in the AIRSDB_PACKETS directory. The Level 1B files containing science data are named AIRS.yyyy.mm.dd.ggg.L1B.AIRS_Rad.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.AMSU_Rad.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.HSB_Rad.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.VIS_Rad.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L2.CC.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L2.RetStd.v4.0.9.102.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L2.RetSup.v4.0.9.102.DYYDDDHHMMSS.hdf where yyyy is the year mm is the month dd is the day of the month ggg is the granule number of the day (there are 240 x 6-minute granules per day) AIRS is the AIRS spectrometer radiances AMSU is the AMSU brightness temperatures HSB is the HSB brightness temperatures VIS is the AIRS visible/near-IR imager radiances YY is the last two digits of the year when file was created DDD is the day of year when the file was created HHMMSS is the UTC hours/minutes/seconds when the file was created All the files are in HDF-EOS format, identical to the GSFC DAAC format. For more information on decoding the files, including links to reader software, see http://daac.gsfc.nasa.gov/atmodyn/airs/ To extract information from the Level 1B output files, a HDF reader is required. A freely available reader is supplied with the HDF4 distribution available at ftp://hdf.ncsa.uiuc.edu/HDF/HDF4.1r5/bin/ After downloading and installing the HDF4 package for your system, there are two utilities you can use to extract data from a HDF file. First, the 'ncdump' utility will list a directory of the contents of a HDF file, e.g., $ ncdump -h AIRS.2003.09.02.175.L1B.AIRS_Rad.v4.0.9.102.D05259171938.hdf | head -20 netcdf AIRS.2003.09.02.175.L1B.AIRS_Rad.v3.0.8.102.D04324135403 { dimensions: GeoTrack_L1B_AIRS_Science = 135 ; GeoXTrack_L1B_AIRS_Science = 90 ; Channel_L1B_AIRS_Science = 2378 ; SpaceXTrack_L1B_AIRS_Science = 4 ; MaxRefChannel_L1B_AIRS_Science = 100 ; variables: double Latitude(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; double Longitude(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; double Time(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float radiances(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science, Channel_L1B_AIRS_Science) ; float scanang(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; long ftptgeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; short zengeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; short demgeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float satzen(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float satazi(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float solzen(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; The 'hdp' utility will extract one variable into a binary file, e.g., $ hdp dumpsds -n radiances -b -o radiances.dat \ AIRS.2003.09.02.175.L1B.AIRS_Rad.v4.0.9.102.D05259171938.hdf extracts a file containing the AIRS spectrometer radiances into a binary file with 2378x90x135 32-bit float values. 8.0 AIRS CHANNEL PROPERTIES FILE -------------------------------- The AIRS team distributes an "AIRS Channel Properties File", separate from the Level 1B HDF files. The channel properties file is an ASCII text file containing ancilary information about the AIRS radiance data. It is recommended as a way to determine - the central frequency for each AIRS channel (column #2) - an indication of which AIRS channels contain good data (column #13) The current version of the file (v8.1.0) is available at http://disc.gsfc.nasa.gov/AIRS/L2.chan_prop.2003.11.19.v8.1.0.txt The current version is valid for all AIRS data collected from 19 November 2003 onwards. If you wish to process data from an earlier date, please obtain the appropriate channel properties file from the site mentioned above. 9.0 TROUBLESHOOTING ------------------- Listed below are some reasons why the package may fail to run correctly: Problem: leapsec.dat and/or utcpole.dat files are out of date. Solution: Update the files from the site mentioned in section 4.0. Problem: Input Level 0 PDS files contain bad packets. Solution: Ensure that Reed-Solomon error correction is enabled when creating the Level 0 PDS files. Problem: DEM files were not found at runtime. Solution: Check that the DEM files were uncompressed correctly in the directory identifed by DEM_DATA. Problem: Output files appear in the directory identified by AIRS_PACKETS. Solution: Edit the run script to move the Level 1B output files (AIRS*.hdf) to a different directory when processing is complete.