AIRS/AMSU/HSB LEVEL 1 PROCESSING PACKAGE FOR DIRECT BROADCAST ------------------------------------------------------------- Version 1.0 31 September 2003 Liam.Gumley@ssec.wisc.edu 1.0 INTRODUCTION ---------------- This package contains the software needed to process Aqua direct broadcast AIRS, AMSU-A, and HSB(*) data from raw packets in PDS (Production Data Set) format into Level 1B calibrated radiances. It uses v2.7.12 of the AIRS Science Processing SYstem (SPS). Information on the AIRS instrument is available at http://www-airs.jpl.nasa.gov/ and information on AIRS Level 1 and Level 2 products is available at http://daac.gsfc.nasa.gov/atmodyn/airs/ Platforms supported in this release are - Red Hat Intel Linux 7.3, 8.0, 9.0 - Sun Solaris (SPARC) 7, 8, 9 Approximately 10 Gigabytes of disk space are required for the software and associated ancillary data files. This is a binary-only package (no source code). Future releases will include Level 2 algorithms for creating geophysical products, and will include source code, making it possible to port the package to other platforms. (*) HSB ceased operating on 5 February 2003 at 21:50 UTC. 2.0 PACKAGE DETAILS ------------------- This package produces Level 1B output files in HDF-EOS swath format, identical to the GSFC DAAC format. While Aqua passes of arbitrary length may be used as input to the package, output files are always broken into 6 minute granules (the AIRS default). Output granules at the start or end of a direct broadcast pass will typically contain some fill values. For information on decoding and interpreting AIRS Level 1 files, see http://daac.gsfc.nasa.gov/atmodyn/airs/ The Solaris distribution is version 2.7.12.0. The Linux distribution is v2.7.12.101. This version is modified from v2.7.12.0 with various changes to make the software compile under Linux. The prinicpal change is that v2.7.12.101 does not include the Visible/near-infrared cloud discrimination module, so all cloud-related fields in the Linux products contain fill values. The disclaimer at http://www-airs.jpl.nasa.gov/images_data/level_1b_data_disclaimer.html describes the liens against this release. 3.0 PACKAGE CONTENTS -------------------- The distribution package contains the following files: Name Size (bytes) ---- ---- AIRS_LINUX.tar.gz 25482057 AIRS_SUN.tar.gz 24879094 AIRS_input.tar.gz 54353761 AIRS_output_LINUX.tar.gz 179934789 AIRS_output_SUN.tar.gz 179934482 DEM30ARC.tar 717107200 STD30ARC.tar 237957120 AIRS_LINUX.tar.gz contains the executable code, scripts, and coefficient files for Linux. This file is required for Linux platforms. AIRS_SUN.tar.gz contains the executable code, scripts, and coefficient files for Sun. This file is required for Sun platforms. AIRS_input.tar.gz contains a sample input data set (from SSEC direct broadcast). This file is required for both platforms to verify the installation of the package. AIRS_output_LINUX.tar.gz contains a sample output data set for Linux. This file is optional. AIRS_output_SUN.tar.gz contains a sample output data set for Sun, without cloud discrimination. This file is optional. DEM30ARC.tar and STD30ARC.tar contain digital elevation model data in HDF format. Both these files are required for all platforms. Note that the DEM files will occupy approximately 10 Gigabytes of disk space when uncompressed. The following two files are required if you wish to run the cloud discrimination module, which is only available on Sun platforms in this release of the package. Name Size (bytes) ---- ---- AIRS_anc_AVHRR_NDVI_2.7.12.0.tar.Z 840194437 AIRS_output_SUN_cloud.tar.gz 179189888 AIRS_anc_AVHRR_NDVI_2.7.12.0.tar.Z contains ancillary data required to run the cloud discrimination algorithm. AIRS_output_SUN.tar.gz contains a sample output data set for Sun with the cloud discrimination algorithm results included. NOTE: The cloud/clear algorithm will be available in a future release of the Linux version of the package. No other additional ancillary data or toolkits are required to run the package. 4.0 INSTALLATION ---------------- This package requires the following utilities to be installed: - Korn shell (ksh) - Python 1.5 or higher - Perl 5.0 or higher To check if these utilities are installed on your system: $ ls -l /bin/ksh $ python -V $ perl -v The AIRS package is distributed via anonymous FTP at ftp.ssec.wisc.edu in the directory named pub/gumley/IMAPP/AIRS The username is 'anonymous', and you may use any password. (1) Download the required files for your platform in BINARY mode. (2) Go to the directory where you want the AIRS software to be installed, and then unpack the software package, e.g., $ cd $HOME $ gzip -dc AIRS_LINUX.tar.gz | tar xvf - A new directory named 'AIRS' containing the software package will be created. (3) Go to the directory where you want the DEM data to be stored, and then unpack the DEM files, e.g., $ mkdir /data1/AIRS $ cd /data1/AIRS $ tar xvf $HOME/DEM30ARC.tar $ tar xvf $HOME/STD30ARC.tar $ gzip -vd * NOTE: About 10 GB of disk space is required for the uncompressed DEM files. A new directory named 'DEM' containing the DEM files will be created. Note that the DEM files do *not* need to be stored in the same directory as the software. (4) Go to the directory where you want the sample input data to be stored, and then unpack the sample data files, e.g., $ cd /data1/AIRS $ gzip -dc $HOME/AIRS_input.tar.gz | tar xvf - A new directory named 'input' containing the sample input data files will be created. (5) Download updated versions of the ancillary data files leapsec.dat and utcpole.dat from ftp://daac.gsfc.nasa.gov/pub/data/dbs/ancillary/ and store them in the following directories AIRS/toolkit/database/common/TD/leapsec.dat AIRS/toolkit/database/$OS/TD/leapsec.dat AIRS/toolkit/database/common/CSC/utcpole.dat AIRS/toolkit/database/$OS/CSC/utcpole.dat where $OS is your platform name ('linux' or 'SUN') NOTE: The package will crash if the leapsec.dat and utcpole.dat files are out of date. We recommend that you update the leapsec.dat and utcpole.dat files once per week. The files won't change every week, however this way you will be sure to catch the new versions if and when they are updated. (6) If you are running the Sun Solaris version of the package, and you wish to enable the cloud discrimination algorithm, you must unpack the required ancillary files in the AIRS/STORE/ancillary directory, e.g., $ cd $HOME/AIRS/store/ancillary $ gzip -dc $HOME/AIRS_anc_AVHRR_NDVI_2.7.12.0.tar.Z | tar xvf - 5.0 INPUT FILES --------------- The package requires AIRS, AMSU, HSB, and GBAD Level 0 PDS files for input, where each file contains time ordered packets for a single Application Process ID (APID). The APIDs required from each instrument are as follows: AMSU => APIDs 261, 262, 290 HSB => APIDs 342 AIRS => APIDs 404, 405, 406, 407, 414, 415 GBAD => APID 957 Therefore eleven separate Level 0 input files are required if all instruments are processed. The HSB file may be omitted if desired. The package expects the input files to have filenames formatted as follows: PsssppppAAAAAAAAAAAAAAyyjjjhhmmss001.PDS where sss is the spacecraft id (154 = Aqua) pppp is the 4-digit APID yy is the last two digits of the year jjj is the day of year hhmmss is the UTC time (hours, minutes, seconds) at the start of the pass NOTE: Only the spacecraft and APID fields are actually used. The other fields must be present, but their values are ignored. For example, the following list of file names is valid for input to the package: P1540261AAAAAAAAAAAAAA00000000000001.PDS P1540262AAAAAAAAAAAAAA00000000000001.PDS P1540290AAAAAAAAAAAAAA00000000000001.PDS P1540342AAAAAAAAAAAAAA00000000000001.PDS P1540404AAAAAAAAAAAAAA00000000000001.PDS P1540405AAAAAAAAAAAAAA00000000000001.PDS P1540406AAAAAAAAAAAAAA00000000000001.PDS P1540407AAAAAAAAAAAAAA00000000000001.PDS P1540414AAAAAAAAAAAAAA00000000000001.PDS P1540415AAAAAAAAAAAAAA00000000000001.PDS P1540957AAAAAAAAAAAAAA00000000000001.PDS If your Level-0 processing creates filenames in a different format, you can simply create soft links to the filenames shown above. The package also requires a construction record file and a metadata file for each input Level 0 PDS file, with names formatted as shown below: PsssppppAAAAAAAAAAAAAAyyjjjhhmmss000.PDS PsssppppAAAAAAAAAAAAAAyyjjjhhmmss001.PDS.met If the construction record and metadata files are not found, they will be created automatically by the AIRS package when needed. This option is recommended even if your system creates the construction record file or metadata files, because it ensures that the construction record and metadata files are created in the format expected by the AIRS package. For near real-time applications, the package uses the GSFC GBAD processor to convert Level 0 files containing APID 957 to Aqua attitude and ephemeris files. The GBAD processor also requires a NORAD two-line elements (TLE) file to be available. The latest NORAD TLE file is available at http://www.celestrak.com/NORAD/elements/resource.txt The default TLE file name expected by the GBAD processor is 'noradfile'. Note that only the orbit number is extracted from the TLE file, so it is only necessary to update the file once per week. The TLE file is *not* used to determine the position or attitude of the Aqua spacecraft. If you wish to use post-processed attitude and ephemeris files (typically these are delayed by 12 hours or more), simply move them to the same directory as the Level 0 PDS input files. Recent attitude and ephemeris files for Aqua are available at ftp://aqua.ssec.wisc.edu/pub/aqua/ephemeris/ and a deeper archive is available at ftp://g0dps01u.ecs.nasa.gov/Ancillary_Other/ 6.0 RUNNING THE SOFTWARE ------------------------ The package is invoked by the following Python script: AIRS/script/AirsDb.py Before this script is called, environment variables must be set to tell the script where the input files, binary exectuables, and ancillary datasets are located. For this purpose, a sample run script is included named Run.csh.linux or Run.csh.sun The Linux version of the script is shown below: ---start of script--- #!/bin/csh -f # Set location of input files setenv AIRSDB_PACKETS /data1/gumley/AIRS/input setenv GBAD_DATA /data1/gumley/AIRS/input # Set location of output files setenv RUN_DIR /data1/gumley/AIRS/output # Set AIRS DB environment variables setenv AIRSROOT /home/gumley/AIRS setenv AIRSDB $AIRSROOT setenv AIRSDB_HOME $AIRSROOT setenv ARCH linux setenv ARCH_TOOLKIT linux set path = ($path $AIRSROOT/$ARCH/bin $AIRSDB/script) # Set location of DEM files setenv DEM_DATA /data1/gumley/AIRS_geodata # Set SDP toolkit environment variables setenv PGSHOME /home/gumley/AIRS/toolkit setenv HDFHOME /dev/null setenv HDFEOS_HOME /dev/null # Run the AIRS processing script cd $RUN_DIR $AIRSDB/script/AirsDb.py if ($status != 0) then echo "Error: AIRS processing script failed" exit(-1) endif ---end of script--- An explanation of each environment variable follows: AIRSDB_PACKETS Directory for AIRS, AMSU, HSB and GBAD Level-0 PDS files GBAD_DATA Directory for NORAD TLE file ('noradfile') RUN_DIR Directory where the package will create runtime files AIRSROOT Directory where the AIRS package is installed ARCH Architecture identifier for scripts (linux or SUN) ARCH_TOOLKIT Architecture identifier for toolkit (linux or sun5) DEM_DATA Directory containing DEM files PGSHOME Directory containing 'toolkit' subdirectory Edit the script to set the environment variables appropriately for your system. To run the sample script, simply type $ ./Run.csh.linux or $ ./Run.csh.sun depending on your operating system. By default, all input files found in the AIRSDB_PACKETS directory will be processed. If only a subset of the input set is to be processed, the following command line pptions for the AirsDb.py script may be used: -D YYYY-MM-DD : specify the processed date -G XXX : specify the starting granule (between 1 and 240) -N X : specify the number of granules to be processed. To see the list of options available, use $ AIRS/AirsDb.py -h 7.0 OUTPUT FILES ---------------- Level 1B output files are created in the AIRSDB_PACKETS directory. The files containing science data are named AIRS.yyyy.mm.dd.ggg.L1B.AIRS_Rad.v2.7.12.0.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.AMSU_Rad.v2.7.12.0.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.HSB_Rad.v2.7.12.0.DYYDDDHHMMSS.hdf AIRS.yyyy.mm.dd.ggg.L1B.VIS_Rad.v2.7.12.0.DYYDDDHHMMSS.hdf where yyyy is the year mm is the month dd is the day of the month ggg is the granule number of the day (there are 240 x 6-minute granules per day) AIRS is the AIRS spectrometer radiances AMSU is the AMSU brightness temperatures HSB is the HSB brightness temperatures VIS is the AIRS visible/near-IR imager radiances YY is the last two digits of the year when file was created DDD is the day of year when the file was created HHMMSS is the UTC hours/minutes/seconds when the file was created All the files are in HDF-EOS format, identical to the GSFC DAAC format. For more information on decoding the files, including links to reader software, see http://daac.gsfc.nasa.gov/atmodyn/airs/ To extract information from the Level 1B output files, a HDF reader is required. A freely available reader is supplied with the HDF4 distribution available at ftp://hdf.ncsa.uiuc.edu/HDF/HDF4.1r5/bin/ After downloading and installing the HDF4 package for your system, there are two utilities you can use to extract data from a HDF file. First, the 'ncdump' utility will list a directory of the contents of a HDF file, e.g., $ set AIRSFILE = AIRS.2003.03.12.205.L1B.AIRS_Rad.v2.7.12.102.D03259152507.hdf $ ncdump -h $AIRSFILE | head -20 netcdf AIRS.2003.03.12.205.L1B.AIRS_Rad.v2.7.12.102.D03259152507 { dimensions: GeoTrack_L1B_AIRS_Science = 135 ; GeoXTrack_L1B_AIRS_Science = 90 ; Channel_L1B_AIRS_Science = 2378 ; SpaceXTrack_L1B_AIRS_Science = 4 ; MaxRefChannel_L1B_AIRS_Science = 100 ; variables: double Latitude(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; double Longitude(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; double Time(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float radiances(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science, Channel_L1B_AIRS_Science) ; float scanang(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; long ftptgeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; short zengeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; short demgeoqa(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float satzen(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float satazi(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; float solzen(GeoTrack_L1B_AIRS_Science, GeoXTrack_L1B_AIRS_Science) ; The 'hdp' utility will extract one variable into a binary file, e.g., $ hdp dumpsds -n radiances -b -o radiances.dat $AIRSFILE extracts a file containing the AIRS spectrometer radiances into a binary file with 2378x90x135 32-bit float values. The Sun version of the AIRS Level 1 Processing Package runs a cloud discrimination algorithm using the visible and near-infrared channels of AIRS. Within the AIRS Level 1 radiance product (AIRS.*.AIRS_Rad.*.hdf), the following variables contain the clear/cloud flag information: PrelimCldQA PrelimCldPrcVis PrelimCldPrcVisErr PrelimCldMapVis The Linux version of the package does not have this capability, so these variables will contain missing data on Linux platforms. The cloud discrimination algorithm will be added in a future release of the package. 8.0 AIRS CHANNEL PROPERTIES FILE -------------------------------- The AIRS team distributes an "AIRS Channel Properties File", separate from the Level 1B HDF files. The channel properties file is an ASCII text file containing ancilary information about the AIRS radiance data. It is recommended as a way to determine - the central frequency for each AIRS channel (column #2) - an indication of which AIRS channels contain good data (column #13) The current version of the file (v6.6.7) is available at http://daac.gsfc.nasa.gov/atmodyn/airs/airs_chan_prop.html The current version is valid for all AIRS data collected from January 10, 2003 onwards. If you wish to process data from an earlier date, please obtain the appropriate channel properties file from the site mentioned above. 9.0 TROUBLESHOOTING ------------------- Listed below are some reasons why the package may fail to run correctly: Problem: leapsec.dat and/or utcpole.dat files are out of date. Solution: Update the files from the site mentioned in section 4.0. Problem: Input Level 0 PDS files contain bad packets. Solution: Ensure that Reed-Solomon error correction is enabled when creating the Level 0 PDS files. Problem: DEM files were not found at runtime. Solution: Check that the DEM files were uncompressed correctly in the directory identifed by DEM_DATA. Problem: Output files appear in the directory identified by AIRS_PACKETS. Solution: Edit the run script to move the Level 1B output files (AIRS*.hdf) to a different directory when processing is complete.