MODIS LEVEL 2 PRODUCT SOFTWARE PACKAGE FOR DIRECT BROADCAST ----------------------------------------------------------- Version 2.0 28 August 2007 Kathy.Strabala@ssec.wisc.edu This package contains the software needed to process Aqua and Terra Level 1B direct broadcast (DB) data into Level 2 MODIS cloudmask (MOD35), cloudtop top properties and cloud phase (a portion of MOD06), atmospheric profiles (MOD07), aerosol (MOD04), sea surface temperatures and near infrared total column water vapor in binary and HDF formats. This software has been completely repackaged to run as one inline set of executables called from one script. A package containing software which will create images of key product parameters from the Level 2 HDF output files is also available for downloading. These routines are McIDAS-Lite based and are called from a shell script to automatically create the image files (.GIF and .JPG formats) within the main processing script of MODIS level 2 products (modis_level2.csh). The McIDAS-Lite that has been packaged for IMAPP requires approval from SSEC prior to any repackaging or redistribution (contact the McIDAS Help Desk via the email form at http://www.ssec.wisc.edu/mcidas/misc/contact.html?mclite). For more information about the Man computer Interactive Data Access System (McIDAS) and McIDAS-Lite, please refer to: http://www.ssec.wisc.edu/mcidas/ and http://www.ssec.wisc.edu/mcidas/software/mclite/ The steps in the creation of the products are: 1) Run flatfile extractors on input MODIS L1B and Geolocation data sets to produce binary files which are used as input to level 2 processing scripts. 2) Run individual product scripts which include identifying the matching date/time ancillary data sets and fetching them from a remote ftp site if necessary. These are driven by the modis_level2.csh script located within the imapp_modisl2/scripts directory. If there are products that you do no wish to create, you can simply change the names of the executables in the imapp_modisl2/bin directory. For example, if you do not wish to create the sst product, you can rename the imapp_modisl2/bin/sst.exe file and the other products will still be created. NOTE: Most MODIS products require the modis cloudmask as an input. Please do not rename this executable (cloudmask.exe). DOCUMENTATION ------------- Individual README files for each separate software module can be found in the imapp_modisl2/src directories. These provide more information on each algorithm and how you would execute each stand alone executable. Detailed output file format information can be found for each separate product in the imapp_modisl2/doc directory. QUICK GUIDE TO RUNNING THE IMAPP MODIS LEVEL 2 PRODUCT SOFTWARE --------------------------------------------------------------- More detailed instructions are available following this section. The software is supported on the following Linux operating systems using the g77 gnu (gcc 3.2.1) compiler: Red Hat Linux x86 7.2, Red Hat Enterprise Linux 3 x86_64 and Linux CentOS 4.2 x86_64 on Sun Operton platforms 1) Go to th IMAPP web site http://cimss.ssec.wisc.edu/imapp and download and unpack the required IMAPP_MODISL2.tar.gz file In this example, we unpack the files in $HOME cd $HOME gzip -dc IMAPP_MODISL2.tar.gz | tar xvf - 2) OPTIONAL: Download the optional input/output test data set and unpack. In this example case, we will unpack the data in the /imapp_modisl2/input_data directory. mv IMAPP_MODISL2_input_output.tar.gz $HOME/imapp_modisl2/input_data cd $HOME/imapp_modisl2/input_data gzip -dc IMAPP_MODISL2_input_output.tar.gz | tar xvf - Required inputs are MODIS L1B Aqua or Terra 1 km, 500m, 250m and geolocation files. The software will process DAAC or direct broadcast L1B files, but the automated scripts require IMAPP naming conventions (a1 or t1 1000m.hdf, 500m.hdf, 250m.hdf, geo.hdf). One set is included in the input_output test data set package. All files should be placed in the same directory. It should be noted that the original IMAPP MODIS formatted L1B files cannot be used with the automated scripts for this release. You must use the SeaDAS modisl1db (http://oceancolor.gsfc.nasa.gov/seadas/modisl1db/) output HDF L1B direct broadcast files or the DAAC HDF MODIS L1B files as input. 3) OPTIONAL: Download the optional MODIS Level 2 product imaging software that allows you to automatically create product browse images. You will be required to sign a non-distribution agreement prior to downloading. Unpack the software in the imapp_modisl2 directory. mv McLITE-linux-imapp-1.5.tar.gz $HOME/imapp_modisl2 cd $HOME/imapp_modisl2 gzip -dc McLITE-linux-imapp-1.5.tar.gz | tar xvf - 4) Set the MODIS_L2_HOME environmental variable in the script that matches your shell. These variable setting scripts are located in the imapp_modisl2/env directory (imapp_modisl2.bash_env or imapp_modisl2.csh_env) for bash and csh users (bash_env should work for ksh users too). Source the file after you edit it. This will set the environmental variables. In our example, for a bash shell user, we would edit the imapp_modisl2.bash_env script as: export MODIS_L2_HOME=$HOME/imapp_modisl2 save the file and then type: source imapp_modisl2.bash_env This will allow access of the imapp_modisl2 scripts and executables through your $PATH. 5) Create MODIS Level 2 products by running the main processing script from any directory: modis_level2.csh where: is the satellite platform name (aqua or terra). is the path and name of the MODIS L1B 1km hdf file. is the name of the directory you want the output data to be placed in. So for our example, we choose to place the output files into the imapp_modisl2/products directory. We would type: modis_level2.csh aqua $MODIS_L2_HOME/input_data/a1.07066.2038.1000m.hdf $MODIS_L2_HOME/products A temporary directory will be created in the current directory where all intermediate files are created and execution takes place. In our example, this directory name will be: 2007_066_Aqua_2038_hhmmss, where hhmmss is the hh-hour mm-minute ss-second that the package was run. This directory can be automatically deleted during runtime by editing the imapp_modisl2/scripts/modis_level2.csh script and removing the "#" in these lines: 262 #/bin/rm * 265 #cd $STARTDIR 266 #/bin/rmdir $TMPDIR 6) Since the default OUTPUT_TYPE variable in the env scripts is set to 2, only HDF files will be copied into the /products directory. Set OUTPUT_TYPE to 1 if you want binary files only (.img and .hdr files) and 3 if you would like both binary and HDF files copied to the output directory. The product imaging software (McIDAS-Lite) will create Browse images only from the HDF output products. 7) If you are using the imaging software, the images will be placed in the $MODIS_L2_HOME/images directory. This can be set in the environmental scripts also. 8) Required ancillary data files are fetched automatically from the IMAPP ancillary ftp site. You can set the place to look for ancillary data locally and where to look for it externally in the env scripts. In our test case, it will fetch the data from the the IMAPP ftp site and download it into the /imapp_modisl2/ancillary directory (these are the env script defaults). The ancillary data directories that are created upon download are not deleted upon completion. You must delete the directories and files within them if you do not wish to archive them. 9) If you get an error message, then you may need to compile the software for your platform/OS combination. You will also need the HDF4 libraries for the compilation. Please see the instructions below for more information. 1.0 INTRODUCTION ---------------- System requirments: (1) The software is supported on the following Linux operating systems using the g77 gnu (gcc 3.2.1) compiler: Red Hat Linux x86 7.2, Red Hat Enterprise Linux 3 x86_64 and Linux CentOS 4.2 x86_64 on Sun Operton platforms The binaries have been statically linked, and should work on most Linux OS/platforms. (2) This release requires ~2.7 GB of disk space for the software, the supporting ancillary data files and the input/output test data sets. (3) Execution of the software will require ~500MB of memory. Software requirements: (1) All software required to execute the IMAPP MODIS L2 science product software and ancillary data fetching scripts are included with this release. Required binaries include hdp, ncftpget, ncdump, wgrib and a date/time utility dateplus.exe. All of these are located within the imapp_modisl2/bin directory. All have been compiled statically and tested on the supported platforms. If you get an error message when executing hdp or ncdump, these utilities are a part of the hdf distribution, and pre-compiled binaries can be acquired from: ftp://ftp.ncsa.uiuc.edu/HDF/HDF/HDF_Current/bin The ncftpget utility is also available for different platforms via download from this web site: http://www.ncftp.com/ncftp/ Finally, wgrib, the utility to unpack fields from the ancillary GRIB files can be acquired from http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html (2) Two configuration scripts which set the correct environmental variables for your computer platforms/OS are included in the imapp_modisl2/env directory. Both bash and csh shells are supported. The path to the main directory (IMAPP_L2_HOME) must be set prior to the execution of the run scripts. (3) Statically pre-compiled versions of the science product executables are included within the imapp_modisl2/bin directory. A script to manually compile the executables is also included (imapp_modisl2/scripts/make_imapp_executables.sh). The script will attempt to compile all of the source code located within the imapp_modisl2/src directory and copy the executables into the bin directory. Individual make files can also be used to compile the code. These are located within the main source directory for each program. You will need to set the HDF_HOME directory in most of the make files before execution of the make files or make script. (4) If you are going to compile any of the software in this package you will need to install the HDF4.2r1 libraries on your system. HDF4.2r1 can be obtained in pre-compiled binary form from: http://www.hdfgroup.org/release4/obtain.html Make sure you get the SZIP, ZLIB and JPEG libraries also, and place all of the library and include files into the main hdf/lib and hdf/inc directories. Input data requirements: (1) MODIS Level 1B data All of the IMAPP MODIS Level 2 products require the MODIS 1km Level 1B and geolocation files as input. The MODIS cloudmask and aerosol products use the 250 m file as input and the aerosol product also requires the MODIS 500 m file as input. Although the software can work on the DAAC L1B files with the MYD and MOD naming conventions, the automated processing scripts will only recognize the t1 and a1 direct broadcast naming conventions. If you wish to use these scripts to create MODIS Level 2 IMAPP products, then copy or link the files to the standard db file naming conventions. For example, the standard DAAC L1B 1km files are named: (Prefix MYD-aqua, MOD-terra yyyy - year ddd - Julian Day hh-hour mm-minute) MYD021KM.Ayyyyddd.hhmm.xxxxxxxxxxxxx.hdf 1km MYD02HKM.Ayyyyddd.hhmm.xxxxxxxxxxxxx.hdf 500m MYD02QKM.Ayyyyddd.hhmm.xxxxxxxxxxxxx.hdf 250m MOD03.Ayyyyddd.hhmm.xxxxxxxxxxxxx.hdf geolocation and the standard IMAPP Level 1B files are named: (Prefix a1-aqua, t1-terra yy - year last 2 digits ddd - Julian Day hh-hour mm-minute) a1.yyddd.hhmm.1000m.hdf 1km a1.yyddd.hhmm.500m.hdf 500m a1.yyddd.hhmm.250m.hdf 250m a1.yyddd.hhmm.geo.hdf geolocation (2) Ancillary data Static ancillary data files are located within the imapp_modisl2/coeff directory. They are linked to this directory from the /src directories. Dynamic ancillary data files are now identified and fetched automatically within the product run scripts. Please see the ftp://ftp.ssec.wisc.edu/pub/kathys/ancillary/IMAPP_L2_ANCILLARY.txt file for more information on ancillary data. Remote and local ancillary data locations must be set prior to MODIS L2 script execution. These are set within .bash and .csh environmental variable scripts located in the imapp_modisl2/env directory. 2.0 PACKAGE DETAILS ------------------- This package produces Level 2 MODIS products in binary and HDF format similar to but not identical to the official MODIS HDF products created by the Goddard Space Flight Center (GSFC) DAAC. The main script included with this release will run all of the executables in sequence and copy the output files to the destination directory specified in an argument to the main run script, modis_level2.csh. The IMAPP_MODISL2 package (not the IMAPP McLITE imaging software) is distributed under the GNU GENERAL PUBLIC LICENSE agreement included at this site. We include the NO WARRANTY section of the LICENSE below: NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 3.0 PACKAGE CONTENTS -------------------- The distribution package consists of the following files: Name Size (bytes) Unpacked Compressed ---- -------- ---------- 00README_MODISL2.txt IMAPP_MODISL2.tar.gz 1.109 GB 381 MB IMAPP_MODISL2_input_output.tar.gz 1.659 GB 867 MB 00README_MODISL2.txt Contains documentation on the IMAPP MODIS direct broadcast Level 2 release. IMAPP_MODISL2.tar.gz contains statically compiled binary executables, source code, run scripts, coefficient files, environmental variable scripts, static ancillary files, configuration files and statically compiled HDF 4.2r1 libraries for Linux platforms. This is a required file. IMAPP_MODISL2_input_output.tar.gz contains sample input/output data files (from UW SSEC direct broadcast). These files are required to verify the installation of the package. NOTE: There are dynamic ancillary data files that are required in order to run the software. They are included in the package but not needed unless you wish to place them in your local ancillary data directory. Scripts which automatically fetch the files from the IMAPP ancillary data archive are provided and execute automatically within the product run scripts. 4.0 INSTALLATION ---------------- The IMAPP_MODISL2 direct broadcast package is distributed via download from the IMAPP web page: http://cimss.ssec.wisc.edu/imapp (1) Download the required files (2) Go to the directory where you want the software to be installed, and then unpack the software package, e.g., $ cd $HOME $ gzip -dc IMAPP_MODISL2.tar.gz | tar xvf - A new directory named 'imapp_modisl2' containing the software package will be created, along with these subdirectories: /ancillary - Default directory for your local ancillary data archive /bin - Contains all binary files used in product generation aerosol.exe - MODIS aerosol retrieval executable cloudmask.exe - Cloud mask executable cloudtop.exe - Cloud Top Properties executable profiles.exe - Atmospheric Profiles executable sst.exe - Sea Surface Temperatures executable wvnir.exe - Near Infrared Water Vapor executable create_fake_mod04.exe - Converts binary aerosol files to hdf create_fake_mod06.exe - Converts binary cloudtop files to hdf create_fake_mod07.exe - Converts binary profiles files to hdf dateplus.exe - Date utility extract_ncep_gdas1.csh - Uses wgrib to extract gdas fields from grib file extract_ncep_ice.csh - Uses wgrib to extract global ncep ice concentration fields from grib file extract_nsidc_nise.csh - Extracts global snow and ice cover arrays from NISE global HDFEOS file extract_tovs_ozone.csh - Uses wgrib to extract global total ozone field from grib file hdiff - HDF utility to compare hdf files hdp - HDF utility to dump out values from an hdf file modis_extract_1km.exe - Extracts 1km radiance/reflectance data from an input 1km MODIS hdf file and writes it to an output binary (.img) and header file (.hdr) modis_extract_geo.exe - Extracts geolocation data from an input MODIS geolocation hdf file and writes it to an output binary (.img) and header file (.hdr) modis_extract_hkm.exe - Extracts 500m radiance/reflectance data from an input 500m MODIS hdf file and writes it to an output binary (.img) and header file (.hdr) modis_extract_met.exe - Extracts scan type and mirror side information from an input 1km MODIS hdf file and writes it to an output binary (.img) and header file (.hdr) modis_extract_qkm.exe - Extracts 250m radiance/reflectance data from an input 250m MODIS hdf file and writes it to an output binary (.img) and header file (.hdr) ncdump - HDF utility to dump values from an hdf file ncftpget - Utility to fetch data from an ftp site wgrib - Utility that extracts fields from grib files /coeff - Contains static coefficient files used in product generation. These files are statically linked from the product source directories aerosol_land_map.v2, big_v1c1.dat.v2, big_v2c1.dat.v5, big_v3c1.dat.v2, lookup_land_w0466.v1, lookup_land_w0553.v1, lookup_land_w0644.v1, lookup_land_w2119.v1, mod04ocean.in.v1, small_v1c1.dat.v2, small_v2c1.dat.v3, small_v3c1.dat.v2 - Static coefficient files used by the aerosol algorithm goge1_2_img - Cloud mask global 1km ecosystem file thresholds.dat.Aqua - Contains thresholds for individual Aqua spectral tests determining cloud/clear per pixel thresholds.dat.Terra - Contains thresholds for individual Terra spectral tests determining cloud/clear per pixel MODIS_Aqua_regcoef_2004_06_02_header.hdf - Aqua regresstion coefficient file for atmospheric profiles package MODIS_Terra_regcoef_2004_06_02_header.hdf - Terra regresstion coefficient file for atmospheric profiles package MODIS_senzen.bin - Atmospheric profiles sensor zenith angle file Aqua_bias.dat - Atmospheric profiles Aqua MODIS channel bias file Terra_bias.dat - Atmospheric profiles Terra MODIS channel bias file Aqua_det.dat - Atmospheric profiles Aqua MODIS channel detector file Terra_det.dat - Atmospheric profiles Aqua MODIS channel detector file modisdet.dry.101.v2 - Cloudtop dry gas coefficient file modisdet.ozo.101.v2 - Cloud top ozone coefficient file modisdet.wco.101.v2 - Cloud top wet gas 1 coefficient file modisdet.wtl.101.v2 - Cloud top wet gas 2 coefficient file modisdet.wts.101.v2 - Cloud top water vapor coefficient file /config - Contains individual product default configuration files. These will be filled with input/output file names upon execution cloudmask.cfg - List of inputs/outpus that are filled automatically using the run_modis_cloudmask.csh script. It is an argument to cloudmask.exe aerosol.cfg - List of inputs/outpus that are filled automatically using the run_modis_aerosol.csh script. It is an argument to aerosol.exe cloudtop.cfg - List of inputs/outpus that are filled automatically using the run_modis_cloudtop.csh script. It is an argument to cloudtop.exe profiles.cfg - List of inputs/outpus that are filled automatically using the run_modis_profiles.csh script. It is an argument to profiles.exe sst.cfg - List of inputs/outpus that are filled automatically using the run_modis_sst.csh script. It is an argument to sst.exe wvnir.cfg - List of inputs/outpus that are filled automatically using the run_modis_wvnir.csh script. It is an argument to wnvir.exe /doc - Contains the output product file descriptions Aerosol_Output_Description.txt Cloud_Mask_Bit_Description.txt Cloud_Top_Property_Description.txt Flatfile_Output_Description.txt Profiles_Output_Description.txt SST_Output_Description.txt WVNIR_Output_Description.txt /env - Contains files to source to set environmental variables. (imapp_modisl2.bash_env imapp_modisl2.csh_env) These set the following variables - shown here with the default paths: MODIS_L2_HOME= (set this to the package home directory) HDF_HOME=${MODIS_L2_HOME}/hdf (HDF4 home directory) LOCAL_ANC_DIR=${MODIS_L2_HOME}/ancillary (PATH to local ancillary repository) REMOTE_ANC_DIR=ftp://ftp.ssec.wisc.edu/pub/eosdb/ancillary site of remote ancillary data directory MODIS_L2_CFG=${MODIS_L2_HOME}/config (location of config files) MODIS_L2_COEFF=${MODIS_L2_HOME}/coeff (Static coefficient files) MODIS_L2_BIN=${MODIS_L2_HOME}/bin (Location of executables) MODIS_L2_IMAGES=${MODIS_L2_HOME}/images (path to output images) MODIS_L2_XSIZE=900 (output image size - X direction) (500-2000) MODIS_L2_YSIZE=900 (output image size - Y direction) (500-2000) OUTPUT_TYPE=2 (output product format where: 1 = binary only, 2 = hdf only, 3 = binary and hdf) PATH=.:${MODIS_L2_BIN}:${MODIS_L2_HOME}/scripts:${PATH} (Adds /bin and /script directories to your $PATH) Note: MODIS_L2_IMAGES, MODIS_L2_XSIZE and MODIS_L2_YSIZE are used only with the McIDAS-Lite IMAPP imaging package /hdf - Contains a statically compiled version of the HDF 4.2r1 libraries, include files and binaries for the supported platforms /input_data - empty - you can use this directory to place the L1B input test files /products - empty - the test run script can be used to place the output products here /scripts - Contains the scripts which run the ancillary data fetchers, the main modisl2 script (modis_level2.csh), individual product run scripts (i.e., run_modis_cloudmask.csh) and the script which will execute the make files for all programs in /src (make_imapp_executables.sh) get_anc_gdas_gfs.csh - Fetches matching gdas ancillary file get_anc_icec.csh - Fetches matching ice concentration file get_anc_nise.csh - Fetches matching NISE ancillary file get_anc_rsst_wday.csh - Fetches matching SST ancillary file get_anc_toast_ozone.csh - Fetches matching TOAST file get_anc_tovs_ozone.csh - Fetches matching TOVS ancillary file get_nearest_gdas.csh - Fetches closest matching gdas file make_imapp_executables.sh - Script to compile all source code found in MODIS_L2_HOME/src directory modis_level2.csh - Main product run script run_modis_aerosol.csh - Executes aerosol retrieval. This is called by modis_level2.csh run_modis_cloudmask.csh - Executes cloudmask retrieval. This is called by modis_level2.csh run_modis_flatfile.csh - Executes flatfile extractions. This is called by modis_level2.csh run_modis_cloudtop.csh - Executes cloudtop retrievals. This is called by modis_level2.csh run_modis_profiles.csh - Executes atmospheric profiles retrievals. This is called by modis_level2.csh run_modis_sst.csh - Executes sea surface temperature retrieval. This is called by modis_level2.csh run_modis_wvnir.csh - Executes near-infrared water vapor retrieval. This is called by modis_level2.csh make_mclite_images.sh - Script which creates product images automatically using McIDAS-Lite software /src - Contains the source code, include files, static ancillary files and make files for each MODIS product Source code included in this release: flatfile - Software to extract MODIS L1B data into binary flatfiles, the native IMAPP format used as input to science data product software cloudmask - Collect 5 MODIS cloudmask software (MOD35) aerosol - Collect 5 MODIS aerosol MOD04 product software. Uses MODIS cloud mask binary files as inputs. cloudtop - Collect 5.1 MODIS cloudtop software (MODCT06) profiles - Collect 5 MODIS profiles software (MOD07) sst - UW SSEC Sea Surface Temperature software wnvir - UW SSEC Near infrared water vapor software dateplus - date utility - used by product run scripts (3) Set your environmental variables needed for executing the run scripts correctly. Edit the files found in the /env file that matches your shell (either bash or csh). Set the top level MODIS_L2_HOME directory in the file along with any changes to the LOCAL_ANC_DIR (the directory on your local machine that you would like to check for ancillary data) and the REMOTE_ANC_DIR (the place to look for the ancillary data remotely) and then source it. The default location for the remote ancillary site is the IMAPP ftp site: ftp://ftp.ssec.wisc.edu/pub/eosdb/ancillary It should be noted that the automated scripts will look for ancillary data in a directory structure that is similar to that found on the IMAPP ancillary data ftp site, i.e. LOCAL_ANC_DIR/yyyy_mm_dd_ddd . You will also need to set the HDF_HOME directory if you intend to compile the software manually on a non-supported platform/OS. For example, to set the environmental variables for the bash shell, type: source imapp_modisl2.bash_env Sourcing this script will set those environmental variables and add the /bin and /scripts directories to your PATH. You can add the source command to your profile so that it will be executed upon log in. (4) If you are not going to use the pre-compiled binaries located in the /bin directory, you will need to compile the code manually. If you wish to compile the code yourself, cd into the individual product src or cmd directory and type "make" and copy the executable into the /bin directory. You may need to set the $HDF_HOME environmental variable first. This can be done manually, within the make file, or through sourcing one of the environmental variable scripts (see (3) above). A version of the hdf libraries are included with this release. This is set as the default HDF_HOME directory. It was compiled with gcc 3.2.1. It may or may not be compatible with the gcc libraries on your local machine. You can also execute the /scripts/make_imapp_executables.sh script which will attempt to compile all the source code by using all the make files found within the /src directory. You will still need to set the HDF_HOME and MODIS_L2_HOME environmental variables prior to executing the script. The make file uses the g77 compiler in an attempt to compile the source code for the detected platform. The scripts are set up to try and compile on IRIX64, SunOS, AIX, Linux and HP-UX operating systems. Note: For other than the supported platforms, you will need to get the correct versions of the files in the /bin directory for your platform/OS. See section 1.0, software requirements. The executables and test output products were all created on a little endian platform. Run: As long as you have set the correct directory for MODIS_L2_HOME in the environmental variables source script, and sourced it, you can now execute the run script from any directory, however you must have all of the L1B input files (1km, 500m, 250m and geolocation) located within the same directory. To create the products type: modis_level2.csh where: is the satellite platform name (aqua or terra). is the path and name of the MODIS L1B 1km hdf file. is the full path name of the directory you want the output data to be placed in. The output files that are moved to the output directory upon completion depend upon the output type that you have chosen. If you choose to produce both binary and hdf files, then the files moved will be: (Where: s =t for terra or a for aqua yy=year -2000 ddd=Julian Day hh=hour mm=minute taken from the direct broadcast MODIS 1km L1B file) s1.yyddd.hhmm.mod35.hdf - Cloud mask HDF file s1.yyddd.hhmm.mod35.hdr - Cloud mask header file s1.yyddd.hhmm.mod35.img - Cloud mask binary image file s1.yyddd.hhmm.mod35qa.hdr - Cloud mask QA header file s1.yyddd.hhmm.mod35qa.img - Cloud mask QA binary image file s1.yyddd.hhmm.mod35debug.txt- Cloud mask scene statistics file s1.yyddd.hhmm.mod35.cfg - Cloud mask configuration file - list of inputs/output file names s1.yyddd.hhmm.mod04.hdf - Atmospheric aerosol HDF file s1.yyddd.hhmm.mod04.hdr - Atmospheric aerosol header file s1.yyddd.hhmm.mod04.img - Atmospheric aerosol image file s1.yyddd.hhmm.mod04debug.txt- Aerosol nadir statistics file s1.yyddd.hhmm.mod04.cfg - Aerosol configuration file - list of inputs/output file names s1.yyddd.hhmm.mod06ct.hdf - Cloud Top Properties/Phase HDF file s1.yyddd.hhmm.mod06.hdr - Cloud Top Properties/Phase header file s1.yyddd.hhmm.mod06.img - Cloud Top Properties/Phase binary file s1.yyddd.hhmm.mod06.hdr - Cloud Top Properties/Phase QA header file s1.yyddd.hhmm.mod06.img - Cloud Top Properties/Phase QA binary file s1.yyddd.hhmm.mod06debug.txt- Cloud Top Properties/Phase debug text file s1.yyddd.hhmm.mod06.cfg - Cloud Top Properties/Phase configuration file - list of inputs/output file names s1.yyddd.hhmm.mod07.hdf - Atmospheric Profiles hdf file s1.yyddd.hhmm.mod07.hdr - Atmospheric Profiles header file s1.yyddd.hhmm.mod07.img - Atmospheric Profiles binary file s1.yyddd.hhmm.mod07debug.txt- Atmospheric Profiles debug text file s1.yyddd.hhmm.mod07.cfg - Atmospheric Profiles configuration file - list of inputs/output file names s1.yyddd.hhmm.mod28.hdf - Sea Surface Temperatures hdf file s1.yyddd.hhmm.mod28.hdr - Sea Surface Temperatures header file s1.yyddd.hhmm.mod28.img - Sea Surface Temperatures binary file s1.yyddd.hhmm.mod28debug.txt- Sea Surface Temperatures debug text file s1.yyddd.hhmm.mod28.cfg - Sea Surface Temperatures configuration file - list of inputs/output file names s1.yyddd.hhmm.wvnir.hdf - Near infrared water vapor hdf file s1.yyddd.hhmm.wvnir.hdr - Near infrared water vapor header file s1.yyddd.hhmm.wvnir.img - Near infrared water vapor binary file s1.yyddd.hhmm.wvnir.cfg - Near infrared water vapor configuration file - list of inputs/output file names Intermediate files will remain in the temporary directory that is created during processing. For the input test data set, these files are: a1.07066.2038.1000m.hdr MODIS extracted Level 1B 1km header file a1.07066.2038.1000m.img MODIS extracted Level 1B 1km binary image file a1.07066.2038.250m.hdr MODIS extracted Level 1B 250m header file a1.07066.2038.250m.img MODIS extracted Level 1B 250m binary image file a1.07066.2038.500m.hdr MODIS extracted Level 1B 500m header file a1.07066.2038.500m.img MODIS extracted Level 1B 500m binary image file a1.07066.2038.geo.hdr MODIS extracted geolocation header file a1.07066.2038.geo.img MODIS extracted geolocation binary image file a1.07066.2038.met.hdr MODIS extracted Level 1B met header file a1.07066.2038.met.img MODIS extracted Level 1B met binary image file (scan type and mirror side info) eng.070307.bin Ice concentration binary grid field extracted from grib file gdas1.PGrbF00.070307.18z.bin Global model analysis binary grid fields extracted from grib file NISE_SSMIF13_20070307.HDFEOS_NORTH.bin Northern Hemisphere Snow/ice binary extracted fields NISE_SSMIF13_20070307.HDFEOS_SOUTH.bin Southern Hemisphere Snow/ice binary extracted fields TOAST16_070307.GRB.bin Total ozone binary grid field extracted from grib file In addition, the run script will automatically include downloading the original ancillary data files from the IMAPP ancillary data ftp site to the location you set for LOCAL_ANC_DIR in the /env script. This defaults to MODIS_L2_HOME/ancillary. The downloaded files for our test data set, which are included in the IMAPP_MODISL2_input_output.tar.gz file are: eng.070307 Ice concentration glopal grib file gdas1.PGrbF00.070307.18z Global model analysis grib file NISE_SSMIF13_20070307.HDFEOS Snow/ice cover HDFEOS file TOAST16_070307.GRB Total ozone global grib file oisst.20070307 Reynolds blended SST global binary file These files will not be deleted automatically in case you wish to archive the files. If you want the directories and files to be deleted, you must delete them yourselves. If you choose to create HDF file formatted output files, the run scripts will automatically call C based executables (create_fake_mod06.exe, create_fake_mod07.exe and create_fake_mod04.exe) to create a cloud top properties and cloud phase HDF file, an atmospheric profiles HDF file and an aerosol HDF file from the standard binary product. The MODIS cloudmask, sea surface temperature and near infrared water vapor software use the $OUTPUT_TYPE variables as an argument and will produce either binary or HDF outfiles from the standard software alone. All IDL routines used to convert from binary to HDF that were used in the past have been replaced. NOTE: The IMAPP output HDF files are similar to BUT NOT IDENTICAL to the official DAAC products. They are intended to provide enough information for a viewing tool to display the most important parameters located within the files. For instance, the IMAPP aerosol product contains only a subset of the arrays contained within the DAAC MOD04 product. For a description of the output products, please see the product descriptions located in the imapp_modisl2/doc directory. If you download and unpack the IMAPP McIDAS-Lite imaging software within the imapp_modisl2 directory, a imapp_modis/mclite directory will be created. This will allow the make_mclite_images.sh script to execute in the processing chain and create images in .GIF and .JPG formats using the HDF output files. No images are created from the binary files. The image size is set within the imapp_modisl2/env scripts (MODIS_L2_XSIZE, MODIS_L2_YSIZE) along with the output image file directory (MODIS_L2_IMAGES which defaults to ${IMAPP_L2_HOME}/images). The mclite script will automatically scale the output image to the resolution needed to fit within the size limits. Optimal image X and Y sizes are between 500 and 2000, and must be divisble by 4. The number of products created will depend upon the products you choose to create and whether the input data set is a day or night product. As an example, the sample data set included will result in these image products (the full set) being created from the HDF IMAPP product output files (daytime pass): AQUA_07_Mar_2007_2038_AerosolOpticalDepth.GIF AQUA_07_Mar_2007_2038_Band_01.GIF AQUA_07_Mar_2007_2038_Band_07.GIF AQUA_07_Mar_2007_2038_Band_26.GIF AQUA_07_Mar_2007_2038_Band_27.GIF AQUA_07_Mar_2007_2038_Band_31.GIF AQUA_07_Mar_2007_2038_CloudMask.GIF AQUA_07_Mar_2007_2038_CloudPhase.GIF AQUA_07_Mar_2007_2038_CloudTopPressure.GIF AQUA_07_Mar_2007_2038_RGB.JPG (True color image) AQUA_07_Mar_2007_2038_SeaSurfaceTemperatures.GIF AQUA_07_Mar_2007_2038_WaterVapor.GIF (MOD07 infrared product) This sample set of images is included in the IMAPP_MODISL2_input_output.tar file. 4.0 VERIFYING THE OUTPUT PACKAGE: --------------------------------- One test MODIS Aqua data set located in the IMAPP_MODISL2_input_output file. Download the file from the IMAPP web site http://cimss.ssec.wisc.edu/imapp and unpack the data sets. For test purposes, you may want to untar the files in the imapp_modisl2/input_data directory, e.g., cd $HOME/imapp_modisl2/input_data gzip -dc IMAPP_MODISL2_input_output.tar.gz | tar xvf - This will unpack all the input files needed to create the MODIS level2 IMAPP products: a1.07066.2038.1000m.hdf - Aqua MODIS 1km L1B input file a1.07066.2038.250m.hdf - Aqua MODIS 250m L1B input file a1.07066.2038.500m.hdf - Aqua MODIS 500m L1B input file a1.07066.2038.geo.hdf - Aqua MODIS 1km geolocation file NOTE: You must make sure that all of the input files are located within the same directory. Create the MODIS IMAPP Level 2 products for this example by typing: modis_level2.csh aqua $MODIS_L2_HOME/input_data/a1.07066.2038.1000m.hdf $MODIS_L2_HOME/products NOTE: You can execute the software from any directory as long as the $MODIS_L2_HOME environmental variable has been set correctly and you include the path to the input L1B 1km file. A directory will be created with the starting date/time of the run included in tha name. This directory is where execution takes place, and all intermediate files will be placed here. The final output products will be copied into the output directory read in as the third argument to modis_level2.csh. Output files will be created for the chosen products. A complete set of output product files is available in the IMAPP_MODISL2_input_output.tar.gz file for the test input granule. You can compare the output results by using the HDF4 "hdiff" utility. One statically compiled copy for the supported platforms is included within the /bin directory. To compare the HDF files created on your platform/OS with the packaged benchmark versions, from the /input_data directory, where the IMAPP_MODISL2_input_output files were copied to, type: hdiff -b -S a1.07066.2038.mod35.hdf ../products/a1.07066.2038.mod35.hdf You can type in similar commands for other file types. There will be statistics that show how many pixel values differ over the entire data set for each array located within the file. There should be few, if any, significant differences. If you see many differences, please let us know. Because the output binary files are created in ENVI compatible format, the binary product files can be viewed and compared using ITT's ENVI tool (http://www.ittvis.com/envi/index.asp) or its free counterpart tool FreeLook which can be downloaded from: http://www.ittvis.com/download/download.asp Using the Product dropdown menu, choose ENVI - FreeLook and then select the submit button. Follow the instructions for your platform. Direct comparison can be attempted by using the unix cmp command on the binary .img files and the unix diff command on the text output header files. From the imapp_modisl2/product directory, you may type (as an example): diff a1.07066.2038.mod04.hdr ../benchmark/a1.07066.2038.mod04.hdr cmp a1.07066.2038.mod04.img ../benchmark/a1.07066.2038.mod04.img diff a1.07066.2038.mod04debug.txt ../benchmark/a1.07066.2038.mod04debug.txt Again, these files were created on a little endian machine. Binary files created on a big endian machine must be byte flipped prior to any comparison.