MS2GT: The MODIS Swath-to-Grid Toolbox

Version History

The current version of MS2GT is 0.5 released May 31, 2001

Overview

What the Software Does

The MODIS Swath-to-Grid Toolbox (MS2GT) is a set of software tools that can be used to read HDF-EOS files containing MODIS swath data and produce flat binary files containing gridded data in a variety of map projections. Multiple input files corresponding to successively acquired 5 minute MODIS "scenes" can be processed together to produce a seamless output grid.

MS2GT consists of three perl programs that make calls to several standalone IDL and C programs: mod02.pl which reads MOD021KM, MOD02HKM, or MOD02QKM Level 1b files, mod10_l2.pl which reads MOD10_L2 snow cover files, and mod29.pl which reads MOD29 sea ice files.  All three Perl programs can optionally read MOD03 files for geolocation information; in addition, mod02.pl can extract "ancillary" data (such as illumination and viewing angles) from MOD03 files.

How the Software Works

Each of the three Perl scripts provided with MS2GT make similar calls to the IDL and C programs outlined below: It should relatively easy for a user who is familiar with Perl and IDL programming to modify the Perl scripts and IDL procedures in order to process other kinds swath data using many of the same standalone programs provided with MS2GT. Further details can be found in the Tutorials.

Requirements

In its current form, MS2GT runs on a Unix workstation with a standard C compiler. It also requires Perl 5.0 or higher and IDL 5.0 or higher. The installation instructions assume that the user is running csh or tcsh as the Unix shell. Users of other shells (e.g. bash, ksh, sh, etc.) may have to make slight changes to the installation instructions.

MS2GT has been developed and tested on an SGI O2 workstation having 192 Mbytes of memory and running IRIX 6.5, perl 5.004_04, and IDL 5.4. Care has been taken to minimize memory requirements at the expense of increased temporary disk storage requirements and slightly slower speed. However, in its current form, MS2GT does require that two times a grid's worth of single precision floating-point data for one channel fit into virtual memory. The result is that fairly large gridded images can be created on a machine with a modest amount of memory in a fairly short time. For example, the creation of two 1316 x 1384 250 m grids containing MODIS channels 1 and 2 reflectance data derived from a single MOD02QKM file takes about three minutes on the above machine and requires about 2 * 1316 * 1384 * 4 = 15 MB of free virtual memory.
 

Supported Data Sets

See the MODIS Home Page for general information on the MODIS instrument. See MODIS Snow and Ice Products for information on MODIS snow and ice products distributed by NSIDC.
Short Name Long Name Channel Data Ancillary Lat-lon
MOD021KM MODIS/TERRA CALIBRATED RADIANCES 5-MIN L1B SWATH 1KM 1-36 @ 1 km 7 @ 5 km 5 km
MOD02HKM MODIS/TERRA CALIBRATED RADIANCES 5-MIN L1B SWATH 500M 1-7 @ 500 m none 1 km
MOD02QKM MODIS/TERRA CALIBRATED RADIANCES 5-MIN L1B SWATH 250M 1-2 @ 250 m none 1 km
MOD03 MODIS/TERRA GEOLOCATION FIELDS 5-MIN L1A SWATH 1KM none 8 @ 1 km 1 km
MOD10_L2 MODIS/TERRA SNOW COVER 5-MIN L2 SWATH 500M 2 @ 500 m none 5 km
MOD29 MODIS/TERRA SEA ICE EXTENT 5-MIN L2 SWATH 1KM 6 @ 1 km none 5 km

Installation Instructions

Obtaining the Software

The software and associated documentation can be downloaded from ftp://baikal.colorado.edu/pub/NSIDC/ms2gt.0.5.tar.gz, where 0.5 is the current version and release. Save this file into some directory where you want the MS2GT software installed. If you have a previous version of MS2GT installed in the same directory, you should either remove it or rename it. For example:

rm -fr ms2gt

or something like:

mv ms2gt ms2gt.old

Then type:

gunzip -dc ms2gt0.5.tar.gz | tar xvf -

This will create a directory in the current directory called ms2gt which will contain several subdirectories. Further instructions on the installation and use of MS2GT can be then found in html files in the ms2gt/doc subdirectory. At this point you can delete the tar.gz file if you want. Point your browser to ms2gt/doc/index.html (this document).

Building the Executables

Change the current working directory to the ms2gt/src directory and type:

make all

This will build and install the MS2GT executables under the ms2gt directory. The only write privileges required are the ability to write into the ms2gt directory and its subdirectories.

Verifying your Perl Installation

Type the command:

perl -v

If you see something like:

This is perl, version 5.004_04 built for irix-n32

Copyright 1987-1997, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5.0 source kit.

then you know perl has been installed ok. Just make sure you have at least perl version 5 installed. However, if you see something like:

perl: Command not found

then you need to contact your system administrator to get perl installed.

Assuming you have perl installed, then type:

/usr/bin/perl -v

If you see the same message as before, then you have a proper link to your installed version of perl. However, if you see something like:

/usr/bin/perl: Command not found

or you see mention of a lower version of perl, then you need to contact your system administrator and have a link called /usr/bin/perl created that points to your installed version of perl. For example, if you type:

which perl (or type perl if you're running bash, ksh, or sh)

and you see:

/usr/sbin/perl

then the command to create the link would be:

ln -s /usr/sbin/perl /usr/bin/perl

but you'll probably need to be logged in as root to be able to execute this command successfully.

Setting up the MS2GT Environment

Edit your .cshrc or your .login file and insert the following two lines:

setenv MS2GT_HOME $HOME/ms2gt
source $MS2GT_HOME/ms2gt_env.csh

If you installed ms2gt into some directory other than $HOME, then change the first line accordingly. If your .cshrc or your .login includes lines such as:

setenv IDL_DIR /usr/local/rsi/idl
source $IDL_DIR/bin/idl_setup
setenv IDL_PATH ...

or a line such as

setenv PATHMPP ...

then the two MS2GT lines above should be placed after these IDL and PATHMPP lines in the appropriate file. Once your .cshrc or your .login file has been editted, then logout and login again, or else type the following two lines:

source ~/.cshrc or
source ~/.login
rehash

Finally, create a writeable directory in your home directory called tmp that will be used by a perl script called idl_sh.pl. This directory will be used for holding temporary shell scripts for running IDL programs:

mkdir ~/tmp

Verifying the MS2GT Installation

Try typing in the following three commands to get the syntax of each of the three perl scripts (note that the name of the first script contains a zero (0), followed by a two (2); note that the name of the second script contains a one (1), followed by a zero (0), followed by an underbar (_), followed by a lower case L (l), followed by a two (2):

mod02.pl
mod10_l2.pl
mod29.pl

You should see a usage message that describes the syntax of each perl script that is essentially the same as mod02_usage, mod10_l2_usage and mod29_usage, respectively. If you get "command not found" or something to that effect, it probably means that your $path is being set after the source $MS2GT_HOME/ms2gt_env.csh command in your .cshrc or .login file. Find the line in your .cshrc or .login that looks like:

set path = <something>

and change it to:

set path = ($path <something>)

Then logout and login and try the commands again.

Next, verify that the environment variable $IDL_PATH is set correctly. Type:

echo $IDL_PATH

You should see a directory that looks like:

<something>/ms2gt/src/idl

as one of the directories in the IDL path. If not, then make sure that you placed the source $MS2GT_HOME/ms2gt_env.csh command after setting $IDL_PATH in your .login or .cshrc as decribed above.

Finally, verify that the environment varialble $PATHMPP is set correctly. Type:

echo $PATHMPP

You should see a directory that looks like:

<something>/ms2gt/grids

as one of the directories in PATHMPP. If not, then make sure that you placed the source $MS2GT_HOME/ms2gt_env.csh command after setting $PATHMPP in your .login or .cshrc as decribed above.

Using the MS2GT Software

In most cases, users will need to perform the following steps in order to use the MS2GT software (detailed examples of these steps can be found in the Tutorials):

Searching for the Data

The user needs to use the EDG or a similar tool to find the swath data to be gridded.

Ordering and Downloading the Data

The user needs to use the EDG or a similar tool to order and download the swath data to be gridded.

Creating a MS2GT Command File

The user needs to use a text editor to create a command file for running the particular MS2GT script: mod02.pl, mod10_l2.pl, or mod29.pl . This command file contains an invocation of the particular script together with its associated parameters. The parameters to a particular script contain the names of miscellaneous text files (described below) as well as other parameters used to control the operation of the script including a specification of which data arrays in the input file(s) are to be gridded. Alternatively, the invocation of the script and its parameters may be simply typed in on the command line once the miscellaneous text files have been created.

Creating Miscellaneous Text Files

The user needs to use a text editor to create miscellaneous text files used to control the operation of the particular mod02.pl, mod10_l2.pl, or mod29.pl script.

Each script requires a listfile which contains a list of the filenames of the HDF-EOS files to be gridded. Each of these HDF-EOS files contains MODIS swath data acquired over a five minute interval. Consecutively acquired files can be gridded together to produce a seamless gridded result.

Each script also requires a Grid Parameters Definition (gpd) file, which in turn requires an associated Map Projection Parameters (mpp) file. These files are used by the mapx library software (developed at NSIDC) to define the grid into which the swath data is to be mapped. The format of these files and an explanation of the mapx library is provided in Points, Pixels, Grids, and Cells.

Other miscellaneous text files used by some of the scripts include a chanfile, an ancilfile, and a latlonfile.

The format and use of each miscellaneous file is described below as well as in the Tutorials.

Running the MS2GT Command File

Once all the miscellaneous text files have been created, the MS2GT command file can be invoked to actually perform the gridding, or the invocation of the particular MS2GT script and its associated parameters may be simply typed in on the command line.

The MS2GT script runs several IDL and C programs to actually perform the gridding as outlined above. Further explanations are also provided in the Tutorials.

Examining the Results

Once the particular MS2GT script has completed processing, there will exist a single binary flat file containing the gridded data for each channel specified in the MS2GT command file. The filenames for each flat file contain fields that specify the type of data the file contains as well as the number of rows and columns in the grid. A detailed description of the filenames created by each script is provided in the Tutorials. The MS2GT software does not currently provide any facility for visualizing these data, but they can be easily imported into a variety of visualization programs.

Geolocating the Results

The MS2GT software contains a utility called gridloc which can be used to create binary floating-point flat files containing the latitude and longitude of each cell in the grid. An example of how to use gridloc can be found at the end of Tutorial 1.

Script Descriptions and Usage

mod02.pl

This script can process MOD021KM, MOD02HKM, MOD02QKM, or MOD03 swath files to produce flat binary gridded files containing MODIS Level 1B channel data and/or ancillary data. The channel data consist of up to 36 spectral bands described in MODIS Technical Specifications. See Supported Data Sets for a table summarizing which channels are available at which resolutions in which files. The table also shows that ancillary data are available in MOD021KM and MOD03 files at 5 km and 1 km resolutions, respectively. Finally the table shows geolocation (lat-lon) data are available at either 1 km or 5 km resolutions. Use of 1 km ancillary data and/or 1 km geolocation data will minimize interpolation error.

The mod02.pl script has the following usage:

mod02.pl dirinout tag listfile gpdfile chanfile
                [ancilfile [latlon_src [ancil_src [keep [rind]]]]]
       defaults:   none          1          1       0     50

mod02.pl usage provides a cursory explanation of the mod02.pl script parameters. A more expanded discussion of each parameter is provided here. Note that the first five parameters to mod02.pl are required; the rest are optional.

  1. dirinout - the directory containing all input files required by mod02.pl and all output files created by mod02.pl. If input files must reside in some directory other that the one used for output, then the user will need to create links to each of these files in the dirinout directory.
  2. tag - a string used in constructing the names of output files. Each output filename will begin with the tag string.
  3. listfile - the name of a text file containing the names of the MOD021KM, MOD02HKM, MOD02QKM, or MOD03 swath files to be gridded. If multiple files are present in listfile, then the files must all be of the same type (e.g. if the first file is a MOD021KM file, then all subsequent files must also be MOD021KM files). Multiple files must have been acquired at subsequent acquisition times without any gaps. If chanfile (below) is "none", then listfile must contain a list of one or more MOD03 files from which the 1 km ancillary data specified in ancilfile (below) will be read. If chanfile is not "none", then listfile must contain a list of one or more MOD021KM, MOD02HKM, or MOD02QKM files from which the 1 km, 500 m, or 250 m channel data, respectively, specified in chanfile will be read. If, in this latter case, ancilfile is also not "none", then the 5 km data specified in ancilfile will be read from MOD03 files whose filenames are constructed from the corresponding filenames in chanfile by substituting the MOD021KM, MOD02HKM, or MOD02QKM filename prefix with MOD03.
  4. gpdfile - the name of a Grid Parameters Definition (gpd) file that defines the desired output grid. See Points, Pixels, Grids, and Cells and the Tutorials for more details.
  5. chanfile - the name of a text file containing a list of the channels to be gridded, one line per channel.  If chanfile is "none" then no channel data will be gridded and ancilfile must not be "none". Note that if the first file in listfile is a MOD03 file, then chanfile must be "none" and both latlon_src and ancil_src (described below) are forced to 3. The channels specified in chanfile are always read from the files specified in listfile. Each line in chanfile can have up to four fields. The first field is mandatory; the rest are optional. The fields are:
      6. chan conversion weight_type fill
    and are defined as follows:
  6. ancilfile - the name of a text file containing a list of ancillary parameters to be gridded, one line per parameter. The default is "none" indicating that no ancillary parameters should be gridded. Each line in ancilfile can have up to four fields. The first field is mandatory; the rest are optional. The fields are:
      7. param conversion weight_type fill
    and are defined as follows:
  7. latlon_src - a single character code specifying from where the latitude and longitude data should be read. The codes are as follows:
    8. Note that if latlon_src is set to 3, then ancil_src (below) is forced to 3. Note also that if the first file specified in listfile is a MOD02HKM, MOD02QKM, or MOD03 file, then latlon_src is forced to H, Q, or 3, respectively.
  8. ancil_src - a single character code specifying from where the ancillary data should be read. The codes are as follows:
    9. Note that if ancil_src is set to 3, then latlon_src is forced to 3. Note also that if the first file specified in listfile is a MOD03 file, then ancil_src is forced to 3.
  9. keep - a single character code indicating whether intermediate files should or should not be deleted. The codes are as follows:
  10. rind - number of pixels to add around intermediate grid to eliminate holes in final grid. Default is 50 which should be adequate in most cases. If you see pixels being set to the fill value along the seam joining adjacent scenes, then you might try increasing the value of rind.

mod10_l2.pl

This script can process MOD10_L2 swath files to produce flat binary gridded files containing MODIS snow cover data. The snow cover data consist of two 500 m resolution "channels" described in mod10_l2 usage and MOD10_L2. The geolocation information stored in MOD10_L2 files is 5 km resolution, but 1 km resolution geolocation information extracted from MOD02HKM, MOD02QKM, or MOD03 files if available can be used by mod10_l2.pl via the latlonlistfile parameter (described below). Use of 1 km geolocation information will minimize geolocation interpolation error.

The mod10_l2.pl script has the following usage:

mod10_l2.pl dirinout tag listfile gpdfile
                  [chanlist [latlonlistfile [keep [rind]]]]
       defaults:      1          none         0     50

mod10_l2 usage provides a cursory explanation of the mod10_l2.pl script parameters. A more expanded discussion of each parameter is provided here. Note that the first four parameters to mod10_l2.pl are required; the rest are optional.

  1. dirinout - the directory containing all input files required by mod10_l2.pl and all output files created by mod10_l2.pl. If input files must reside in some directory other that the one used for output, then the user will need to create links to each of these files in the dirinout directory.
  2. tag - a string used in constructing the names of output files. Each output filename will begin with the tag string.
  3. listfile - the name of a text file containing the names of the MOD10_L2 swath files to be gridded. Multiple files must have been acquired at subsequent acquisition times without any gaps.
  4. gpdfile - the name of a Grid Parameters Definition (gpd) file that defines the desired output grid. See Points, Pixels, Grids, and Cells and the Tutorials for more details.
  5. chanlist - string specifying channel numbers to be gridded. The default string is 1, i.e. grid channel 1 only. The channel numbers are:
    6. The chanlist string should not contain any embedded blanks. Thus to specify that all channels should be gridded, chanlist should be 12.
  6. latlonlistfile - text file containing a list of MOD02HKM, MOD02QKM, or MOD03 files whose latitude and longitude data should be used in place of the latitude and longitude data in the corresponding MOD10_L2 files in listfile. The default is "none" indicating that the latitude and longitude data in each MOD10_L2 file should be used without substitution.
  7. keep - a single character code indicating whether intermediate files should or should not be deleted. The codes are as follows:
  8. rind - number of pixels to add around intermediate grid to eliminate holes in final grid. Default is 50 which should be adequate in most cases. If you see pixels being set to the fill value along the seam joining adjacent scenes, then you might try increasing the value of rind.

mod29.pl

This script can process MOD29 swath files to produce flat binary gridded files containing MODIS sea ice extent data. The sea ice extent data consist of six 1 km resolution "channels" described in mod29 usage and MOD29. The geolocation information stored in MOD29 files is 5 km resolution, but 1 km resolution geolocation information extracted from MOD02HKM, MOD02QKM, or MOD03 files if available can be used by mod29.pl via the latlonlistfile parameter (described below). Use of 1 km geolocation information will minimize geolocation interpolation error.

The mod29.pl script has the following usage:

mod29.pl dirinout tag listfile gpdfile
                [chanlist [latlonlistfile [keep [rind]]]]
       defaults:    1          none         0     50

mod29 usage provides a cursory explanation of the mod29.pl script parameters. A more expanded discussion of each parameter is provided here. Note that the first four parameters to mod29.pl are required; the rest are optional.

  1. dirinout - the directory containing all input files required by mod29.pl and all output files created by mod29.pl. If input files must reside in some directory other that the one used for output, then the user will need to create links to each of these files in the dirinout directory.
  2. tag - a string used in constructing the names of output files. Each output filename will begin with the tag string.
  3. listfile - the name of a text file containing the names of the MOD29 swath files to be gridded. Multiple files must have been acquired at subsequent acquisition times without any gaps.
  4. gpdfile - the name of a Grid Parameters Definition (gpd) file that defines the desired output grid. See Points, Pixels, Grids, and Cells and the Tutorials for more details.
  5. chanlist - string specifying channel numbers to be gridded. The default string is 1, i.e. grid channel 1 only. The channel numbers are:
    6. The chanlist string should not contain any embedded blanks. Thus to specify that all channels should be gridded, chanlist should be 123456.
    NOTE: During twilight and nighttime conditions, "dark" MOD29 granules may be created that do not include Sea Ice by Reflectance, Sea Ice by Reflectance PixelQA, nor Combined Sea Ice channels. Gridded arrays containing the fill value of 255 are returned when these channels are specified for dark MOD29 granules.
  6. latlonlistfile - text file containing a list of MOD02HKM, MOD02QKM, or MOD03 files whose latitude and longitude data should be used in place of the latitude and longitude data in the corresponding MOD29 files in listfile. The default is "none" indicating that the latitude and longitude data in each MOD29 file should be used without substitution.
  7. keep - a single character code indicating whether intermediate files should or should not be deleted. The codes are as follows:
  8. rind - number of pixels to add around intermediate grid to eliminate holes in final grid. Default is 50 which should be adequate in most cases. If you see pixels being set to the fill value along the seam joining adjacent scenes, then you might try increasing the value of rind.
Last updated: May 31, 2001 by
Terry Haran
NSIDC-CIRES
449 UCB
University of Colorado
Boulder, CO 80309-0449
303-492-1847
tharan@colorado.edu