mgd77manage(1gmt)
NAME
mgd77manage - Manage the content of MGD77+ files
SYNOPSIS
mgd77manage NGDC-ids [ -A[+]a|c|d|D|e|E|g|i|n|t|Tfileinfo ] [ -Cf|g|e ] [ -Dabbrev1,abbrev2,...) ] [ -Eempty ] [ -F ] [ -Iabbrev/name/unit/t/scale/offset/comment ] [ -Nunit ] [ -Rregion ] [ -V[level] ] [ -bibinary ] [ -dinodata ] [ -nflags ] Note: No space is allowed between the option flag and the associated arguments.
DESCRIPTION
mgd77manage deals with maintaining extra custom columns in MGD77+ netCDF files. You can either delete one or more columns, add a new column, update an existing column with new data, or supply error correction information (*.e77 files). New data may come from a table (ASCII unless -bi is used), be based on existing columns and certain theoretical expressions, or they may be obtained by sampling a grid (choose between GMT grid or a Sandwell/Smith Mercator *.img grid) along track. The new data will be appended to the MGD77+ file in the form of an extra data column of specified type. The data file will be modified; no new file will be created. For the big issues, see the DISCUSSION section below.
REQUIRED ARGUMENTS
NGDC-ids
Can be one or more of five kinds of specifiers:
1. 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
2. 2-character <agency> codes which will return all cruises from
each agency.
3. 4-character <agency><vessel> codes, which will return all
cruises from those vessels.
4. =<list>, where <list> is a table with NGDC IDs, one per line.
5. If nothing is specified we return all cruises in the data
base.
(See mgd77info -L for agency and vessel codes). The ".mgd77" or
".nc" extensions will automatically be appended, if needed (use
-I to ignore certain file types). Cruise files will be looked
for first in the current directory and second in all directories
listed in $MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set
it will default to $GMT_SHAREDIR/mgd77].
OPTIONAL ARGUMENTS
-A[+]a|c|d|D|e|E|g|i|n|t|Tfileinfo
Add a new data column. If an existing column with the same
abbreviation already exists in the file we will cowardly refuse
to update the file. Specifying -A+ overcomes this reluctance
(However, sometimes an existing column cannot be upgraded
without first deleting it; if so you will be warned). Select a
column source code among a, c, d, D, e, g, i, n, t, or T;
detailed descriptions for each choice follow:
a Append filename of a single column table to add. File must
have the same number of rows as the MGD77+ file. If no file is
given we read from stdin instead.
c Create a new column that derives from existing data or
formulas for corrections and reference fields. Append c for the
Carter corrections subtracted from uncorrected depths, g for the
IGF gravity reference field (a.k.a "normal gravity"), m for the
IGRF total field magnetic reference field, and r for recomputed
magnetic anomaly (append 1 or 2 to specify which total field
column to use [1]). For gravity we choose the reference field
based on the parameter Gravity Theoretical Formula Code in the
cruise's MGD77 header. If this is not set or is invalid we
default to the IGF 1980. You can override this behavior by
appending the desired code: 1 = Heiskanen 1924, 2 =
International 1930, 3 = IGF1967, or 4 = IGF1980.
d Append filename of a two-column table with the first column
holding distances along track and the second column holding data
values. If no file is given we read from stdin instead. Records
with matching distances in the MGD77+ file will be assigned the
new values; at other distances we set them to NaN.
Alternatively, give upper case D instead and we will interpolate
the column at all record distances. See -N for choosing distance
units and -C for choosing how distances are calculated.
e Expects to find an e77 error/correction log from mgd77sniffer
with the name NGDC_ID.e77 in the current directory or in
$MGD77_HOME/E77; this file will examined and used to make
modifications to the header values, specify a systematic
correction for certain columns (such as scale and offset),
specify that a certain anomaly should be recalculated from the
observations (e.g., recalculate mag from mtf1 and the latest
IGRF), and add or update the special column flag which may hold
bitflags (0 = GOOD, 1 = BAD) for each data field in the standard
MGD77 data set. Any fixed correction terms found (such as
needing to scale a field by 0.1 or 10 because the source agency
used incorrect units) will be written as attributes to the
netCDF MGD77+ file and applied when the data are read by
mgd77list. Ephemeral corrections such as those determined by
crossover analysis are not kept in the data files but reside in
correction tables (see mgd77list for details). By default, the
first character of each header line in the e77 file (which is ?,
Y or N) will be consulted to see if the corresponding adjustment
should be applied. If any undecided settings are found (i.i, ?)
we will abort and make no changes. Only records marked Y will be
processed. You can override this behavior by appending one or
more modifiers to the -Ae command: h will ignore all header
corrections, f will ignore all fixed systematic trend
corrections, n, v, and s will ignore bitflags pertaining to
navigation, data values, and data slopes, respectively. Use -A+e
to replace any existing E77 corrections in the file with the new
values. Finally, e77 corrections will not be applied if the E77
file has not been verified. Use -AE to ignore the verification
status.
g Sample a GMT geographic (lon, lat) grid along the track given
by the MGD77+ file using bicubic interpolation (however, see
-n). Append name of a GMT grid file.
i Sample a Sandwell/Smith Mercator *.img grid along the track
given by the MGD77+ file using bicubic interpolation (however,
see -n). Append the img grid filename, followed by the
comma-separated data scale (typically 1 or 0.1), the IMG file
mode (0-3), and optionally the img grid max latitude [80.738].
The modes stand for the following: (0) Img files with no
constraint code, returns data at all points, (1) Img file with
constraints coded, return data at all points, (2) Img file with
constraints coded, return data only at constrained points and
NaN elsewhere, and (3) Img file with constraints coded, return 1
at constraints and 0 elsewhere.
n Append filename of a two-column table with the first column
holding the record number (0 to nrows - 1) and the second column
holding data values. If no file is given we read from stdin
instead. Records with matching record numbers in the MGD77+
file will be assigned the new values; at other records we set
them to NaN.
t Append filename of a two-column table with the first column
holding absolute times along track and the second column holding
data values. If no file is given we read from stdin instead.
Records with matching times in the MGD77+ file will be assigned
the new values; at other times we set them to NaN.
Alternatively, give upper case T instead and we will interpolate
the column at all record times.
-Cf|g|e
Append a one-letter code to select the procedure for along-track
distance calculation when using -Ad|D (see -N for selecting
distance units):
f Flat Earth distances.
g Great circle distances [Default].
e Geodesic distances on current GMT ellipsoid.
-Dabbrev1,abbrev2,...)
Give a comma-separated list of column abbreviations that you
want to delete from the MGD77+ files. Do NOT use this option to
remove columns that you are replacing with new data (use -A+
instead). Because we cannot remove variables from netCDF files
we must create a new file without the columns to be deleted.
Once the file is successfully created we temporarily rename the
old file, change the new filename to the old filename, and
finally remove the old, renamed file.
-Eempty
Give a single character that will be repeated to fill empty
string values, e.g., "9" will yield a string like "99999..."
[9].
-F Force mode. When this mode is active you are empowered to delete
or replace even the standard MGD77 set of columns. You better
know what you are doing!
-Iabbrev/name/unit/t/scale/offset/comment
In addition to file information we must specify additional
information about the extra column. Specify a short (16 char or
less, using lower case letters, digits, or underscores only)
abbreviation for the selected data, its more descriptive name,
the data unit, the data type 1-character code (byte, short,
float, int, double, or text) you want used for storage in the
netCDF file, any scale and offset we should apply to the data to
make them fit inside the range implied by the chosen storage
type, and a general comment (< 128 characters) regarding what
these data represent. Note: If text data type is selected then
the terms "values" in the -A discussion refer to your text data.
Furthermore, the discussion on interpolation does not apply and
the NaN value becomes a "no string" value (see -E for what this
is). Place quotes around terms with more than one word (e.g.,
"Corrected Depth").
-Nunit Append the distance unit (see UNITS). [Default is -Nk (km)].
Only relevant when -Ag|i is selected.
-R[unit]xmin/xmax/ymin/ymax[r] (more ...)
Specify the region of interest. Only relevant when -Ag|i is
selected.
-V[level] (more ...)
Select verbosity level [c].
-bi[ncols][t] (more ...)
Select native binary input. This applies to the input 1- or
2-column data files specified under some of the -A options. The
binary input option is only available for numerical data
columns.
-dinodata (more ...)
Replace input columns that equal nodata with NaN.
-n[b|c|l|n][+a][+bBC][+c][+tthreshold] (more ...)
Select interpolation mode for grids.
-^ or just -
Print a short message about the syntax of the command, then
exits (NOTE: on Windows use just -).
-+ or just +
Print an extensive usage (help) message, including the
explanation of any module-specific option (but not the GMT
common options), then exits.
-? or no arguments
Print a complete usage (help) message, including the explanation
of options, then exits.
UNITS
For map distance unit, append unit d for arc degree, m for arc minute, and s for arc second, or e for meter [Default], f for foot, k for km, M for statute mile, n for nautical mile, and u for US survey foot. By default we compute such distances using a spherical approximation with great circles. Prepend - to a distance (or the unit is no distance is given) to perform "Flat Earth" calculations (quicker but less accurate) or prepend + to perform exact geodesic calculations (slower but more accurate).
CONSEQUENCES OF GRID RESAMPLING
Resample or sampling of grids will use various algorithms (see -n) that may lead to possible distortions or unexpected results in the resampled values. One expected effect of resampling with splines is the tendency for the new resampled values to slightly exceed the global min/max limits of the original grid. If this is unacceptable, you can impose clipping of the resampled values values so they do not exceed the input min/max values by adding +c to your -n option.
EXAMPLES
To append Geosat/ERS-1 gravity version 11.2 as an extra data column in
the cruises 01010047.nc and 01010008.nc, storing the values as mGal*10
in a 2-byte short integer, try
gmt mgd77manage 01010047 01010008 -Ai10/1/grav.11.2.img \
-Isatgrav/"Geosat/ERS-1 gravity"/"mGal"/s/10/0/"Sandwell/Smith version 11.2" -V
To append a filtered version of magnetics as an extra data column of
type float for the cruise 01010047.nc, and interpolate the filtered
data at the times given in the MGD77+ file, try
gmt mgd77manage 01010047 -ATmymag.tm -Ifiltmag/"Intermediate-wavelength \
magnetic residuals"/"nTesla"/f/1/0/"Useful for looking for isochrons" -V
To delete the existing extra columns satfaa, coastdist, and satvgg from
all MGD77+ files, try
gmt mgd77manage =allmgd77.lis -Dsatfaa,coastdist,satvgg -V
To create a 4-byte float column with the correct IGRF reference field
in all MGD77+ files, try
gmt mgd77manage =allmgd77.lis -Acm -Iigrf/"IGRF reference \
field"/"nTesla"/f/1/0/"IGRF version 10 for 1990-2010" -V
DISCUSSION
1. Preamble
The mgd77 supplement is an attempt to (1) improve on the limited
functionality of the existing mgg supplement, (2) incorporate some of
the ideas from Scripps' gmt+ supplement by allowing extra data columns,
and (3) add new capabilities for managing marine geophysical trackline
data stored in an architecture-independent CF-1.0- and COARDS-compliant
netCDF file format. Here are some of the underlying ideas and steps you
need to take to maintain your files.
2. Introduction
Our starting point is the MGD77 ASCII data files distributed from NGDC
on CD-ROMS, DVD-ROMS, and via FTP. Using Geodas to install the files
locally we choose the "Carter corrected depth" option which will fill
in the depth column using the two-way travel-times and the Carter
tables if twt is present. This step yields ~5000 individual cruise
files. Place these in one or more sub-directories of your choice, list
these sub-directories (one per line) in the file mgd77_paths.txt, and
place that file in the directory pointed to by $MGD77_HOME; if not set
this variable defaults to $GMT_SHAREDIR/mgd77.
3. Conversion
Convert the ASCII MGD77 files to the new netCDF MGD77+ format using
mgd77convert. Typically, you will make a list of all the cruises to be
converted (with or without extension), and you then run
mgd77convert =cruises.lis -Fa -Tc -V -Lwe+ > log.txt
The verbose settings will ensure that all problems found during
conversion will be reported. The new *.nc files may also be placed in
one or more separate sub-directories and these should also be listed in
the mgd77_paths.txt file. We suggest you place the directories with
*.nc files ahead of the *.mgd77 directories. When you later want to
limit a search to files of a certain extension you should use the -I
option.
4. Adding new columns
mgd77manage will allow you to add additional data columns to your *.nc
files. These can be anything, including text strings, but most likely
are numerical values sampled along the track from a supplied grid or an
existing column that have been filtered or manipulated for a particular
purpose. The format supports up to 32 such extra columns. See this man
page for how to add columns. You may later decide to remove some of
these columns or update the data associated with a certain column. Data
extraction tools such as mgd77list can be used to extract a mix of
standard MGD77 columns (navigation, time, and the usual geophysical
observations) and your custom columns.
5. Error sources
Before we discuss how to correct errors we will first list the
different classes of errors associated with MGD77 data: (1) Header
record errors occur when some of the information fields in the header
do not comply with the MGD77 specification or required information is
missing. mgd77convert will list these errors when the extended verbose
setting is selected. These errors typically do not affect the data and
are instead errors in the meta-data (2). Fixed systematic errors occur
when a particular data column, despite the MGD77 specification, has
been encoded incorrectly. This usually means the data will be off by a
constant factor such as 10 or 0.1, or in some cases even 1.8288 which
converts fathoms to meters. (3) Unknown systematic errors occur when
the instrument that recorded the data or the processing that followed
introduced signals that appear to be systematic functions of time along
track, latitude, heading, or some other combination of terms that have
a physical or logical explanation. These terms may sometimes be
resolved by data analysis techniques such as along-track and
across-track investigations, and will result in correction terms that
when applied to the data will remove these unwanted signals in an
optimal way. Because these correction terms may change when new data
are considered in their determination, such corrections are considered
to be ephemeral. (4) Individual data points or sequences of data may
violate rules such as being outside of possible ranges or in other ways
violate sanity. Furthermore, sequences of points that may be within
valid ranges may give rise to data gradients that are unreasonable. The
status of every point can therefore be determined and this gives rise
to bitflags GOOD or BAD. Our policy is that error sources 1, 2, and 4
will be corrected by supplying the information as meta-data in the
relevant *.nc files, whereas the corrections for error source 3
(because they will constantly be improved) will be maintained in a
separate list of corrections.
6. Finding errors
The mgd77sniffer is a tool that does a thorough along-track sanity
check of the original MGD77 ASCII files and produces a corresponding
*.e77 error log. All problems found are encoded in the error log, and
recommended fixed correction terms are given, if needed. An analyst may
verify that the suggested corrections are indeed valid (we only want to
correct truly obvious unit errors), edit these error logs and modify
such correction terms and activate them by changing the relevant code
key (see mgd77sniffer for more details). mgd77manage can ingest these
error logs and (1) correct bad header records given the suggestions in
the log, (2) insert scale/offset correction terms to be used when
reading certain columns, and (3) insert any bit-flags found. Rerun this
step if you later find other problems as all E77 settings or flags will
be recreated based on the latest E77 log.
7. Error corrections
The extraction program mgd77list allows for corrections to be applied
on-the-fly when data are requested. First, data with BAD bitflags are
suppressed. Second, data with fixed systematic correction terms are
corrected accordingly. Third, data with ephemeral correction terms will
have those corrections applied (if a correction table is supplied). All
of these steps require the presence of the relevant meta-data and all
can be overruled by the user. In addition, users may add their own
bitflags as separate data columns and use mgd77list's logical tests to
further dictate which data are suppressed from output.
CREDITS
The IGRF calculations are based on a Fortran program written by Susan Macmillan, British Geological Survey, translated to C via f2c by Joaquim Luis, and adapted to GMT style by Paul Wessel.
SEE ALSO
mgd77convert, mgd77list, mgd77info, mgd77sniffer mgd77track x2sys_init
REFERENCES
The Marine Geophysical Data Exchange Format - MGD77, see http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt IGRF, see http://www.ngdc.noaa.gov/IAGA/vmod/igrf.html
COPYRIGHT
2016, P. Wessel, W. H. F. Smith, R. Scharroo, J. Luis, and F. Wobbe
Opportunity
Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.
Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.
Free Software
Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.
Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.
Free Books
The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.
Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.
Education
Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.
Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.
