Extract from IOC Training Manual Vol. II, 1994.

5. DATA PROCESSING

In the 1985 IOC Manual, Section 4 on data reduction covered all aspects regarding the

processing of chart/graphical records. A variety of typical problems were discussed and

solutions suggested. Whilst some of these problems were specific to chart records

(e.g. continuity) the majority could equally well apply to the processing of digital records.

5.1 PC BASED SOFTWARE

The aim of data processing software should be to ensure the scientific validity of the data.

Three main aspects should be considered :

a) linking of the data to a reference level e.g. permanent benchmark

b) timing; identification of missing scans etc.

c) correction of spikes and filling of short gaps

Many organisations have developed their own processing software to validate incoming data

in varied formats and media and are specific to their requirements. However two

organisations have developed PC based software as a contribution to GLOSS and other

programs with the aim of enabling participating countries to be able to process and validate

their own records. Contact names and addresses are given in Appendix 2.

5.1.1 TOGA SOFTWARE

The Tropical Ocean Global Atmosphere (TOGA) Programme Sea Level

Centre (TSLC) (Honolulu, USA) in collaboration with the US National Oceanographic Data

Centre (NODC) has prepared a package (Caldwell, 1991) for sea level data processing

designed for any IBM PC or compatible microcomputers under the DOS operating system

(DOS 3.1 or later). The package is geared towards people with some experience of DOS and

sea level data processing; it is interactive and self-descriptive. The package and

accompanying manual are freely available and updates and modifications are delivered to

users. The software occupies 1.9 Mbytes of hard disk, although not all of the programs need

to be loaded at the same time. The goal of the software is the establishment of a permanent

archive of hourly, daily and monthly data, written in a standardised format suitable for

incorporation into international archives where sea level data are available to the scientific

community for exchange and analysis.

The package includes software for:

a) Tidal Analysis and Prediction

This software facilitates the use of the tidal analysis and prediction programs of the Institute of

Ocean Sciences, Victoria, British Colombia (Foreman, 1977). It consists of self-descriptive,

interactive batch jobs and programs which prepare the input and output to the Foreman

programs.

b) Quality Control

Quality control ensures the scientific validity of the data. The software contains plotting

programs which are considered to be an integral part of the package as they are the primary

means used to quality control the input data and verify the processed data. This component

consists of three sections:

· Inspection of reference level stability: This allows comparison with tide staff

measurements and is usually carried out on a monthly basis, with values

compared month to month.

· Correction of timing errors: This is normally carried out annually; timing problems

are detected on plots of residuals and corrections of a whole number of hours can

be applied.

· Filling gaps and correcting data spikes: The software produces a list of gaps;

where possible these are replaced by data from auxiliary gauges. If this is not

possible gaps can be filled by interpolation. It is recommended that gaps less

than 24 hours only are replaced. This method can also be used for correcting

individual incorrect points (spikes) and glitches (1 - 6 consecutive obviously

wrong points).

c) Filtering Software

Programs are provided for obtaining daily values from hourly sea level data by a two-stage

process. Firstly the dominant diurnal and semi-diurnal components are removed and

secondly a 119-point convolution filter, centred on noon, is applied to remove the remaining

high frequency energy and to prevent aliasing. Monthly values are calculated from the daily

values with a simple average. Programs to enable comparison of two daily or monthly series

are also included.

5.1.2 FIAMS SOFTWARE

The Flinders Institute for Atmospheric and Marine Sciences (FIAMS) has prepared timeseries

software (FIAMS, July 1990) for sea level processing designed for use on Personal

Computers under the DOS operating system. The package includes software for:

a) Data Entry and Utilities

A program is provided for manual entry of data. There is also the capability to reformat data

files and the facility to change the time zone and units of constituents. Other utilities include

comparison of two card-image tidal level files and software to check for obvious errors in the

data. Simple statistics can be calculated (i.e. maximum, minimum, mean for each file) and

data files can be split into monthly files.

b) Analysis and Prediction

This software facilitates the use of the extensively modified tidal analysis (TIRA) and

prediction (ELSIE) programs first developed at the Proudman Oceanographic Laboratory,

Bidston Observatory, Birkenhead, Merseyside, UK (Murray, 1963) and consists of selfdescriptive,

interactive batch jobs and programs which prepare input and output to the

programs. Hourly tidal predictions or high and low water predictions can be calculated.

Residuals can be computed and statistics and histograms of the residuals produced.

Generation and display of constituent differences from two appropriately formatted

constituent files is also possible. Programs are also provided to compute and plot the results

from the spectral analysis of time-series.

c) Quality Control

Quality control ensures the scientific validity of the data. The software contains plotting

programs which are the primary means used to quality control the input data and verify the

processed data by comparison with predictions. Data can be compared by plotting the

difference between two series and/or observed and predicted levels. A program for filling

gaps using a Cosine-Lanczos filter in Fourier Transform space is provided.