Archive formats for SEDNA data
From APLIS07
Time series and model data related to SEDNA should be placed into NetCDF format, while HDF5 is the recommended format for all satellite data products generated in this project. This will enable the broadest possible use of our data in the Earth Science community. As the satellite data products are specific to Dr. Cathy Geiger's group, no further explanation is supplied here of the HDF5 format. However several groups will be using NetCDF, and a step-by-step guide is provided here for archiving SEDNA data using standard conventions.
Contents |
A brief introduction to NetCDF
For those not familiar with NetCDF, it is an efficient self-describing binary format frequently used in Meteorology and Oceanography and is easily ingested and created from within many common graphics and data processing packages including Matlab, NCAR graphics, IDL, PV-Wave and Ferret. The binary data may easily be converted to ASCII using the simple but effective ncdump routine and converted from ASCII using ncgen, however there are also C, Fortran and Java libraries to write directly from within programs, and most common graphics and processing packages have their own in-built functions available for the same task.
A complete description of everything to do with NetCDF is available at Unidata, including software for generating NetCDF and information on using it from within graphics and processing packages. If you are a beginner with NetCDF, it is highly recommended you visit this website before proceeding further.
Writing NetCDF using the CF-1.0 convention
It is easy to write any data to NetCDF providing you can define the dimension(s) of the data. Of greater importance, however, is to write the data using a predefined convention recognized by graphics and data processing packages that enable the packages to easily pluck self-describing information from the file. The COARDS convention has been used for this purpose in the past, but the most up-to-date standard is the CF-1.0 convention. A complete description of this convention is available at this Lawrence Livermore National Laboratory site. A hands-on description is provided here of how to apply the CF-1.0 convention for the SEDNA dataset.
It is strongly recommended that as part of your CF-1.0 compliant archive you use UTC "hours since 2000-01-01 00:00:0.0" as your units of time because this will be easily interpreted by most graphics and data processing packages and removes all ambiguity often surrounding time formats in data files.
It is also suggested you end all files with the .nc extension as is common for NetCDF files. It is suggested you use the following naming conention: SEDNA_{data_name}_{further_data descriptors}.nc. Where, using the example of the weather station data, there are two files available, one called SENDA_WX.nc (WX abbreviates weather), and the other SEDNA_WX_SUMMARY.nc.
SEDNA global attributes
The global attributes for each SEDNA NetCDF file should include these fields:
title - A succinct description of what is in the dataset.
source - The method of production of the original data.
If it is model-generated, source should name the model and its,
version as specifically as is useful. If it is an observational,
source should characterize this (e.g., "surface observation" or
"radiosonde"). This attribute could also include specifics of
of the instrumentation used to produce the data:
1) satellite/remotely sensed data: include summary
of the satellite, instrument and processing algorithms.
2) GPS/positional data: include processing procedure and
retrieval method.
3) directly sensed data: exact details on sensors used (model
numbers, sensor types)
4) model generated data: computer on which the data was
generated.
5) conglomerate dataset (such as climatology or assimilated):
list (within reasonable bounds) all sources of data and
methods/models used to compile the dataset.
Or the this listed information can be included as a comment. See
The working example below for more information.
references - Include the SEDNA data report along with other references of your choice.
comment - Miscellaneous information about the data or methods used to produce it. See
the working example below for ideas on what should be included as comments.
institution - Your institution name.
history - The date and time of file creation, and who created it.
SEDNA variable attributes
Each variable in the dataset must first have their dimensions defined (see the Unidata website for further explanation). Following this, it is recommended you include each of these fields for every variable of your dataset:
long_name - text string of the long version of the variable name
units - text string specifying the data units. If the variable
is time then it is suggested to use the Gregorian Calendar
to calculate the time archive with units of
"hours since 2000-01-01 00:00:0.0" for all SEDNA data.
dimension(s) - the coordinate variables defining this data. For a 1D
time series this will be 'time', for a 2D spatial dataset
this might be 'latitude' and 'longitude', for example.
data - data values to the same precision as nc_type. Do not
compress data using add_offset and scale_factor.
missing_value - the value provided for missing values along any defined
NetCDF dimension for the particular variable. If there are
no missing values this may be omitted. Note that the missing
value must have the same nc_type as the data.
nc_type - the data must be assigned a type when writing to netcdf.
If the variable is time, the units suggested above require
NC_FLOAT to be the type. NC_DOUBLE (double precision),
NC_INT (integer) are other common archive forms.
A working example of a SEDNA NetCDF archive
Below is the NetCDF header information from the SEDNA_WX.nc weather station data archived in CF-1.0 NetCDF format. The following is a summary obtained using ncdump -h. Note that the data is self describing, and a reference is provided for where to find more information if required:
netcdf SEDNA_WX {
dimensions:
time = UNLIMITED ; // (1855 currently)
variables:
float time(time) ;
time:long_name = "time" ;
time:units = "hours since 2000-01-01 00:00:0.0" ;
float latitude(time) ;
latitude:long_name = "latitude within 407 meters of weather station" ;
latitude:units = "degrees North" ;
float longitude(time) ;
longitude:long_name = "longitude within 407 meters of weather station" ;
longitude:units = "degrees East" ;
float battery(time) ;
battery:long_name = "10 minute average of battery voltage" ;
battery:units = "Volts" ;
float panel(time) ;
panel:long_name = "10 minute average electronics panel temperature" ;
panel:units = "Degrees Celsius" ;
float bar(time) ;
bar:long_name = "10 minute sampled barometric pressure" ;
bar:units = "hPa" ;
bar:missing_value = 99999.9f ;
float w3dir(time) ;
w3dir:long_name = "10 minute sample of wind direction 3.1m above the surface" ;
w3dir:units = "degrees clockwise from North" ;
float w3mean(time) ;
w3mean:long_name = "10 minute mean wind speed 3.1m above the surface" ;
w3mean:units = "m/s" ;
float w3max(time) ;
w3max:long_name = "10 minute maximum wind speed 3.1m above the surface" ;
w3max:units = "m/s" ;
float w3min(time) ;
w3min:long_name = "10 minute minimum wind speed 3.1m above the surface" ;
w3min:units = "m/s" ;
float w3std(time) ;
w3std:long_name = "10 minute standard deviation of wind speed 3.1m above the surface" ;
w3std:units = "m/s" ;
float t3mean(time) ;
t3mean:long_name = "10 minute mean air temperature 3.1m above the surface" ;
t3mean:units = "degrees Celsius" ;
float t3max(time) ;
t3max:long_name = "10 minute maximum air temperature 3.1m above the surface" ;
t3max:units = "degrees Celsius" ;
float t3min(time) ;
t3min:long_name = "10 minute minimum air temperature 3.1m above the surface" ;
t3min:units = "degrees Celsius" ;
float t3std(time) ;
t3std:long_name = "10 minute standard deviation air temperature 3.1m above the surface" ;
t3std:units = "degrees Celsius" ;
float rh3mean(time) ;
rh3mean:long_name = "10 minute mean relative humidity 3.1m above the surface" ;
rh3mean:units = "percent" ;
rh3mean:missing_value = 99999.9f ;
float rh3max(time) ;
rh3max:long_name = "10 minute maximum relative humidity 3.1m above the surface" ;
rh3max:units = "percent" ;
rh3max:missing_value = 99999.9f ;
float rh3min(time) ;
rh3min:long_name = "10 minute minimum relative humidity 3.1m above the surface" ;
rh3min:units = "percent" ;
rh3min:missing_value = 99999.9f ;
float rh3std(time) ;
rh3std:long_name = "10 minute standard deviation relative humidity 3.1m above the surface" ;
rh3std:units = "percent" ;
rh3std:missing_value = 99999.9f ;
float dew3mean(time) ;
dew3mean:long_name = "10 minute mean dew point temperature 3.1m above the surface" ;
dew3mean:units = "degrees Celsius" ;
float dew3max(time) ;
dew3max:long_name = "10 minute mean dew point temperature 3.1m above the surface" ;
dew3max:units = "degrees Celsius" ;
float dew3min(time) ;
dew3min:long_name = "10 minute minimum dew point temperature 3.1m above the surface" ;
dew3min:units = "degrees Celsius" ;
float dew3std(time) ;
dew3std:long_name = "10 minute standard deviation dew point temperature 3.1m above the surface" ;
dew3std:units = "degrees Celsius" ;
float w2dir(time) ;
w2dir:long_name = "10 minute sample of wind direction 1.9m above the surface" ;
w2dir:units = "degrees clockwise from North" ;
float w2mean(time) ;
w2mean:long_name = "mean wind speed 1.9m above the surface over averaging period" ;
w2mean:units = "m/s" ;
float w2max(time) ;
w2max:long_name = "10 minute maximum wind speed 1.9m above the surface" ;
w2max:units = "m/s" ;
float w2min(time) ;
w2min:long_name = "10 minute minimum wind speed 1.9m above the surface" ;
w2min:units = "m/s" ;
float w2std(time) ;
w2std:long_name = "10 minute standard deviation of wind speed 1.9m above the surface" ;
w2std:units = "m/s" ;
float t2mean(time) ;
t2mean:long_name = "10 minute mean air temperature 1.9m above the surface" ;
t2mean:units = "degrees Celsius" ;
float t2max(time) ;
t2max:long_name = "10 minute maximum air temperature 1.9m above the surface" ;
t2max:units = "degrees Celsius" ;
float t2min(time) ;
t2min:long_name = "10 minute minimum air temperature 1.9m above the surface" ;
t2min:units = "degrees Celsius" ;
float t2std(time) ;
t2std:long_name = "10 minute standard deviation of air temperature 1.9m above the surface" ;
t2std:units = "degrees Celsius" ;
float rh2mean(time) ;
rh2mean:long_name = "10 minute mean relative humidity 1.9m above the surface" ;
rh2mean:units = "percent" ;
rh2mean:missing_value = 99999.9f ;
float rh2max(time) ;
rh2max:long_name = "10 minute maximum relative humidity 1.9m above the surface" ;
rh2max:units = "percent" ;
rh2max:missing_value = 99999.9f ;
float rh2min(time) ;
rh2min:long_name = "10 minute minimum relative humidity 1.9m above the surface" ;
rh2min:units = "percent" ;
rh2min:missing_value = 99999.9f ;
float rh2std(time) ;
rh2std:long_name = "10 minute standard deviation of relative humidity 1.9m above the surface" ;
rh2std:units = "percent" ;
rh2std:missing_value = 99999.9f ;
float dew2mean(time) ;
dew2mean:long_name = "10 minute mean dew point temperature 1.9m above the surface" ;
dew2mean:units = "degrees Celsius" ;
float dew2max(time) ;
dew2max:long_name = "10 minute maximum dew point temperature 1.9m above the surface" ;
dew2max:units = "degrees Celsius" ;
float dew2min(time) ;
dew2min:long_name = "10 minute minimum dew point temperature 1.9m above the surface" ;
dew2min:units = "degrees Celsius" ;
float dew2std(time) ;
dew2std:long_name = "10 minute standard deviation of dew point temperature 1.9m above the surface" ;
dew2std:units = "degrees Celsius" ;
float surftmean(time) ;
surftmean:long_name = "10 minute mean surface temperature" ;
surftmean:units = "degrees Celsius" ;
float surftmax(time) ;
surftmax:long_name = "10 minute maximum surface temperature" ;
surftmax:units = "degrees Celsius" ;
float surftmin(time) ;
surftmin:long_name = "10 minute minimum surface temperature" ;
surftmin:units = "degrees Celsius" ;
float surftstd(time) ;
surftstd:long_name = "10 minute standard deviation of surface temperature" ;
surftstd:units = "degrees Celsius" ;
// global attributes:
:title = "Sea Ice Experiment: Dynamic Nature of the Arctic (SEDNA) Weather Station Data" ;
:institution = "International Arctic Research Center" ;
:source = "surface weather measurements" ;
:references = "Hutchings et al, 2007: SEDNA Data Report, International Arctic Research Center" ;
:comment = "SEDNA WEATHER STATION DESCRIPTION: \n",
" \n",
"Instrumentation: \n",
" 03101 R.M. Young Wind Sentry Anemometer \n",
" 03301 R.M. Young Wind Sentry Vane \n",
" Vaisala HMP45C-L Temperature and Humidity Probe housed in \n",
" Campbell Scientific 41003-5 10-Plate Gill Radiation Shield \n",
" IRR-P Apogee Precision Infrared Temperature Sensor \n",
" CS1000 Campbell Scientific Barometric Pressure Sensor \n",
" CR1000 Campbell Scientific Data Logger \n",
" LLAC4 Campbell Scientific AC Conversion Module \n",
" \n",
"Three-level configuration: \n",
" *3.1meters above ground: \n",
" Temperature, Humidity, Wind speed and direction \n",
" *1.9meters above ground: \n",
" Temperature, Humidity, Wind speed and direction \n",
" *Ground level: \n",
" Downard looking Infrared Radiometer mounted 1 meter above \n",
" ground with ~0.8 meter (22 degree half angle) footprint \n",
" diameter over snow and sea ice surface detecting surface \n",
" temperature. \n",
" \n",
"Weather station site: \n",
" Weather station was sited >170 meters from camp structures \n",
" on flat multiyear sea ice having about 10cm mean snow \n",
" depth. Site was approximately 400 meters from GPS position \n",
" reported in this file. This distance remained constant \n",
" during the timeseries represented in this file. \n",
" \n",
"Weather station sampling procedure: \n",
" The following were sampled every 5 seconds and output means \n",
" every 10 minutes: Battery voltage & electronics temperature. \n",
" The following were sampled every 10 minutes and output as \n",
" samples: barometric pressure & wind direction. \n",
" The following were sampled every 5 seconds and output \n",
" means, maximums, minimums and standard deviations every \n",
" 10 minutes: wind speed, temperature & humidity. \n",
" \n",
"GPS sampling and wind direction calculation procedure: \n",
" GPS positions provided in this file are from the \n",
" receiver located at a constant distance of ~400m from the \n",
" weather station, sampling position every 10 seconds. \n",
" This has been subsampled with linear interpolation to the \n",
" station sampling times in this file. Wind direction has been \n",
" calculated using the two GPS receivers each 200m from camp and \n",
" in a fixed triangle with the weather station. Estimated error \n",
" in wind direction using this GPS \"compass\" is +/-10 degrees. \n",
" \n",
"" ;
:Conventions = "CF-1.0" ;
:history = "22-Oct-2007 18:24:07: File created by Andrew Roberts" ;
- Andrew Roberts, October 2007
