SAF/GRAS/METO/SRN/ROPP/001 - Version 6.0 - 31 October 2011

Software Release Notes for ROPP-6 (v6.0)

Contents

1. Introduction
2. Release file set
3. Previous versions
4. Third-party software dependencies
5. Unpacking and building ROPP
6. Building ROPP manually
7. Testing
8. Portability 
9. Known problems
10. Limitations
11. And finally...

1. Introduction

This document outlines the sixth major release of the Radio Occultation Processing Package (ROPP) Version 6.0 - hereafter referred to as 'ROPP-6'. It describes the file set, documentation, dependency packages, a summary 'Quick Start' guide to build/installation, a list of supported platforms and known problems.

ROPP is a collection of software modules (provided as source code), supporting build scripts, data files and documentation. The complete package is split into several modules. Users may wish to integrate a subset of ROPP code into their own software applications, individually linking modules to their own code. These users may not require the complete ROPP-6 distribution package. Alternatively, users may wish to use the executable tools provided as part of each module as stand-alone applications for RO data processing. These users should download the complete ROPP-6 release. Section 5 has guidance for unpacking and building ROPP.

The download website is organized as:

  ROPP Download (root) page
  |-- ROPP-6
      |-- Documents
  |-- Dependency Packages

The root directory contains the Release Notes (this file), Change Log for this and previous releases and a compressed tar archive file (ROPP-6.0.tar.gz) which contains the complete release for ROPP-6.

Directory ROPP-6 contains compressed tar archive files for the individual components (source and build modules) for this release. These are provided should users wish to access individual modules rather than download the complete package. Related documentation is also available separately in the Documents sub-directory. Third-party dependency packages (not part of the ROPP distribution and for those whose licences permit it) can be downloaded from the Dependency Packages directory. See the README files in these directories for more information.

ROPP-6 IO, PP, FM and 1DVAR modules have been fully validated using operational Level 1b and Level 2 GRAS data. As such, it is guaranteed for operational use. All parts of this release have been tested against CHAMP, GRACE-A, COSMIC and GRAS data.  In addtion,  limited samples of data from TerraSAR-X, C/NOFS and SAC-C have been tested with the BUFR encoding and decoding tools and appropriate GFZ-to-ROPP and UCAR-to-ROPP converters.

Almost all files described here can be downloaded by registered and logged-in users from this website (http://www.grassaf.org > User Login > ROPP > ROPP Download). Any exceptions are noted below.

Before being able to download any ROPP-6 files, you will have accepted the Terms and Conditions of the full ROPP Software Licence, a copy of which is available in the ROPP-6 directory for reference. Note also the contents of the COPYRIGHT file.

All comments on the ROPP software should, in the first instance, be reported via the GRAS SAF Helpdesk at http://www.grassaf.org.
top

2. Release file set

The 'Status' column in the following tables shows the current status of files since the previous release (v5.0), by a traffic light system:

A summary of the main changes to this package (v6.0) from the previous release (v5.0) are documented in the Change Log.

2.1 ROPP root directory

2.2 ROPP-6 directory

For users not requiring the complete distribution package, this directory contains the component module files:

Note that the collective content of these separate module tarfiles is contained in the main tarfile in the root directory.

2.3 ROPP-6/Documents directory

The ROPP User Guide (in three parts) and documentation on ROPP code (Reference Manuals), file formats (netCDF and BUFR), Technical Notes and other related items, can be found here.

Please refer to the Overview document for the general content of the individual modules, and to the User Guide for details of the installation and usage of the package.

For users wishing to write their own interfaces to the ROPP routines, or to modify the ROPP code, detailed user-documentation in the form of Reference Manuals, one for each module, are provided; these Reference Manuals are also contained in the main ROPP tarfile. Stand-alone application tools also have Unix-style 'man' pages which are installed during the ROPP module building process.

2.4 Dependency Packages directory

ROPP uses some standard third-party packages. Most of these packages are open source and their licences permit re-distribution. Where licences permit, the latest stable versions of these packages known to be compatible with ROPP are available for download here as a convenience to users.

Updated versions may become available from the noted source websites; we do not guarantee that these will work with ROPP. If later versions are proved to be compatible, the new distribution files will be posted here.

1. netCDF-4 is the current release family; ROPP uses the 'classic' netCDF model, for which netCDF-4 is equivalent to netCDF-3. As from ROPP-6 v6.0, building netCDF-3 (v3.x.x) from source is no longer supported, but any release from v4.0.0 to v4.1.3 can be built for use with ROPP, as long as the option to build a separate Fortran object library is used.
   
2. The latest Met Office BUFR library is bufr-19.1. Any v19.x release is required to build the BUFR tools in ROPP-6 as earlier v18.x releases are incompatible. v19.0 brings support for encoding BUFR Edition 4 messages and includes some recently ratified RO satellite IDs to the code tables; v19.1 include support for additional compilers.
   
3. The ROPP-6 BUFR encoder and decoder can be used with either the Met Office or ECMWF BUFR libraries but only one need be installed. The ROPP build system will detect which is present and make the executable with the appropriate source code. If both libraries are installed, the MetDB one will be used in preference.
4. The ECMWF BUFR C code may fail to compile or run the package tests with certain versions of gcc under Cygwin. Such failures should be reported to ECMWF, not the GRAS SAF.
   

Sample build and configuration scripts are available to assist in building these packages consistently for use with ROPP; see Section 5.4.
top

3. Previous versions

If you have previously installed v1.x, v2.0, v3.0, v4.x or v5.0 we recommend that you install v6.0 to a separate target path - do not just install v6.0 over an existing installation. This is especially important if the v5.0 (or earlier) tools or applications built on those libraries are being used for operational production. Only swing over to using v6.0 tools and libraries when you have validated them locally and you are satisfied that they meet your requirements.

Support for v4.x and v5.0 will be limited to problems which can be reproduced also with v6.0. If reported problems with v4.x or v5.0 have been already fixed at v6.0, we recommend upgrading to v6.0. ROPP Versions v1.x, v2.0 and v3.0 are no longer supported.

All previous Beta release versions of ROPP, up to and including v6.0-beta, are obsolete and are completely unsupported. If you have previously installed any version of ROPP for Beta testing, under the terms of the Beta licence (to which you agreed when first requesting access to the software) the right to use this software terminated after one year. Any and all ROPP Beta files downloaded for testing purposes are unsupported and should be deleted.
top

4. Third-party software dependencies

Some third-party packages used by ROPP-6 are noted above and elsewhere on this website. How to obtain this code is detailed in the README.deps, ROPP Overview and User Guide documents. Where licences permit, the latest versions known to be compatible with ROPP can be downloaded from this website - see Section 2.4.

Object library and header, include and module files generated from the third-party source code should be placed in the path being used to build ROPP. Alternatively, soft links to their actual locations may be placed in this path. See Section 5 for installation tips. Unless existing installations are known to have been compiled with the same compiler and options as being used for ROPP, we strongly recommend building these dependency packages specifically for ROPP.
top

5. Unpacking and building ROPP

This section provides guidance on building the complete ROPP package as a standalone installation for evaluation and testing or to build the application tools for local use. ROPP is designed to be highly modular and key components of the software may be extracted for embedding in user applications via user-callable application program interface (API). In the latter case, the 3^rd party dependency packages may not be required. See the User Guide for guidance on use of the ROPP main APIs.

5.1 Unpacking

For simplicity, we recommend downloading the main ropp-6.0.tar.gz compressed tar archive (approx. 25Mb) which contains the complete ROPP-6 (v6.0) package set. Download (or copy) this file to a suitable directory, where the content can be extracted with:

> tar -zxvf ropp-6.0.tar.gz

> gunzip ropp-6.0.tar.gz
> tar -zxvf ropp-6.0.tar.

This will create a sub-directory with the following structure:

ropp-6.0
|- configure
|- ropp_1dvar
|- ropp_fm
|- ropp_io
|- ropp_pp
|- ropp_utils

The ropp-6.0 top directory contains additional README files and example build scripts and the configure sub-directory contains a number of mini-scripts for local build configuration. The other ropp_* sub-directories contain the source code, makefiles, test data, etc., for each of the main ROPP modules.

Alternatively, if the complete package is not required, the user can manually create the ropp-6.0 directory (or one with any other suitable name), download the individual module tarfiles from the website's ROPP-6 directory to it and unpack them here. For example:

> mkdir -p $HOME/ROPP/ropp-6.0
> cd $HOME/ROPP/ropp-6.0
...download or copy required tarfiles here...
> tar -zxvf ropp_build-6.0.tar.gz
> tar -zxvf ropp_utils-6.0.tar.gz
...etc

Note that unpacking ropp_build will create the configure sub-directory and build scripts as above. Since the buildpack script will unpack a module's tarfile if the source tree is not found, it is not necessary to manually unpack the other ROPP modules if using this tool.

We suggest that any required dependency packages also be downloaded here; they can then be unpacked and built using the buildpack tool before building the ROPP modules

5.2 Preparing to build the ROPP code

The full package tarfile contains a plain-text file called MANIFEST which lists all of the files in the distribution. The individual module tarfiles each contain a similar file for their own module.

Third-party dependency libraries must be installed before attempting to build any ROPP code. The configure sub-directory contains some configuration mini-scripts and there are higher-level shell scripts to assist in building the netCDF and BUFR (both Met Office and ECMWF ) packages. Review and edit these to suit your local requirements before running them. Refer to the README.build, README.compilers, README.unix and README.cygwin files included in the package for  further information.

Any third-party packages already installed (e.g. for ROPP v4.x or v5.0 or for other applications) do not normally need to be re-built but should be in the expected ROPP build path for libraries, as real files or as softlinks. Note that this release may require a minimum version of a dependency package. Third-party packages may have been updated since the previous release of ROPP; those available on this website in the Dependency Packages directory are the latest versions supported by ROPP and the sample build scripts. We recommend installing these latest versions, but in general it is not necessary to do so. Rebuilding is also recommended if your C or Fortran compiler version has changed since the original build.

It is highly recommended that third-party dependency packages are compiled using the same Fortran compiler being used for the ROPP code. Mixing code at object level, even using different versions of compilers from the same vendor, can cause linking or run-time problems. C code in the dependency packages (there is none in the ROPP package) must be compiled using the cfortran.h system or equivalent to ensure calling interface compatibility with the Fortran compiler's object code, which means that the package must be configured for use with a particular Fortran interface, even if the package itself contains no Fortran.

The example configure/build scripts use the environment variable ROPP_ROOT as the default installation path, and this should be set prior to running these scripts. See Section 6 for an example.

5.3 Environment variables

> export ROPP_ROOT=/usr/local

The ROPP BUFR encoder and decoder may be interfaced with either the Met Office 'MetDB' or ECMWF BUFR kernel libraries. A BUFR library is optional, and if neither is present, the encoder and decoder application programs in the ROPP_IO module will not be built. If both libraries are detected, the MetDB one will be used in preference. If a BUFR library is to be used, the appropriate environment variable must be pre-set to the directory path for the run-time tables. For instance, if using the Met Office 'MetDB' library:

> export BUFR_LIBRARY=$ROPP_ROOT/data/bufr/

5.4 Example configuration scripts

The ropp_build module (content included within the main tarfile) contains example build and configure shell mini-scripts for installing not only the ROPP code but also the dependent third-party packages in a compatible way. These files are extracted into the configure sub-directory and follow the naming style <package>_configure_<compiler>_<os> where <package> is the package name (ropp, netcdf, ecbufr...), <compiler> is the compiler ID (ifort, nag, pgf, g95, gfortran, xlf95...) and <os> is the operating system ID, as output by the uname(1) command but entirely in lower case (linux, cygwin, hp-ux, sunos, aix...). These configure mini-scripts are used by the high-level buildpack script.

The example configure scripts for specific platforms and compilers may need to be edited for optimal local use, or users may create their own following one of the examples. We would be pleased to receive back equivalent scripts if you have built the packages using different compilers and/or platforms and which could be included in future updates. We can't formally support such scripts since their use is highly dependent on the user's target system and compiler setting preferences and we have no means of testing them ourselves.

5.5 BUILDPACK script

Included in the main package (and in ropp_build) is a BASH shell script file buildpack. The user need not be using Bash as an interactive shell, but Bash needs to be installed (usually at /bin/bash) for the script to work; almost all POSIX-based systems should have Bash installed by default, and for many Bash is the login shell. On the rare systems that do not have Bash installed, follow the main commands for the desired package installation section of this script after reading Section 6.

This script may be used to automate the build of any ROPP module or dependency package in a consistent way, using the above mentioned configure scripts. Summary usage can be obtained by:

> buildpack -h

This script assumes that all archive (tar & zip) files are placed in the same working directory and the configure scripts are in a configure sub-directory. Packages will be decompressed here and resulting files installed to the ROPP_ROOT/<compiler> target directory, as described in Section 6. In general, to build and install a package, it is only necessary to give the command:

> buildpack <package> <compiler> [[NO]CLEAN]

where <package>is one of the supported ROPP module or package names (ropp_io, netcdf, mobufr, ecbufr...) and <compiler> is the required compiler (ifort, nag, g95, gfortran, xlf95...) which matches an appropriate configure script. The <os> part is detected automatically. 

The script's main steps are:

1. check that ROPP_ROOT has been set (or is given a default), and that it exists (or is created) and the user has write access
2. detect the latest (or only) existing source installation directory (decompressed archive) for the package
3. optionally delete the tree to force a clean build (if CLEAN is present)
4. if no source tree is present (first install or deleted at step 2), detect the latest archive file (compressed tar or zip file) and extract the contents
5. if starting from an existing installation directory , clean it (or optionally skip this step if NOCLEAN is present)
6. for those packages requiring a configure step, run the appropriate mini-configure script if found
7. run the appropriate build and install command(s) (usually make and make install)
8. for packages providing the facility, run test procedures (usually make test)

Since some modules are dependent on other modules, the recommended build order is:

1. netCDF
2. BUFR   (optional - one of Met Office or ECMWF packges)
   
3. ROPP_UTILS
4. ROPP_IO
5. ROPP_PP
6. ROPP_FM
7. ROPP_1DVAR

Note that this Bash script has only been tested under Linux (RHEL4, RHEL6 and OpenSUSE 11), IBM AIX and PC/Windows (Cygwin) and may need minor modifications to run on other systems. See README.build for more information.

5.6 Other build scripts

Other shell wrapper scripts build* are provided which can be used to further automate the build process by calling buildpack with a pre-determined sequence of packages or compilers. Review and edit to suit your requirements.
top

6. Building ROPP manually

The low-level build sequence is described in the User Guide (see also README.unix), but in summary, consists of the common configure / make / make install commands. If not using the supplied buildpack script, this section gives some guidance on the manual procedure.

We suggest that a ROPP 'root' directory be created in a suitable place, for example:

> cd <directory-of-your-choice>
> mkdir ropp-6.0
> cd ropp-6.0
> export ROPP_ROOT=$PWD

though of course ROPP_ROOT can be set to any existing directory to which the user has write access.

As provided, the configure scripts will install the package files to:

where <compiler> is an ID (normally the command line name) for a particular Fortran compiler. If only one compiler is to be used, the appropriate configure script can be edited to remove this extra directory layer.

Then copy the required tar files here and unpack, e.g.:

> tar -zxvf ropp_build-6.0.tar.gz
> tar -zxvf ropp_utils-6.0.tar.gz

Select a suitable configure script from the configure sub-directory. E.g. if using the Intel ifortcommand (on our RHEL6 systems this defaults to v12) compiler under Linux this would be ropp_configure_ifort_linux. Review/edit this file to suit your local installation directories and compiler flags. In this example, by default, the ROPP libraries, include, F90 module files, etc will be installed into the ROPP_ROOT/ifort tree and the configure will expect similar pre-built third-party dependency files to be found here also.

To build the ROPP_UTILS library:

> cd ropp_utils-6.0
> ../configure/ropp_configure_ifort_linux
> make
> make install
> make test

Repeat for any other ROPP modules you wish to use. If your application only requires (say) the I/O support (e.g. only the BUFR encoder/decoder tools are wanted), then it is not necessary to build the PP, FM or 1DVAR modules. Alternatively, if only the PP, FM or 1DVAR callable-API routines are to be implemented in a user's own applications, then the IO module (and netCDF support) need not be built. The configure script will then only build those parts of the ROPP code with no dependence on ROPP_IO (i.e. stand-alone tools with netcdf read/write are not built).

Note that to use the ROPP BUFR encoder and decoder, the appropriate environment variable must be pre-set to the desired directory path to install and use the run-time tables.

7. Testing

All ROPP modules have a basic user-test option; this is not intended to be a comprehensive test of the software, but just to demonstrate that the package has been correctly installed and programs generate data which can be validated against a reference dataset (provided). The tests can be run by:

> cd tests
> make test

The screen output (and in some cases log files or graphics) should be reviewed as to whether the test(s) were successful or not. Example log files and graphics are provided to show the output for a successful test. Graphical output is only generated if IDL is installed. (Tip:

> make test > test.log 2>&1

to save a log file which can be examined at leisure.)

If using the buildpack script, testing will be performed automatically if the build process completed successfully.

Some of the tests (notably in ROPP_PP, ROPP_FM and ROPP_1DVAR) generate graphics files using the IDL system. If IDL is not installed, the test scripts will note the fact, and skip the graphics generation and display. Obviously in this case, the user will not be able to visually verify the tests. We are investigating using an alternate graphical tool, such as Python+matplotlib, for a future release.  If ROPP is able to generate and display graphical results, each test result and a comparison reference figure (part of the distribution) will be displayed simultaneously.  By setting the environment variable $ROPP_PAUSE to TRUE, the user can examine the two figures at leisure, before allowing the test script to move on to the next figure by hitting any key.

top

8. Portability

A prime requirement of ROPP is that it must be portable so as to compile and run under a range of compilers and platforms. Nevertheless, ROPP can only be supported on platforms also supported by the required third-party packages. In practice this means POSIX-compliant systems (Unix, Linux, MS Windows/Cygwin, SunOS, OS/X...) and ISO-compliant F95 & C compilers. ROPP depends on some F95 features, so older F90-only compilers cannot be supported. In addition, the ROPP code can only be fully validated only on platforms and with compilers available to the Development Team.

Target compiler/platform combinations we aim to support are listed in the ROPP Overview document. However, difficulties have been encountered with certain compilers, mostly due to explicit or implicit bugs in the compilers themselves, or some limitations in their functionality or interpretation of Fortran standards. Work is on-going to investigate work-arounds, though it is not our policy to provide different code for different compilers just because certain compilers do not adhere to accepted standards or have internal bugs.

For this release, the following OS/compiler combinations have passed all of our comprehensive internal testing ('Test Folder' system) and so should work on similar systems elsewhere. Unless noted, this release of ROPP works with netcdf-4.1.3, MetDB  bufr-19.1and ECMWF bufr_000387. The RHEL4 platform is 32-bit OS and compilers while RHEL6 is 64-bit OS and compilers (unless noted). Future releases of ROPP will not be tested on RHEL4.

For this release, the following OS/compiler combinations have failed some element of our comprehensive internal testing ('Test Folder' system):

In addition, all dependency code and ROPP modules have been successfully built and passed basic user-tests (as provided in the distribution tarfiles) under the following OS/compiler combinations:

The Linux builds have been successfully re-created on a 64-bit laptop (Intel Core2 Duo T9400) and the Cygwin builds are also successful under a 32-bit Windows XP Pro setup.

1. The ECMWF example test program fails to run to completion using the Portland Group compilers, but nevertheless generates correct BUFR files and ROPP tests also pass.
   
2. A C routine in the ECMWF low-level I/O interface code fails to compile on this particular version of gcc on Cygwin. The problem appears to be related to a missing #define in a standard system header file, which the ECMWF code expects to be set, but isn't. The problem occurs on both 64- and 32-bit versions of gcc on Cygwin Hence ROPP has not been tested with the ECMWF BUFR library on this platform. The simplest work-around is to use the MetDB BUFR library.
3. Due to non-POSIX-style command line syntax and file naming conventions, native Windows compilers such as Intel, Salford and others - though perfectly good compilers - generally do not work with POSIX-style configure scripts (as used by netCDF and ROPP) under Cygwin, so are not supported.

top

9. Known problems

• Despite being the focus of much of the development work in going from ROPP-5 to ROPP-6, there remain occasional unresolved differences between the bending angles (and output geopotential heights) generated by the pre-processing module of ROPP, ropp_pp, and the OCC pre-processing code on which it is based, and which it is intended to emulate. The reasons for this are still being investigated, but mean differences in bending angle are usually less than 0.1% and the large majority of profiles are essentially identical. The discrepancy is not considered critical.
  
• As noted above, the ECMWF BUFR library may not fully build on Cygwin with the default gcc. Since this package is supported by ECMWF, such problems should be reported there, not to the GRAS SAF. The suggested work-around is to use the MetDB BUFR library. Upgrading to the very latest version of gcc (either a separate Cygwin binary build or from the latest source) may cure the problem, but we have not tried this ourselves.
• Recent validation testing has revealed occasional difficulties when using the logp or logq options with the default minropp minimization scheme in ropp_1dvar. Until these are resolved, we recommend using minropp without the logp,q options, or the Levenberg-Marquardt minimiser (with or without the logp,q options). The two minimisers are compared and contrasted in GRAS SAF Report 11, which the user should consult for further information.
  top

  ---

  10. Limitations

  • ROPP is being implemented in a phased approach, and while all functionality planned for ROPP-6 has been implemented, not all of the total planned functionality for the package is yet available. See the ROPP Overview document for more details.
  • Support for previous versions will be limited to problems which can also be reproduced with the latest version. We recommend that users should upgrade to the latest version when released. In general, only the current (viz ROPP-6) release is fully supported with the immediately preceding releases (viz ROPP-5) having limited support.
  • ROPP supports writing of more than one RO profile to a netCDF file. It is assumed that the number of data points in the first profile is sufficient to hold all subsequent profiles to be written to the file (this is a fundamental limitation of saving 2D arrays in netCDF). As a consequence, should the first file written to a multifile have no valid samples in the profile (say, all refractivities are invalid and set missing), then that 'Level 2a' section will be written with zero data points. Subsequent profiles having at least one valid sample will then not be correctly saved in the multifile. See the ROPP User Guide Part I for more details.
  • All ROPP stand-alone tools use the ROPP_IO routines to read data from input ROPP format netCDF files. By default, the data are checked against their valid range attributes, and each data level tested for the availability of valid height (or time) values. All input files are therefore required to have valid time, impact parameter or geopotential height information to pass the range checks on reading for Level1a, Level1b and Level2 data respectively. See the ROPP User Guide Part I for more details.
  • ROPP_PP contains routines to process GRAS 'Raw Sampling' (RS) data (strictly, this is full-sampling closed loop data). While this code works, the pre-processing of GRAS RS data itself is still in development by EUMETSAT, file formats are liable to change, and the ROPP_PP code has not been extensively tested. The GRAS RS functionality in the ROPP_PP code therefore has only 'unvalidated prototype' status.
    

  top

  ---

  11. And finally...

  Data files, documentation, etc, in this release are not definitive and not all final intended functionality may be present, since ROPP is being developed and released in planned stages. The software is not guaranteed to be bug-free, but has been tested with operational GRAS data (Level 1b from EUMETSAT and GRAS SAF Level 2 GRM-01 Refractivity products) and also with COSMIC, CHAMP, GRACE-A (and to a limited extent, TerraSAR-X, C/NOFS and SAC-C) occultations. Nevertheless, the code - with the portability and other limitations noted above - have passed a defined set of tests and the complete package has been formally reviewed and approved for general release as software having operational status.
  
  Nevertheless, we encourage you to carefully check the building and installation and test the running of the ROPP software and to review the package in its entirety and to provide feedback to the ROPP Development Team.
  
  Updates to the ROPP code to:

  • fix any reported bugs
  • extend portability
  • improve the user-build experience
    
  • enhance functionality
  • improve the science

  will be released from time to time as minor or major releases. Registered users will be notified when an update is available. This may be in the form of a complete replacement file, a patch file or - in the case of trivial changes - instructions for local file editing. Implementation of such updates is at the discretion of the user, but support for older releases will become increasingly limited.
  
  All comments on the ROPP software should, in the first instance, be reported via the GRAS SAF Helpdesk at http://www.grassaf.org > Helpdesk > New enquiry.

  
  top

  ---

  GRAS SAF ROPP Development Team
  http://www.grassaf.org