Error covariance

Error Covariance Matrices


The error covariance matrix is a dataset that specifies the correlations in the observation errors between all possible pairs of vertical levels. It is given as a two-dimensional array, of size NxN , where N is the number of vertical levels in the sounding data products.

Specifically, the ECM for variable X is given by E[(X - E[X])(X - E[X])T], where in practice the expectation value is estimated by an average over a representative sample.  It is a matrix because levels are treated independently: X is a vector of length equal to the number of vertical levels.

X could be an observation (refractivity N or bending angle α), in which case the ECM is sometimes called O, or a background state ({T, q, p*} for ECMWF-like models; {p, q} for Met Office-like models), in which case the ECM is sometimes called B. Both need to be specified in the minimisation of the usual 1dvar cost function

J(x) = ½(x-b)TB-1(x-b) + ½(H(x)-y)TO-1(H(x)-y))

which is needed to produce a retrieved analysis x.

ECMs are positive semi-definite matrices which can be factorised as

ECM = DT C D

where the diagonal matrix D comprises the standard deviations of X, and defines the "size" of the variance, while the correlation matrix C describes the "shape" of the variance of X. By definition of standard deviation, the diagonal terms of C are all equal to 1.

The ROPP user has a choice of whether to use fixed standard deviations D, read from an auxiliary covariance file, or to allow them to vary profile-by-profile, by including them in the observation or background file. At present, correlations C can only be read from the same auxiliary file. It is the content of these auxiliary covariance files which is descibed here.

In ROPP, the correlations in the covariance files are stored in packed, "triangular" format, so that successive elements of the variable "corr" hold the following matrix elements:

1 2 4 7 ...
3 5 8 ...
6 9 ...
10 ...
...

By exploiting the symmetry of the correlation coefficients, only ½N(N+1) of them need to be stored. 

The correlation matrices can be binned (in the same file) according to latitude. In principle this could be extended to seasonality. Standard deviations can also be held in the same file.  These, rather than the profile-by-profile SDs, wil be used in the retrieval if  bg/obs_covar_method = FSFC or FSDC.

Further details on all the above can be found in the ROPP FM and 1DVAR User Guide.

There are two basic error-covariance matrices:
  • Covariance matrix for NRT sounding data products
  • Covariance matrix for Offline sounding data products
Each matrix can be provided in several versions reflecting variations with geographical areas (e.g. latitude) and with season.

 NRT STATUS:
 Released


Description Image (click to expand) File
Background (courtesy J Nielsen, DMI)

Suitable for 91L ECMWF background state.

Currently, the T and p* standard deviations are the ones shown in the file.  The standard deviation of q is calculated profile-by-profile, using the background q multiplied by a fixed <dq/q>.  These standard deviations are then used to calculate the covariances using the given correlation matrix as ECM = D C D.

The block diagonal structure of correlation matrix arises from use of a 3 element state vector: {T, q, p*}.

NRT_bg_cov DMI_ECMWF91L.nc
Observations (courtesy J Nielsen, DMI)

Suitable for 175L refractivity retrievals. (Bending angle correlation matrices are usually assumed to be diagonal.)

Note that there is a minimum (N) of 0.02 N-units above about 25km.

NRT Obs cov DMI_N175L.nc



 Offline STATUS:
 Development
Some example ECMs are available in the ropp_1dvar/errors directory of the ROPP distribution, for the benefit of the users.  They are descibed below.

Description Image (click to expand) File
Background (author H Lewis, Met Office, UK)

Suitable for (early) 60L ECMWF background state.

Note that correlation matrix assumes level 1 at the top of the atmosphere.

ROPP_60L_ECMWF ropp_bg_ecmwf_error_corr_L60.nc
Background (authors H Lewis and I Culverwell, Met Office, UK)

Suitable for (current) 91L ECMWF background state.

Note that correlation matrix assumes level 1 at the bottom of the atmosphere.

ROPP_91L_ECMWF ropp_bg_ecmwf_error_corr_L91.nc
Background (courtesy H Lewis, Met Office, UK)

Suitable for (early) 50L Met Office background state {p, q}.

Three latitude bands: -90 to -20, -20 to 20, 20 to 90N.















ROPP_50L_METO_SH

ROPP_50L_METO_TR

ROPP_50L_METO_NH
ropp_bg_meto_error_corr_L50.nc
Background (courtesy C Burrows, Met Office, UK)

Suitable for (current) 70L Met Office background state {p, q}.

Three latitude bands: -90 to -20, -20 to 20, 20 to 90N.











ROPP_70L_UKMO_TR

ROPP_70L_UKMO_TR

ROPP_70L_UKMO_NH
ropp_bg_meto_error_corr_L70.nc
Observation (author= "ROPP")

Suitable for 300L bending angles. (Note that C is the identity matrix).



ROPP_300L_bangle ropp_ob_bangle_error_corr.nc
Observation (author= "GRAS SAF / ROPP v 4.0")

Suitable for standard 247L refractivities.

ROPP_247L_REFRAC ropp_ob_refrac_error_corr.nc

DMI's logo IEEC's logo Met Office's logo
ECMWF's logo