Skip to content

Data

Data

The top-level component data contains an array of data sets in struct format. Each data set needs to contain the components type and name. Other components are dependent on the type of data set as demonstrated below:

  • name: custom string
  • type: string that determines the format of the observations
  • ...: each type of observations has different parameter keys. Some of these are optional and marked accordingly in the more detailed description below

A detailed description of the different types with examples can be found below. While data, in most settings, has no uncertainty attached to it, this standard does allow to provide data with uncertainty, as there are some settings in which uncorrelated or correlated errors need to be taken into account. These include cases such as

  • generated data obtained from importance sampling, including a (potentially Gaussian) uncertainty from the frequency weights
  • unfolded data, resulting from arbitrarily complex transformation functions involving statistical models folding some degree of uncertainty into the data points themselves

While it should always be preferred to publish "raw" data, allowing to include pre-processed data with corresponding uncertainties expands the possible applications considerably.

Data sets associate their contents with variables via axes. Depending on the type of data set, these are continuous, binned or categorical axes, as specified in Axis Specifications.

Forbidden attributes: The const attribute must not appear in the axes of data sets. All axes of data sets define observable domains, never fit parameters.

Point Data

Point data describes a measurement of a single number, with a possible uncertainty (error).

  • name: custom string
  • type: point
  • value: value of this data point
  • axes: (optional) array of continuous axes as defined in Continuous Axes. When present, associates the point measurement with a specific observable domain.
  • uncertainty: (optional) uncertainty of this data point
Example: Point Data
"data":[
    {
        "name":"data1",
        "type":"point",
        "value":0.,
        "uncertainty":1.,
        "axes":[
            { "name":"obs_x", "min":-5, "max":5 }
        ]
    }
]

Unbinned Data

Unbinned data describes a measurement of multiple data points in a possibly multi-dimensional space of variables. These data points can be weighted.

  • name: custom string
  • type: unbinned
  • entries: array of arrays containing the coordinates/entries of the data. For a categorical axis, the corresponding column holds the respective row's category as one of the strings listed in categories.
  • axes: array of continuous or categorical axes as defined in Axis Specifications.
  • weights: (optional) array of values containing the weights of the individual data points, to be used for \(\chi^2\) comparisons and fits. If this component is not given, weight 1 is assumed for all data points. If given, the array needs to be of the same length as entries.
  • entries_uncertainties: (optional) array of arrays containing the errors/uncertainties of each entry. If given, the array needs to be of the same shape as entries.

Example: Unbinned Data
"data":[
  {
    "name":"data1",
    "type":"unbinned",
    "weights":[ 9.0, 18.4 ],
    "entries":[ [1,3], [2,9] ],
    "entries_uncertainties":[ [0.3], [0.6] ],
    "axes":[
      { "name":"variable1", "min":1, "max":3 },
      { "name":"variable2", "min":-10, "max":10 }
    ]
  }
]
Unbinned data sets with a categorical axis store the category string in the corresponding column of entries:
Example: Unbinned Data with categorical axis
"data":[
  {
    "name":"data2",
    "type":"unbinned",
    "weights":[12.0, 4.3, 2.1],
    "entries":[["SR",7,0.1], ["SR",9,-0.2], ["CR",4,0.9]],
    "axes":[
      {"name":"cat1", "categories":["SR","CR"]},
      {"name":"variable1", "min":3, "max":20},
      {"name":"variable2", "min":-1, "max":1}
    ]
  }
]
Here the first column of entries refers to the cat1 axis, giving the category of each row directly as "SR" or "CR".

Binned Data

Binned data describes a histogram of data points with bin contents in a possibly multi-dimensional space of variables. Entries that fall precisely on an interior bin boundary may be assigned to either of the two adjacent bins, at the discretion of the creator of the document; this standard does not define which. Both bounds of an axis are inclusive: entries at exactly the lower bound of the axis belong to the first bin and entries at exactly the upper bound of the axis belong to the last bin, so that no entry inside the bounds is lost. Consequently, consumers that re-bin unbinned data should not expect bit-identical histograms across implementations.

  • name: custom string
  • type: binned
  • contents: array of values representing the contents of the binned data set, flattened as defined in Flattening of Binned Contents
  • axes: array of binned or categorical axes as defined in Axis Specifications
  • uncertainty: (optional) struct representing the uncertainty of the contents. It consists of up to three components:
    • type: denoting the kind of uncertainty, for now only Gaussian distributed uncertainties denoted as gaussian_uncertainty are supported
    • sigma: array of the standard deviation of the entries in contents. Needs to be of the same length as contents
    • correlation: (optional) array of arrays denoting the correlation between the contents in matrix format. Must be of dimension length of contents \(\times\) length of contents. It can also be set to 0 to indicate no correlation.
Example: Binned Data
"data":[
  {
    "name":"data3",
    "type":"binned",
    "contents":[ 9.0, 18.4 ],
    "axes":[ { "name":"variable1", "nbins":2, "min":1, "max":3 } ]
  },
  {
    "name":"asimov_data3",
    "type":"binned",
    "contents":[ 9.0, 18.4, 13, 0. ],
    "axes":[
      { "name":"variable1", "nbins":2, "min":1, "max":3 },
      { "name":"variable2", "edges":[0,10,100] }
    ]
  }
]

The contents array is flattened as defined in Flattening of Binned Contents: the last axis listed in axes varies fastest, so in example asimov_data3 above the values belong to the bins \((i_0, i_1) = (0,0), (0,1), (1,0), (1,1)\), in this order. A categorical axis with \(n\) categories behaves like a binned axis with \(n\) bins, the bin index being the position of the category in categories.

Example: Binned Data with categorical axis
"data":[
  {
    "name":"data4",
    "type":"binned",
    "contents":[9.0, 8.4, 4.0, 7.0],
    "axes":[
      {"name":"channel", "categories":["SR","CR"]},
      {"name":"variables1", "nbins":2, "min":0, "max":10}
    ]
  }
]

This type can also be used to store pre-processed data utilizing the uncertainty component

Example: Pre-processed binned Data
"data":[
  {
    "name":"data5",
    "type":"binned",
    "contents":[ 9.0, 18.4 ],
    "uncertainty" : {
      "type": "gaussian_uncertainty",
      "correlation" : 0,
      "sigma" : [ 3, 4 ]
     },
    "axes":[
      { "name":"variable1", "nbins":2, "min":1, "max":3 }
    ]
  }
]

Combined data set

Combined data sets merge multiple data structs into one. The sub-data sets can be of any type described above. Their axes names are pairwise distinct, since each sub-data set is paired with a component of a combined distribution and those components share no variate.
A combined data set is defined with the components:

  • name: custom string
  • type: combined
  • columns: array of names. Each name is a selection column whose allowed labels are declared by a categorical axis of that name in the domains, see Selection keys; the axis declares labels only and introduces no variable or observable. The labels of a single column, or the combinations of labels of several columns, are the possible values of a key. axes must not be present.
  • components: array of structs, each assigning one sub-data set to one key. Each struct must have the components:
    • key: array of strings, one per entry of columns. No two entries of components may have the same key.
    • dataset: name of the referenced data set

components need not cover every possible key.

Example: Combined data set
"data":[
  {
    "name":"combined_data",
    "type":"combined",
    "columns":["regions","observables"],
    "components":[
      { "key":["SR","physics"], "dataset":"data_sr" },
      { "key":["CR","physics"], "dataset":"data_cr" },
      { "key":["VR","physics"], "dataset":"data_vr" },
      { "key":["SR","global"],  "dataset":"global_obs_sr" },
      { "key":["CR","global"],  "dataset":"global_obs_cr" }
    ]
  },
  { "name":"data_sr", "type":"binned", "contents":[...], "axes":[...] },
  { "name":"data_cr", "type":"unbinned", "entries":[...], "axes":[...] },
  { "name":"data_vr", "type":"unbinned", "entries":[...], "axes":[...] },
  { "name":"global_obs_sr", "type":"unbinned", "entries":[...], "axes":[...] },
  { "name":"global_obs_cr", "type":"unbinned", "entries":[...], "axes":[...] }
]

Note that region VR has no global entry in components: the combination (VR, global) is simply absent, which is valid.