Skip to content

Data

Data

The top-level component data contains an array of data sets in struct format. Each data set needs to contain the components type and name. Other components are dependent on the type of data set as demonstrated below:

  • name: custom string
  • type: string that determines the format of the observations
  • ...: each type of observations has different parameter keys. Some of these are optional and marked accordingly in the more detailed description below

A detailed description of the different types with examples can be found below. While data, in most settings, has no uncertainty attached to it, this standard does allow to provide data with uncertainty, as there are some settings in which uncorrelated or correlated errors need to be taken into account. These include cases such as

  • generated data obtained from importance sampling, including a (potentially Gaussian) uncertainty from the frequency weights
  • unfolded data, resulting from arbitrarily complex transformation functions involving statistical models folding some degree of uncertainty into the data points themselves

While it should always be preferred to publish "raw" data, allowing to include pre-processed data with corresponding uncertainties expands the possible applications considerably.

Axis Specifications

Axes define the observable domain for data and PDFs. Every axis is either continuous or categorical. The kind of an axis is determined structurally: an axis struct must contain exactly one of the field pair min/max (continuous axis) or the field categories (categorical axis). Continuous axes additionally have type-specific requirements based on the data type they describe.

Continuous Axes

A continuous axis describes a real-valued observable. All continuous axes must include:

  • name: identifier matching the observable in the PDF
  • min: lower bound of the observable domain
  • max: upper bound of the observable domain

These fields define the domain over which PDFs are normalized.

Note on normalization: Any PDF depending on a continuous observable \(x\) must be normalized over the domain \([\text{min}, \text{max}]\) defined by that observable's axis, not over \((-\infty, \infty)\).

Categorical Axes

A categorical axis describes a discrete-valued observable that takes one of a finite set of named states. A categorical axis must include:

  • name: identifier matching the categorical observable in the PDF
  • categories: array of unique strings enumerating the allowed values. categories defines a set: the order in which values appear in the array must not affect evaluation.

The categories array itself defines the domain of the axis; the fields min, max, nbins and edges must not be present.

Forbidden Attributes

The const attribute must not appear in axis specifications. All axes define observable domains, never fit parameters.

Axis Types by Data Type

For Point Data:

  • Only continuous axes are permitted, carrying only the base fields (name, min, max)
  • Binning fields (nbins, edges) must not be present

For Unbinned Data:

  • Continuous axes are required to carry base fields (name, min, max)
  • Binning fields (nbins, edges) must not be present
  • Categorical axes are permitted; the corresponding column of entries holds the respective row's category as one of the strings listed in categories

For Binned Data:

  • Continuous axes are required to carry base fields (name, min, max)
  • Each continuous axis must carry exactly one binning specification:
    1. Regular binning: nbins (integer number of equal-width bins)
    2. Irregular binning: edges (array of length \(n+1\) bin boundaries)
  • Both nbins and edges must not be specified simultaneously
  • Neither nbins nor edges must not be omitted on a continuous axis
  • Categorical axes are permitted. A categorical axis acts as its own binning, with one bin per category in the order given by categories

For irregular binning, the edges array must satisfy: - edges[0] == min - edges[-1] == max - All values must be in strictly ascending order

Point Data

Point data describes a measurement of a single number, with a possible uncertainty (error).

  • name: custom string
  • type: point
  • value: value of this data point
  • axes: (optional) array of axis structs. Each struct must contain name, min, max as defined in Axis Specifications. When present, associates the point measurement with a specific observable domain.
  • uncertainty: (optional) uncertainty of this data point
Example: Point Data
"data":[
    {
        "name":"data1",
        "type":"point",
        "value":0.,
        "uncertainty":1.,
        "axes":[
            { "name":"obs_x", "min":-5, "max":5 }
        ]
    }
]

Unbinned Data

Unbinned data describes a measurement of multiple data points in a possibly multi-dimensional space of variables. These data points can be weighted.

  • name: custom string
  • type: unbinned
  • entries: array of arrays containing the coordinates/entries of the data
  • axes: array of axis structs. Each struct must be a valid continuous or categorical axis as defined in Axis Specifications. Binning fields (nbins, edges) must not be present.
  • weights: (optional) array of values containing the weights of the individual data points, to be used for \(\chi^2\) comparisons and fits. If this component is not given, weight 1 is assumed for all data points. If given, the array needs to be of the same length as entries.
  • entries_uncertainties: (optional) array of arrays containing the errors/uncertainties of each entry. If given, the array needs to be of the same shape as entries.

Example: Unbinned Data
"data":[
  {
    "name":"data1",
    "type":"unbinned",
    "weights":[ 9.0, 18.4 ],
    "entries":[ [1,3], [2,9] ],
    "entries_uncertainties":[ [0.3], [0.6] ],
    "axes":[
      { "name":"variable1", "min":1, "max":3 },
      { "name":"variable2", "min":-10, "max":10 }
    ]
  }
]
Unbinned data sets with a categorical axis store the category string in the corresponding column of entries:
Example: Unbinned Data with categorical axis
"data":[
  {
    "name":"data2",
    "type":"unbinned",
    "weights":[12.0, 4.3, 2.1],
    "entries":[["SR",7,0.1], ["SR",9,-0.2], ["CR",4,0.9]],
    "axes":[
      {"name":"cat1", "categories":["SR","CR"]},
      {"name":"variable1", "min":3, "max":20},
      {"name":"variable2", "min":-1, "max":1}
    ]
  }
]
Here the first column of entries refers to the cat1 axis, giving the category of each row directly as "SR" or "CR".

Binned Data

Binned data describes a histogram of data points with bin contents in a possibly multi-dimensional space of variables. Whether entries that fall precisely on the bin boundaries are sorted into the smaller or larger bin is under the discretion of the creator of the model and thus not defined.

  • name: custom string
  • type: binned
  • contents: array of values representing the contents of the binned data set
  • axes: array of axis structs. Each struct must be a valid continuous or categorical axis as defined in Axis Specifications. Each continuous axis must carry exactly one binning specification:
    1. Regular binning: nbins (number of equal-width bins)
    2. Irregular binning: edges (array of length \(n+1\) bin boundaries)
  • uncertainty: (optional) struct representing the uncertainty of the contents. It consists of up to three components:
    • type: denoting the kind of uncertainty, for now only Gaussian distributed uncertainties denoted as gaussian_uncertainty are supported
    • sigma: array of the standard deviation of the entries in contents. Needs to be of the same length as contents
    • correlation: (optional) array of arrays denoting the correlation between the contents in matrix format. Must be of dimension length of contents \(\times\) length of contents. It can also be set to 0 to indicate no correlation.
Example: Binned Data
"data":[
  {
    "name":"data3",
    "type":"binned",
    "contents":[ 9.0, 18.4 ],
    "axes":[ { "name":"variable1", "nbins":2, "min":1, "max":3 } ]
  },
  {
    "name":"asimov_data3",
    "type":"binned",
    "contents":[ 9.0, 18.4, 13, 0. ],
    "axes":[
      { "name":"variable1", "nbins":2, "min":1, "max":3 },
      { "name":"variable2", "min":0, "max":100, "edges":[0,10,100] }
    ]
  }
]

The contents of the data sets are represented in a flattened one-dimensional array of bin-contents. Here the behaviour is the same for a categorical axis with \(n\) categories as for regularly-binned continuous axis with \(n\) bins: each category corresponds to one bin along the respective dimension of the flattened contents array.

Example: Binned Data with categorical axis
"data":[
  {
    "name":"data4",
    "type":"binned",
    "content":[9.0, 8.4, 4.0, 7.0],
    "axes":[
      {"name":"channel", "categories":["SR","CR"]},
      {"name":"variables1", "nbins":2, "min":0, "max":10}
    ]
  }
]

This type can also be used to store pre-processed data utilizing the uncertainty component

Example: Pre-processed binned Data
"data":[
  {
    "name":"data5",
    "type":"binned",
    "contents":[ 9.0, 18.4 ],
    "uncertainty" : {
      "type": "gaussian_uncertainty",
      "correlation" : 0,
      "sigma" : [ 3, 4 ]
     },
    "axes":[
      { "name":"variable1", "nbins":2, "min":1, "max":3 }
    ]
  }
]