Data
Data¶
The top-level component data contains an array of data sets in struct format. Each data set needs to contain the components type and name. Other components are dependent on the type of data set as demonstrated below:
name: custom stringtype: string that determines the format of the observations...: each type of observations has different parameter keys. Some of these are optional and marked accordingly in the more detailed description below
A detailed description of the different types with examples can be found below. While data, in most settings, has no uncertainty attached to it, this standard does allow to provide data with uncertainty, as there are some settings in which uncorrelated or correlated errors need to be taken into account. These include cases such as
- generated data obtained from importance sampling, including a (potentially Gaussian) uncertainty from the frequency weights
- unfolded data, resulting from arbitrarily complex transformation functions involving statistical models folding some degree of uncertainty into the data points themselves
While it should always be preferred to publish "raw" data, allowing to include pre-processed data with corresponding uncertainties expands the possible applications considerably.
Data sets associate their contents with variables via axes. Depending on the type of data set, these are continuous, binned or categorical axes, as specified in Axis Specifications.
Forbidden attributes: The const attribute must not appear in the axes of data sets. All axes of data sets define observable domains, never fit parameters.
Point Data¶
Point data describes a measurement of a single number, with a possible uncertainty (error).
name: custom stringtype:pointvalue: value of this data pointaxes: (optional) array of continuous axes as defined in Continuous Axes. When present, associates the point measurement with a specific observable domain.uncertainty: (optional) uncertainty of this data point
"data":[
{
"name":"data1",
"type":"point",
"value":0.,
"uncertainty":1.,
"axes":[
{ "name":"obs_x", "min":-5, "max":5 }
]
}
]
Unbinned Data¶
Unbinned data describes a measurement of multiple data points in a possibly multi-dimensional space of variables. These data points can be weighted.
name: custom stringtype:unbinnedentries: array of arrays containing the coordinates/entries of the data. For a categorical axis, the corresponding column holds the respective row's category as one of the strings listed incategories.axes: array of continuous or categorical axes as defined in Axis Specifications.weights: (optional) array of values containing the weights of the individual data points, to be used for \(\chi^2\) comparisons and fits. If this component is not given, weight 1 is assumed for all data points. If given, the array needs to be of the same length asentries.entries_uncertainties: (optional) array of arrays containing the errors/uncertainties of each entry. If given, the array needs to be of the same shape asentries.
"data":[
{
"name":"data1",
"type":"unbinned",
"weights":[ 9.0, 18.4 ],
"entries":[ [1,3], [2,9] ],
"entries_uncertainties":[ [0.3], [0.6] ],
"axes":[
{ "name":"variable1", "min":1, "max":3 },
{ "name":"variable2", "min":-10, "max":10 }
]
}
]
entries:
"data":[
{
"name":"data2",
"type":"unbinned",
"weights":[12.0, 4.3, 2.1],
"entries":[["SR",7,0.1], ["SR",9,-0.2], ["CR",4,0.9]],
"axes":[
{"name":"cat1", "categories":["SR","CR"]},
{"name":"variable1", "min":3, "max":20},
{"name":"variable2", "min":-1, "max":1}
]
}
]
entries refers to the cat1 axis, giving the category of each row directly as "SR" or "CR".
Binned Data¶
Binned data describes a histogram of data points with bin contents in a possibly multi-dimensional space of variables. Entries that fall precisely on an interior bin boundary may be assigned to either of the two adjacent bins, at the discretion of the creator of the document; this standard does not define which. Both bounds of an axis are inclusive: entries at exactly the lower bound of the axis belong to the first bin and entries at exactly the upper bound of the axis belong to the last bin, so that no entry inside the bounds is lost. Consequently, consumers that re-bin unbinned data should not expect bit-identical histograms across implementations.
name: custom stringtype:binnedcontents: array of values representing the contents of the binned data set, flattened as defined in Flattening of Binned Contentsaxes: array of binned or categorical axes as defined in Axis Specificationsuncertainty: (optional) struct representing the uncertainty of the contents. It consists of up to three components:type: denoting the kind of uncertainty, for now only Gaussian distributed uncertainties denoted asgaussian_uncertaintyare supportedsigma: array of the standard deviation of the entries incontents. Needs to be of the same length ascontentscorrelation: (optional) array of arrays denoting the correlation between the contents in matrix format. Must be of dimension length ofcontents\(\times\) length ofcontents. It can also be set to 0 to indicate no correlation.
"data":[
{
"name":"data3",
"type":"binned",
"contents":[ 9.0, 18.4 ],
"axes":[ { "name":"variable1", "nbins":2, "min":1, "max":3 } ]
},
{
"name":"asimov_data3",
"type":"binned",
"contents":[ 9.0, 18.4, 13, 0. ],
"axes":[
{ "name":"variable1", "nbins":2, "min":1, "max":3 },
{ "name":"variable2", "edges":[0,10,100] }
]
}
]
The contents array is flattened as defined in Flattening of Binned Contents: the last axis listed in axes varies fastest, so in example asimov_data3 above the values belong to the bins \((i_0, i_1) = (0,0), (0,1), (1,0), (1,1)\), in this order. A categorical axis with \(n\) categories behaves like a binned axis with \(n\) bins, the bin index being the position of the category in categories.
"data":[
{
"name":"data4",
"type":"binned",
"contents":[9.0, 8.4, 4.0, 7.0],
"axes":[
{"name":"channel", "categories":["SR","CR"]},
{"name":"variables1", "nbins":2, "min":0, "max":10}
]
}
]
This type can also be used to store pre-processed data utilizing the uncertainty component
"data":[
{
"name":"data5",
"type":"binned",
"contents":[ 9.0, 18.4 ],
"uncertainty" : {
"type": "gaussian_uncertainty",
"correlation" : 0,
"sigma" : [ 3, 4 ]
},
"axes":[
{ "name":"variable1", "nbins":2, "min":1, "max":3 }
]
}
]
Combined data set¶
Combined data sets merge multiple data structs into one. The sub-data sets can be of any type described above. Their axes names are pairwise distinct, since each sub-data set is paired with a component of a combined distribution and those components share no variate.
A combined data set is defined with the components:
name: custom stringtype:combinedcolumns: array of names. Each name is a selection column whose allowed labels are declared by a categorical axis of that name in the domains, see Selection keys; the axis declares labels only and introduces no variable or observable. The labels of a single column, or the combinations of labels of several columns, are the possible values of akey.axesmust not be present.components: array of structs, each assigning one sub-data set to onekey. Each struct must have the components:key: array of strings, one per entry ofcolumns. No two entries ofcomponentsmay have the samekey.dataset:nameof the referenced data set
components need not cover every possible key.
"data":[
{
"name":"combined_data",
"type":"combined",
"columns":["regions","observables"],
"components":[
{ "key":["SR","physics"], "dataset":"data_sr" },
{ "key":["CR","physics"], "dataset":"data_cr" },
{ "key":["VR","physics"], "dataset":"data_vr" },
{ "key":["SR","global"], "dataset":"global_obs_sr" },
{ "key":["CR","global"], "dataset":"global_obs_cr" }
]
},
{ "name":"data_sr", "type":"binned", "contents":[...], "axes":[...] },
{ "name":"data_cr", "type":"unbinned", "entries":[...], "axes":[...] },
{ "name":"data_vr", "type":"unbinned", "entries":[...], "axes":[...] },
{ "name":"global_obs_sr", "type":"unbinned", "entries":[...], "axes":[...] },
{ "name":"global_obs_cr", "type":"unbinned", "entries":[...], "axes":[...] }
]
Note that region VR has no global entry in components: the combination (VR, global) is simply absent, which is valid.