Summary statistics [RSS]

[jpullmann]

Summary statistics [RSS]

Express summary statistics and descriptive metrics to characterize a Dataset.

Related use cases: Summarization/Characterization of datasets [ID33] 

[makxdekkers]

DQV should be able to meet these requirements.

[dr-shorthair]

Do we also need statistics on distributions? This requirement is suggested by the comment submitted by Daniel Pop [1]. Of course it also depends on how we resolve the matter of 'information equivalence' of different distributions.

[1] https://lists.w3.org/Archives/Public/public-dxwg-comments/2019Jan/0013.html

[andrea-perego]

I think we shouldn't prevent this - as done for other information. The question is how to put this option in the spec.

I'm a bit reluctant to explicitly add properties in class definitions where we don't have real-world use cases and/or implementation evidence. So, this could be included in the "guidance" part of the spec.

About "information equivalence", (again) -1 to it. This ends up to be a matter of the "granularity" of the notion of dataset, which is mainly a data provider choice (possibly also based on the requirements of the intended users).

[makxdekkers]

@dr-shorthair I am not quite sure how you derive a requirement for statistics on datasets? If there is a need for it, maybe we could refer to DQV or Data Cube?
In my mind, Daniel's point (1) could be resolved by modelling the real-time data stream as a dcat:DataService and modelling the CSVs as separate datasets.

[dr-shorthair]

The statistic that Daniel mentioned is the frequency or spacing of members in a time series, where various distributions might have fixed spacing that is different (usually coarser) than what is available from the underlying dataset. I was on the point of creating an explicit issue for this aspect alone, but since this is an aspect of dataset statistics I thought it would be best to open the discussion here first.

[dr-shorthair]

@makxdekkers I did not derive a new requirement for dataset statistics - this was one of the original requirements taken from UCR.

However, I do wonder if time-series are such a common case that they might deserve special treatment. i.e. complement dct:temporal (coverage) with one more number - the item-accrual-periodicity. And since dct:accrualPeriodicity has been hijacked (in the DCAT context) to describe the publication period, it might have to be a new property? See #728

[smrgeoinfo]

If I understand correctly, the concept @dr-shorthair is looking for is named temporalResolution in ISO19115-1, and is important for evaluating datasets that have temporal coverage. There is a corresponding spatialResolution property that is equally important if you're evaluating spatial data.

[dr-shorthair]

@smrgeoinfo yes - I think we need to pair

• dct:temporal - Temporal Coverage - i.e. the temporal extent of the dataset, the time interval that this dataset describes
  with
• dcat:temporalResolution - smallest time period resolvable in the data; e.g. temporal spacing of a regular time series

And, while I'm a little wary of treading too far down a path that should be managed through a geospatial profile, since we already have

• dct:spatial - Geospatial coverage - i.e. the spatial extent of the dataset
  it is not much of a stretch to match this with
• dcat:spatialResolution - smallest distance separating items in the data

(and stop there).

[andrea-perego]

For spatial / temporal resolution, see UC15, which describes the general context and provides the relevant references.

These topics were discussed by the SDW WG, and then with the DWBP WG (in particular, with @aisaac and @riccardoAlbertoni ), which led to a proposal on how to specify it by using DQV.

The proposal is included as an example (focussing on spatial resolution only) in DQV, §6.13 (Express dataset precision and accuracy), which was in turn re-used into SDW's Best Practice 14 (Describe the positional accuracy of spatial data).

We should therefore re-use and consolidate that approach.

About consolidation, I summarised what I see as issues to be addressed in the context of the possible revisions to GeoDCAT-AP ( see SEMICeu/GeoDCAT-AP#3).

For our convenience, I copy-paste below the relevant text from SEMICeu/GeoDCAT-AP#3:

Basically, DQV models this information as observations / measurements of a given quality metric (which corresponds to a given type of resolution).

[...]

[Adopting] This [solution] would however require the definition of two groups of individuals:

1. Those corresponding to the different types of resolution (denoting a quality metric).
2. Those corresponding to each of the different levels of resolution (denoting the measurement of a specific quality metric).

As far as the first group is concerned (i.e., the different types of resolution), these individuals can be defined in DQV as follows:

:SpatialResolutionAsEquivalentScale a dqv:Metric;
  skos:definition "Spatial resolution of a dataset expressed as equivalent scale,
	  by using a representative fraction (e.g., 1:1,000, 1:1,000,000)."@en ;
  dqv:expectedDataType xsd:decimal ;
  dqv:inDimension dqv:precision .
    
:SpatialResolutionAsDistance a dqv:Metric;
  skos:definition "Spatial resolution of a dataset expressed as distance"@en ;
  dqv:expectedDataType xsd:decimal ;
  dqv:inDimension dqv:precision .

This initial list can be further extended. E.g.:

:SpatialResolutionAsHorizontalGroundDistance a dqv:Metric;
  skos:definition "Spatial resolution of a dataset expressed as horizontal ground distance"@en ;
  dqv:expectedDataType xsd:decimal ;
  dqv:inDimension dqv:precision .
    
:SpatialResolutionAsVerticalDistance a dqv:Metric;
  skos:definition "Spatial resolution of a dataset expressed as vertical distance"@en ;
  dqv:expectedDataType xsd:decimal ;
  dqv:inDimension dqv:precision .
    
:SpatialResolutionAsAngularDistance a dqv:Metric;
  skos:definition "Spatial resolution of a dataset expressed as angular distance"@en ;
  dqv:expectedDataType xsd:decimal ;
  dqv:inDimension dqv:precision .    

The question is in which space such individuals should be defined [...].

The definition of individuals in the second group is however more problematic, since the level of resolution and unit of measurement are arbitrary (1:1000, 1:100, 1m, 1km, 100m, 10 decimal degrees, etc.).

Possible options include the following ones:

1. Define only the individuals corresponding to the types of spatial / temporal resolution, whereas the individuals expressing the actual resolution will be defined at the data level. This solution is not optimal, since it will result in multiple definitions of the same individuals.
2. Define individuals only for some levels of resolution and units of measurements - e.g., the most common ones. This solution may address the majority of (but not all) the cases.
3. Set up a URI space supporting arbitrary levels of resolution and units of measurements. This register will dynamically generate the corresponding individuals based on information included in their URI.

An example of the last option, including also a proposal for how these individuals could be defined, is available at:

http://geodcat-ap.semic.eu/id/resolution/