Linked SDMX Data

As a demonstration of the SDMX-ML to RDF transformations, the selection of datasets here are from the following organizations. Instead of regurgitating publisher's profile about themselves, and to keep this part brief, I'll quote the first sentences from their about pages:

OECD

The mission of the Organisation for Economic Co-operation and Development (OECD) is to promote policies that will improve the economic and social well-being of people around the world from OECD Our mission.

BFS

Swiss Statistics, the Federal Statistical Office’s web portal. Offering a modern and appealing interface, our website proposes a wide range of statistical information on the most important areas of life: population, health, economy, employment, education and much more from BFS Welcome.

FAO

Achieving food security for all is at the heart of FAO's efforts - to make sure people have regular access to enough high-quality food to lead active, healthy lives from FAO's mandate.

ECB

Whose main task is to maintain the euro's purchasing power and thus price stability in the euro area from ECB home.

IMF

Working to foster global monetary cooperation, secure financial stability, facilitate international trade, promote high employment and sustainable economic growth, and reduce poverty around the world from IMF home.

UIS

The primary source for cross-nationally comparable statistics on education, science and technology, culture, and communication for more than 200 countries and territories from UIS home.

FRB

The Federal Reserve, the central bank of the United States, provides the nation with a safe, flexible, and stable monetary and financial system from FRB home.

BIS

The Bank for International Settlements (BIS) is an international organisation which fosters international monetary and financial cooperation and serves as a bank for central banks from BIS home.

ABS

We assist and encourage informed decision making, research and discussion within governments and the community, by leading a high quality, objective and responsive national statistical service from ABS mission statement.

The OECD, FAO, ECB, IMF, UIS, BIS, ABS datasets consisted of observational and structural data. The OECD, ECB, UIS, BIS, ABS provided complete coverage (to the best of our knowledge), whereas FAO had partial fishery related data, and IMF partial data over their REST service. BFS had all of their classifications available, with no observational data in SDMX-ML.

As SDMX-ML publishers have their own publishing processes, availability and accessibility of the data varied. After obtaining the dataset codes, names, and URLs with common Linux command-line work, a Bash script was created to retrieve the data.

On how to retrieve all of the datasets from the OECD website was not entirely clear. In order to automatically get a hold of list of datasets, I copied the innerHTML of the DOM tree that contained all the dataset codes from OECD.StatExtracts to a temporary file. This was done due to the fact that a simple scrape of an HTTP GET wasn't possible as the data on the page was populated via JavaScript on document ready. After constructing a list of datasets and structures to get, two REST API endpoints were called.

BFS offered a Microsoft Excel document which contained a catalog of their classifications and URLs for retrieval.

After searching for keywords along the lines of SDMX site:fao.org at a search-engine nearby, FAO Fisheries and data.fao.org SDMX Registry and Repository, and its children pages were marked for SDMX-ML retrieval.

ECB had a similar REST API to OECD. Additionally, SDMX Dataflows was retrieved to get a primary list of datasets to retrieve. Some of the large datasets was retrieved by making multiple smaller calls to the API using a call per refereance area.

IMF had the same procedure as ECB and OECD.

FRB had the same procedure as above.

UIS had the same procedure as above.

BIS had the same procedure as above.

ABS had the same procedure as above.

We now discuss provenance at: retrieval, transformation, and post-processing.

By in large, there was no need to pre-process the data as the transformation dealt with the data as it was. However, some non-vital SDMX components were omitted from the output. For instance, one type of attribute in OECD and ECB observations contained free-text as opposed to its corresponding code from a codelist. Since the RDF Data Cube required codes as opposed to free-text for dimension values, some attributes were excluded. The decision here was to trade-off some precision in favour of retaining the dataset.

This section goves over several areas which are at the heart of representing statistical data in SDMX-ML as Linked Data. The approach taken was to provide a level of consistency for data consumers and tool builders for all statistical Linked Data with its origins from data in SDMX-ML.

• Transforms SDMX KeyFamilies, ConceptSchemes and Concepts, CodeLists and Codes, Hierarchical CodeLists, DataSets.
• Configurability for SDMX publisher's needs.
• Detection and referencing CodeLists and Codes of external agencies.
• Support of interlinking publisher-specific annotation types.
• Support for omission of components.
• Inclusion of provenance data.

It comes with scripts and sample data:

• XSLT 2.0 templates to transform Generic SDMX-ML data and metadata. It includes the main XSL template for generic SDMX-ML, an XSL for common templates and functions, and an RDF/XML configuration file to set preferences like base URIs, delimiters in URIs, how to map annotation types.
• Bash script that transforms sample data using saxonb-xslt.
• Sample SDMX Message and Structure retrieved from those organizations that are initially involved in the SDMX standard, as well as from BFS.

The requirements for the Linked SDMX toolkit are an XSLT 2.0 processor to transform, and optionally to configure some of the settings in the transformation with provided config.ttl (in RDF Turtle) and transforming that to an abbreviated version of RDF/XML. In sequel some of they key features are described in more detail.

The transformation follows some common Linked Data practices as well as other ones out of thin air.

The config file is used to pre-configure some of the decisions that are made in XSL templates. Here is an outline for some of the noteworthy things.

<http://ecb.270a.info/property/OBS_STATUS>
<http://purl.org/linked-data/cube#codeList>
<http://eurostat.270a.info/code/1.0/CL_OBS_STATUS>

Here are some statistics for the transformations.

The command-line tool saxonb-xslt was used to conduct the XSL transformations. 12000M of memory was allocated on a machine with Intel(R) Xeon(R) CPU E5620 @ 2.40GHz. Linux kernel 3.2.0-33-generic was used. Table [Transformation time] provides information on datasets; input SDMX-ML size, output RDF/XML size, their size difference in ratio, and the total amount transformation time.

Table [Transformed data] provides data on the transformed data; number of triples it contains, as well as the number of qb:Observations, and the ratio.

Table [Resource counts] provides further statistics on prominent resources. It gives a contrast between the classifications and the dataset.

Initial interlinking is done among the classifications themselves in the datasets. The OECD and UIS classifications in particular contained highly similar codes (in some cases the same) throughout its codelists. Majority of the interlinks are between skos:Concepts with link relation skos:exactMatch.

The consequent interlinking was done with DBpedia, World Bank Linked Data, Transparency International Linked Data, and EUNIS using LInk discovery framework for MEtric Spaces (LIMES): A Time-Efficient Hybrid Approach to Link Discovery (Ngonga Ngomo, A.-C., 2011)

Figure [SDMX concept links] gives an overview of the complete connectivity of a concept that's linked internally, externally, and with sdmx-codes where applicable, as well as the interlinking that was done to an external concept.

Apache Jena’s TDB storage system is used to load the RDFized data using TDB’s incremental tdbloader utility. tdbstats, the tool for TDB Optimizer is executed after a complete load to internally update the count of resources in order for TDB to make the best decision to come up with future query results.

Individual datasets from each organization were transformed to N-Triples format before loading into the store. Each RDF Data Cube dataset was imported into its own NAMED GRAPH in the store. Given the significant load speed on an empty database, N-Triples were ordered from largest to smallest, and then loaded.

As VoID file is generally intended to give an overview of the dataset metadata i.e., what it contains, ways to access it or query it, each dataspace contains Vocabulary of Interlinked Datasets (VoID) files accessible through their .well-known/void. Each OECD, BFS, FAO, ECB, IMF, UIS, FRB, BIS, ABS VoID contains locations to RDF datadumps, named graphs that are used in the SPARQL endpoint, used vocabularies, size of the datasets, interlinks to external datasets, as well as the provenance data which was gathered through the retrieval and transformation process. The VoID files were generated automatically by first importing the LODStats information into respective graph/void into the store, and then a SPARQL CONSTRUCT query to include all triples as well as additional ones which could be actively created based on the available information in all graphs.

Dataset statistics are generated and are included in the VoID file using LODStats, LODStats – An Extensible Framework for High-performance Dataset Analytics (Demter, J., 2012).

The HTML pages are generated by the Linked Data Pages framework, where Moriarty, Paget, and ARC2 does the heavy lifting for it. Given the lessons learned over the years about Linked Data publishing, there is a consideration to either take Linked Data Pages further (originally written in 2010), or to adapt one of the existing frameworks after careful analysis.

Apache Jena Fuseki is used to run the SPARQL server for the three datasets. SPARQL Endpoints are publicly accessible and read only at their respective /sparql and /query locations for OECD, BFS, FAO, ECB, IMF, UIS, FRB, BIS, ABS. Currently, 12000MB of memory is allocated for the single Fuseki server running all datasets.

The data dumps for the datasets are available from their respective /data/ directories: OECD, BFS, FAO, ECB, IMF, FRB, UIS, BIS, ABS. Additionally, they are mentioned in the VoID files. The the Data Hub entries (see below) also contains links to the dumps.

The code for transformations is at csarven/linked-sdmx, and for retrieval and data loading to RDF store for OECD is at csarven/oecd-linked-data, for BFS is at csarven/bfs-linked-data, for FAO is at csarven/fao-linked-data, for ECB is at csarven/ecb-linked-data, for IMF is at csarven/imf-linked-data, for UIS is at csarven/uis-linked-data, for FRB is at csarven/frb-linked-data, for ABS is at csarven/abs-linked-data. All using the Apache License 2.0.

All published Linked Data adheres to original data publisher’s data license and terms of use. Additionally attributions are given on the websites. The Linked Data version of the data is licensed under CC0 1.0 Universal (CC0 1.0) Public Domain Dedication.

For other ways for these datasets to be discovered, they are announced at mailing lists, status update services, and at the Data Hub: OECD is at oecd-linked-data, BFS is at bfs-linked-data, FAO is at fao-linked-data, ECB is at ecb-linked-data, IMF is at imf-linked-data, UIS is at uis-linked-data, FRB is at frb-linked-data, BIS is at bis-linked-data, ABS is at abs-linked-data.