Developer Tools

METIS provides a number of tools for developers that wish to automate their access to METIS data, concepts and other services.   Developers may want to explore the three areas below:

• APIs for Data Access
  • Accessing METIS Data Product Metadata
  • Downloading Product Data Files
  • Searching for METIS Data Products
    • Find All METIS Products
    • Find METIS Products Related to a Keyword Phrase
    • Find all METIS Products Related to a Specific CHIPS Research Topic
• Software Tools and Libraries
  • pdrdownload: a Python Script for Downloading Data Products
  • Client Libraries
• Leveraging METIS Semantic Assets

APIs for Data Access

The METIS portal is built on top of the NIST Public Data Repository (PDR) and, consequently, shares the same programmatic interfaces for doing such things as:

• accessing metadata descriptions of METIS data publications
• retrieving lists of downloadable files for a publication and their URLs
• searching for data publications based on text matching or complex metadata values

Accessing METIS Data Product Metadata

The METIS Data Portal web page provides a dynamic listing of all METIS data products. Clicking on one of the datasets will display the home page for the product. The URL for accessing the product home page (like this one) can also be used to access its JSON-encode metadata by appending “?format=nerdm” to it.

Alternatively, the PDR URLs support content negotiation in which an HTTP client can provide its format preference via the the HTTP Accept header. The JSON-encoded metadata is returned by default when the header is not given or specifies “application/json”. This can be seen in the output from these example curl commands which both return JSON:

curl https://data.nist.gov/od/id/mds2-3331
curl -H 'Accept: application/json' https://data.nist.gov/od/id/mds2-3331

The JSON data returned conforms to the NERDm family of schemas which serves as the PDR’s native metadata format. If you are new to NERDm or JSON data, this Reader’s Guide to NERDm may be helpful.

Other formats are supported as well as summarized below:

Downloading Product Data Files

The key to downloading files from a METIS data product programmatically is by accessing and interpreting the product’s NERDm metadata description as described above. The NERDm JSON record will include a components property whose value is an array of descriptions of the different component parts that make up the product. There are different types of components, of which the DataFile type represents a downloadable data file that is included in the product.

For example, a product’s components property might look something like this:

"components": [
  {
    "@id": "cmps/Supplementary_Figure6_Optica/Supplementary_Figure6b/propagation_loss.csv",
    "@type": [
      "nrdp:DataFile",
      "nrdp:DownloadableFile",
      "dcat:Distribution"
    ],
    "_extensionSchemas": [
       "https://data.nist.gov/od/dm/nerdm-schema/pub/v0.7#/definitions/DataFile"
    ],
    "filepath": "Supplementary_Figure6_Optica/Supplementary_Figure6b/propagation_loss.csv",
    "downloadURL": "https://data.nist.gov/od/ds/ark:/88434/mds2-3331/Supplementary_Figure6_Optica/Supplementary_Figure6b/propagation_loss.csv",
    "mediaType": "text/csv",
    "size": 10176,
    "checksum": {
      "hash": "06184ef31373a0ec67ea56c861348ed19b8889b15f5ac82b11663e371e937174",
      "algorithm": {
        "tag": "sha256",
        "@type": "Thing"
      }
    }
  },
  ...
]

Each JSON object in the components array represents an accessible part of the data product. The object’s @type array property will include "nrdp:DataFile" if the object describes a downloadable data file. The object will also include a downloadURL property which provides the URL that can be used to download the file.

Other properties that may be of interest for programmatically downloading files:

• filepath – Product data files can be organized into “folders” and “subfolders”. This property gives file-system-like path to the file within the product. This value can be used as the relative path name for saving the file to local disk.
• size – the size of the file in bytes
• mediaType and format – these properties give information on what kind of file it is.
• checksum – provides checksum information for checking the integrity of the downloaded file (with the checksum hash contained in the hash subproperty). As of this writing, all METIS checksums are SHA-256 hashes.

The PDR provides a Python script, pdrdownload, which takes advantage of all these features to download data product files robustly on the command line. You can view this script’s code as an example for taking advantage of product metadata.

Searching for METIS Data Products

The PDR provides a rich API for searching for data products which is described in detail in the PDR’s API documentation. Here, we show and explain a few common example uses.

Find All METIS Products

First, to get a listing of all the METIS data products, one can use this curl-based query:

curl "https://data.nist.gov/rmm/records?isPartOf.@id=ark:/88434/pdr0-0002"

The query string, isPartOf.@id=ark:/88434/pdr0-0002, illustrates how to place a search constraint on a specific NERDm property. The query result looks something like this:

{
  "ResultCount": 17,
  "PageSize": 0,
  "ResultData": [
    ...
  ]

The ResultData array will contain a list of objects where each one is a NERDm description of one product in the METIS collection. ResultCount will indicate the number of products there.

Find METIS Products Related to a Keyword Phrase

A special search parameter in the PDR search syntax, searchPhrase, will look for an arbitrary text string in any value within a NERDm record. For example, we can find METIS products related to “machine learning”:

curl "https://data.nist.gov/rmm/records?isPartOf.@id=ark:/88434/pdr0-0002&searchphrase=machine+learning"

Find all METIS Products Related to a Specific CHIPS Research Topic

As described in the next section, METIS products are tagged with terms from the CHIPS Metrology taxonomy. In this example, we find all METIS products related to the CHIPS Grand Challenge 4:

curl "https://testdata.nist.gov/rmm/records?isPartOf.@id=ark:/88434/pdr0-0002&topic.tag=Grand+Challenge:+4"

If you examine the topic tags in the NERDm records returned by this query, you will notice that the full topic tag is “Metrology: Grand Challenge: 4-Modeling and Simulating Semiconductor Materials, Designs, and Components”. Thus, you can see that use of the topic.tag search constraint looks for the given substring in topic.tag values.

Software Tools and Libraries

pdrdownload: a Python Script for Downloading Data Products

The PDR provides a stand-alone Python script called pdrdownload that can be used to download products from the repository robustly from the command line. In particular, when given a product identifier, it can:

• summarize the data files available without downloading them
• download all of the files that are part of the data product
• download a selected portion of the product based on folder names (useful for very large products)
• use checksum hashes to ensure that files were downloaded without corruption
• restart a download request after a download failure, picking up where it left off

Client Libraries

Convenient client libraries that allow a developer to easily leverage the PDR APIs are currently under development. Watch this space for links to available libraries and tools.

Leveraging METIS Semantic Assets

We are continually enhancing the METIS platform to make CHIPS data products not only easier to find but also more machine actionable and usable in automated systems. This includes capturing more research-specific metadata and tagging with discipline-specific terms. Watch this space for links to semantic resources--such as taxonomies, schemas, and ontologies--that we’re using to enhance our data, along with tools that can help you make the most of them.