Here we define ?s ?p ?o as the pattern we are looking for.  The ? is a variable that can be used to match any value.  In this case, we are looking for all the triples in the graph.  We have used s, p and o to as simple names for our subject, predicate, object elements of the RDF triple.
Since this would match ALL the values in the graph, which would be a lot, we have added a LIMIT option to reduce the number of resutls returned.
You could also try these queries at: https://oceans.collaborium.io/sparql.html
s1 = """
    SELECT ?s ?p ?o
    WHERE {
        ?s ?p ?o .
    }
    LIMIT 3
"""
Run the query#
We will run this query, and print out the results
from icecream import ic
for row in g.query(s1):
    ic(row.asdict())
ic| row.asdict(): {'o': rdflib.term.Literal('2014-06-02'),
                   'p': rdflib.term.URIRef('https://schema.org/startDate'),
                   's': rdflib.term.BNode('n0dfe00a0174f48a7b9fb702b5f5b4ecdb236')}


ic| row.asdict(): {'o': rdflib.term.Literal(' Copenhagen Denmark '),
                   'p': rdflib.term.URIRef('https://schema.org/address'),
                   's': rdflib.term.BNode('n0dfe00a0174f48a7b9fb702b5f5b4ecdb496')}
ic| row.asdict(): {'o': rdflib.term.Literal('Introduction to Ocean Data Management for Students of the Environment Group II'),
                   'p': rdflib.term.URIRef('https://schema.org/name'),
                   's': rdflib.term.BNode('n0dfe00a0174f48a7b9fb702b5f5b4ecdb1745')}
Nicer formatting#
We can do a bit of formatting to see our results in a nicer way.
for row in g.query(s1):
    s = row["s"].n3()
    p = row["p"].n3()
    o = row["o"].n3()
    ic(s, p, o)
ic| s: '_:n0dfe00a0174f48a7b9fb702b5f5b4ecdb236'
    p: '<https://schema.org/startDate>'
    o: '"2014-06-02"'


o: '"2014-06-02"'
ic| s: '_:n0dfe00a0174f48a7b9fb702b5f5b4ecdb496'
    p: '<https://schema.org/address>'
    o: '" Copenhagen Denmark "'
ic| s: '_:n0dfe00a0174f48a7b9fb702b5f5b4ecdb1745'
    p: '<https://schema.org/name>'
    o: ('"Introduction to Ocean Data Management for Students of the Environment Group '
        'II"')
KGLab#
As mentioned, this intro is based on the nice work done by KGLab (https://derwen.ai/docs/kgl/) both in their library and documentation.  We wll import kglab here and take advantage of some of their nice features.  In particular some nice functioons to load our graph and query it with results into a Dataframe.
import kglab
namespaces = {
    "schema":  "https://schema.org/",
    }
kg = kglab.KnowledgeGraph(
    name = "Demonstration Graph",
    base_uri = "https://oceaninfohub.org/id/",
    namespaces = namespaces,
    )


namespaces = namespaces,
    )
kg.load_rdf("../tooling/notebooks/data/oceanexperts_graph.ttl")
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/__init__.py:177: UserWarning: Code: dateTimeStamp is not defined in namespace XSD
  from . import DatatypeHandling, Closure
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: dateTimeStamp is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: length is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: maxExclusive is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples


/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: maxInclusive is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: maxLength is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: minExclusive is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: minInclusive is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples


/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: minLength is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/RDFSClosure.py:40: UserWarning: Code: pattern is not defined in namespace XSD
  from owlrl.AxiomaticTriples import RDFS_Axiomatic_Triples, RDFS_D_Axiomatic_Triples
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/OWLRL.py:53: UserWarning: Code: dateTimeStamp is not defined in namespace XSD
  from .XsdDatatypes import OWL_RL_Datatypes, OWL_Datatype_Subsumptions
/home/fils/.conda/envs/kglab/lib/python3.8/site-packages/owlrl/OWLRLExtras.py:64: UserWarning: Code: dateTimeStamp is not defined in namespace XSD
  from .RestrictedDatatype import extract_faceted_datatypes
<kglab.kglab.KnowledgeGraph at 0x7f470860bf40>
PREFIX#


PREFIX#
Let’s do another query.  We will use the PREFIX keyword to define a prefix for the namespace.  This is a way to leverage namespace in our query.  Note the use of schema:name vs the https://schema.org/location.  The latter is a URI and the former is a prefix.  The prefix is defined in the PREFIX keyword.   The schema:name is the name of the property that we are querying, the prefix can expand it to a URI.  The https://schema.org/location is the URI of the property directly, without use of a prefix.   Both work, the prefix can simply make life easier for you and also help avoid hard to find typos where you may miss-type a URI.
Let’s do a query to see all the location names in our group for training.
s2 = """
PREFIX schema: <https://schema.org/>
SELECT DISTINCT ?locname
  WHERE {
      ?s rdf:type <https://schema.org/Course> .
      ?s  <https://schema.org/location> ?location .
      ?location schema:name ?locname .
  }
  """
import pandas as pd
pd.set_option("max_rows", None)
df = kg.query_as_df(s2)
df.head(5)
locname
0
Lima Peru
1
Qingdao China
2
Russia
3


0
Lima Peru
1
Qingdao China
2
Russia
3
Playa del Secreto Puerto Morelos, Mexico
4
University of Ghent  Ghent Belgium
Location Counts#
The above is nice, but it doesn’t tell us much about the number of events in each location. We can modify our query to do this.  We can leverage the COUNT and GROUP BY capacity of SPARQL to do this.  We can then add in ORDER BY to sort the results.
s3 = """
SELECT DISTINCT ?locname (COUNT(?locname) AS ?count)
  WHERE {
      ?s rdf:type <https://schema.org/Course> .
      ?s  <https://schema.org/location> ?location .
      ?location <https://schema.org/name> ?locname .
  }
  GROUP BY ?locname
  ORDER BY DESC(?count)
  """
import pandas as pd
pd.set_option("max_rows", None)
df = kg.query_as_df(s3)
df.head(5)
locname
count
0
UNESCO/IOC Project Office for IODE Wandelaark...
28
1
Russia
27
2
UNESCO/IOC Project Office for IODE Wandelaark...
22
3
Wandelaarkaai 7 8400 Oostende Belgium
13
4


Wandelaarkaai 7 8400 Oostende Belgium
13
4
Belgium
10
FILTER#
Graphs are great for showing relations between objects but our approaches to searching or selecting data can often leverage pattern matching on text strings.  We do have some tools in SPARQL for this in the form of FILTER.  We can use regex pattern in the FILTER to match text.  Let’s use this to find all the locations with “Peru” in the name.
In the following we will do a simple FILTER with a regular expression, but you can also do things like:
SELECT DISTINCT ?s
  WHERE {
      ?s rdf:type <https://schema.org/Course> .
      FILTER NOT EXISTS { ?s  <https://schema.org/hasCourseInstance> ?instance . }
  }
which in a SPARQL query would require that a hasCourseInstance property is not defined for the Course.  That is, Courses that have had no instances made yet.
s4 = """
SELECT DISTINCT ?locname (COUNT(?locname) AS ?count)
  WHERE {
      ?s rdf:type <https://schema.org/Course> .
      ?s  <https://schema.org/location> ?location .
      ?location <https://schema.org/name> ?locname .


?location <https://schema.org/name> ?locname .
      FILTER regex(?locname, ".Peru.", "i") .
  }
  GROUP BY ?locname
  """
import pandas as pd
pd.set_option("max_rows", None)
df4 = kg.query_as_df(s4)
df4.head(5)
locname
count
0
Lima Peru
1
1
Dirección Hidrografía y Navegación de la Arma...
1
2
Hotel Palmetto Lima Perú Peru
1
PERU#
So we see we have three entries in our graph that have Peru in the name.  This is where connecting our group to a shared concept of what Peru is like the WikiData entry for Peru at https://www.wikidata.org/wiki/Q419 would help.  That’s a topic for another day.
Dates#
Let’s look at the dates that are available in the graph.
s5 = """
SELECT DISTINCT ?sdate (COUNT(?sdate) AS ?count)
  WHERE {
      ?s rdf:type <https://schema.org/Course> .
      ?s  <https://schema.org/hasCourseInstance> ?instance .
      ?instance  <https://schema.org/startDate> ?sdate .
  }
  GROUP BY ?sdate
  """


}
  GROUP BY ?sdate
  """
import pandas as pd
pd.set_option("max_rows", None)
df4 = kg.query_as_df(s5)
df4.head(5) # show some results
sdate
count
0
2020-06-08
1
1
2011-05-03
1
2
2012-07-16
1
3
2007-07-12
1
4
2019-04-02
1
A bit of Pandas#
Here we will mix in a bit of Pandas to convert our dates to Pandas dates, group and plot.  This is not
SPARQL but rather just some post processing in Pandas.  It does still give us an idea of some of the
information we can extract from our graph.
Also, when the graph is composed of multiple source we can still conduct the same search across
a shared concept like https://schema.org/Course with CourseInstance connected to a date via startDate.
This shared approach by Ocean InfoHub is developed to support the shared data and integration
goals among the participants.
df4['count'] = df4["count"].astype(int) # convert count c to int
df4pd = df4.to_pandas()  # convert cudf dataframe to pandas dataframe


df4pd = df4.to_pandas()  # convert cudf dataframe to pandas dataframe
df4pd['sdate'] = pd.to_datetime(df4pd['sdate'], format='%Y-%m-%d')  # convert date to datetime
courseByYear = df4pd.groupby(pd.Grouper(key='sdate', freq='Y')).size() # group by year
ax = courseByYear.plot.bar(rot=80, stacked=True, figsize=(15, 5)) # plot
More on Query#
Here we have gone through an introdcution to the SPARQL query language and how to use it to query our graph.
We can look at the SPARQL we use on the larger Ocean Infohub graph.
References#
This is a just a quick overview of the SPARQL language.  You can find a large amount of information online.
https://derwen.ai/docs/kgl/ex4_0/
https://www.stardog.com/tutorials/getting-started-1/
https://www.dataversity.net/introduction-to-sparql/
https://www.slideshare.net/olafhartig/an-introduction-to-sparql


OIH SPARQL#
About#
This page will hold some information about the SPARQL queries we use and
how they connect with some of the profile guidance in this document.    We will
show how this relates to and depends on the Gleaner prov as well as the
Authoritative Reference elements of the patterns.  It is expected that the Gleaner
prov will be present, though this can be made optional in case other
indexing systems are used that do not provide this prov shape.    The SPARQL will
be looking for both Gleaner prov and the Authroitative Reference elements.
This will be different for different patterns.  For example, it might
relate to the publisher provider elements for Creativeworks, but to
the identity element for People and Organizations.
 1prefix prov: <http://www.w3.org/ns/prov#>
 2PREFIX con: <http://www.ontotext.com/connectors/lucene#>
 3PREFIX luc: <http://www.ontotext.com/owlim/lucene#>
 4PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
 5PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
 6PREFIX schema: <https://schema.org/>


6PREFIX schema: <https://schema.org/>
 7PREFIX schemaold: <http://schema.org/>
 8PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 9
10SELECT DISTINCT ?g  ?s  ?wat ?orgname ?domain ?type ?score ?name ?url ?lit ?description ?headline
11WHERE {
12   ?lit bds:search "coral" .
13   ?lit bds:matchAllTerms "false" .
14   ?lit bds:relevance ?score .
15   graph ?g {
16    ?s ?p ?lit .
17    ?s rdf:type ?type .
18    OPTIONAL { ?s schema:name ?name .   }
19    OPTIONAL { ?s schema:headline ?headline .   }
20    OPTIONAL { ?s schema:url ?url .   }
21    OPTIONAL { ?s schema:description ?description .    }
22  }
23   ?sp prov:generated ?g  .
24   ?sp prov:used ?used .
25   ?used prov:hadMember ?hm .
26   ?hm prov:wasAttributedTo ?wat .
27   ?wat rdf:name ?orgname .
28   ?wat rdfs:seeAlso ?domai
29}


28   ?wat rdfs:seeAlso ?domai
29}
30ORDER BY DESC(?score)
31LIMIT 30
32OFFSET 0
Lines 12-14#
It should be noted that the above SPARQL is not standards compliant.  It leverages some
vender specific syntax that is not part of the SPARQL standard.  This is not uncommon
as groups will often add their own syntax to offer additional functionality.
A common one is what is seen here where a full text index is leveraged to allow for more complex
and faster searches than can be done with FILTER regex.  These three lines will
only work in the current OIH triplestore (Blazegraph).   Other triplestores like Jena
and other do similar built in function extensions.
Lines 18-21#
These line demonstrate the use of the OPTIONAL keyword.  These triples are not required
to be present in a resource.  If they are, they will be returned.
Lines 23-28#
These lines are standard SPARQL but are searching across triples not from the provider
graphs.  Rather, they are looking at triples generated by the OIH indexing program
used, Gleaner.
Note, that Gleaner is not a dependency of this project and other
indexing approaches and software could be used.  As pointed out in the documentation,


indexing approaches and software could be used.  As pointed out in the documentation,
this approach is based on structured data on the web and web architecture approaches.
So, any indexing system following those approaches can be used.
These triples are used to track the indexing event and the sources indexed.  It provides
some additional provenance to the information collected, but does not change or even
extend what the providers are publishing.
As such, these statements could be removed and all that would be lost of indexing
activity information.
Lines 30-32#
These lines represent three specific SPARQL parameters.
First is the ORDER BY directive.  This is used to order the results by one of the
returned variables.  In this case we are using the ?score variable which comes from
the vendor specific syntax noted in lines 11-13.  This score is the ranking score
for a resource search against the full text index.  However, this could be any
variable coming from standards compliant SPARQL calls too.   Sorting can be done
on alphanumeric values in ascending (ASC) or descending (DESC) order.
The  LIMIT is used to limit the number of results returned.  We follow this with,
OFFSET which is used to skip the first n results.  These two are useful for pagination when
combined with the ORDER BY directive.


Users#
About#
The reference client is available at
search.oceaninfohub.org.
Using OIH search in Chrome#
It is possible to set Ocean InfoHub as a search shortcut in Chrome.
To do this go to the Mange Search … section of your settings.
There you will see a button “Add” in the Site Search section.
You can make an entry like: https://search.oceaninfohub.org/results/?search_text=%s&page=0
The shortcut, here oih can be used in the address bar to quickly invoke this search.
Simple type: oih or whatever you set the shortcut to and then hit the tab key.  The address bar will convert to a search mode where you can type your search and hit return.


APIs#
About#
The Ocean InfoHub graph is accessible through several approaches.
The graph is implemented in RDF and expressed through a standards compliant triplestore.
This triplestore exposes a SPARQL endpoint that can be queried using the SPARQL 1.1 Query Language.
To do this you can visit a web based query interface as discussed in other sections of this document.
It is also possible to access this service following RESTful principles.
SPARQL HTTP Protocol#
One approach to this RESTful approach is to use the
SPARQL 1.1 Graph Store HTTP Protocol.
This is a simple HTTP protocol that allows you to query the graph using SPARQL and obtain the results in JSON.
The OIH triplestore exposes the graph following this pattern for queries.
Examples#
curl -X POST https://graph.collaborium.io/blazegraph/namespace/aquadocs/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' -H 'Accept:application/sparql-results+json'
If run this from the command line we will get something like the following.


If run this from the command line we will get something like the following.
✗ curl -X POST https://graph.collaborium.io/blazegraph/namespace/aquadocs/sparql --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' -H 'Accept:application/sparql-results+json'
{
  "head" : {
    "vars" : [ "s", "p", "o" ]
  },
  "results" : {
    "bindings" : [ {
      "s" : {
        "type" : "uri",
        "value" : "https://hdl.handle.net/1834/10030"
      },
      "p" : {
        "type" : "uri",
        "value" : "https://schema.org/propertyID"
      },
      "o" : {
        "type" : "literal",
        "value" : "https://hdl.handle.net/"
      }
    } ]
  }
}
While this is unlikely how you will want to interact with the graph,
it desmonstrates the HTTP based access API that can be used in tools, notebooks
or other applications.
This is the same basic approach the used in the web client.  There the


This is the same basic approach the used in the web client.  There the
axios library (https://axios-http.com/) is used with a code snippet like:
 axios.get(url.toString())
      .then(function (response) {
        // handle success
        console.log(response);
        const el = document.querySelector('#container2');
        render(showresults(response), el);
      })
      .catch(function (error) {
        // handle error
        console.log(error);
      })
      .then(function () {
        // always executed
      });


References#
A broad collection of references.
General#
Science on Schema
BioSchemas
Ocean Best Practices on Schema
PID policy for European Open Science Cloud
DCAT Schema.org mappings
DCAT US Data.gov reference
FAIR Semantics
Developer References#
Schema.org releases
Schema.org RDF graph (turtle format)
JSON-LD Playground
Google Developers Search Gruid
Google Developers Fact Check
Structured Data Testing Tool
Rich Results Testing Tool
JSON-LD
Ruby JSON-LD
schema.org Java
Perl classes for schema.org markup
Content Management and Web Server support#
Drupal Support
Wordpress Claim Review
Organizations#
Google Open Source
DataCommons  & DataCommons REST
Indexers#
Gleaner
BMUSE
Graph tools#
Wikipedia SPARQL implementation list
RDFlib
Any23
rdfjs
shemaram
validatingrdf
Structured Data Linter
Blogs and Press Releases#
Yandex: What is Scheme.org
Bing: Fact Check Label
Bing: Contextual Awareness
Facebook: Marketing API
Facebook: fact checking
Amazon: Alex skills
Google Developers mail invoice
Not Categorized Yet#
Lighthouse Plugins
Lighthouse
Science on Schema
BioSchemas
CodeMeta
Linter Structured Data
JSON-LD Playground
JSON-LD.org
SHACL playground
Google Structured Data testing tool (


JSON-LD.org
SHACL playground
Google Structured Data testing tool (
Google Dataset for  developers
Press  article
Rich results
SchemaApp.com
Yandex
Schema dev
Chromeextension
Google Rich Results
Datashapes
ACL Web Alexa Meaning Representation Language
hash.aio Volcano schema
Yoast Structured Data Guide
Schema App
Springer: Scigraph
iPhylo Biodiversity KG
ozymandias
RDA group meeting notes
RDA Plenary meeting


Registries#
Documents and Datasets#
DOI
A not-for-profit membership organization that is the governance and management body for the federation of Registration Agencies providing Digital Object Identifier (DOI) services and registration, and is the registration authority for the ISO standard (ISO 26324) for the DOI system. The DOI system provides a technical and social infrastructure for the registration and use of persistent interoperable identifiers, called DOIs, for use on digital networks.
Datacite
Locate, identify, and cite research data with the leading global provider of DOIs for research data.
Archival Resource Key or ARK:  and
N2T ARKs and Names to Thinkgs
People#
Orcid
ORCID’s mission is to enable transparent and trustworthy connections between researchers, their contributions, and their affiliations by providing a unique, persistent identifier for individuals to use as they engage in research, scholarship, and innovation activities.
Organizations#
re3data
A registry of research data repositories
ROR
ROR is a community-led project to develop an open, sustainable, usable, and unique identifier for every research organization in the world.
Grid
GRID is a free and openly available global database of research-related organisations, cataloging research-related organisations and providing each with a unique and persistent identifier. With GRID you have over 99,609 carefully curated records at hand, enabling you to identify and distinguish research-related institutions worldwide.


Physical Samples#
IGSN
The objective of the IGSN e.V. is to implement and promote standard methods for identifying, citing, and locating physical samples with confidence by operating an international IGSN registration service.


Known Issues#
About#
This document will collect some of the various issues we have encounter in publishing
the JSON-LD documents.
control characters in URL string for sitemap or in the JSON-LD documents#
Make sure there are no control characters such as new line, carriage returns,
tabs or others in the document.  These can be problematic both for processing and
display.
context is a map (changed from 1.0)#
Be sure to use a context style like:
    "@context": {
        "@vocab": "https://schema.org/"
    },
The context section must be a map starting in JSON-LD 1.1
data graphs need @id#
Be sure to include an @id in your graph that points to the identifier or the
web address of the resource providing the metadata.  This is not the material the
metadata is about, but rather the metadata record itself.
string literals must be valid#
The string literals must be sure to not have quotation marks or other invalid
characters without escaping or encoding them.


Appendix#
About#
A collection of items related to the OIH development.  Mostly related
to examples around representing concepts in the graph such as
date and time, language etc.
As these develop they may be moved into other sections of the book.


Controlled Vocabularies#
About#
See also:  ODIS-ARCH Vocabularies
A list of possible controlled vocabularies to use in the schema.org documents.
Many such resources can be found by searching at BARTOC.org
or the UNESCO Thesaurus.
Note, at present this is an exploration and there is not yet a recommendation
for use in OIH.
List#
Essential Climate Variables
Vocabulary based on DFG’S Classification of Subject Area, Review Board, Research Area and Scientific Discipline (2016 - 2019)
Nature Subjects Ontology


Indexing with Gleaner#
Gleaner (app)#
The Gleaner applications performs the retrieval and loading of JSON-LD documents
from the web following structured data on the web patterns.  Gleaner is available for Linux, Mac OS X and Windows.
While Gleaner is a stand alone app, it needs to interact with
an object store to support data storage and other operations.  These dependencies are met within the
Gleaner Indexing Services or Data Service Docker compose files.
Warning
This documentation is in development.  The primary testing environments are Linux and other UNIX based platforms
such as Mac OS X.   If you are on Windows, there may be some issues.  If you can use a Linux subsystem on Windows,
you may experience better results.  We will test with Windows eventually and update documentation as needed.
Quick Start steps#
This quick start guide is focused on setting up and testing Gleaner in a local environnement.  It is similar to
how you might run Gleaner in a production environment but lacks the routing and other features likely desired for
such a situation.
Note
This documentation assumes a basic understanding of Docker and experience with basic Docker activities like
starting and stopping containers.  It also assumes an understanding of using a command line interface and
editing configuration files in the YAML format.
Command
From this point down, the documentation will attempt to put all commands


Command
From this point down, the documentation will attempt to put all commands
you should issue in this admonition style box.
In the end, this is the table of applications and config files you will need.  In this guide we will go through
downloading, setting them up and running Gleaner to index documents from the web.
Table 1 Required Applications and Their Config Files#
Gleaner
Docker
Minio Client
config.yaml
setenv.sh
load2blaze.sh
schemaorg-current-https.jsonld
gleaner-DS-NoRouter.yml
Grab Gleaner and the support files we need#
We will need to get the Gleaner binary for your platform and also the Gleaner configuration file
template.  To do this, visit the Gleaner Releases page
and pick the release Ocean InfoHubdev rc1.  Under the Assets drop down you should see the files we need.  Get:
Gleaner for your platform
Gleaner config template: template_v2.0.yaml
Gleaner indexing service compose file: gleaner-IS.yml
Helper environment setup script: setenvIS.sh
For this demonstration, we will be running on linux, so this would look something like:
Command
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/gleaner


curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/gleaner-IS.yml
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/setenvIS.sh
curl -L -O https://github.com/earthcubearchitecture-project418/gleaner/releases/download/2.0.25/template_v2.0.yaml
Note
You can download these with any tool you wish or through the browser.  Above we downloaded used the command
line curl tool.  For GitHub, be sure to add the -L to inform curl to follow redirects to the object to download.
Command
You may need to change the permission on your gleaner file to ensure it can be run.   On Linux this would
look something like the following.
chmod 755 gleaner
We then need to visit Schema.org for Developers to pull down the
appropriate JSON-LD context.  For this work we will want to pull down the schemaorg-current-https in JSON-LD format.
It also should work to do something similar to the following:
Command
curl -O https://schema.org/version/latest/schemaorg-current-https.jsonld
About the compose file(s)#
The above steps have collected the resources for the indexer.   We now want to set up the services that


Gleaner will use to perform the indexing.  To do that we use Docker or an appropriate run time alternative like
Podman or others.   For this example, we will assume you are using the Docker client.
As noted, a basic understanding of Docker and the ability to issue Docker cli commands to start and stop
containers is required. If you are new do Docker, we recommend you visit and read:
Get Started with Docker.
We need to select the type of services we wish to run.  The various versions of these Docker compose
file can be found in the Gleaner-compose deployment directory.
Why pick one over the other?
Choose Gleaner IS if you simply wish to retrieve the JSON-LD into a data warehouse to use in your own workflows
Choose Gleaner DS if you wish to build out a graph and want to use the default contains used by Gleaner.
Note
We wont look at this file in detail here since there will hopefully be no
required edits.  You can see the file in detail in the Index Services
section.
Edit environment variables setup script#
We have Docker and the appropriate compose file.  The compose files require a set of environment variables
to be populated to provide the local hosts information needed to run.  You can set these yourself or
use or reference the setenv.sh file in the Gleaner-compose repository in the
Gleaner-compose deployment directory.


Gleaner-compose deployment directory.
You may also need to visit information about permissions at
Post-installation steps for Linux if you are
having permission issues.
Let’s take a look at the script.
 1#!/bin/bash
 2
 3# Object store keys
 4export MINIO_ACCESS_KEY=worldsbestaccesskey
 5export MINIO_SECRET_KEY=worldsbestsecretkey
 6
 7# local data volumes
 8export GLEANER_BASE=/tmp/gleaner/
 9mkdir -p ${GLEANER_BASE}
10export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
11export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
You may wish to edit file to work better with your environment.  By default it will attempt to
use localhost to resolve with and host local runtime data in a /tmp/gleaner directory.
Spin up the containers#
Load our environment variables to the shell:
Command
source setenv.sh
Then start the containers:
Command
docker-compose -f gleaner-IS.yml up -d
If all has gone well, you should be able to see your running containers with
Command
docker ps
and see results similar to:
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                    NAMES


CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                    NAMES
c4b7097f5e06        nawer/blazegraph                 "docker-entrypoint.s…"   8 seconds ago       Up 7 seconds        0.0.0.0:9999->9999/tcp   test_triplestore_1
ca08c24963a0        minio/minio:latest               "/usr/bin/docker-ent…"   8 seconds ago       Up 7 seconds        0.0.0.0:9000->9000/tcp   test_s3system_1
24274eba0d34        chromedp/headless-shell:latest   "/headless-shell/hea…"   8 seconds ago       Up 7 seconds        0.0.0.0:9222->9222/tcp   test_headless_1
Edit Gleaner config file#
We have all the files we need and we have our support services running.  The next and
final step is to edit our Gleaner configuration file.  This will let Gleaner know
the location of the support services, the JSON-LD context file and the locations
of the resources we wish to index.
Let’s take a look at the full configuration file first and then break down each section.
 1---
 2minio:


1---
 2minio:
 3  address: 0.0.0.0
 4  port: 9000
 5  accessKey: worldsbestaccesskey
 6  secretKey: worldsbestsecretkey
 7  ssl: false
 8  bucket: gleaner
 9gleaner:
10  runid: oih # this will be the bucket the output is placed in...
11  summon: true # do we want to visit the web sites and pull down the files
12  mill: true
13context:
14  cache: true
15contextmaps:
16- prefix: "https://schema.org/"
17  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
18- prefix: "http://schema.org/"
19  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
20summoner:
21  after: ""      # "21 May 20 10:00 UTC"
22  mode: full  # full || diff:  If diff compare what we have currently in gleaner to sitemap, get only new, delete missing
23  threads: 1


23  threads: 1
24  delay: 0  # milliseconds (1000 = 1 second) to delay between calls (will FORCE threads to 1)
25  headless: http://0.0.0.0:9222  # URL for headless see docs/headless
26millers:
27  graph: true
28  #geojson: false
29sitegraphs:
30- name: aquadocs
31  url: https://oih.aquadocs.org/aquadocs.json
32  headless: false
33  pid: https://www.re3data.org/repository/aquadocs
34  properName: AquaDocs
35  domain: https://aquadocs.org
36sources:
37- name: samplesearth
38  url: https://samples.earth/sitemap.xml
39  headless: false
40  pid: https://www.re3data.org/repository/samplesearth
41  properName: Samples Earth (DEMO Site)
42  domain: https://samples.earth
43- name: marinetraining
44  url: https://www.marinetraining.eu/sitemap.xml
45  headless: false
46  pid: https://www.re3data.org/repository/marinetraining
47  properName: Marine Training EU


47  properName: Marine Training EU
48  domain: https://marinetraining.eu/
49- name: marineie
50  url: http://data.marine.ie/geonetwork/srv/eng/portal.sitemap
51  headless: true
52  pid: https://www.re3data.org/repository/marineie
53  properName: Marine Institute Data Catalogue
54  domain: http://data.marine.ie
55- name: oceanexperts
56  url: https://oceanexpert.org/assets/sitemaps/sitemapTraining.xml
57  headless: false
58  pid: https://www.re3data.org/repository/oceanexpert
59  properName: OceanExpert UNESCO/IOC Project Office for IODE
60  domain: https://oceanexpert.org/
61# - name: obis
62#   url: https://obis.org/sitemap/sitemap_datasets.xml
63#   headless: false
64#   pid: https://www.re3data.org/repository/obis
65#   properName: Ocean Biodiversity Information System
66#   domain: https://obis.org
Object store#
1minio:
2  address: 0.0.0.0
3  port: 9000
4  accessKey: worldsbestaccesskey


3  port: 9000
4  accessKey: worldsbestaccesskey
5  secretKey: worldsbestsecretkey
6  ssl: false
7  bucket: gleaner
The minio section defines the IP and port of the object store.  For this case, we are
using minio and these are the IP and port from our docker compose steps above.  Note,
if you were to use Ceph or AWS S3, this section is still labeled minio.  You simply
need to update the property values.
Gleaner#
1gleaner:
2  runid: oih # this will be the bucket the output is placed in...
3  summon: true # do we want to visit the web sites and pull down the files
4  mill: true
This passes a few high level concpets.
runid:
summon
mill
Context sections#
1context:
2  cache: true
3contextmaps:
4- prefix: "https://schema.org/"
5  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
6- prefix: "http://schema.org/"
7  file: "./jsonldcontext.json"  # wget http://schema.org/docs/jsonldcontext.jsonld
Comments for the context sections
Summoner section#
1summoner:


Comments for the context sections
Summoner section#
1summoner:
2  after: ""      # "21 May 20 10:00 UTC"
3  mode: full  # full || diff:  If diff compare what we have currently in gleaner to sitemap, get only new, delete missing
4  threads: 1
5  delay: 0  # milliseconds (1000 = 1 second) to delay between calls (will FORCE threads to 1)
6  headless: http://0.0.0.0:9222  # URL for headless see docs/headless
Comments for the summoner sections
Millers section#
1millers:
2  graph: true
3  #geojson: false
Comments for the miller sections
Site graphs section#
1sitegraphs:
2- name: aquadocs
3  url: https://oih.aquadocs.org/aquadocs.json
4  headless: false
5  pid: https://www.re3data.org/repository/aquadocs
6  properName: AquaDocs
7  domain: https://aquadocs.org
Comments for the sitegrpah sections
Sources section#
 1sources:
 2- name: samplesearth
 3  url: https://samples.earth/sitemap.xml


3  url: https://samples.earth/sitemap.xml
 4  headless: false
 5  pid: https://www.re3data.org/repository/samplesearth
 6  properName: Samples Earth (DEMO Site)
 7  domain: https://samples.earth
 8- name: marinetraining
 9  url: https://www.marinetraining.eu/sitemap.xml
10  headless: false
11  pid: https://www.re3data.org/repository/marinetraining
12  properName: Marine Training EU
13  domain: https://marinetraining.eu/
14- name: marineie
15  url: http://data.marine.ie/geonetwork/srv/eng/portal.sitemap
16  headless: true
17  pid: https://www.re3data.org/repository/marineie
18  properName: Marine Institute Data Catalogue
19  domain: http://data.marine.ie
20- name: oceanexperts
21  url: https://oceanexpert.org/assets/sitemaps/sitemapTraining.xml
22  headless: false
23  pid: https://www.re3data.org/repository/oceanexpert
24  properName: OceanExpert UNESCO/IOC Project Office for IODE
25  domain: https://oceanexpert.org/
26# - name: obis


25  domain: https://oceanexpert.org/
26# - name: obis
27#   url: https://obis.org/sitemap/sitemap_datasets.xml
28#   headless: false
29#   pid: https://www.re3data.org/repository/obis
30#   properName: Ocean Biodiversity Information System
31#   domain: https://obis.org
Comments for the sources sections
Run gleaner#
For this example we are going to run Gleaner directly.  In a deployed instance you may
run Gleaner via a script or cron style service.  We will document that elsewhere.
We can do a quick test of the setup.
Command
 ./gleaner -cfg template_v2.0 -setup
For now, we are ready to run Gleaner.  Try:
Command
 ./gleaner -cfg template_v2.0
Note
Leave the suffix like .yaml off the name of the config file.  The config system can also read
json and other formats.  So simply leave the suffix off and let the config code inspect the
contents.
Load results to a graph and test#
You have set up the server environment and Gleaner and done your run.  Things look good
but you don’t have a graph you can work with yet.    You need to load the JSON-LD into


the triplestore in order to start playing.
Minio Object store#
To view the object store you could use your browser and point it on the default minio
port at 9000.  This typically something like localhost:9000.
If you wish to continue to use the command line you can use the Minio client at
Minio Client Quickstart guide.
Once you have it installed and working, you can write an entry for our object store with:
Command
 ./mc alias set minio http://0.0.0.0:9000 worldsbestaccesskey worldsbestsecretkey
Load Triplestore#
We now want to load these objects, which are JSON-LD files holding RDF based graph
data, into a graph database.  We use the term, triplestore, to define a graph database
designed to work with the RDF data model and provide SPARQL query support over that
graph data.
Simple script loading
Nabu
Try out a simple SPARQL query
References#
The following are some reference which may provide more information on the various
technologies used in this approach.
Google: Understanding how structured data works
Google Dataset Search By the Numbers
Google Dataset Search: Building a search engine for datasets in an open Web ecosystem
W3C SPARQL
SHACL
Triplestores


Interfaces#
About#
In the end the goal is to provide use of the generated index.  There are several
possible used for an index.
Web UI such as the reference client at oceans.collaborium.io
A variation on this is the development of web components that can be easily included in
domain sites to perform operations on the OIH index
graph access via SPARQL
access to the graph and objects via workflows like Jupyter notebooks
Gleaner Web UI (WUI)#
The user of the index may take several forms.  A user may be a software developer creating a web based
interface to the generated index.  It may also be an end user accessing the index (indexes) through notebooks
or special clients.
Those wishing to run a web site can augemnt the compose files to run their
preferred web server, object server (to serve files from the object store) or
software such as node or others to support their deployment pattern.
Fig. 6 Gleaner Optional Web UI#


Alternatives#
Options#
While Gleaner will be used during initial OIH development it is not
the only or required approach.   The web architecture foundation means there are many other tools that
can be used and might be leveraged in a production environment including:
Extrunct
BioSchemas Tools
LDSpider
StormCrawler
Squirrel
Nutch (Apache)
Laundromat
DataArchiver
OD Archiver
These different tools may better fit into the workflow and available
skill sets for a group.


Graph First Approach#
About#
During the early adopters meetings and in discussion with others an alternative publication pattern came up.  This is the pattern where it is not possible to update the web resources with the metadata content.  This may be due to access or technical issues.  Regardless, what was possible was to generate the metadata in bulk locally and make the resulting document available.
This approach is not ideal since it is a non-standard pattern and makes the data and information more obscure to other users.  However, it is one the OIH architecture can adapt to and is preferable to the option of excluding those partners in this activity.
As such, we are making some changes to allow for this pattern.  This means documenting the published graph structure based on the existing thematic patterns and some updates in the indexing workflow to obtain and integrate these graphs into the OIH graph.
Warning
Anti-pattern:
Using the approach here is not in alignment with
Google guidance nor with W3C patterns for structured
data on the web.
It is documented here for edge cases
where this is the minimum viable approach.  The hope is it
could act as a gateway to a more standards aligned implementation later.
Graph Only#
There are cases where it is only possible to generate the graph
based on the metadata.  Access to the HTML pages is either difficult or the process of inserting the
data into the pages is not supportable.


data into the pages is not supportable.
For this case the goal is to create a simple graph in JSON-LD.  To do this we need a collection
approach that is valid for a range of Things.
For this it is proposed to use ItemList which can be used on a list of type Thing, ie anything
type in the Schema.org vocabulary.
This would define a ListItem with item of any type.  Below is an example for a CreativeWork (map)
and a Course.  Once you are in a “item” any of the details from the other thematic type descriptions
can be used.
{
  "@context": "https://schema.org/",
  "@type": ["ItemList", "CreativeWork"],
  "@id": "https://example.org/id/graph/X",
  "name": "Resource collection for site X",
  "author": "Creator of the list",
  "itemListOrder": "https://schema.org/ItemListUnordered",
  "numberOfItems": 2,
  "itemListElement": [
    {
      "@type": "ListItem",
      "item": {
           "@id": "ID_for_this_metadata_record1",
           "@type": "Map",
            "@id": "https://example.org/id/XYZ",
            "name": "Name or title of the document",


"name": "Name or title of the document",
            "description": "Description of the map to aid in searching",
            "url":  "https://www.sample-data-repository.org/creativework/map.pdf"
      }
    },
    {
      "@type": "ListItem",
      "item": {
           "@id": "ID_for_this_metadata_record2",
            "@type": "Course",
            "courseCode": "F300",
            "name": "Physics",
            "provider": {
                "@type": "CollegeOrUniversity",
                "name": "University of Bristol",
                "url": {
                    "@id": "/provider/324/university-of-bristol"
                }
            }
        }
    }
  ]
}
In the case of schema:Dataset one might use schema:DataCatlogue for the following approach.  However,
since OIH is addressing a wide range of types a more generic collection of Things or CreativeWorks
approach is needed.
Item Catalogue Page#
It’s not hard to generate a simple HTMl page based on the structured metadata file.  This doesn’t
alter the content of the graph, just builds an automated HTMl page around it.
Publishing and referencing#
Testing#


Publishing and referencing#
Testing#
Since we are now dealing with a graph that is pulled as a complete entity there are a few thoughts.
How do ensure a connection between a record in the list and a resolvable URL?  Do we need
to:
ensure each record has a IRI it is subject of
in the case where IRI is or can be URL, do a validation of at least a 200 on it
How do we publish this?
entry in robots.txt (might be able due to reasons above?)
published and provided to OIH
Need guidance on format and structure


Data Services#
The typical functional goal of this work is the development and use of a Graph that can be accessed via a triplestore (Graph Database).  To do that we need a set of additional containers to support this and expose these services on the web through a single domain with https support.
Object Store
An S3 compliant object store supporting S3 APIs including S3Select.  For open source this is best satisfied with the Minio Object Store.  For commercial cloud AWS S3 or hosted Ceph services will work.
Graph Database
Web Router (technically optional)
Gleaner Data Services (DS)#
If you wish to work with a triplestore and wish to use the default app used by OIH
you can use the compose file that sets up the Gleaner Data Services environment.
This adds the Blazegraph triplestore to the configuration along with the object store.
The details of the OIH data services are found in the
Data Services section.
Fig. 3 Gleaner Data Service Activity Workflow#
Typically, a user would wish to run the full Gleaner DS stack which supports
both the indexing process and the serving of the resulting data warehouse and graph
database capacity.
Combined, these would then look like the following where the indexing and
data services shared a common object store.
Fig. 4 Gleaner Indexing and Data Service Combined#
Object store pattern#


Fig. 4 Gleaner Indexing and Data Service Combined#
Object store pattern#
Within in the object store the following digital object pattern is used.
This is based on the work of the RDA Digital Fabric working group.
Fig. 5 Gleaner Digital Object Pattern#
At this point the graph and data warehouse (object store) can be exposed to the net for use by clients such as jupyter notebooks or direct client calls to the S3 object APIs and SPARQL endpoint.
Gleaner Data Services (DS) Environment Variables
The Docker Compose file used to launch the Gleaner DS has a set of configurable elements that can be set and passed to the orchestration system with environment variables.
These can be set manually or through the command line.  A simple script to set the variables could look like:
– Environment Var settings script
The actual services can be deployed via a Docker Compose file (also works with Podman).  An example of that file and details about it follow.
Let’s take a look at this.
 1#!/bin/bash
 2
 3# domains
 4export GLEANER_ADMIN_DOMAIN=admin.local.dev
 5export GLEANER_OSS_DOMAIN=oss.local.dev
 6export GLEANER_GRAPH_DOMAIN=graph.local.dev
 7export GLEANER_WEB_DOMAIN=web.local.dev


7export GLEANER_WEB_DOMAIN=web.local.dev
 8export GLEANER_WEB2_DOMAIN=web2.local.dev
 9
10# Object store keys
11export MINIO_ACCESS_KEY=worldsbestaccesskey
12export MINIO_SECRET_KEY=worldsbestsecretkey
13
14# local data volumes
15export GLEANER_BASE=/tmp/gleaner/
16export GLEANER_TRAEFIK=${GLEANER_BASE}/config
17export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
18export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
19
– Break down the compose file here
  1version: '3'
  2
  3# ${GLEANER_ADMIN_DOMAIN}
  4# ${GLEANER_OSS_DOMAIN}
  5# ${GLEANER_GRAPH_DOMAIN}
  6# ${GLEANER_WEB_DOMAIN}
  7# ${GLEANER_WEB2_DOMAIN}
  8# ${MINIO_ACCESS_KEY}
  9# ${MINIO_SECRET_KEY}
 10#
 11# ${GLEANER_TRAEFIK}
 12# ${GLEANER_OBJECTS}
 13# ${GLEANER_GRAPH}
 14
 15services:
 16  triplestore:


14
 15services:
 16  triplestore:
 17    image: nawer/blazegraph
 18    environment:
 19      JAVA_XMS: 2g
 20      JAVA_XMX: 8g
 21      JAVA_OPTS: -Xmx6g -Xms2g --XX:+UseG1GC
 22    ports:
 23      - 9999:9999
 24    labels:
 25      - "traefik.enable=true"
 26      - "traefik.http.routers.triplestore.entrypoints=http"
 27      - "traefik.http.routers.triplestore.rule=Host(`${GLEANER_GRAPH_DOMAIN}`)"
 28      - "traefik.http.middlewares.triplestore-https-redirect.redirectscheme.scheme=https"
 29      - "traefik.http.routers.triplestore.middlewares=triplestore-https-redirect"
 30      - "traefik.http.routers.triplestore-secure.entrypoints=https"
 31      - "traefik.http.routers.triplestore-secure.rule=Host(`${GLEANER_GRAPH_DOMAIN}`)"
 32      - "traefik.http.routers.triplestore-secure.tls=true"


33      - "traefik.http.routers.triplestore-secure.tls.certresolver=http"
 34      - "traefik.http.routers.triplestore-secure.service=triplestore"
 35      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowmethods=GET,OPTIONS,PUT,POST"
 36      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolalloworigin=*"
 37      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolmaxage=200"
 38      - "traefik.http.middlewares.triplestore-secure.headers.addvaryheader=true"
 39      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowcredentials=true"
 40      - "traefik.http.middlewares.triplestore-secure.headers.accesscontrolallowheaders=Authorization,Origin,Content-Type,Accept"
 41      - "traefik.http.middlewares.triplestore-secure.headers.customresponseheaders.Access-Control-Allow-Headers=Authorization,Origin,Content-Type,Accept"
 42      - "traefik.http.routers.triplestore-secure.middlewares=triplestore-secure@docker"
 43      - "traefik.http.services.triplestore.loadbalancer.server.port=9999"


44      - "traefik.docker.network=traefik_default"
 45    volumes:
 46      - ${GLEANER_GRAPH}:/var/lib/blazegraph
 47    networks:
 48      - traefik_default
 49
 50  s3system:
 51    image: minio/minio:latest
 52    ports:
 53      - 9000:9000
 54    labels:
 55      - "traefik.enable=true"
 56      - "traefik.http.routers.s3system.entrypoints=http"
 57      - "traefik.http.routers.s3system.rule=Host(`${GLEANER_OSS_DOMAIN}`)"
 58      - "traefik.http.middlewares.s3system-https-redirect.redirectscheme.scheme=https"
 59      - "traefik.http.routers.s3system.middlewares=s3system-https-redirect"
 60      - "traefik.http.routers.s3system-secure.entrypoints=https"
 61      - "traefik.http.routers.s3system-secure.rule=Host(`${GLEANER_OSS_DOMAIN}`)"
 62      - "traefik.http.routers.s3system-secure.tls=true"


63      - "traefik.http.routers.s3system-secure.tls.certresolver=http"
 64      - "traefik.http.routers.s3system-secure.service=s3system"
 65      - "traefik.http.services.s3system.loadbalancer.server.port=9000"
 66      - "traefik.docker.network=traefik_default"
 67    volumes:
 68      - ${GLEANER_OBJECTS}:/data
 69    environment:
 70      - MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY}
 71      - MINIO_SECRET_KEY=${MINIO_SECRET_KEY}
 72    networks:
 73      - traefik_default
 74    command: ["server", "/data"]
 75
 76  features:
 77    image: fils/grow-general:latest
 78    ports:
 79      - 8080:8080
 80    environment:
 81      - S3ADDRESS=s3system:9000
 82      - S3BUCKET=sites
 83      - S3PREFIX=domain
 84      - DOMAIN=https://${GLEANER_WEB_DOMAIN}/
 85      - S3KEY=${MINIO_ACCESS_KEY}
 86      - S3SECRET=${MINIO_SECRET_KEY}
 87    labels:


86      - S3SECRET=${MINIO_SECRET_KEY}
 87    labels:
 88      - "traefik.enable=true"
 89      - "traefik.http.routers.features.entrypoints=http"
 90      - "traefik.http.routers.features.rule=Host(`${GLEANER_WEB_DOMAIN}`, `${GLEANER_WEB2_DOMAIN}`)"
 91      - "traefik.http.middlewares.features-https-redirect.redirectscheme.scheme=https"
 92      - "traefik.http.routers.features.middlewares=features-https-redirect"
 93      - "traefik.http.routers.features-secure.entrypoints=https"
 94      - "traefik.http.routers.features-secure.rule=Host(`${GLEANER_WEB_DOMAIN}`,`${GLEANER_WEB2_DOMAIN}`)"
 95      - "traefik.http.routers.features-secure.tls=true"
 96      - "traefik.http.routers.features-secure.tls.certresolver=http"
 97      - "traefik.http.routers.features-secure.service=features"
 98      - "traefik.http.services.features.loadbalancer.server.port=8080"
 99      - "traefik.docker.network=traefik_default"


99      - "traefik.docker.network=traefik_default"
100      - "traefik.http.middlewares.features.headers.accesscontrolallowmethods=GET,OPTIONS,PUT,POST"
101      - "traefik.http.middlewares.features.headers.accesscontrolalloworigin=*"
102      - "traefik.http.middlewares.features.headers.accesscontrolmaxage=100"
103      - "traefik.http.middlewares.features.headers.addvaryheader=true"
104      - "traefik.http.middlewares.features-secure.headers.accesscontrolallowheaders=*"
105      - "traefik.http.middlewares.features-secure.headers.customresponseheaders.Access-Control-Allow-Headers=*"
106    networks:
107      - traefik_default
108
109networks:
110  traefik_default:
111
NOTE:  DS also needs the object -> graph sync (via Nabu)
NOTE:  Should also add in (here or to the side) the ELT local Data Lake to Data Warehouse path (ala CSDCO VaultWalker)


Indexing Services#
Gleaner can not run alone and relies on a couple of Open Container Initiative (OCI) containers to support it.  For this document, we will assume you are using Docker but this will work with Podman or other OCI compliant orchestration environments.   These Gleaner Indexing Services are necessary to use Gleaner.   The exception to this would be if you are using a 3rd party objects store like AWS S3 or Wasabi.
Object Store
An S3 compliant object store supporting S3 APIs including S3Select.  For open source this is best satisfied with the Minio Object Store.  For commercial cloud AWS S3 or hosted Ceph services will work.
Headless Chrome (technically optional)
This is only needed where you expect the sources you index to use Javascript to include the JSON-LD in the pages.  If you know your sources do not use this publishing pattern and rather include the JSON-LD in the static page, then you don’t need this container running.
IS represent the minimum required services to support Gleaner.  With IS you have an object
store in the form of Minio and a headless chrome server in the form of
chromedp/headless-shell.
As shown in the figure below, and support the basic harvesting of resources with Gleaner
and loading the JSON-LD objects into Minio.


and loading the JSON-LD objects into Minio.
It does not result in these objects ending up in a graph / triplestore.   You would use
this option if you intend to work on the JSON-LD objects yourself.  Perhaps loading
them into a alternative graphdb like Janus or working on them with python tooling.
Fig. 2 Basic Gleaner Indexing Service Activity Workflow#
Gleaner Indexing Services (IS) Environment Variables
The Docker Compose file used to launch the Gleaner IS has a set of configurable elements that can be set and passed to the orchestration system with environment variables.
These can be set manually or through the command line.  A simple script to set the variables could look like:
#!/bin/bash
# domains
export GLEANER_ADMIN_DOMAIN=admin.local.dev
export GLEANER_OSS_DOMAIN=oss.local.dev
export GLEANER_GRAPH_DOMAIN=graph.local.dev
export GLEANER_WEB_DOMAIN=web.local.dev
export GLEANER_WEB2_DOMAIN=web2.local.dev
# Object store keys
export MINIO_ACCESS_KEY=worldsbestaccesskey
export MINIO_SECRET_KEY=worldsbestsecretkey
# local data volumes
export GLEANER_BASE=/tmp/gleaner/
export GLEANER_TRAEFIK=${GLEANER_BASE}/config


export GLEANER_TRAEFIK=${GLEANER_BASE}/config
export GLEANER_OBJECTS=${GLEANER_BASE}/datavol/s3
export GLEANER_GRAPH=${GLEANER_BASE}/datavol/graph
The actual services can be deployed via a Docker Compose file (also works with Podman).  An example of that file and details about it follow.
– Break down the compose files here  link to them


Aggregator#
Intoduction#
This section introduces the the OIH approach to indexing. Currently,  OIH is
using the Gleaner software to do the indexing and leverages the Gleaner IO
gleaner-compose Docker
Compose files for the server side architecture.  For more information on Docker Compose files visit the
Overiew of Docker Compose.  The gleaner-compose repository holds Docker compose files that can set up
various environments that Gleaner needs.
The figure below gives a quick overview of the various compose options for setting up
the supporting architecture for Gleaner. A fully configured system where all the indexing
and data services are running and exposing services to the net, a total of five containers are run.  In many case
you may run fewer than this.
Fig. 1 The various compose options for Gleaner#
Container overview#
S3 (Minio / AWS): This is the only container that is required in all cases to run.  Gleaner needs an S3 compatible object store.  By default
we use the Minio object store
Chrome Headless: In cases where providers place the JSON-LD documents into the pages with Javascript, we need to render the page before
reading and accessing the DOM.  This is done using Chrome Headless


reading and accessing the DOM.  This is done using Chrome Headless
Graph data base:  Gleaner extracts JSON-LD documents from resources.  These JSON-LD documents are representations of the RDF data mode.  To
queries on them at scale, it easiest to load the triples into a compatible graph database.  Sometimes we call this a triplestore.  For OIH we
use the Blazgraph triplestore.
Router: If we wish to deploy this setup onto the net, we will route to route all the services through a single domain.  To do this network routing
we use Traefik.  This router is not required for local use and alternative routers like Caffdy
or nginx are also valid options.
Web Server: If you wish to serve a web UI for the index, then you can also leverage this setup to serve that.  Again, this is optional and your web site
may be hosted elswhere and simply call to the index in compliance with CORS settings.  There is an example web server that leverages the object store
available in this setup.
Gleaner#
As mentioned Gleaner is a single binary app (ie, one file).  It can be run on Linux, Mac OS X or Windows.  It does
not need to be run on the same machine as the supporting services as it can connect to them


not need to be run on the same machine as the supporting services as it can connect to them
over the network.  So, for example, they could be hosted in commercial cloud services or
on remote servers.
You can download and compile the code from the previously mentioned github repository or
the releases page.
A single configuration file provides the settings Gleaner needs.  Additionally, a local
copy of the current schema.org context file should be downloaded and available to the app.  This file is needed
for many operations and access it over the net is slow and often rate limited depending on the
source.   You can download the file at the Schema.org for Developers page.
This setup show in the above figure is the typical setup for Gleaner and is
detailed in the Quick Start section.
Indexing Services#
This is the basic indexing service requirements.  At a minimum we need the object store and the Chrome headless containers
scoped in the Indexing Sevices box above. More details on this set can be found in Indexing Servives.
Data Services#
A more expanded set of services is defined in the Data Services section.  This section discussion a setup more
designed to address a server setup tht will support indexing and also present the resulting indexes to the broader internet.
Web UI#


Web UI#
As mentioned, if you wish to serve a web UI for the index, then you can leverage this setup to serve that.  Again, this is optional and your web site
can be hosted elsewhere and simply call to the index in compliance with CORS settings. Some tailes on this can be found in the Interfaces section.
Alternatives#
Note, the Gleaner ecosystem is not a requirement.  OIH follows the structured data on the web and data on the web best practices patterns.  Being web architecture based, there are many open source tools and scripting solutions you might use.  You may wish to explore the Alternative Approaches section for more on this.
What follows is a bit more detail on the setup used by Gleaner.  Experienced users will
see where they can swap out elements for their own preference.  Like a different
triplestore, or wish to leverage a commercial object store?  Simply modify the architecture
to do so.
ODIS Catalog as Index Source#
Before we discuss indexing source a key question is what source will be indexed.
OIH is not a web crawl in that it doesn’t move from source to source based on
the content of those sources.
Rather, the OIH index is based on a list of sources selected ahead of time.  At
this time that set of sources if based on those partners engaged in the


this time that set of sources if based on those partners engaged in the
development phase of OIH.  As the work moves to a more routine operation
the sources will come from the ODIS Catalog.
The ODIS Catalog will then act as a curated source of domains for inclusion
in the Ocean InfoHub.   This will provide a level of curation and vetted of sources
and ensure sources are aware of the technical requirements for inclusion in the
OIH index.


Gleaner CLI Docker#
About#
This is a new approach for quick starts with Gleaner.  It is a script that exposes
a containerized version of Gleaner as a CLI interface.
You can use the -init flag to pull down all the support files you need including
the Docker Compose file for setting up the object store, a triplestore and
the support for headless indexing.
Prerequisites#
You need Docker installed.  Later, to work with the results and load them into a
triplestore, you will also need an S3 compatible client.  We will use the Minio
client, mc, for this.
Steps#
Download the script gleanerDocker.sh from https://github.com/earthcubearchitecture-project418/gleaner/tree/master/docs/cliDocker You may need to make it run-able with
curl -O https://raw.githubusercontent.com/earthcubearchitecture-project418/gleaner/master/docs/cliDocker/gleanerDocker.sh
chmod 755 gleanerDocker.sh
Next you can run the script with the -init flag to pull down all the support files you need.
./gleanerDocker.sh -init
This will also download the needed docker image and the support files.
Your directory should look like this now:
fils@ubuntu:~/clidocker# ls -lt
total 1356


fils@ubuntu:~/clidocker# ls -lt
total 1356
-rw-r--r-- 1 fils fils    1281 Aug 15 14:07 gleaner-IS.yml
-rw-r--r-- 1 fils fils     290 Aug 15 14:07 setenvIS.sh
-rw-r--r-- 1 fils fils    1266 Aug 15 14:07 demo.yaml
-rw-r--r-- 1 fils fils 1371350 Aug 15 14:07 schemaorg-current-https.jsonld
-rwxr-xr-x 1 fils fils    1852 Aug 15 14:06 gleanerDocker.sh
Let’s see if we can setup our support infrastructure for Gleaner.  The
file gleaner-IS.yml is a docker compose file that will set up the object store,
and a triplestore.
To do this we need to set up a few environment variables.  To do this we can
leverage the setenvIS.sh script.  This script will set up the environment we need.
Note you can also use a .env file or other approaches.  You can references
the Environment variables in Compose documentation.
root@ubuntu:~/clidocker# source setenvIS.sh
root@ubuntu:~/clidocker# docker-compose -f gleaner-IS.yml up -d


root@ubuntu:~/clidocker# docker-compose -f gleaner-IS.yml up -d
Creating network "clidocker_traefik_default" with the default driver
Creating clidocker_triplestore_1 ... done
Creating clidocker_s3system_1    ... done
Creating clidocker_headless_1    ... done
Note:  In a fresh run all the images will be pulled down.  This may take a while.
In the end, you should be able to see these images running:
root@ubuntu:~/clidocker# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS                                              NAMES
a26f7c945479        nawer/blazegraph                 "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:9999->9999/tcp                             clidocker_triplestore_1
f3a4197c42be        minio/minio:latest               "/usr/bin/docker-ent…"   About a minute ago   Up About a minute   0.0.0.0:9000->9000/tcp, 0.0.0.0:54321->54321/tcp   clidocker_s3system_1


062f029462b1        chromedp/headless-shell:latest   "/headless-shell/hea…"   About a minute ago   Up About a minute   0.0.0.0:9222->9222/tcp
At this point we should be able to do a run.  During the init process a
working config file was downloaded.
Note:  This config file will change…  it’s pointing to an OIH partner
and I will not do that for the release.  I have a demo site I will use.
Next we need to setup our object for Gleaner.  Gleaner itself can do this
task so we will use
root@ubuntu:~/clidocker# ./gleanerDocker.sh -setup -cfg demo
main.go:35: EarthCube Gleaner
main.go:110: Setting up buckets
check.go:58: Gleaner Bucket gleaner not found, generating
main.go:117: Buckets generated.  Object store should be ready for runs
Note:  Here is where we go off the rails.  The config file uses 0.0.0.0 as the
location and this is not working.   You need to edit the config file with the
“real” IP of the host machine.  In may case is this 192.168.122.77.  This is


obviously still a local network IP but it does work.  I am still investigating
this issue.
We can now do a run with the example template file.
Note:  Best to delete the “sitegraph” node, I will do that soon.  It should
work, but is currently slow and gives little feedback
If everything goes well, you should see something like the following:
root@ubuntu:~/clidocker# ./gleanerDocker.sh -cfg demo
main.go:35: EarthCube Gleaner
main.go:122: Validating access to object store
check.go:39: Validated access to object store: gleaner.
org.go:156: Building organization graph (nq)
org.go:163: {samplesearth  https://samples.earth/sitemap.xml false https://www.re3data.org/repository/samplesearth Samples Earth (DEMO Site) https://samples.earth}
main.go:154: Sitegraph(s) processed
summoner.go:16: Summoner start time: 2021-08-15 14:34:08.907152656 +0000 UTC m=+0.067250623
resources.go:74: samplesearth : 202


resources.go:74: samplesearth : 202
 100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (202/202, 20 it/s)
summoner.go:34: Summoner end time: 2021-08-15 14:34:20.36804137 +0000 UTC m=+11.528139340
summoner.go:35: Summoner run time: 0.191015
webfeed.go:37: 1758
millers.go:26: Miller start time: 2021-08-15 14:34:20.368063453 +0000 UTC m=+11.528161421
millers.go:40: Adding bucket to milling list: summoned/samplesearth
millers.go:51: Adding bucket to prov building list: prov/samplesearth
 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (202/202, 236 it/s)


graphng.go:77: Assembling result graph for prefix: summoned/samplesearth to: milled/samplesearth
graphng.go:78: Result graph will be at: results/runX/samplesearth_graph.nq
pipecopy.go:16: Start pipe reader / writer sequence
graphng.go:84: Pipe copy for graph done
millers.go:80: Miller end time: 2021-08-15 14:34:21.84702814 +0000 UTC m=+13.007126109
millers.go:81: Miller run time: 0.024649
Working with results#
If all has gone well, at this point you have downloaded the JSON-LD documents into Minio or
some other object store.Next we will install a client that we can use to work with these objects.
Note, there is a web interface exposed on the port mapped in the Docker compose file.
In the case of these demo that is 9000.  You can access it at
http://localhost:9000/ with the credentials set in the environment variable file.
However, to work with these objects it would be better to use a command line tool, like mc.
The Minio Client, can be installed
following their Minio Client Quickstate Guide.
Be sure to place it somewhere where it can be seen from the command line, ie, make sure it


is in your PATH variable.
If you are on Linux this might look something like:
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help
There is also a Minio Client Docker image
that you can use as well but it will be more difficult to use with the following scripts due
to container isolation.
To man an entry in the mc config use:
mc alias set oih  http://localhost:9000 worldsbestaccesskey worldsbestsecretkey
We should now be able to list our object store.  We have set it up using the alias oih.
user@ubuntu:~/clidocker# mc ls oih
[2021-08-15 14:31:20 UTC]     0B gleaner/
user@ubuntu:~/clidocker# mc ls oih/gleaner
[2021-08-19 13:36:04 UTC]     0B milled/
[2021-08-19 13:36:04 UTC]     0B orgs/
[2021-08-19 13:36:04 UTC]     0B prov/
[2021-08-19 13:36:04 UTC]     0B results/


[2021-08-19 13:36:04 UTC]     0B summoned/
You can explore mc and see how to copy and work with the object store.
Loading to the triplestore#
As part of our Docker compose file we also spun up a triplestore.  Let’s use that now.
Now Download the minio2blaze.sh script.
curl -O https://raw.githubusercontent.com/earthcubearchitecture-project418/gleaner/master/scripts/minio2blaze.sh
chmod 755 minio2blaze.sh
The content we need to load into the triplestore needs to be in RDF for Blazegraph.  We also need
to tell the triplestore how we have encoded that RDF.   If look in the object store at
mc ls oih/gleaner/milled
[2021-08-19 13:26:52 UTC]     0B samplesearth/
We should see a bucket that is holding the RDF data converted from the JSON-LD.  Let’s use this
in our test.   We can pass this path to the minio2blaze.sh script.  This script will go looking
for the mc command we installed above, so be sure it is in a PATH location that script can see.
./minio2blaze.sh oih/gleaner/milled/samplesearth


./minio2blaze.sh oih/gleaner/milled/samplesearth
...   lots of results removed
If all has gone well, we should have RDF in the triplestore.  We started our triplestore as part of
the docker-compose.yml file.  You can visit the triplestore at http://localhost:9999/blazegraph/#splash
Note, you may have to try other addresses other than localhost if your networking is a bit different with
Docker.  For me, I had to use a real local IP address for my network, you might also try 0.0.0.0.
Hopefully you will see something like the following.
We loaded into the default kb namespace, so we should be good there.  We can see that is listed
as the active namespace at the Current Namespace: kb report.
Let’s try a simple SPARQL query.  Click on the Query tab to get to the query user interfaced. We
can use something like:
select *
where
{
  ?s ?p ?o
}
LIMIT 10
A very simple SPARQL to give us the first 10 results from the triplestore.  If all has gone well,
we should see something like:
You can explore more about SPARQL and the wide range of queries you can do with it


You can explore more about SPARQL and the wide range of queries you can do with it
at the W3C SPARQL 1.1 Query Language reference.
Conclusion#
We have attempted here to give a quick introduction to the use of Gleaner in a Docker
environment.  This is a very simple example, but it should give you an idea of the approach
used.  This approach can then be combined with other approaches documented to establish a
more production oriented implementation.  Most of this documentation will be located
at the Gleaner.io GitHub repository and Gleaner repository.
Note: The plan is to merge the Gleaner.io GitHub repository into the first.


Prov#
About#
This is the start of some discussion on issues around prov tracking in OIH.
This may take two paths.  One would be the prov tracking indexers might do
and the other prov that providers would encode to provide specific prov
the community requests.
Gleaner Prov#
The Gleaner application generates a prov graph of the activity of accessing
and indexing provider resources.  The main goal of this prov is to connect
an indexed URL to the digital object stored in the object store.  This
digital object should be the JSON-LD data graph presented by the provider.
By contrast, the authoritative reference in the various profiles will connect
the the data graph ID, or in the absence of that the data graph URL or the
referenced resources URL by gleaner, to another reference.  This may be
an organization ID or a PID of the connected resource.
 1{
 2    "@context": {
 3        "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
 4        "prov": "http://www.w3.org/ns/prov#",
 5        "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
 6    },
 7    "@graph": [
 8        {


6    },
 7    "@graph": [
 8        {
 9            "@id": "https://www.re3data.org/repository/obis",
10            "@type": "prov:Organization",
11            "rdf:name": "Ocean Biodiversity Information System",
12            "rdfs:seeAlso": "https://obis.org"
13        },
14        {
15            "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
16            "@type": "prov:Entity",
17            "prov:wasAttributedTo": {
18                "@id": "https://www.re3data.org/repository/obis"
19            },
20            "prov:value": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2"
21        },
22        {
23            "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55",
24            "@type": "prov:Collection",
25            "prov:hadMember": {


25            "prov:hadMember": {
26                "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2"
27            }
28        },
29        {
30            "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55",
31            "@type": "prov:Entity",
32            "prov:value": "7c1eaa1aaed95861330109026c42e57a31ecae55.jsonld"
33        },
34        {
35            "@id": "https://gleaner.io/id/run/7c1eaa1aaed95861330109026c42e57a31ecae55",
36            "@type": "prov:Activity",
37            "prov:endedAtTime": {
38                "@value": "2021-04-20",
39                "@type": "http://www.w3.org/2001/XMLSchema#dateTime"
40            },
41            "prov:generated": {


40            },
41            "prov:generated": {
42                "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55"
43            },
44            "prov:used": {
45                "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55"
46            }
47        }
48    ]
49}
import json
from rdflib.extras.external_graph_libs import rdflib_to_networkx_multidigraph
from rdflib.extras.external_graph_libs import rdflib_to_networkx_graph
from pyld import jsonld
import graphviz
import os, sys
currentdir = os.path.dirname(os.path.abspath(''))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
from lib import jbutils
with open("../../../odis-in/dataGraphs/indexing/prov/graphs/gleaner.json") as dgraph:
    doc = json.load(dgraph)
frame = {
  "@context": {"@vocab": "https://schema.org/",
  "prov": "http://www.w3.org/ns/prov#"},
  "@explicit": "false",


"@explicit": "false",
  "@type":     "prov:Activity",
   "prov:generated": {},
   "prov:endedAtTime": {},
   "prov:used": {}
}
context = {
  "@vocab": "https://schema.org/",
  "prov": "http://www.w3.org/ns/prov#"
}
compacted = jsonld.compact(doc, context)
framed = jsonld.frame(compacted, frame)
jd = json.dumps(framed, indent=4)
print(jd)
{
    "@context": {
        "@vocab": "https://schema.org/",
        "prov": "http://www.w3.org/ns/prov#"
    },
    "@id": "https://gleaner.io/id/run/7c1eaa1aaed95861330109026c42e57a31ecae55",
    "@type": "prov:Activity",
    "prov:endedAtTime": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2021-04-20"
    },
    "prov:generated": {
        "@id": "urn:gleaner:milled:obis:7c1eaa1aaed95861330109026c42e57a31ecae55",


"@type": "prov:Entity",
        "prov:value": "7c1eaa1aaed95861330109026c42e57a31ecae55.jsonld"
    },
    "prov:used": {
        "@id": "https://gleaner.io/id/collection/7c1eaa1aaed95861330109026c42e57a31ecae55",
        "@type": "prov:Collection",
        "prov:hadMember": {
            "@id": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
            "@type": "prov:Entity",
            "prov:value": "https://obis.org/dataset/9381239f-3d64-48b4-80c9-b9ebb674edc2",
            "prov:wasAttributedTo": {
                "@id": "https://www.re3data.org/repository/obis",
                "@type": "prov:Organization",
                "http://www.w3.org/1999/02/22-rdf-syntax-ns#name": "Ocean Biodiversity Information System",
                "http://www.w3.org/2000/01/rdf-schema#seeAlso": "https://obis.org"
            }


}
        }
    }
}
Nano Prov#
This is a basic nanoprov example. Note, this is a draft and
the ID connections and examples have not been made yet.
 1{
 2    "@context": {
 3        "gleaner": "https://voc.gleaner.io/id/",
 4        "np": "http://www.nanopub.org/nschema#",
 5        "prov": "http://www.w3.org/ns/prov#",
 6        "xsd": "http://www.w3.org/2001/XMLSchema#"
 7    },
 8    "@set": [
 9        {
10            "@id": "gleaner:nanopub/XID",
11            "@type": "np:NanoPublication",
12            "np:hasAssertion": {
13                "@id": "gleaner:nanopub/XID#assertion"
14            },
15            "np:hasProvenance": {
16                "@id": "gleaner:nanopub/XID#provenance"
17            },
18            "np:hasPublicationInfo": {
19                "@id": "gleaner:nanopub/XID#pubInfo"
20            }
21        },
22        {


20            }
21        },
22        {
23            "@id": "gleaner:nanopub/XID#assertion",
24            "@graph": {
25                "@id": "DataSetURI",
26                "@type": "schema:Dataset",
27                "description": "This is where you would put corrections or annotations",
28                "identifier": [
29                    {
30                        "@type": "schema:PropertyValue",
31                        "name": "GraphSHA",
32                        "description": "A SHA256 sha stamp on the harvested data graph from a URL",
33                        "value": "{{SHA256 HASH HERE}}"
34                    },
35                    {
36                        "@type": "schema:PropertyValue",
37                        "name": "ProviderID",
38                        "description": "The id provided with the data graph by the provider",
39                        "value": "{{re3 or URL noted in config}}"
40                    },
41                    {
42                        "@type": "schema:PropertyValue",
43                        "name": "URL",
44                        "description": "The URL harvested by gleaner",
45                        "value": "{{The URL the JSON-LD came from}}"
46                    }
47                ]
48            }
49        },
50        {


46                    }
47                ]
48            }
49        },
50        {
51            "@id": "gleaner:nanopub/XID#provenance",
52            "@graph": {
53                "@id": "URIforprovondataset",
54                "prov:wasGeneratedAtTime": {
55                    "@value": "dateDone",
56                    "@type": "xsd:dateTime"
57                },
58                "prov:wasDerivedFrom": {
59                    "@id": "IDHERE"
60                },
61                "prov:wasAttributedTo": {
62                    "@id": "IDHERE"
63                }
64            }
65        },
66        {
67            "@id": "gleaner:nanopub/XID#pubInfo",
68            "@graph": {
69                "@id": "IDHERE",
70                "prov:wasAttributedTo": {
71                    "@id": "gleaner:tool/gleaner"
72                },
73                "prov:generatedAtTime": {
74                    "@value": "2019-10-23T14:38:00Z",
75                    "@type": "xsd:dateTime"
76                }
77            }
78        }
79    ]
80}
import json


77            }
78        }
79    ]
80}
import json
from pyld import jsonld
import os, sys
currentdir = os.path.dirname(os.path.abspath(''))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
from lib import jbutils
with open("../../../odis-in/dataGraphs/indexing/prov/graphs/nanoprov.json") as dgraph:
    doc = json.load(dgraph)
context = {
    "@vocab": "https://schema.org/",
}
compacted = jsonld.compact(doc, context)
jbutils.show_graph(compacted)
Fontconfig error: "/etc/fonts/conf.d/30-0-google-crosextra-caladea-fonts.conf", line 1: Extra content at the end of the document
Fontconfig error: "/etc/fonts/conf.d/60-open-sans.conf", line 1: Extra content at the end of the document
Fontconfig error: "/etc/fonts/conf.d/62-google-crosextra-caladea-fonts.conf", line 1: Extra content at the end of the document
Refs#
Nanopubs Guidance


Tooling#
About#
The tooling section is a collection of tools, scripts, notebooks and other software that could
be of use to the various personas of Ocean InfoHub
OpenRefine#
In this section you will find some details around
the Open Refine project and how to use it to
generate JSON-LD documents.
Notebooks#
In this section you will find some Jupyter Notebooks
that demonstrate working with JSON-LD in various
ways.  These can be copied and used locally to
explore or implement workflows.
On-line tooling#
Schema.org Validator
json-ld playgroud
SHACL Playground
json-lint
f-uji
Dev#
json-ld
f-uji dev
jq
jello (python) and
Practical JSON at the command line using jello
Python Extruct


Personas#
About#
During the design process of the Ocean InfoHub (OIH), many of the design approached leverage three
personas that help define the various archtypes of people who engage with OIH.  It should not be assumed
these scope all the potential persona or that a person or organization scope only one.  It is quite possible
to many.   These are simply design approaches representing potential models or characters.   They
are tools used in the design process of OIH.
Publisher: A key persona whose activities are covered in detail in Publishing patterns for OIH
Aggregator: Leverages web architecture to retrieve structured data on the web and generate usable indexes.
User: The end user of the publishing and aggregation activities.  May leverage
the web for discovery or tools such as Jupyter for analytics and visualization.
Persona: Publisher#
In OIH the Publisher is engaged authoring the JSON-LD documents and publishing them
to the web.  This persona is focused on describing and presenting structured data on the web
to aid in the discovery and use the resources they manage.
Details on this persona can be found in the Publisher section.
Additionally, this persona would be leveraging this encoding described in the JSON-LD Foundation section and the
profiles described in the Thematic Patterns.
Persona: Aggregator#
In OIH the Aggregator is a person or organization who is indexing resources on the


In OIH the Aggregator is a person or organization who is indexing resources on the
web using the structured data on the web patterns described in this documentation.
Their goal is to efficiently and efficiently index the resources exposed by the Publisher
persona and generate usable indexes.  Further, they would work to exposed these indexes in
a manner that is usable by the User persona.
Details on the approach used by OIH and potential alternatives can be found in the
Aggregator section.
Persona: User#
The user is the individual or community who wished to leverage the indexes generated
as a result of the publishing and aggregation activities. The user may be using the
developed knowledge graph or some web interface built on top of the knowledge graph or
other index.  They may also use query languages like SPARQL or other APIs or even
directly work with the underlying data warehouse of collected data graphs.
User tools may be web sites or scientific notebooks.  Some examples of these
user experiences are described in the User section.