Any source, one model
SQL databases, Kafka topics, JSON, CSV, XML, spreadsheets, REST APIs and documents. Each has its own Transformer and they all map to the same model, so adding a fourth source does not mean a fourth idea of what a customer is.
Knowledge graph ETL
ETL for a graph is not ETL for a table
Loading a warehouse means getting rows into the right columns. Loading a knowledge graph means deciding what every row means, and then saying it the same way every time, from every source. Graph.Build Transformers are the part that does that.
What changes
The hard part is identity, not throughput
Two systems both know about the same customer and neither agrees with the other about what to call them. A table can hold both. A graph has to choose.
the difference
The mapping is where the meaning gets decided
A graph ETL job resolves an identity for every record, then emits triples, or nodes and edges if you are building a property graph. Get the identity right and data from four systems converges on one node. Get it wrong and you have four nodes that look like a network and are actually a filing error. Graph.Build handles that in the mapping rather than in code: the mappings are RML and R2RML, which are open W3C specifications, so what decides the meaning of your data is a file you can read and diff. Over a hundred built-in functions cover the usual value cleaning, and you can write your own for anything specific to you.
The consequence: the logic that decides what your data means is reviewable, rather than buried in an ingest application.
customers.json · transform
mapping customer-v14
gbo:Customer/88213
rdf:type
gbo:Customer
gbo:Customer/88213
hasLegalName
Ardent Logistics
gbo:Customer/88213
holdsAccount
gbo:Account/GB-4471
gbo:Account/GB-4471
openedOn
2019-04-02 (date)
n-triples
11 emitted from one record, checked against customer.ontology
all passed
unmapped: 0 · a mapping that does not conform to the model is flagged before it runs
What it gives you
Six things that are painful to build and dull to maintain
Every graph project ends up needing all six. The question is only whether you write them yourself.
Scale by adding containers
Transformers are stateless containers reading from a queue. When a load is too slow you run more of them, which is a deployment change rather than a rewrite.
scaling horizontal
Consistency you can check
The ontology constrains the ETL, so a mapping that does not fit the model is caught as a warning at design time rather than as a strange node three months later.
validated against the model
It stays current
Runs can write straight into your graph. With change data capture, a create, update or delete in the source database moves through Kafka and reaches the graph without waiting for tonight's rebuild.
latency batch or live
No target lock-in
The same transformed output can be written to a semantic graph or a property graph, over SPARQL, Cypher or Gremlin. Changing your mind about the database does not mean rebuilding the pipeline.
targets RDF and LPG
Provenance, if you want it
Every statement can carry where it came from and when, using W3C PROV. That is the difference between a graph an auditor accepts and one they ask questions about.
standard PROV-O
Can we not just use our existing ETL tool?
You can get data out of it, and people do. What a general purpose tool has no concept of is the model: it will happily load two nodes for the same customer, because as far as it is concerned those were two rows. The graph-specific work is identity resolution, ontology conformance and emitting statements rather than records, and that is what you would end up writing by hand on top of it.
Where does the mapping live?
In RML and R2RML files, which are open specifications rather than our format. They are produced from the model in Studio, and they are readable, diffable and version-controlled. If you stopped using Graph.Build tomorrow, the description of what your data means would still be yours and still be standard.
Does it handle messy data?
Within reason. There are over a hundred built-in functions for string handling, dates, arithmetic, hashing and lookups, and you can add your own. Records that cannot be made to fit go to a failure topic with the reason attached, rather than being silently dropped or silently loaded.
How does the loaded graph stay up to date?
A transform can write straight to the graph when it runs. For continuous updates, change data capture watches the source database and feeds every change through Kafka to a Transformer, so the graph moves when the source moves. Setups are documented for MySQL, Postgres and MongoDB.
Where does it run?
In your own infrastructure. The components are containers, deployed on AWS, on Google Cloud, or on hardware you own. Your data does not leave your account to be transformed.
Next
The ETL this page is about is called Transformers
Bring a source you already have and we will map it in front of you, which settles the question faster than any page can.
- SQL, Kafka, files, REST and documents
- Open mappings in RML and R2RML
- Runs in your account, not ours