You are here
The graph engineering lifecycle
Design, configure, engineer, iterate and automate graph model production: the whole lifecycle, without writing bespoke ingest code for each project.
Transformers
A Transformer is an independent processor that reads one kind of source and emits the graph data your model describes. It is configured against the model you published in Studio, not written against a schema, so refining the model does not mean rewriting an application.
How it works
A Transformer has no opinion about your domain. It carries the mapping you published and applies it, record by record, until the source is exhausted.
mechanism
You point a Transformer at a source and hand it a mapping. For each record it resolves an identity, applies the mapping and emits triples, or nodes and edges if you are building a property graph. Where a value needs work before it fits, over a hundred built-in functions cover the usual cases, string handling, dates, arithmetic, hashing and lookups, and you can write your own for anything specific to you. The mappings themselves are RML and R2RML, which are open specifications, so what decides the meaning of your data is a file you can read and diff rather than a codebase you have to maintain.
The consequence: changing your mind about the model costs an afternoon of configuration, not a development cycle.
n-triples
11 emitted from one record, validated against customer.ontology
all passed
unmapped: 0 · records that do not fit go to the failure topic with their reason
Choosing
Each Transformer specialises in how it reaches data, not in what the data means. Meaning comes from the model, which is why a setup can start as one container against one source and grow without the model being touched.
JSON, XML, CSV, XLSX and ODS. Point it at a path or a URL and it reads the file; give it a Kafka topic and it reads messages as they arrive. Several mappings can run from one source, so one file can populate more than one part of the model.
reaches files · URLs · topics
Anything with a JDBC driver, which in practice means MySQL, SQL Server, Oracle, Postgres and the rest. Queries can be iterated in pages so a table too large to hold in memory is still a single configured job rather than a script somebody babysits.
reaches any JDBC source
A RESTful endpoint that returns JSON or XML is a URL, and a URL is something the semi-structured Transformer already reads. Cloud applications and internal services come in the same way a file does, with the same mapping and the same output.
reaches any JSON or XML API
The other option
Everything above copies your source into a graph. There is a second way to use the same model, and it is worth knowing about before you assume the copy is compulsory.
materialise or virtualise
A materialised graph is built by a Transformer and loaded by a Writer. The data is copied, it is fast to traverse, and it is as current as its last run or its change feed. Virtualisation uses the same Studio model and the same R2RML mapping, and copies nothing: a SPARQL query is translated into SQL against your relational database as it is asked, and the answer comes back shaped by the ontology. Build the model, connect the database, write the query. It is relational sources only, because R2RML maps tables, and the things that come with an ingest do not apply. There is no provenance, no change data capture, and no graph left behind when you close the connection.
The consequence: nothing to store and nothing to keep in step, in exchange for your source database's own query performance. It suits data that is not allowed to move, and it suits trying a model out before committing a database to it.
Change data capture
A graph that is rebuilt every night is a graph people learn not to trust between rebuilds. Change data capture removes the gap.
staying current
Debezium watches the source database and writes every create, update and delete to Kafka. Because that payload arrives as JSON, the semi-structured Transformer picks it off the topic and maps it exactly as it maps anything else, producing RDF or CSV for just the part that moved. The output lands as a file, its URL goes back onto a topic, and the Writer applies it to your graph. Setups are documented for MySQL, Postgres and MongoDB.
The consequence: the same mapping serves the first full load and every change after it. There is no second pipeline to write and no second thing to keep correct.
Whichever you run
These are properties of the platform rather than of any one connector, which is the point of the connectors being interchangeable.
Every ingest records where the data came from and when, using PROV-O rather than a convention of ours. You get a record of the data over time, so what the graph said last quarter is still answerable this quarter.
model W3C PROV-O
Transformers are separate containers with no shared state, so a load that is taking too long is answered by running more of them. Volume is a deployment question rather than a rewrite.
unit one container
Over a hundred built-in functions handle the transformations that come up in every project. When your domain needs something they do not cover, you add a custom function rather than pre-processing the source somewhere else.
built in 100+
Records that fail the mapping are not dropped and do not stop the run. They go to a Kafka failure topic with the reason attached, so they can be reviewed and processed later once the model or the source is corrected.
on failure review, then re-run
Also available
These sit outside the standard set and are scoped with you rather than listed with a price. If one of them is the source that matters to you, a conversation is the right place to start.
For cloud applications, internal services and third-party APIs where the endpoint needs more than a URL: authentication, pagination, rate limits and the particular shape a given vendor returns. Configured against the same model as every other Transformer, producing the same output.
availability talk to us
For DOCX and PDF, where there is no schema to map because the structure is prose. It reads the documents with an LLM and pulls out a graph, and the thing that makes that trustworthy rather than creative is that it is aimed at an ontology you have already published. The model still comes first. The LLM is finding your classes and properties in the text, not deciding what the classes should be.
availability talk to us
The fastest way to judge this is to watch one of your own files, tables or endpoints become graph data against a model of your domain.