RDF and property graph
Semantic and labelled property graph databases are both first-class targets. The model you built decides which output the transformation produces, and the Writer emits the load that target expects.
Writers
The model is the asset. The database is a decision you can change.
A Writer takes the graph data a transformation produced and lands it in your database, using that database's own native load. Anything speaking SPARQL, openCypher or Gremlin is a target as standard, and both RDF and property graph models are supported, from the same model.
How it works
Loading is a run with a number, not a migration
The Writer is handed the output of a transformation and applies it. That is the whole of its job, which is why it can be pointed somewhere else without anything upstream noticing.
mechanism
Two modes as standard, and a third that arrives with change data capture
Give a Writer the file a Transformer produced, RDF as n-quads or nodes and edges as CSV, and it writes it. There are two modes. Insert adds what it is given. Update reconciles it against what is already there, so a re-run corrects rather than accumulates. Deleting is the one thing a Writer will not do off its own bat, because a file of graph data says what exists and not what has stopped existing. That information only comes from change data capture, where the source database reports a deletion and the Writer applies it. Writers hold no state between runs and scale horizontally, so a first load that is taking too long is answered by running more of them rather than by waiting.
The consequence: you can afford to reload. When the model changes, putting the corrected graph in front of people is a run, and a run is minutes.
writer · run 5120 · memgraph
update mode
09:41:02
insert · nodes
+ 1,904
09:41:03
insert · edges
+ 4,118
09:41:03
update · nodes
~ 312
09:41:04
update · edges
~ 96
09:41:04
run complete
00:00:02
11:04:22
delete · via cdc
- 1 node · - 3 edges
The component
Four properties, and the fourth is the one people ask about
Writers are containers like everything else in the platform, deployed on-premise or in your own cloud, and added to a setup whenever you are ready for one.
Insert or update, and they answer different questions
Insert accumulates, so with provenance switched on you get a record of what the data said and when, and can ask what the graph looked like last quarter. Update keeps one current picture, which is what an application wants. Change data capture is update mode running continuously, so the picture is current to the minute rather than to the last batch.
modes insert · update · cdc
Horizontally scalable
No shared state and no coordinator, so throughput is a question of how many Writers you run. The size of the first load stops being an argument about whether the project is feasible.
unit one container
Optional, in the literal sense
You do not need a Writer, or a database, to get value out of a model. The transformation output stands on its own as files, and Writers can be added to an existing setup at any point.
add whenever you like
Vendor independence
Test two databases with one model, in an afternoon
The usual way to choose a graph database is to read benchmarks somebody else ran on somebody else's data. The better way is to run your own model on both and look. That is only affordable if the model is not written in either vendor's tooling.
- Run two Writers, one per candidate, against the same transformation output. Same model, same data, two databases to query.
- Rework the domain as a property graph as well as RDF and run all four combinations, if the formalism is the part you are unsure about.
- Move later without rebuilding anything upstream. The model does not know which database it is going to, so changing that is configuration and a re-run.
- Nothing about your model is held in a vendor's proprietary tool, so there is nothing to unpick from one before you can leave.
one model · four targets
customer-v14
Neo4j
openCypher
Amazon Neptune
Gremlin
Memgraph
openCypher
GraphDB
SPARQL
Same model, four generated loads. Changing target is a configuration change and a re-run.
A common question
What if we have not picked a graph database yet?
This is the position most teams are in, and it is a better position than it feels like.
if you have not chosen
The output is a file before it is a database
A transformation produces RDF as n-quads for semantic models, or node and edge CSV for property graphs, and those files are yours whether or not a Writer ever runs. You can download them, keep them, load them by hand, or hold them while the procurement conversation finishes. When you are ready, a Writer is added to the setup and given the same files.
The consequence: the database decision stops blocking the modelling work, and gets made later on evidence from your own graph.
transform output
run 5120
transform output run 5120
customers.nq n-quads · RDF
nodes.csv property graph
edges.csv property graph
provenance.nq PROV-O
download them, or hand any of them to a Writer
Is our database on the list?
If it speaks SPARQL, openCypher or Gremlin then it is supported as standard, which covers most of the market. The compatibility page lists the ones we see most often, grouped by query language, and it is added to regularly. If yours is not there, ask, because the answer is usually yes.
Does a re-run duplicate everything?
Less than people expect, and the detail is worth knowing. A semantic graph is a set of statements, so re-inserting a statement it already holds changes nothing: the same subject, predicate and object is stored once however many times you send it. What insert mode will do is add a second value where the object differs. If a supplier's address is inserted again with a new value, the supplier ends up holding both, which is the behaviour you want if you are keeping history and provenance and not the behaviour you want if you are keeping one current picture. Update mode is the second case: it reconciles rather than accumulates.
How fast is the first load?
It depends on your database more than on us, because we emit its native load rather than issuing statements one at a time. Where the Writer is the constraint, it scales sideways: run more of them. We would rather show you a run on your own data than quote a number from ours.
Can we write to two databases at once?
Yes, and it is a normal thing to do. Some organisations run one database for an application and another for analytics, fed from the same model. Others do it temporarily while migrating. Both are two Writers pointed at two places.
Where does it run?
In your account, on your infrastructure. Writers are Docker containers deployed on-premise or in AWS, Azure or Google Cloud alongside the rest of the platform. Your database credentials stay on your side and nothing is routed through us.
What happens to deletes?
They come from change data capture, and only from there. A file of graph data describes what exists, so nothing in a normal run tells a Writer that a record has gone away. Put the source under CDC and the source database reports the deletion itself, at which point the Writer applies it. This is the part usually missing from a nightly rebuild and the reason stale entities accumulate in graphs that are otherwise well kept.
Load your model into two databases and compare
Bring a domain and a shortlist. We will model it, transform a sample of your data and write it to both, so the comparison is yours rather than a vendor's.
- Any database speaking SPARQL, openCypher or Gremlin
- Runs in your account, on your infrastructure
- No obligation to buy anything