You are here
The graph engineering lifecycle
Design, configure, engineer, iterate and automate graph model production: the whole lifecycle, without writing bespoke ingest code for each project.
The graph engineering lifecycle
There are eight steps, and a practitioner runs all of them every time. Traditionally most of the work sat in software development rather than in modelling, which is why knowledge graphs have cost more and taken longer than the thinking behind them would suggest. This is the whole lifecycle, and an honest account of which parts the platform takes off you.
What the lifecycle is
Nobody gets a knowledge graph model right first time, and nobody should be expected to. What matters is what a lap round the loop costs, because that is what decides how many you can afford.
the shape of it
You identify a domain, define its entities and relationships, decide which attributes to record, model the graph, map your data into it, validate what came out and refine what was wrong. Then you go again, because validation always tells you something. Traditionally each lap meant changing a bespoke ingest application written for the previous version of the model, so the cost of a knowledge graph scaled with the number of times you learned something, which is a terrible property for a thing you are trying to get right.
The consequence: teams stopped iterating long before the model was finished, because every improvement had to be justified as a development project.
repeats until the model is right
then, once it is right
Four stages, eight steps, one arrow back. Everything inside the ring repeats, and what it costs to go round decides how good the model gets.
The eight steps
The first four are thinking about your domain and cannot be automated, nor should they be. The last four are where the engineering used to be, and they are the ones the platform is built to make cheap.
Decide what the graph is about and, more usefully, what it is not about. For a retailer that might be products, customers and orders, and the relationships between them. Scope set badly here is the most expensive mistake available.
step 01
Name the things you want to hold information about. These become the classes in your model: Product, Customer, Order, Store. This is the vocabulary the rest of the organisation will end up using, so it is worth arguing about.
step 02
Say how the entities connect. Purchased, viewed, recommended, supplied by. These are the edges, and they are what makes a graph worth having, because they are the part a relational schema makes expensive to traverse.
step 03
Decide what you record about each entity and each relationship. Name, description, price and category on a product; the date and time on a purchase. Attributes on relationships are the ones most often forgotten and most often needed.
step 04
Turn the decisions above into a schema: an ontology for a semantic graph, or a property graph schema. This is the blueprint, and the thing every later step is checked against. In Studio it is drawn rather than written, and the formal artefact is generated from it.
step 05
Connect real sources to the model: which column is that property, which table is that class, which nested field is that relationship. Historically the longest step by a wide margin, because it meant writing an application per source. Here it is configuration against the model.
step 06
Check that what came out represents the domain and answers the questions it was built for. Real output, in a real database, queried the way the application will query it. A model validated on a diagram has not been validated.
step 07
Change the schema, the entities, the relationships or the attributes in light of what validation told you, and run the lifecycle again. How cheap this step is determines how good the finished model gets.
step 08
What runs each step
Nothing here is a separate product to integrate. The model authored in step five is the same artefact the transformation executes in step six and the same one you validate against in step seven.
Steps 1 to 6. The domain, the entities, the relationships, the attributes, the model and the mapping, drawn rather than coded, with the ontology generated from what you drew.
outputs · model + mapping
Step 6, executed. The mapping runs against your real sources, so what you validate in step 7 is real output and not a sample somebody prepared.
outputs · RDF or CSV
Steps 7 and 8. The graph lands in a database you can query, and refining it is the same handle turned again rather than a new piece of work.
outputs · your graph database
What changes
The point of automating the lifecycle is not that the same project finishes sooner. It is that a different, better project becomes possible.
the payoff
When a lap costs an afternoon instead of a quarter, you stop defending the model and start improving it. Ideas get tried rather than debated, because trying one is faster than arguing about it. The people who understand the domain can be in the room for step seven rather than being shown a report about it two months later. And the modeller spends their time on the part that needs expertise, which is the first five steps, rather than on the mechanics of getting data to fit.
The consequence: improvements and corrections during design happen in days rather than months, and the model that reaches production is the one you actually meant.
the lifecycle 8 steps
1 to 4 · define your domain experts
5 · model Studio, visually
6 · map Studio, configured
7 · validate on real output
8 · refine and run it again
the thinking stays yours, the engineering does not
The honest answer is that the first version is rarely the hard part; the versions after it are. A first model over a bounded domain and a couple of sources is usually a matter of days. What used to take months was the loop, and that is the part that changes.
For steps one to four, someone has to think properly about the domain, and that skill is real whatever it is called. It does not have to be a hire. Plenty of teams do it with a subject-matter expert and an analyst, because the modelling is visual and does not require a query language to follow. We can also sit alongside you for the first one.
Yes, and it is often the better order. Steps one to six do not involve a database at all, and the transformation output stands on its own as RDF or node and edge CSV. Choosing the database once you have a model to test with is a much easier decision.
The mapping is configuration against the model, so a changed source is a changed mapping rather than a changed application. Where the source is a database under change data capture, the graph keeps itself in step without anyone re-running anything.
Step seven, done properly: load it and ask it the questions it was built to answer, using your own data. A model that looks right on a canvas and cannot answer the question the project was funded for is a common and expensive outcome, and the only defence is validating early and often enough that it surfaces while it is still cheap to fix.
No. The same model can produce a semantic graph or a labelled property graph, and you can build both from one domain if you are still deciding which suits the work.
Bring something somebody has already tried to model, an OWL file, a database schema or a whiteboard photograph, and we will take it round the loop with you.