Inside Orion DB

An 84-byte record header, a write protocol that self-heals, an O(1) seek table that one line had switched off, and a timestamp that drifted

An 84-byte record header, a write protocol that self-heals, an O(1) seek table that one line had switched off, and a timestamp that drifted

Orion DB is the storage engine we wrote ourselves. It is not PostgreSQL with extensions and it is not a key-value store with a query layer bolted on. It is a single engine that serves five roles that a conventional stack needs five separate products to cover: a hierarchical object store, an ACID transactional database, a filesystem, a vector/RAG store for local AI, and a schemaless fast path for real-time reads.

I want to show you what is actually inside it — at the byte level — and then tell you about three bugs, because the bugs are where you learn whether someone really owns their storage layer or just claims to.

Everything below was measured against the running engine source or read out of a live database file. None of it is inferred.

The record header is 84 bytes, and MVCC lives in it

Every object in Orion is a record with a packed 84-byte header, then a body. Here is the layout:

Three things worth pulling out of that table.

MVCC is not a layer, it is three header fields. TransTime, EndTransTime, and TransNumber are in every record. A row is not overwritten in place — a new version is appended with its own transaction stamps, and a read walks versions and picks the one visible to the reader's transaction. There is no separate undo log, no rollback segment, no vacuum daemon. The version chain is the record chain.

A record belongs to three linked lists at once. One threads every object written by the same transaction. One threads objects in OID order. One threads objects within their collection. Different access patterns walk different chains, and they are independent — which turns out to matter enormously (see below).

**A pointer is gate * 0x1000 + offset.** Page size is 4096 bytes; a gate is a page number and the offset is page-relative. That is the whole addressing scheme.

One consequence that shapes everything else: after the header come TemplNum and RevNum, and then a template-driven body. A record's physical length is only knowable via its template. You cannot skip a record you cannot resolve the template for. Remember that.

The write protocol is crash-safe by construction

Writing an object is four steps:

1. Header first, with Flag = (1<<31) ^ F_OBJ_BEGIN. Bit 31 set means "still joining."

2. Body next — TemplNum, RevNum, field values.

3. Header rewritten to F_OBJ_PROCESSING, which clears bit 31.

4. At commit, PROCESSING flips to F_OBJ_WRITTEN and EndTransTime is stamped.

Now here is why a crash in the middle of a write cannot corrupt anything.

An interrupted write leaves a record sitting at F_OBJ_BEGIN. The enumerator treats F_OBJ_BEGIN as end-of-data — a begin record means "the list ends here." And because an append always happens at the tail, stopping at the torn record loses exactly nothing that was ever committed. The next process to open the file sees a clean list. Records left at F_OBJ_PROCESSING are validated against their transaction's commit state, and uncommitted ones are skipped.

So a crash mid-write cannot produce a committed record with a garbage body sitting in the middle of a list. The commit is a single-word flip, and it happens last.

I mention this because during one investigation a session proposed rewriting the protocol to "write the header last" for safety. That was wrong: the engine already commits last, and the proposal would have broken a property it was trying to add. The correction came from reading AddToList, not from arguing about it. When you own the engine, "read the function" is always available as a tiebreaker.

Enumeration is a physical page walk — and that has teeth

GetFirst/GetNext do not follow the per-record next-pointers for a full scan. They walk the collection's pages in physical order: inside a page, advance record by record (reading TemplNum to know how far to skip); across pages, follow the page link; and stop when the walk reaches the collection's recorded last page and offset, which live in the first page's header.

Two consequences fall straight out of this, and both bit us:

If the last-page metadata is stale, the walk stops early and silently. Records physically written past that point are fully valid and completely invisible to a sequential scan.

The OID index is a separate structure. WHERE obj_id = N consults an index that has nothing to do with the page walk. So a row can be index-visible and enumeration-invisible at the same time.

That asymmetry is a diagnostic signature, and we learned to read it the hard way. One of our git repositories started serving a stale master ref. Investigation showed the pushed ref was physically written, valid, and returned correctly by WHERE obj_id = 51. But the sequential enumeration — which the ref-advertisement path used — stopped at an earlier record, because the collection's last-page pointer was stale from an earlier incident. And because the ref upsert found existing refs by walking that same broken enumeration, it never found master, so it inserted a duplicate. The old ref stayed enumerable at a low OID; the new one sat unreachable at a high OID. The advertisement kept serving the old SHA.

Note what the failure was not. It was not a torn write — the protocol above self-heals those, and we proved it on a copy. It was not the malformed record we initially blamed; flipping that record's flag on a snapshot did not restore the walk, and when that experiment failed we should have dropped the theory immediately instead of rationalizing it. The rule we wrote down afterward: records present-but-unenumerable means suspect the enumeration metadata, not the record bytes.

Files are binary members, and they are seekable

The claim "the filesystem is the database" is easy to say and usually means "we store a blob and a path." In Orion it means something specific: a binary member supports offset-addressed random access read and write.

OPEN_BIN cid:<c>.oid:<o>.mm:raw TO &h;
int off = 0;
while(off < total){
    int want = chunk; if(total - off < want) want = total - off;
    int sz = want;
    if(elast_read_buf(h, buf, sz, off) != 1) break;
    /* process buf[0..sz) */
    off = off + sz;
}
CLOSE_BIN <h>;

elast_read_buf(handle, buffer, length, offset) and elast_write_buf(handle, buffer, length, offset) take an offset and go straight there. So an arbitrarily large object is stored as one binary member and streamed in bounded chunks. No large-object side table, no chunk-splitting layer, no external blob store.

We validated it on a real 14 MB stored object: read in 256 KB chunks by offset, 55 chunks, 9 milliseconds, reassembled byte-exact, with a 256 KB working set instead of a 14 MB one.

The O(1) seek table that one line had switched off

Here is the part I find most instructive.

A binary member can carry a page-index accelerator: an offset→page map, built by CreatePgTable, that lets a seek jump directly to the page containing a target offset. The read fast path checks for it and uses it. Both halves are complete code — not stubs.

It is opt-in via a mode flag. And the function that resolves that flag contained this:

int bList = bListW;
if(bListW == 3) bList = 0;   // strips the accelerator request

Unconditionally. Every request for the accelerator was silently downgraded, so no binary member in any database ever got the table.

Nothing broke. For years. Because the only access pattern in use was a whole-buffer read, which seeks once — to offset 0 — and a linear page walk from the start to offset 0 costs nothing.

Then we changed the access pattern. Chunked streaming seeks to increasing offsets. Without the table, every seek is a linear page walk from the beginning, so reading an n-byte blob in fixed chunks becomes O(n²/chunk) instead of O(n). For a 703 MB blob that is roughly 2,800 seeks, each one walking deeper than the last — tens of billions of page steps. With the table it is O(1) per seek.

The lesson is not "we had a bug." The lesson is that a disabled optimization is invisible until the access pattern that needs it shows up, and the symptom then presents as a performance cliff in brand-new code rather than as a regression in old code. If we had not owned the engine we would have spent that investigation profiling our own loop.

The bug that needed four root causes: read-your-own-writes

This is the best debugging story in the engine, and the first three answers were all wrong.

The problem. An import did roughly 678 inserts per document — a word-index fan-out — at about 290 ms per insert, because each row committed on its own and paid its own fsync. Times 618 documents, that is hours. The fix is obvious: wrap each document in one transaction, one fsync per document, and it becomes minutes.

It did not work. Every document failed with No one item updated.

Root cause #1: MVCC visibility. IsValidObject, the version walk, only treated a version as visible if its transaction was F_TRANSAC_COMMITED. A row inserted inside the currently open transaction is F_OBJ_PROCESSING — so it was judged invisible to its own transaction. An UPDATE ... WHERE obj_id issued moments after the INSERT, in the same transaction, found nothing.

That is a real gap, and the fix is called read-your-own-writes: if a processing version belongs to the reader's own transaction, treat it as visible and let it shadow committed versions. We matched the transaction by pointer identity. Built it. It did not fire.

Root cause #2: the read path was in a different transaction. If the pointer comparison never matches, maybe the read handle carries a different transaction than the insert did. And indeed — START TRANSACTION put one global transaction on the shared root, but the function every read and write went through started a separate per-collection transaction whenever a fresh handle had none. So the INSERT and the UPDATE's read genuinely were in different transactions. (That also meant per-operation writes would not commit with the global commit — a latent data-loss bug in its own right.) We attached the global transaction instead. It still did not fire.

Root cause #3: wrong layer entirely. New hypothesis, and it fit every symptom: the version walk only runs after an index lookup, and an in-transaction OID is not in the index until commit. So the lookup returns not-found and bails out before the version walk ever executes — which would explain why the own-write branch never ran, and why a non-transactional update of the same row also failed as "not on committed disk." Coherent, consistent with the evidence, and also wrong.

The actual root cause. The pointer-identity check was correct all along. It was just sitting inside a guard that compared the transaction's begin time against the record's TransTime. And the function that attached transactions to handles re-stamped hTrans->Phys.BeginTime with the current time on every operation. So by the time the UPDATE's read ran, the transaction's begin time had drifted away from the value stamped into the row at insert. The guard rejected the own-write even though the pointer match had already returned true.

An instrumented build printed both values side by side — txn_begin != obj_tr_time — and that single line ended a six-day investigation. The fix was to move the own-write check outside the timestamp guard. Result: 10 documents, 0 failures, 0 errors. Transactional batching works.

And a coda that matters more than the fix. With the engine correct, the import was still slow — because the word-index fan-out was routed out to a separate service on a separate connection, and a transaction cannot span that. The engine bug was real and the engine fix was necessary, and it was not sufficient. The architecture had to follow the engine.

Four hypotheses, three of them wrong, one line of instrumentation that settled it. Everything I now believe about debugging storage engines comes from that ratio.

The allocator was never capped — its failure policy was the bug

One more, because it is a good example of misreading a symptom.

The process died at around 2.8 GB and the obvious conclusion was a memory cap. It was not. The page allocator reserves memory in 1.5 GB arena blocks — mmap with PROT_NONE, then commit with mprotect — and it holds a table of up to a thousand such blocks, so it can address hundreds of gigabytes. There was no OS limit in play: no cgroup ceiling, unlimited rlimits, ~22 GB free.

What actually happened was a failed commit. Under the kernel's default overcommit heuristic, a large contiguous commit gets refused when committed memory is already high, even with plenty of free RAM. And the allocator's policy on a failed commit was to print a message and call exit(). One oversized allocation killed the entire service.

The oversized allocation itself came from code that buffered whole objects and whole packs in RAM — a 13 MB stored object inflating into a buffer that doubled its way to a gigabyte. The storage layer was fine. The offset API from earlier in this article was right there, unused.

So the real defect was three defects wearing one symptom: an allocator whose failure mode is process death instead of a NULL return, calling code that allocated in whole-object units, and an accelerator that would have made the streaming alternative viable, switched off by one line.

What owning the database actually buys

Nothing above is exotic computer science. MVCC, page-indexed seeks, hazard-free append protocols — these are textbook. The point is not novelty.

The point is that when the ref advertisement served a stale SHA, we read the 84-byte header on the live file and compared it against the enumerator's source, and we knew within a session that the records were written and the metadata was stale. When streaming got slow, we found the one-line downgrade rather than profiling our own loop for a week. When a transaction could not see its own insert, we changed what "visible" means inside a transaction — in the engine, that afternoon.

If Orion were PostgreSQL, every one of those would have been a mailing-list thread and a version upgrade. Some of them would have been "won't fix."

And the discipline that came out of it, written down after we wasted hours on a wrong theory:

• Measure before theorizing. Every claim in this article is either read from the source or measured on a copy of a real database. The claims that were guessed are the ones that turned out to be wrong.
• When a fix experiment fails, believe the failure. The malformed-record theory was tested and disproved early, then rationalized instead of dropped. That cost hours.
• Know the mechanism before proposing a change. The write-protocol rewrite died the moment somebody read the function it wanted to replace.
• Instrument on attempt one, not attempt four. Two printed values ended a six-day hunt. They were available on day one.

Orion DB is the storage engine inside the Elastic Platform. We built the compiler (EPL), the database, the protocol (TProtocol), the HTTP server (HSRV), and the GUI framework from scratch. Nine products ship on it; 20 AI agents build on it daily.

More: elastcode.com/investors