How We Replaced a Global Semaphore With Hazard Pointers

A 335-crash-per-session bug, a non-reentrant lock from 2010, and why "just use a mutex" is the wrong answer in a JIT compiler runtime

A 335-crash-per-session bug, a non-reentrant lock from 2010, and why "just use a mutex" is the wrong answer in a JIT compiler runtime

We run our own compiler. EPL compiles to native x64 machine code and executes it. The runtime manages objects, garbage collection, memory allocation — the same things a JVM or V8 does, but for our language.

For years, a global semaphore protected the runtime's shared data structures. It worked. Then we added concurrent AI workers — 20 agents building and running code simultaneously — and it stopped working.

This is the story of the crash, the investigation, the design, and the fix.

The crash: 335 cascading failures in one session

The symptom was a printf storm:

Fatal Error: SetSemaphore is called twice (Counter=1)
Fatal Error: SetSemaphore is called twice (Counter=1)
Fatal Error: SetSemaphore is called twice (Counter=1)

Hundreds of times in a single burst. Then the process exits. The watchdog restarts it in 3 seconds. Under concurrent load, it crashes again within 60 seconds.

In one 4-hour window: 335 cascade events across 11 crash cycles. Plus 7 SIGSEGV signals in the memory allocator. The system was functional — the watchdog masked the crashes — but every crash killed an in-flight AI response.

Idle? Zero crashes for hours. Concurrent AI workers? Crash within seconds.

What the semaphore actually protected

The global semaphore (SEM_STRUC in BaseLib/Parse/Memory.cpp) was a non-reentrant lock. The common belief was "it protects the stack allocator, and the per-thread TC made the stack per-thread, so why is it still there?"

Half right, and importantly wrong.

The stack allocator WAS per-thread now. The semaphore was no longer needed for that. But it still protected three other things:

1. The object body table. This is the core data structure — a growable array of slots holding every live EPL object. The garbage collector frees slots. Other threads allocate slots. The semaphore serialized lookups against concurrent frees.

2. The output-variable chain. A process-global linked list mutated by every data-segment allocation. Historical — every entry was owned by the allocating thread, but the head pointer was global. No genuine cross-thread sharing, just an artifact of the original single-threaded design.

3. GC destructor invocation. The garbage collector held the semaphore while running object destructors. If the destructor executed EPL bytecode that needed the semaphore — and it almost always did — the inner acquire found Counter at 1 instead of 0. Non-reentrant lock + reentrant code = fatal.

This was the actual crash mechanism: the GC reaper holds the lock → invokes a destructor → the destructor runs EPL code → that code tries to acquire the same lock → Counter goes from 1 to 2 → the non-reentrant check fires → process exits.

Why "just use a recursive mutex" is wrong

The obvious fix: replace the non-reentrant semaphore with pthread_mutex_t using PTHREAD_MUTEX_RECURSIVE. Same thread can acquire it multiple times. "Called twice" can't fire. One-day patch.

We rejected it.

A recursive mutex preserves the global lock on the read path. Every GetObject(idx) — which the runtime calls thousands of times per second — takes and releases the lock. Under concurrent AI worker load, that's thousands of threads contending on a single cache line. The crash goes away but gets replaced by latency: lock contention serializes all object lookups across all threads.

A reader-writer lock is better — multiple readers, exclusive writer. But it still bounces a shared counter on every read. Under high concurrency, that counter's cache line ping-pongs between cores.

We needed a design where the read path has zero shared cache-line writes.

The design: hazard pointers for the body table

Hazard pointers are a lock-free memory reclamation scheme. The idea: each thread publishes the pointer it's about to dereference. The writer checks all published pointers before freeing anything. If a pointer is published by any thread, it's not freed yet.

Each thread gets a small TLS array of hazard slots:

#define HP_PER_THREAD 4
__thread Table *Hazard[HP_PER_THREAD];
__thread int    HazardDepth;

The read path:

Table* PinCurrentTable() {
    int slot = HazardDepth++;
retry:
    Table *T = atomic_load_acquire(&GlobalTablePtr);
    Hazard[slot] = T;                          // publish
    Table *T2 = atomic_load_acquire(&GlobalTablePtr);
    if (T2 != T) goto retry;                   // grew between load and publish
    return T;
}

The recheck is the entire correctness argument. Between our first load and our publish, a writer might have swapped the global pointer, retired the old table, and freed it. The recheck detects that case and retries.

The write (grow) path:

void GrowTable(size_t new_cap) {
    Table *Old = atomic_load(&GlobalTablePtr);
    Table *New = AllocTable(new_cap);
    memcpy(New->Slots, Old->Slots, Old->Cap * sizeof(Slot));
    atomic_store_release(&GlobalTablePtr, New);
    RetireQueue.push(Old);
    ScanAndFreeRetired();
}

ScanAndFreeRetired() snapshots all hazard slots across all threads. Any retired table that's still published by a reader stays on the queue. Everything else gets freed. Cost: O(threads × HP_PER_THREAD) — cheap, and grows are geometric (O(log N) over process lifetime).

The key property: the read path does zero shared writes. It writes only to thread-local memory (the hazard slot). No cache-line bouncing. No lock. No contention. Exactly what we needed.

Why not RCU? Why not epoch-based reclamation?

We considered both.

Epoch-based reclamation (QSBR) is cheaper for readers — just bump a per-thread epoch counter. But it requires every thread to periodically pass through a "quiet point" where it holds no pinned pointers. Our EPL workers can sit in long-running bytecode execution for minutes. No guaranteed quiet point. Hazard pointers don't have this dependency.

RCU has the same quiet-point requirement (grace periods). It's ideal for read-mostly kernel data structures. Not ideal for a JIT runtime where threads hold references for unpredictable durations.

Leak-on-grow (never free old tables, geometric expansion bounds total leak at ~2× current size) — rejected by me. Bounded leaks are still leaks, and this is infrastructure that runs for months.

The body allocator: per-thread caches + lock-free global

The body table holds slots. The slots hold bodies (the actual EPL objects). Bodies are variable-sized — the size depends on the class's data members. The existing lifetime primitive (__sync_sub_and_fetch(&Obj->RefNum, 1)) was already atomic. We needed a lock-free storage primitive.

Design: per-size-class buckets, per-thread cache, ABA-tagged Treiber stack as the global slow path.

1. Round up body size to one of ~12 size classes (16, 32, 48, 64, ... 1024, then page-size).

2. Each thread holds a TLS cache per size class (~32 nodes). Alloc/free hits the cache — no atomics, no shared cache lines. The hot path is a few instructions.

3. Cache empty → take a batch (16-32) from the global free-list via one CAS.

4. Cache full → return a batch via one CAS.

5. The global free-list is a Treiber stack with tagged pointer (top 16-bit version counter) to defeat ABA. Single lock cmpxchg per batch.

The body allocator becomes the only place the semaphore touches at all, and even there the per-thread cache means the global head is touched 1/Nth as often.

The other two fixes

Output-variable chain → per-thread. The chain head was global for historical reasons. Every entry was owned by the allocating thread. Fix: move the head into the per-thread ThreadContext. ~20 lines changed. Zero shared state, zero lock needed.

GC destructor: release-before-call. The reaper releases the semaphore before invoking the destructor, re-acquires after. The destructor runs lock-free. Any internal allocation that needs the lock takes it cleanly — no nesting, no over-increment. Safe because by the time we're running the destructor, refcount hit zero on this thread and nobody else can have pinned the body.

Migration sequence

Each step validated on dev for 24+ hours before the next. None deployed during production.

The result

After the full migration: zero cascade crashes under the same concurrent AI worker load that previously produced 335 events per session.

The read path — GetObject(idx), called thousands of times per second across 20+ threads — now involves zero locks, zero shared cache-line writes, and zero contention. A thread pins the current table, reads the slot, AddRefs the body, unpins. The GC reaper retires tables safely, frees bodies when no hazard pointer references them.

The semaphore that protected three things now protects nothing. The three things protect themselves: the table with hazard pointers, the bodies with per-thread caches, the chain with per-thread storage.

What this tells you about the platform

This is not a story about hazard pointers. Every systems textbook covers hazard pointers.

This is a story about what happens when you own the runtime. When the crash is in your memory allocator, inside your JIT compiler, guarding your object table — you can't file a bug report with someone else. You can't wait for the next JVM release. You can't "upgrade the dependency."

You read the six layers of C++ between the semaphore and the page allocator. You instrument. You find that the non-reentrant lock is held by the GC reaper while a destructor calls back into bytecode that acquires the same lock. You design a replacement that eliminates the lock from the read path entirely. You land it in five incremental steps, each validated for 24 hours.

The fix was not one line. The investigation was not one afternoon. But the result — zero crashes where there were 335, zero contention where there was serialization — is what "owning the stack" actually means.

EPL is the compiler at the core of the Elastic Platform. We built the compiler, database (Orion DB), protocol (TProtocol), HTTP server (HSRV), and GUI framework from scratch. 9 products ship on it. 20 AI agents build on it daily.

More: elastcode.com/investors