Engine Handbook — build serverless apps on PilserLabs

POST /api/srv/<board>/tables/todos/submit looks trivial. Underneath, a precise pipeline runs — this post walks it end to end, because understanding it explains the engine’s performance characteristics and its failure modes.

The pipeline

HTTP handler (tokio worker)
  │ collect body, CORS, admission permit          ← backpressure gate
  ▼
spawn_blocking ──► route_blocking (blocking pool)
  │ engine.Mutex.lock()                            ← THE serialization point
  ▼
record_insert:
  ├─ load_table            config lookup (cached)
  ├─ prepare_payload       computed fields + JSON-schema validation
  ├─ check_unique          unique-key duplicate guard
  ├─ allocate_seqs         ATOMIC counter (CAS on Helix counter node)
  ├─ stored_record_json    {board_id, table, seq, payload, created_at}
  ├─ db.insert             → write_node: upsert by _srv_key+table
  ├─ audit::append         audit trail row (if board.audit)
  ├─ automation::dispatch  Phase A: match recipes, run DB-side actions,
  │                        defer $call HTTP → Phase B executes AFTER unlock
  ├─ webhooks::fire_hooks  enqueue signed deliveries (worker POSTs later)
  └─ emit → broker         SSE subscribers get record.created instantly

The three details worth knowing

1. Seq allocation is atomic — by design

Older versions computed max(seq)+1 with a query. That is only correct if a single lock serializes all writes; the moment you allow parallel writers, two inserts can race and silently overwrite each other. The fix shipped as Database::allocate_seqs: the Helix adapter keeps one counter node per (tenant, table) and increments it with an optimistic compare-and-swap (set_property guarded by counter == expected, retry-on-race). Allocation is now correct even under full write parallelism — measured 32 parallel inserts at ~3–5ms/request.

2. Recipe HTTP does not hold the lock

A recipe’s $call action performs outbound HTTP. If that ran while holding the global engine Mutex, one slow webhook target would freeze every request in the daemon — this exact failure froze production during a bulk migration. The dispatch is now phased:

Phase A (locked): match recipes, run DB-side actions, collect resolved HTTP plans
Phase B (unlocked): execute the network calls
Phase C (locked): write results back

Same pattern for cron jobs. The lesson generalizes: never hold a coarse lock across network I/O.

3. Reads are cached and coalesced

CachedDatabase fronts the backend with a 30s result cache plus singleflight: when N clients request the same expired query simultaneously, one scan runs and the rest wait for its verdict. Reproduced before the fix: 10 identical counts = 10 scans, 8.6s tail. After: values identical, scans coalesced. Admission control bounds how many engine operations run concurrently at all (SRV_MAX_DB_CONCURRENCY, default 8), and shed excess load with 503 Retry-After.

Why a single Mutex at all?

Honesty section. The engine still serializes operations through one Mutex<ServerlessEngine>. It is the design’s central tradeoff: the database handle is not thread-safe for mutation, and wrapping everything in one lock made the first version correct immediately. The refactor path (documented in the repo’s docs/engine-mutex-refactor-plan.md) removes lock-held-I/O stage by stage while keeping semantics: assets already bypass it entirely, recipe/cron HTTP phases around it, and seq allocation no longer depends on it. What remains behind the lock is short, DB-bound critical sections.

Failure modes this design accepts

A slow Helix scan delays other DB-bound requests (bounded by admission control; static files unaffected — they never touch the lock)
Cache misses after 30s TTL cost one scan per distinct query (coalesced when concurrent)
Webhook delivery is eventually-consistent by queue, never inline

Where to look in the source

Write chain: crates/engine/src/crud.rs (record_insert)
Phased dispatch: crates/engine/src/automation.rs
CAS allocator: crates/server/src/db/helix.rs (counter_allocate)
Cache/singleflight: crates/engine/src/storage/cache.rs

Next: what the storage layer looks like underneath — HelixDB nodes, SlateDB LSMs, and why MinIO holds the truth.

The write path: what happens when you submit a record