Real-Time Dealer Gamma Engine

The vendor allows a single WebSocket connection, so a relay owns it and routes frames by root symbol without decoding them. Each consumer has a bounded drop-oldest queue, so a slow engine can never back-pressure the others.

Two engine shards (indices; single names) consume from the relay. A front service merges their snapshots into one API used by the site, a snapshot recorder, the watchdog and the alerter. A separate tick recorder writes hourly Hive-partitioned Parquet that is archived nightly.

Quotes update a quote cache; trades are classified (aggressor flags, position in the spread, quote imbalance, size, clustering). Multi-leg prints are buffered and paired into spreads before they reach the books.

At snapshot time profiles are built against the current SVI surface. Session flow is rolled into the baseline at end of day so the book carries across sessions.

Hypotheses are pre-registered with dev, hold-out and burn-in partitions enforced in code, and results are scored against matched controls (including round-number levels).

• Classification: a LightGBM dealer-direction model trained against measured Cboe Open-Close dealer volume, with 5-fold-by-day cross-validation so no day leaks across folds. Sign accuracy beats the Lee-Ready baseline on SPX (0.65 vs 0.54), SPY (0.65 vs 0.63) and QQQ (0.63 vs 0.61), AUC ≈ 0.70.
• Known limits stated up front: with only top-of-book NBBO, the documented accuracy ceiling is about 0.74. Train/serve parity is enforced by tests that check the streaming accumulator against the batch pipeline.
• Volatility surface: Gatheral SVI calibrated with the Zeliade quasi-explicit method plus a vega-weighted robust polish, a Durrleman butterfly-arbitrage check, and forwards inferred from put-call parity.
• Performance work driven by profiling: SVI fits moved off the GIL into a process pool (with tests proving identical results to the threaded path), the gamma build vectorised (31% faster), and HTTP’s share of the event loop cut from 13% to 4.5%.
• Reliability: a watchdog with two-stage escalation and a circuit breaker, per-symbol starvation detection, and an alerter that deliberately shares no code with what it monitors.
• A shadow engine runs alongside production and is compared snapshot for snapshot (SPXW net-GEX correlation median 1.000; total-GEX sign agreement 100%).

• The research programme reports null results as well as positive ones. Example: across 223 node touches over 99 days, only distance to the neighbouring node predicted overshoot (ρ 0.36–0.44); node size, sign and velocity were null.
• Gamma reconstruction is trusted only within ±0.5% of spot — documented, not hidden.