Start with the contract
Clients ask for logical inference or object work. They never select a pool, vdev, DPU, queue pair, rail, or NVMe namespace.
See the contract boundary ->REQUEST TO EVIDENCE01Ingress02Translation03Coherence04Transport05Persistence
PROJECT COHERENT STORAGE / APPLICATION VERSION 10
From client request to durable evidence.
A control-plane-guided, memory-first storage mesh for inference state. The design connects an OpenAI-compatible client boundary to Coherence-CE, governed memory tiers, DPU-mediated NVMe-oF, and mirrored OpenZFS while keeping physical selectors below the logical service contract.
PLANNING TARGETS not production SLOs
Local memory hit5-20 usCoherence owner
Remote memory hit20-50 usMesh + RoCEv2
Mirrored write110-150 usLocal + remote persistence
ORDER OF OPERATIONSOne request. Five visible responsibility changes.
The supplied color order follows the request from authorized ingress to durable evidence. Color encodes sequence; line style and geometry encode subcomponents.
- 01
#A0005FAuthorizeIngress->TLS, API token, and HAProxy establish the service boundary.
- 02
#4B0F3BNormalizeTranslation->OpenAI/OpenAPI requests become logical Coherence operations.
- 03
#1B1B24PlaceCoherence->Ownership, policy, and memory locality determine where work lives.
- 04
#C97AA8MoveTransport->RoCEv2, RDMA, DPUs, and NVMe-oF carry registered data.
- 05
#F6F0F4ProvePersistenceMirrored OpenZFS commits durable state and returns evidence.
BASELINE READING MODEL / FIVE QUESTIONS
What must be understood before the numbers matter?
Use these five questions as the minimum mental model. Each opens the next level of detail without requiring the entire stack at once.
Separate the two planes
The data plane moves state. The control plane validates intent, capability, health, rollback, and evidence.
Peel the architecture ->Follow completion
A memory arrival, an RDMA completion, and a durable mirrored write are different events with different failure semantics.
Inspect equations ->Scale by bounded units
Pods repeat known placement, fabric, DPU, and mirror patterns. Capacity expands without turning every node into every other node's failure domain.
Configure a design ->Require evidence
Fresh observations, failure drills, SLO boundaries, energy, cost, and recovery state turn a complex design into an operable platform.
Open impact evidence ->CHOOSE YOUR LENS
Distinguished / Principal Storage EngineerCURRENT PERSPECTIVE
Trace every acknowledgement to a durable boundary.
Inspect partition ownership, remote mirror completion, tail amplification, failure locality, and the provider contracts that keep platform exceptions explicit.
Correctness and performance meet at the completion semantic: remote memory arrival is not remote persistence.
p99/p99.9, ZIL timing, RDMA CQ depth, NUMA locality, ARC behavior, capability state, and recovery evidence.
A repeatable failure drill preserves acknowledged data and returns the pod to its declared protection state.
WHY THIS EXISTS
Make specialized infrastructure behave like a governed platform.
The architecture separates intent, capability, data movement, and durability so each concern can evolve without making the whole system opaque.
Inference repeats expensive context work
Partitioned HotCache keeps reusable prefix and KV state in memory, with a Redis-compatible access surface.
Local planning target: 5-20 usFast acknowledgement can hide weak durability
A write completes only after primary and backup memory plus local and remote persistent paths reach their required boundary.
Mirrored-write target: 110-150 usAll-to-all storage fabrics amplify failure
Strict ZFS mirror pairs bound remote namespace visibility while Coherence partitions the working set across nodes.
Scale by pod, fail by pairHardware diversity leaks into every workflow
A versioned API, reconciler, capability registry, and provider contracts normalize control without pretending every platform can do everything.
Explicit supported / degraded / unavailable statesTWO COORDINATED SYSTEMS
Trace the path, then inspect the boundary.
NEW / ADR-MODELED VIEWOpen the 12-layer spatial architecture explorer->
MODULAR DESIGN LAB
Change the system. Watch the constraint move.
Relative planning model derived from the attached architecture targets. Values expose design direction and bottlenecks; they are not benchmark claims.
CURRENT DESIGN
36 nodes across 4 pods
Host OS: RHEL/Rocky LinuxACTIVE CONSTRAINTNVMe media
Higher offload headroom; validate failure boundary. Host profile: RPM / systemd / SELinux / kABI. Model ceilings: fabric 2879 GB/s, media 332 GB/s, CPU/software 387 GB/s.
Modeled p99 path latencymicroseconds / relative estimate
Local memory
8 us
Remote memory
29 us
Mirrored write
110 us
Cold NVMe read
113 us
ECN mark estimate2.5%
Posturebalanced queues
Balanced thresholds trade some queue depth for sustained utilization under mixed traffic.
4 pods
8 pods
12 pods
16 pods
20 pods
24 pods
28 pods
32 pods
Planning model only. Coefficients are synthetic and intentionally transparent; validate against version-pinned hardware, firmware, topology, queue configuration, object sizes, and workload traces.
OPTIONAL SPATIAL PLAN / TOP-DOWN SYSTEMS CAMPUS
One complex. Seven coordinated structures.
The detailed campus drawing remains available as a second spatial lens. Open it when adjacency, repetition, and physical scale matter; use the Architecture page to peel through logical responsibility.
Open campus plan
OPTIONAL SPATIAL PLAN / TOP-DOWN SYSTEMS CAMPUS
One complex. Seven coordinated structures.
PCS / DATACENTER CAMPUS PLANSCHEME 04.09
L-03 / POD DISTRICT4 PODS / 36 NODES
Plan geometry communicates boundaries, adjacency, repetition, and scale. It is a conceptual systems diagram, not a rack elevation or construction drawing.
OPTIONAL OPERATIONAL DRILL
From alert to evidence, without guessing.
Open a bounded failure workflow when the question shifts from architecture awareness to on-call response.
Open incident workflow
OPTIONAL OPERATIONAL DRILL
From alert to evidence, without guessing.
Detect
Coherence membership changes; a ZFS remote mirror leg and provider endpoint become unavailable.
Contain
Keep requests inside the pod, preserve primary/backup ownership, and block changes that would reduce protection again.
Recover
Validate surviving durable state, replace or recover the node, reconnect namespaces, resilver, then reconcile desired state.
Prove
NODE-SAFE status, zpool state, acknowledged-write audit, provider observations, resilver completion, and recovery timing.
SYSTEMS-THEORY POSTURE
Complex does not have to mean complicated.
Local rules, global behavior
Partition ownership, pairwise mirroring, and provider contracts compose into cluster behavior without requiring one global storage lock.
Boundaries before features
Memory, persistence, transport, offload, and control each expose a completion rule and an observable health state.
Evidence before confidence
Capabilities are lab-validated, adapter-validated, design-only, or excluded - never silently promoted by an optimistic UI.