Correctness and performance meet at the completion semantic: remote memory arrival is not remote persistence.
PROJECT COHERENT STORAGE / VERSION 10
Complex systems.
Clear boundaries.
A control-plane-guided, memory-first storage mesh for inference state - connecting an OpenAI-compatible client boundary to Coherence-CE, governed memory tiers, DPU-mediated NVMe-oF, and mirrored OpenZFS without leaking physical complexity into the service contract.
PLANNING TARGETS not production SLOs
Local memory hit5-20 usCoherence owner
Remote memory hit20-50 usMesh + RoCEv2
Mirrored write110-150 usLocal + remote persistence
CHOOSE YOUR LENS
Distinguished / Principal Storage EngineerCURRENT PERSPECTIVE
Trace every acknowledgement to a durable boundary.
Inspect partition ownership, remote mirror completion, tail amplification, failure locality, and the provider contracts that keep platform exceptions explicit.
p99/p99.9, ZIL timing, RDMA CQ depth, NUMA locality, ARC behavior, capability state, and recovery evidence.
A repeatable failure drill preserves acknowledged data and returns the pod to its declared protection state.
WHY THIS EXISTS
Make specialized infrastructure behave like a governed platform.
The architecture separates intent, capability, data movement, and durability so each concern can evolve without making the whole system opaque.
Inference repeats expensive context work
Partitioned HotCache keeps reusable prefix and KV state in memory, with a Redis-compatible access surface.
Local planning target: 5-20 usFast acknowledgement can hide weak durability
A write completes only after primary and backup memory plus local and remote persistent paths reach their required boundary.
Mirrored-write target: 110-150 usAll-to-all storage fabrics amplify failure
Strict ZFS mirror pairs bound remote namespace visibility while Coherence partitions the working set across nodes.
Scale by pod, fail by pairHardware diversity leaks into every workflow
A versioned API, reconciler, capability registry, and provider contracts normalize control without pretending every platform can do everything.
Explicit supported / degraded / unavailable statesTWO COORDINATED SYSTEMS
Trace the path, then inspect the boundary.
NEW / ADR-MODELED VIEWOpen the 12-layer spatial architecture explorer->
MODULAR DESIGN LAB
Change the system. Watch the constraint move.
Relative planning model derived from the attached architecture targets. Values expose design direction and bottlenecks; they are not benchmark claims.
CURRENT DESIGN
36 nodes across 4 pods
Host OS: RHEL/Rocky LinuxACTIVE CONSTRAINTNVMe media
Higher offload headroom; validate failure boundary. Host profile: RPM / systemd / SELinux / kABI. Model ceilings: fabric 2879 GB/s, media 332 GB/s, CPU/software 387 GB/s.
Modeled p99 path latencymicroseconds / relative estimate
Local memory
8 us
Remote memory
29 us
Mirrored write
110 us
Cold NVMe read
113 us
ECN mark estimate2.5%
Posturebalanced queues
Balanced thresholds trade some queue depth for sustained utilization under mixed traffic.
4 pods
8 pods
12 pods
16 pods
20 pods
24 pods
28 pods
32 pods
Planning model only. Coefficients are synthetic and intentionally transparent; validate against version-pinned hardware, firmware, topology, queue configuration, object sizes, and workload traces.
OPTIONAL SPATIAL PLAN / TOP-DOWN SYSTEMS CAMPUS
One complex. Seven coordinated structures.
The detailed campus drawing remains available as a second spatial lens. Open it when adjacency, repetition, and physical scale matter; use the Architecture page to peel through logical responsibility.
Open campus plan
OPTIONAL SPATIAL PLAN / TOP-DOWN SYSTEMS CAMPUS
One complex. Seven coordinated structures.
PCS / DATACENTER CAMPUS PLANSCHEME 04.09
L-03 / POD DISTRICT4 PODS / 36 NODES
Plan geometry communicates boundaries, adjacency, repetition, and scale. It is a conceptual systems diagram, not a rack elevation or construction drawing.
OPTIONAL OPERATIONAL DRILL
From alert to evidence, without guessing.
Open a bounded failure workflow when the question shifts from architecture awareness to on-call response.
Open incident workflow
OPTIONAL OPERATIONAL DRILL
From alert to evidence, without guessing.
Detect
Coherence membership changes; a ZFS remote mirror leg and provider endpoint become unavailable.
Contain
Keep requests inside the pod, preserve primary/backup ownership, and block changes that would reduce protection again.
Recover
Validate surviving durable state, replace or recover the node, reconnect namespaces, resilver, then reconcile desired state.
Prove
NODE-SAFE status, zpool state, acknowledged-write audit, provider observations, resilver completion, and recovery timing.
SYSTEMS-THEORY POSTURE
Complex does not have to mean complicated.
Local rules, global behavior
Partition ownership, pairwise mirroring, and provider contracts compose into cluster behavior without requiring one global storage lock.
Boundaries before features
Memory, persistence, transport, offload, and control each expose a completion rule and an observable health state.
Evidence before confidence
Capabilities are lab-validated, adapter-validated, design-only, or excluded - never silently promoted by an optimistic UI.