# Simulation Model

## 1. Event/state structure

The OCaml executable is a deterministic discrete-time simulation with an SST-compatible event vocabulary. Each tick emits event rows to `sst_events.csv` and component rows to `sst_components.csv`.

The modeled component set is:

- `WorkloadGen`
- `Scheduler`
- `MetadataRouter`
- `GpuExecutor`
- `RdmaNic`
- `RoceSwitch`
- `ZfsPool`
- `BackpressureController`
- `ScalingController`
- `StorageAdapter`

The modeled event kinds are:

- `WorkloadArrival`
- `AdmissionDecision`
- `MetadataLookup`
- `GpuKernelSubmit`
- `RdmaPacketBatch`
- `ZfsTxgWrite`
- `ZfsTxgStateTransition`
- `ScaleEvaluate`
- `ScaleApply`
- `BackpressureSignal`

The module names and event names are intentionally aligned with SST component/link/event concepts so the data model can later be ported into SST-Core components.

## 2. RoCE-v2 / RDMA packet model

The simulator models RDMA traffic as packet batches. A packet batch is a set of same-class RDMA packets emitted during a tick. The model records:

- traffic class
- opcode: `RDMA_READ`, `RDMA_WRITE`, or `SEND`
- queue pair number
- PSN start
- payload bytes
- packet count
- wire bytes
- MTU
- rail count
- Gbps per rail
- utilization
- serialization latency
- queueing latency
- base NIC/switch/propagation latency
- p99 latency
- ECN marks
- CNP feedback packets
- PFC pause time
- retransmits

The model uses:

```text
payload_per_packet = mtu_bytes - roce_header_bytes
packet_count       = ceil(payload_bytes / payload_per_packet)
wire_bytes         = payload_bytes + packet_count * (roce_header_bytes + preamble_ifg_bytes)
bandwidth_bytes_us = rdma_rails * roce_gbps_per_rail * 125
serialization_us   = wire_bytes / bandwidth_bytes_us
latency_us         = nic + switch_hops + propagation + serialization + queue + pfc_pause
```

ECN marks appear when utilization exceeds `roce_ecn_util_threshold`. PFC pause time appears when utilization exceeds `roce_pfc_util_threshold`. The model treats these as telemetry, not as correctness proofs.

## 3. OpenZFS TXG model

The simulator models transaction groups using the phases:

```text
OPEN -> QUIESCING -> SYNCING -> SYNCED/COMMIT
```

A TXG closes when:

- `zfs_dirty_data_max_mb` is reached; or
- `zfs_txg_timeout_us` expires and the TXG has dirty data.

Sync time is modeled as a function of:

- dirty MB
- storage node count
- write Gbps per node
- sync threads per node
- checksum/compression cost
- flush/FUA cost
- DPU/RDMA path capacity limit

The simulator allows a new open TXG while older TXGs are syncing, which is the behavior required to study writeback pressure and lag.

## 4. GPU kernel model

The simulator models kernel phases instead of real CUDA/HIP execution. Each kernel row records:

- kernel type
- batch size
- total GPUs and nodes
- FLOPs in teraFLOPs
- HBM bytes
- RDMA bytes
- occupancy
- compute time
- HBM time
- RDMA wait
- queue time
- total duration

Kernel duration is:

```text
duration_us = launch_us + max(compute_us, memory_us) + rdma_wait_us + queue_us
```

PCS scenarios include inference-oriented kernels:

- `prefill_attention_gemm`
- `decode_attention_kv`
- `batch_prefill_gemm`

USM scenarios additionally use:

- `checksum_compress_or_vector_index`

## 5. Scaling controller

The scaling controller evaluates rolling p99 metrics and applies actions after the configured cooldown.

Scale dimensions:

- **Network / RoCE-v2**
  - horizontal: add RDMA rail or endpoint
  - vertical: increase Gbps per rail and use qualified MTU profile
  - both: apply both, bounded by max limits

- **Compute / GPU**
  - horizontal: add GPU nodes per rack
  - vertical: increase GPUs per node and GPU TFLOP profile
  - both: apply both

- **Storage / OpenZFS TXG**
  - horizontal: add storage nodes
  - vertical: increase write Gbps per node and sync threads per node
  - both: apply both

- **USM metadata mesh**
  - horizontal: add metadata domains, API nodes, orchestrators
  - vertical: add shards per domain and workers
  - both: apply both

## 6. Output interpretation

Use `summary.json` for run-level assessment. Use the CSV files for detailed per-model analysis:

- RoCE-v2 congestion: `rdma_packets.csv`
- TXG lifecycle and writeback lag: `openzfs_txg.csv`
- GPU kernel latency and queue pressure: `gpu_kernels.csv`
- Component/event behavior: `sst_events.csv`
- Scaling explanation: `scaling_decisions.csv`
