Model-read storage layout investigation#

Status: Postmortem Investigation date: 2026-09-22 Scope: storage layouts observed in schema-v3 ComfyUI benchmarks with verified cache eviction Superseded: The purchase recommendation in this document was superseded on 2026-09-23 by the dual-workload single-versus-RAID experiment, which measured a single Gen5 NVMe above 10 GiB/s and found no repeated application-level RAID advantage.

Outcome#

Every host that sustained at least approximately 10 GiB/s while reading the real model set used two NVMe devices in parallel. No measured single-drive host crossed that threshold through the complete filesystem/container/model-loading path.

The closest proven layout to the planned Ryzen 9 9950X machine was:

CPU-connected M.2 slot ── 4 TB WD Black SN850X, PCIe 4.0 x4 ─┐
                                                               ├─ Linux mdadm RAID 0
CPU-connected M.2 slot ── 4 TB WD Black SN850X, PCIe 4.0 x4 ─┘  512 KiB chunks

Chipset-connected M.2 slot ── separate 1 TB OS drive

This layout appeared on two MSI MAG X870 Tomahawk systems with Ryzen 9950X-class CPUs. It sustained 11.11-11.54 GiB/s on the model-file benchmark. The result supports using two 2 TB Samsung 9100 Pros as a striped, disposable model cache and a separate Gen4 NVMe drive for the operating system and application data.

What was measured#

The primary metric is not the disk speed advertised by Vast or a synthetic fio headline. It is model_file_read_benchmark from the schema-v3 result files:

• The persisted input and model set is 42,471,968,783 bytes (42.47 GB decimal, 39.56 GiB).
• Every file is flushed and evicted with fsync plus POSIX_FADV_DONTNEED.
• The files are then read normally in 8 MiB buffered chunks.
• /proc counters verify that the reads reached physical storage rather than Linux page cache.
• Three samples are recorded independently of the end-to-end cold/warm workflow pairs.

The fixed end-to-end protocol separately runs five cycles of:

evict resolved files -> cold run in a fresh ComfyUI process
                     -> warm run with a different seed in the same process
                     -> stop ComfyUI

All cold samples in the results below passed physical-read verification. Every warm sample recorded zero physical disk reads.

Layouts that crossed the threshold#

fio values below use decimal GB/s, as reported from bytes per second. Model reads use GiB/s.

Two Gen5 T710 drives#

Machine 55868 used a Gigabyte TRX50 AERO D and Ryzen Threadripper 9960X. Both 4 TB Crucial T710 drives negotiated PCIe 5.0 x4 on separate root ports. Linux exposed them as /dev/nvme0n1 and /dev/nvme2n1, combined into /dev/md0:

• RAID level: 0
• Metadata: 1.2
• Chunk size: 512 KiB
• Array size: approximately 8 TB
• Read-ahead: 2 MiB

This was the fastest confirmed data path. Its 15.59 GiB/s model read remained substantially below the 29.71 GB/s high-queue-depth fio result, demonstrating why the model-file measurement is the acceptance metric.

Two Gen4 SN850X drives on AM5#

Machines 112410 and 37297 both used an MSI MAG X870 Tomahawk with a Ryzen 9950X-class CPU. Their model arrays consisted of two 4 TB SN850X drives, each negotiating PCIe 4.0 x4, combined with mdadm RAID 0 and 512 KiB chunks. A third 1 TB SN850X was connected through the chipset and carried the host operating system.

The PCIe topology placed the GPU and both model-array members on independent CPU root ports. The 1 TB OS drive traversed the chipset. This is the most transferable result for the planned AM5 build because it matches its CPU and motherboard topology rather than relying on Threadripper's additional lanes.

Machine 112410 was measured twice. Its model-file means were 11.48 GiB/s and 11.54 GiB/s, which provides same-host repeatability evidence for the layout.

Two Gen5 Samsung 9100 Pros#

Machine 149412 used an ASUS Pro WS WRX90E-SAGE SE and two 2 TB Samsung 9100 Pros, each negotiating PCIe 5.0 x4 on a separate root port.

The container exposed a small 210 GB mdadm RAID 1 system partition. That visible mirror was not the high-speed data path. The data-volume counters showed a separate virtual device with its reads and writes divided almost exactly in half between the two physical SSDs. Aggregate direct reads reached 29.21 GB/s and writes reached 24.50 GB/s. Those write counters and rates are inconsistent with either a single SSD or a mirrored data volume; the host was using an additional striped layer that Vast did not expose fully to the container.

This result is evidence for a two-drive 9100 Pro data path, not evidence that one 9100 Pro sustains the same model-read rate.

Evidence that both RAID members were active#

For each confirmed striped host, the storage-probe cgroup counters showed approximately:

aggregate array or virtual device: 213.4 GB read
first physical NVMe:                106.7 GB read
second physical NVMe:               106.7 GB read

The totals correspond to the repeated 23.62 GB fio read probes. The near-perfect 50/50 member split proves that the probe path was served by both drives rather than a RAM disk or an unrelated device. The probe file and model files both live below /root/ComfyUI on the same container overlay, so they share the same backing data path.

For the explicit RAID hosts, /proc/mdstat and captured mdadm metadata independently identify the two members and RAID level. For the dual-9100 host, the balanced physical counters and aggregate virtual-device counters establish effective striping even though the host-managed layer itself was hidden.

Counterexamples and limits#

One 4 TB Samsung 9100 Pro#

Machine 150656 exposed one 4 TB Samsung 9100 Pro on a PCIe 5.0 x4 link with no RAID:

Samsung rates the 2 TB 9100 Pro for up to 14,800 MB/s sequential reads, but that device specification is not a guarantee for a containerized, buffered, multi-file workload. The observed single-drive host nearly reached the target with direct fio yet missed it materially on the real model path. See Samsung's 9100 Pro specification.

This does not prove that a properly configured local 9100 Pro cannot sustain 10 GiB/s. It means the current evidence does not demonstrate it.

Four SN850X drives are not automatically faster#

Machine 33260 exposed four 2 TB SN850X drives in an explicit RAID 0 array and advertised more than 26 GB/s. It nevertheless produced:

• 13.19 GB/s direct QD32 fio
• 4.07 GB/s buffered fio
• 3.49 GiB/s model-file reads
• A 32.36-second cold penalty

The capture did not include cgroup I/O mapping for this host, so it cannot prove that the ComfyUI overlay actually used the visible four-drive array. This is still a useful negative result: drive count and an advertised RAID topology do not establish application throughput. The exact mounted path, controller topology, container layer, throttling, thermal state, and real model read must all be verified.

Recommended layout for the planned machine#

The planned MSI MAG X870E Tomahawk Max WiFi provides two CPU-connected Gen5 x4 M.2 slots and two chipset-connected Gen4 x4 slots. The RTX 5090 retains its CPU-connected Gen5 x16 link. M2_2 shares its four lanes with rear USB4: with both active they default to x2 each; selecting full x4 for M2_2 in BIOS disables the rear 40 Gbps USB-C ports. See the MSI storage and lane-sharing specification.

Ryzen 9 9950X
├── PCIe 5.0 x16 ── RTX 5090
│
├── M2_1, CPU Gen5 x4 ── Samsung 9100 Pro 2 TB ─┐
│                                                ├─ mdadm RAID 0
├── M2_2, CPU Gen5 x4 ── Samsung 9100 Pro 2 TB ─┘  512 KiB chunks
│                                                   /srv/model-cache
│
├── M2_3, chipset Gen4 x4 ── OS, containers, applications, downloads and outputs
└── M2_4, chipset Gen4 x4 ── optional future scratch or output drive

Configuration decisions:

• Configure M2_2 for full Gen5 x4 and accept losing the rear USB4 ports.
• Use Linux mdadm RAID 0. The successful explicit arrays used software RAID, not a hardware controller.
• Use 512 KiB chunks and begin with 2 MiB array read-ahead, matching the captured fast arrays.
• Use XFS or ext4 with 1 MiB partition alignment and noatime.
• Use periodic fstrim; avoid continuous discard on the hot path.
• Put the OS, Docker/container layers, application code, downloads and outputs on the separate Gen4 drive so incidental writes do not contend with cold model reads.
• Cool both Gen5 drives with the motherboard heat spreaders and direct case airflow. Validate sustained temperature rather than a short peak benchmark.
• Treat the RAID 0 volume strictly as a disposable hot cache. Authoritative models remain on local HDD or NAS and can repopulate the array after a drive failure.

At the quoted equal total price for 2x 2 TB versus 1x 4 TB, the two-drive layout is the better-supported choice when the objective is reliably exceeding 10 GiB/s. It keeps the same 4 TB usable capacity while trading redundancy for throughput on data that is already recoverable elsewhere.

Acceptance test for the physical build#

The build should not be accepted from product specifications or one fio number. Validate it in this order:

1. PCIe topology
   • Confirm both 9100 Pros negotiate 32.0 GT/s at x4.
   • Confirm the RTX 5090 remains at its intended x16 width.
   • Confirm the model drives are on the two CPU-connected M.2 paths, not both behind the chipset.
2. Array configuration
   • Confirm two healthy RAID 0 members, 512 KiB chunks and the expected 4 TB usable capacity.
   • Confirm read counters rise approximately equally on both physical members during a large read.
3. Synthetic storage floor
   • Direct 1 MiB QD32 reads should comfortably exceed 20 GB/s.
   • Buffered single-stream reads after cache eviction should exceed approximately 12 GB/s.
4. Application storage target
   • Run the benchmark's normal buffered model-file reader after fsync and POSIX_FADV_DONTNEED.
   • Require at least 10 GiB/s across repeated samples, not only the best sample.
5. End-to-end target
   • Run the fixed five-pair ComfyUI protocol.
   • Require every cold sample to pass physical-read verification and every warm sample to record zero physical reads.
   • Target a cold-to-warm penalty of approximately 5-8 seconds for the current workflow.
6. Thermal stability
   • Read at least 200 GB continuously while recording both NVMe temperatures and negotiated links.
   • Reject a configuration that crosses the throughput target briefly and then throttles below it.

Interpretation boundaries#

• The results measure one approximately 42.47 GB model set and its current ComfyUI loading behavior.
• Results from different GPU hosts remain confounded by CPU, motherboard, kernel, filesystem, container storage and background load. Disk layout conclusions rely on the dedicated storage probes and captured topology, not cold-runtime comparisons across GPU classes.
• A result above 10 GiB/s demonstrates the complete observed path, not a universal guarantee for every filesystem or workload.
• RAID 0 improves aggregate throughput but doubles the number of devices whose failure destroys the array. That trade is acceptable only because the proposed volume is a cache with a separate source of truth.
• The exact filesystem under the Vast Docker overlay was not visible from inside every container. The block topology, RAID metadata, PCIe links, cgroup counters and application-level measurements are the evidence available.