RTX 5090 single-NVMe versus RAID 0 dual-workload experiment#

Status: Complete for the RAID purchase decision; exact 2 TB and 4 TB Samsung 9100 Pro MiniMax replications collected Started: 2026-09-23 Decision: One 4 TB Gen5 NVMe versus two 2 TB Gen5 NVMe devices in RAID 0 for the planned RTX 5090 service node

Question#

Does two-drive RAID 0 materially reduce real ComfyUI cold-start latency compared with a sufficiently fast single Gen5 NVMe, or can one drive already feed model weights quickly enough that GPU loading and model initialization become the limiting path?

The evidence available at the start was not conclusive. MiniMax H3 runs associated two-drive RAID 0 with much lower cold penalties, but those results came from different hosts and different container storage implementations. The newer Krea 2 result demonstrated 10.80 GiB/s model-file reads and a 5.22-second cold penalty from one WD Black SN8100. A controlled, replicated comparison must retain the full application benchmark instead of treating fio or a sequential file reader as a proxy for ComfyUI.

Hypotheses#

• Single-drive-sufficient hypothesis: a correctly configured single Gen5 NVMe sustains approximately 10 GiB/s through the actual model path, and its end-to-end cold penalty is within one second or 10% of a comparable RAID 0 host on both workloads.
• RAID-beneficial hypothesis: two-drive RAID 0 repeatedly saves more than two seconds of end-to-end cold latency on both workloads after accounting for warm GPU performance and host differences.
• Synthetic throughput alone cannot decide between these hypotheses. The primary measurement is paired cold-versus-warm ComfyUI runtime with verified physical reads.

Workloads#

Both workloads run during one provisioned-host session. ComfyUI, Python, PyTorch, CUDA dependencies and benchmark tooling are installed once. The union of both model sets is downloaded before either benchmark. Each workload nevertheless produces its own complete schema-v3 JSON result and ComfyUI log evidence.

At least 160 GB of container storage will be allocated so both sets remain present while the sustained storage probes create their temporary test file.

Full per-workload benchmark#

Neither workload is reduced to a storage microbenchmark. Each independent result contains:

1. Physical GPU, CPU, RAM, motherboard, negotiated PCIe links, PCIe ancestry and storage topology.

2. CPU memory-copy and pinned CPU-to-GPU/GPU-to-CPU transfer measurements.

3. Sustained direct QD1, direct QD32 and cache-evicted buffered storage probes.

4. Five repetitions of:

   fsync and evict every resolved workload file
       -> launch a fresh ComfyUI process
       -> cold run, requiring physical reads >= 80% of resolved bytes
       -> warm run with a different seed in the same process
       -> stop ComfyUI
   

5. Three normal buffered reads of the workload's real model/input set after page-cache eviction, including /proc physical-read verification.

6. Per-sample wall time, execution time, seeds, process resource traces and ComfyUI model-loading evidence.

The default resident ComfyUI cache is used. --cache-none is not used.

Experimental units and host selection#

The physical host, rather than each of its five workflow pairs, is the independent experimental unit. Five pairs quantify within-host noise; they do not substitute for replication across hosts.

The minimum primary matrix is:

Primary hosts must expose one allocated physical GPU (gpu_frac=1) negotiating PCIe 5.0 x16, matching the intended build. Shared multi-GPU systems, GPUs reduced to x8 and four-drive arrays are excluded from the primary comparison. Results from those machines may be retained as supporting evidence but cannot alone decide the purchase.

Known candidates at the start of the experiment:

• Single 4 TB Samsung 9100 Pro: machines 147169 and 150656.
• Two 4 TB WD Black SN850X RAID 0: machine 112410.
• A second comparable dedicated RAID 0 RTX 5090 host must be acquired.
• Machine 137184's single SN8100 result is supporting evidence because the physical host has two GPUs.

All primary runs must use the same exact ComfyUI tag, Torch/CUDA build, cache policy and benchmark code. GPU power limit, PCIe width, CPU allocation, RAM, container runtime and filesystem path are captured rather than inferred from Vast's advertised disk-speed field.

PCIe topology controls#

Drive model and count do not fully specify a storage layout. Two nominally identical RAID members can each negotiate Gen5 x4 yet still share a chipset uplink, while two CPU-direct M.2 slots can each have an independent x4 path. Those configurations must not be treated as equivalent. The GPU link is another independent variable: a 5090 negotiating x8 is supporting evidence for a planned x16 machine, even when its warm runtime happens to be close.

Every new result therefore records, for the GPU and each NVMe endpoint:

• current and maximum PCIe generation/speed and link width;
• the resolved sysfs device path, PCI address and ordered PCIe ancestry;
• the number and identity of upstream bridges;
• shared upstream bridges between every pair of NVMe devices and between each NVMe and the GPU;
• the complete lspci tree and path-oriented lspci -DPPnn listing.

The ancestry is evidence, not an automatic CPU-direct/chipset label. Final classification should combine it with the motherboard or platform block diagram. A primary single-versus-RAID comparison must match GPU link width and, as far as possible, CPU socket/NUMA placement. A RAID result whose members share one chipset uplink answers a different question from a RAID result whose members use independent CPU root ports.

PCIe 5.0 x8 and PCIe 4.0 x16 have similar nominal one-direction link bandwidth, but they are not assumed to be application-equivalent. That claim needs a later controlled validation, preferably on the same 5090 host with x16 versus x8 selected through slot placement or BIOS bifurcation. The validation must include pinned host-to-device transfers and both full ComfyUI workloads, not only a PCIe microbenchmark. Until then, every x8 GPU result is supporting-only regardless of its observed warm runtime.

For the planned MSI MAG X870E Tomahawk Max build, the physical acceptance test must be performed with every intended device installed, and the 5090 must negotiate Gen5 x16 under load. In any two-drive comparison, the model drives in M2_1 and M2_2 must each negotiate Gen5 x4 on their CPU paths, the system drive must remain on its chipset path, and M2_2 must be configured for full x4 even though that disables the shared rear USB4 function. The recommended single-drive configuration uses M2_1 and leaves rear USB4 available; only a later justified RAID trial should consume M2_2 and disable USB4. If adding a second model drive reduces the GPU to x8 or puts either RAID member behind a shared chipset uplink, the machine is not the topology this experiment tested.

Session and order controls#

The suite runner must:

• provision and install once;
• download the de-duplicated union of resolved files before the first workload;
• run both complete benchmarks sequentially;
• fetch and persist the first result before starting the second;
• use distinct remote logs and local JSON filenames;
• attach a shared suite/session ID, workload order, setup versions and prepared-file totals to both results;
• refuse to overwrite an existing result.

Workload order is counterbalanced across hosts to expose heating or order effects:

Page-cache eviction and physical-read verification remain mandatory for the second workload. Storage temperatures and device counters should be captured around each workload so a warmed or throttled SSD is visible in the result.

Analysis#

Results remain separate by workload. For each host and workload calculate:

• median and mean paired cold wall - warm wall;
• cold and warm means, standard deviations and individual samples;
• actual model-file read mean, minimum and coefficient of variation;
• physical bytes and effective physical-read rate during each cold run;
• warm-run physical reads;
• CPU-to-GPU bandwidth and warm runtime as host/GPU matching checks;
• negotiated GPU width, storage ancestry, shared bridges, NUMA placement and CPU-direct versus chipset path;
• device temperatures and throttling indicators when exposed.

Do not pool the five samples from different hosts as independent observations. Compare host-level summaries, show every host, and report layout ranges. A RAID claim requires the direction and practical magnitude to replicate across independent machines and both workloads.

Decision thresholds#

A single Gen5 NVMe is considered sufficient when both independent single-drive hosts satisfy all of the following for both workloads:

• all five cold samples pass physical-read verification;
• all warm samples show no material physical model reads;
• the real model-file reader stays around 10 GiB/s or higher without large degradation;
• the median paired cold penalty is within one second or 10% of comparable RAID hosts;
• warm GPU runtime is close enough to rule out a materially different GPU floor.

RAID 0 is recommended only if it repeatedly saves more than approximately two seconds of end-to-end cold latency on both workloads, or if single-drive throughput is unstable enough to miss the service target. Higher fio bandwidth without an application-level improvement is not sufficient.

Validation and launch log#

Before parallel acquisition, one dual-workload session will be monitored manually. Validation requires:

• one ComfyUI/PyTorch installation;
• both model sets present before the first measured suite;
• two successful, separate JSON outputs with the same suite ID;
• five cold and five warm samples in each output;
• every cold sample physically verified;
• the first result safely persisted before the second begins;
• no existing benchmark file overwritten;
• expected instance retention or teardown behavior.

After the RAID replicate completed, machines 147169 and 150656—the two initially known dedicated Ryzen 9950X, Gen5 x16, single-4-TB-Samsung-9100-Pro hosts—remained unavailable through 60 consecutive checks over about 32 minutes. A later search deliberately relaxed country, advertised speed and verification filters, while still manually rejecting shared multi-GPU hosts. That found machines 150183 and 148782. The second is supporting rather than primary evidence because Vast marks it unverified; its result is retained with that caveat instead of being discarded.

Results#

Suite validation: RAID host 147512#

The first dual-workload session used one dedicated physical RTX 5090, a Threadripper 7960X, 128 GB RAM and a PCIe 5.0 x16 GPU link. Physical inspection found a separate 1 TB WD Black SN7100 OS drive and two 4 TB Crucial T705 drives, each negotiating PCIe 5.0 x4. The T705 partitions formed a clean mdadm RAID 0 array with 512 KiB chunks and 2 MiB read-ahead. The PCI tree places the two T705 endpoints on separate root ports (0000:00:01.1 and 0000:00:01.2) in PCI domain/NUMA side 0, while the GPU is under root port 0000:40:01.1 in domain/NUMA side 40. This is not a shared-chipset-uplink RAID, but the cross-domain storage and GPU placement is another host-level difference that must remain visible in comparisons.

Both result files share suite ID vast-20260923T044419Z. The first workload downloaded the complete 61,108,590,469-byte de-duplicated file set; the second reported zero download time and found all eight files ready. ComfyUI was installed once at v0.37.0 with Torch 2.9.1+cu130. The MiniMax result was persisted before Krea began. The retained validation instance was explicitly destroyed after both results were inspected.

Results: MiniMax H3 JSON and Krea 2 JSON.

Validation invariants passed for both workloads:

• five cold and five warm samples were recorded;
• every cold sample passed the 80% physical-read requirement on its first attempt;
• every warm sample recorded zero physical reads;
• the files use separate names and neither replaced an older benchmark;
• suite metadata records one setup invocation, workload order and the shared prepared-file set.

NVMe SMART temperature data was unavailable because the container exposed NVMe sysfs topology but not the corresponding /dev/nvme*n* device nodes. The failed command and error are retained in both JSON files rather than silently treated as a temperature reading.

These values validate the runner but do not resolve the storage decision. Machine 147512 is one RAID host, and its Threadripper platform and cross-domain GPU/storage paths are not a matched replacement for the planned Ryzen 9950X system.

Supporting single-drive host 137184#

The counterbalanced session used a Ryzen 9950X and one WD Black SN8100 negotiating Gen5 x4. The SSD and tested GPU occupy separate CPU root ports (0000:00:01.2 and 0000:00:01.3) with no chipset bridge in either visible path. The physical host contains two 5090s and the tested GPU negotiated Gen5 x8, however, so this host cannot be a primary match for the intended one-GPU x16 build. Its median pinned host-to-device transfer was 26.89 GiB/s, versus 48.86-49.65 GiB/s on the x16 RAID host. This large transfer difference did not produce a meaningful warm-runtime difference in either tested workflow, but that observation does not replace a controlled x8-versus-x16 validation.

The single drive's 10.19-11.00 GiB/s real model reads also remained well below the measured 26.89 GiB/s host-to-device transfer rate. That makes the x8 link unlikely to be the immediate ceiling on its storage-fed load path and, directionally, an x16 production link should not make the single-drive result worse. Model loading includes CPU transforms, allocation and synchronization, however, so this is an inference rather than a substitute for the matched x16 run.

Both result files share suite ID vast-20260923T051235Z. Krea ran first and MiniMax second, the complete model union was ready before Krea, MiniMax downloaded zero additional bytes, and all cold samples passed on their first attempt. Every warm sample recorded zero physical reads.

Results: MiniMax H3 JSON and Krea 2 JSON.

First cross-host comparison#

Positive latency values mean the two-drive RAID host was slower. In this first comparison the single drive was faster on both cold means and actual model-file reads, despite the RAID host reaching 26.0-28.0 GB/s in direct QD32 fio. Warm times were effectively identical. This is strong evidence that the RAID array's headline parallel throughput did not help these application paths, but it is not yet a conclusive purchase result: there is only one host per layout, their CPUs and PCIe paths differ, and the single host's GPU is x8.

Independent AM5 RAID host 112410#

The second RAID suite used a Ryzen 9950X-class CPU, 128 GB RAM and a dedicated 575 W 5090 negotiating Gen5 x16. Its two 4 TB SN850X RAID members each negotiate Gen4 x4 on independent AM5 CPU root ports: 0000:00:01.2 and 0000:00:03.1. The new ancestry capture confirms that the members share no upstream PCI bridge with each other or the GPU. A separate 1 TB SN850X system device traverses three upstream bridges behind the chipset. This is the closest measured topology to the intended build, although its RAID media are Gen4 rather than Samsung 9100 Pro Gen5 drives.

Both result files share suite ID vast-20260923T054900Z. The runner installed ComfyUI v0.37.0 and Torch 2.9.1+cu130 once, prepared the complete model union before Krea, persisted Krea before MiniMax, and downloaded zero additional bytes for MiniMax. All ten cold samples passed on their first attempt and every warm sample recorded zero physical reads. The automatically provisioned instance was destroyed after both JSON files were saved.

Results: MiniMax H3 JSON and Krea 2 JSON.

The Krea means reproduce this same host's earlier independent v0.37.0 result to 15 ms cold and exactly at the reported millisecond warm mean. That repeatability makes the large host-to-host difference between the two RAID machines a real platform/path effect rather than five-sample noise.

Supporting same-media RAID host 55019#

Machine 55019 adds a useful negative control. Physical inspection found two 2 TB WD Black SN8100 drives, each negotiating Gen5 x4 on a separate root port, combined into a clean two-member mdadm RAID 0 with 512 KiB chunks and 2 MiB read-ahead. The array reached 24.28 GiB/s direct QD32 on MiniMax H3. Actual model reads nevertheless reached only 4.32 GiB/s and the cold penalty was 17.10 seconds. Krea 2 reproduced the same 4.30 GiB/s model-read ceiling and an 8.84-second cold penalty.

Results: MiniMax H3 JSON and Krea 2 JSON.

All ten cold samples passed physical-read verification on their first attempt and all warm samples read zero model bytes. The host is supporting evidence only: it is a four-GPU EPYC 9135 system, the offer allocated one quarter of the host's GPU/CPU/RAM resources, and the selected GPU was limited to 450 W. Its two storage endpoints and selected GPU also reside in different PCI domains. Those differences prevent a causal single-versus-RAID comparison with machine 137184. They do, however, prove that two matching SN8100 drives and very high direct-QD32 throughput are not sufficient to produce fast buffered model reads or a small ComfyUI cold penalty.

Exact single Samsung 9100 Pro MiniMax validation#

Two later MiniMax H3 runs validate the exact SSD family without replacing any earlier host result. Machine 150183 is a verified, dedicated 575 W RTX 5090 host with a Ryzen 9950X3D, 64 GB RAM, Gen5 x16 GPU and one 2 TB 9100 Pro on a direct CPU Gen5 x4 root port. Machine 148782 is a separate, dedicated 600 W RTX 5090 host with a Ryzen 9950X3D, approximately 96 GB RAM, Gen5 x16 GPU and one 4 TB 9100 Pro on another direct CPU Gen5 x4 root port. Vast marks machine 148782 unverified but not VM-deverified, so it is supporting evidence.

Results: 2 TB machine 150183 JSON and 4 TB machine 148782 JSON.

Both results contain five cold/five warm pairs with no rejected attempts. Every cold pass physically read 42.72-42.98 GB, and every warm pass read zero model bytes apart from an immaterial 1.39 MB in machine 150183's first warm sample. The drives have distinct serial numbers, the same 0B2QNXH7 firmware, and different motherboard/kernel environments. A single 9100 Pro therefore does not have the approximately 5 GiB/s ceiling observed on the older machines, and 4 TB capacity is not the cause of that anomaly.

The 4 TB host's separate sequential reader was slower than the 2 TB host's, yet its full-workload cold penalty was 3.088 seconds smaller. This reinforces that fio and a plain sequential reader are diagnostics, not substitutes for the verified ComfyUI cold/warm experiment. The source-level investigation of ComfyUI's actual eight-worker, per-tensor file path and the unresolved old-host hypotheses are recorded in the Samsung 9100 Pro investigation journal.

Completed dual-workload v0.37.0 matrix#

The single-drive result falls inside the two independent RAID hosts' range for cold time, cold penalty and real model-read rate on both workloads. Across the RAID hosts, Krea cold time varies by 1.417 s and MiniMax cold time by 1.334 s despite both using explicit two-drive arrays. The storage-layout effect, if any, is therefore smaller than ordinary cross-host platform/path variation in the completed matrix.

Within each host, the paired cold-minus-warm sample standard deviation was small relative to those cross-host differences:

Final finding and recommendation#

The completed fixed v0.37.0 runs do not show a practical RAID 0 advantage. The single SN8100 lands inside the two RAID hosts' observed range on both full workloads, while the RAID layouts range from faster than the single host to slower than it. Neither workload shows the predeclared, repeated improvement of more than two seconds. The fastest synthetic array result—approximately 28 GB/s direct QD32—did not produce the fastest model reads or cold starts. The two later exact-9100 MiniMax runs independently show that one 9100 Pro can also produce a small cold penalty without striping.

For the equal quoted total cost of one 4 TB Samsung 9100 Pro versus two 2 TB devices, the current purchase recommendation is therefore one 4 TB 9100 Pro, without RAID 0 initially. It preserves the second CPU-connected M.2 path and rear USB4 function, avoids array/topology/thermal complexity, and has a smaller cache-loss surface. The evidence does not justify consuming those resources for synthetic throughput that has not translated into application latency.

Install the single 4 TB device in M2_1 and accept the physical node only after checking all of the following with every intended device present:

1. The 9100 Pro negotiates Gen5 x4 on a direct CPU path.
2. The RTX 5090 negotiates Gen5 x16 under load.
3. The system SSD is on its intended chipset path and does not share the model-drive path.
4. The real model-file reader repeatedly sustains about 10 GiB/s or more after verified page-cache eviction.
5. Both full five-pair workloads reproduce cold times inside the new matrix, with every cold run physically verified and every warm run reading zero model bytes.

If the physical 9100 Pro build misses that storage target, diagnose slot placement, negotiated widths, filesystem/container path, thermal throttling and CPU-to-GPU transfer first. Add a second drive only after a same-host before/after test demonstrates a repeatable application improvement—not merely a higher fio number—and preferably clears the same two-second threshold on both workloads.

The exact 4 TB 9100 Pro has now been validated with the heavier MiniMax workload on a second physical host. An exact-drive Krea run was not collected because the later acquisitions intentionally focused on the open MiniMax anomaly rather than reinstalling the dual-workload suite. That narrower remaining gap does not change the RAID decision: RAID failed to show a repeated benefit across two complete application workloads, and a single exact-product drive has now avoided the old failure mode at both 2 TB and 4 TB capacities.

This recommendation supersedes the earlier provisional two-drive recommendation in model-read-storage-layouts.md, which was based on pre-matrix results where storage layout and host effects were not separated as rigorously.