Samsung 9100 Pro cold-load investigation journal#

Status: Original slow machine 147169 fully traced and destroyed; dedicated 400 W adjacent control running Started: 2026-09-23 Scope: RTX 5090 MiniMax H3 runs with verified page-cache eviction, with emphasis on why some single Samsung 9100 Pro hosts reached only approximately 5 GiB/s through the real model path despite healthy direct-I/O results

This is the durable evidence journal for the storage-path investigation. It intentionally records negative results, source-level details and unresolved alternatives as the work proceeds. Purchase conclusions remain in the single-NVMe versus RAID 0 experiment.

2026-09-23: question and prior anomaly#

Two earlier dedicated RTX 5090 hosts exposed one 4 TB Samsung 9100 Pro on a direct CPU PCIe 5.0 x4 root port while the GPU negotiated PCIe 5.0 x16:

• machine 147169: 18.397-second cold penalty and a 5.11 GiB/s mean normal model-file read;
• machine 150656, first run: 21.317-second cold penalty and a 5.93 GiB/s mean model-file read;
• machine 150656, repeat: 27.779-second cold penalty and a 4.81 GiB/s mean model-file read.

Those runs used ComfyUI v0.36.0 and comfy-aimdo 0.5.3. Their direct 1 MiB QD1 fio reads were much faster, generally approximately 10-14 GB/s. The discrepancy establishes a bottleneck somewhere in the buffered/application path; it does not by itself establish a physical limit of one 9100 Pro.

Both drives reported firmware 0B2QNXH7, PCIe 5.0 x4, scheduler none, 128 KiB read-ahead and a 128 KiB maximum request size. Neither host exposed usable NVMe temperature telemetry in its saved result. Both had Ryzen 9950X-class CPUs, approximately 128 GB RAM and dedicated 575-600 W GPUs. No cgroup io.max throttle was found. CPU memory-copy and pinned host-to-device measurements were healthy.

2026-09-23: exact ComfyUI and aimdo I/O path#

The source was inspected from temporary, detached clones at these exact revisions:

• ComfyUI v0.37.0: 73c9bad4d21e7addbe1d13bc92eee0f1431b017d;
• comfy-aimdo v0.5.5: 3b8e8c162efeb9470d912609a7a6e7a2b1c693ec.

ComfyUI v0.37.0 pins comfy-aimdo==0.5.5. It memory-maps safetensors, retains the backing file descriptor and annotates each tensor storage with its exact file slice. When an eligible tensor is copied, the normal fast path calls read_file_to_device rather than first faulting the tensor through the mmap.

The Linux aimdo transfer implementation is more concurrent than an ordinary one-thread buffered reader:

• xfer-file.c defines eight workers, 2 MiB tasks and a 256-entry queue;
• one tensor slice is divided into 2 MiB tasks and the caller waits for all of that tensor's tasks to finish before returning (task submission and wait);
• hostbuf-file-reader.c stages reads through 64 MiB pinned-host windows and issues asynchronous host-to-device copies;
• Linux reads use pread. No POSIX_FADV_SEQUENTIAL or POSIX_FADV_WILLNEED hint is issued. The only file advice in that path is optional POSIX_FADV_DONTNEED after a read when mark_cold is true (implementation);
• mmap bounce() is exactly MADV_DONTNEED (implementation). It applies to mmap-backed fallback copies marked dirty, not the successful direct file-slice path.

The relevant Linux I/O files in aimdo v0.5.3 and v0.5.5 are byte-identical: src/xfer-file.c, src-posix/xfer-file-plat.c, src/hostbuf-file-reader.c and src-posix/model-mmap.c have identical SHA-256 hashes across the two tags. The ComfyUI v0.36.0-to-v0.37.0 diff also leaves comfy/memory_management.py's direct file-slice reader unchanged. Therefore, the old slow results cannot be explained by a change from aimdo 0.5.3 to 0.5.5. v0.37.0 did change pin-management policy, so old and new end-to-end runtimes should still be labelled by version, but the underlying file-read mechanism is the same.

Actual MiniMax tensor sizes#

Safetensors headers were range-read from the exact persisted MiniMax source files. For each tensor, the available aimdo worker count is min(8, ceil(tensor_bytes / 2 MiB)). The last column weights that count by bytes rather than by the number of tensor records.

Many tensor records are tiny, but they account for very few bytes. The byte-dominant tensors can use most or all eight workers. This rules out the simple hypothesis that MiniMax is slow because aimdo can only issue one storage request at a time. It does not prove that the current worker/window pipeline is optimal.

2026-09-23: relaxed single-9100 replication, machine 150183#

A broader Vast search relaxed country, power and host-selection requirements, then manually rejected shared multi-GPU offers. Machine 150183 was selected because it was a dedicated physical GPU host with a 575 W RTX 5090, Ryzen 9950X3D, 64 GB RAM, PCIe 5.0 x16 GPU and one advertised 2 TB Samsung 9100 Pro.

Result: machine 150183 JSON.

Validation passed: five cold/five warm pairs, no rejected cold attempts, all cold runs physically read 42.72-42.89 GB, and warm samples read zero model bytes except 1.39 MB of immaterial process I/O in the first warm sample. The drive was a distinct 2 TB 9100 Pro with the same 0B2QNXH7 firmware. It was on direct CPU root port 0000:00:01.2, negotiated PCIe 5.0 x4, and shared no upstream bridge with the GPU. The system drive was a separate Gen4 Seagate device behind the chipset.

The 9100 queue still exposed 128 KiB read-ahead, max_sectors_kb=128 and max_hw_sectors_kb=128. Therefore, neither 128 KiB read-ahead nor the 128 KiB hardware request limit is sufficient to explain the older approximately 5 GiB/s results. This host nearly reached 10 GiB/s through a single drive with those same values.

A separate 15-second sysfs monitor ran throughout the workload. The 9100 composite temperature rose from approximately 46°C to a sampled maximum of 64°C; its hottest sampled sensor reached 71.85°C. Sysfs exposed 83.85°C maximum and 87.85°C critical thresholds, and no alarm was observed. The drive therefore remained below its exposed thermal limits while preserving fast end-of-run reads.

The cold resource trace is also instructive. Physical reads span almost the entire 71-second execution window because weights are demanded throughout initialization and inference, but actual read-active intervals average only 12.71 seconds. The simple end-of-run model reader takes approximately 4.13 seconds. Storage I/O is interleaved with model setup, GPU transfers and compute rather than forming one isolated bulk load at the start.

What machine 150183 establishes#

• A single 9100 Pro can sustain approximately 9.6 GiB/s through the benchmark's real model files.
• RAID 0 is not required merely to cross the older hosts' approximately 5 GiB/s ceiling.
• aimdo version, 128 KiB block read-ahead and the 128 KiB request limit do not individually explain that ceiling.
• The older runs were bottlenecked by a host/path condition, not the advertised SSD model alone.

What it does not establish#

• It is a 2 TB drive, while the older anomalous hosts used 4 TB drives.
• It uses a B850/9950X3D platform and Linux 7.0, while the older hosts used X870E/9950X and Linux 6.8.
• It does not reveal temperatures from the old hosts, so thermal throttling remains plausible but unproven.
• It is one independent host. Host storage contention and container backing-path effects can still vary.

2026-09-23: relaxed 4 TB replication, machine 148782#

A second relaxed-support run completed successfully on Vast instance 52190732:

• offer 51046112, machine 148782, Finland;
• dedicated 600 W RTX 5090, PCIe 5.0 x16;
• Ryzen 9950X3D, approximately 96 GB RAM;
• one advertised Samsung 9100 Pro 4 TB, firmware 0B2QNXH7, direct CPU PCIe 5.0 x4;
• MSI MPG X870E Carbon WiFi;
• Vast verification is unverified, is_vm_deverified=false, reliability 0.989559.

This host is supporting rather than primary evidence because of the unverified status, but it matches the old drive capacity and intended GPU topology unusually well. Manual inspection before measurement found 128 KiB read-ahead but max_sectors_kb=max_hw_sectors_kb=512, unlike the 128 KiB limit on the previous 9100 hosts. The updated benchmark captures queue configuration and block counters before and after the full run even when /dev/nvme* is hidden from the container. This particular host exposes neither a usable NVMe device node nor NVMe hwmon temperature sensors to the container, so its saved temperature-sensor list will be empty. It can replicate the performance result but cannot directly settle the thermal hypothesis.

Result: machine 148782 JSON.

Validation passed: five cold/five warm pairs, no rejected attempts, all cold runs physically read 42.82-42.98 GB and every warm run recorded zero physical reads. Cold wall times were 65.913-67.139 seconds; the result is not driven by one lucky sample. The instance was destroyed automatically after the result was persisted.

The late model-file reader ran only after the synthetic storage probes and five full cold passes. It still sustained 7.585 GiB/s, materially above the older 4.81-5.93 GiB/s 4 TB results. That rules out 4 TB 9100 Pro capacity as an inherent approximately 5 GiB/s limitation. It also provides an indirect negative check against severe late-run throttling on this host, although the absent temperature telemetry prevents a direct thermal conclusion.

The result exposes an equally important measurement limit. Machine 148782's plain sequential model reader was slower than machine 150183's 9.581 GiB/s, yet its cold penalty was 3.088 seconds smaller and its physical reads occupied 3.169 fewer seconds. The normal 8 MiB reader and fio are not exact proxies for aimdo's eight-worker, per-tensor pread path. The 4 TB host's 512 KiB request limit, versus 128 KiB on machine 150183, is one plausible advantage for aimdo's concurrent requests, but it remains confounded with the kernel, filesystem, motherboard and host.

Across the six independent v0.37.0 RTX 5090 MiniMax hosts now available, DuckDB reports:

• correlation of cold penalty with cold physical-read active time: 1.000;
• correlation of cold penalty with the separate model-reader rate: -0.821;
• correlation of cold penalty with major page faults: -0.117.

These are observational correlations over only six heterogeneous hosts, not causal estimates. They do show that physical-read active time from the full workflow is the closest current storage diagnostic. The earlier apparent major-fault relationship does not survive the cleaner v0.37.0-only matrix.

Combined single-9100 interpretation#

The two independent machines use different drive serial numbers, motherboard models and kernels. Together they establish that both 2 TB and 4 TB single-drive configurations can avoid the old failure mode. The unverified 4 TB host remains supporting evidence, but the verified 2 TB result independently establishes the core claim that a single 9100 Pro can feed this workload near 10 GiB/s without RAID 0.

Current hypothesis ranking#

1. A time-varying host or storage-path condition on the older machines — most consistent with the evidence, but not uniquely identified. Machine 150656's early direct fio reached approximately 10-14 GB/s while its late normal model read fell to 4.81-5.93 GiB/s, and the same physical host became slower on its repeat. Contention, filesystem/container backing behavior and controller state can all produce this shape. The existing old results do not contain enough phase-level telemetry to separate them.
2. Late-run thermal or controller throttling on the older hosts — plausible, not proven. The old runs applied sustained synthetic I/O and five 42 GB cold passes before the late reader, but did not expose temperature sensors. The 2 TB replication stayed below its reported thermal thresholds and remained fast late. The 4 TB replication also remained fast late, but its container hid temperature telemetry. This means 4 TB capacity is not inherently slow; it does not prove the old drives never throttled.
3. Filesystem, kernel or request-shaping differences — plausible. The successful hosts differ from the old hosts in kernel, motherboard and backing filesystem. The 4 TB success exposes a 512 KiB request limit while the old hosts and successful 2 TB host expose 128 KiB. Because the 2 TB host was fast at 128 KiB, that limit is not a sufficient cause; a same-host intervention is required to test whether 512 KiB still benefits aimdo.
4. Aimdo queue depth is inherently too low — contradicted. Source and tensor-byte distributions show that most bytes can use roughly seven to eight workers.
5. A 4 TB 9100 Pro is inherently limited to approximately 5 GiB/s — contradicted. The distinct 4 TB drive on machine 148782 sustained 7.585 GiB/s in the late plain reader and a 5.732-second full-workload cold penalty.
6. 128 KiB read-ahead/request limits impose a 5 GiB/s ceiling — contradicted as a sufficient cause. Machine 150183 retained both 128 KiB values and reached 9.581 GiB/s.
7. GPU PCIe width or host-to-device bandwidth caused the old file-reader result — contradicted. The independent normal model reader does not copy to the GPU, and all measured 9100 hosts used x16 GPUs.
8. RAID 0 is intrinsically required for the workload — contradicted. The single SN8100 result and both single 9100 replications reach application performance comparable to successful arrays.

Cross-host factor audit#

The investigation is a multivariable observational problem, not a one-axis SSD comparison. The current single-9100 matrix exposes this factor pattern:

The overlay implementation is a candidate because it perfectly separates the old and new 9100 results, but it is not established as the cause. Machine 55019 is a counterexample to a general “Docker overlay2 is fast” rule: its two-drive SN8100 RAID also used Docker overlay2 and reached only 4.323 GiB/s in the model reader. Runtime/overlay must therefore remain one covariate alongside device state, filesystem, kernel, queue behavior and host contention.

ComfyUI version has a useful same-machine control on RAID host 112410:

Across those three runs, model-read rate varies by only 1.4% from minimum to maximum and cold penalty by 0.55 seconds. This does not prove version can never interact with a different path, but it strongly argues against v0.36-to-v0.37 being the broad cause of the 9100 improvement. It agrees with the source comparison: the relevant aimdo Linux I/O files and ComfyUI direct file-slice reader did not change.

Further analysis will keep the physical host as the independent unit and retain these dimensions: storage model/count/array and PCIe ancestry, kernel, overlay/runtime path, queue limits, CPU/RAM, GPU power and link, host-to-device bandwidth, model-reader rate, full-workload physical-read active time, cold penalty and warm GPU floor. With the current small and heterogeneous sample, descriptive stratification and same-host comparisons are valid; fitting a many-variable causal regression is not.

Next discriminating evidence#

1. If a slow 4 TB host becomes available, run a same-host temporal sequence: model reader before synthetic load, sustained read/write heat load, then model reader after load while sampling all NVMe sensors.
2. Test the 128 KiB versus 512 KiB request-limit hypothesis on one controllable local system, preferably by lowering the soft limit on the same drive when Linux permits it. Do not infer causality from the two Vast hosts alone.
3. Add an aimdo-shaped storage probe: 2 MiB pread tasks, eight workers, per-tensor barriers using the actual safetensors slice list. Plain buffered and direct fio are not exact substitutes.
4. Preserve every independent host result. Do not overwrite or average away slow hosts; host-level variance is itself part of the deployment risk.

2026-09-23: original slow machine 147169 reacquired#

The exact physical machine from the first anomalous run became rentable again after the two independent replications completed. Vast offer 51432163 was acquired as instance 52195702 at 08:16 UTC:

• verified dedicated host in Oklahoma, reliability 0.9814785;
• Ryzen 9950X, 126 GB advertised RAM;
• 600 W RTX 5090, advertised PCIe 5.0 x16;
• advertised storage nvme, 8.676 GB/s, with the prior result identifying its 4 TB Samsung 9100 Pro;
• ComfyUI pinned to v0.37.0, resident cache policy, full five-pair MiniMax H3 benchmark;
• instance retained after the standard run for same-host phase and storage-path probes.

This is the highest-value remaining comparison because it holds the physical drive, motherboard and host constant while updating ComfyUI and the benchmark telemetry. If the slow result repeats, the retained host can be probed directly. If it disappears, the earlier limitation was temporal or software/runtime-state dependent rather than an invariant property of the drive or topology.

Adjacent-host collection and dedicated-host guard#

The wider factor matrix may include lower-power 5090 hosts as supporting observations, provided GPU power, CPU, RAM, PCIe link and storage topology remain explicit columns. An attempted 450 W acquisition exposed an important Vast selection ambiguity: offer 49868233 reported num_gpus=1 but gpu_frac=0.25. That means the offer allocates one GPU from a four-GPU physical host; it does not mean a quarter of one GPU, but CPU, RAM, storage and platform resources still belong to the shared host class this experiment intends to separate. Instance 52198067 was cancelled and destroyed before setup or measurement.

benchmark_vast.ts now requires gpu_frac=1 in both offer search and post-selection validation. The only override is an explicit --allow-multi-gpu-host, intended for clearly labelled supporting runs. This closes the gap where num_gpus=1 alone could silently select one allocated GPU on a multi-GPU machine.

The replacement adjacent control is Vast machine 108899, instance 52198443:

• verified single-GPU physical host (gpu_frac=1), reliability 0.998713;
• Ryzen 9950X, 64 GB advertised RAM and B850 Gaming Plus WiFi;
• 400 W RTX 5090, advertised PCIe 5.0 x16;
• approximately 6 GB/s advertised NVMe;
• ComfyUI v0.37.0 and the same full five-pair MiniMax H3 benchmark.

This host is not a matched SSD comparison. Its purpose is to add a lower-power observation with a close CPU class and full-width GPU link, so warm runtime can expose the GPU-power effect while paired cold penalty and captured topology describe the storage path.

Critical discovery: Vast's container path is double-buffered through a loop file#

The retained machine 147169 exposed enough host-visible sysfs metadata to trace the model path more deeply. The container root is overlayfs, but its writable container storage lives on a 2.9 TB XFS loop device:

• /sys/class/block/loop0/loop/backing_file is /var/lib/docker-loop.xfs;
• /sys/class/block/loop0/loop/dio is 0, so the loop backing file uses buffered rather than direct I/O;
• /etc/hosts exposes /dev/loop0 as XFS, while the overlay upperdir is under the same host containerd storage tree;
• cgroup io.stat separately attributes large read totals to device 7:0 (loop0) and much smaller totals to 259:0 (the Samsung NVMe namespace).

The dedicated aimdo-shaped probe made the accounting mismatch explicit. Each repetition evicted the four model files, then /proc/self/io recorded approximately 42.47 GB of process block reads. Across the same repetitions, Samsung nvme0n1 counters increased by only 0-328 MB, not 42 GB. The reads were charged to the loop-backed container path, while the underlying physical-device counters barely moved.

A final one-pass backing-path probe captured every counter around the same 42.47 GB read. The raw evidence is in the storage-backing-path probe JSON:

The logical pass took 5.461 seconds, or 7.244 GiB/s. Exact equality between the process and loop read totals, together with zero NVMe read sectors, shows that this pass missed the inner XFS/model-file cache but was served from the buffered /var/lib/docker-loop.xfs backing file in the host page cache. Incidental NVMe writes occurred concurrently, so device busy-time deltas alone would have been misleading; the read-sector and cgroup read-byte deltas are the relevant evidence.

This means the existing field named physical_read_bytes is too strong semantically on these Vast hosts. Linux /proc/<pid>/io read_bytes proves that the process missed its immediate file page cache and caused block-layer I/O; it does not prove the bytes reached physical NAND when a buffered loop backing file adds another cache layer. POSIX_FADV_DONTNEED on the inner model file does not guarantee eviction of the outer /var/lib/docker-loop.xfs backing-file pages.

Same-host result under the corrected interpretation#

The reacquired v0.37.0 run on machine 147169 completed successfully:

Every new cold pass recorded approximately 43 GB of process block reads and every warm pass recorded zero, but paired penalties declined sharply within one session: 12.485, 14.497, 9.343, 7.170 and 7.104 seconds. Read-active time declined in the same direction: 16.837, 17.836, 12.829, 10.826 and 10.625 seconds. Fixed PCIe topology, kernel, SSD, firmware and ComfyUI cannot explain that within-session change.

The post-run probe compared three cache-evicted one-thread 8 MiB sequential reads with three approximate aimdo reads using eight workers, 2 MiB pread tasks and per-tensor barriers:

The approximation omits aimdo's pinned buffers and GPU transfers, but it shows that eight-worker scheduling does not bypass this host's container-storage ceiling. NVMe sensors remained approximately 42.85°C through all six reads, directly arguing against thermal throttling during this probe.

Consequence for prior conclusions#

• The results remain valid measurements of the Vast container storage path experienced by ComfyUI.
• They are not clean physical-SSD experiments, even when lsblk shows one 9100 Pro or an mdadm array.
• “Single 9100,” “single SN8100” and “RAID” must be treated as host topology labels until block counters show that model reads reach those devices rather than an outer loop cache.
• The weak relationship between advertised disk speed, direct fio and cold penalty is now unsurprising: those measurements can exercise different layers of the path.
• The bare-metal purchase must be accepted on its own direct filesystem with both process I/O and physical NVMe counters. Vast cannot, by itself, settle whether a direct local 9100 Pro needs RAID 0.

The collector must capture all loop devices, loop backing-file and dio settings, cgroup device counters, and before/after counters for every relevant block layer. Future wording should call /proc values “process block-read bytes” rather than “physical reads” unless underlying-device deltas corroborate them.

That collector change is now implemented as benchmark schema v4. New results retain the old physical_read_* summary aliases only for compatibility, explicitly mark their semantics, and add:

• process_block_read_* names and verification wording;
• all top-level block devices with device numbers, counters and queue settings;
• loop backing file, buffered/direct-I/O flag, offset and size limit;
• cgroup I/O counters and block-layer snapshots around every cold and warm sample;
• the same block/cgroup snapshots around each independent model-file reader pass.

This is the minimum evidence needed to distinguish an inner file-cache miss from a read that actually reaches the advertised NVMe or RAID devices.

Schema-v4 validation on the retained host#

Before destroying instance 52195702, a one-pair MiniMax validation exercised the new collector. The complete result is machine 147169 schema-v4 validation JSON. It recorded a 71.596-second cold pass, 60.234-second warm pass and 11.362-second paired penalty. The run is a collector validation, not another five-pair host replicate.

The per-layer counters expose partial and variable service from the physical SSD:

Process, loop and cgroup counters agree closely at the inner layer. The changing NVMe share proves that the same nominal cache-eviction operation creates a mixture of physical storage reads and outer-cache hits. It also explains the otherwise puzzling reader variation on one unchanged SSD: the slowest reader pulled the largest fraction from NVMe, while the fastest pulled none. This is stronger than a cross-host correlation because filesystem, kernel, drive, firmware, CPU and PCIe topology are held constant.

The warm pass charged zero read bytes to either cgroup device. Small host-wide sysfs activity occurred on both devices during that minute, which demonstrates why cgroup attribution should accompany global device counters on a busy host.

The schema-v4 JSON contains loop backing metadata, cgroup counters and before/after snapshots at every sample and model-reader boundary. Instance 52195702 was then destroyed, and the Vast API confirmed that it was no longer present.