Should I train many epochs directly off the mount?

No. For many epochs of random access at full speed, stage the data on the local NVMe of the GPU box. The mount is for the part before that: inspecting, sampling, the first pass, and getting checkpoints back.

Is this cheaper than object storage?

You pay nothing to store the dataset, because it stays on hardware you already own. Putting it in a bucket means paying every month for a second copy of data you already have. Egress is a different matter: several GPU providers and object stores charge none, so transfer fees are not the saving here.

Mount the dataset. Skip the bucket.

Q: Does it work with WebDataset shards?

The mount is a normal path, so a shard directory works with a standard file based loader. Sequential reads of large tar shards are the access pattern that suits a streaming mount best.

Your data stays on the machine that already holds it.

By Marius-Florin Cristian · 18 August 2026

Mount a workstation or lab NAS on the rented GPU box. Shards read on demand. Checkpoints write back to the workstation during the run.

Training starts on the data where it already lives. Storage stays on hardware you own.

What the numbers on this page are. Filesystem measurements: metadata cost, changed-region transfer, session stability. Dataloader throughput, epoch times and checkpoint figures are being recorded now. They go on this page when they exist, and the form at the bottom sends them to you.

Suited to

Inspect and sample

Shard listing, a few examples, label distribution, a sanity pass. Reads pull what they read.

Start the first pass early

Begin while a copy would still be running. Large sequential shard reads suit it best.

Storage you already own

The dataset stays on the machine that holds it. Bucket storage bills monthly for a duplicate.

Checkpoints go home

Write into the mount and each checkpoint reaches the workstation as it is written.

Then stage locally

For many epochs of random access at full speed, copy to the instance NVMe. That stays the right answer.

Measured, on the filesystem

Measurement	Result	Conditions
10,000 file directory usable on the peer	116 ms	Loopback, metadata only, about 73 bytes per file
Re-read after a small change	18.56 MB for a 2 MB change in a 1 GB file	Loopback, byte exact by sha256, floor about 16 MB
Continuous session	102 of 102 cycles byte exact, 0 reconnects	17 hours across two runs, wide area, bridged

The metadata figure matters for a shard directory: the listing arrives first, content follows on read. The change based figure describes the filesystem, and a checkpoint benchmark is a separate measurement.

Usage

A standard file based loader takes the mount with no special client:

ls ~/KeibiDrop/Mount/datasets/imagenet-wds/ | head
python -c "import webdataset as wds; \
  ds = wds.WebDataset('~/KeibiDrop/Mount/datasets/imagenet-wds/train-{0000..1023}.tar')"

Checkpoints go back the same way:

torch.save(state, "~/KeibiDrop/Mount/runs/exp-14/ckpt-step-2000.pt")

The kd command prints one JSON line per call, so a script or an agent can drive mounts and transfers directly.

Limits

Stage locally for heavy training. Many epochs of small random reads belong on the instance NVMe.
The saving is storage. RunPod, Lambda, Voltage Park, Salad, R2 and Wasabi all charge zero egress, so transfer fees are a wash. The monthly bill for a duplicate copy is what goes away.
Training loop numbers are in progress. They appear here with their conditions once measured.
Two machines per connection. A share joins two peers. Fanning one dataset out to many workers is a separate job.

Common questions

Should I train many epochs off the mount?

Stage to local NVMe for that. The mount earns its place on inspection, the first pass, and checkpoints.

Is it cheaper than object storage?

Storage costs you zero, because the dataset stays on hardware you own. A bucket bills monthly for a second copy of the same data.

Does it work with WebDataset shards?

Yes. The mount is a normal path and large sequential shard reads suit it.

Can an agent drive it?

Yes. kd prints one JSON line per command.

Which measurement would decide it for you?

We are choosing what to benchmark first: dataloader throughput on shards, checkpoint write back during a run, or time to first batch against an upload. Mail marius@keibisoft.com with the one that decides it for you.

A form for this arrives once the endpoint is live.

Use cases · The kd agent CLI · Eager metadata at 100,000 files