mux

How mux works

A picture-first walk through one artifact, from seed to a finished swarm. For module layout see architecture; for the bytes on the wire see protocol.

The problem

When many machines download the same thing from one place, they split that place's uplink between them. Ten machines each get a tenth.

 direct download                    mux

      source                        source ─── A ─── C
   ┌──┬──┼──┬──┐                       │   ╲   │   ╱ │
   A  B  C  D  E                       B ───── D ─── E

 five machines share one uplink     every machine is also a source

In a swarm, a machine starts serving a chunk the moment it has verified it. Adding machines adds upload capacity, so the rollout does not slow down as the fleet grows.

1. Seeding turns files into a ticket

mux seed reads the files once, splits each into fixed-size chunks and hashes every chunk. The list of files and chunk hashes is the manifest. The hash of the manifest is the artifact id.

 weights/                     manifest
 ├── config.json    ──┐    ┌───────────────────────┐
 ├── model-00.bin   ──┼──▶ │ files:  path, size    │ ──BLAKE3──▶ artifact id
 └── model-01.bin   ──┘    │ chunks: hash, hash, … │
                           └───────────────────────┘

Nothing is copied. The seeder serves straight from the original files.

2. The ticket says what, who and where

mux://450b9c6b…6d2bfc54?peer=35964d95…f14dd0e7@10.0.0.5:4433
         └───────┬───────┘      └───────┬───────┘ └─────┬─────┘
           what to fetch          who to trust    where they are
      (hash of the manifest)    (node public key)

3. The swarm fills in

Each row is a moment in time. Each bar is one machine's copy of an eight-chunk artifact.

 t0   origin ████████    A ░░░░░░░░    B ░░░░░░░░    C ░░░░░░░░
 t1   origin ████████    A ██░░░░░░    B ░░█░░░░░    C ░░░░░██░
 t2   origin ████████    A ████░█░░    B ░░███░█░    C █░░░░███
 t3   origin  offline    A ████████    B ███████░    C █░██████

4. Nothing unverified reaches the disk

                                           ┌─ equal ────▶ write to the part file,
 peer ── chunk 17 ──▶ BLAKE3 ──▶ compare ──┤              offer it to others
                                 with the  │
                                 manifest  └─ different ▶ discard, ban the peer,
                                                          ask someone else

Integrity does not depend on trusting peers. A peer that lies wastes one request and is never contacted again.

A file is written as <name>.mux-part and gets its real name only when every chunk of it has been verified:

 downloading   model.bin.mux-part   ██████░░██░░░░░░
 last chunk    model.bin.mux-part   ████████████████   flush to disk
 whole         model.bin            ████████████████   renamed in one step

So a file under its real name is always complete and correct, however the download ended. A file that was already there is replaced by that one rename or left alone, never changed in place.

5. Resume is a re-hash

There is no state file. On start, get hashes whatever is already in the destination and requests only what does not match.

 after a crash     ██████░░██░░░░░░     re-hash the part file
 run `get` again   ██████▒▒██▒▒▒▒▒▒     fetch only the gaps

The same mechanism repairs a damaged copy or updates an old one: point get at it, and the chunks that are already right are copied from it instead of being fetched. The old file stays as it is until the new one is whole.

What this buys as the fleet grows

This is a model, not a measurement. mux has so far been run with three nodes on one machine.

Assume a 50 GB artifact and a 1 Gbit/s link on every machine, source included.

MachinesDirect, one sourceSwarm, best caseServed by the source: directServed by the source: swarm, at least
101 h 7 minabout 7 min500 GB50 GB
10011 habout 7 min5 TB50 GB
50056 habout 7 min25 TB50 GB

The shape matters more than the exact numbers: direct time grows with every machine added, swarm time stays roughly flat, and the source ships about one copy instead of one per machine.

It does not help when there is one downloader, when the source link is already faster than the downloaders can write, or when machines cannot reach each other.