mux

Commands

This describes the commands of release 0.3. Release 0.2.0 had seed, get and id only, under the name p2rent.

Progress and notes for people go to standard error. What a command produced goes to standard output, so it can be piped.

mux cp

$ mux cp <SOURCE>... <DESTINATION>

Copies each source to the destination. A source is a web address, a file, a folder, or a ticket for what another machine offers. The destination is a folder, or peer:, which offers the sources to other machines.

$ mux cp https://example.org/big.iso ./downloads/      # ./downloads/big.iso
$ mux cp https://example.org/big.iso ./image.iso       # under a name of its own
$ mux cp ./weights ./notes.txt /mnt/backup/            # a folder and a file
$ mux cp ./weights peer:                               # offer it, and print a ticket
$ mux cp 'mux://450b9c6b…' ./models/                   # ./models/weights, from whoever has it

The destination is read as cp reads it. If it is a folder, or is written with a slash at its end, the copy goes inside it under the name the source has. Otherwise it is the name the copy is given.

OptionMeaning
--hash <DIGEST>For a source on the web. The digest the source must have: blake3: or sha256: and 64 hex characters. A copy that does not have it is not kept.
--mirror <URL>For a source on the web. Another address that holds the same file, read together with the source. Repeatable. It needs a digest of the file, from --hash or from a server that sends one: nothing else tells that two servers hold the same bytes.
--streams <N>The most requests to make at once to each source, each for a part of a file. The default is 8. A server that refuses some of them is used with those it takes. With 1, a file is one request.
--stall-timeout <SECONDS>Give up after this long with next to nothing received: under about a kilobyte a second on every stream. A slow link is waited for. The default is 60.
--dry-runSay what would be copied, in the same form as a result, and copy nothing. A source on the web is asked what it holds and for none of the file.
--seedWhen the copy is whole, go on offering it to other machines until stopped: see below.
--jsonWrite for a program: see below.

A flag may be written before, between or after the places.

What a copy does

A folder is copied with everything in it: its files, the folders that hold nothing, links that stay inside it, permissions and the time each file was last changed. Special files, links that lead out of the folder, and copies that were never finished are left out, and a note says how many. Every file is written under a part name first, as above.

Copying a folder again copies all of it again, unless every file is already there with its size and time. Copying only what has changed is what sync will do. A folder cannot be copied into itself.

Standard output

One line for each source, with tabs between its three parts: the path that was written, its size in bytes, and how far it was checked.

CheckedMeaning
verifiedEvery byte was compared with a digest given for the file.
consistentNo digest was known. The source stayed what it was throughout. For a folder, nothing checks that: its files are read once, as they are.
already thereEvery file was there with the size and time the source gives, and nothing was read.

To and from another machine

$ mux cp ./weights peer:
seeding  weights  (400.68 MiB, 402 chunks)
mux://450b9c6b…6d2bfc54?peer=35964d95…f14dd0e7@https://euc1-1.relay.n0.iroh.link./

mux cp <PATH>... peer: offers files or folders on this machine to others. It writes one ticket for each path to standard output, and goes on serving until it is stopped with Ctrl-C. Only what is on this machine can be offered.

$ mux cp 'mux://450b9c6b…6d2bfc54?peer=35964d95…f14dd0e7@https://euc1-1.relay.n0.iroh.link./' ./models/
./models/weights	420143104	verified

mux cp <TICKET> <FOLDER> fetches what the ticket names, from every machine that holds it.

The web and other machines at once

The second machine to want a file from the web need not load the server again. The first copies it with --seed:

$ mux cp https://example.org/big.iso ./downloads/ --seed
./downloads/big.iso	1288490188	consistent
mux://450b9c6b…6d2bfc54?peer=35964d95…f14dd0e7@https://euc1-1.relay.n0.iroh.link./&url=https://example.org/big.iso

After the line that says what was written comes a ticket, and the command goes on offering the file. The ticket names this machine and the web address the file came from. Every other machine copies with it:

$ mux cp 'mux://450b9c6b…?peer=35964d95…&url=https://example.org/big.iso' ./downloads/ --seed
./downloads/big.iso	1288490188	verified

--seed offers what was copied from a folder too, as copying it to peer: would.

A ticket is all another machine needs, and whoever has it can fetch the data. It names the content by its hash and each machine by its key, with where that machine can be reached: a relay, as in the examples here, or an address. The ids are shortened in this page: each is 64 characters.

These flags say how the two machines reach each other. They are the same for cp, probe, seed and get:

Flag
--listen <ADDR>The address to listen on. By default every address, at a port the system picks.
--relay <URL>Reach machines behind routers through this relay, in place of the public ones.
--no-relayUse no relay and no directory. A machine is then reached at an address or not at all, and a ticket carries this machine's own.
--congestion <cubic|bbr>How a connection decides how fast to send. cubic, the default, halves its rate at each lost packet. bbr holds the rate the link carries whatever is lost: many times faster over a link that loses one packet in a hundred, and somewhat slower than cubic on a link that loses none. It is the sender's choice, so give it to the machine that offers.
--key <FILE>This machine's key, which is its name to other machines. Also MUX_KEY. It is made the first time a command has to do with another machine.

mux seed and mux get

mux seed <PATH>... is mux cp <PATH>... peer:, and mux get <TICKET> --out <FOLDER> is mux cp <TICKET> <FOLDER>/. They came first and are the same code: what one prints, takes with --json and ends with, the other does. Three flags are theirs alone:

Flag
seed --advertise <HOST[:PORT]>An address to put in the ticket for other machines to dial. Repeatable.
seed --chunk-size <SIZE>4K to 16M. The default is 1M.
get --peer <NODE[@HOST:PORT]>Another machine to try, as well as those in the ticket. Repeatable. A ticket can name it too: …&peer=NODE@HOST:PORT.

mux probe

$ mux probe <PLACE>

Says what is at a web address, a file, a folder or a ticket, and reads none of it. A web server is asked for the first byte of the file. The machines a ticket names are asked what it holds, and how much of it each of them has.

$ mux probe https://example.org/big.iso
name     big.iso
kind     file
size     1288490188 bytes (1.20 GiB)
files    1
checked  consistent, if it is copied
parts    read several at once

checked is how far a copy of the place would be checked. It is verified if a digest of every file is known: one given with --hash, or one a web server sends, which is then listed. parts says whether the place can be read at several points at once, or only in order from its start.

$ mux probe 'mux://450b9c6b…6d2bfc54?peer=35964d95…f14dd0e7@https://euc1-1.relay.n0.iroh.link./'
name     weights
kind     folder
size     420143104 bytes (400.68 MiB)
files    12
checked  verified, if it is copied
parts    read several at once
holder   35964d95  all of it

For a ticket there is a holder line for each machine the ticket names: all of it, how many bytes of it, or could not be asked for a machine that did not answer. If the ticket names web addresses, each has an origin line that says whether a copy would read from it, and if not, why.

mux sync

$ mux sync <SOURCE> <DESTINATION>

Makes the destination hold what the source holds, and reads from the source only what the destination lacks or has in another version. The source is a ticket, web address, file or folder. The destination is the folder, or the file, that is made to match: mux sync ./a ./b makes ./b match ./a, where mux cp ./a ./b/ puts a copy of a inside b.

$ mux sync ./weights /mnt/backup/weights --delete
remove	old-notes.txt
add	run-7/config.json
replace	run-7/model.bin

On the sending machine, run mux cp ./photos peer: and keep it running. On the receiving machine, use the ticket it prints.

mux sync "$TICKET" ./backup/photos --dry-run
mux sync "$TICKET" ./backup/photos

The ticket identifies one snapshot. To pull later edits, offer the updated folder and sync its new ticket to the same destination. Mux compares the verified manifest with the destination, then fetches the missing chunks of selected files from peers and any available web origins. It checks chunks recovered from an interrupted transfer or a previous version against the same manifest, including chunks shifted by an insertion. Add --delete to remove paths absent from the snapshot.

Standard output has one line for each path where there was something to do, with a tab between the step and the path. . is the destination itself.

StepMeaning
addThe destination had nothing there.
replaceThe destination had something else there.
touchIt held the same, with other permissions or another time.
removeThe source has nothing there. Only with --delete.
blockedThe destination has a folder where the source has a file or a link. Without --delete nothing is removed, so it is left as it is, and the command ends with 1.
OptionMeaning
--dry-runSay what would be done, and do none of it.
--deleteTake away what the destination has and the source has not. Without it nothing is ever removed.
--forceGo on though more than half of what the destination has would be removed.
--no-trashKeep nothing of what is replaced or removed.
--streams, --stall-timeoutAs for cp.
--listen, --relay, --no-relay, --congestionHow to reach the peers of a ticket, as for cp.
--jsonWrite for a program: a step line for each step, the lines of the copy, and a synced line at the end, or sync_plan with --dry-run.

What keeps a sync from losing files

What counts as the same

Two files are taken to hold the same if they have the same size and were last changed in the same second. This is the check rsync makes by default. Parts of a second are not compared, because file systems keep them to different lengths. A source that gives no time for a file, as some web servers do, is replaced every time.

What a system cannot keep is not compared. On Windows that is a file's permissions beyond whether it may be written, and a folder's permissions and time.

A file is synced with a file and a folder with a folder. Name the destination directly if its final path component is a symbolic link. Mux reserves the root-level name .mux-trash in every letter case and requires a real directory when it stores trash. The per-path rules are covered by the mirror proofs.

For a program

With --json, a command writes one JSON object on each line of standard output and nothing else there. Every line has v, the version of the shape of the lines, which is 1, and event. A later version of mux may add fields to a line without changing v. It will change v before it takes a field away or changes what one means.

eventWhenFields
startA copy begins.source and destination as they were typed, bytes and files to copy
progressFrom time to time, and when all of it is there.bytes at the destination, of how many
fileA file is whole and has just been given its final name. Each file that is copied gets one line: a file with no bytes as the copy begins. A file that was there already and is kept gets none.path
doneA copy is finished.path, bytes, checked, fetched, and kept from an earlier copy
planIn place of all the above, with --dry-run.source, destination, path, bytes, files, checked
probeThe answer of mux probe.place, name, kind, bytes, files, checked, parts, and digests, a list of path and digest. For a ticket also holders, a list of node and bytes: how much of it that machine has, or null if it could not be asked; and origins, a list of address and unused: why a copy would not read from that web address, or null if it would
offerA path is offered to other machines: by cp to peer:, by seed, or after a copy with --seed.ticket, name, bytes, files
stepOne thing a sync does, or would do with --dry-run.step and path, as on standard output without --json
syncedA sync is finished.source, destination, how many paths got each step (add, replace, touch, remove, keep, blocked), bytes in files added or replaced, fetched bytes read from suppliers, kept bytes reused in those files, and trash holding replacements and deletions (or null)
sync_planIn place of synced, with --dry-run.The planned counts and bytes
errorThe command failed. It is the last line.code, the exit code; kind, the same in a word; message; hint, or null

checked is verified, consistent or already there, as on standard output without --json. The words for kind are failed, usage, absent, unreachable, changed and corrupt, in the order of the exit codes from 1.

$ mux cp --json https://example.org/a.bin ./out/
{"v":1,"event":"start","source":"https://example.org/a.bin","destination":"./out/","bytes":1100000,"files":1}
{"v":1,"event":"progress","bytes":1100000,"of":1100000}
{"v":1,"event":"file","path":"./out/a.bin"}
{"v":1,"event":"done","path":"./out/a.bin","bytes":1100000,"checked":"consistent","fetched":1100000,"kept":0}

A person's error message is still written to standard error.

Exit codes

They are the same for every command and do not change between versions.

CodeMeaningWorth running again
0Done.
1It failed for a reason not listed below.
2The command line is wrong, or names a place that cannot be used.No.
3Nothing is at the source.No.
4The source could not be reached, or stopped sending.Yes: it carries on with what it stored.
5The source changed while it was being read.Yes: it copies the source as it is now.
6What arrived is not what was asked for: it does not have the digest given.No.

An error is one line on standard error beginning error:. Where there is something to do about it, a line beginning hint: follows.

2 is for a command that asks for what mux does not do, however often it is run: too few places, a place that is not one, two places that cannot be copied between, a flag that is not for that kind of place, a folder copied into itself, a new name for what a ticket names. The command has to be written another way. A command refused like this copies nothing, opens no socket and makes no key.

A sync that stops because of what it found ends with 1: a folder in the way, more than half of the destination to remove, a file where a folder is. The message says which flag goes on, if that is what is meant.