The architecture in three planes, the package map, every decision on record with its reasoning, how to build and test it, and a plain list of what has been proven on real infrastructure and what has only been written.
Go, cgo, userspace WireGuard, Linux and Android.
The front page says they are deliberately separate. Here is what each one actually is, because the separation is the whole design and most of the confusion about mesh VPNs comes from conflating them.
Userspace WireGuard, the real protocol. One WireGuard device per mesh, on its own interface and its own UDP port. Direct peer to peer where NAT allows, through a relay you run where it does not.
Logos Delivery pub/sub, used intermittently: cold start, network change, partition repair. Rotating content topics that stay on one shard. It never carries traffic.
Announcements, revocations, credentials, service lists. Signed by the publishing device, encrypted to the mesh, padded to a fixed size, and carried over the rendezvous plane.
internal/wg embeds wireguard-go and shares its UDP
socket with our own control protocol. This is not an optimisation. NAT
traversal and the tunnel must use the same socket, or the
reflexive address you discover is not the port your data actually arrives
on and a hole punch lands on the wrong mapping. Kernel WireGuard owns its
socket and will not share it, which is the reason the data plane is in
userspace at all
(ADR-002).
The demux follows Tailscale's magicsock discriminators and separates on the first two bytes:
WireGuard : msg[0] in 0x01..0x04 and msg[1:4] == 0x000000
disco : msg[0] == 0x54 # Tailscale's
STUN : msg[1] == 0x01 with the magic cookie at offset 4
# ours therefore starts with a byte > 0x04 and != 0x54
It wraps the ReceiveFuncs returned by StdNetBind
and filters in place, following NetBird's ICEBind. That preserves batching
and GSO/GRO offload — the data path never goes through a channel, which is
what makes go-libp2p's shared-conn approach unsuitable here
(ADR-001).
On that shared socket, internal/disco sends small probes.
WireGuard holds exactly one endpoint per peer, so you cannot spray
handshakes at five candidates and let the best win — you would be
overwriting the endpoint under yourself. Probe with cheap packets first,
then set the endpoint that answered
(ADR-009).
Every pong echoes the source address it was observed at, so with N peers you
get N-1 vantage points on your own public address and need no STUN server.
Probes are encrypted, not merely authenticated, under a key derived from the network key. Authenticating alone would leave the sender's device public key in cleartext on every packet — a stable 32-byte identifier correlating a device across every network it joins. Every packet is the same size regardless of type, so an observer cannot tell a ping from a pong by length.
The content topic is derived from the network key and a time epoch, so an observer without the key cannot find the mesh's traffic, and linkability is bounded to one epoch. The trick that makes rotation cheap is that autosharding hashes only the application and version fields of the content topic:
shard = sha256(application ‖ version) mod numShardsInCluster
So rotating the {name} field changes the content topic while
leaving the shard, and therefore the underlying gossipsub topic, fixed.
Peers stay in one mesh and no subscribe/unsubscribe traffic announces the
rotation to neighbours. Rotating the pubsub topic instead would emit visible
subscription churn and drop the mesh into a fresh anonymity set of one
(ADR-006).
Rendezvous is used, not depended on. Once tunnels exist they sustain themselves, because WireGuard relearns a peer's endpoint from any correctly authenticated packet. The announce loop exists to bootstrap, to repair after a move, and to carry membership changes (ADR-003).
Every control message is signed by the publishing device and encrypted under a per-epoch key derived from the network key. The signature lives inside the ciphertext: the relay layer uses StrictNoSign at libp2p to preserve what weak sender anonymity exists, and signing outside would undo that. Messages are padded to a fixed size, so "device came online" and "device changed IP" are indistinguishable from a steady-state heartbeat on the wire.
An announce is 512 or 1024 bytes every 45 s — the larger size once a mesh uses credentials, which do not fit beside the endpoints in the smaller one. A relay is not a separate message type: it is a flag on the ordinary announce, so relays inherit endpoint validation and path probing unchanged and there are no relay addresses to distribute (ADR-014).
There is no IPAM, no DHCP and no allocator, because there is nothing to allocate. Both halves of the address are a hash:
prefix = fd || SHA256("mesh/v1/ula" || NK)[0:5] → a /48
addr = prefix : SHA256("mesh/v1/addr" || device_pub)[0:10] → /128
The prefix comes from the network key, the host bits from the device key.
Every node computes every other node's address locally from a public key it
already holds, which also makes WireGuard's AllowedIPs
self-enforcing with zero shared state. No allocation messages, no conflict
resolution, no leases, no split-brain when two nodes join at once. Collision
probability across ten devices is about 4×10⁻²³
(ADR-005).
Eighty host bits are ample because the address is not an
authenticator. cjdns and Yggdrasil commit 113–120 bits because
their addresses are the security boundary; here authentication
comes from WireGuard's handshake and the admin-signed credential, and the
address is only a stable, collision-free name. The prefix is inside
fd00::/8 per RFC 4193 — deliberately not fc00::/8,
which cjdns squats, nor 0200::/7, which is Yggdrasil's.
What it costs. An address is a function of identity, so
it changes when identity changes. Rotating the network key re-addresses
every node; losing /var/lib/shrooms gives that node a new
address and makes it a stranger to every peer. There is also no
human-friendly addressing — names come from the announce and are advisory —
and anyone holding a device's public key can compute its address. That last
one is not a leak, since the address was never secret, but it is worth
knowing.
Because the host bits are a hash of the device key, reusing one identity across two meshes would carry the same 80-bit suffix into both and let anyone in either correlate you. That is why each mesh gets its own identity rather than sharing one (ADR-015).
The overlay is IPv6-only, because derived addressing needs 128 bits. That
breaks browsers on v4-only networks: Chromium stops sending AAAA queries
when its IPv6 probe fails, asks only for A records, gets a correct empty
answer and gives up. So internal/v4 gives each peer a synthetic
IPv4 alias and translates at the tun. The alias never leaves the machine, is
not announced, and two devices may pick different aliases for the same peer
without anything noticing — which is what keeps it from needing the
coordination this project exists to avoid
(ADR-021).
One binary, cmd/shrooms, with subcommands: daemon, CLI and
relay mode. Everything below it is in internal/, and everything
in internal/ is platform-neutral — no netlink, no
/proc, no systemd assumptions outside cmd/ and
packaging/. That rule is why the Android port was cheap: the
app runs the same core through gomobile rather than a reimplementation
(ADR-016).
| Package | What it is |
|---|---|
mesh | Wires Delivery discovery to the WireGuard data plane. The announce loop that bootstraps, repairs after a move, and carries membership changes. |
wg | Embeds wireguard-go and shares its UDP socket with the control protocol, filtering in place so batching and GSO/GRO survive. |
disco | Socket-level discovery: small encrypted probes over the same socket as WireGuard, for liveness and for reflexive address discovery. |
control | The control-plane message types: signed inside the ciphertext, encrypted per epoch, padded to a fixed size. |
topic | Derives the rotating rendezvous content topics, and computes the shard they land on independently of the library. |
waku | The cgo binding over liblogosdelivery. Targets the packaged build, whose API differs from upstream master in ways that break compilation rather than behaviour. |
rendezvous | Adapts a Delivery node to the interfaces the protocol packages define, so invite can stay free of cgo and stay testable without a node. |
identity | Device keys, the network key, and the derived addressing scheme. Two keypairs per device: Ed25519 for signing and addressing, X25519 for the tunnel. |
cred | Separates authority from participation: admin keys, credentials and the checks over them. No wire format and no I/O — it changes nothing that runs until something calls it. |
invite | One-time enrolment tokens. Crypto and wire format only: no Delivery, no files, no clock beyond what it is handed. |
relay | A DERP-style packet reflector keyed by WireGuard public key. It never terminates a tunnel, holds no session key, and cannot read what it forwards. |
portmap | Asks the local router for an inbound UDP mapping. PCP first, NAT-PMP as fallback, both on UDP 5351; UPnP-IGD deliberately not implemented. |
dns | Serves names for the mesh and nothing else. Authoritative for one suffix, and the rule that matters is what it refuses to do. |
hosts | Renders and applies /etc/hosts entries. The zero-dependency fallback, deliberately not the long-term answer. |
service | Publishes local ports under their own names, forwarding from the overlay address to loopback. A forwarder, not a proxy: bytes are copied, TLS is not terminated. |
listeners | Finds what is already bound to this device's mesh addresses, so it can be announced. Only sockets on exactly a mesh address count. |
v4 | A synthetic IPv4 address per peer, translated before WireGuard sees it. Purely local; no peer has to agree to it. |
state | On-disk config and device state, separated by who writes them: config.toml is human-edited, state.json is daemon-owned. |
This is where the project's reasoning lives. The code says what happens; these say why, and what would change our mind. Each is one page — context, decision, consequences — and most rest on research that would otherwise be lost: measured throughput figures, NAT prevalence data, source-level findings in nwaku. Where a number appears, its source is named.
If you are going to read one thing before touching the code, read these. Several of them exist specifically to record a decision not to build something, which is the part of a design that is otherwise invisible.
Alongside them: DESIGN.md for the architecture and the research behind each decision, PROTOTYPE.md for the build plan and what each milestone proved, SECURITY.md for what is protected, what leaks and what is deferred, and TESTING.md for the scenarios containers cannot honestly reproduce.
Go 1.23 or later with cgo and a C toolchain. The core links liblogosdelivery, for which there is no canonical distribution, so the first step is to get a copy:
git clone https://github.com/vpavlin/shrooms && cd shrooms
make deps-release # download a prebuilt copy — works anywhere
make shrooms # -> bin/shrooms
make deps-basecamp reuses the copy Logos Basecamp installs, and
make LD_DIR=/path/to/lib points at your own build.
make deps builds it from source in a pinned container and
currently fails, upstream, at both master and the revision
the Android bindings pin — which is why deps-release exists at
all. Every target that links the library runs check-lib first
and tells you to run deps-release rather than failing at the
linker.
| Target | What it does, and why it is shaped that way |
|---|---|
make shrooms | Builds the single binary — daemon, CLI and relay mode — into bin/, with the version stamped from git describe. |
make test | The whole suite with -race, about twenty seconds. The race detector is not optional: the failure that motivated it was a counter incremented on the dispatch path while a read lock was held, which CI caught and a plain non-race run had been passing over for a day. |
make test-unit | The packages that do not link liblogosdelivery, also with -race, about five seconds. This is what CI runs on every push. CGO_LDFLAGS is cleared deliberately, because inheriting -llogosdelivery would fail the link when the library is absent — which is exactly CI's situation before the build job runs. |
make build-all | Builds every package, including the nested mobile module. test-unit skips the cgo-bound packages, so without this an API change can break a command with nothing noticing — which is what happened to cmd/m0demo when the control-plane signature changed. |
make vet-cgo | Vets the packages that link the library, which the no-cgo vet in CI cannot reach. Without it internal/mesh is never vetted at all. |
make image | Builds the container image, which is how this gets deployed: liblogosdelivery needs glibc 2.38, so a bare binary fails on Debian 12 and works only on Ubuntu 24.04 and later. Publish once from a machine that has the library; every other machine pulls. |
make apk | Builds the Android app, via make aar. The gomobile binding is built in a container because gomobile needs a JDK and Go 1.25 or later, which the core deliberately does not. |
make basecamp-check | Loads the real Basecamp view offscreen against a fixture and asserts it reads a status snapshot. There is no display on a build machine and the QML runtime does not need one — and "the QML looks right" is not a test. |
make basecamp-lgx | Builds the Basecamp module as a portable LGX, the installable package format. Portable rather than plain, because the plain output can reference paths in the nix store of the machine that built it: fine for a dev loop, useless as a download. Needs nix with flakes; CI builds it on every push. |
The milestone and spike targets reproduce the results below rather than
describing them. make s1 does a publish-and-receive round trip
over the real fleet — it is also the fastest way to tell whether the
rendezvous plane is the problem. make s3 checks that six
rotated topics land on one shard. make m0 needs no root at all,
since it uses a netstack TUN. make m1, make m2 and
make m3 run containerised nodes and need docker and
/dev/net/tun. make m3-remote HOST=user@vps runs the
relay test over the real internet instead.
Containers hide anything timing-dependent, repeatedly and expensively. Two bugs found only on real infrastructure: peers recreated on every sync, where a handshake arriving mid-recreation always won the race at 1 ms container RTT and always lost at 100–450 ms; and paths re-probed only after they had already expired, invisible when a round trip costs nothing. Both were sitting inside a passing test. Prefer hardware for anything timing-, NAT- or loss-dependent.
The distinction that matters is between code that exists and behaviour that has been observed. This is the same table the README keeps, and it is kept honest on purpose.
| State | |
|---|---|
| S1 cgo binding to Logos Delivery | publish → receive over the real fleet |
| S3 rotating topics stay on one shard | 6 epochs, all to /waku/2/rs/2/3 |
| M0 WireGuard sharing a socket with control traffic | tunnel and control packets on one socket, no root |
| M1 discovered peers replace static config | over the real internet: NATed laptop ↔ VPS, direct tunnel, ssh across it |
| M2 NAT traversal | reflexive discovery proven on real NAT; punching between two NATed nodes still unproven |
| M3 relay fallback | carrying real traffic: a phone on carrier NAT reached its peers through a VPS relay it discovered itself |
| M4 seamless operation | roaming survives and the mesh repairs itself after a rendezvous outage; switching networks is still rough on the phone |
| M5 credentials | issued, carried, verified, revoked and renewed by a sweep — though no credential has yet been watched roll over on live devices |
| M6 name resolution | resolver registered with the host, on Linux and Android, including <service>.<device>.mesh |
| services | local ports and LAN devices published by name, including things that never joined the mesh |
| Android | a full participant, in daily use — tunnels, names, roaming between wifi and mobile data |
| multi-mesh | several meshes in one daemon and one app, each with its own identity, addresses, relay and services |
| invites | one device at a time, fifteen minutes, no key pasted |
| Basecamp module | published: reads the mesh and changes what the daemon can change alone — settings, services, meshes, joining, leaving, restarting — through shrooms_core, which reaches the control socket from outside the QML sandbox. Never admission, which needs the admin key |
| S2 fleet Store retention | never run. Android shipped without depending on it, so it left the critical path unmeasured |
| S4 WireGuard throughput baseline | never run, minor |
Relaying works on real infrastructure, as of 2026-08-11. A
phone on mobile data reached a peer through a relay on a public VPS, over
the real internet, with the endpoint showing as
relay:<key>@<vps>:51820. Relays are discovered from
the roster, so nothing is told where one is. Relaying is per mesh: a mesh
joined by invite has no relay until a member is told to be one.
A NATed node can now be reached from outside without a relay, as of 2026-08-12 — by asking the router rather than by punching. A laptop behind a domestic NAT was granted a mapping over NAT-PMP, announced it, and a phone on mobile data dialled it directly, on a mesh with no publicly reachable member. Whether your router answers is between you and your router; when it does not, nothing is worse than before.
Punching between two NATed peers is still unproven. There is a plan for it now — ADR-027, a simultaneous open coordinated over the relay the pair already use, with every part in place except the coordination. It is a different mechanism from the above. Three reasons it is hard, all learned rather than assumed:
advertise has to be set by hand.
The test rig is part of the problem. Plain Linux MASQUERADE is
not endpoint-independent — measured here, it allocates a different external
port per destination, so a harness built on it tests the hard case while
claiming to test the easy one. A synthetic NAT built to be cooperative
proves only that the code works against a NAT we made cooperative.
The rendezvous plane can fail while everything looks fine. Tunnels keep carrying traffic, the roster stops changing, and peers age out one at a time until the status page is a list of healthy devices marked offline. Both the daemon and the app now watch for it. Two watchdogs: nothing arriving for ten minutes, and — the case a "nothing has arrived" check sails straight past — deaf for twelve minutes to a specific peer whose WireGuard tunnel is still rekeying, which proves that peer's daemon is running and therefore announcing. Recovery is a restart in both places, because the delivery library keeps process-global state and has never survived being restarted inside a live process.
Renewal has now been run against a real mesh. On 2026-08-13 a sweep reissued credentials on a three-member mesh: the admin key signed, the daemon published, and a remote device — a phone, untouched — verified against the same admin keys and stored its own, going from 28 days left to 30. The half that had never been exercised was that one.
The first run renewed nothing, correctly: every member had 28 days and the window opens at 10. The one member that did not renew was off the network entirely — renewal travels over the rendezvous plane, so a device whose tunnel is healthy and whose Delivery connection is not cannot be renewed.
A fleet migration broke everything once, silently. On
2026-08-07 logos.dev moved to cluster 3 while the preset compiled into the
pinned liblogosdelivery still said cluster 2. Every peer connected, compared
metadata, disagreed and hung up — which looks exactly like an outage. The
default is now logos.test, whose preset is correct, and
cluster_id exists as an override for the next time a fleet
moves ahead of the library. make s1 detects this.
There is no contribution ceremony. There are three habits the codebase keeps, and keeping them is most of what a review will ask for.
Not what the code does — the code says that. Why it is shaped this
way, and the specific failure that made it so. The Makefile's own
comments are the model: -race is explained by the counter
it caught, build-all by the command that broke unnoticed.
One page: context, decision, consequences, and what would change our mind. Where a number appears, name its source. A decision not to build something counts, and several of the existing records are exactly that.
TestMeshNamesAreNeverForwarded,
TestRoutesBySNIWithoutTouchingTheBytes,
TestPortAlreadyHeldIsNotAnError. A failing test should say
what broke without anyone opening the file.
Two practical constraints. Everything in internal/ must be
platform-neutral, because Android runs the same code and a
netlink call there is a port rather than a patch. And control
messages must stay small and idempotent, because the Android path will
deliver them late, out of order, and sometimes not at all.
Run make test before pushing, and make build-all
if you have touched anything a command uses — CI's unit run cannot reach the
cgo-bound packages. If a change is timing-, NAT- or loss-dependent, get it
onto real hardware; the container harness has hidden that class of bug twice
and both times it looked like a passing test.