COMPAX,
DOCUMENTED

GETTING STARTED

Start here

• Redis
• Telegram Bot API
• Rust

What runs is three processes and two stores. The delivery service orders messages and hands them out; it keeps rooms and their logs in Postgres and shares presence, rate buckets and its live channels through Redis, so more than one instance of it is a deployment decision rather than a rewrite. Beside it the Bot API gateway speaks Telegram’s surface to whatever bots you run.

The encryption is MLS, RFC 9420 — a standard rather than something of ours, with forward secrecy and post-compromise security, and it is what makes a group conversation cost one message rather than one per member. The service never holds a key that opens anything: it orders ciphertext, and the order is all it contributes. Both it and the gateway are Rust; the app is Flutter, one codebase for the phones, the desktops and the web.

A deployment is a container each and a Postgres and a Redis to point them at — the compose file is in the repository and builds the image itself, along with the migrations the service runs against its own database on start.

On its first run the gateway mints @botfather and prints its identity to the log. That account is where bots come from, and it is an ordinary bot on the same surface as yours — it sends messages, hangs keyboards, answers taps and edits its own messages through the same code path. Whatever is broken about buttons is broken for it too, which is the point of building it that way.

BOTS

Bots

• Telegram Bot API
• Python
• Node.js
• Rust
• MLS

The strong reading of “Telegram compatible” is the only one that is testable: a bot written with python-telegram-bot, aiogram, telegraf or grammY runs against Compax with the base URL changed and nothing else. That is what the gateway is built to be — not a lookalike of Telegram’s buttons, but the wire protocol those libraries already speak.

A bot is a member of the group

On Telegram a bot is an account on Telegram’s servers, and the Bot API is a view onto a database somebody else owns. Here the server holds ciphertext and an epoch counter; there is no plaintext anywhere but on a member’s device. So a bot has its own MLS device and its own identity key, and it joins the group as a member . There is no other place plaintext exists for it to read.

Three consequences, all of them visible in the app rather than hidden by it:

• Adding a bot is adding a member. It shows up in the member list, because MLS owns the roster and the app asks rather than keeping its own copy.
• A bot reads every message in the group, always. Telegram has privacy mode; we cannot implement it, because the bot holds the group’s keys and has decrypted the message before any rule could apply. getMe reports can_read_all_group_messages: true and that is a fact about the architecture, not a setting.
• Removing a bot is a real MLS removal, which rotates keys forward. “Kicked but still holding the keys” is not a state that gets presented as removed.

Making a bot

Find @botfather in the app’s chat search and say /newbot. It asks for a name, a username ending in bot — matching Telegram’s own rule, so bot code doing username.endswith("bot") is right — and then for the bot’s key, which your app derives from your own recovery phrase. The gateway never sees that phrase and cannot make the key itself. It starts the bot’s device and tells you what the bot is:

@orders_bot is 9f2c4d19b7e05a3c9d61f8a24b7e0c93d15641ac8e37b2f0a94c6d1e5b83702f.
└─ the bot's public key — thirty-two bytes, hex

That is a public key, not a secret. It names the bot in a URL and proves nothing on its own, so a paste of that line, a proxy log or a browser history hands nobody anything. What proves a request is a signature, made with a device your own application issues for the backend — see pointing a backend at it.

Registering is the same rule one step earlier. The device your application sends carries an attestation: the bot’s identity key signing this device may speak for me. @botfather verifies that signature before it registers anything, so claiming a key you do not hold fails here rather than later — you would have to forge a signature by the identity you are claiming to be. The record’s own device key is checked against the attested one too, which is what makes the two halves one device rather than two.

There is no token here and nothing that stands in for one. The shape this replaced was Telegram’s — 146492815:AAHq…, an id and thirty-five characters — and it was a password in a URL path: whoever held it could speak as the bot in every group the bot was in, and it travelled wherever a backend’s configuration travelled. A signature cannot be replayed, cannot be read out of a path, and can be taken back.

• /newbot, in the app @botfather refuses to do this in a group and says why. Nothing it prints is a secret any more, but making a bot is still an act with an owner, and a group log is a poor place to settle who that is.
• Your app makes the key, not the gateway A bot is derived from your recovery phrase at a random index, so the gateway holds a public key and a number — neither of which reproduces the bot. It used to generate a phrase per bot and keep it in the clear, which meant whoever read that file was every bot in it, and losing the file lost them for good. Now the phrase you already have brings any of them back.
• Add the bot to a conversation Creating a bot is not adding it to anything. Joining a group is an MLS commit, and only a member can make one — so a member adds it, from the group’s member list. Nothing can let itself in.
• Issue a device, and run your backend against that Below. Your application exports a device for the backend; the bot’s identity stays in the application, which is what makes a leaked backend key a revocation rather than a funeral. /mybots stops this gateway serving with a device, and your app issues another at a raised version — the bot stays the same member of the same chats. /delete takes the bot off this gateway — its record, its device, its name — and its leaf stays in every group it was in, silent, until somebody removes it there. What /delete no longer does is end the bot: it is derived from your phrase, so the same key comes back at the same index. That used to be a one-way door, because the only copy of a bot’s identity was the one on the gateway’s disk.

The gateway’s own registry lists every bot it knows: the public key, and the device it is being served with. Both are public, so reading it is a status check rather than a way to get at anything — which is new. That file used to hold a working credential for every bot on the machine, and twenty-four words that reproduced each of them.

Pointing a backend at it

Every method lives at http://127.0.0.1:8081/bot<key>/<method>, where <key> is the bot’s public key rather than a credential. GET and POST both work, parameters arrive as JSON, form-encoded or query string, and the answer is the usual {"ok": true, "result": …} envelope.

What a request must also carry is a signature: an x-chat-relay header covering the method, the path and query, the body, a timestamp and a nonce, made with the device key your app exported. It is the same envelope this deployment’s servers use to talk to each other, so there is one construction to get right rather than two.

That is the one place a Telegram bot library needs help: it will build the path and the body correctly and will not sign anything. A few lines of middleware in whatever HTTP client it uses is the whole adaptation — and in exchange, nothing your bot holds can be read out of a URL, a proxy log or a paste.

Two things, and only one of them is a secret

A backend is given one file — what your application exported — and everything comes out of it:

• bot — the public key. It goes in the path, and that is all it does. Nothing is ever signed with it: a public key cannot sign, which is what makes it safe to put in a URL.
• record.signing_seed — the device’s private key, thirty-two bytes. Every request is signed with this, and it never leaves the backend. This is the secret; the file holding it is the thing to keep.
• record.attestation — the bot’s identity key saying this device may speak for me. That is what the gateway checked when the device was installed.

{
  "bot": "9f2c4d19…",         // public — this goes in the path
  "base_url": "http://127.0.0.1:8081",
  "record": {
    "device_id": "4b1f…",
    "signing_seed": "a3f0…",  // PRIVATE — this signs every request
    "attestation": { …the identity's signature over this device… }
  }
}

# Ed25519: the seed is the private key, and the key in the path is its
# public half. The gateway holds only the public half and checks the
# signature against it — so possession of the private key is the proof,
# and it is the one thing that never travels.

Signing a request, in full

Twenty lines and no dependencies. This is the whole of authorising: lay out the bytes, sign them, put the result in a header.

Why the three prefixes. Without them a longer path and a shorter body lay out identically to a shorter path and a longer one — so one signed request could be read as a different request, with the signature still checking out. The length in front of each variable field is what makes the concatenation readable in exactly one way.

• The body is the exact bytes you will send. Not a hash of them, and not a re-serialisation — a JSON library that reorders a key or changes a space produces a signature over something other than what arrives.
• The nonce is what stops a replay. The signature proves who wrote the request and nothing about when; the nonce is single-use and issued_at is checked against a skew window of a minute, so the same signed bytes cannot be presented twice.
• The literal is signed first, on purpose. A signature made for this envelope cannot be replayed as a signature for another protocol that happens to lay its fields out the same way.

import { createPrivateKey, sign, randomBytes } from "node:crypto";

// An Ed25519 seed is a raw 32 bytes; node wants PKCS#8, which is that
// seed behind a sixteen-byte header that never changes.
const PKCS8 = Buffer.from("302e020100300506032b657004220420", "hex");
const keyFromSeed = (seed) =>
  createPrivateKey({ key: Buffer.concat([PKCS8, seed]), format: "der", type: "pkcs8" });

const u32 = (n) => { const b = Buffer.alloc(4); b.writeUInt32BE(n); return b; };
const u64 = (n) => { const b = Buffer.alloc(8); b.writeBigUInt64BE(BigInt(n)); return b; };

function relayHeader({ seed, publicKey, home, method, path, body }) {
  const issuedAt = Math.floor(Date.now() / 1000);
  const nonce = randomBytes(16);

  const signed = Buffer.concat([
    Buffer.from("chat-relay-envelope-v1"),
    publicKey, Buffer.from(home, "hex"),
    u32(Buffer.byteLength(method)), Buffer.from(method),
    u32(Buffer.byteLength(path)), Buffer.from(path),
    u32(body.length), body,
    u64(issuedAt), nonce,
  ]);

  const signature = sign(null, signed, keyFromSeed(seed));  // the private key

  return Buffer.from(JSON.stringify({
    device_pubkey: [...publicKey],                          // an array of numbers, not hex
    home, method, path,
    issued_at: issuedAt,
    nonce: [...nonce],                                      // likewise
    signature: signature.toString("hex"),                   // this one is hex
  })).toString("base64url");                                // no padding
}

import base64, json, os, struct, time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey

def relay_header(seed, public_key, home, method, path, body):
    issued_at, nonce = int(time.time()), os.urandom(16)
    signed = (b"chat-relay-envelope-v1" + public_key + bytes.fromhex(home)
        + struct.pack(">I", len(method)) + method
        + struct.pack(">I", len(path)) + path
        + struct.pack(">I", len(body)) + body
        + struct.pack(">Q", issued_at) + nonce)

    signature = Ed25519PrivateKey.from_private_bytes(seed).sign(signed)

    envelope = json.dumps({
        "device_pubkey": list(public_key), "home": home,
        "method": method.decode(), "path": path.decode(),
        "issued_at": issued_at, "nonce": list(nonce),
        "signature": signature.hex(),
    }).encode()
    return base64.urlsafe_b64encode(envelope).rstrip(b"=").decode()

And used — the body is rendered once and then both signed and sent. Rendering it twice is how a signature comes to cover something the request does not contain, and the refusal that follows names the signature rather than the line that caused it.

const export_ = JSON.parse(fs.readFileSync(process.env.BOT_DEVICE, "utf8"));
const seed = Buffer.from(export_.record.signing_seed, "hex");
const publicKey = createPublicKey(keyFromSeed(seed))
  .export({ type: "spki", format: "der" }).subarray(12);
const home = export_.record.attestation.attestation.home;

const path = `/bot${export_.bot}/sendMessage`;
const body = Buffer.from(JSON.stringify({ chat_id: -1001, text: "Hello." }));

await fetch(export_.base_url + path, {
  method: "POST",
  headers: {
    "x-chat-relay": relayHeader({ seed, publicKey, home, method: "POST", path, body }),
    "content-type": "application/json",
  },
  body,
});

export = json.load(open(os.environ["BOT_DEVICE"]))
seed = bytes.fromhex(export["record"]["signing_seed"])
public_key = Ed25519PrivateKey.from_private_bytes(seed).public_key().public_bytes(
    serialization.Encoding.Raw, serialization.PublicFormat.Raw)
home = export["record"]["attestation"]["attestation"]["home"]

path = f"/bot{export['bot']}/sendMessage"
body = json.dumps({"chat_id": -1001, "text": "Hello."}).encode()

requests.post(
    export["base_url"] + path,
    headers={
        "x-chat-relay": relay_header(seed, public_key, home, "POST", path, body),
        "content-type": "application/json",
    },
    data=body,                              # the same bytes that were signed
)

from telegram.ext import ApplicationBuilder, CommandHandler
from nacl.signing import SigningKey                                   # any Ed25519 will do

export = json.load(open(os.environ["BOT_DEVICE"]))
device = SigningKey(bytes.fromhex(export["record"]["signing_seed"]))  # the private key

app = (
    ApplicationBuilder()
    .token(export["bot"])             # the PUBLIC key — it only
                                      # builds the path
    .base_url("http://127.0.0.1:8081/bot")
    .request(SigningRequest(device))  # the private key signs here
    .build()
)

async def start(update, ctx):
    await update.message.reply_text("Hello from Compax.")

app.add_handler(CommandHandler("start", start))
app.run_polling()

from aiogram import Bot, Dispatcher
from aiogram.client.session.aiohttp import AiohttpSession
from aiogram.client.telegram import TelegramAPIServer

session = AiohttpSession(
    api=TelegramAPIServer.from_base("http://127.0.0.1:8081")
)
# export["bot"] is public and addresses the bot; the seal below is what
# proves anything, and it is made with export["record"]["signing_seed"].
bot = Bot(token=export["bot"], session=session)
# session.middleware(…) is where the x-chat-relay header goes on
dp = Dispatcher()

// The public key goes where the token went — it addresses, it does not
// authenticate. signing() below holds record.signing_seed and seals each
// request with it.
const bot = new Bot(export.bot, {
  client: {
    apiRoot: "http://127.0.0.1:8081",
    fetch: signing(device),   // adds x-chat-relay to each request
  },
});

// telegraf
const bot = new Telegraf(export.bot, {
  telegram: { apiRoot: "http://127.0.0.1:8081", agent: signing(device) },
});

BASE=http://127.0.0.1:8081/bot$BOT_KEY

# Every request needs an Ed25519 signature over the method, the path,
# the body, a timestamp and a nonce — so curl on its own cannot make
# one, and an unsigned request is answered 404 like any other stranger.
# $SIGN below stands for whatever produces that header for you.

curl -s -X POST $BASE/sendMessage \
  -H "x-chat-relay: $(SIGN POST /bot$BOT_KEY/sendMessage "$BODY")" \
  -H 'content-type: application/json' \
  -d "$BODY"

# the header is base64url of this, signed with the exported device key:
# {"device_pubkey":…,"home":…,"method":"POST","path":"/bot…/sendMessage",
#  "issued_at":1731000000,"nonce":…,"signature":…}

// crates/botapi/examples/demo_bot.rs — reqwest and serde_json
// against a base URL, plus the one helper that signs.

// What your application exported: which bot this is, where it lives,
// and the device that speaks for it. No identity key is in the file.
let export: signing::Export =
    serde_json::from_slice(&std::fs::read(std::env::var("BOT_DEVICE")?)?)?;

// Rendered once, then both signed and sent. Rendering twice is how a
// signature ends up covering something the request does not contain.
let path = format!("/bot{}/{method}", hex::encode(export.bot));
let body = serde_json::to_vec(&params)?;
let signed = signing::header(&device, home, "POST", &path, &body);

let answered: Value = http
    .post(format!("{base}{path}"))
    .header(protocol::relay::RELAY_HEADER, signed)
    .header(CONTENT_TYPE, "application/json")
    .body(body)
    .send().await?.json().await?;

The update loop

Poll with getUpdates. An update is confirmed by the next call’s offset, the way Telegram does it: limit defaults to 100 and is clamped there, timeout is a long poll of up to 300 seconds, and allowed_updates filters.

{
  "update_id": 41,
  "message": {
    "message_id": 17,                  // the group log's own seq
    "date": 1755993600,
    "chat": { "id": -1001, "type": "group", "title": "Ridge Trip" },
    "from": { "id": 5501, "is_bot": false, "first_name": "Ada" },
    "text": "/ask something",
    "entities": [{ "type": "bot_command", "offset": 0, "length": 4 }]
  }
}

Those entities are not decoration. CommandHandler in python-telegram-bot and CommandStart in aiogram never look at the text — they find a bot_command entity at offset 0 and take the command from the span. The gateway computes them in UTF-16 code units, because that is the unit every library slices with, and one emoji before a command shifts everything after it by two.

Identifiers

Every Bot API library types chat.id and from.id as 64-bit integers, not opaque strings, and real bot code branches on the sign. Our identifiers are a 32-byte identity hex and a UUID, so the gateway keeps a persistent registry that mints them once and never reuses one:

Keyboards and callbacks

An inline keyboard is Telegram’s own JSON, passed through verbatim rather than re-modelled — a round trip that loses a field is a conformance bug we want a test to see, and a second schema hides those by construction. Hang it on a message, and a tap comes back as a callback_query.

POST /bot$KEY/sendMessage
{
  "chat_id": -1001,
  "text": "Flat white or latte?",
  "reply_markup": { "inline_keyboard": [[
      { "text": "Flat white", "callback_data": "cup:flat" },
      { "text": "Latte", "callback_data": "cup:latte" }
  ]] }
}

Somebody presses one. Nothing on their screen changes — the app has sent the string and is waiting, which is the one thing this exchange is about. What the bot receives is not a request it made:

{ "update_id": 42, "callback_query": {
    "id": "cb-7", "data": "cup:latte",
    "from": { "id": 5501, "first_name": "Ada" },
    "message": { "message_id": 18, "chat": { "id": -1001 } } } }

Two calls answer it. The first stops the spinner and is the only thing that can; the second edits the message that is already in the conversation, so the keyboard cannot be pressed a second time.

POST /bot$KEY/answerCallbackQuery
{ "callback_query_id": "cb-7", "text": "Ordering…", "show_alert": false }

POST /bot$KEY/editMessageText
{ "chat_id": -1001, "message_id": 18, "text": "Latte — ready in about four minutes." }

Supported on an InlineKeyboardButton today: text, callback_data, url and copy_text. web_app, switch_inline_query* and login_url are later stages; pay and callback_game are refused — see below. Reply keyboards, ForceReply and ReplyKeyboardRemove are drawn by the app; the gateway does not yet turn a keyboard tap or a picked poll back into an update.

Native widgets

A keyboard is rows of buttons, so every question that is not which of these buttons becomes a conversation: one message per value, and no way at all to collect a set, a quantity or a day in place. A widget is the answer to that: a card the app draws from JSON the bot sends, in the same message, with the same callback channel underneath.

Every control there is, what each one sends back, and a constructor that writes the JSON and draws the card as you turn its knobs, are on a page of their own. They were four hundred lines in the middle of this one, which is a reference section standing where a tutorial should be.

The API, method by method

Generated from nothing — this is read off crates/botapi/src/api.rs, and a method not named here answers 404.

Where this is not Telegram

Four things are not “not yet”. They stand on a platform that does not exist here, and a well-formed error is the honest answer rather than a half-truth:

• Payments. Telegram’s model is a provider token and Telegram taking custody. Mapping sendInvoice onto our invoices would hand bots a payment they cannot reconcile. Refused, with a native extension offered instead.
• login_url. It hands a website a Telegram user id signed by Telegram. There is no Telegram user id here; identity is a key pair, which is what the dApp browser already signs logins with. The native mapping is strictly better and the Telegram-shaped one does not exist.
• Channels, supergroups, admin rights, join requests. MLS groups are flat. chat.type is private or group, and the chat-administration methods answer 400.
• Stickers, games, business accounts, Premium, boosts, gifts. No counterpart and no plan.

And two that are degraded rather than absent: request_contact returns a Contact with a user_id and names but an empty phone_number, because identity here is a key pair and there are no phone numbers to share; and privacy mode is permanently off, for the reason in the first section.

Before you run it for real

• Poll from one process. Two pollers for one bot steal each other’s updates. The gateway does not stop you; the symptom is a bot that answers every second message.
• Advance offset before you handle , not after, so a handler that panics does not put the bot in a loop answering the same message forever.
• Treat the exported device as the secret. The key in the URL is public and safe to log; the file your application exported is not. Keep it out of images and CI variables — and if it does get out, /mybots takes it back, which is the whole reason the backend holds a device and not the bot.
• Say what the bot is for when it joins. It reads every message in the group from the moment it is added; the app shows that, and a bot that explains itself first is the difference between a member and a surprise.
• Re-check every event on arrival. A spec that hides a button is a drawing, not an enforcement.

SELF-HOSTING

Run your own server

• Docker
• PostgreSQL
• Redis
• Prometheus
• Grafana

Everything above talks to a server. It can be ours or yours, and nothing in a bot changes between the two — the gateway is the same binary either way. The delivery service, the client SDK and the wire formats are one repository, askucher/compax‑core, and the compose file beside them raises everything it needs: Postgres, Redis, blob storage, two instances and a proxy in front of them.

git clone https://github.com/askucher/compax-core
cd compax-core/deploy

# placeholders that start; each line says why it is not one to deploy
cp .env.example .env

# the first build compiles the workspace, so it is not quick
docker compose up -d --build
curl localhost:8080/health

One image holds both binaries — the delivery service and the Bot API gateway — built from one workspace and one lockfile, so the schema, the code applying it and the thing talking to it cannot disagree about which of them is older. Migrations run on boot: the first up is also the install.

• Two instances, by default. Everything correctness‑bearing lives in Postgres, so they are interchangeable and the proxy needs no sticky routing. One instance is the case that works by accident.
• The proxy speaks plaintext until you give it an address. Which is right behind a load balancer that terminates TLS and wrong as a public edge — SITE_ADDRESS in .env is the whole of turning that around, certificates included, and the section below is what else that takes.
• The placeholders are placeholders. Compose refuses to start without a Postgres password, a metrics token and the SFU credentials, and .env.example ships values for all of them so the stack comes up before you have real ones. They are not values to deploy, and the file says so beside each.

On another server

The same commands on a host that runs Docker, plus a domain. The address is what turns a laptop into a deployment: set it and the proxy obtains and renews that domain’s certificate itself, leave it empty and it serves plain HTTP, which is right behind a load balancer that terminates TLS and wrong as a public edge.

ssh you@your-server
git clone https://github.com/askucher/compax-core
cd compax-core/deploy
cp .env.example .env

# in .env -- the A record must already point at this host
#   SITE_ADDRESS=chat.example.com
#   HTTP_PORT=80        the certificate challenge is answered here
#   HTTPS_PORT=443
# and real values for the eight the file marks as required

docker compose up -d --build
docker compose logs -f server-a server-b

Point clients at https://chat.example.com. The certificate is asked for when the proxy starts rather than on the first request, so the domain has to resolve here before the first up; until it does, the site answers on plain HTTP and the challenge is retried in the background.

• Two ports, and two more only for calls. 80 and 443 are the deployment — the instances themselves publish on loopback and are reached through the proxy. --profile calls adds 7880 and 7881 with 50000–50100/udp for the media server, and 3478 with 49160–49200/udp for the relay. Set TURN_EXTERNAL_IP to the host’s public address when you raise it: left empty the relay hands out a private one, which answers and never relays.
• Prometheus and Grafana publish on every interface, unlike everything else here. Behind a firewall or over an ssh -L tunnel they are what they were meant to be; open to the internet, --profile observability is an admin login on port 3000.
• Updating is git pull and the same up. Migrations run on boot, so there is no second step and no window where the schema and the code disagree.
• Back up the db volume and botstate. The first is everything correctness‑bearing; the second holds the operator phrase and every bot’s keys, and losing it is losing the bots.

A second server, and how the two see each other

federationmimipeertls

Two instances of one deployment need no configuration at all: they share Postgres and Redis, and everything that decides anything is in the first of those. Two providers are the other thing entirely — separate machines, separate databases, and nothing in common but what each has been told about the other.

They do not discover each other. There is no registry, no gossip and nothing to sign up to: each side is told, in its own .env, and that is the whole mechanism.

Two homes, written out. Each file names the other and nothing about itself beyond its own label and url:

HOME_LABEL=a1b2c3
PUBLIC_URL=https://a.example.com
SERVER_KEY=1f4c…                      # openssl rand -hex 32
HOME_DIRECTORY={"homes":{"d4e5f6":{"url":"https://b.example.com","key":"9f2c…"}}}

HOME_LABEL=d4e5f6
PUBLIC_URL=https://b.example.com
SERVER_KEY=9f2c…                      # its own, and the public half is what A holds
HOME_DIRECTORY={"homes":{"a1b2c3":{"url":"https://a.example.com","key":"1f4c…"}}}

The key in each entry is the public half of the other side’s SERVER_KEY, and it is what a peer request is checked against — so a directory that lied about somebody else’s key would be a directory whose requests fail a signature check rather than one that succeeds. What each deployment holds is published at /.well-known/chat-homes, so the other side can fetch it rather than be told it twice.

Then raise it with the profile that writes the key: docker compose --profile federation up -d. Everything above is empty by default, and empty is a deployment that is alone — which holds every group it is asked about anyway.

Scaling it to millions

postgresrediss3proxy

One rule makes the rest of this section short: nothing decisive lives in a process. Sessions are rows, routing is Redis, attachments are wherever both instances agreed to put them — so a second instance is a deployment decision and a twentieth is the same decision again. What follows is the order in which things stop being enough, because that order is not obvious and every one of these steps is wasted if it is taken first.

The backend, which is the easy half

Every instance is interchangeable and none of them is anybody's home: a client may open its socket on any one, and the fanout finds it through the registry rather than by knowing where it went. Two numbers decide how many you can run.

• Sixteen connections each. That is max_connections(16) in main.rs, so ten instances is a hundred and sixty and a stock Postgres accepts a hundred. Past about five instances the answer is a pooler in transaction mode rather than a larger Postgres, because a connection costs memory whether or not it is doing anything. chat_db_pool_connections by state is the graph that says which of the two you need.
• One socket per device, not per person. A phone, a desktop and a tab are three, and each one is a file descriptor and a buffer for as long as it is open. chat_ws_sockets_open against the box's own limit is the ceiling; raise nofile before it is reached rather than after.

Attachments have to be shared before the second instance, not after: BLOB_STORE=s3 with S3_BUCKET, and S3_ENDPOINT for MinIO or R2, which speak the same protocol and differ only in that address. A service that finds a live instance keeping its bytes somewhere else refuses to start, and that is deliberate: with two disk directories every download is a coin flip on which instance serves it, and a 404 for bytes that exist is a bug nobody can reproduce.

Redis is per region, and never between two

The registry is written on every connect, and a mobile client reconnects on every change of network — so it is a write-heavy keyspace with a millisecond budget, and a region reaching across an ocean for it turns every reconnection into a round trip. Each region gets its own, and what crosses between regions is the peer lane, which is built for a link that is slow and sometimes down.

The database, in three moves and in this order

First, the partitions you already have. The log is partitioned by month and retention drops a partition rather than deleting rows, so the largest table in the service is bounded by a setting rather than by traffic — see the chart under Storage. Nothing else needs doing until that is true and the graph is flat.

Second, a replica for what may be stale. Less than it sounds: the log is read to catch up, so a replica behind by a second is a client that misses the message it was told about. What moves safely is what nobody is waiting on — the directory reads behind a profile, the public group listings, and the metrics scrapes.

Third, more than one home rather than more than one shard. This is the part where an honest section is worth more than a complete one: the data is cut along two keys already — rooms and identities — and a room fits inside one piece whole, but there is one DATABASE_URL in this code and no router in front of it. What is built is federation, and it divides the same way: a second deployment with its own label, its own database and its own Redis, ordering its own rooms and reaching the first over the peer lane. It is the same cut, made where the code already supports it.

The one thing that does not divide

A room is ordered by one server and one row: one commit per epoch, a compare-and-swap, and every commit in that room waits behind it. Millions of people in millions of rooms is a flat problem — spread them across homes and every home is busy in parallel. A million people in one room is not, and no amount of hardware changes it, because the serialisation is what MLS requires rather than what this implementation chose. That is the number to design the product around, and it is the reason a broadcast channel and a group conversation are not the same feature.

Storage

postgresredisdiskblobsbackup

Two stores, and one question decides which: would anybody want it back after a restart? The log, the directory and the inboxes would, so they are in Postgres. Who is connected, and where to push what to them, would not — that is Redis, and losing all of it costs a reconnect. There is no third store and nothing decisive is cached in a process, which is the whole reason two instances are interchangeable.

Twenty-one tables, sixteen indexes and nine foreign keys — and the arrows are the half a list cannot say: eight of the nine land on devices, every one of them on delete cascade. Deleting a device takes its sessions, its key packages, its wrapped state keys, its push token and the blobs it uploaded with it, because a revocation that left any of those behind would be a revocation that did not revoke.

Every line below is read off migrations/, and what each table says about itself is its own comment in the schema — a map written from memory is a map that is wrong by the second migration. The two marked in accent are the ones every message goes through.

Two things worth noticing in the shape of it. group_log is partitioned by month, so dropping history is dropping a partition rather than deleting a hundred million rows — which is also why the window is a product decision, how long a device may be away, rather than an ops knob. And several tables carry no index on purpose: sessions is swept four times a day by a scan Postgres would have chosen anyway, and an index there would be maintained on every login to serve four queries.

What it will cost is a shape before it is a number, and the shape has a corner in it. Every table here answers to one of two things — how much was said, or how many people there are — and only the second kind keeps rising. The log stops a month in because from then on a day arriving drops a day off the far end, which is retention seen from the outside rather than a second mechanism.

The arithmetic under the top line is one multiplication. An entry is its ciphertext plus the row around it — call it a kilobyte for a message, which is generous — so ten thousand messages a day is about ten megabytes a day, and the plateau is thirty of those: three hundred megabytes, and it stays there whatever else happens. Attachments are the same sum against a much larger multiplier and they leave with the entry that referred to them. Sealed history has its own knob, ARCHIVE_RETENTION_DAYS, which defaults to the same month for a different reason: how far back a device linked today can see.

The bottom line is the one to watch, because nothing sweeps it. A device costs its row, its key packages, its wrapped state key and a push token — a few kilobytes, once — and a person costs that per device they own. At a hundred thousand devices it is still under a gigabyte, which is why this is a line and not a section: the traffic plateaus and the directory is small, so the forecast that matters is the one for attachments.

Monitoring

prometheusgrafanametricsalerts

Most of what goes wrong in this service goes wrong quietly. A push that cannot be sent is dropped, because clients reconcile. A rate bucket that cannot be read allows the request. A socket queue that fills discards events. Every one of those is a warning in a log at best, and the service keeps working — so the counter beside it is the only thing that says it is happening. That is what these numbers are for, and it is why the alert list is short: each one fires on something otherwise invisible.

The endpoint, and the token on it

Metrics are served on a listener of their own — 127.0.0.1:9091 by default — rather than as a route on the public one, so nothing about them shares a middleware, a rate limiter or a port with what clients talk to. It is always behind a bearer token, loopback included, and that has no exceptions: on a shared host loopback is not a boundary. A token shorter than sixteen bytes is refused at startup.

{
  "metrics": { "bind": "0.0.0.0:9091" }
}

// The token comes from exactly one of three places, and two of
// them being wrong is a startup error rather than a silent default:
//
//   "token":      "<the token>"           here, in this file
//   "token_file": "/run/secrets/metrics"  the shape Docker and
//                                         Kubernetes use
//   METRICS_TOKEN in the environment      what docker-compose does
//
// openssl rand -hex 32

Bringing up Prometheus and Grafana

Both are in the compose file already, behind a profile, so they are one word on the command line rather than a second stack to assemble. Nothing about them starts unless you ask.

echo "METRICS_TOKEN=$(openssl rand -hex 32)" >> .env
echo "GRAFANA_PASSWORD=$(openssl rand -hex 16)" >> .env

docker compose --profile observability up -d

# Prometheus  http://localhost:9090
# Grafana     http://localhost:3000   the dashboard is already there

There is a one-shot container between those two lines and Prometheus, and it is worth knowing why it exists: Prometheus reads its credential from a file and expands no variables, so the token in .env becomes a file exactly once, in a volume the two of them share.

What Prometheus is told

Every instance is its own target, scraped directly rather than through the proxy. That is the one line of this config worth explaining: the proxy balances, so a metrics endpoint behind it answers for whichever instance it felt like — a gauge that halves and doubles at random. Scrape each, and sum() at query time is what makes them a fleet again.

global:
  scrape_interval: 15s
  # equal to the scrape interval: an alert is only ever as
  # fresh as the sample it reads
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: chat-delivery
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/metrics_token
    static_configs:
      - targets: ["server-a:9091", "server-b:9091"]

The bot gateway and the faucet export the same way, with the same header and the same metric names, and neither is in that compose file — so their jobs are written out in the comments there rather than guessed at. Give each its own token when they are not one deployment; credentials_file is per job for exactly that reason.

Grafana, provisioned rather than clicked

The datasource and the dashboard are files, mounted read-only, with allowUiUpdates: false. A dashboard whose datasource was chosen in a browser is a dashboard that breaks on the next fresh volume; edits in the browser are for trying things, and the file is what the next deployment gets. What arrives is chat — delivery service, twenty-seven panels: is it up, instances answering against instances in the registry, live sockets, time since housekeeping finished, requests per second and 95th percentile by route, throttling, delivery.

chat — delivery serviceLast 6 hours15s

Is it up

2Instances answering

2Instances in the registry

1 284Live sockets

41mSince housekeeping

7Requests in flight

3Bots served

Traffic, and delivery

Requests per second, by route

/groups/:id/log/inbox/connect

sum by (route) (rate(chat_http_requests_total[5m]))

95th percentile, by route

/groups/:id/append/blobs

histogram_quantile(0.95, sum by (le, route) (rate(chat_http_request_duration_seconds_bucket[5m])))

The group log

messagecommitepoch conflicts

sum by (kind) (rate(chat_log_appends_total[5m])) sum(rate(chat_epoch_conflicts_total[5m]))

Live events

sentdropped: slow socket

sum(rate(chat_ws_events_sent_total[5m])) sum by (reason) (rate(chat_ws_events_dropped_total[5m]))

What is exported

Ninety-one metrics, and they group into eleven families. Nearly two thirds are counters, because most of what is worth knowing here is a rate rather than a level — and a counter survives a restart being read as a reset, which a gauge does not.

What is worth waking somebody for

Fourteen rules, and the test each one had to pass is the same: it fires on something that would otherwise stay invisible. InstanceDown and more than one percent of requests failing are the two that page. The rest are tickets — the instance registry disagreeing with the scrape list, the limiter failing open, fanout flapping, sockets dropping events, housekeeping stopped, an epoch-conflict storm, key packages exhausted, a provider unreachable, blobs missing, the database pool saturated, and somebody probing the metrics endpoint.

- alert: FleetDisagrees
  # The registry's own count against the number of targets that
  # answered. They disagree when an instance died without
  # unregistering, or somebody scaled out and did not tell Prometheus.
  expr: max(chat_fleet_instances) != count(up{job="chat-delivery"} == 1)
  for: 15m
  labels: { severity: ticket }

- alert: ServerErrors
  expr: |
    sum(rate(chat_http_requests_total{status=~"5.."}[5m]))
      / sum(rate(chat_http_requests_total[5m])) > 0.01
  for: 5m
  labels: { severity: page }

One rule about the metric names themselves: no device id, no identity, no token, no slug, no name. A label that names a person is a label that turns a dashboard into a record of who talked to whom — see the header of crates/observe/src/lib.rs.

HOW THE SERVER WORKS

What the service is, and what it refuses to be

• PostgreSQL
• MLS

One sentence, and everything else follows from it: the delivery service orders messages and cannot read them. It is not a store of conversations, not an authority on who is in a room, and not a party to anybody’s encryption. Every design decision below is what that costs or what it buys.

One order per room, and where it comes from

MLS requires every member to apply the same commits in the same order. A commit accepted out of turn splits a group into two incompatible key states, and unlike a database there is no rollback — applying a commit destroys the previous epoch’s secrets by design.

So the whole ordering guarantee is one comparison: a room’s row is locked, the sender’s epoch is compared to the room’s, and a sender whose epoch is stale is refused rather than merged. That is it. There is no consensus, no resolution, no reconciliation — the cheapest correct thing, and the reason a busy room’s throughput is a number worth knowing before somebody discovers it.

The honest cost: a commit needs a round trip to the room’s home server, and two members committing at once means one of them is told to try again. A design that merged instead would be a design that could not use MLS.

Following a room without holding a key

The server has to know who is in a room — that is who a message is fanned out to. It used to be told: the sender named the devices it had just added, and the server believed it. Two lies were available to any modified client. Naming a device that is not in the ratchet tree subscribes it to every ciphertext the room produces forever; it cannot read them, but it learns who talks to whom and how often. Omitting a device that is in the tree stops its fanout while it remains a member, so it falls behind and its next commit is refused on the epoch barrier — one member evicting another, silently.

Handshake messages travel in the clear already, so the server reads the roster from the tree itself and neither lie stays expressible. It learns nothing it did not already have on disk, and it still holds no key.

Shadow first

New rules that could refuse something run for a while without refusing anything. They run, they agree or they disagree, and the disagreement is counted. A rule switched on the day it is written is a rule whose false refusals arrive as somebody’s conversation breaking; a rule that has disagreed with reality zero times for a month is one that can be switched on.

The log is a catch-up buffer, not an archive

This is the decision most likely to surprise, so it is worth stating plainly. The service keeps a room’s log long enough for a device that was away to read it, and no longer — thirty days by default. Once every member has read past an entry it is dead weight, and forward secrecy means nobody could decrypt it later anyway.

So retention is not an ops knob dressed as a product one. It is the longest a device may stay offline before it must re-join its rooms, said in days. History lives in the application, on the devices, sealed — or it does not exist.

A cursor, not a queue

Nothing here holds a copy of anything for anybody. A device catching up reads the log from where it left off; another provider being caught up is a mark saying how far it has been told. There is no outbox to fill, nothing to leak if it is never drained, and nothing to reconcile if two of them disagree — the log is the truth and a position in it is the whole of the state.

A live socket is a doorbell, not the delivery. A push notification is an empty wake-up. Both exist to shorten the wait; the log is what is actually read, which is why a missed push costs nothing and why a push can carry no content at all.

What a device that was away comes back to

A client that reconnects does not ask anybody what it missed. The live channel begins by reading the mailbox and then each room’s log from the last sequence number it holds, so everything that happened while it was away arrives as the same messages it would have received at the time, in the order they were written. Control messages are part of that: a room renamed, a room’s picture changed, somebody’s profile announced — their name, their photograph, the line they wrote about themselves — a reel posted, a reel watched. They are applied in sequence, so the state a device ends up in is the state everybody else has been in all along.

An announcement that changes nothing is applied and not reported. The room says what it is called and what it looks like again every time somebody is added — a member who joins at epoch nine cannot read what was said at epoch one — so everybody already in it hears it a second time and a third. The client writes a line into the conversation only when the value it holds actually changes.

What replaying the log cannot repair is what is no longer in it. Retention is a limit on how long a device may be away, not a limit on history: past it, the device re-joins its rooms rather than reading them, and an announcement old enough to have been trimmed can never be replayed — a copy of a profile that went stale behind it would stay stale for ever. Reels expire on their own after a day, so one posted and gone while a device was off is simply never seen, which is the point of them.

So each device republishes its own profile when it connects, and at most once a day. That is one control message per room, from the only party that knows the facts, and it means any stale copy of a name, a photograph or a description repairs itself within a day of both ends having been online — without anybody asking anybody for anything, and without the service holding a directory of who is called what.

Two instances, and why the second one is free

Everything correctness-bearing is in Postgres, so instances are interchangeable and the proxy in front of them needs no sticky routing. One instance is the case that works by accident.

The one thing that does not follow from that is fanout: a message landing on instance B has to reach a socket held by A. It was done through Postgres notifications, and the argument was good — nothing new to operate. What it missed is that a notification is a broadcast: every instance parsed every event in the deployment to find the few it held sockets for, so ten instances did ten times the work to deliver the same messages. That is the exact opposite of what a second instance is bought for. It is addressed now, and the rate buckets moved the same way — one bucket per subject for the whole fleet, so the numbers in the file are the numbers in force however many instances are running.

Sharding, which is not federation

Two different things are easy to confuse here, and the difference is who is on the other side. Federation is two providers who do not trust each other exchanging signed messages — different operators, different databases, a protocol between them. Sharding is one provider whose database is cut into pieces: one operator, one deployment, one set of rules, and the pieces answer to the same code. A room does not know which shard it is on, and no client ever addresses one.

What makes the cut possible is the same fact that makes the room’s home work: the only guarantee that has to be strict is one commit per epoch, and that is a compare-and-swap on one row. Everything the write path touches — the room, its roster, its log — is keyed by the room. So a room fits inside one piece whole, and no transaction has to cross anything.

There are two natural keys, not one: rooms, and people. Devices, sessions, key packages, sealed state and the inbox all follow the identity that owns them, so logging in and checking whether a device is still one its identity stands behind stay in one place too. The seam is where a question about a room needs an answer about a person, and there was one such question on every read and every append — is this device’s owner in this room. The roster stores the identity now, so the room answers it alone.

That was worth doing before any piece was cut. It removes a join from the hottest query in the service, which is a plain win on a single database — and the ability to divide the data later is the consequence rather than the reason.

When the request came to the wrong server

A room has one home, written into its identifier. A request for a room somebody else orders is answered 421 with the label of the home that does — not 404, because the room exists and the request is well formed, and not forwarded, because a server that quietly ordered somebody else’s room would produce a working conversation nobody else could see. That is the worst shape a routing bug can take. The client re-reads the directory and asks the right one; nobody is told anything.

Retention and cleanup

Housekeeping runs every six hours. Old log data leaves by dropping a whole monthly partition rather than deleting rows — the difference between instant and a vacuum backlog that arrives days later as mysterious slowness.

Every one of those is counted, and the counts are the point rather than a nicety: a number that stops moving is how an operator finds out a sweep stopped running. A sweep that silently stopped looks exactly like a deployment nobody is using.

LIMITATIONS

Limitations

• Redis
• Proof of work

Enrolment is open and self-service. An attestation is self-signed, so making a device costs nothing and making a thousand of them costs nothing either — none of that is a break-in, and all of it is somebody else’s machine full. Three separate things stand in the way, and they answer three different questions.

Work, and only when somebody is being expensive

Below the allowance the demand is zero — not a small puzzle, no puzzle, no header, no cost. Past it each step of volume adds a bit, and a bit is a doubling. The shape is the point: somebody typing has a rate a human hand sets, somebody pasting has a size a human patience sets, and a script has neither.

The defaults are read from published usage rather than guessed. A messaging user sends on the order of forty messages a day over about thirty-five minutes in the application — roughly one a minute while actually using it. And message length has been measured repeatedly in the same place: mean 58.7 characters, median 50.5, mode 32. So the allowance sits at sixty messages and 256 KiB a minute: about fifty times what a person does, and under a second’s work for a loop.

A proof covers the server’s rotating seed, the device, and the exact bytes being sent. Without the seed it could be computed a week early; without the device one solved puzzle would spend for a thousand of them; without the body it would be reusable, which is the flood again with one puzzle solved. The refusal carries all three so a client can answer rather than guess — and it answers once, because a second demand means the difficulty rose while the work was being done, and retrying that forever is a phone getting hot instead of an error.

This is not a defence against somebody willing to spend money on hardware, and does not claim to be. What it changes is the economics of the cheap case — a laptop, a loop, and nothing to lose — which is the case that actually shows up.

Giving things up in order, rather than falling over

A server out of CPU does not refuse politely. It gets slow, then slower, then the health check times out and something restarts it in the middle of everybody’s conversation. Refusing a video upload is a thing one person notices for a minute; falling over is a thing everybody notices for ten.

Three readings, and the worst of them decides rather than the average — a machine with plenty of CPU and no disk is not half fine. Load average per core, so one file means the same on a laptop and on a sixty-four core machine. Free space where attachments are written, which is the one with a floor under it because a full disk fails for everything at once, including the database. And bytes this service moved — not what the network card did, which includes everybody else on the machine and is not ours to answer for.

The order is the argument: largest and least urgent first, and text is never refused at any level. It has no knob in the file, which is how it is guaranteed rather than configured — a messenger that cannot carry a sentence has not degraded, it has stopped.

A voice message is the one the server cannot judge for itself: it holds ciphertext, and a voice note is not distinguishable from any other audio by size. So /info says what is not on offer and the application hides the control, with the size ceiling as the backstop rather than the rule. A button that fails when pressed is indistinguishable from a bug.

One file, and it can be TOML

All three live in the same file — pointed at by CHAT_SERVER_CONFIG, or ./server.json if that is unset. The language is chosen by the file’s name rather than by trying one and falling back: a TOML file with a mistake in it reported as bad JSON is an error about the wrong language, on the wrong line, for somebody who never wrote any JSON.

[pow]
free_messages = 60              # ~50x what a person does in a minute
free_bytes = 262144             # a message is tens of bytes of text
messages_per_bit = 20           # each further step doubles the work
max_bits = 22                   # a few seconds on a phone, and no more

[congestion]
cpu_critical = 4.0              # load average per core
disk_free_critical = 536870912  # 512 MiB — the last chance to stop
max_upload_busy = 8388608       # 8 MiB — a photo goes, a video does not
refuse_calls_at = "congested"
refuse_voice_at = "critical"

• Every field is optional. Name the ones you want to move; a file that is not there at all is the ordinary case and means the defaults. A file that is there and will not parse is a startup error, because limits that silently failed to load are limits nobody is running.
• The congestion section can only narrow. It never raises a ceiling — otherwise the real maximum lives in two places, and the smaller one is not always the one in force.
• These numbers are placeholders with their reasoning attached. Published averages are other people’s users on other people’s products. The service records what yours do — chat_messages_per_device_minute, chat_message_bytes, chat_pressure_level — so the next set can be read off a running deployment instead of cited. Refusals climbing while those histograms sit low means the allowance is too tight, not that somebody is flooding.

FEDERATION

More than one server, and what has to be true between them

• MIMI
• MLS

Federation here is not a network of equals passing copies around. It is one sentence: every room has exactly one server deciding its order, and everything else is delivery. MLS requires one commit per epoch, and that guarantee is a compare-and-swap on a single row — it cannot be distributed, only assigned. So the interesting question is never who agrees; it is who was assigned, how everybody else finds out, and what happens to the ones who are somewhere else.

Three protocols, and which one is spoken depends on who is at each end. A device talks to the provider it has an account at over ordinary HTTPS, with one WebSocket for what it has to be told rather than ask for. It talks to another provider the same way and directly — opening an account there is an attestation and a signature, not a favour anybody has to grant. Providers talk to each other in draft-ietf-mimi-protocol over mutual TLS, at endpoints each reads out of the other’s directory.

The dashed line is the only fallback in the picture: a home this device cannot reach at all is reached through its own, which carries a request it cannot read, alter or replay. Read the solid axis left to right and it is also the other arrangement — a room another provider orders, followed here on the device’s behalf and read locally like any other. Which of the two a room uses is settled at the invitation, not by its identifier.

A room has one home, and its name says which

The assignment has to be readable by anyone holding the room’s identifier — otherwise there is a global registry of rooms somewhere, which is the thing being avoided. So it is in the identifier, in bytes both sides read the same way.

The home is an opaque three-byte label, not a domain. A domain would be permanent: the identifier lives in the MLS group context and is signed into every commit, so renaming or moving an operator would mean re-creating every room it holds. It would also tell everyone who ever sees a commit which operator runs the room. A label costs a small directory mapping it to an address, and that directory is a file anyone may edit.

The version byte is there for the change nobody survives: rules that shift under bytes that still decode. Nothing else here catches it — the server does not read what is inside a room, and MLS does not tell an application message from any other. So the rules a room was made under travel with its name, and a byte declares rather than enforces: the refusals sit at the two doors that enter a room, not on the read path, because a rule applied to a room already full of conversation takes the conversation with it.

Ask the wrong server and it answers 421 with the label of the one that does — never 404, and never by forwarding, for the reasons under the wrong server. That refusal is also the one a client can act on alone: the identifier names its home in bytes both sides agree on, so a 421 cannot mean the client got the home wrong — only that the label resolved to the wrong address. It re-reads the directory and asks again, once. A second 421 after a fresh directory is a fleet disagreeing with itself, and retrying into that produces a client that hangs instead of one that reports something an operator can act on.

The lane between providers

Two instances of one deployment are interchangeable and share a database; two operators share nothing. The second case is a wire protocol, and the one being spoken is draft-ietf-mimi-protocol — the IETF’s work on messaging interoperability. Paths are not hard-coded: a peer reads /.well-known/mimi-protocol-directory and finds the endpoints there. Every field in it is optional, which is how a peer discovers what a provider does not offer — more useful than finding out from a 404 halfway through a flow.

Authentication is mutual TLS, and the peer roots are the deployment’s own, not the system’s. A peer is somebody this operator agreed to federate with, and “any certificate the world will issue” is not that agreement.

• A half-configured lane is a startup error, named variable by variable and all at once. The alternative is a lane that fails at the first peer connection — where the failure belongs to somebody else’s request and looks like their problem.
• A domain with no lane running serves an empty directory, and warns. Advertising endpoints nobody can connect to is the one promise the directory exists not to make.
• The certificate says which provider, not who asked. So claiming key material must additionally be signed by the key it names. A claim spends a single-use package: unchecked, a peer could compose requests for any of its users and burn through this deployment’s people’s packages. What the signature establishes is stated no more strongly than it is — that whoever composed the request holds that private key. Not who the key belongs to, because a credential issued by another provider is one this deployment cannot verify.
• Addresses may be told rather than resolved. MIMI_PEER_ADDRESSES pins domain=host:port where DNS is not the answer. It pins an address and never an identity — the certificate is still checked against the name, so pointing a domain somewhere does not make whatever answers there that domain.

Adding somebody at another provider

None of the hops is one the adding client could make itself. Their key material is claimed through their own provider, their leaf is committed into the tree here and ordered at the hub, and only then does the Welcome go out to be carried.

Order is the whole of it. Claim, commit, then the Welcome — somebody who accepted an invitation to an epoch the hub never accepted is a member of a group nobody else is in.

• Who to notify is read, not accepted. The providers a room touches come out of the ratchet tree, never out of the request. A client naming its own list could have this server deliver a room’s Welcomes to a provider nobody in the room belongs to — telling them the room exists, who is in it, and handing them ciphertext to keep.
• Who a Welcome is for is worked out, not stated. Its secrets are addressed to key package references, which is MLS’s own name for a joiner, so the fanout names no recipient and does not have to. A reference nothing matches is skipped — that is the ordinary shape of a Welcome addressed to several people at several providers.
• A foreign key package takes the same road as a local one, so it meets exactly the validation a local one does. That is what makes a foreign leaf safe to have in the tree at all.
• Fanout accepts Welcomes and refuses the rest, loudly. A commit or an application message would need this deployment to hold the log of a room it does not order. A peer told its messages arrived and finding nobody read them has been lied to.

Following a room somebody else orders

A room this deployment does not order is followed, and everything follows from that one distinction. There is no second barrier: the barrier exists so one server decides the order, and for a followed room that server is the hub. Entries arrive from it, are appended in the order they came, and are numbered locally, because a local sequence is what a client here reads by. The contents are not taken on trust — every commit and message is validated by each member’s own MLS state, exactly as if it had been ordered here.

• Its own copy has to come back recognisable. The hub returns entries with no sender, which is right for everybody else’s and wrong for one’s own: MLS refuses to let a sender read their own ciphertext. So the one thing this provider knows — that it carried these exact bytes for this device — is remembered under the ciphertext’s hash for the length of the round trip.
• The client has to be told where to read. Nothing in an identifier could say it: the question is not whose label it is but who is holding a copy. The flag is persisted at join, from the Welcome.
• A commit is not pushed at the provider it adds. They have not been told the room exists — the invitation is a separate hop and has to be — so the fanout uses the tree from before the entry. They get the tree with the Welcome.
• A change to the room goes through the same barrier. What the hub checks is what MLS checks: that the asking provider has somebody in this room. No more, because within a room any member may commit and what a commit may do is decided by every member validating it, not by a server’s opinion. No less, because otherwise it is ordering strangers’ commits into somebody’s room.
• The joiner is told where to start reading, from just after the commit that added them — which only the hub knows. Sending them to the start of the log hands them messages from epochs they were never in, which MLS refuses, correctly and unhelpfully.

A provider that was down

Everything about federating was first proved inside one process: two servers, two databases, loopback ports, a certificate authority the test invents. Running them as two real processes showed something that arrangement could not. Stop one provider, send a message to a room somebody there is in, start it again: the message is gone, permanently.

The reasoning that allowed it is right about the case it was written for. A fanout has never been the delivery guarantee, because the log is — a member who misses a push reads the log and catches up. For a member at another provider it does not hold: their provider’s copy of the room is those pushes. There is no second source to catch up from, so a push that failed while they were down is not a delayed message but a lost one, and nothing said so.

The fix is a cursor, not a queue: one number per room and provider saying how far that provider has been told. A push advances it only when the entry it delivered is exactly the next one — a success out of order leaves it alone, so a hole can never be stepped over — and a sweep offers everything from behind, in order, stopping at the first refusal. Nothing queues a payload, because the log already holds every entry; the log’s own retention is therefore the bound, and an entry that has aged out was never going to be delivered by any amount of retrying. Rows are taken with for update skip locked, so instances share the pass rather than racing, and a provider whose last member leaves is forgotten on the next commit — the tree is the only thing that knows.

The part worth keeping is not the cursor. It is that a rule which is correct in one place was carried to a place where it is not, and that only two real processes could show it.

When a client cannot reach a home

A device that cannot open a connection to a room’s home is carried there by its own. This was refused for a long time on a good argument — talking to a home directly needs no trust between servers, while forwarding appeared to need it — and the argument turns out to be against vouching, a server asserting “this is Bob”. It does not touch a design where the device signs for itself.

The envelope covers the method, the path and query, the body, a timestamp, a nonce and the home it is addressed to, signed with the same device key that proves possession at login. A carrier holds bytes: it cannot forge a request, alter one, move it to another route or another home, or send it twice — and it cannot sign for the device whose key it can plainly see in every envelope passing through it.

• Direct first, carried on failure. Only a failure to reach a home falls back. A home that answered — with a refusal, a 401, anything — has given an answer, and asking again through a longer pipe would get the same one.
• A route that dies mid-conversation recovers too. That is the ordinary state of a phone changing networks, so it is the case that matters most: a connection that stops answering is replaced by a carried one, and the carried one replaces the cached entry, because a route that has just failed is not a route to keep trying first.
• One request is lost when a route dies, deliberately. A connection that died halfway through sending is not an answer either, but it is still not safe to repeat: the bytes may have arrived and been acted on. So the request in flight is reported to the caller and only the ones after it are carried — whether to send it again is the caller’s decision, because the caller is the only layer that knows whether doing so is harmless.
• No new trust is needed, for three reasons. A device’s identifier lives inside its attestation, signed by the root key, so a roster means the same thing at every home. A device row’s keys are never rewritten, so the far home reads which device a key belongs to out of its own records rather than the sender’s claim. And registration is anonymous, so the first carried request from an unknown key is the one that creates the row — it passes as nobody, and an authenticated route then refuses it for having no session.

What carrying does not hide: the carrier sees which homes this device has business with, when, and how much. That is the cost of the fallback and the reason it is a fallback rather than the normal road. Always carrying would trade the metadata the other way, and it is a different product decision rather than a missing piece of this one.

What this does not do

• A room cannot be moved. Its home is in its identifier and that identifier is signed into every commit it will ever make. What exists instead is succession: a new room at the new home with the same people, announced from inside the old one over a channel MLS has already authenticated. It does not need the old home except to say so — every member’s key material lives at their home, and their home is in the attestation in the tree the migrating device already holds. Planned moves are handled completely; a home that vanishes without warning leaves the people still pointed at it untold, and no protocol fixes that from this side.
• Consent is types, not policy. Within one deployment anybody may claim anybody’s key packages, and the argument is that a claim is a drain rather than a disclosure. Between providers that reasoning stops holding: a stranger’s provider can drain somebody’s packages all day and learn who their devices are while doing it. The request and answer are written; what is not written is the rule for deciding.
• Franking is an open question, and may be answered no. Reporting abuse to a third party requires cryptographic proof that a message was sent — a property this project has so far declined to offer, and one MIMI requires for interoperability.
• A credential from another provider is not verified. Signatures are checked; who a key belongs to is only established among peers using this project’s own attested credentials. That is MIMI’s general problem rather than this deployment’s, and the checks are documented at the strength they actually have.

MONEY A GROUP HOLDS

The group wallet, and what signs for it

• Multisig
• Bitcoin
• Ethereum
• TRON
• Solana
• Safe
• Squads

A wallet a group holds is the one thing on this list that is not ours. Every chain already has a way for several people to hold one address, and this uses that way rather than inventing a construct on top of it: P2WSH on Bitcoin, Safe on the EVM chains, the account’s own permission system on Tron, Squads V4 on Solana. Four crates under wallet/crates/multisig/, one per chain, behind a single JSON‑RPC entry point where every method is prefixed by the chain it belongs to — bitcoin_*, evm_*, solana_*, tron_*.

What matters for everything else on this page is where the keys are. There is no wallet service and no custody: each signer’s key stays on their own device, the address is built out of all of them at once, and the threshold is the group’s rule rather than a setting on a server. Nothing this deployment runs can move that money, because nothing it runs has ever held a key to it — the same sentence as the one about the delivery service holding ciphertext it cannot read, said about coins instead of messages.

Four steps, and only the last is on a chain: the group fixes the rule, somebody proposes a payment, each signer signs on their own device, and at the threshold it goes out. The three before the last are messages in the conversation the payment was argued about in — which is why a payment here still has its reason attached to it a month later, and a payment from an exchange does not.

The whole of it — the flow drawn out, what each chain calls “several people, one address”, and the deployments this application actually talks to — is a page of its own. It is a product page rather than a reference section, and folding it in here would put a chapter about money in the middle of a document about servers.

THE DAPP BROWSER

A browser inside the app, and pages that ask the wallet to sign

• Ethereum
• EIP-1193
• TRON
• Solana
• Multisig

Status, so the rest of this is not read as shipped: phases 0 to 4 are written and pass their tests, and no real dapp has been opened with it yet. What the unit tests cannot cover is the part that has to be right — injection timing, the handler round trip, and whether a real bundle recognises the provider.

A web page cannot reach a private key and must not be able to. What it can reach is an object put on window before the page loads, whose every method is a message to something outside the page. So the interesting part of this work is not the WebView. It is the router that decides which of a dapp’s many possible requests are answered from local state, which go to a public node, and which stop and wait for a human.

How a wallet listens to a page

There are two mechanisms and they coexist, because the older one cannot be withdrawn.

• The global, read the moment the page loads. A page that asks “is there a wallet here?” synchronously at load reads window.ethereum and gets whatever is there at that instant. Which is why injection has to happen at document start, in the page’s own world, before any page script runs — a provider that appears after the bundle has already asked is a provider that dapp will never see. It is defined with Object.defineProperty and non-configurable, so a page shipping its own shim cannot quietly replace it.
• The event pair, for when more than one wallet is present. EIP-6963 exists because a single global cannot hold two wallets: whoever wrote it last wins, silently. So a wallet announces itself on load, and announces again every time the page fires eip6963:requestProvider — each announcement carrying a name, an icon and a reverse-DNS id, which is what modern connect modals list. The icon is a data URI, so listing wallets fetches nothing from the network.

Which is also why isMetaMask is false here. Many dapps still branch on it, and setting it true is a lie the page then acts on — the lie every other wallet tells. There is a per-site switch, off by default and labelled as what it is, for pages that hard-gate on the flag. A global default of true would tell every page something untrue whether or not it needed to be told.

What the injected object is, and is not

A thin, boring shim: an EIP-1193 object with the legacy send / sendAsync / enable shims older libraries still call, state cached so the common questions cost no round trip, and events pushed out from the app rather than inferred by the script. chainChanged is emitted after the app has switched, never optimistically.

Every call becomes one message across the bridge and comes back as an EIP-1193 result or an EIP-1193 error — {code, message, data}, with the standard codes used honestly: 4001 only when a person actually pressed Reject, 4100 when the origin has no accounts granted, 4200 for a method we do not implement, -32602 for malformed parameters.

The script never holds a secret and never makes a decision. If a rule appears there and not in the app, the rule does not exist — the page shares a JavaScript context with it and can call the handler directly.

Four answers, and a whitelist

The classification is a whitelist rather than a blocklist: a method nobody has thought about is refused, never forwarded and never guessed at.

• An unconnected origin is told nothing. eth_accounts returns an empty list until the origin has been granted one — that is the specified behaviour rather than a courtesy.
• Forwarding introduces no second setting. Read-only calls go out over the endpoint already configured for that chain, the same one the balances use, including any override.
• Two refusals are named rather than merely unlisted. eth_sign signs an opaque 32-byte hash, which is the primitive behind the drain-everything phishing page and which no legitimate dapp needs. eth_sendRawTransaction is publishing, which is the wallet’s job — a page that wants a transaction sent asks for one to be signed. Each refusal has a test of its own, because the way this class of code fails open is somebody adding a passthrough default.

What a person is shown before signing

Everything the wallet signed before this was composed on its own screen. Signing what a stranger asked for is a different activity wearing the same word, so what is known is shown and what is not known is said rather than left out.

• A sign-in whose domain is not this site is stopped. When a message parses as EIP-4361 its domain is compared with the origin, and a mismatch stops the sheet rather than warning underneath it: that signature is the one that logs somebody into a site they are not on.
• An unlimited approval is the standard way wallets are drained. If one call is decoded it is this one, and approve at the maximum value is warned about explicitly. Anything not decoded is shown as an unknown call with its selector and its raw data — honest, rather than a friendly summary of something nobody read.
• The origin is re-read immediately before signing. Not trusted from when the sheet opened, and the request is cancelled if the page navigates or the origin changes while it is open.

Origins are scheme, host and port, computed from the URL the WebView actually committed and never from anything the page says about itself. HTTPS only, plus the loopback for development — a page delivered over plain HTTP can be rewritten in flight by anyone on the path, and this one is being handed an account. A grant and a site’s stored data are two separate things to clear, and the settings say so.

What it deliberately does not do

• No group wallets, in this version. A multisig address is not an ordinary account: it cannot answer a personal signature at all, and a dapp that assumes it can is a dapp that fails quietly. Connecting one means EIP-1271 and a proposal round through the group. The connect sheet says which account it is offering.
• No phishing blocklist. Somebody else’s list, fetched at runtime, leaks every site visited to whoever hosts it. What is done instead: the full origin always visible, a first visit marked, international domains shown decoded and raw, and plain HTTP refused outright.
• No Solana or Tron provider yet, and no tabs. A second chain is a second script and a second method table rather than a rewrite — but shipping three at once triples the surface with keys behind it. One page at a time is a complete browser for this purpose.

WALLETCONNECT

The same signatures, reached from a site across the room

• WalletConnect
• TRON
• Solana

Status: the signing crate and the naming layer are built and tested; pairing, requests, sessions and the end-to-end test are not written. This section is what is being built toward, and it says which is which.

The dapp browser answers a page inside the app. WalletConnect answers one that is not — a site open on a desktop, paired by scanning a code. Different transport, different session model, the same four signatures at the end of it.

The rule that shapes the pairing screen is that a session is a set of chains and accounts, and which addresses are exposed is chosen rather than assumed. A session that can see all of somebody’s chains because that was the easy default is a privacy decision nobody made.

Which chains dapps actually ask for

The protocol is namespace-agnostic — a session is a set of CAIP-2 chains and a wallet advertises whichever it can serve — so the protocol supports everything. The ecosystem does not, and that is what decides the order of work.

Tron is where the promise breaks, and not because the wallet could not answer: tvm has been a draft since 2023 and its own text says there is no central registry of chain ids, so what little asks may disagree with us about which chain it means. Doing all four at once spends the most effort on the least used.

Worth recording because it is the kind of thing that survives review: Solana’s devnet and testnet identifiers are two similar base58 strings, and writing them from memory got them the wrong way round. Reading them from the registry is what caught it.

The long pole is the signing, not the SDK

Every signing method this wallet had signed a transfer this app composed. A dapp asks for the opposite shape: arbitrary bytes, a structured document, and a transaction carrying arbitrary calldata. None of those were expressible, so any estimate that starts with the SDK has the schedule backwards.

• Signing is held to published vectors, never to fixtures. A plausible implementation passes a smoke test and produces a signature that recovers to somebody else’s address. Expected values come from the specifications themselves and from an independent implementation of the same specification — including a Cyrillic message, because an implementation that writes the character count into the EIP-191 prefix passes every ASCII test and is wrong.
• The package choice changed on measurement. The wallet-side SDK declares Android and iOS only, through a payments module this app has no use for; the two packages that carry the protocol support desktop as well. Building on those is what keeps the desktop layout — and it reversed an assumption made before measuring.
• A relay sees who pairs with whom. That is a second place data leaves the device unencrypted, and it belongs in settings written in the same voice as the first one rather than in a footnote.

Four things are recorded as refusals now, so that they are not quietly done later: no auto-approval of anything, including “just a signature”; no exposing every address by default; no silent chain switching, because it moves which chain a later transaction lands on; and no half-decoding — an honest “this is a contract call and this wallet cannot read it” beats a friendly summary of something nobody parsed.