Multi-hub sync (CRDT) research

Synapse today is single-hub authoritative: one hub owns presence, claims, the board, and the durable event log, and every agent talks to it. This research lane asks whether several hubs — across hosts or domains — could synchronise their coordination state while keeping the two invariants that make Synapse worth using: claim safety (authoritative hubs do not grant overlapping live file scopes) and local-first (a hub stays correct and usable offline for the work it owns).

The honest answer drives the whole design: most coordination state can be merged conflict-free, but claims cannot — granting a lease is mutual exclusion, the exact opposite of a conflict-free merge. So the design splits state by what is actually CRDT-suitable and routes claims through ownership, not gossip.

Runtime status

What exists is single-hub plus operator-managed peering, not state sync:

• One SynapseHub is the authoritative source of truth; the durable SQLite event log is replayed on restart so a single hub already survives crashes.
• Multi-host deployments can use mutual TLS peer trust bundles (see signed events and mTLS) to authenticate peer hubs, and the federated trust model scopes which domain owns which project namespaces.
• The relay log and synapse ingest export the event stream read-side, which is the seam a downstream consumer (or a peer) could replay.

The full multi-host sync now routes a real claim to its owning hub and detects an ownership partition at runtime from the peers observed asserting a namespace. The hub populates that feed itself: synapse hub --multihub-watch PEER=URI (repeatable, requiring --namespace-owner) runs a standing follower (core/multihub_watch.py) that polls each named peer on a bounded interval and folds the observed claims into the assertions the ownership gate consumes. Naming a peer is the operator confirmation for the always-on outbound connection — nothing is auto-discovered — and a failed poll keeps the last successful observation, so an outage never lets a contested namespace silently resume granting. When the hub has a durable journal, entering or changing a contested namespace appends multihub_partition at FULL durability; a successful, non-empty history refresh from every persisted contestant that proves the namespace uncontested appends multihub_heal. An empty or failed refresh is not healing evidence. An unresolved partition is restored into the refusing assertion feed after restart, and an entirely failed poll cannot manufacture a heal. Both events project through the universal-receipt view as federation evidence without changing coordination replay. An operator can still wire the feed by hand (asserting_owners over a follower's view) in library deployments. The honest-peer read-side merge layer, the cross-host event-log pull that lets one hub observe another over a real connection, the serving-side gate that lets a hub refuse to serve its log to an untrusted peer from the live connection, the claim-forwarding path that routes a claim to its owning hub and relays the verdict, and runtime partition detection that refuses a contested namespace, have shipped:

• core/multihub_merge.py — the content-bound event-log union: it tags each event with its authoring hub, binds (hub_id, seq) to a canonical fingerprint of the complete event, merges honest append-only logs into a grow-only set, replays them in the deterministic (ts, hub_id, seq) order, and reports the per-hub resume cursor. Exact duplicates collapse; different content at one identity raises a typed equivocation error and no arrival-order or timestamp winner is selected.
• core/multihub_fold.py — folds that merged order into the observed mergeable view: the board (verified causal heads, then display-only last-write-wins among unresolved heads), the grow-only progress ledger, and the observed claim view — the latest claim each peer reports, tagged with its hub, marked advisory, cleared on release, and never granted. The board contract is explicitly non-authoritative: JSON exposes its policy, complete event fingerprint, and each displayed task's winning (timestamp, hub_id, seq) provenance. Task writes may name one content-bound parent. The fold verifies the parent event identity, complete-event fingerprint, and task id before removing that proven ancestor from the candidate heads; a unique causal head wins even when its clock is behind. It emits a payload-free unresolved conflict when remaining heads have different task-record fingerprints. Missing, malformed, or mismatched parents suppress nothing. The conflict proves disagreement, not concurrency: absent edges and NTP/clock-skew metadata are not vector clocks or causal proof. Partition detection uses a stricter sibling fold keyed by (hub_id, task_id), so a release from one hub cannot erase another hub's equal-named task from the authority signal even though the general display view remains last-writer-wins by task id.
• core/multihub_follower.py — a read-only MultiHubFollower that tracks a per-peer seq cursor, fetches a peer's events past it through an injected transport (store_fetcher reads a peer EventStore over its read_since seam), folds the union, and returns the observed view. Network fetchers also expose the peer's latest wire-version negotiation and welcome-frame clock skew, so a follower can report whether timestamp-ordered evidence is being read across materially skewed hubs. Observe-only by construction: it grants no claim and, losing a peer, simply stops advancing that peer's cursor — the fail-closed posture. The complete fetched batch is validated before publication: an equivocation freezes the prior view and cursor, quarantines the peer, and, when a journal is configured, records bounded digest-only evidence. Quarantine survives restart and only an explicit operator recovery naming a new log generation or accepted checkpoint clears it. Gaps and unseen rollback sequences also fail before publication. It is exposed to operators as synapse multihub observe (the walkthrough below).
• core/multihub_wire.py, core/handlers/multihub.py, and core/multihub_transport.py — the cross-host pull: a request/snapshot message pair on the hub server lets a peer ask for the events past a cursor, and network_fetcher drops a network reader into the same follower in place of store_fetcher. core/multihub_federation.py gates the pull deny-by-default (federation policy composed with mTLS peer verification), so a follower only pulls from a granted, cert-pinned peer. Exposed as synapse multihub follow.
• core/multihub_serving.py — the serving-side mirror of that gate. A hub configured with a MultiHubServingPolicy reads the certificate the peer presents on the live mutual-TLS connection and runs the same authorise_multihub_peer composition the following side enforces; a peer with no operator grant, no client certificate, or an untrusted pin is answered with an empty snapshot — the same shape as "no new events", so the refusal leaks nothing. A hub with no policy refuses every peer — the gate is fail-closed by default.

The production CLI loads that policy through --multihub-serving-policy FILE. The strict version-one JSON names one audited federation store, one client CA, and exact sender/domain/namespace/signing-key grants. Relative paths resolve beside the policy; unsafe, symlinked, duplicate, unknown, or inconsistent input refuses startup. Native TLS requests a client certificate for multi-hub frames while ordinary local clients may still connect without one; the multi-hub gate then denies a missing, revoked, wrongly pinned, or out-of-scope identity. The policy, federation store, and client-CA file must each be owned by the effective hub service user and use mode 0400 or 0600. Core captures all three through full-component no-follow descriptor reads; OpenSSL receives the captured CA bytes and never reopens the configured path.

Content identity, quarantine, and recovery

The honest-peer convergence contract requires one immutable value for every (hub_id, seq). Core fingerprints the authoring hub, sequence, exact timestamp, event kind, and recursively type-tagged payload. Mapping insertion order does not change the digest; payload type changes do. Wall-clock time remains display metadata and never resolves a collision.

If an authenticated, pinned, or signed peer presents a second fingerprint for an accepted identity, authentication does not excuse the conflict. The follower validates the whole candidate batch before replacing its event union, cursor, protocol metadata, or clock-skew metadata. It then refuses the batch, retains the prior observed state, and blocks subsequent automatic polls for that peer. A durable watcher appends one multihub_equivocation record with the peer, sequence, both SHA-256 fingerprints, local detection time, and observer identity; it never stores the event payload. These rows and explicit recoveries project through universal-receipts as federation evidence.

Recovery is a ceremony, not a reconnect. The operator must identify who accepted the recovery, provide a reason (stored only as a digest), and name the log generation or checkpoint the operator accepted. Core commits multihub_equivocation_recovery before discarding that peer's old observed history and cursor. The next poll begins from zero after that recorded ceremony; the current wire does not independently attest the generation label. A stronger signed hash chain with previous-event digests and checkpoint verification remains future work; this containment detects collisions and cursor-shape violations but does not claim Byzantine consensus.

For the built-in durable watcher, stop the local hub and run:

synapse multihub recover --journal ./hub.db --peer-id west \
  --operator incident-commander --reason "checkpoint verified" \
  --new-log-generation west-generation-2

Restart only after that command succeeds. It refuses an unknown or already recovered peer and cannot silently clear a live process's in-memory quarantine.

{
  "version": 1,
  "federation_store": "federation.json",
  "client_ca_file": "client-ca.pem",
  "grants": [
    {
      "sender": "syn-west",
      "domain_id": "west.example",
      "namespace": "PROJECT",
      "signing_key_id": "PROJECT:hub:2026-07"
    }
  ]
}

Outbound secure routes share one paired owner-only identity configured by --multihub-client-certfile and --multihub-client-keyfile. Every such watch, claim, or relay route must also carry its named SHA-256 server pin. Claim and relay maps stay independent, and the relay connector is also used for the content-free dead-letter escalation pointer. Captured client material is loaded at process startup; rotate it by restarting the Core hub. Use --require-relay-reason to make every governed action explain itself and --require-two-person-relay to hold it for a distinct trust-domain principal; these policies apply equally to direct and routed relays. On a journal-backed hub, a validated inbound pending audit restores only the first-principal quorum after restart. A later terminal audit removes it, and malformed or contradictory evidence restores no quorum. Keyed pending, applied, and completed-quorum no-op verdicts exact-replay only after the current peer certificate, relay scope, and namespace ownership are authorised again.

• core/multihub_watch.py — the hub's own standing follower: polls each operator-named peer over that same pull, folds the observed claims with the gate's namespace derivation, and holds the per-namespace asserting-hub view partition detection consumes. Opt-in via --multihub-watch; fail-closed for authority (a failed poll keeps the last successful observation).
• observed_peers.py — the operator-surface adapter for the same read-only pull: synapse who, synapse status, synapse state, and synapse dashboard accept repeatable --observed-peer HUB=URI flags, fold each peer locally, and render the result as observed@HUB advisory state without changing the local hub. When the peer advertises a usable welcome timestamp, the row and JSON payload include local-minus-peer clock skew alongside cursor lag.

Observing a peer — a two-hub walkthrough

The read-side layer above lets one operator observe another hub's coordination with no cross-hub service running. On a single machine — or any shared filesystem — run two hubs with separate event stores, do some work on each, and read the other's state.

1. Run two hubs

Each hub owns its own durable event store (--db):

synapse hub --port 8876 --db ./east.db &
synapse hub --port 8877 --db ./west.db &

2. Coordinate on each

Declare a task on each hub, and claim a file scope on one:

synapse task declare build --title "Build the wheel" --uri ws://localhost:8876
synapse git-claim build --paths src/ --uri ws://localhost:8876

synapse task declare docs --title "Write the docs" --uri ws://localhost:8877

3. Observe the peer

East's operator reads west's coordination, read-only, straight from its event store:

synapse multihub observe --peer-db ./west.db --peer-id west

observing peer 'west' — 1 tasks, 0 progress notes, 0 observed claims
board (verified causal heads; display LWW among unresolved heads — non-authoritative):
  [open] docs — Write the docs [display source west#1 @ 1]

And west's operator observes east — including east's claim, which appears as an observed claim, never granted locally:

synapse multihub observe --peer-db ./east.db --peer-id east

observing peer 'east' — 1 tasks, 0 progress notes, 1 observed claims
board (verified causal heads; display LWW among unresolved heads — non-authoritative):
  [open] build — Build the wheel [display source east#1 @ 1]
observed claims (advisory — not granted):
  build -> <agent> @ east

observe reads the peer's event store through the same read_since seam the follower uses — SQLite WAL lets it read alongside the live peer hub — and prints the folded state. It grants nothing: a peer's claim is advisory here, and a real claim is still made on the owning hub. Add --json for a machine-readable ObservedState. The JSON retains the existing board records and adds board_policy plus per-task board_provenance, including the complete event_fingerprint needed by task declare|update --causal-parent. Consumers must not use the display winner as authoritative task truth. A causal: true provenance means a verified parent chain selected a unique head, not that the observed board grants local authority.

4. Follow a peer over the network

When the peer is on another host with no shared filesystem, follow pulls its event log over a real connection instead of reading a file. It asks the peer for the events past a cursor and folds the same observed view:

synapse multihub follow --peer-uri wss://west.example:8876/ --peer-id west

observing peer 'west' — 1 tasks, 0 progress notes, 0 observed claims
board (verified causal heads; display LWW among unresolved heads — non-authoritative):
  [open] docs — Write the docs [display source west#1 @ 1]

follow is the network counterpart of observe: it drops a network fetcher into the same follower, so it grants nothing either and a peer's claim stays advisory. Pass --token for a secured peer hub, --limit to bound the batch, and --json for the machine-readable ObservedState. Whether a follower may pull from a peer at all is gated deny-by-default by the federation/mTLS policy (see Boundaries); the library API peer_authoriser composes it and the fetcher fails closed for an ungranted peer.

For a wss:// peer whose certificate is self-signed or private-CA, pass --pin sha256:<hex> to trust the peer by its live certificate pin instead of a CA chain — the same pin recorded in the peer's federation bundle and printed by synapse federation fetch:

synapse multihub follow --peer-uri wss://west.example:8876/ --peer-id west \
  --pin sha256:9b1f63a2c8d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0

The pin is compared against the certificate presented on the TLS socket before any snapshot frame is decoded; a mismatch, a missing certificate, or a plaintext ws:// URI fails the pull closed and the follower's cursor stays unadvanced. The library counterpart is synapse_channel.core.multihub_transport.pinned_connector.

On connect, the network fetcher also reads the peer hub's advertised welcome.protocol_version. A mismatch does not fail the pull: the fetcher logs an operator-visible warning, records the negotiated lowest-common wire version for the MultiHubFollower, and keeps optional features gated to that effective version. Peers that omit the field are treated as legacy wire version 1.

4a. Add observed peers to operator surfaces

The day-to-day operator surfaces can show the same advisory peer state without switching to the dedicated multihub command:

synapse who --observed-peer west=wss://west.example:8876/
synapse status --json --observed-peer west=wss://west.example:8876/
synapse state --observed-peer west=wss://west.example:8876/
synapse dashboard --observed-peer west=wss://west.example:8876/

Each command fetches the peer with the multi-hub log request path, folds the events locally, and labels output observed@west. A peer outage renders an unreachable peer row; it does not make the local hub unhealthy and does not grant, revoke, or route a local claim. Use --observed-token for secured peers and --observed-timeout to bound each pull. For a self-signed wss:// peer, add --observed-pin HUB=sha256:<hex> (repeatable) — the same pull-side certificate pin as multihub follow --pin; a pin naming a hub that --observed-peer does not fetch is refused, so a typo cannot silently leave the real peer unpinned. When the peer's welcome frame carries a finite timestamp, who rows include skew=+/-Ns, status includes the largest observed absolute skew, and JSON outputs carry clock_skew_seconds.

5. Trace causality across the hubs

With both logs at hand, synapse causality causes ./east.db west:9 --peer west=./west.db answers the causality queries over the merged logs: events keep their global identity hub:seq, the recorded relations (lifecycle, dependency, contention) are derived over the deterministic merged order, and an edge whose endpoints two different hubs authored is tagged federation with the underlying relation as its basis — so a release on one hub that satisfied a depends_on claimed on another renders as federation:dependency. The same honesty boundary as the fold applies: cross-hub precedence is ordered by event timestamps across the hubs' clocks (there is no shared sequence), so a federation edge is clock-ordered evidence, and the query observes and grants nothing. See the CLI guide for a worked cross-hub example.

If the operator has measured peer skew from synapse doctor --federation-peer or an observed-peer pull, pass it into the offline causality report:

synapse causality causes ./east.db west:9 \
  --peer west=./west.db \
  --clock-skew west=-6.4 \
  --skew-warn-seconds 5

The report then adds a clock-skew warning when the absolute skew exceeds the threshold; --json carries the same warning in clock_skew, and --dot emits it as a Graphviz comment. The warning does not change the graph. It tells the operator that cross-hub ordering is timestamp evidence outside the configured clock-agreement bound.

Where this stops

The cross-host event-log pull above now ships, so a hub can observe another over a real connection, and the federation/mTLS gate is now enforced on both sides — the following side before it pulls, and the serving side before it serves (MultiHubServingPolicy, see Boundaries). Opt-in claim forwarding also ships: a hub configured with --namespace-owner and --claim-peer HUB_ID=URI routes a remote-owned claim to that namespace's owning hub and relays the owner's verdict. What remains outside this design is automatic peer discovery, global consensus, and a merge that could turn observed peer state into local claim authority.

State, split by what merges

Coordination state is not one thing; each kind has a different merge story.

• Durable event log — append-only and the natural sync unit. Each hub's events carry a hub id and a per-hub monotonic sequence, so the union of two logs is a grow-only set keyed by (hub_id, seq). Replaying the union in (timestamp, hub_id, seq) order is deterministic, but that total order is not a vector clock and does not establish cross-hub causality. The grow-only union is the one piece that is genuinely CRDT-shaped.
• Presence — last-writer-wins per agent, keyed by hub id; an agent is present on the hub it connected to, and a peer view is advisory.
• Progress notes and the board plan — notes are grow-only. The observed board verifies optional content-bound, same-task parent references and removes proven ancestors. It uses whole-record LWW ordered by (timestamp, hub_id, seq) only among the remaining unresolved heads, with visible winner provenance. It is useful for a convergent operator view but is not authoritative task state; unrelated or incompletely linked heads remain unresolved and are not called concurrent. Their fallback LWW display selection remains non-causal. A clock-ahead older ancestor cannot override its verified causal descendant.
• Capability cards — LWW per agent id, mergeable.
• Claims — not mergeable. A claim is a lease that must be unique per file scope; two hubs independently granting the same scope is precisely the collision the claim exists to prevent. Claims need ownership or consensus, never a merge.

Claims without a merge

Because claims are mutual exclusion, the design routes them by namespace ownership, reusing the domain model the federated trust model already defines:

• Each project namespace has exactly one authoritative hub at a time. Claims within a namespace are granted only by its owning hub, so there is never a conflicting grant to merge.
• A peer hub does not grant claims for a namespace it does not own; it can read the owning hub's claim state (advisory, eventually consistent) but routes a real claim request to the owner.
• Cross-namespace work that needs scopes from two owners is a coordinated, explicit hand-off between owners, not a silent merge.
• If two hubs ever both believe they own a namespace (a partition), the safe default is refuse to grant until ownership is re-established — claim safety fails closed, never open.

This keeps the strong invariant local: each hub grants claims authoritatively for its own namespaces with no network round-trip, and only the observed view of other namespaces is eventually consistent.

The resolution behind this rule ships in core/namespace_ownership.py: a NamespaceOwnership map records the single owning hub per namespace and resolves a namespace to local (grant here), remote (a named peer owns it), ungoverned, or partitioned, the last two failing closed. A hub configured with such a map enforces it on the grant path — a claim whose namespace (derived from the agent identity, as the ACL derives it) the hub does not own is refused with a claim_denied naming the owning hub, so the caller knows where to route it; a hub with no map grants every namespace, as a single hub does today. The networked half now ships too, opt-in: a hub configured with claim_peers (a route to each owning hub) forwards a remote-owned claim over a connection through core/multihub_claim_transport.py, the owner grants it on the serving side (core/handlers/multihub_claim.py), and the verdict is relayed to the claimant — a grant carrying the owner's authentic lease. An unreachable owner falls back to the refusal that names the owner, fail-closed. Runtime partition detection ships too, opt-in: a hub wired with an observed_asserting_hubs feed resolves a namespace a peer is observed contesting to partitioned and refuses the claim, even on its own local grant path; asserting_owners builds that feed from a follower's observed claims. What is not yet built is the hub auto-discovering peers: the hub populates the feed only from the operator-named standing follower configured by repeatable --multihub-watch PEER=URI flags.

Sync transport

Sync rides the existing seams rather than inventing a new protocol surface: a peer replays another hub's event log from a cursor (the ingest/relay seam), authenticated by the mTLS peer trust bundle, and folds the mergeable state in. Because an honest append-only log is the CRDT-shaped unit, "sync" is mostly "replay the peer's log since my cursor, verify its content bindings, and apply the folds"; conflicting identities fail closed instead of merging, and only namespace-ownership changes need an explicit, operator-confirmed step.

Local-first guarantee

Every hub stays fully usable for the namespaces it owns with no peer reachable: claims, presence, board, and log all work offline. Sync adds an observed view of peer namespaces and a merged history; it never makes a hub depend on a peer to do its own work, and it never lets a peer grant a claim inside a namespace it does not own.

Boundaries

The read-side (merge, fold, follower), the cross-host event-log pull (observe and follow), the deny-by-default federation/mTLS gate on both the following and the serving side (MultiHubServingPolicy reads the peer's live certificate), the namespace-ownership resolution with its local grant-path enforcement (NamespaceOwnership), the cross-hub claim forwarding that routes a remote-owned claim to its owning hub and relays the verdict (claim_peers + forward_claim + handle_multihub_claim_request), and runtime partition detection that refuses a contested namespace when fed observed assertions (observed_asserting_hubs + asserting_owners) are implemented — and so is the hub feeding those assertions itself from a standing follower of its own (core/multihub_watch.py, opt-in via --multihub-watch). The design stays deliberately conservative.

• Claims are not a CRDT. Mutual exclusion is not conflict-free; the design uses single-owner-per-namespace, not claim merging, and fails closed on an ownership partition. The ownership resolution, the local refusal of an unowned namespace, the forwarding of a remote-owned claim to its owner, refusing a contested namespace on observed assertions, the hub's own standing follower feeding those assertions (--multihub-watch), and durable partition/heal transition evidence all ship.
• The merge assumes honest append-only history, not Byzantine consensus. Exact duplicates are idempotent. A reused (hub_id, seq) with different timestamp, kind, or payload is durable equivocation evidence: the candidate view and cursor stay unpublished and the peer is quarantined. mTLS, pins, and frame signatures authenticate the presenter but do not choose a winning history.
• The observed board is not authoritative. A valid task parent is a bounded causal assertion: it is accepted only when the named complete event is present, content-bound, and for the same task. That edge can suppress its proven ancestor; it cannot grant a claim. Remaining heads still use deterministic LWW only for display. Missing edges do not prove concurrency, and synchronized clocks, including NTP, do not establish a parent relation. Local board truth, claim authority, and the fail-closed namespace ownership resolver do not consume this display choice.
• It does not add a new always-on wire surface casually — the pull is a request/snapshot message pair on the existing hub server, reusing the event log, read_since seam, and mTLS peer bundles; it adds no always-on cross-hub service to the local core.
• It does not weaken local-first: a hub never depends on a peer to grant its own claims or run its own work.
• It does not introduce a global consensus cluster. There is no single global leader; authority is partitioned by namespace, each hub local-authoritative for its own.
• It makes no multi-host claim-safety guarantee for two default/unconfigured hubs: the cross-host pull is observe-only. Multi-host mutual exclusion requires an explicit, consistent namespace-owner map plus the standing watch; the durable transition events evidence that posture but do not replace it with consensus.