Skip to content

7. Moving from CRDT to Files

Date: 2026-01-16

Status

Accepted

Supercedes 2. State-based CRDT

Supercedes 5. Chronology through trusted attestation

Supercedes 6. Announcing hostnames

Context

As the needs of Clan grow, we wish to place more kinds of state, with differing merge semantics, into Data Mesher. This requires moving away from the bespoke CRDT for distributed DNS already implemented, to a more generic schema-driven approach.

But as we have learned from the first implementation, reasoning about CRDTs is not trivial. For this reason, Pinpox experimented with a simpler, file-based approach in the form of Data Smasher.

Instead of a CRDT, users sign a file and present it to Data Smasher for distribution. This is checked against a pre-defined set of signing keys (per file name), and if valid, the file is accepted and distributed to all other nodes.

No requirements are placed on the format of the file. Processes that are interested in the file's contents can watch for changes via mechanisms such as systemd.path.

Decision

Having reviewed the implementation of Data Smasher, we have decided to move forward with the file-based approach. We feel it provides a simpler model which is easier to reason about.

The DNS-focused CRDT will be removed entirely along with its associated network and host configuration.

nss-datamesher will be extracted to a separate repository and refactored to use the file-based approach.

A new files config section will be added, which will map file names to signing keys:

1
2
3
[files]
"dns:sol" = ["P6AE0lukf9/qmVglYrGPNYo5ZnpFrnqLeAzlCZF0lTk="]
"dns:vulcan" = ["ZasdhiAVJTa5b2qG8ynWvdHqALUxC6Eg8pdn6RVXuQE=", "1ru2QQ1eWV7yDlyfTTDEml3xTiacASYn0KprzknN8Pc="]

A new PUT /files/:name endpoint will be added to the HTTP API, which will accept a binary body and the following headers:

  • Content-Type: application/octet-stream
  • X-Signedat: <unix timestamp in seconds>:
  • X-Signedby: <base64-encoded ED25519 public key>
  • X-Signature: <base64-encoded signature>

Submitted files will have their :name compared against the list of allowed files, and their signature checked against the list of allowed signing keys. If either check fails, the file will be rejected.

If the check succeeds, the file will be written to disk and its signature recorded in the signature database.

Periodically, during the push-pull loop within memberlist, signature databases will be exchanged and compared against the local database:

  • Any files configured locally (files config) but not present in the local database will be downloaded
  • Any files with a later signature than the local database entry will be downloaded.
  • Signatures will be validated on import, and if they fail validation, the file will be rejected.
  • All other file signatures will be ignored.

Consequences

Data Mesher becomes agnostic to the underlying state which it distributes, decoupling it from any particular use case.

The file system essentially becomes the plugin API, with plugins responsible for implementing the semantics of their own state.

Ensuring the chronology of file updates becomes the responsibility of the signing keys configured on a per-file basis.

It is up to the signers to agree upon what latest means in their respective use cases and to ensure Data Mesher is presented with a valid signature with a suitable SignedAt field which ensures the latest state is distributed.