9. Per-file TTL via Signed ValidFor

Date: 2026-04-10

Status

Accepted

Context

Data Mesher files persist indefinitely until explicitly deleted via a signed tombstone. This requires the file author to actively manage the lifecycle of every file they publish. For some use cases such as ephemeral DNS records, short-lived configuration, time-bounded announcements and so on, the author knows the file should expire, but there is no mechanism to express this.

Without TTL support, files accumulate across the cluster until someone issues a tombstone. In a decentralized mesh where the publishing node may go offline, that tombstone may never happen, leaving stale data visible to consumers indefinitely.

A TTL mechanism must satisfy two constraints:

1. Tamper-proof: A relay or malicious peer must not be able to extend or shorten a file's lifetime. The expiry must be committed by the author's signing key at publication time.

2. Distributed-clock-tolerant: Nodes in the mesh have unsynchronized clocks. Expiry enforcement must tolerate reasonable clock skew without causing files to oscillate between visible and invisible across peers.

Decision

Signed ValidFor field

Add a ValidFor field (time.Duration) to the Signature struct. When non-zero, it represents a TTL committed into the ed25519-signed buffer at publication time.

The file is considered expired on every node once SignedAt + ValidFor is in the past.

When ValidFor is zero, the file has no expiry and persists until explicitly deleted via tombstone. This is the default and preserves backward compatibility.

Signature format

The signed buffer is constructed as:

type (1) | network_id (32) | name (var) | signed_at (15) | size (8) | hash (32) [ | valid_for (8) ]

The valid_for tail is conditional: it is only appended when ValidFor > 0. This means signatures without a TTL produce identical signed buffers to the pre-TTL format, ensuring wire compatibility between old and new nodes for non-expiring files.

When ValidFor > 0, the 8-byte big-endian encoding of the duration in nanoseconds is appended. Because this changes the signed buffer length, a Man In The Middle that strips ValidFor (setting it to zero to make the file appear non-expiring) will cause verification to fail.

Expiry semantics

Expired(now time.Time) bool returns true when ValidFor > 0 && now.After(SignedAt + ValidFor).

Expired files should be hidden from consumers immediately.

The on-disk file and signature may persist briefly after expiry. A background sweeper routine walks the signature store at a configurable interval (sweep_interval, default 60s), deleting expired signatures and their corresponding files.

Sweeper design

The sweeper should operate in three passes within a single invocation:

1. Collect: walk all signatures, identify those where Expired(now) is true.
2. Delete signatures: remove all expired signatures from BoltDB and commit the transaction.
3. Remove files: best-effort os.Remove for each expired file. Failures are logged but do not fail the sweep.

This ordering ensures the only possible inconsistency is orphaned files on disk for signatures that are already gone. Since readers check the signature store first, orphaned files are invisible. The integrity checker cleans them up on the next restart.

The sweeper should not produce tombstones. Expiry is independently evaluated by every node from the signed metadata -- there is nothing to gossip. This avoids amplifying cluster traffic for what is a local garbage-collection operation.

Clock skew tolerance

Different nodes have different clocks. Without tolerance, a file that just expired on one node might be re-imported from a peer whose clock is slightly behind, then expire again, causing unnecessary churn.

The gossip ingress filter should apply a configurable clock_skew_tolerance (default 2 minutes): a remote signature is rejected only if its effective expiry (SignedAt + ValidFor) is more than clock_skew_tolerance in the past relative to the local clock.

The HTTP PUT endpoint (local client to local daemon) should use no tolerance -- the client and daemon share a clock.

Boundary validation

ValidFor should be validated at all ingress points:

Boundary	Validation
CLI (--expires-in)	>= 0, <= max_valid_for
HTTP PUT (X-Validfor header)	>= 0, <= max_valid_for, not already expired
Gossip import	>= 0, <= max_valid_for, not expired beyond clock skew tolerance

max_valid_for is a configurable field (default 30 days) that caps the maximum TTL a client or peer can set, preventing absurdly large durations.

Configuration

Three new configuration fields should be added to Config:

Field	Default	Description
sweep_interval	60s	How often the sweeper runs
clock_skew_tolerance	2m	Slack for rejecting expired remote sigs
max_valid_for	30 days	Upper bound on per-file TTLs

CLI

The file update command should gain an --expires-in flag:

dm file update --expires-in 10m myfile.txt

A value of 0 (the default) means no expiry.

Consequences

Authors of ephemeral files will be able to express their intent at publication time. Files will expire automatically across the entire cluster without requiring a follow-up tombstone or the publishing node to remain online.

The signature format will remain backwards compatible for non-expiring files. Nodes that do not use TTLs will see no change in wire format or behaviour.

Expired files will be hidden immediately on read but cleaned up lazily by the sweeper. The maximum staleness window will be bounded by sweep_interval for on-disk cleanup, but reads will always be consistent because Get() checks Expired() before returning.

The clock skew tolerance trades a small window of continued visibility (up to clock_skew_tolerance after true expiry) for stability -- files will not bounce in and out of visibility as they propagate through peers with slightly different clocks.

The max_valid_for cap will prevent a client or peer from setting a TTL so large it is indistinguishable from no expiry while still carrying the overhead of sweeper evaluation.