3. Store state in a JSON file

Date: 2025-02-27

Status

Accepted

Context

As the name indicates, Data Mesher will need to store data persistently.

The volume of data is expected to be low, on the order of megabytes, not gigabytes. More reads are expected than writes, with expected throughput on the order of 10/s.

Initially, our state will be relatively simple. Eventually, it is intended to become more flexible with a dynamic schema.

Decision

Several options were considered, such as Bolt DB and sqlite but in the end, we have decided to store state in-memory, with changes flushed to a JSON file when changes need to be made persistent.

Consequences

This has several benefits:

• simpler to reason about.
• avoids third-party dependencies, as this can be implemented with the Go standard library.
• makes it easy to introspect our state on disk, and to manipulate it offline if required.
• amenable to dynamic schema in the future.
• file-format can be re-used for transport.