4. Memberlist for clustering

Date: 2025-02-27

Status

Accepted

Context

We need to periodically merge state across participating nodes within a cluster. However, efficiently distributing state within a decentralised system is challenging and fraught with nuance.

The prototype, written in Python, relied upon a naive HTTP-based polling mechanism. We would prefer a robust and proven solution for state distribution.

Decision

After reviewing available solutions, we have decided to proceed with memberlist.

It is designed with eventually consistent state distribution in mind and handles cluster membership for us.

Hashicorp has a solid track record with infrastructure-level projects such as this, which gives confidence that it has been implemented well.

Consequences

By leaning on memberlist to handle cluster membership and state distribution, we can instead focus on how that state is represented and the ways in which it is merged.

It does, however, restrict us from implementing merge strategies which cannot be supported by memberlist's API and design choices.

Experience Report

Overall, our experience with memberlist has been positive. It is performing its role well, and as expected, allowing us to focus on other aspects of the project.

It has, however, taken time to understand its clustering model. There are still some open questions with respect to cluster joining and reconnecting which need to be resolved.