docs/p2_p3/M23-e2e-encryption.md

M23 — End-to-End Encryption

Spec version: v1.0 (Phase 2) Depends on: M01 (identity, key derivation), M07 (blobs, for prekey bundles), X02 (events, for prekey/session events), X04 (config), pynacl Depended on by: M10 chat (extended), M25 group chat, optionally M07 file encryption

---

1. Responsibility

Provide end-to-end encryption between community members for:

• 1:1 chat (via M10 extension): every chat message encrypted with a per-sender ratchet
• Group chat (M25): per-thread sender keys
• File envelopes (M07 extension): chunks optionally wrapped in a per-recipient envelope

The cryptographic design borrows from Signal but stays simpler:

• X3DH for initial key agreement (one identity key + one signed prekey + one one-time prekey)
• Double Ratchet for per-session forward-secrecy
• Sender keys (Signal-style) for group threads
• Per-blob envelope for file encryption

This module owns the crypto primitives and session state. M10 and M25 own the message protocol that calls into M23.

---

2. File layout

hearthnet/crypto/
├── __init__.py
├── kem.py             # X25519 key agreement (X3DH)
├── ratchet.py         # Double Ratchet, per-session
├── sender_keys.py     # Group sender keys (M25 helper)
├── envelope.py        # File envelope encryption (chunks)
└── prekeys.py         # Prekey bundle storage and publication

The hearthnet/crypto/ directory is a NEW top-level package in Phase 2.

---

3. Public API

3.1 kem.py — X3DH

# hearthnet/crypto/kem.py
from dataclasses import dataclass

@dataclass(frozen=True)
class X25519KeyPair:
    private: bytes        # 32 bytes
    public:  bytes        # 32 bytes "x25519:<base64>"

def x25519_generate() -> X25519KeyPair: ...
def x25519_dh(our_priv: bytes, their_pub: bytes) -> bytes:
    """Computes the shared secret. 32 bytes."""

@dataclass(frozen=True)
class PrekeyBundle:
    """What a recipient publishes so senders can establish a session without them being online."""
    node_id_full:        str
    identity_pub:        bytes               # long-lived; derived from Ed25519 identity via SignedConversion
    signed_prekey_pub:   bytes
    signed_prekey_sig:   bytes               # Ed25519 signature over signed_prekey_pub by identity Ed25519 key
    one_time_prekeys_pub: list[bytes]        # depleted on use
    published_at:        int                 # unix seconds

def derive_identity_x25519_from_ed25519(ed_kp: KeyPair) -> X25519KeyPair:
    """Use the standard nacl conversion. Single x25519 identity key per device."""

def build_prekey_bundle(
    ed_kp: KeyPair,
    *,
    num_one_time: int = E2E_PREKEY_BUNDLE_SIZE,
) -> tuple[PrekeyBundle, X25519KeyPair, list[X25519KeyPair]]:
    """Returns (bundle, signed_prekey_full, one_time_prekeys_full).
       Caller persists the private halves; publishes only the bundle."""

def x3dh_initiator(
    our_identity_x: X25519KeyPair,
    our_ephemeral: X25519KeyPair,
    their_bundle: PrekeyBundle,
) -> tuple[bytes, dict]:
    """Returns (shared_secret, session_init_message).
       session_init_message includes our identity_pub + ephemeral_pub + used_otp_index."""

def x3dh_responder(
    our_identity_x: X25519KeyPair,
    our_signed_prekey: X25519KeyPair,
    our_one_time_prekey: X25519KeyPair | None,
    their_identity_pub: bytes,
    their_ephemeral_pub: bytes,
) -> bytes:
    """Returns shared_secret."""

3.2 ratchet.py — Double Ratchet

# hearthnet/crypto/ratchet.py
@dataclass
class RatchetState:
    """One session, one direction. There are two per session (send + receive)."""
    root_key:      bytes
    chain_key:     bytes
    counter:       int
    epoch:         int
    skipped_messages: dict[tuple[int, int], bytes]  # (epoch, counter) → message_key
    dh_keypair:    X25519KeyPair | None
    remote_dh_pub: bytes | None

@dataclass
class RatchetSession:
    """A bidirectional encrypted session between two NodeIDs."""
    peer_node_id_full: str
    send_state:    RatchetState
    recv_state:    RatchetState

    def is_established(self) -> bool: ...

def init_session_initiator(shared_secret: bytes, peer_dh_pub: bytes) -> RatchetSession: ...
def init_session_responder(shared_secret: bytes, our_dh_kp: X25519KeyPair) -> RatchetSession: ...

@dataclass(frozen=True)
class RatchetMessageHeader:
    dh_pub:     bytes       # current sender's DH pub
    epoch:      int
    counter:    int

def encrypt_message(
    session: RatchetSession,
    plaintext: bytes,
    *,
    aad: bytes = b"",
) -> tuple[RatchetMessageHeader, bytes]:
    """Returns (header, ciphertext). Mutates session.send_state."""

def decrypt_message(
    session: RatchetSession,
    header: RatchetMessageHeader,
    ciphertext: bytes,
    *,
    aad: bytes = b"",
) -> bytes:
    """Verifies + decrypts. Mutates session.recv_state.
       Tolerates up to E2E_RATCHET_MAX_OUT_OF_ORDER out-of-order messages via skipped_messages."""

class RatchetError(Exception):
    """code in {
       'session_not_established','decrypt_failed','out_of_order_too_far',
       'message_too_old','aad_mismatch'}"""
    code: str

# Persistence: sessions are serialised via dataclasses_json into a SQLite table.

3.3 sender_keys.py — Group ratchet

# hearthnet/crypto/sender_keys.py
@dataclass
class SenderKeyState:
    """Per (group, sender) sender-key chain. Each sender broadcasts a chain key
       to all group members at thread create / join."""
    thread_id:       str
    sender_node_id:  str
    chain_key:       bytes
    counter:         int
    signature_keypair: tuple[bytes, bytes] | None    # ed25519, for message signing

@dataclass
class GroupSession:
    """One per thread; holds all members' SenderKeyStates."""
    thread_id:    str
    sender_keys:  dict[str, SenderKeyState]    # sender_node_id → state

def init_sender_key(thread_id: str, sender_node_id: str) -> SenderKeyState: ...
def encrypt_for_group(session: GroupSession, sender_node_id: str, plaintext: bytes) -> tuple[dict, bytes]: ...
def decrypt_for_group(session: GroupSession, header: dict, ciphertext: bytes) -> bytes: ...

def serialise_sender_key_distribution(state: SenderKeyState) -> bytes:
    """Serialise a sender key for sending to other group members.
       MUST be sent inside a pairwise Double Ratchet session, not in cleartext."""

def consume_sender_key_distribution(bytes_blob: bytes, session: GroupSession) -> None: ...

3.4 envelope.py — File envelope

# hearthnet/crypto/envelope.py
@dataclass(frozen=True)
class FileEnvelopeHeader:
    recipient_node_ids: list[str]
    wrapped_keys:       dict[str, bytes]      # node_id_short → wrapped symmetric key
    nonce:              bytes                  # 12 bytes
    chunk_size_bytes:   int

def encrypt_blob_for(
    recipients: list[str],                    # NodeIDs full
    plaintext: bytes,
    sender_kp: KeyPair,
    sessions_provider: Callable[[str], RatchetSession | None],
) -> tuple[FileEnvelopeHeader, bytes]:
    """1. Generate random 32-byte symmetric key
    2. For each recipient: encrypt key with their ratchet session (one-shot use of next message key)
    3. ChaCha20-Poly1305-encrypt plaintext with the symmetric key + nonce
    4. Return (header, ciphertext)"""

def decrypt_blob_for_self(
    header: FileEnvelopeHeader,
    ciphertext: bytes,
    our_node_id_full: str,
    sessions_provider: Callable[[str], RatchetSession | None],
) -> bytes:
    """1. Find our wrapped key in header.wrapped_keys
    2. Decrypt symmetric key via sender's ratchet session
    3. ChaCha20-Poly1305-decrypt"""

3.5 prekeys.py — Publication and consumption

# hearthnet/crypto/prekeys.py
class PrekeyStore:
    """Persists this node's private prekey material and the bundles we've consumed for others."""

    def __init__(self, db_path: Path):
        ...

    def publish_self(
        self,
        ed_kp: KeyPair,
        event_log: EventLog,
        *,
        num_one_time: int = E2E_PREKEY_BUNDLE_SIZE,
    ) -> None:
        """1. Build PrekeyBundle (kem.build_prekey_bundle)
        2. Persist private halves locally
        3. Emit e2e.prekeys.published event with public halves"""

    def get_peer_bundle(self, peer_node_id_full: str) -> PrekeyBundle | None:
        """Look in local cache; if absent, fetch from peer via bus.
        Returns one bundle including one consumable one-time prekey if available."""

    def consume_one_time_prekey(self, our_otp_index: int) -> X25519KeyPair | None:
        """Server-side: when someone uses one of our one-time prekeys, return + remove it."""

    def refill_one_time_prekeys_if_low(self, ed_kp: KeyPair, event_log: EventLog) -> int:
        """If fewer than E2E_PREKEY_BUNDLE_SIZE / 4 remain, publish a new bundle.
        Returns count added."""

---

4. Behaviour

4.1 Session establishment lifecycle (1:1)

Alice wants to send Bob an encrypted message; no session exists.
  ↓
PrekeyStore.get_peer_bundle("ed25519:bob") → fetch from Bob's most recent e2e.prekeys.published event
  ↓
x3dh_initiator(alice_identity_x, alice_ephemeral, bob_bundle) → (shared_secret, init_msg)
  ↓
init_session_initiator(shared_secret, bob_signed_prekey_pub) → RatchetSession
  ↓
encrypt_message(session, plaintext) → (header, ciphertext)
  ↓
Alice sends: chat.message.sent event with data.body = {
   "e2e": true,
   "header": { x3dh_init: init_msg, ratchet_header: header },
   "ciphertext": "<base64>"
}
  ↓
Bob receives event.
   x3dh_responder(...) → shared_secret
   init_session_responder(...) → RatchetSession
   decrypt_message(...) → plaintext
   Emit e2e.session.established event so Alice can clean up retries
  ↓
Subsequent messages: just header + ciphertext (no x3dh_init).

4.2 Group session establishment

Thread creator emits chat.thread.created with members and an ed25519:thread_signing_root.
  ↓
Each member generates a SenderKeyState for themselves in this thread.
  ↓
Each member, in a pairwise loop, sends their sender key distribution to each other member
   inside their 1:1 ratchet sessions (so non-thread-members never see it).
  ↓
Once everyone has everyone's sender keys, encrypt/decrypt happens with sender keys
   (chain ratchet only; no DH ratchet on the group session itself).

When a member is added later, the inviter must re-distribute all existing senders' current chain states to the new member (rewinds to the message they should start being able to read — usually the current state, not history).

When a member is removed, existing sender keys are still known to them. All members must rotate their sender keys to achieve forward secrecy after removal. UI prompts this.

4.3 Out-of-order messages

Up to E2E_RATCHET_MAX_OUT_OF_ORDER (32) skipped message keys are cached per session. Beyond that, out_of_order_too_far is raised; the message is dropped and the sender notified (out-of-band) that they should rekey.

4.4 Rekeying

After E2E_RATCHET_REKEY_AFTER_MESSAGES messages on the same DH ratchet, the next message includes a new DH ephemeral. Standard Double Ratchet behaviour. Transparent to users.

4.5 Session loss recovery

If a node's session state is lost (disk corruption, fresh install with same keys), the peer doesn't know — messages will fail to decrypt. Recovery flow:

1. Decrypting node returns e2e_decrypt_failed via pubsub
2. Sending node sees this and re-initiates X3DH
3. New session replaces old; resends recent messages

UI shows "session was reset" so users know context might have been lost.

4.6 Identity X25519 derivation

We derive a per-device X25519 identity key from the Ed25519 identity key, using libsodium's crypto_sign_ed25519_pk_to_curve25519. This way:

• Only one identity key to maintain
• Anyone with the public Ed25519 (in the community manifest) can derive the X25519 pub
• Signed prekey signatures use the Ed25519 key (already established as device identity)

4.7 Prekey publication

Each node publishes a fresh e2e.prekeys.published event on startup if their last one is > 24h old. The event contains:

• identity_pubkey (X25519 form)
• signed_prekey (with Ed25519 signature)
• one_time_prekeys[] (up to E2E_PREKEY_BUNDLE_SIZE = 20)

Consumers find a peer's bundle by reading their latest e2e.prekeys.published event from the log.

4.8 What is NOT E2E

Even with M23 active:

• Event envelope (sender, recipient, lamport, event_type, wall_clock) is cleartext within the community
• Signatures over events remain valid for community-level audit
• Message metadata leaks to community members (who talked to whom and when), just not content

This is intentional: communities are trust roots; complete anonymity within a community is not a goal.

4.9 File envelope

For file blobs, encrypt_blob_for(recipients, plaintext, ...) produces a single ciphertext, with a small per-recipient header. Senders pick recipients explicitly (e.g. group thread members for an attachment). Bystanders cannot decrypt even if they fetch the blob via M07 file.read.

The blob's CID is the hash of the ciphertext, so the same plaintext sent to different recipient sets has different CIDs. Costs more storage; needed for security.

---

5. Persistence

5.1 Sessions table

CREATE TABLE ratchet_sessions (
  peer_node_id_full TEXT PRIMARY KEY,
  session_blob      BLOB NOT NULL,        -- serialised RatchetSession
  established_at    INTEGER NOT NULL,
  last_used         INTEGER NOT NULL
);

5.2 Group sessions table

CREATE TABLE group_sessions (
  thread_id        TEXT PRIMARY KEY,
  session_blob     BLOB NOT NULL,
  updated_at       INTEGER NOT NULL
);

5.3 Prekey private halves

CREATE TABLE prekey_private (
  kind TEXT NOT NULL,                     -- 'identity'|'signed_prekey'|'one_time'
  index_or_id TEXT NOT NULL,              -- '0' for identity; 'spk_v1' for signed; otp index for OTPs
  private_key BLOB NOT NULL,
  consumed_at INTEGER,                    -- only set for one-time, when used
  PRIMARY KEY (kind, index_or_id)
);

Files locked at 0600. Backed up nightly via hearthnet export (encrypted with user passphrase).

---

6. Errors

RatchetError codes (M23-internal):

• session_not_established
• decrypt_failed
• out_of_order_too_far
• message_too_old
• aad_mismatch

Wire mapping per CAP2 §9:

• e2e_session_missing ← session_not_established
• e2e_decrypt_failed ← decrypt_failed, aad_mismatch
• ratchet_out_of_order ← out_of_order_too_far

---

7. Configuration

config.e2e.enabled                       = True
config.e2e.chat_default_enabled          = True       # new 1:1 chats default to E2E
config.e2e.group_default_enabled         = True
config.e2e.file_default_enabled          = False      # opt-in per blob
config.e2e.prekey_refill_count           = E2E_PREKEY_BUNDLE_SIZE
config.e2e.rekey_after_messages          = E2E_RATCHET_REKEY_AFTER_MESSAGES
config.e2e.max_out_of_order              = E2E_RATCHET_MAX_OUT_OF_ORDER

---

8. Tests

Unit

• test_x25519_dh_symmetric
• test_x3dh_initiator_responder_agree
• test_ratchet_encrypt_decrypt_roundtrip
• test_ratchet_out_of_order_within_window
• test_ratchet_out_of_order_too_far_rejected
• test_rekey_after_n_messages
• test_group_sender_key_distribution_pairwise_only
• test_blob_envelope_recipient_only_can_decrypt

Integration

• test_two_node_first_message_x3dh_session_persists
• test_session_recovery_after_disk_wipe
• test_group_add_member_can_decrypt_subsequent
• test_group_remove_member_cannot_decrypt_after_rotation
• test_file_envelope_2_recipients

Adversarial

• test_replay_old_ratchet_message_rejected
• test_modified_ciphertext_decrypt_fails
• test_one_time_prekey_consumed_once

---

9. Cross-references

What	Where
e2e.* events	CAP2 §7
Encrypted chat body envelope	CAP2 §1.1, §7.2 chat.message.sent
Chat service hook	M10 ext — Phase 2 extension
Group chat	M25
File envelope use	M07 ext
Identity key conversion	M01

---

10. Open questions

1. Post-quantum readiness. X25519 + ChaCha20-Poly1305 is not PQ-safe. Hybrid (X25519 + ML-KEM-768) is Phase 3.
2. Verification of session identity. Signal does safety numbers; HearthNet can do the same. UI ergonomics deferred.
3. Multi-device per identity. If a user has anchor + mobile + laptop, do they share keys or have separate ones? Currently separate (each device is a separate NodeID; group threads include all of them). Could unify with a "linked devices" Phase 3 feature.
4. Forward secrecy on group membership change. Current spec asks members to rotate sender keys on removal. UX of forcing this needs design.
5. Cryptographic auditing. This module should be reviewed by a real cryptographer before going to civil-defence pilots. Listed in THREAT_MODEL_v2.md.