Skip to content
AITroveRead. Build. Understand.
Reading list
Make this comfortable

Framed journals: detect a torn tail before replay

Last updated: 7 Oct 20269 min read
tutorial
IntermediateBy AITrove Editorial

A framed append journal stores each transaction as a length, checksum, and complete payload. This single-writer example puts all page replacements for one transaction into one frame, writes it to an existing file, then calls fsync before acknowledging the append. Recovery scans a contiguous valid prefix. A short final header or payload represents a torn tail and is truncated before another append; a full frame with a bad checksum raises for manual recovery instead of silently discarding potentially committed records after it. A recovered map is rebuilt solely from complete validated frames. The example does not write separate data pages, so it needs no undo for uncommitted page contents.

Operational case

The first complete frame replaces P-19 and creates P-26. Eleven bytes of a later frame for P-52 are injected without a complete payload. Recovery yields P-19 and P-26, removes the incomplete tail, and leaves P-52 absent. The demonstration controls byte truncation; it does not simulate a real power cut or prove the storage device honored its flush request. A transaction frame that is fully present and checksum-valid may still have been written without a successful fsync in another failure schedule, so acknowledgment and replay policy must be defined together.

Working Python program

python
"""Single-writer append journal: fsync complete frames, truncate torn tail."""

import json
import os
import struct
import tempfile
import zlib
from pathlib import Path

HEADER = struct.Struct(">II")
MAX_RECORD = 1_048_576


def encode_transaction(changes):
    payload = json.dumps(changes, sort_keys=True, separators=(",", ":")).encode("utf-8")
    if len(payload) > MAX_RECORD:
        raise ValueError("transaction too large")
    return HEADER.pack(len(payload), zlib.crc32(payload)) + payload


def append_transaction(journal_path, changes):
    frame = encode_transaction(changes)
    with open(journal_path, "ab", buffering=0) as journal:
        written = journal.write(frame)
        if written != len(frame):
            raise OSError("short journal write")
        os.fsync(journal.fileno())


def recover(journal_path):
    pages = {}
    with open(journal_path, "r+b", buffering=0) as journal:
        while True:
            frame_start = journal.tell()
            header = journal.read(HEADER.size)
            if not header:
                break
            if len(header) < HEADER.size:
                journal.truncate(frame_start)
                os.fsync(journal.fileno())
                break
            length, checksum = HEADER.unpack(header)
            if length > MAX_RECORD:
                raise ValueError("invalid record length")
            payload = journal.read(length)
            if len(payload) < length:
                journal.truncate(frame_start)
                os.fsync(journal.fileno())
                break
            if zlib.crc32(payload) != checksum:
                raise ValueError("checksum mismatch; manual recovery required")
            changes = json.loads(payload)
            if not isinstance(changes, dict) or not all(isinstance(key, str) and isinstance(value, str) for key, value in changes.items()):
                raise ValueError("invalid transaction")
            pages.update(changes)
    return pages


with tempfile.TemporaryDirectory() as temporary_directory:
    journal_path = Path(temporary_directory) / "pages.log"
    append_transaction(journal_path, {"P-19": "balanced", "P-26": "new-leaf"})
    with open(journal_path, "ab") as journal:
        journal.write(encode_transaction({"P-52": "unfinished"})[:11])
    print(recover(journal_path))
    print(journal_path.stat().st_size == len(encode_transaction({"P-19": "balanced", "P-26": "new-leaf"})))

Output

Output
{'P-19': 'balanced', 'P-26': 'new-leaf'}
True

Time, space, and tradeoff

Encoding a transaction with B payload bytes takes O(B) time and space; appending writes O(B) bytes and pays a storage-dependent fsync latency. Recovery scans O(L) bytes for a journal of length L and stores O(P) latest page values plus one bounded record buffer, where P is the number of distinct pages. Replaying the entire file on startup grows with history until a separately designed checkpoint is added. CRC detects many accidental corruptions but is not an authentication code and cannot repair data. The model assumes one writer, an existing durable filename, a file system with meaningful fsync behavior, and no concurrent reader that races truncation. It is a small disk-backed journal, not a complete database transaction engine.

Common Mistakes

  • Do not acknowledge a frame before checking that its write and fsync succeeded.
  • Do not skip over a checksum failure in the middle and silently apply later records.
  • Do not append new frames behind a torn tail before recovery trims that tail.
  • Do not claim that CRC or fsync alone guarantees safety against every hardware fault or directory-entry loss.

Connected lessons

Test this contract in the live depot audit project, then check the operations quiz.

Sorted runs and tombstones: model an LSM read path adds a related operation contract.

Merkle trees: verify an indexed scan record adds a related operation contract.

data structures
range-query-structures
Storage details