The Question Was Simple
A landlord asked whether we could prove that payment records had not been changed after the fact.
That question matters more than it sounds. It is not really about one payment. It is about whether the financial record can be trusted when there is pressure on it.
At the time, we had ordinary application protections. That was not enough.
The First Step Was Immutability At The Application Layer
The initial improvement was straightforward: financial ledger entries should not be updated or deleted in normal application flow. Corrections should happen through reversal entries, not edits.
That follows normal accounting logic. If something was wrong, the record of the mistake should remain visible and the correction should be visible too.
class ImmutableQuerySet(models.QuerySet):
def update(self, **kwargs):
raise TypeError("This record is immutable. Use a reversal entry.")
def delete(self):
raise TypeError("This record is immutable. You can't delete history.")
And at the instance level:
class LedgerEntry(models.Model):
def save(self, *args, **kwargs):
if not self._state.adding:
raise ValueError("LedgerEntry is immutable — mistakes get reversed, not erased.")
super().save(*args, **kwargs)
We Also Needed Tampering To Be Detectable
Application-level immutability helps, but it does not answer a harder question: what if someone changes the underlying row directly?
To make tampering detectable, each entry includes a hash of its own contents plus the hash of the entry before it.
That creates a chain. If an earlier entry is altered later, the chain stops verifying cleanly.
def compute_entry_hash(
*,
organization_id,
amount,
currency,
reference_type,
reference_id,
timestamp,
previous_hash,
) -> str:
payload = "|".join([
str(organization_id),
str(amount),
currency,
reference_type,
str(reference_id),
timestamp.isoformat(),
previous_hash or "",
])
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
Verification then becomes a simple walk through the sequence:
def verify_ledger_integrity(organization) -> bool:
expected_previous = ""
for entry in LedgerEntry.objects.filter(
organization=organization
).order_by("created_at", "id"):
if entry.previous_hash != expected_previous:
return False # Chain is broken — something was changed
expected_hash = compute_entry_hash(...)
if entry.entry_hash != expected_hash:
return False # This entry was tampered with
expected_previous = entry.entry_hash
return True
Then Concurrency Broke The Assumption
The next problem came from concurrent writes. Two payment events for the same organization could arrive close together and both try to append to the chain at nearly the same time.
If both workers read the same "last entry" before writing, they can compute hashes from the same parent and create an invalid branch.
The Fix Was Write Serialization
To prevent that, writes for a given organization are serialized with a row lock:
@transaction.atomic
def post_entry(organization, amount, ...):
Organization.objects.select_for_update().filter(pk=organization.pk).get()
previous_hash = get_last_entry_hash(organization)
entry_hash = compute_entry_hash(..., previous_hash=previous_hash)
LedgerEntry.objects.create(..., entry_hash=entry_hash, previous_hash=previous_hash)
That forces the second write to wait until the first finishes. Once the first commit lands, the next write reads the new last entry and computes the correct next hash.
What This Means In Practice
For a landlord or operator, the important part is simple: the financial record is append-only, corrections are visible, and integrity checks can detect whether the chain has been altered.
For an engineer, the important part is also straightforward: application immutability is not enough on its own. You need tamper evidence and write serialization too.
That landlord's original question still works as the best test: if someone asks whether the payment history could have been changed after the fact, can the system answer with evidence instead of confidence?