Notes from the backend

Architecture, code, data stores and the infrastructure underneath — written down while the details were still fresh.

Deep dives

  • An MCP server for internal schema inspection: security, limits, and lessons

    In early 2026 we built an internal Model Context Protocol (MCP) server allowing engineering assistants to inspect production-replica database schemas, table relationships, and index definitions. The goal was to stop…

  • The RabbitMQ cluster that deadlocked on unroutable confirms

    On a Tuesday evening, our message publishing pipeline locked up completely. Producers publishing transactional events began timing out after five seconds, worker queues starved, and web processes backed up until…

  • Zero-downtime column rename under 400 writes a second

    In MySQL 8, RENAME COLUMN is theoretically an instantaneous metadata-only operation with ALGORITHM=INPLACE. In practice, running it against a table handling four hundred writes a second during business hours is…

  • The circuit breaker that tripped on its own health check

    On a Thursday afternoon in mid-September, our checkout service tripped its circuit breaker to the upstream payment provider. For forty-two minutes, every customer payment was rejected with an immediate 503…

  • A second year of asymmetric visibility in a domain model

    Twelve months ago we converted forty aggregate roots from boilerplate getter methods to PHP 8.4 asymmetric visibility. At the time it felt like an unambiguous cleanup: deleting twenty-four hundred lines…

  • An incident with no engineer

    An outage at 04:00 in August, resolved before anybody woke up. A worker crash loop that would have been a two-hour outage in 2023.


Recent notes


Latest snippets