Work

A production Postgres kept breaking under the reads on its hot path. Caching was the obvious fix and invalidation the risk, so the team was careful about it, with good reason. There was talk of heavier machinery, Kafka among it, which I thought was more than the problem called for.

Rather than argue the point, I built a prototype the team could pull apart, and we went through it over a few technical sessions. The dataset was small and stable enough to sit in memory, so invalidation could be driven straight off the Postgres replication log. It came out ahead of every alternative on cost, on how far it scaled, and on how much there was to maintain.

Peaked at 300k requests per second on two commodity vCPUs. The incidents stopped.

A set of legacy TypeScript services, bundled into one NestJS monolith, misbehaved under load and was hard to run on-prem.

I laid the groundwork to port them into a fully instrumented Go monorepo: a data access layer that minimized roundtrips and eliminated excess queries, plain hand-written SQL that both the query planner and people could reason about, and CLI tooling that makes the services straightforward to run.

The services moved one at a time. Each went out first as a passthrough proxy in front of its TypeScript upstream, which gave it its first metrics and traces, taken from production traffic before a line of logic moved. That took the guesswork out of the migration. The Go implementation landed behind the proxy later, in increments.

Each small Go pod replaced five large TypeScript ones. The callers never knew the port happened.

A micro-mobility operator ran tens of thousands of scooters and e-bikes across several makes of hardware, each speaking its own protocol. Their Rails backend carried all of it: wired into every make, model and protocol, and swamped with the traffic.

The backend had no need to know what sat on the other end. I wrote a set of Go servers, one per protocol, each translating a vehicle's own protocol into a single common one, then refactored Rails to speak only that and stay blind to the hardware underneath. After the cutover I ran, supported and extended the platform.

Cut over with the fleet in service across the US, without taking a single vehicle off the street.

A distributed system of around thirty gRPC and HTTP Go services exported nothing more than stock Kubernetes metrics and non-uniform logs. Instrumenting them one by one would have given thirty different answers, drifting apart with every release.

Instead of instrumenting services, I changed how they get built. The framework builds a program out of named components: it runs each one, wires up health, startup and readiness probes, and owns the lifecycle down to graceful shutdown. One import and a few lines, and the program traces, measures and logs like every other one in the system. It covers HTTP and gRPC on both the server and the client side, so a request stays traced across every hop, the client end included, which most setups leave dark.

On its fifth major version, running tens of Go services. OpenTelemetry is no longer wired in service by service.

The services in a distributed stack had drifted onto contracts that no longer matched.

I put every protobuf definition in one repository, then built a pipeline on top of git and GitHub Actions that generates each language's client library and publishes it to its own repo. Consumers pull them like any other git dependency.

Contracts ship as commits to one repository, and every caller has a matching client on the next CI run, git log included.

Every engineer develops against their own Kubernetes cluster. The Python tool that set one up required tearing the whole loop down to change which version of a service was running, so trying a colleague's branch meant losing the session you already had going.

I replaced it with a Go tool on a new design. Each project can be pointed at its catalog default, at a branch, or at a directory on your machine, and only that project rebuilds while the rest of the stack keeps serving. Projects load and build on their own, so a slow one holds nobody up and a broken one doesn't take the session down with it.

Trying a feature now takes a single command.

[redacted] link <project> git|dir <REF|DIR>