Reliability
How we stay up on the one morning everyone logs in at once
Result day is the moment a school ERP earns or loses trust: every parent opens the app within the same ten minutes. Here's what we found when we tried to break ours on purpose, what we changed, and exactly how far that gets us today — sourced, and reproducible by anyone with the repository.
What we measured
We built a load test that reproduces result day directly: marks are published, report cards are generated for real (a real headless-browser render, not a mock), and then parents open the app at a steadily increasing rate while buses are still pinging their location and staff are still working the console — because on a real result day, none of that traffic politely waits its turn. The metric we report is p95 latency (95% of requests faster than this) on the parent's results read, against an 800 ms bar.
One CPU core was the ceiling
The first thing we found was not a database problem. A single app process pins one CPU core while the database and the rest of the machine sit mostly idle — a database connection-pool cap made this look like a database issue at first, until sizing the pool properly exposed the real ceiling underneath it. Uncached, on one core, we broke the 800 ms bar at 120 opens/second (p95 948 ms in our own re-run).
Caching the composed read
A parent's results read does several queries and some formatting work every time, which is exactly the CPU cost the one-core ceiling above is scarce on. Caching the composed result (not the raw queries) cut that cost enormously: the same 120 opens/second, same core, same hour, dropped from p95 948 ms to p95 22 ms — and held there past 200/second before failing.
Scaling out multiplies the ceiling
Because the constraint is CPU cores, not the database, running more app processes — one per core, behind a load balancer — multiplies capacity rather than fighting for the same core. Four processes with caching held p95 at 59 ms at 300 opens/second (over 1,000 parents in a five-minute window) with zero request failures, and broke somewhere between 300 and 450.
Report cards, generated in bulk
Report-card PDFs were originally rendered by launching a fresh headless browser per student. Reusing one browser across a class's worth of students measurably cuts that cost — in our own re-run, roughly 2.2-2.6x faster than the old per-student approach for a 50-student class.
The architecture underneath
Every school's data lives in the same database, isolated by Postgres row-level security (RLS): every tenant-scoped table carries a policy that binds queries and writes to the tenant on the connection, and the application's database role cannot bypass it. That isolation is enforced at the database layer, not left to every query remembering aWHERE tenant_id = … clause.
The result-read cache we added for this work extends that same boundary rather than opening a gap in it: every cache key is built by one function that takes the tenant from the same request-scoped source RLS uses, cannot be called without it, and cannot be constructed by hand elsewhere in the codebase — a lint rule enforces that at review time, and a cross-tenant test exercises it end-to-end, including a sweep asserting every live cache key actually carries a tenant. A cache sitting next to a multi-tenant database is exactly the kind of place a tenancy bug hides, so this was built and tested as a security boundary, not bolted on as a pure performance detail.
What this is, and isn't
This is a load test on hardware sized like our production box, not a production traffic record. We don't have a live paying school yet, so there is no production uptime or latency history to publish — only what we can measure and reproduce ourselves before that day comes. Every number above comes from one session's own test run against a 4-core sandbox; the code paths and the architectural findings (one core per process, caching removes CPU work, scaling out multiplies capacity) are facts about the software and transfer directly. The exact opens-per-second figures are specific to that sandbox on that day and do not.
We know that because we checked: re-running the identical test in a later session produced a different absolute number for the same configuration — not once, but on two separate occasions. All three sessions agree on the mechanism and the direction of every fix; none of them agree with each other on the millisecond figure. We're publishing that rather than picking whichever run reads best, because a reliability claim that hides its own measurement noise is worse than no claim at all.
Full methodology, every run's raw output, and the exact commands to reproduce it yourself: P3.11-load-results.md, alongside the two prior sessions'P3.9andP3.10results in the same repository.
Live status and the source
Our status page shows whether our own systems are responding right now — a live check, not a monitoring platform, and we say so plainly there too. The load-test code itself is open in the repository linked above; nothing on this page asks you to take our word for it.