Runtime and api-gateway systems

Context

Every customer request passes through the API gateway. What starts as a single request can involve another deployment, a route, a permission check, or a production incident. How the gateway behaves under load and failure affects every client.

Alongside this, I built storage quotas, worked on selective versioned deployments of asynchronous workers during Helm upgrades, and wrote a proposal for complement authorization pushdown using OPA and Rego.

Approach

One commit or one rollback. The earlier version of api-gateway service could leave inserts and other operations partially applied when a request failed unexpectedly or the service restarted midway through. I centralized transaction handling so that all operations within a request either commit together or roll back together, rather than leaving the system in a inconsistent state.

Freeing up the main thread. The request path relied on synchronous APIs, with several points stalling on thread contention, and caused low CPU util through GIL (Global Interpreter Lock). I moved it to async, using Locust load tests and profiling to identify bottlenecks and guide each change. The result was higher throughput with fewer guvicorn workers (we use FastAPI). Combined with the api-gateway refactor, these changes increased request throughput by 60% in internal Locust benchmarks.

Costs savings in zen mode. Per-user code servers use dynamic routing, with KEDA scaling pods based on Prometheus metrics. When a pod has no active user and no background script running, it can scale down. I profiled the idle resource footprint of these pods and used that measured baseline to set the scale-down threshold. Customers can confidently leave their desk, knowing that we don’t shutdown their running workloads, if the pods ever goes away, it is always when it’s idle.

Writeups for informed decisions. I wrote a proposal for pushing authorization checks down to the gateway using OPA and Rego. It covers the problems this would solve, the considerable integrations points introduced by Rego policy, and how it could coexist with the existing authorization system as an optional integration. The implementation decision remains with the product team as of date. Certrain decisions are made better with data that services own themselves, an api-gateway cannot ask for it and it is an antipattern leads to unnecessary coupling, in those cases OPA could help moving decision point elsewhere without compromising integrity.

Boundaries

• The 60% throughput improvement comes from internal Locust benchmarks and reflects the combined changes, not the async migration alone.
• The scale-down threshold is based on the measured idle resource footprint of the per-user code server pods.
• The authorization pushdown remains a written proposal, pending a product decision.