Observability and research

Context

Many of my assignments started with questions: where is the extra time is spent, would this tool fit, and what happens under production-like traffic? I build the smallest thing (or large, it depends) that can answer the question, then document what I find.

Alongside this, I built a diagnostics suite that automates call stack inspection and visualizes latency bottlenecks per sql query from a http request. I also standardized logging, simulated production-like traffic with Locust, and ran high-availability experiments on open-source systems.

Approach

Bird’s eye view. Every customer request passes through the API gateway, but there was no single dashboard for its traffic. I built one using Grafana, Loki, and Prometheus to track client traffic, HTTP status codes, and captured errors. It supports searches by request ID and correlation ID, filtering by service-specific labels I added, and narrowing results by client or email. These filters can be combined to trace problems across requests and services.

Test the assumption with the cheapest experiment. We manage trino for federated and it was setup aith auth & authz by poining it to our api-gateway. Thereafter, even trivial queries were taking 150–300 ms on average. I suspected the requests were passing through the API gateway, so I captured packets in a debug pod for a few seconds after a fresh start and analyzed it with wireshark. They weren’t. That ruled out the gateway path as the suspected source of latency, without spending more time investigating the wrong component.

Use the tool, then document the decision. Every proof of concept gets a written assessment of its benefits, integration cost, complexity, product compatibility, and benchmarks. For Linkerd, I measured the latency overhead introduced by its mTLS proxies and how it accumulates across chained service calls. For Superset, I explored integration with our existing data services to query our Vue platform-specific data, with a plan to inherit roles and permissions from the api-gateway. The goal was to establish what each tool would cost to adopt, not just whether it worked.

Cast guidance on agents. I built aidlc, a Claude skill for our monorepo that the internal team has adopted. Given a detailed issue, it clarifies open questions with the developer, runs through the development loop, and raises a concise draft PR. Its instructions stay deliberately light: prefer the programming lang’s standard library, sanitize large changes at the end, and tier based test which ultimately run against a development kubernetes cluster. It also keeps brief notes on developer corrections to avoid repeating mistakes as content in the monorepo. Heavy-handed instructions tend to fight the model’s existing everimproving capabilities, so the skill adjusts its defaults rather than trying to control every step. This skill is necessary to improve team’s cadence by not repeating themselves everytime when a new task is taken up.

Boundaries

• The dashboard covers API gateway traffic. Sentry-style error tracking and distributed trace spans are outside its scope.
• Linkerd, Superset, and Trino were evaluated in development but never deployed to production because there were no active first-party consumers. The findings remain documented for adoption.
• AI-DLC deliberately adds only a small layer of preferences over the model’s defaults. As those defaults improve, parts of the skill may become unnecessary.