Architecture — Tanishq Kumar

Note 1: Own the agent loop

99 percentfunction-calling success rate

Agent harness · Anthropic and OpenAI SDKs

What I ran into: Wrapper frameworks added dependencies and unpredictability, and hid the steps I most needed to see when an agent went wrong.
What I did: Wrote an agent harness from scratch over the native Anthropic and OpenAI SDKs rather than on a general-purpose agent framework, replacing an earlier DSPy pipeline with a single AgentBuilder factory. Constrained decoding against Pydantic v2 schemas reached a 99% function-calling success rate and removed 35% of runtime dependencies.
What I took from it: A framework is a fine way to start and an expensive thing to debug. Once agents are the product, owning the loop, the tool calls, and the output schemas is what makes them predictable.

from 15 minutes to under 100 millisecondsbuild time

Lexer, parser · AST · DAG

What I ran into: Pipelines assembled by an LLM took around 15 minutes to build and could hallucinate schemas that didn’t exist.
What I did: Replaced generative assembly with a custom lexer and recursive descent parser. Business formulas are tokenized into typed ASTs and evaluated as in-memory DAGs, which removed schema hallucination and cut build time from 15 minutes to under 100 ms.
What I took from it: Models are good at understanding what someone means; they are a poor way to produce something that must be exactly right. Where correctness is the requirement, a deterministic compiler wins on speed and on trust.

40 percent loweragent failure rate

MCP · one tool layer

What I ran into: Agents needed real backend capabilities, but tool calls leaked UI side effects into the agent’s context.
What I did: Built one tool layer that serves both the in-app agent and external clients (Claude Desktop, Cursor, CLI agents) through MCP servers. An abstract ToolSpec keeps UI side-channels out of agent payloads; reads are idempotent and every write is a staged, transactional diff. Agent failure rates dropped by 40%.
What I took from it: A tool is a contract, not an integration. Keep its side effects out of the model’s context, and where it runs, in the app or over MCP, becomes a transport detail.

under 20 millisecondscached full-text search

FastAPI · PostgreSQL

What I ran into: Endpoints slow down for predictable reasons: blocking I/O, chatty queries, repeated reads, and responses that load more than the client needs.
What I did: Kept I/O async from FastAPI handlers to the database, pooled connections, optimized queries (indexes, no N+1s, only the columns a request uses), cached hot reads, and paged large results with cursors and streamed responses. Cached PostgreSQL full-text search answers in under 20 ms.
What I took from it: An API’s speed is mostly decided by how it waits, what it asks the database for, and how much it sends back. Get those right by default and most endpoints never need tuning one by one.

Rust · Axum · SQLx · Tokio

What I ran into: A few services sat on the critical path, where every request’s latency and every core’s cost added up.
What I did: Wrote the performance-critical services in Rust with Axum, SQLx, and Tokio, while the rest of the platform stayed in Python on FastAPI. The Rust services delivered more throughput on less compute.
What I took from it: Python is the right default for moving fast, and Rust earns its place on the hot path. Choosing the language per service, not per platform, is where the performance came from.

84 percentprompt-cache hit rate on OpenAI

Phoenix · Prometheus · Grafana

What I ran into: LLM latency and spend were invisible per tenant, so there was no way to tell which change made things faster or cheaper.
What I did: Ran Arize Phoenix, Prometheus, and Grafana on Kubernetes via Helm, with OpenInference spans and custom instrumentation across services. Restructuring prompts around stable prefixes reached an 84% prompt-cache hit rate on OpenAI, under per-tenant budget caps and p90 tool latency of 50 ms.
What I took from it: Cost and latency are shaped by how prompts and tools are structured, not tuned afterwards. Measure them per request and per tenant from the start, and they become something you design for.