Straxion AI: Building a Multi-Agent System That Doesn't Suck

The Brief (and Why It Scared Me)

"We need an AI platform. Think ChatGPT, but for our specific domain. Can you build it in two weeks?"

I've learned to never say no to ambitious timelines. But this one made me pause. They wanted:

• Multi-agent orchestration (agents calling other agents)
• RAG pipeline with domain-specific knowledge
• Real-time responses under 2 seconds
• Scale to 10K requests/day from day one

Oh, and their current prototype? Held together with duct tape and prayer, dying at 800 requests/day.

Week 1: The Architecture That Almost Killed Us

Day 1-2: The Agent Design Debate

My first instinct was to use LangChain. Everyone uses LangChain, right?

Wrong.

After 6 hours of wrestling with LangChain's callback hell and opaque debugging, I threw it out. Switched to LangGraph - Anthropic's newer framework that actually makes sense.

Here's the agent setup we landed on:

import { StateGraph } from "@langchain/langgraph";
import { ChatAnthropic } from "@langchain/anthropic";

const model = new ChatAnthropic({
  modelName: "claude-opus-3-5",
  temperature: 0.3,
  maxTokens: 4096,
});

// Router agent - decides which specialized agent to call
const routerNode = async (state) => {
  const { query } = state;

  const classification = await model.invoke([
    { role: "system", content: "Classify this query into: research, analysis, or synthesis" },
    { role: "user", content: query }
  ]);

  return { agentType: classification.content };
};

// Build the graph
const workflow = new StateGraph({
  channels: {
    query: null,
    agentType: null,
    context: null,
    response: null,
  }
});

workflow.addNode("router", routerNode);
workflow.addNode("research", researchAgent);
workflow.addNode("analysis", analysisAgent);
workflow.addNode("synthesis", synthesisAgent);

// Conditional routing based on agent type
workflow.addConditionalEdges(
  "router",
  (state) => state.agentType,
  {
    research: "research",
    analysis: "analysis",
    synthesis: "synthesis",
  }
);

This failed on Day 3.

Day 3: When Everything Broke

The routing was too slow. Each classification took 800ms. Multiply that by potential multi-hop reasoning? We were looking at 3-5 second responses.

Unacceptable.

The fix: Stop trying to be clever. Implemented a simple keyword + embedding similarity hybrid router. 120ms average. Sometimes boring wins.

// The "boring" router that actually works
async function fastRouter(query: string) {
  // Quick keyword checks first
  if (query.match(/how|what|why/i)) return "research";
  if (query.match(/analyze|compare/i)) return "analysis";

  // Fall back to embedding similarity for edge cases
  const embedding = await embedQuery(query);
  const similar = await vectorStore.similaritySearch(embedding, 1);

  return similar[0].metadata.agentType;
}

Day 4-5: RAG Pipeline (Or: How I Learned to Stop Worrying and Love Pinecone)

Initially tried building a custom vector store. Bad idea. Wasted 8 hours.

Switched to Pinecone. Their serverless tier is magic for MVPs:

import { Pinecone } from "@pinecone-database/pinecone";

const pc = new Pinecone({
  apiKey: process.env.PINECONE_API_KEY,
});

const index = pc.index("straxion-knowledge");

// Chunking strategy that actually worked
function chunkDocument(doc: string, chunkSize = 512, overlap = 64) {
  const chunks = [];
  let start = 0;

  while (start < doc.length) {
    const end = Math.min(start + chunkSize, doc.length);
    chunks.push(doc.slice(start, end));
    start += (chunkSize - overlap);
  }

  return chunks;
}

// Hybrid search: semantic + keyword
async function retrieveContext(query: string) {
  const [semanticResults, keywordResults] = await Promise.all([
    index.query({
      vector: await embed(query),
      topK: 5,
      includeMetadata: true,
    }),
    // Pinecone doesn't do keyword search, so we cheat
    index.query({
      vector: await embed(query),
      topK: 10,
      filter: { keywords: { $in: extractKeywords(query) } },
    }),
  ]);

  // Merge and dedupe
  return mergeResults(semanticResults, keywordResults);
}

Week 2: The Performance Nightmare

Day 6-7: Why Is Everything So Slow?

Production traffic came early. 2K requests on Day 6. Our beautiful architecture was averaging 4.2 seconds per request.

The culprit? Sequential processing everywhere.

// BAD: Sequential hell
async function handleQuery(query: string) {
  const context = await retrieveContext(query);      // 800ms
  const classification = await classifyQuery(query); // 120ms
  const response = await generateResponse(query, context); // 2100ms

  return response; // Total: ~3000ms
}

// GOOD: Parallelize everything possible
async function handleQueryFast(query: string) {
  const [context, classification] = await Promise.all([
    retrieveContext(query),
    classifyQuery(query),
  ]);

  const response = await generateResponse(query, context, classification);

  return response; // Total: ~2100ms (context retrieval happens during classification)
}

900ms shaved off. But not enough.

Day 8-9: Streaming Saved Us

Instead of waiting for complete responses, we streamed tokens:

import { streamText } from "ai";

export async function POST(req: Request) {
  const { query } = await req.json();

  const context = await retrieveContext(query);

  const result = streamText({
    model: anthropic("claude-opus-3-5"),
    messages: [
      {
        role: "system",
        content: `You are an expert assistant. Use this context:\n${context}`,
      },
      { role: "user", content: query },
    ],
  });

  return result.toDataStreamResponse();
}

Perceived latency: 180ms (time to first token) Actual latency: Still 2.1s, but users don't care anymore

This is the dirty secret of "fast" AI apps. Make users feel like it's instant.

Day 10-11: Caching & Cost Optimization

Our Claude API bill was looking scary. $400 in 3 days of testing.

Solution: Aggressive prompt caching.

// Cache the RAG context - it rarely changes
const response = await anthropic.messages.create({
  model: "claude-opus-3-5",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: "You are a helpful assistant...",
    },
    {
      type: "text",
      text: `Here is the knowledge base:\n${knowledgeBase}`,
      cache_control: { type: "ephemeral" }  // Cache this!
    }
  ],
  messages: [{ role: "user", content: query }],
});

Result: API costs dropped 68%. Same responses, way cheaper.

Day 12-13: The Bugs You Don't See in Demos

Bug 1: Agents would occasionally hallucinate and route to non-existent agents. Fix: Added a fallback "general" agent for unknown classifications.

Bug 2: Rate limiting wasn't per-user, it was global. One power user killed the API for everyone. Fix: Implemented per-user rate limiting with Upstash Redis.

import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

const ratelimit = new Ratelimit({
  redis: Redis.fromEnv(),
  limiter: Ratelimit.slidingWindow(10, "60 s"),
  analytics: true,
});

export async function POST(req: Request) {
  const userId = req.headers.get("x-user-id");
  const { success } = await ratelimit.limit(userId);

  if (!success) {
    return new Response("Too many requests", { status: 429 });
  }

  // ... rest of handler
}

Bug 3: We had a memory leak. Long conversations would crash the Node process. Fix: Truncate conversation history to last 10 messages. Controversial, but it worked.

Day 14: Ship It

Final metrics after launch:

• Average response time: 1.8s (down from 4.2s)
• P95 response time: 2.4s
• Requests handled: 12K/day (150% above target)
• API cost per 1K requests: $2.40
• Uptime: 99.8% (one Redis outage on Day 3 of production)

What I'd Do Differently

1. Start with streaming from Day 1. Trying to optimize for actual speed first was a mistake. Perceived speed is what matters.

2. Don't build custom infrastructure. We wasted time on a custom vector store. Pinecone worked fine.

3. Prompt caching should be table stakes. We left money on the table for a week.

4. Test at scale earlier. We only load-tested on Day 10. Should've been Day 4.

The Tech Stack That Worked

• LLM: Claude Opus 3.5 (via Anthropic API)
• Framework: LangGraph for agent orchestration
• Vector DB: Pinecone serverless
• Embeddings: text-embedding-3-large (OpenAI)
• Frontend: Next.js 15 with App Router
• Caching: Upstash Redis
• Deployment: Vercel (API routes on Edge)
• Monitoring: Axiom (logs) + Highlight.io (session replay)

Real Talk

This project was chaos. We shipped in 14 days because we cut scope ruthlessly and accepted "good enough."

The first version had bugs. The agents weren't perfect. The UI was basic.

But it worked. And that's what mattered.

If you're building AI products, ship imperfect things fast. You'll learn more from 100 real users than 1000 hours of planning.

---

Building something similar? Let's talk. We've made all the mistakes so you don't have to.