Essay
·Why We Deliberately Sunset Our First Production AI Agent
Architectural post-mortem of Jarvis: supervisor routing pitfalls, multi-agent chaining, dual memory with vector DBs, and retiring bespoke bots for enterprise platforms.
In mid-2024, our infrastructure engineering team deployed Jarvis: our company’s first production AI support agent.
Operating inside our internal developer support channels on Slack, Jarvis served as the first line of defense against incoming infrastructure questions. Built in Python with LangGraph and Model Context Protocol (MCP) integrations, Jarvis autonomously resolved roughly 25% of all inbound developer support requests. It queried live on-call rosters, inspected service catalog metadata, diagnosed CI/CD pipelines, and assisted during active production incident triage. It passed a formal security review and an internal Red Team engagement.
In June 2026, we decommissioned it.
Sunsetting a production system that works reliably and solves real problems can feel counterintuitive. In platform engineering, however, knowing when to decommission software is just as critical as knowing when to build it.
Here is the architectural post-mortem of what we built, the failure modes we encountered, the hard lessons we learned, and why sunsetting bespoke agents is often the mark of a maturing platform.
Incoming Slack Question
│
▼
┌─────────────────┐
│ Supervisor │ (Lightweight Routing Model)
└────────┬────────┘
│
┌─────┴───────────────────────────────┐
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ OpsLevel Agent │ │ PagerDuty Agent │
│ (Service Catalog MCP)│ │ (On-Call Schedule) │
└──────────┬───────────┘ └──────────▲───────────┘
│ │
└────── Step 1 ──> Step 2 ────┘
(Sequential Chaining Pattern)The Architecture: Multi-Agent Supervisor
Jarvis was built around a LangGraph multi-agent supervisor architecture. Rather than relying on a single monolithic system prompt, we decomposed responsibilities into specialized subagents:
- PagerDuty Agent: Queried active schedules and primary/secondary on-call rosters.
- OpsLevel Agent: Inspected our internal service catalog to map service names to owning teams and repositories.
- Coralogix & Observability Agent: Queried log streams and active alert monitors.
- Atlassian Agent: Searched Jira backlogs and Confluence architecture documentation.
- GitHub & ArgoCD Agents: Inspected repository configurations, deployment manifests, and sync statuses.
Each subagent had its own domain-specific system prompt, tool definitions, and MCP connection. The top-level supervisor ingested the user’s initial query and routed the conversation to the relevant expert subagent.
Architectural Learnings and Failure Modes
Operating a multi-agent system in a busy internal engineering channel taught us several non-obvious lessons.
1. The 6-Subagent Problem (Supervisor Routing Noise)
Our first major architectural misstep was introducing too much structural complexity too early.
When a user asked “Who is on-call for user-comms-platform?”, the supervisor frequently made an incorrect routing decision on the first turn. Because the query mentioned “on-call”, the supervisor forwarded the prompt directly to the PagerDuty agent.
However, in our infrastructure ecosystem, PagerDuty does not maintain service-to-team mappings; our internal service catalog (OpsLevel) does. The PagerDuty agent had no concept of which team owned user-comms-platform, and failed to answer.
To resolve this, we learned that sequential chaining beats single-shot routing. Rather than forcing the supervisor to pick a single tool out of six in one shot, we structured multi-step chains:
- First, call the service catalog agent to resolve the owning team.
- Second, pass the identified team context to the PagerDuty agent to fetch the active on-call engineer.
Key Takeaway: Start with a tightly scoped, single-purpose agent before expanding into multi-agent topologies. When multi-agent coordination is required, chain subtasks sequentially rather than expecting a supervisor to route complex queries in a single jump.
2. Heterogeneous Model Orchestration
Not every component of an agentic system requires the largest, most expensive frontier model.
In Jarvis, the supervisor’s only responsibility was classification and intent routing. Using a flagship reasoning model for that step was wasteful. We assigned lightweight, high-throughput models to the supervisor, while routing specialized analytical tasks (such as log parsing or code analysis) to GPT-4 Turbo.
For specialized domain workloads like document parsing and OCR extraction, we routed subtasks to Gemini on Vertex AI. LangGraph made it straightforward to orchestrate heterogeneous models across different nodes in the execution graph, keeping end-to-end latency low and operating costs predictable.
The Dual-Memory Architecture
Because conversational LLMs and stateless containers do not maintain persistence between HTTP requests, we engineered a dual-memory system for Jarvis:
User Interaction
│
┌─────────────────┴─────────────────┐
▼ ▼
[ Short-Term Memory ] [ Long-Term Memory ]
│ │
PostgreSQL Thread Store Engineer Emojis on Slack
(Session state & history) │
│ ▼
30-Day TTL Purge "Learnings" Subagent
(Slack Data Retention) (Extracts clean Q&A)
│
▼
OpenAI Embeddings
│
▼
Weaviate Vector DB
│
▼
Dynamic Context Injection
(Overrides Outdated Docs)Short-Term Memory (Session State)
We stored conversational thread state in PostgreSQL using LangGraph checkpointers. When an engineer replied inside a Slack thread, Jarvis loaded previous message history directly from the database rather than making repetitive calls to the Slack API.
To maintain compliance with company security and data retention policies, all conversation threads were automatically purged on a 30-day sliding TTL.
Long-Term Memory & Continuous Learning
Internal Confluence documentation is notoriously prone to falling out of date. When an engineer changed an infrastructure procedure, documentation updates often lagged behind reality.
To solve this, we implemented a continuous learning loop:
- When Jarvis encountered a question it could not answer, an infrastructure engineer stepped into the Slack thread to provide the correct solution.
- The engineer reacted to the thread with a designated emoji.
- The emoji reaction triggered an asynchronous Learnings Agent that analyzed the thread, extracted a clean question-and-answer pair, and computed dense vector embeddings via OpenAI.
- The vectors were indexed in a Weaviate vector database.
When subsequent questions arrived, Jarvis performed a semantic similarity search against Weaviate and dynamically injected the verified engineer answers into the system prompt (RAG). This dynamic prompt injection allowed Jarvis to answer new questions accurately without inflating static system prompts or waiting for wiki pages to be rewritten.
Why We Decommissioned Jarvis
If Jarvis was resolving 25% of developer support tickets, had continuous memory, and passed security reviews, why did we sunset it?
Between 2024 and 2026, our enterprise AI platform matured rapidly:
- The Claude Platform Rollout: We deployed governed Claude workspaces across the entire company with centralized token vending and single-sign-on.
- The MCP Connector Catalog: We built centralized Model Context Protocol connectors for internal APIs, databases, and monitoring systems.
- The AI Skills Monorepo: We established an org-wide skills repository where every engineering team maintains their own domain tooling under
teams/<team>/<skill>.
Suddenly, every engineer at the company had direct, governed access to infrastructure tooling, runbooks, and service catalog data inside standardized AI interfaces.
The Cost of Bespoke Bots
Maintaining a custom Slack bot alongside a centralized AI platform creates subtle but compounding technical debt:
- Fragmented Governance: A custom bot requires its own IAM roles, secret rotation pipelines, and telemetry stack, duplicating platform capabilities.
- Inconsistent Developer Experience: Having an infrastructure assistant in Slack that behaves differently from Claude creates confusing operational boundaries.
- Maintenance Drag: Keeping custom Slack event handlers, dependency trees, and container runtimes up to date diverted engineering hours away from core platform improvements.
The Platform Lifecycle
Rather than preserving Jarvis as legacy software, we formally decommissioned the service.
Jarvis fulfilled its exact purpose: it proved the business value of agentic support automation, surfaced critical learnings around tool chaining and dynamic memory, and paved the way for our centralized AI platform.
True platform engineering means building for immediate leverage and having the discipline to retire custom tools when standard platform abstractions arrive.