Essay
·Designing an Enterprise Agent Skills Repo: Preventing Tool Sprawl with Embedding CI
How we structured an org-wide skills repository across 500+ engineers with team-scoped CODEOWNERS, dynamic GitHub MCP retrieval, and automated embedding similarity checks.
When enterprise engineering organizations adopt AI coding assistants like Claude and Cursor at scale, an initial burst of productivity is usually followed by a subtle operational challenge: agent tool sprawl.
Without a unified standard, individual engineers and teams build disconnected markdown rules, ad-hoc prompt snippets, and custom scripts. Within months, three different teams write three slightly different “Deploy to Staging” or “Create Jira Bug” skills. Each skill consumes different token budgets, formats responses inconsistently, and occasionally provides conflicting guidance to the underlying model.
To solve this, we designed a centralized AI Skills Repository that balances autonomous team contributions with strict architectural governance.
ai-skills Repository
│
┌────────────────────────┴────────────────────────┐
▼ ▼
org-skills/ teams/
(Pre-installed in Claude; (Team-owned domain skills;
maintained by Platform & Security) governed by CODEOWNERS)
│ │
┌──────────────┐ teams/data-eng/snowflake-query/
│skill-manager │ (Global orchestrator) teams/payments/stripe-reconcile/
└──────┬───────┘ teams/infra/eks-log-dump/
│ │
│ (Dynamic in-session fetch via GitHub MCP) │
└────────────────────────┬────────────────────────┘
│
▼
GitHub Actions CI Pipeline
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Format & Lint │ │ Semantic CI │ │ Catalog Index │
│ (Frontmatter) │ │ (Deduplication│ │ (skills-index)│
└───────────────┘ └───────┬───────┘ └───────────────┘
│
Cosine Similarity Checks
(text-embedding-3-small)
│
┌────────────────────────┴────────────────────────┐
▼ ▼
Score ≥ 85% Score 70% – 84%
❌ Blocking Failure ℹ️ Advisory PR Comment
(Prevents duplicate tools) (Surfaces related skills)The Monorepo Layout: Centralized Discovery, Decentralized Ownership
A central platform team cannot review every domain-specific skill across hundreds of engineers. At the same time, giving everyone write access to a single unpartitioned directory quickly devolves into chaos.
We structured the repository around clear ownership boundaries:
ai-skills/
├── org-skills/ # Universal skills pre-loaded into Claude
│ ├── incident-response/
│ │ └── SKILL.md
│ ├── security-review/
│ │ └── SKILL.md
│ └── skill-manager/ # Global discovery and in-session loading skill
│ └── SKILL.md
├── teams/ # Team-owned domain skills listed in the catalog
│ ├── analytics/
│ │ └── dbt-backfill/SKILL.md
│ ├── payments/
│ │ └── stripe-webhook-debug/SKILL.md
│ └── platform/
│ └── tgw-route-inspect/SKILL.md
├── CODEOWNERS # Maps team directories to GitHub teams
└── skills-index.json # Auto-generated catalog consumed by AI agentsDecentralized Governance via CODEOWNERS
By mapping teams/<team-name>/ to respective GitHub engineering teams in CODEOWNERS, product teams review and approve their own domain skills autonomously. Platform and security teams only gate changes to org-skills/ and core CI automation.
The Anatomy of a Production Skill
Every skill is defined in a single SKILL.md file with strict frontmatter requirements:
---
name: stripe-webhook-debug
description: Investigates failing Stripe webhook events, extracts error payloads, and suggests replay curl commands.
---
# Stripe Webhook Debugger
## When to Use
Activate this skill when debugging 4xx/5xx responses from Stripe webhook endpoints or inspecting dropped events in payment queues.
## Required Tools & MCP Endpoints
- `payments-mcp`: `fetch_webhook_payload`, `get_event_trace`
- `datadog-mcp`: `query_logs`
## Instructions
1. Extract the `event_id` (e.g. `evt_1N...`) from the user prompt or log snippet.
2. Query `payments-mcp` to retrieve the payload and signature verification status.
3. Validate if the error is caused by schema mismatch, idempotency conflict, or downstream timeout.
4. Output a structured root-cause summary and provide a sanitized curl snippet for local replay.
## Guardrails & Non-Goals
- Do NOT replay live payments against production without explicit user confirmation.
- Never log raw customer credit card tokens or PII.A dedicated CI linter (validate_skill_format.py) enforces that every file has valid YAML frontmatter, non-empty description strings, and structured instructions.
Semantic CI: Preventing Duplication with Vector Embeddings
The most innovative component of the platform is automated semantic similarity checking in GitHub Actions.
When an engineer opens a pull request adding a new skill, CI runs similarity_check.py. It converts the proposed SKILL.md into a dense vector embedding (text-embedding-3-small) and calculates cosine similarity against all existing skills in the repository index.
import math
from openai import OpenAI
def cosine_similarity(a: list[float], b: list[float]) -> float:
dot = sum(x * y for x, y in zip(a, b))
mag_a = math.sqrt(sum(x * x for x in a))
mag_b = math.sqrt(sum(x * x for x in b))
return dot / (mag_a * mag_b) if mag_a and mag_b else 0.0
def evaluate_skill_similarity(new_embedding, indexed_skills):
REPORT_THRESHOLD = 0.70 # Include in PR comment table
BLOCK_THRESHOLD = 0.85 # Fail the CI check
matches = []
for path, existing_embedding in indexed_skills.items():
score = cosine_similarity(new_embedding, existing_embedding)
if score >= REPORT_THRESHOLD:
matches.append((score, path))
matches.sort(reverse=True)
is_blocking = bool(matches and matches[0][0] >= BLOCK_THRESHOLD)
return is_blocking, matchesThe Tiered Enforcement Model
-
Blocking Overlap (≥ 85% Similarity): If the new skill matches an existing indexed skill with ≥ 85% cosine similarity, the CI check fails. The bot comments on the PR:
🔴 Blocking: This skill is 89% similar to
teams/analytics/dbt-backfill/SKILL.md. Please extend the existing skill or explain why a separate tool is required. -
Advisory Warning (70% – 84% Similarity): If the similarity is moderate, CI passes with an informational PR comment showing a visual score bar:
🔍 Skill Similarity Check
Similarity Matching Skill 🟠 81% [████████░░]teams/infra/eks-log-dump/SKILL.md🟡 72% [███████░░░]teams/observability/datadog-query/SKILL.md -
Unique Tooling (< 70% Similarity): CI passes cleanly with no comment noise.
Dynamic Catalog Generation (skills-index.json)
To ensure AI assistants can discover available skills without recursively crawling hundreds of GitHub directories at runtime, our post-merge workflow automatically regenerates a centralized skills-index.json on main:
[
{
"name": "stripe-webhook-debug",
"description": "Investigates failing Stripe webhook events, extracts error payloads, and suggests replay curl commands.",
"path": "teams/payments/stripe-webhook-debug/SKILL.md",
"team": "payments"
}
]When an agent needs to discover what capabilities exist across the company, it reads this single pre-compiled JSON catalog, keeping latency and token overhead minimal.
Eliminating Friction: skill-manager and Dynamic In-Session Retrieval
Even with a well-governed monorepo, forcing non-technical colleagues (editorial staff, SEO analysts, performance marketers, legal counsel) to clone repositories, write YAML, or manually download and upload ZIP files creates an immediate adoption barrier.
To solve this, we deployed skill-manager as an org-shipped global skill pre-installed in Claude for all employees.
1. Zero-Install Dynamic Skill Loading via GitHub MCP
Instead of asking employees to permanently install dozens of individual skills into their personal Claude profiles, we guided users to connect GitHub MCP once through company single-sign-on (SSO).
With GitHub MCP enabled, users never need to install individual skills manually. When an employee asks:
“Load the skill for paid marketing” or “Help me check SEC compliance disclosures”
Claude triggers skill-manager, reads skills-index.json to resolve the matching directory, and dynamically fetches the SKILL.md content via GitHub MCP (get_file_contents). The instructions are loaded directly into the active chat session inline on demand without file handling or context clutter.
2. Creating New Skills Conversationally from Chat
skill-manager also makes contribution effortless for non-engineers. An employee does not need a local development environment or terminal to contribute:
- The user tells Claude: “I want to create a new skill for our team to triage partner API webhook drops.”
skill-managercollaborates conversationally to draft theSKILL.md, formatting the frontmatter, instructions, and error-handling guardrails.- Once approved by the user,
skill-manageruses GitHub MCP tools (create_branch,push_files,create_pull_request) to cut a feature branch, commit the file, and open a GitHub pull request automatically. - From there, the CI pipeline automatically runs formatting and semantic similarity checks, notifying the appropriate
CODEOWNERSfor approval.
Engineering Takeaways
- Decentralize content, centralize linting: Let domain teams own their tools via
CODEOWNERS, but use automated linters to enforce format consistency and token budget limits. - Semantic checks beat string matching: Simple keyword searches miss duplicate tools written with different phrasing. Dense vector embeddings catch functional overlap before technical debt compounds.
- Dynamic retrieval beats manual installation: Using GitHub MCP to fetch skills dynamically in-session allows non-technical employees to leverage company-wide tooling without file management or installation fatigue.
- Treat prompts and skills as first-class software: AI tooling requires the same engineering rigor as production code: version control, code review, automated testing, and deprecation lifecycles.