Unsupervised Defect Telemetry: Clustering 26,000 Support Tickets with Embeddings and DBSCAN

How we used dense vector embeddings, cosine distance thresholds, and LLM-powered deep research to catch silent production bugs months before traditional alerts.

In modern distributed systems, production observability is heavily geared toward catastrophic failure: HTTP 500 error spikes, failing Kubernetes readiness probes, and database connection pool exhaustion. When an infrastructure component crashes, Datadog, Sentry, and PagerDuty alert on-call engineers within seconds.

Subtle logical defects behave differently.

When a client-side update breaks a specific dropdown on a single mobile OS version, or when a third-party partner API silently changes its response schema, traditional telemetry often stays green. The application still returns HTTP 200 responses, but users cannot complete their workflows.

Instead of registering in APM dashboards, these failures manifest as a trickle of confusion inside customer support queues. Because users describe symptoms with wildly differing vocabulary (“the submit button is greyed out”, “cannot tap continue after entering zip code”, “app hangs on step 3”), manual triage often fails to connect the dots until weeks or months have passed.

To solve this, we built an automated defect telemetry engine that streams support escalations, embeds them into vector space, and runs density-based clustering to detect emerging production defects autonomously.

Incoming Support Tickets (Zendesk / Intercom)
                     │
                     ▼
       ┌───────────────────────────┐
       │ PII Redaction & Scrubbing │
       └───────────────────────────┘
                     │
                     ▼
       ┌───────────────────────────┐
       │ OpenAI Embedding Pipeline │ (text-embedding-3-small)
       └───────────────────────────┘
                     │
                     ▼
       ┌───────────────────────────┐
       │   PostgreSQL + pgvector   │ (Vector 1536)
       └───────────────────────────┘
                     │
         ┌───────────┴───────────┐
         ▼                       ▼
 ┌───────────────┐       ┌───────────────┐
 │ Real-Time Bug │       │ Weekly Batch  │
 │  Track (k-NN) │       │ Defect Track  │
 └───────┬───────┘       └───────┬───────┘
         │ (≥ 5 in 4h)           │ (DBSCAN eps=0.30)
         └───────────┬───────────┘
                     │
                     ▼
       ┌───────────────────────────┐
       │ Cluster Deep Research     │
       │ (LangGraph + MCP Agents)  │
       └─────────────┬─────────────┘
                     │
                     ▼
 ┌───────────────────────────────────────┐
 │ Synthesized Jira Bug & Root Cause Doc │
 └───────────────────────────────────────┘

Why Supervised Classification and K-Means Fail

When approaching automated ticket classification, two conventional techniques usually fall short:

1. Supervised Multi-Class Classifiers

Supervised machine learning models require predefined taxonomies and labeled training sets. While effective for stable categories (e.g., “Billing Inquiries” vs. “Password Reset”), supervised models cannot detect previously unknown defects. By definition, a zero-day production bug does not exist in your training labels.

2. Centroid-Based Clustering (K-Means)

K-Means clustering requires specifying the number of clusters (k) in advance, which is impossible in a dynamic support queue where the number of active issues changes daily. Furthermore, K-Means assumes spherical cluster geometries and forces every single data point into a centroid. In support data, where 85% of incoming tickets are idiosyncratic one-off questions, K-Means pollutes legitimate bug clusters with unrelated noise.

Density-Based Spatial Clustering (DBSCAN)

To discover arbitrary, dense groupings while filtering out noise, we selected DBSCAN (Density-Based Spatial Clustering of Applications with Noise) paired with cosine metric distance thresholds.

DBSCAN works on two fundamental parameters:

eps (ε): The maximum distance between two points to be considered neighbors.
min_samples: The minimum number of points required to form a dense core region.

Points that do not belong to a dense neighborhood are classified as noise (assigned cluster ID -1) and ignored.

import numpy as np
from sklearn.cluster import DBSCAN

def cluster_ticket_embeddings(
    embeddings: np.ndarray, 
    eps: float = 0.30, 
    min_samples: int = 5
) -> np.ndarray:
    """
    Cluster ticket vectors using cosine distance metric.
    eps represents max cosine distance (1.0 - cosine_similarity).
    """
    clustering = DBSCAN(
        eps=eps,
        min_samples=min_samples,
        metric="cosine",
        n_jobs=-1
    )
    return clustering.fit_predict(embeddings)

By setting metric="cosine" and tuning eps to 0.30 (equivalent to a minimum cosine similarity of 0.70), the algorithm isolates tightly correlated user complaints regardless of the specific words used, while safely casting aside non-actionable noise tickets.

Dual-Track Architecture

The system operates across two distinct operational cadences:

Track 1: Real-Time Rapid Bug Alerts (k-NN Search)

For immediate regression detection, every incoming ticket webhook is stripped of PII, converted to an embedding with text-embedding-3-small, and stored in PostgreSQL using pgvector.

The system immediately queries for neighboring vectors created within a sliding 4-hour lookback window:

SELECT id, ticket_body, 
       1 - (embedding <=> :incoming_vector) AS cosine_similarity
FROM support_tickets
WHERE created_at >= NOW() - INTERVAL '4 hours'
  AND (1 - (embedding <=> :incoming_vector)) >= 0.85
ORDER BY cosine_similarity DESC
LIMIT 10;

If the count of semantically similar tickets exceeds the alert threshold (e.g., 5 similar reports within 4 hours), the system triggers an immediate developer alert for real-time investigation.

Track 2: Weekly Latent Defect Mining (DBSCAN Batch)

Every week, an automated batch job pulls all unlinked tickets across the prior 7 days and runs DBSCAN clustering over the entire vector space. This catches low-velocity, high-severity bugs: defects that affect 15 to 30 users over a week, but never generated more than one ticket in any single hour.

Deep Research and Root Cause Synthesis

Isolating a cluster of 25 vector IDs is not enough; engineers need actionable context without reading 25 raw transcripts.

When a valid cluster is formed, an automated LangGraph synthesis agent analyzes the cluster. Equipped with Model Context Protocol (MCP) integrations for Jira and GitHub, the agent performs four automated steps:

Centroid Extraction: Selects the top 5 tickets closest to the cluster centroid to serve as canonical exemplars.
Issue Deduplication: Queries the Jira MCP connector to check if an open ticket with similar semantics already exists.
Root-Cause Hypothesis: Prompts an LLM with the sanitized ticket excerpts, user device metadata, and system error strings to generate a concise summary of the failure mode.
Automated Ticket Drafting: Creates a structured Jira Task containing the failure hypothesis, affected user count, customer verbatim quotes, and a suggested engineering fix plan.

Business and Engineering Impact

We backtested the pipeline against 26,000 historical support escalations and evaluated it live in production:

100% Defect Rediscovery: The system autonomously detected 100% of previously confirmed production bugs between 15 and 75 days earlier than manual support-to-engineering escalation paths.
107-Day Bug Caught: One silent, edge-case authentication loop that had lingered in the backlog for 107 days was isolated as a distinct, 14-ticket cluster within minutes of ingestion.
Noise Compression: Compressed tens of thousands of messy escalations into 20 actionable clusters, identifying 8 critical production bugs over a four-month operating window.
Executive Visibility: Demonstrated directly to the Chief Product Officer (CPO) and Product Platform leadership as a foundational pattern for Voice-of-Customer telemetry.

Key Takeaways

Support queues are early-warning telemetry: If telemetry only monitors server metrics, you miss failures where the server responds correctly to broken client requests.
Density clustering beats arbitrary partitioning: DBSCAN is the right tool for unstructured support data because it treats the vast majority of one-off inquiries as noise without skewing cluster boundaries.
Synthesized output drives engineering adoption: Engineers do not want raw clusters or CSV exports. Delivering an automated Jira ticket with verified customer quotes, reproduction steps, and root-cause hypotheses turns abstract data science into immediate operational fixes.

← All essays Home