Drift Detection Types

FlautoPsy monitors your AI workflows for 5 types of drift. Each detector uses a different algorithm and threshold to catch meaningful changes.

1. PROMPT_DECAY

What it detects: Your prompt has lost key constraints or instructions.

Algorithm: Jaccard distance on keyword set

How it works:

  1. FlautoPsy extracts key terms from your baseline prompt
  2. On new traces, it compares the keyword set to the baseline
  3. If the similarity drops below 80%, an alert is triggered

Example:

Original prompt:

text
Classify this email as SPAM, LEGITIMATE, or MARKETING.
Do not respond with anything else.
Return JSON format: {"class": "..."}

Keywords:

text
["SPAM", "LEGITIMATE", "MARKETING", "JSON", "class"]

Modified prompt:

text
Is this email spam?

Keywords:

text
["email", "spam"]
(lost 3 out of 5 key terms)

Similarity: 40% → PROMPT_DECAY Alert

When it's useful: Catches accidental prompt changes, injection attacks, or misconfigured workflows.

Evidence JSON:

json
{
  "detector": "PROMPT_DECAY",
  "baseline_keywords": ["SPAM", "LEGITIMATE", "MARKETING", "JSON"],
  "current_keywords": ["email", "spam"],
  "similarity_score": 0.40,
  "threshold": 0.80,
  "alert": true
}

2. OUTPUT_DRIFT

What it detects: Your LLM's output is significantly longer or shorter than baseline.

Algorithm: Z-score on output length

How it works:

  1. Baseline: mean output length = 150 tokens, std dev = 20 tokens
  2. New trace output: 400 tokens
  3. Z-score = (400 - 150) / 20 = 12.5σ
  4. Threshold: |Z| > 3σ → OUTPUT_DRIFT Alert

Example:

Your email classifier normally outputs 10-20 tokens:

json
{"class": "spam", "confidence": 0.95}

But a new prompt causes it to output 500 tokens (full reasoning):

json
{
  "class": "spam",
  "confidence": 0.95,
  "reasons": ["Contains urgency words like...", "Sender domain is...", ...],
  "analysis": "..."
}

Z-score = 18σ → OUTPUT_DRIFT Alert

When it's useful: Catches model changes, prompt rewrites, or token limit adjustments.

Evidence JSON:

json
{
  "detector": "OUTPUT_DRIFT",
  "baseline_mean_tokens": 150,
  "baseline_std_dev": 20,
  "current_tokens": 400,
  "z_score": 12.5,
  "threshold": 3.0,
  "alert": true
}

3. COST_SPIKE

What it detects: Your LLM calls are using significantly more tokens than baseline.

Algorithm: EWMA (exponential weighted moving average) on cost

How it works:

  1. Baseline EWMA cost = 1000 tokens
  2. New trace cost = 5500 tokens
  3. Ratio = 5500 / 1000 = 5.5x baseline
  4. Threshold: > 5x baseline → COST_SPIKE Alert

Example:

Your workflow normally uses 1000 tokens per call:

Baseline traces (tokens): 950, 1020, 980, 1050, 990, 1010

EWMA ≈ 1000

New trace: 5200 tokens (5.2x baseline) → COST_SPIKE Alert

Possible causes:

  • Model changed to a larger model (GPT-4 vs GPT-3.5)
  • Prompt became longer or more complex
  • Context window grew (more input data)

When it's useful: Catches unexpected cost increases before they hit your bill.

Evidence JSON:

json
{
  "detector": "COST_SPIKE",
  "baseline_ewma_tokens": 1000,
  "current_tokens": 5200,
  "ratio": 5.2,
  "threshold": 5.0,
  "alert": true
}

4. LATENCY_SPIKE

What it detects: Your LLM calls are significantly slower than baseline.

Algorithm: EWMA on latency

How it works:

  1. Baseline EWMA latency = 300ms
  2. New trace latency = 1500ms
  3. Ratio = 1500 / 300 = 5x baseline
  4. Threshold: > 5x baseline → LATENCY_SPIKE Alert

Example:

Your email classifier normally responds in 250-350ms:

Baseline traces (ms): 280, 310, 295, 305, 320, 290

EWMA ≈ 300ms

New trace: 1800ms (6x baseline) → LATENCY_SPIKE Alert

Possible causes:

  • LLM provider experiencing outages
  • Your workflow has rate limiting
  • Network latency increased
  • Model took longer to think (e.g., o1 with extended thinking)

When it's useful: Catches infrastructure issues and poor user experience before it impacts customers.

Evidence JSON:

json
{
  "detector": "LATENCY_SPIKE",
  "baseline_ewma_ms": 300,
  "current_ms": 1800,
  "ratio": 6.0,
  "threshold": 5.0,
  "alert": true
}

5. TOPOLOGY_CHANGE

What it detects: Your workflow's structure has changed (different modules, different order).

Algorithm: Weisfeiler-Lehman graph hash

How it works:

  1. Baseline: workflow is
    text
    [input] → GPT-4 → sentiment_analyzer → [output]
  2. FlautoPsy computes a graph hash of this structure
  3. New workflow:
    text
    [input] → GPT-4 → [output]
    (sentiment analyzer removed)
  4. New graph hash doesn't match → TOPOLOGY_CHANGE Alert

Example:

Baseline workflow:

  1. User input
  2. OpenAI GPT-4 (classify email)
  3. Send to Slack
  4. Log to database

Modified workflow:

  1. User input
  2. Anthropic Claude (different model!)
  3. Send to Slack
  4. Log to database

Module changed from GPT-4 to Claude → TOPOLOGY_CHANGE Alert

When it's useful: Catches accidentally breaking your workflow or unintended changes by collaborators.

Evidence JSON:

json
{
  "detector": "TOPOLOGY_CHANGE",
  "baseline_graph_hash": "abc123...",
  "current_graph_hash": "def456...",
  "hash_match": false,
  "alert": true,
  "changes": ["Module at position 2: OpenAI → Anthropic"]
}

Alert Severity & Thresholds Summary

| Detector | Threshold | Severity | Action | |----------|-----------|----------|--------| | PROMPT_DECAY | < 80% similarity | Medium | Review prompt changes | | OUTPUT_DRIFT | > 3σ (z-score) | Low | Monitor output length | | COST_SPIKE | > 5x baseline | High | Check token usage | | LATENCY_SPIKE | > 5x baseline | High | Check provider status | | TOPOLOGY_CHANGE | Graph hash mismatch | Critical | Review workflow changes |

Tuning Alerts

In FlautoPsy settings, you can adjust sensitivities:

  • Cost sensitivity: Reduce threshold from 5x to 3x if you want earlier warnings
  • Latency sensitivity: Increase threshold from 5x to 10x if your workflow naturally varies
  • Prompt decay threshold: Increase from 80% to 90% for stricter matching
  • Output drift threshold: Change from 3σ to 2σ for earlier detection

See FAQs for how to disable specific detectors.

FAQ

Why 5x for cost and latency, but 3σ for output?

Cost and latency are easier to understand (e.g., "5x is always bad"). Output length uses statistical z-scores because outputs have more natural variation.

Can I get fewer alerts?

Yes. Go to Settings → Alert Tuning and increase thresholds, or disable specific detectors.

What if my workflow is naturally variable?

That's why we use baselines! FlautoPsy learns what "normal variation" is for your specific workflow. If you normally range 200-500ms, we won't alert until you hit 2000ms+.

Do all detectors fire for every workflow?

No. For example:

  • TOPOLOGY_CHANGE only fires if you change your workflow structure
  • PROMPT_DECAY only fires if you change your prompt
  • COST_SPIKE fires if your tokens suddenly increase

Can I alert on custom metrics?

Not yet, but it's on the roadmap. Email support@stralocroft.com if you need custom detectors.

Next Steps