hermes-config/skills/software-development/systematic-debugging/SKILL.md

---
name: systematic-debugging
description: "4-phase root cause debugging: understand bugs before fixing."
version: 1.2.0
author: Hermes Agent (adapted from obra/superpowers)
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [debugging, troubleshooting, problem-solving, root-cause, investigation]
    related_skills: [test-driven-development, writing-plans, subagent-driven-development]
---

# Systematic Debugging

## Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

## The Iron Law

```
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
```

If you haven't completed Phase 1, you cannot propose fixes.

## When to Use

Use for ANY technical issue:
- Test failures
- Bugs in production
- Unexpected behavior
- Performance problems
- Build failures
- Integration issues

**Use this ESPECIALLY when:**
- Under time pressure (emergencies make guessing tempting)
- "Just one quick fix" seems obvious
- You've already tried multiple fixes
- Previous fix didn't work
- You don't fully understand the issue

**Don't skip when:**
- Issue seems simple (simple bugs have root causes too)
- You're in a hurry (rushing guarantees rework)
- Someone wants it fixed NOW (systematic is faster than thrashing)

## The Four Phases

You MUST complete each phase before proceeding to the next.

---

## Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

### 1. Read Error Messages Carefully

- Don't skip past errors or warnings
- They often contain the exact solution
- Read stack traces completely
- Note line numbers, file paths, error codes

**Action:** Use `read_file` on the relevant source files. Use `search_files` to find the error string in the codebase.

### 2. Reproduce Consistently

- Can you trigger it reliably?
- What are the exact steps?
- Does it happen every time?
- If not reproducible → gather more data, don't guess

**Action:** Use the `terminal` tool to run the failing test or trigger the bug:

```bash
# Run specific failing test
pytest tests/test_module.py::test_name -v

# Run with verbose output
pytest tests/test_module.py -v --tb=long
```

### 3. Check Recent Changes

- What changed that could cause this?
- Git diff, recent commits
- New dependencies, config changes

**Action:**

```bash
# Recent commits
git log --oneline -10

# Uncommitted changes
git diff

# Changes in specific file
git log -p --follow src/problematic_file.py | head -100
```

### 4. Gather Evidence in Multi-Component Systems

**WHEN system has multiple components (API → service → database, CI → build → deploy):**

**BEFORE proposing fixes, add diagnostic instrumentation:**

For EACH component boundary:
- Log what data enters the component
- Log what data exits the component
- Verify environment/config propagation
- Check state at each layer

Run once to gather evidence showing WHERE it breaks.
THEN analyze evidence to identify the failing component.
THEN investigate that specific component.

### 5. SPA Auth Redirect Loops — Full-File Sweep First

**WHEN the symptom is a redirect loop between pages (login ↔ dashboard, etc.):**

**CRITICAL: DO NOT rename files, swap pages, or edit HTML nav links as a first response.** This is the #1 cause of user frustration in redirect-loop debugging. The visible entries (anchor tags, button hrefs) are almost never the real redirect source. The real redirects come from auth listeners (`onAuthStateChanged`), guard functions, inline localStorage scripts, keyboard shortcut handlers, and form-submit callbacks — many of which are buried in JS files you haven't read yet. Renaming files creates a cascade of broken references that must be undone when you find the real cause, while the loop keeps running untouched. **Users correctly perceive this as you not reading the files.**

**THE MOST COMMON ROOT CAUSE:** `onAuthStateChanged` listeners in both pages firing with opposite values. A stale PocketBase/Firebase/Supabase auth token in localStorage causes the listener to fire `callback(user)` on one page and `callback(null)` on the other (or fire twice — once sync with user, then async with null after a failed API call). Each page redirects to the other, creating the loop. The fix is usually ONE of:
- Remove the redirect from the listener on the page that should NOT redirect (e.g., login page should show the form, not bounce to dashboard)
- Use an inline `<script>` with a raw `localStorage.getItem()` check instead of the auth SDK listener (deterministic, no async state jitter)
- Clear the stale token from localStorage

**First action — before any file writes or renames:** grep EVERY JS and HTML file for ALL redirect sources in one sweep:

```bash
grep -rn "window\.location\|location\.replace\|location\.href\|meta.*refresh\|onAuthStateChanged" --include="*.js" --include="*.html" .
```

This reveals ALL redirect paths at once: `onAuthStateChanged` listeners, keyboard shortcut handlers, inline localStorage scripts, form-submit redirects, `meta` refresh tags. Missing a single hidden redirect from an auth state listener is the #1 cause of failed fixes — the user sees you editing files while the real loop persists untouched.

Only AFTER you have the complete map of every redirect source — and have identified which specific listeners are firing with different values on each page — should you plan and apply fixes.

### 6. Trace Data Flow

**WHEN error is deep in the call stack:**

- Where does the bad value originate?
- What called this function with the bad value?
- Keep tracing upstream until you find the source
- Fix at the source, not at the symptom

**Action:** Use `search_files` to trace references:

```python
# Find where the function is called
search_files("function_name(", path="src/", file_glob="*.py")

# Find where the variable is set
search_files("variable_name\\s*=", path="src/", file_glob="*.py")
```

### Phase 1 Completion Checklist

- [ ] Error messages fully read and understood
- [ ] Issue reproduced consistently
- [ ] Recent changes identified and reviewed
- [ ] Evidence gathered (logs, state, data flow)
- [ ] **For redirect loops**: EVERY JS/HTML file grepped for `window.location`, `location.replace`, `location.href`, `meta.*refresh` — all redirect sources mapped
- [ ] Problem isolated to specific component/code
- [ ] Root cause hypothesis formed

**STOP:** Do not proceed to Phase 2 until you understand WHY it's happening.

---

## Phase 2: Pattern Analysis

**Find the pattern before fixing:**

### 1. Find Working Examples

- Locate similar working code in the same codebase
- What works that's similar to what's broken?

**Action:** Use `search_files` to find comparable patterns:

```python
search_files("similar_pattern", path="src/", file_glob="*.py")
```

### 2. Compare Against References

- If implementing a pattern, read the reference implementation COMPLETELY
- Don't skim — read every line
- Understand the pattern fully before applying

### 3. Identify Differences

- What's different between working and broken?
- List every difference, however small
- Don't assume "that can't matter"

### 4. Frontend Missing-Element Debugging (Multi-Layer Comparison)

**WHEN a UI feature (menu, button, panel, modal) works on one page but not another:**

**⚠️ FIRST ACTION — Compare working vs broken page immediately.** When the user says "it works on page A but not page B," or when YOU discover this asymmetry, the VERY NEXT tool call should compare the two pages. Do NOT spend turns hypothesizing and applying fixes without first diffing the two pages' HTML/scripts/CSS. The user telling you to compare them is a correction signal — they already identified the right approach and you wasted time guessing.

**Compare across these layers:**

1. **HTML structure** — Search for the element's ID/class in both pages. The working page has it, the broken page may not. Also compare the modal's button attributes (`onclick`, `data-close-modal`, `aria-label`).
2. **CSS styles** — Check if the working page has CSS rules (z-index, animations, visibility) for the element that the broken page lacks.
3. **Script includes** — The JS handler that toggles the element may exist in a shared file that the broken page never loads. Check both pages' `<script>` tags at the bottom. Also check for IIFE inline scripts — a page-specific capture-phase click handler on `document` can silently kill button behavior.
4. **Module chain** — If one page loads the same JS modules as another but they behave differently, check whether the broken page loads the modules in a different order or has an additional module that could be interfering.

**Action:** diff the pages. Search all three layers systematically. Compare script tag blocks at the bottom of each page, search for element IDs/classes, check style sections and inline IIFE scripts. Most cases of works on page A, not on page B are: missing a script include, missing the HTML element, missing CSS, or a page-specific IIFE that catches clicks and stops propagation sometimes all four.

**⚠ Capture-phase click handlers (the silent button killer):** An IIFE with document.addEventListener(click, handler, true) capture phase fires BEFORE target-phase handlers like inline onclick or addEventListener. If it calls e.stopPropagation(), the event never reaches the button's own handlers. Buttons with data-close-modal or aria-label=Close attributes are common matches. To diagnose: check for addEventListener with a third true argument in the page's inline scripts. The fix: dont tag save buttons with data-close-modal so the capture handler ignores them. This is the single most common cause of clicks that "do nothing" on one page but work fine on another.

### 5. Dynamic JS-Generated HTML

**WHEN editing HTML/classes/styling produces no visible change:**

The UI may be generated at **runtime by JavaScript**, not sourced from static HTML files. Common patterns:

- **`innerHTML = \`template literal\``** — JS creates full HTML with hardcoded Tailwind classes (e.g., `text-yellow-800 dark:text-yellow-200`)
- **`document.createElement()` + `.className = '...'`** — JS sets classes programmatically
- **`insertAdjacentHTML()`** — JS injects HTML into the DOM

**Diagnosis — search for the offending classes or text in JS files too:**

```bash
# Search for hardcoded Tailwind classes in JS template literals
search_files("text-yellow-800", path="src/", file_glob="*.js")
search_files("bg-yellow-50", path="src/", file_glob="*.js")
```

**Common pitfall: editing static HTML `<div class="...">` but the element is re-created by JS on every render.** The static HTML only acts as a placeholder that JS immediately overwrites. The fix is in the JS template literal, not the HTML file.

**Multi-layer cross-reference pattern:**
1. Search static HTML for the element's ID/content
2. Search ALL JS files for the same ID/content
3. Search for CSS class definitions that the JS references
4. If found in JS → edit the JS template literal. If only in static HTML → edit the HTML.
5. Search the CSS file for class definitions to see if the style layer is also wrong.

### 6. Understand Dependencies

- What other components does this need?
- What settings, config, environment?
- What assumptions does it make?

---

## Phase 3: Hypothesis and Testing

**Scientific method:**

### 1. Form a Single Hypothesis

- State clearly: "I think X is the root cause because Y"
- Write it down
- Be specific, not vague

### 2. Test Minimally

- Make the SMALLEST possible change to test the hypothesis
- One variable at a time
- Don't fix multiple things at once

### 3. Verify Before Continuing

- Did it work? → Phase 4
- Didn't work? → Form NEW hypothesis
- DON'T add more fixes on top

### 4. When You Don't Know

- Say "I don't understand X"
- Don't pretend to know
- Ask the user for help
- Research more

---

## Phase 4: Implementation

**Fix the root cause, not the symptom:**

### 1. Create Failing Test Case

- Simplest possible reproduction
- Automated test if possible
- MUST have before fixing
- Use the `test-driven-development` skill

### 2. Implement Single Fix

- Address the root cause identified
- ONE change at a time
- No "while I'm here" improvements
- No bundled refactoring

### 3. Verify Fix

```bash
# Run the specific regression test
pytest tests/test_module.py::test_regression -v

# Run full suite — no regressions
pytest tests/ -q
```

### 4. If Fix Doesn't Work — The Rule of Three

- **STOP.**
- Count: How many fixes have you tried?
- If < 3: Return to Phase 1, re-analyze with new information
- **If ≥ 3: STOP and question the architecture (step 5 below)**
- DON'T attempt Fix #4 without architectural discussion

### 5. If 3+ Fixes Failed: Question Architecture

**Pattern indicating an architectural problem:**
- Each fix reveals new shared state/coupling in a different place
- Fixes require "massive refactoring" to implement
- Each fix creates new symptoms elsewhere

**STOP and question fundamentals:**
- Is this pattern fundamentally sound?
- Are we "sticking with it through sheer inertia"?
- Should we refactor the architecture vs. continue fixing symptoms?

**Discuss with the user before attempting more fixes.**

This is NOT a failed hypothesis — this is a wrong architecture.

---

## Red Flags — STOP and Follow Process

If you catch yourself thinking:
- "Quick fix for now, investigate later"
- "Just try changing X and see if it works"
- "Add multiple changes, run tests"
- "Skip the test, I'll manually verify"
- "It's probably X, let me fix that"
- "I don't fully understand but this might work"
- "Pattern says X but I'll adapt it differently"
- "Here are the main problems: [lists fixes without investigation]"
- Proposing solutions before tracing data flow
- **"One more fix attempt" (when already tried 2+)**
- **Each fix reveals a new problem in a different place**

**ALL of these mean: STOP. Return to Phase 1.**

**If 3+ fixes failed:** Question the architecture (Phase 4 step 5).

### Tool Incompatibility vs. Data Corruption

**Critical Phase 1 pitfall:** When a tool fails to open a file, do NOT conclude the file is corrupt or damaged.

Multiple tools can fail on a perfectly valid file:
- `rawpy`/LibRaw may not support specific camera RAW formats (DNG variants, newer NEF versions)
- Language-native parsers often trail format updates
- Permissions or path encoding can cause opaque errors

**Investigate before declaring data loss:**
1. Try a different tool for the same file type (`dcraw` for RAW photos, ImageMagick for images, ffprobe for media)
2. Check if the user can open the file (they know their own data)
3. Read the error message carefully — "data error" from a library ≠ disk corruption

**When in doubt, the user knows their files.** If they say "I can open them fine," believe them and find the right tool rather than insisting the data is lost.

This pattern applies beyond photos: try multiple parsers/readers before declaring any file corrupted.

## Common Rationalizations

| Excuse | Reality |
|--------|---------|
| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question the pattern, don't fix again. |

## Quick Reference

| Phase | Key Activities | Success Criteria |
|-------|---------------|------------------|
| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence, trace data flow | Understand WHAT and WHY |
| **2. Pattern** | Find working examples, compare, identify differences | Know what's different |
| **3. Hypothesis** | Form theory, test minimally, one variable at a time | Confirmed or new hypothesis |
| **4. Implementation** | Create regression test, fix root cause, verify | Bug resolved, all tests pass |

## Hermes Agent Integration

### Investigation Tools

Use these Hermes tools during Phase 1:

- **`search_files`** — Find error strings, trace function calls, locate patterns
- **`read_file`** — Read source code with line numbers for precise analysis
- **`terminal`** — Run tests, check git history, reproduce bugs
- **`web_search`/`web_extract`** — Research error messages, library docs

### Post-Fix: Browser Cache

After fixing static HTML/CSS/JS files, the user's browser may still show the old version. Add cache-busting meta tags to the `<head>` and ask for a hard refresh.

### Readability Checks

After fixing logic bugs in UI code, check for readability issues. The most common pattern: **same-tone text on same-tone backgrounds** (e.g., `text-yellow-600` on `bg-yellow-50`, or `text-yellow-800 dark:text-yellow-200` on a yellow gradient background).

These are invisible to the debugger but obvious to users.

**Search pattern for same-tone readability bugs:**

```bash
# Find potential yellow-on-yellow
search_files("text-yellow", file_glob="*.html")
search_files("text-yellow", file_glob="*.js")

# Find potential amber-on-amber
search_files("text-amber", file_glob="*.html")
search_files("text-amber", file_glob="*.js")
```

**Check ALL layers that affect the element:**
1. Static HTML classes (`.html` files)
2. JS template-literal classes (`.js` files with `innerHTML`)
3. CSS class definitions (`.css` files, `<style>` blocks)
4. Dynamic JS color assignments (`.js` files — code that sets `className` or constructs class strings)

**Fix approach — use contrasting colors:**
- Light mode: dark text (`text-gray-900`, `text-red-800`, `text-amber-700`) on white/light bg (`bg-white`, `bg-red-50`, `bg-amber-50`)
- Dark mode: light text (`text-white`, `text-red-200`, `text-amber-200`) on dark bg (`bg-gray-800`, `bg-red-900/20`, `bg-amber-900/20`)
- Avoid similarly-toned pairs like `text-yellow-700` on `bg-yellow-50` or `text-amber-600` on `bg-amber-100`
- When using colored labels beneath badges, prefer neutral grays (`text-gray-500`) over colored text
- For alert/notification boxes: use a **colored left border accent** with a neutral white/dark card background — it looks modern and the text is always readable

### Z-Index & Overflow Clipping

Another common UI bug: a dropdown or popup renders but is cut off by its parent container. See `references/css-z-index-overflow-clipping.md` for the diagnosis checklist and fixes.

### Forcing Dark Mode via CSS Overrides

When a user wants the site to always use dark-mode colors regardless of the `.dark` class toggle, use CSS `!important` overrides in a shared stylesheet to map light-mode Tailwind classes to dark-mode colors. See `references/css-force-dark-mode-overrides.md`.

**With delegate_task**

For complex multi-component debugging, dispatch investigation subagents:

```python
delegate_task(
    goal="Investigate why [specific test/behavior] fails",
    context="""
    Follow systematic-debugging skill:
    1. Read the error message carefully
    2. Reproduce the issue
    3. Trace the data flow to find root cause
    4. Report findings — do NOT fix yet

    Error: [paste full error]
    File: [path to failing code]
    Test command: [exact command]
    """,
    toolsets=['terminal', 'file']
)
```

### With test-driven-development

When fixing bugs:
1. Write a test that reproduces the bug (RED)
2. Debug systematically to find root cause
3. Fix the root cause (GREEN)
4. The test proves the fix and prevents regression

## Real-World Impact

From debugging sessions:
- Systematic approach: 15-30 minutes to fix
- Random fixes approach: 2-3 hours of thrashing
- First-time fix rate: 95% vs 40%
- New bugs introduced: Near zero vs common

**No shortcuts. No guessing. Systematic always wins.**