From 006016cc2d1a1a670c44112d032154d22a9b2aca Mon Sep 17 00:00:00 2001
From: Dan Guido <dan@trailofbits.com>
Date: Thu, 7 Aug 2025 11:06:23 -0700
Subject: [PATCH] Add mutation testing guidance to CLAUDE.md
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Added a section on writing effective tests that emphasizes the importance
of verifying that tests actually detect failure cases. This lightweight
mutation testing approach ensures:

- Tests catch the specific bugs they're designed to prevent
- We avoid false confidence from tests that always pass
- Test purposes are clear and documented
- Both success and failure cases are validated

The guidance includes a concrete example from our recent inline comment
detection test, showing how to verify both the problematic pattern
(should fail) and the fixed pattern (should pass).

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CLAUDE.md | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/CLAUDE.md b/CLAUDE.md
index ea6abaaa..b2440584 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -141,6 +141,34 @@ shellcheck *.sh
 pwsh -Command "Invoke-ScriptAnalyzer -Path ./algo.ps1"
 ```
 
+### Writing Effective Tests - Mutation Testing Approach
+
+When writing tests, **always verify that your test actually detects the failure case**. This is a form of lightweight mutation testing that ensures tests add real value:
+
+1. **Write the test for the bug/issue you're preventing**
+2. **Temporarily introduce the bug** to verify the test fails
+3. **Fix the bug** and verify the test passes
+4. **Document what specific issue the test prevents**
+
+Example from our codebase:
+```python
+def test_regression_openssl_inline_comments():
+    """Tests that we detect inline comments in Jinja2 expressions."""
+    # This pattern SHOULD fail (has inline comments)
+    problematic = "{{ ['DNS:' + id,  # comment ] }}"
+    assert not validate(problematic), "Should detect inline comments"
+    
+    # This pattern SHOULD pass (no inline comments)  
+    fixed = "{{ ['DNS:' + id] }}"
+    assert validate(fixed), "Should pass without comments"
+```
+
+This practice ensures:
+- Tests aren't just checking happy paths
+- Tests will actually catch regressions
+- The test's purpose is clear to future maintainers
+- We avoid false confidence from tests that always pass
+
 ## Common Issues and Solutions
 
 ### 1. Ansible-lint "name[missing]" Warnings