mirror of
https://github.com/trailofbits/algo.git
synced 2025-09-03 10:33:13 +02:00
Add mutation testing guidance to CLAUDE.md
Added a section on writing effective tests that emphasizes the importance of verifying that tests actually detect failure cases. This lightweight mutation testing approach ensures: - Tests catch the specific bugs they're designed to prevent - We avoid false confidence from tests that always pass - Test purposes are clear and documented - Both success and failure cases are validated The guidance includes a concrete example from our recent inline comment detection test, showing how to verify both the problematic pattern (should fail) and the fixed pattern (should pass). 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
26f349837d
commit
006016cc2d
1 changed files with 28 additions and 0 deletions
28
CLAUDE.md
28
CLAUDE.md
|
@ -141,6 +141,34 @@ shellcheck *.sh
|
|||
pwsh -Command "Invoke-ScriptAnalyzer -Path ./algo.ps1"
|
||||
```
|
||||
|
||||
### Writing Effective Tests - Mutation Testing Approach
|
||||
|
||||
When writing tests, **always verify that your test actually detects the failure case**. This is a form of lightweight mutation testing that ensures tests add real value:
|
||||
|
||||
1. **Write the test for the bug/issue you're preventing**
|
||||
2. **Temporarily introduce the bug** to verify the test fails
|
||||
3. **Fix the bug** and verify the test passes
|
||||
4. **Document what specific issue the test prevents**
|
||||
|
||||
Example from our codebase:
|
||||
```python
|
||||
def test_regression_openssl_inline_comments():
|
||||
"""Tests that we detect inline comments in Jinja2 expressions."""
|
||||
# This pattern SHOULD fail (has inline comments)
|
||||
problematic = "{{ ['DNS:' + id, # comment ] }}"
|
||||
assert not validate(problematic), "Should detect inline comments"
|
||||
|
||||
# This pattern SHOULD pass (no inline comments)
|
||||
fixed = "{{ ['DNS:' + id] }}"
|
||||
assert validate(fixed), "Should pass without comments"
|
||||
```
|
||||
|
||||
This practice ensures:
|
||||
- Tests aren't just checking happy paths
|
||||
- Tests will actually catch regressions
|
||||
- The test's purpose is clear to future maintainers
|
||||
- We avoid false confidence from tests that always pass
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### 1. Ansible-lint "name[missing]" Warnings
|
||||
|
|
Loading…
Add table
Reference in a new issue