wallenstein/algo
1
0
Fork 0
mirror of https://github.com/trailofbits/algo.git synced 2025-09-03 10:33:13 +02:00
Code Issues Projects Releases Wiki Activity

Update CLAUDE.md with comprehensive debugging lessons learned

Browse source 
Based on our extensive debugging session, this update adds critical documentation:

## DNS Architecture and Troubleshooting
- Explained the local_service_ip design and why it requires route_localnet
- Added detailed DNS debugging methodology with exact steps in order
- Documented systemd socket activation complexities and common mistakes
- Added specific commands to verify DNS is working correctly

## Architectural Decisions
- Added new section explaining trade-offs in Algo's design choices
- Documented why local_service_ip uses loopback instead of alternatives
- Explained iptables-legacy vs iptables-nft backend choice

## Enhanced Debugging Guidance
- Expanded troubleshooting with exact commands and expected outputs
- Added warnings about configuration changes that need restarts
- Documented socket activation override requirements in detail
- Added common pitfalls like interface-specific sysctls

## Time Wasters Section
- Added new lessons learned from this debugging session
- Interface-specific route_localnet (fails before interface exists)
- DNAT for loopback addresses (doesn't work)
- BPF JIT hardening (causes errors on many kernels)

This documentation will help future maintainers avoid the same debugging
rabbit holes and understand why things are designed the way they are.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Dan Guido 2025-08-17 22:09:34 -04:00
parent 30fb6e6c12
commit 09273f69f4
1 changed files with 140 additions and 34 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

174
CLAUDE.md
View file

@ -176,38 +176,64 @@ This practice ensures:
- Too many tasks to fix immediately (113+)
- Focus on new code having proper names
### 2. dnscrypt-proxy Service Failures
### 2. DNS Architecture and Common Issues
#### Understanding local_service_ip
- Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface
- This IP (`local_service_ip`) is where dnscrypt-proxy should listen
- Requires `net.ipv4.conf.all.route_localnet=1` sysctl for VPN clients to reach loopback IPs
- This is by design for consistency across VPN types (WireGuard + IPsec)
#### dnscrypt-proxy Service Failures
**Problem:** "Unit dnscrypt-proxy.socket is masked" or service won't start
- The service has `Requires=dnscrypt-proxy.socket` dependency
- Masking the socket prevents the service from starting
- **Solution:** Configure socket properly instead of fighting it (see systemd section above)
- **Solution:** Configure socket properly instead of fighting it
### 3. DNS Not Accessible to VPN Clients
**Symptoms:** VPN connects but no internet access
- First check: `sudo ss -ulnp | grep :53` on the server
- If only showing 127.0.0.53 or 127.0.2.1, socket activation is misconfigured
- Check firewall allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}`
- **Never** allow DNS from all sources (0.0.0.0/0) - security risk!
#### DNS Not Accessible to VPN Clients
**Symptoms:** VPN connects but no internet/DNS access
1. **First check what's listening:** `sudo ss -ulnp | grep :53`
- Should show `local_service_ip:53` (e.g., 172.24.117.23:53)
- If showing only 127.0.2.1:53, socket override didn't apply
2. **Check socket status:** `systemctl status dnscrypt-proxy.socket`
- Look for "configuration has changed while running" - needs restart
3. **Verify route_localnet:** `sysctl net.ipv4.conf.all.route_localnet`
- Must be 1 for VPN clients to reach loopback IPs
4. **Check firewall:** Ensure allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}`
- **Never** allow DNS from all sources (0.0.0.0/0) - security risk!
### 4. Multi-homed Systems and NAT
### 3. Multi-homed Systems and NAT
**DigitalOcean and other providers with multiple IPs:**
- Servers may have both public and private IPs on same interface
- MASQUERADE needs output interface: `-o {{ ansible_default_ipv4['interface'] }}`
- Don't overengineer with SNAT - MASQUERADE with interface works fine
- Use `alternative_ingress_ip` option only when truly needed
### 5. Jinja2 Template Complexity
### 4. iptables Backend Changes (nft vs legacy)
**Critical:** Switching between iptables-nft and iptables-legacy can break subtle behaviors
- Ubuntu 22.04+ defaults to iptables-nft which may have implicit NAT behaviors
- Algo forces iptables-legacy for consistent rule ordering
- This switch can break DNS routing that "just worked" before
- Always test thoroughly after backend changes
### 5. systemd Socket Activation Gotchas
- Interface-specific sysctls (e.g., `net.ipv4.conf.wg0.route_localnet`) fail if interface doesn't exist yet
- WireGuard interface only created when service starts
- Use global sysctls or apply settings after service start
- Socket configuration changes require explicit restart (not just reload)
### 6. Jinja2 Template Complexity
- Many templates use Ansible-specific filters
- Test templates with `tests/unit/test_template_rendering.py`
- Mock Ansible filters when testing
### 6. OpenSSL Version Compatibility
### 7. OpenSSL Version Compatibility
```yaml
# Check version and use appropriate flags
{{ (openssl_version is version('3', '>=')) | ternary('-legacy', '') }}
```
### 7. IPv6 Endpoint Formatting
### 8. IPv6 Endpoint Formatting
- WireGuard configs must bracket IPv6 addresses
- Template logic: `{% if ':' in IP %}[{{ IP }}]:{{ port }}{% else %}{{ IP }}:{{ port }}{% endif %}`
@ -289,11 +315,13 @@ Each has specific requirements:
### Time Wasters to Avoid (Lessons Learned)
**Don't spend time on these unless absolutely necessary:**
1. **Converting MASQUERADE to SNAT** - MASQUERADE works fine for Algo's use case
2. **Fighting systemd socket activation** - Configure it properly instead
2. **Fighting systemd socket activation** - Configure it properly instead of trying to disable it
3. **Debugging NAT before checking DNS** - Most "routing" issues are DNS issues
4. **Complex IPsec policy matching** - Keep NAT rules simple
4. **Complex IPsec policy matching** - Keep NAT rules simple, avoid `-m policy --pol none`
5. **Testing on existing servers** - Always test on fresh deployments
6. **Adding `-m policy --pol none`** - This breaks NAT, don't use it
6. **Interface-specific route_localnet** - WireGuard interface doesn't exist until service starts
7. **DNAT for loopback addresses** - Packets to local IPs don't traverse PREROUTING
8. **Removing BPF JIT hardening** - It's optional and causes errors on many kernels
## Working with Algo
@ -329,27 +357,105 @@ ansible-playbook users.yml -e "server=SERVER_NAME"
### Troubleshooting VPN Connectivity
#### "VPN connects but can't route traffic" - Check in this order:
1. **DNS first** - `sudo ss -ulnp | grep :53` - Is dnscrypt-proxy listening on VPN IPs?
2. **Packet counters** - `sudo iptables -L FORWARD -v -n | grep -E '10.49|10.48'` - Are packets reaching the firewall?
3. **NAT counters** - `sudo iptables -t nat -L POSTROUTING -v -n` - Is NAT happening?
4. **Service status** - `sudo systemctl status dnscrypt-proxy` - Is the DNS service running?
#### Debugging Methodology
When VPN connects but traffic doesn't work, follow this **exact order** (learned from painful experience):
**Important:** Most "routing" issues are actually DNS issues. Always check DNS first.
1. **Check DNS listening addresses first**
```bash
ss -lnup | grep :53
# Should show local_service_ip:53 (e.g., 172.24.117.23:53)
# If showing 127.0.2.1:53, socket override didn't apply
```
#### systemd and dnscrypt-proxy
- Ubuntu's dnscrypt-proxy package uses socket activation by default
- The default socket listens on 127.0.2.1:53, NOT the VPN service IPs
- Work WITH systemd, not against it:
```yaml
# Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/override.conf
[Socket]
ListenStream= # Clear defaults
ListenStream=172.x.x.x:53 # Add VPN IP
```
- Use empty `listen_addresses = []` in dnscrypt-proxy.toml when using socket activation
- **Never** use `TriggeredBy=` in systemd units (it's not a valid directive)
- Don't mask sockets that services depend on - just disable them
2. **Check both socket AND service status**
```bash
systemctl status dnscrypt-proxy.socket dnscrypt-proxy.service
# Look for "configuration has changed while running" warnings
```
3. **Verify route_localnet is enabled**
```bash
sysctl net.ipv4.conf.all.route_localnet
# Must be 1 for VPN clients to reach loopback IPs
```
4. **Test DNS resolution from server**
```bash
dig @172.24.117.23 google.com # Use actual local_service_ip
# Should return results if DNS is working
```
5. **Check firewall counters**
```bash
iptables -L INPUT -v -n | grep -E '172.24|10.49|10.48'
# Look for increasing packet counts
```
6. **Verify NAT is happening**
```bash
iptables -t nat -L POSTROUTING -v -n
# Check for MASQUERADE rules with packet counts
```
**Key insight:** 90% of "routing" issues are actually DNS issues. Always check DNS first!
#### systemd and dnscrypt-proxy (Critical for Ubuntu/Debian)
**Background:** Ubuntu's dnscrypt-proxy package uses systemd socket activation which **completely overrides** the `listen_addresses` setting in the config file.
**How it works:**
1. Default socket listens on 127.0.2.1:53 (hardcoded in package)
2. Socket activation means systemd opens the port, not dnscrypt-proxy
3. Config file `listen_addresses` is ignored when socket activation is used
4. Must configure the socket, not just the service
**Correct approach:**
```bash
# Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/10-algo-override.conf
[Socket]
ListenStream= # Clear ALL defaults first
ListenDatagram= # Clear UDP defaults too
ListenStream=172.x.x.x:53 # Add TCP on VPN IP
ListenDatagram=172.x.x.x:53 # Add UDP on VPN IP
```
**Config requirements:**
- Use empty `listen_addresses = []` in dnscrypt-proxy.toml for socket activation
- Socket must be restarted (not just reloaded) after config changes
- Check with: `systemctl status dnscrypt-proxy.socket` for warnings
- Verify with: `ss -lnup | grep :53` to see actual listening addresses
**Common mistakes:**
- Trying to disable/mask the socket (breaks service with Requires= dependency)
- Only setting ListenStream (need ListenDatagram for UDP)
- Forgetting to clear defaults first (results in listening on both IPs)
- Not restarting socket after configuration changes
## Architectural Decisions and Trade-offs
### DNS Service IP Design
Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface for DNS (`local_service_ip`). This design has trade-offs:
**Why it's done this way:**
- Provides a consistent DNS IP across both WireGuard and IPsec
- Avoids binding to VPN gateway IPs which differ between protocols
- Survives interface changes and restarts
- Works the same way across all cloud providers
**The cost:**
- Requires `route_localnet=1` sysctl (minor security consideration)
- Adds complexity with systemd socket activation
- Can be confusing to debug
**Alternatives considered but rejected:**
- Binding to VPN gateway IPs directly (breaks unified configuration)
- Using dummy interface instead of loopback (non-standard, more complex)
- DNAT redirects (doesn't work with loopback destinations)
### iptables Backend Choice
Algo forces iptables-legacy instead of iptables-nft on Ubuntu 22.04+ because:
- nft reorders rules unpredictably, breaking VPN traffic
- Legacy backend provides consistent, predictable behavior
- Trade-off: Lost some implicit NAT behaviors that nft provided
## Important Context for LLMs