mirror of
https://github.com/trailofbits/algo.git
synced 2025-09-03 02:23:39 +02:00
* Fix VPN routing by adding output interface to NAT rules The NAT rules were missing the output interface specification (-o eth0), which caused routing failures on multi-homed systems (servers with multiple network interfaces). Without specifying the output interface, packets might not be NAT'd correctly. Changes: - Added -o {{ ansible_default_ipv4['interface'] }} to all NAT rules - Updated both IPv4 and IPv6 templates - Updated tests to verify output interface is present - Added ansible_default_ipv4/ipv6 to test fixtures This fixes the issue where VPN clients could connect but not route traffic to the internet on servers with multiple network interfaces (like DigitalOcean droplets with private networking enabled). 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix VPN routing by adding output interface to NAT rules On multi-homed systems (servers with multiple network interfaces or multiple IPs on one interface), MASQUERADE rules need to specify which interface to use for NAT. Without the output interface specification, packets may not be routed correctly. This fix adds the output interface to all NAT rules: -A POSTROUTING -s [vpn_subnet] -o eth0 -j MASQUERADE Changes: - Modified roles/common/templates/rules.v4.j2 to include output interface - Modified roles/common/templates/rules.v6.j2 for IPv6 support - Added tests to verify output interface is present in NAT rules - Added ansible_default_ipv4/ipv6 variables to test fixtures For deployments on providers like DigitalOcean where MASQUERADE still fails due to multiple IPs on the same interface, users can enable the existing alternative_ingress_ip option in config.cfg to use explicit SNAT. Testing: - Verified on live servers - All unit tests pass (67/67) - Mutation testing confirms test coverage This fixes VPN connectivity on servers with multiple interfaces while remaining backward compatible with single-interface deployments. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix dnscrypt-proxy not listening on VPN service IPs Problem: dnscrypt-proxy on Ubuntu uses systemd socket activation by default, which overrides the configured listen_addresses in dnscrypt-proxy.toml. The socket only listens on 127.0.2.1:53, preventing VPN clients from resolving DNS queries through the configured service IPs. Solution: Disable and mask the dnscrypt-proxy.socket unit to allow dnscrypt-proxy to bind directly to the VPN service IPs specified in its configuration file. This fixes DNS resolution for VPN clients on Ubuntu 20.04+ systems. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Apply Python linting and formatting - Run ruff check --fix to fix linting issues - Run ruff format to ensure consistent formatting - All tests still pass after formatting changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Restrict DNS access to VPN clients only Security fix: The firewall rule for DNS was accepting traffic from any source (0.0.0.0/0) to the local DNS resolver. While the service IP is on the loopback interface (which normally isn't routable externally), this could be a security risk if misconfigured. Changed firewall rules to only accept DNS traffic from VPN subnets: - INPUT rule now includes -s {{ subnets }} to restrict source IPs - Applied to both IPv4 and IPv6 rules - Added test to verify DNS is properly restricted This ensures the DNS resolver is only accessible to connected VPN clients, not the entire internet. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix dnscrypt-proxy service startup with masked socket Problem: dnscrypt-proxy.service has a dependency on dnscrypt-proxy.socket through the TriggeredBy directive. When we mask the socket before starting the service, systemd fails with "Unit dnscrypt-proxy.socket is masked." Solution: 1. Override the service to remove socket dependency (TriggeredBy=) 2. Reload systemd daemon immediately after override changes 3. Start the service (which now doesn't require the socket) 4. Only then disable and mask the socket This ensures dnscrypt-proxy can bind directly to the configured IPs without socket activation, while preventing the socket from being re-enabled by package updates. Changes: - Added TriggeredBy= override to remove socket dependency - Added explicit daemon reload after service overrides - Moved socket masking to after service start in main.yml - Fixed YAML formatting issues Testing: Deployment now succeeds with dnscrypt-proxy binding to VPN IPs 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix dnscrypt-proxy by not masking the socket Problem: Masking dnscrypt-proxy.socket prevents the service from starting because the service has Requires=dnscrypt-proxy.socket dependency. Solution: Simply stop and disable the socket without masking it. This prevents socket activation while allowing the service to start and bind directly to the configured IPs. Changes: - Removed socket masking (just disable it) - Moved socket disabling before service start - Removed invalid systemd directives from override Testing: Confirmed dnscrypt-proxy now listens on VPN service IPs 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Use systemd socket activation properly for dnscrypt-proxy Instead of fighting systemd socket activation, configure it to listen on the correct VPN service IPs. This is more systemd-native and reliable. Changes: - Create socket override to listen on VPN IPs instead of localhost - Clear default listeners and add VPN service IPs - Use empty listen_addresses in dnscrypt-proxy.toml for socket activation - Keep socket enabled and let systemd manage the activation - Add handler for restarting socket when config changes Benefits: - Works WITH systemd instead of against it - Survives package updates better - No dependency conflicts - More reliable service management This approach is cleaner than disabling socket activation entirely and ensures dnscrypt-proxy is accessible to VPN clients on the correct IPs. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Document debugging lessons learned in CLAUDE.md Added comprehensive debugging guidance based on our troubleshooting session: - VPN connectivity troubleshooting order (DNS first!) - systemd socket activation best practices - Common deployment failures and solutions - Time wasters to avoid (lessons learned the hard way) - Multi-homed system considerations - Testing notes for DigitalOcean These additions will help future debugging sessions avoid the same rabbit holes and focus on the most likely issues first. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix DNS resolution for VPN clients by enabling route_localnet The issue was that dnscrypt-proxy listens on a special loopback IP (randomly generated in 172.16.0.0/12 range) which wasn't accessible from VPN clients. This fix: 1. Enables net.ipv4.conf.all.route_localnet sysctl to allow routing to loopback IPs from other interfaces 2. Ensures dnscrypt-proxy socket is properly restarted when its configuration changes 3. Adds proper handler flushing after socket configuration updates This allows VPN clients to reach the DNS resolver at the local_service_ip address configured on the loopback interface. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Improve security by using interface-specific route_localnet Instead of enabling route_localnet globally (net.ipv4.conf.all.route_localnet), this change enables it only on the specific interfaces that need it: - WireGuard interface (wg0) for WireGuard VPN clients - Main network interface (eth0/etc) for IPsec VPN clients This minimizes the security impact by restricting loopback routing to only the VPN interfaces, preventing other interfaces from being able to route to loopback addresses. The interface-specific approach provides the same functionality (allowing VPN clients to reach the DNS resolver on the local_service_ip) while reducing the potential attack surface. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert to global route_localnet to fix deployment failure The interface-specific route_localnet approach failed because: - WireGuard interface (wg0) doesn't exist until the service starts - We were trying to set the sysctl before the interface was created - This caused deployment failures with "No such file or directory" Reverting to the global setting (net.ipv4.conf.all.route_localnet=1) because: - It always works regardless of interface creation timing - VPN users are trusted (they have our credentials) - Firewall rules still restrict access to only port 53 - The security benefit of interface-specific settings is minimal - The added complexity isn't worth the marginal security improvement This ensures reliable deployments while maintaining the DNS resolution fix. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix dnscrypt-proxy socket restart and remove problematic BPF hardening Two important fixes: 1. Fix dnscrypt-proxy socket not restarting with new configuration - The socket wasn't properly restarting when its override config changed - This caused DNS to listen on wrong IP (127.0.2.1 instead of local_service_ip) - Now directly restart the socket when configuration changes - Add explicit daemon reload before restarting 2. Remove BPF JIT hardening that causes deployment errors - The net.core.bpf_jit_enable sysctl isn't available on all kernels - It was causing "Invalid argument" errors during deployment - This was optional security hardening with minimal benefit - Removing it eliminates deployment errors for most users These fixes ensure reliable DNS resolution for VPN clients and clean deployments without error messages. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Update CLAUDE.md with comprehensive debugging lessons learned Based on our extensive debugging session, this update adds critical documentation: ## DNS Architecture and Troubleshooting - Explained the local_service_ip design and why it requires route_localnet - Added detailed DNS debugging methodology with exact steps in order - Documented systemd socket activation complexities and common mistakes - Added specific commands to verify DNS is working correctly ## Architectural Decisions - Added new section explaining trade-offs in Algo's design choices - Documented why local_service_ip uses loopback instead of alternatives - Explained iptables-legacy vs iptables-nft backend choice ## Enhanced Debugging Guidance - Expanded troubleshooting with exact commands and expected outputs - Added warnings about configuration changes that need restarts - Documented socket activation override requirements in detail - Added common pitfalls like interface-specific sysctls ## Time Wasters Section - Added new lessons learned from this debugging session - Interface-specific route_localnet (fails before interface exists) - DNAT for loopback addresses (doesn't work) - BPF JIT hardening (causes errors on many kernels) This documentation will help future maintainers avoid the same debugging rabbit holes and understand why things are designed the way they are. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
346 lines
11 KiB
Python
346 lines
11 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Test that Ansible templates render correctly
|
|
This catches undefined variables, syntax errors, and logic bugs
|
|
"""
|
|
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
from jinja2 import Environment, FileSystemLoader, StrictUndefined, TemplateSyntaxError, UndefinedError
|
|
|
|
# Add parent directory to path for fixtures
|
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
from fixtures import load_test_variables
|
|
|
|
|
|
# Mock Ansible filters that don't exist in plain Jinja2
|
|
def mock_to_uuid(value):
|
|
"""Mock the to_uuid filter"""
|
|
return "12345678-1234-5678-1234-567812345678"
|
|
|
|
|
|
def mock_bool(value):
|
|
"""Mock the bool filter"""
|
|
return str(value).lower() in ("true", "1", "yes", "on")
|
|
|
|
|
|
def mock_lookup(type, path):
|
|
"""Mock the lookup function"""
|
|
# Return fake data for file lookups
|
|
if type == "file":
|
|
if "private" in path:
|
|
return "MOCK_PRIVATE_KEY_BASE64=="
|
|
elif "public" in path:
|
|
return "MOCK_PUBLIC_KEY_BASE64=="
|
|
elif "preshared" in path:
|
|
return "MOCK_PRESHARED_KEY_BASE64=="
|
|
return "MOCK_LOOKUP_DATA"
|
|
|
|
|
|
def get_test_variables():
|
|
"""Get a comprehensive set of test variables for template rendering"""
|
|
# Load from fixtures for consistency
|
|
return load_test_variables()
|
|
|
|
|
|
def find_templates():
|
|
"""Find all Jinja2 template files in the repo"""
|
|
templates = []
|
|
for pattern in ["**/*.j2", "**/*.jinja2", "**/*.yml.j2"]:
|
|
templates.extend(Path(".").glob(pattern))
|
|
return templates
|
|
|
|
|
|
def test_template_syntax():
|
|
"""Test that all templates have valid Jinja2 syntax"""
|
|
templates = find_templates()
|
|
|
|
# Skip some paths that aren't real templates
|
|
skip_paths = [".git/", "venv/", ".venv/", ".env/", "configs/"]
|
|
|
|
# Skip templates that use Ansible-specific filters
|
|
skip_templates = ["vpn-dict.j2", "mobileconfig.j2", "dnscrypt-proxy.toml.j2"]
|
|
|
|
errors = []
|
|
skipped = 0
|
|
for template_path in templates:
|
|
# Skip unwanted paths
|
|
if any(skip in str(template_path) for skip in skip_paths):
|
|
continue
|
|
|
|
# Skip templates with Ansible-specific features
|
|
if any(skip in str(template_path) for skip in skip_templates):
|
|
skipped += 1
|
|
continue
|
|
|
|
try:
|
|
template_dir = template_path.parent
|
|
env = Environment(loader=FileSystemLoader(template_dir), undefined=StrictUndefined)
|
|
|
|
# Just try to load the template - this checks syntax
|
|
env.get_template(template_path.name)
|
|
|
|
except TemplateSyntaxError as e:
|
|
errors.append(f"{template_path}: Syntax error - {e}")
|
|
except Exception as e:
|
|
errors.append(f"{template_path}: Error loading - {e}")
|
|
|
|
if errors:
|
|
print(f"✗ Template syntax check failed with {len(errors)} errors:")
|
|
for error in errors[:10]: # Show first 10 errors
|
|
print(f" - {error}")
|
|
if len(errors) > 10:
|
|
print(f" ... and {len(errors) - 10} more")
|
|
assert False, "Template syntax errors found"
|
|
else:
|
|
print(f"✓ Template syntax check passed ({len(templates) - skipped} templates, {skipped} skipped)")
|
|
|
|
|
|
def test_critical_templates():
|
|
"""Test that critical templates render with test data"""
|
|
critical_templates = [
|
|
"roles/wireguard/templates/client.conf.j2",
|
|
"roles/strongswan/templates/ipsec.conf.j2",
|
|
"roles/strongswan/templates/ipsec.secrets.j2",
|
|
"roles/dns/templates/adblock.sh.j2",
|
|
"roles/dns/templates/dnsmasq.conf.j2",
|
|
"roles/common/templates/rules.v4.j2",
|
|
"roles/common/templates/rules.v6.j2",
|
|
]
|
|
|
|
test_vars = get_test_variables()
|
|
errors = []
|
|
|
|
for template_path in critical_templates:
|
|
if not os.path.exists(template_path):
|
|
continue # Skip if template doesn't exist
|
|
|
|
try:
|
|
template_dir = os.path.dirname(template_path)
|
|
template_name = os.path.basename(template_path)
|
|
|
|
env = Environment(loader=FileSystemLoader(template_dir), undefined=StrictUndefined)
|
|
|
|
# Add mock functions
|
|
env.globals["lookup"] = mock_lookup
|
|
env.filters["to_uuid"] = mock_to_uuid
|
|
env.filters["bool"] = mock_bool
|
|
|
|
template = env.get_template(template_name)
|
|
|
|
# Add item context for templates that use loops
|
|
if "client" in template_name:
|
|
test_vars["item"] = ("test-user", "test-user")
|
|
|
|
# Try to render
|
|
output = template.render(**test_vars)
|
|
|
|
# Basic validation - should produce some output
|
|
assert len(output) > 0, f"Empty output from {template_path}"
|
|
|
|
except UndefinedError as e:
|
|
errors.append(f"{template_path}: Missing variable - {e}")
|
|
except Exception as e:
|
|
errors.append(f"{template_path}: Render error - {e}")
|
|
|
|
if errors:
|
|
print("✗ Critical template rendering failed:")
|
|
for error in errors:
|
|
print(f" - {error}")
|
|
assert False, "Critical template rendering errors"
|
|
else:
|
|
print("✓ Critical template rendering test passed")
|
|
|
|
|
|
def test_variable_consistency():
|
|
"""Check that commonly used variables are defined consistently"""
|
|
# Variables that should be used consistently across templates
|
|
common_vars = [
|
|
"server_name",
|
|
"IP_subject_alt_name",
|
|
"wireguard_port",
|
|
"wireguard_network",
|
|
"dns_servers",
|
|
"users",
|
|
]
|
|
|
|
# Check if main.yml defines these
|
|
if os.path.exists("main.yml"):
|
|
with open("main.yml") as f:
|
|
content = f.read()
|
|
|
|
missing = []
|
|
for var in common_vars:
|
|
# Simple check - could be improved
|
|
if var not in content:
|
|
missing.append(var)
|
|
|
|
if missing:
|
|
print(f"⚠ Variables possibly not defined in main.yml: {missing}")
|
|
|
|
print("✓ Variable consistency check completed")
|
|
|
|
|
|
def test_wireguard_ipv6_endpoints():
|
|
"""Test that WireGuard client configs properly format IPv6 endpoints"""
|
|
test_cases = [
|
|
# IPv4 address - should not be bracketed
|
|
{"IP_subject_alt_name": "192.168.1.100", "expected_endpoint": "Endpoint = 192.168.1.100:51820"},
|
|
# IPv6 address - should be bracketed
|
|
{
|
|
"IP_subject_alt_name": "2600:3c01::f03c:91ff:fedf:3b2a",
|
|
"expected_endpoint": "Endpoint = [2600:3c01::f03c:91ff:fedf:3b2a]:51820",
|
|
},
|
|
# Hostname - should not be bracketed
|
|
{"IP_subject_alt_name": "vpn.example.com", "expected_endpoint": "Endpoint = vpn.example.com:51820"},
|
|
# IPv6 with zone ID - should be bracketed
|
|
{"IP_subject_alt_name": "fe80::1%eth0", "expected_endpoint": "Endpoint = [fe80::1%eth0]:51820"},
|
|
]
|
|
|
|
template_path = "roles/wireguard/templates/client.conf.j2"
|
|
if not os.path.exists(template_path):
|
|
print(f"⚠ Skipping IPv6 endpoint test - {template_path} not found")
|
|
return
|
|
|
|
base_vars = get_test_variables()
|
|
errors = []
|
|
|
|
for test_case in test_cases:
|
|
try:
|
|
# Set up test variables
|
|
test_vars = {**base_vars, **test_case}
|
|
test_vars["item"] = ("test-user", "test-user")
|
|
|
|
# Render template
|
|
env = Environment(loader=FileSystemLoader("roles/wireguard/templates"), undefined=StrictUndefined)
|
|
env.globals["lookup"] = mock_lookup
|
|
|
|
template = env.get_template("client.conf.j2")
|
|
output = template.render(**test_vars)
|
|
|
|
# Check if the expected endpoint format is in the output
|
|
if test_case["expected_endpoint"] not in output:
|
|
errors.append(
|
|
f"Expected '{test_case['expected_endpoint']}' for IP '{test_case['IP_subject_alt_name']}' but not found in output"
|
|
)
|
|
# Print relevant part of output for debugging
|
|
for line in output.split("\n"):
|
|
if "Endpoint" in line:
|
|
errors.append(f" Found: {line.strip()}")
|
|
|
|
except Exception as e:
|
|
errors.append(f"Error testing {test_case['IP_subject_alt_name']}: {e}")
|
|
|
|
if errors:
|
|
print("✗ WireGuard IPv6 endpoint test failed:")
|
|
for error in errors:
|
|
print(f" - {error}")
|
|
assert False, "IPv6 endpoint formatting errors"
|
|
else:
|
|
print("✓ WireGuard IPv6 endpoint test passed (4 test cases)")
|
|
|
|
|
|
def test_template_conditionals():
|
|
"""Test templates with different conditional states"""
|
|
test_cases = [
|
|
# WireGuard enabled, IPsec disabled
|
|
{
|
|
"wireguard_enabled": True,
|
|
"ipsec_enabled": False,
|
|
"dns_encryption": True,
|
|
"dns_adblocking": True,
|
|
"algo_ssh_tunneling": False,
|
|
},
|
|
# IPsec enabled, WireGuard disabled
|
|
{
|
|
"wireguard_enabled": False,
|
|
"ipsec_enabled": True,
|
|
"dns_encryption": False,
|
|
"dns_adblocking": False,
|
|
"algo_ssh_tunneling": True,
|
|
},
|
|
# Both enabled
|
|
{
|
|
"wireguard_enabled": True,
|
|
"ipsec_enabled": True,
|
|
"dns_encryption": True,
|
|
"dns_adblocking": True,
|
|
"algo_ssh_tunneling": True,
|
|
},
|
|
]
|
|
|
|
base_vars = get_test_variables()
|
|
|
|
for i, test_case in enumerate(test_cases):
|
|
# Merge test case with base vars
|
|
test_vars = {**base_vars, **test_case}
|
|
|
|
# Test a few templates that have conditionals
|
|
conditional_templates = [
|
|
"roles/common/templates/rules.v4.j2",
|
|
]
|
|
|
|
for template_path in conditional_templates:
|
|
if not os.path.exists(template_path):
|
|
continue
|
|
|
|
try:
|
|
template_dir = os.path.dirname(template_path)
|
|
template_name = os.path.basename(template_path)
|
|
|
|
env = Environment(loader=FileSystemLoader(template_dir), undefined=StrictUndefined)
|
|
|
|
# Add mock functions
|
|
env.globals["lookup"] = mock_lookup
|
|
env.filters["to_uuid"] = mock_to_uuid
|
|
env.filters["bool"] = mock_bool
|
|
|
|
template = env.get_template(template_name)
|
|
output = template.render(**test_vars)
|
|
|
|
# Verify conditionals work
|
|
if test_case.get("wireguard_enabled"):
|
|
assert str(test_vars["wireguard_port"]) in output, f"WireGuard port missing when enabled (case {i})"
|
|
|
|
except Exception as e:
|
|
print(f"✗ Conditional test failed for {template_path} case {i}: {e}")
|
|
raise
|
|
|
|
print("✓ Template conditional tests passed")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
# Check if we have Jinja2 available
|
|
try:
|
|
import jinja2 # noqa: F401
|
|
except ImportError:
|
|
print("⚠ Skipping template tests - jinja2 not installed")
|
|
print(" Run: pip install jinja2")
|
|
sys.exit(0)
|
|
|
|
tests = [
|
|
test_template_syntax,
|
|
test_critical_templates,
|
|
test_variable_consistency,
|
|
test_wireguard_ipv6_endpoints,
|
|
test_template_conditionals,
|
|
]
|
|
|
|
failed = 0
|
|
for test in tests:
|
|
try:
|
|
test()
|
|
except AssertionError as e:
|
|
print(f"✗ {test.__name__} failed: {e}")
|
|
failed += 1
|
|
except Exception as e:
|
|
print(f"✗ {test.__name__} error: {e}")
|
|
failed += 1
|
|
|
|
if failed > 0:
|
|
print(f"\n{failed} tests failed")
|
|
sys.exit(1)
|
|
else:
|
|
print(f"\nAll {len(tests)} template tests passed!")
|