mirror of
https://github.com/trailofbits/algo.git
synced 2025-08-09 22:33:03 +02:00
* Implement self-bootstrapping uv setup to resolve issue #14776 This major simplification addresses the Python setup complexity that has been a barrier for non-developer users deploying Algo VPN. ## Revolutionary User Experience Change **Before (complex):** ```bash python3 -m virtualenv --python="$(command -v python3)" .env && source .env/bin/activate && python3 -m pip install -U pip virtualenv && python3 -m pip install -r requirements.txt ./algo ``` **After (simple):** ```bash ./algo ``` ## Key Technical Changes ### Core Implementation - **algo script**: Complete rewrite with automatic uv installation - Detects missing uv and installs automatically via curl - Cross-platform support (macOS, Linux, Windows) - Preserves exact same command interface - Uses `uv run ansible-playbook` instead of virtualenv activation ### Documentation Overhaul - **README.md**: Reduced installation from 4 complex steps to 1 command - **Platform docs**: Simplified macOS, Windows, Linux, Cloud Shell guides - **Removed Python installation complexity** from all user-facing docs ### CI/CD Infrastructure Updates - **5 GitHub Actions workflows** converted from pip to uv - **Docker builds** updated to use uv instead of virtualenv - **Legacy test scripts** (3 files) updated for uv compatibility ### Repository Cleanup - **install.sh**: Updated for cloud-init/bootstrap scenarios - **algo-showenv.sh**: Updated environment detection for uv - **pyproject.toml**: Added all dependencies with proper versioning - **test scripts**: Removed .env references, updated paths ## Benefits Achieved ✅ **Zero-step dependency installation** - uv installs automatically on first run ✅ **Cross-platform consistency** - identical experience on all operating systems ✅ **Automatic Python version management** - uv handles Python 3.11+ requirement ✅ **Familiar interface preserved** - existing `./algo` and `./algo update-users` unchanged ✅ **No breaking changes** - existing users see same commands, same functionality ✅ **Resolves macOS Python compatibility** - works with system Python 3.9 via uv's Python management ## Files Changed (18 total) **Core Scripts (3)**: - algo (complete rewrite with self-bootstrapping) - algo-showenv.sh (uv environment detection) - install.sh (cloud-init script updated) **Documentation (4)**: - README.md (revolutionary simplification) - docs/deploy-from-macos.md (removed Python complexity) - docs/deploy-from-windows.md (simplified WSL setup) - docs/deploy-from-cloudshell.md (updated for uv) **CI/CD (5)**: - .github/workflows/main.yml (pip → uv conversion) - .github/workflows/smart-tests.yml (pip → uv conversion) - .github/workflows/lint.yml (pip → uv conversion) - .github/workflows/integration-tests.yml (pip → uv + Docker fix) - Dockerfile (virtualenv → uv conversion) **Tests (4)**: - tests/legacy-lxd/local-deploy.sh (virtualenv → uv in Docker) - tests/legacy-lxd/update-users.sh (virtualenv → uv in Docker) - tests/legacy-lxd/ca-password-fix.sh (virtualenv → uv in Docker) - tests/unit/test_template_rendering.py (removed .env path reference) **Dependencies (2)**: - pyproject.toml (added full dependency specification) - uv.lock (new uv lockfile for reproducible builds) This implementation makes Algo VPN accessible to non-technical users while maintaining all power and flexibility for advanced users. Closes #14776 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix CI/CD workflow inconsistencies and resolve Claude's code review issues - Fix inconsistent dependency management across all CI workflows - Replace 'uv add' with 'uv sync' for reproducible builds - Use 'uv run --with' for temporary tool installations - Standardize on locked dependencies from pyproject.toml - Fix ineffective linting by removing '|| true' from ruff check in lint.yml - Ensures linting errors actually fail the build - Maintains consistency with other linter configurations - Update yamllint configuration to exclude .venv/ directory - Prevents scanning Python package templates with Ansible-specific filters - Fixes trailing spaces in workflow files - Improve shell script quality by fixing shellcheck warnings - Quote $(pwd) expansions in Docker test scripts - Address critical word-splitting vulnerabilities - Update test infrastructure for uv compatibility - Exclude .env/.venv directories from template scanning - Ensure local tests exactly match CI workflow commands All linters and tests now pass locally and match CI requirements exactly. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove test configuration file * Remove obsolete venvs directory and update .gitignore for uv - Remove venvs/ directory which was only used as a placeholder for virtualenv - Update .gitignore to use explicit .env/ and .venv/ patterns instead of *env - Modernize ignore patterns for uv-based dependency management 🤖 Generated with [Claude Code](https://claude.ai/code) * Implement secure uv installation addressing Claude's security concerns Security improvements: - **Package managers first**: Try brew, apt, dnf, pacman, zypper, winget, scoop - **User consent required**: Clear security warning before script download - **Manual installation guidance**: Provide fallback instructions with checksums - **Versioned installers**: Use uv 0.8.5 specific URLs for consistency across CI/local Benefits: - ✅ Most users get uv via secure package managers (no download needed) - ✅ Clear security disclosure for script downloads with opt-out - ✅ Transparent about security tradeoffs vs usability - ✅ Maintains "just works" experience while respecting security concerns - ✅ CI and local installations now use identical versioned scripts This addresses the unverified download security vulnerability while preserving the user experience improvements from the self-bootstrapping approach. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Major improvements: modernize Python tooling, fix CI, enhance security This commit implements comprehensive improvements across multiple areas: ## 🚀 Python Tooling Modernization - **Eliminate requirements.txt**: Move to pyproject.toml as single source of truth - **Add pytest integration**: Replace individual test file execution with pytest discovery - **Add dev dependencies**: Include pytest and pytest-xdist for parallel testing - **Update documentation**: Modernize CLAUDE.md with uv-based workflows ## 🔒 Security Enhancements (zizmor fixes) - **Fix credential persistence**: Add persist-credentials: false to checkout steps - **Fix template injection**: Move GitHub context variables to environment variables - **Pin action versions**: Use commit hash for astral-sh/setup-uv@v6 (1ddb97e5078301c0bec13b38151f8664ed04edc8) ## ⚡ CI/CD Optimization - **Create composite action**: Centralize uv setup (.github/actions/setup-uv) - **Eliminate workflow duplication**: Replace 13 duplicate uv setup blocks with reusable action - **Fix path filters**: Update smart-tests.yml to watch pyproject.toml instead of requirements.txt - **Remove pip caching**: Clean up obsolete cache: 'pip' configurations - **Standardize test execution**: Use pytest across all workflows ## 🐳 Docker Improvements - **Secure uv installation**: Use official distroless image instead of curl - **Remove requirements.txt**: Update COPY directive for new dependency structure ## 📈 Impact Summary - **Security**: Resolved 12/14 zizmor issues (86% improvement) - **Maintainability**: 92% reduction in workflow duplication - **Performance**: Better caching and parallel test execution - **Standards**: Aligned with 2025 Python packaging best practices 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Complete backward compatibility cleanup and Windows improvements - Fix main.yml requirements.txt lookup with pyproject.toml parsing - Update test_docker_localhost_deployment.py to check pyproject.toml - Fix Vagrantfile pip args with hard-coded dependency versions - Enhance Windows OS detection for WSL, Git Bash, and MINGW variants - Implement versioned Windows PowerShell installer (0.8.5) - Update documentation references in troubleshooting.md and tests/README.md All linters and tests pass: ruff ✅ yamllint ✅ pytest 48/48 ✅ ansible syntax ✅ 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Python version requirement consistency Update test to require Python 3.11+ to match pyproject.toml requires-python setting. Previously test accepted 3.10+ while pyproject.toml required 3.11+. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix pyproject.toml version parsing to not require community.general collection Replace community.general.toml lookup with regex_search on file lookup. This fixes "lookup plugin (community.general.toml) not found" error on macOS where the collection may not be available during early bootstrap. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix ansible version detection for uv-managed environments Replace pip_package_info lookup with uv pip list command to detect ansible version. This fixes "'dict object' has no attribute 'ansible'" error on macOS where ansible is installed via uv instead of system pip. The fix extracts the ansible package version (e.g. 11.8.0) from uv pip list output instead of trying to access non-existent pip package registry. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add Ubuntu-specific uv installation alternatives Enhance the algo bootstrapping script with Ubuntu-specific trusted installation methods when system package managers don't provide uv: - pipx option (official PyPI, ~9 packages vs 58 for python3-pip) - snap option (community-maintained by Canonical employee) - Links to source repo for transparency (github.com/lengau/uv-snap) - Interactive menu with clear explanations - Robust error handling with fallbacks Addresses common Ubuntu 24.04+ deployment scenario where uv is not available via apt, providing secure alternatives to script downloads. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix shellcheck warning in Ubuntu uv installation menu Add -r flag to read command to prevent backslash mangling as required by shellcheck SC2162. This ensures proper handling of user input in the interactive installation method selection. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Major packaging improvements for AlgoVPN 2.0 beta Remove outdated development files and modernize packaging: - Remove PERFORMANCE.md (optimizations are now defaults) - Remove Makefile (limited Docker-only utility) - Remove Vagrantfile (over-engineered for edge case) Modernize Docker support: - Fix .dockerignore: 872MB -> 840KB build context (99.9% reduction) - Update Dockerfile: Python 3.12, uv:latest, better security - Add multi-arch support and health checks - Simplified package dependencies Improve dependency management: - Pin Ansible collections to exact versions (prevent breakage) - Update version to 2.0.0-beta for upcoming release - Align with uv's exact dependency philosophy This reduces maintenance burden while focusing on Algo's core cloud deployment use case. Created GitHub issue #14816 for lazy cloud provider loading in future releases. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Update community health files for AlgoVPN 2.0 Remove outdated CHANGELOG.md: - Contained severely outdated information (v1.2, Ubuntu 20.04, Makefile intro) - Conflicted with current 2.0.0-beta version and recent changes - 136 lines of misleading content requiring complete rewrite - GitHub releases provide better, auto-generated changelogs Modernize CONTRIBUTING.md: - Update client support: macOS 12+, iOS 15+, Windows 11+, Ubuntu 22.04+ - Expand cloud provider list: Add Vultr, Hetzner, Linode, OpenStack, CloudStack - Replace manual dependency setup with uv auto-installation - Add modern development practices: exact dependency pinning, lint.sh usage - Include development setup section with current workflow Fix PULL_REQUEST_TEMPLATE.md: - Fix broken checkboxes: `- []` → `- [ ]` (missing space) - Add linter compliance requirement: `./scripts/lint.sh` - Add dependency pinning check for exact versions - Reorder checklist for logical flow Community health files now accurately reflect AlgoVPN 2.0 capabilities and guide contributors toward modern best practices. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Complete legacy pip module elimination for uv migration Fixes critical macOS installation failure due to PEP 668 externally-managed-environment restrictions. Key changes: - Add missing pyopenssl and segno dependencies to pyproject.toml - Add optional cloud provider dependencies with exact versions - Replace all cloud provider pip module tasks with uv-based installation - Implement dynamic cloud provider dependency installation in cloud-pre.yml - Modernize OpenStack dependency (openstacksdk replaces deprecated shade) This completes the migration from legacy pip to modern uv dependency management, ensuring consistent behavior across all platforms and eliminating the root cause of macOS installation failures. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Update lockfile with cloud provider dependencies and correct version Regenerates uv.lock to include all optional cloud provider dependencies and ensures version consistency between pyproject.toml and lockfile. Added dependencies for all cloud providers: - AWS: boto3, boto, botocore, s3transfer - Azure: azure-identity, azure-mgmt-*, msrestazure - GCP: google-auth, requests - Hetzner: hcloud - Linode: linode-api4 - OpenStack: openstacksdk, keystoneauth1 - CloudStack: cs, sshpubkeys 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Modernize and simplify README installation instructions - Remove obsolete step 3 (dependency installation) since uv handles this automatically - Streamline installation from 5 to 4 steps - Make device section headers consistent (Apple, Android, Windows, Linux) - Combine Linux WireGuard and IPsec sections for clarity - Improve "please see this page" links with clear descriptions - Move PKI preservation note to user management section where it's relevant - Enhance adding/removing users section with better flow - Add context to Other Devices section for manual configuration - Fix grammar inconsistencies (setup → set up, missing commas) - Update Ubuntu deployment docs to specify 22.04 LTS requirement - Simplify road warrior setup instructions - Remove outdated macOS WireGuard complexity notes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Comprehensive documentation modernization and cleanup - Remove all FreeBSD support (roles, documentation, references) - Modernize troubleshooting guide by removing ~200 lines of obsolete content - Rewrite OpenWrt router documentation with cleaner formatting - Update Amazon EC2 documentation with current information - Rewrite unsupported cloud provider documentation - Remove obsolete linting documentation - Update all version references to Ubuntu 22.04 LTS and Python 3.11+ - Add documentation style guidelines to CLAUDE.md - Clean up compilation and legacy Python compatibility issues - Update client documentation for current requirements All documentation now reflects the uv-based modernization and current supported platforms, eliminating references to obsolete tooling and unsupported operating systems. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix linting and syntax errors caused by FreeBSD removal - Restore missing newline in roles/dns/handlers/main.yml (broken during FreeBSD cleanup) - Add FQCN for community.crypto modules in cloud-pre.yml - Exclude playbooks/ directory from ansible-lint (these are task files, not standalone playbooks) The FreeBSD removal accidentally removed a trailing newline causing YAML format errors. The playbook syntax errors were false positives - these files contain tasks for import_tasks/include_tasks, not standalone plays. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix CI test failure: use uv-managed ansible in test script The test script was calling ansible-playbook directly instead of 'uv run ansible-playbook', which caused it to use the system-installed ansible that doesn't have access to the netaddr dependency required by the ansible.utils.ipmath filter. This fixes the CI error: 'Failed to import the required Python library (netaddr)' 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Clean up test config warnings - Remove duplicate ipsec_enabled key (was defined twice) - Remove reserved variable name 'no_log' This eliminates YAML parsing warnings in the test script while maintaining the same test functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add native Windows support with PowerShell script - Create algo.ps1 for native Windows deployment - Auto-install uv via winget/scoop with download fallback - Support update-users command like Unix version - Add PowerShell linting to CI pipeline with PSScriptAnalyzer - Update documentation with Windows-specific instructions - Streamline deploy-from-windows.md with clearer options 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix PowerShell script for Windows Ansible limitations - Fix syntax issues: remove emoji chars, add winget acceptance flags - Address core issue: Ansible doesn't run natively on Windows - Convert PowerShell script to intelligent WSL wrapper - Auto-detect WSL environment and use appropriate approach - Provide clear error messages and WSL installation guidance - Update documentation to reflect WSL requirement - Maintain backward compatibility for existing WSL users 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Greatly improve PowerShell script error messages and WSL detection - Fix WSL detection: only detect when actually running inside WSL - Add comprehensive error messages with step-by-step WSL installation - Provide clear troubleshooting guidance for common scenarios - Add colored output for better visibility (Red/Yellow/Green/Cyan) - Improve WSL execution with better error handling and path validation - Clarify Ubuntu 22.04 LTS recommendation for WSL stability - Add fallback suggestions when things go wrong Resolves the confusing "bash not recognized" error by properly detecting Windows vs WSL environments and providing actionable guidance. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Address code review feedback - Add documentation about PATH export scope in algo script - Optimize Dockerfile layers by combining dependency operations The PATH export comment clarifies that changes only affect the current shell session. The Dockerfile change reduces layers by copying and installing dependencies in a more efficient order. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove unused uv installation code from PowerShell script The PowerShell script is purely a WSL wrapper - it doesn't need to install uv since it just passes execution to WSL/bash where the Unix algo script handles dependency management. Removing dead code that was never called in the execution flow. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Improve uv installation feedback and Docker dependency locking - Track and display which installation method succeeded for uv - Add --locked flag to Docker uv sync for stricter dependency enforcement - Users now see "uv installed successfully via Homebrew\!" etc. This addresses code review feedback about installation transparency and dependency management strictness. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Docker build: use --locked without --frozen The --frozen and --locked flags are mutually exclusive in uv. Using --locked alone provides the stricter enforcement we want - it asserts the lockfile won't change and errors if it would. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix setuptools package discovery error during cloud provider dependency installation The issue occurred when uv tried to install optional dependencies (e.g., [digitalocean]) because setuptools was auto-discovering directories like 'roles', 'library', etc. as Python packages. Since Algo is an Ansible project, not a Python package, this caused builds to fail. Added explicit build-system configuration to pyproject.toml with py-modules = [] to disable package discovery entirely. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Jinja2 template syntax error in OpenSSL certificate generation Removed inline comments from within Jinja2 expressions in the name_constraints_permitted and name_constraints_excluded fields. Jinja2 doesn't support comments within expressions using the # character, which was causing template rendering to fail. Moved explanatory comments outside the Jinja2 expressions to maintain documentation while fixing the syntax error. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance Jinja2 template testing infrastructure Added comprehensive Jinja2 template testing to catch syntax errors early: 1. Created validate_jinja2_templates.py: - Validates all Jinja2 templates for syntax errors - Detects inline comments in Jinja2 expressions (the bug we just fixed) - Checks for common anti-patterns - Provides warnings for style issues - Skips templates requiring Ansible runtime context 2. Created test_strongswan_templates.py: - Tests all StrongSwan templates with multiple scenarios - Tests with IPv4-only, IPv6, DNS hostnames, and legacy OpenSSL - Validates template output correctness - Skips mobileconfig test that requires complex Ansible runtime 3. Updated .ansible-lint: - Enabled jinja[invalid] and jinja[spacing] rules - These will catch template errors during linting 4. Added scripts/test-templates.sh: - Comprehensive test script that runs all template tests - Can be used in CI and locally for validation - All tests pass cleanly without false failures - Treats spacing issues as warnings, not failures This testing would have caught the inline comment issue in the OpenSSL template before it reached production. All tests now pass cleanly. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix StrongSwan CRL reread handler race condition The ipsec rereadcrls command was failing with exit code 7 when the IPsec daemon wasn't fully started yet. This is a timing issue that can occur during initial setup. Added retry logic to: 1. Wait up to 10 seconds for the IPsec daemon to be ready 2. Check daemon status before attempting CRL operations 3. Gracefully handle the case where daemon isn't ready Also fixed Python linting issues (whitespace) in test files caught by ruff. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix StrongSwan CRL handler properly without ignoring errors Instead of ignoring errors (anti-pattern), this fix properly handles the race condition when StrongSwan restarts: 1. After restarting StrongSwan, wait for port 500 (IKE) to be listening - This ensures the daemon is fully ready before proceeding - Waits up to 30 seconds with proper timeout handling 2. When reloading CRLs, use Ansible's retry mechanism - Retries up to 3 times with 2-second delays - Handles transient failures during startup 3. Separated rereadcrls and purgecrls into distinct tasks - Better error reporting and debugging - Cleaner task organization This approach ensures the installation works reliably on fresh installs without hiding potential real errors. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix StrongSwan handlers - handlers cannot be blocks Ansible handlers cannot be blocks. Fixed by: 1. Making each handler a separate task that can notify the next handler 2. restart strongswan -> notifies -> wait for strongswan 3. rereadcrls -> notifies -> purgecrls This maintains the proper execution order while conforming to Ansible's handler constraints. The wait and retry logic is preserved. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix StrongSwan CRL handler for fresh installs The root cause: rereadcrls handler is notified when copying CRL files during certificate generation, which happens BEFORE StrongSwan is installed and started on fresh installs. The fix: 1. Check if StrongSwan service is actually running before attempting CRL reload 2. If not running, skip reload (not needed - StrongSwan will load CRLs on start) 3. If running, attempt reload with retries This handles both scenarios: - Fresh install: StrongSwan not yet running, skip reload - Updates: StrongSwan running, reload CRLs properly Also removed the wait_for port 500 which was failing because StrongSwan doesn't bind to localhost. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
3 lines
134 B
YAML
3 lines
134 B
YAML
---
|
|
# Linode dependencies are now managed via pyproject.toml optional dependencies
|
|
# They will be installed automatically when needed
|