Python3 draft

.travis.yml

@@ -1,6 +1,6 @@
 ---
 language: python
-python: "2.7"
+python: "3.7"
 dist: xenial
 
 services:
@@ -12,7 +12,7 @@ addons:
       - sourceline: 'ppa:ubuntu-lxc/stable'
       - sourceline: 'ppa:wireguard/wireguard'
     packages: &default_packages
-      - python-pip
+      - python3-pip
       - lxd
       - expect-dev
       - debootstrap
@@ -22,7 +22,7 @@ addons:
       - build-essential
       - libssl-dev
       - libffi-dev
-      - python-dev
+      - python3-dev
       - linux-headers-$(uname -r)
       - wireguard
       - libxml2-utils

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:2-alpine
+FROM python:3-alpine
 
 ARG VERSION="git"
 ARG PACKAGES="bash libffi openssh-client openssl rsync tini"
@@ -16,11 +16,11 @@ RUN mkdir -p /algo && mkdir -p /algo/configs
 WORKDIR /algo
 COPY requirements.txt .
 RUN apk --no-cache add ${BUILD_PACKAGES} && \
-    python -m pip --no-cache-dir install -U pip && \
+    python3 -m pip --no-cache-dir install -U pip && \
-    python -m pip --no-cache-dir install virtualenv && \
+    python3 -m pip --no-cache-dir install virtualenv && \
-    python -m virtualenv env && \
+    python3 -m virtualenv env && \
     source env/bin/activate && \
-    python -m pip --no-cache-dir install -r requirements.txt && \
+    python3 -m pip --no-cache-dir install -r requirements.txt && \
     apk del ${BUILD_PACKAGES}
 COPY . .
 RUN chmod 0755 /algo/algo-docker.sh

README.md

@@ -33,12 +33,12 @@ The easiest way to get an Algo server running is to run it on your local machine
 
 2. **[Download Algo](https://github.com/trailofbits/algo/archive/master.zip).** Unzip it in a convenient location on your local machine.
 
-3. **Install Algo's core dependencies.** Open the Terminal. The `python` interpreter you use to deploy Algo must be python2. If you don't know what this means, you're probably fine. `cd` into the `algo-master` directory where you unzipped Algo, then run:
+3. **Install Algo's core dependencies.** Open the Terminal. `cd` into the `algo-master` directory where you unzipped Algo, then run:
 
     - macOS:
       ```bash
-      $ python -m ensurepip --user
+      $ python3 -m ensurepip --user
-      $ python -m pip install --user --upgrade virtualenv
+      $ python3 -m pip install --user --upgrade virtualenv
       ```
     - Linux (deb-based):
       ```bash
@@ -46,20 +46,20 @@ The easiest way to get an Algo server running is to run it on your local machine
           build-essential \
           libssl-dev \
           libffi-dev \
-          python-dev \
+          python3-dev \
-          python-pip \
+          python3-pip \
-          python-setuptools \
+          python3-setuptools \
-          python-virtualenv -y
+          python3-virtualenv -y
       ```
      - Linux (rpm-based): See the pre-installation documentation for [RedHat/CentOS 6.x](docs/deploy-from-redhat-centos6.md) or [Fedora](docs/deploy-from-fedora-workstation.md)
      - Windows: See the [Windows documentation](docs/deploy-from-windows.md)
 
 4. **Install Algo's remaining dependencies.** Use the same Terminal window as the previous step and run:
     ```bash
-    $ python -m virtualenv --python=`which python2` env &&
+    $ python3 -m virtualenv --python="$(command -v python3)" env &&
         source env/bin/activate &&
-        python -m pip install -U pip virtualenv &&
+        python3 -m pip install -U pip virtualenv &&
-        python -m pip install -r requirements.txt
+        python3 -m pip install -r requirements.txt
     ```
     On macOS, you may be prompted to install `cc`. You should press accept if so.
 
@@ -177,7 +177,7 @@ where `user` is either `root` or `ubuntu` as listed on the success message, and
 _If you chose to save the CA key during the deploy process,_ then Algo's own scripts can easily add and remove users from the VPN server.
 
 1. Update the `users` list in your `config.cfg`
-2. Open a terminal, `cd` to the algo directory, and activate the virtual environment with `source env/bin/activate`
+2. Open a terminal, `cd` to the algo directory, and activate the virtual environment with `source .env/bin/activate`
 3. Run the command: `./algo update-users`
 
 After this process completes, the Algo VPN server will contain only the users listed in the `config.cfg` file.

algo-showenv.sh

@@ -68,10 +68,10 @@ elif [[ -f LICENSE && ${STAT} ]]; then
 fi
 
 # The Python version might be useful to know.
-if [[ -x ./env/bin/python ]]; then
+if [[ -x ./.env/bin/python3 ]]; then
-    ./env/bin/python --version 2>&1
+    ./.env/bin/python3 --version 2>&1
 elif [[ -f ./algo ]]; then
-    echo "env/bin/python not found: has 'python -m virtualenv ...' been run?"
+    echo "env/bin/python3 not found: has 'python3 -m virtualenv ...' been run?"
 fi
 
 # Just print out all command line arguments, which are expected

docs/deploy-from-fedora-workstation.md

@@ -7,17 +7,16 @@ These docs were written based on experience on Fedora Workstation 30.
 ### DNF counterparts of apt packages
 
 The following table lists `apt` packages with their `dnf` counterpart. This is purely informative.
-Using `python2-*` in favour of `python3-*` as per [declared dependency](https://github.com/trailofbits/algo#deploy-the-algo-server).
 
 | `apt` | `dnf` |
 | ----- | ----- |
 | `build-essential` | `make automake gcc gcc-c++ kernel-devel` |
 | `libssl-dev` | `openssl-devel` |
 | `libffi-dev` | `libffi-devel` |
-| `python-dev` | `python2-devel` |
+| `python3-dev` | `python3-devel` |
-| `python-pip` | `python2-pip` |
+| `python3-pip` | `python3-pip` |
-| `python-setuptools` | `python2-setuptools` |
+| `python3-setuptools` | `python3-setuptools` |
-| `python-virtualenv` | `python2-virtualenv` |
+| `python3-virtualenv` | `python3-virtualenv` |
 
 ### Install requirements
 
@@ -31,22 +30,20 @@ Next, install the required packages:
 
 ````
 dnf install -y \
-  ansible \
   automake \
   gcc \
   gcc-c++ \
   kernel-devel \
   openssl-devel \
   libffi-devel \
-  libselinux-python \
+  python3-devel \
-  python2-devel \
+  python3-pip \
-  python2-pip \
+  python3-setuptools \
-  python2-setuptools \
+  python3-virtualenv \
-  python2-virtualenv \
+  python3-crypto \
-  python2-crypto \
+  python3-pyyaml \
-  python2-pyyaml \
+  python3-pyOpenSSL \
-  python2-pyOpenSSL \
+  python3-libselinux \
-  python2-libselinux \
   make
 ````
 
@@ -56,7 +53,7 @@ dnf install -y \
 [Download](https://github.com/trailofbits/algo/archive/master.zip) or clone:
 
 ````
-git clone git@github.com:trailofbits/algo.git
+git clone https://github.com/trailofbits/algo.git
 cd algo
 ````
 
@@ -68,24 +65,14 @@ We'll assume from this point forward that our working directory is the `algo` ro
 
 Some steps are needed before we can deploy our Algo VPN server.
 
-### Check `pip`
-
-Run `pip -v` and check the python version it is using:
-
-````
-$ pip -V
-pip 19.0.3 from /usr/lib/python2.7/site-packages (python 2.7)
-````
-
-`python 2.7` is what we're looking for.
-
 ### Setup virtualenv and install requirements
 
-````
+```
-python2 -m virtualenv --system-site-packages env
+python3 -m virtualenv --python="$(command -v python3)" .env
-source env/bin/activate
+source .env/bin/activate
-pip -q install --user -r requirements.txt
+python3 -m pip install -U pip virtualenv
-````
+python3 -m pip install -r requirements.txt
+```
 
 ## Configure
 

docs/deploy-from-redhat-centos6.md

@@ -5,8 +5,8 @@ Many people prefer RedHat or CentOS 6 (or similar variants like Amazon Linux) fo
 ## Step 1: Prep for RH/CentOS 6.8/Amazon
 
 ```shell
-yum -y -q update
+yum -y update
-yum -y -q install epel-release
+yum -y install epel-release
 ```
 
 Enable any kernel updates:
@@ -17,53 +17,64 @@ reboot
 
 ## Step 2: Install Ansible and launch Algo
 
-Fix GPG key warnings during Ansible rpm install:
+RedHat/CentOS 6.x uses Python 2.6 by default, which is explicitly deprecated and produces many warnings and errors, so we must install a safe, non-invasive 3.6 tool set which has to be expressly enabled (and will not survive login sessions and reboots):
 
+- Install the Software Collections Library (to enable Python 3.6)  
 ```shell
-rpm --import https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
+yum -y install centos-release-SCL
+yum -y install \
+  openssl-devel \
+  libffi-devel \
+  automake \
+  gcc \
+  gcc-c++ \
+  kernel-devel \
+  rh-python36-python \
+  rh-python36-python-devel \
+  rh-python36-python-setuptools \
+  rh-python36-python-pip \
+  rh-python36-python-virtualenv \
+  rh-python36-python-crypto \
+  rh-python36-PyYAML \
+  libselinux-python \
+  python-crypto \
+  wget \
+  unzip \
+  nano
 ```
 
-Fix GPG key warning during official Software Collections (SCL) package install:
+- 3.6 will not be used until explicitly enabled, per login session. Enable 3.6 default for this session (needs re-run between logins & reboots)  
-
+```
-```shell
+scl enable rh-python36 bash
-rpm --import https://raw.githubusercontent.com/sclorg/centos-release-scl/master/centos-release-scl/RPM-GPG-KEY-CentOS-SIG-SCLo
 ```
 
-RedHat/CentOS 6.x uses Python 2.6 by default, which is explicitly deprecated and produces many warnings and errors, so we must install a safe, non-invasive 2.7 tool set which has to be expressly enabled (and will not survive login sessions and reboots):
+- We're now defaulted to 3.6. Upgrade required components  
+```
+python3 -m pip install -U pip virtualenv pycrypto setuptools
+```
 
-```shell
+- Download and uzip Algo  
-# Install the Software Collections Library (to enable Python 2.7)
+```
-yum -y -q install centos-release-SCL
+wget https://github.com/trailofbits/algo/archive/master.zip
-
-# 2.7 will not be used until explicitly enabled, per login session		
-yum -y -q install python27-python-devel python27-python-setuptools python27-python-pip
-yum -y -q install openssl-devel libffi-devel automake gcc gcc-c++ kernel-devel wget unzip ansible nano
-
-# Enable 2.7 default for this session (needs re-run between logins & reboots)
-# shellcheck disable=SC1091
-source /opt/rh/python27/enable
-# We're now defaulted to 2.7
-
-# Upgrade pip itself
-pip -q install --upgrade pip
-# python-devel needed to prevent setup.py crash
-pip -q install pycrypto       
-# pycrypto 2.7.1 needed for latest security patch
-pip -q install setuptools --upgrade
-# virtualenv to make installing dependencies easier
-pip -q install virtualenv
-
-wget -q https://github.com/trailofbits/algo/archive/master.zip
 unzip master.zip
 cd algo-master || echo "No Algo directory found"
+```
 
-# Set up a virtualenv and install the local Algo dependencies (must be run from algo-master)
+- Set up a virtualenv and install the local Algo dependencies (must be run from algo-master)  
-virtualenv env && source env/bin/activate
+```
-pip -q install -r requirements.txt
+python3 -m virtualenv --python="$(command -v python3)" .env
+source .env/bin/activate
+python3 -m pip install -U pip virtualenv
+python3 -m pip install -r requirements.txt
+```
 
-# Edit the userlist and any other settings you desire
+- Edit the userlist and any other settings you desire  
+```
 nano config.cfg
-# Now you can run the Algo installer!
+```
+
+- Now you can run the Algo installer!  
+```
 ./algo
 ```
 

docs/deploy-from-windows.md

@@ -21,7 +21,7 @@ Wait a minute for Windows to install a few things in the background (it will eve
 Install additional packages:
 
 ```shell
-sudo apt-get update && sudo apt-get install git build-essential libssl-dev libffi-dev python-dev python-pip python-setuptools python-virtualenv -y
+sudo apt-get update && sudo apt-get install git build-essential libssl-dev libffi-dev python3-dev python3-pip python3-setuptools python3-virtualenv -y
 ```
 
 Clone the Algo repository:

docs/troubleshooting.md

@@ -105,25 +105,13 @@ Command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/p
 Storing debug log for failure in /Users/algore/Library/Logs/pip.log
 ```
 
-You are running an old version of `pip` that cannot download the binary `cryptography` dependency. Upgrade to a new version of `pip` by running `sudo pip install -U pip`.
+You are running an old version of `pip` that cannot download the binary `cryptography` dependency. Upgrade to a new version of `pip` by running `sudo python3 -m pip install -U pip`.
-
-### Error: "TypeError: must be str, not bytes"
-
-You tried to install Algo and you see many repeated errors referencing `TypeError`, such as `TypeError: '>=' not supported between instances of 'TypeError' and 'int'` and `TypeError: must be str, not bytes`. For example:
-
-```
-TASK [Wait until SSH becomes ready...] *****************************************
-An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: must be str, not bytes
-fatal: [localhost -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/x_/nvr61v455qq98vp22k5r5vm40000gn/T/ansible_6sdjysth/ansible_module_wait_for.py\", line 538, in <module>\n    main()\n  File \"/var/folders/x_/nvr61v455qq98vp22k5r5vm40000gn/T/ansible_6sdjysth/ansible_module_wait_for.py\", line 483, in main\n    data += response\nTypeError: must be str, not bytes\n", "module_stdout": "", "msg": "MODULE FAILURE"}
-```
-
-You may be trying to run Algo with Python3. Algo uses [Ansible](https://github.com/ansible/ansible) which has issues with Python3, although this situation is improving over time. Try running Algo with Python2 to fix this issue. Open your terminal and `cd` to the directory with Algo, then run: ``virtualenv -p `which python2.7` env && source env/bin/activate && pip install -r requirements.txt``
 
 ### Error: "ansible-playbook: command not found"
 
 You tried to install Algo and you see an error that reads "ansible-playbook: command not found."
 
-You did not finish step 4 in the installation instructions, "[Install Algo's remaining dependencies](https://github.com/trailofbits/algo#deploy-the-algo-server)." Algo depends on [Ansible](https://github.com/ansible/ansible), an automation framework, and this error indicates that you do not have Ansible installed. Ansible is installed by `pip` when you run `python -m pip install -r requirements.txt`. You must complete the installation instructions to run the Algo server deployment process.
+You did not finish step 4 in the installation instructions, "[Install Algo's remaining dependencies](https://github.com/trailofbits/algo#deploy-the-algo-server)." Algo depends on [Ansible](https://github.com/ansible/ansible), an automation framework, and this error indicates that you do not have Ansible installed. Ansible is installed by `pip` when you run `python3 -m pip install -r requirements.txt`. You must complete the installation instructions to run the Algo server deployment process.
 
 ### Could not fetch URL ... TLSV1_ALERT_PROTOCOL_VERSION
 
@@ -137,9 +125,9 @@ No matching distribution found for SecretStorage<3 (from -r requirements.txt (li
 
 It's time to upgrade your python.
 
-`brew upgrade python2`
+`brew upgrade python3`
 
-You can also download python 2.7.x from python.org.
+You can also download python 3.7.x from python.org.
 
 ### Bad owner or permissions on .ssh
 
@@ -414,32 +402,6 @@ Certain cloud providers (like AWS Lightsail) don't assign an IPv6 address to you
 
 Manually disconnecting and then reconnecting should restore your connection. To solve this, you need to either "force IPv4 connection" if available on your phone, or install an IPv4 APN, which might be available from your carrier tech support. T-mobile's is available [for iOS here under "iOS IPv4/IPv6 fix"](https://www.reddit.com/r/tmobile/wiki/index), and [here is a walkthrough for Android phones](https://www.myopenrouter.com/article/vpn-connections-not-working-t-mobile-heres-how-fix).
 
-### Error: name 'basestring' is not defined
-
-```
-TASK [cloud-digitalocean : Creating a droplet...] *******************************************
-An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NameError: name 'basestring' is not defined
-fatal: [localhost]: FAILED! => {"changed": false, "msg": "name 'basestring' is not defined"}
-```
-
-If you get something like the above it's likely you're not using a python2 virtualenv.
-
-Ensure running `python2.7` drops you into a python 2 shell (it looks something like this)
-
-```
-user@homebook ~ $ python2.7
-Python 2.7.10 (default, Feb  7 2017, 00:08:15)
-[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
-Type "help", "copyright", "credits" or "license" for more information.
->>>
-```
-
-Then rerun the dependency installation explicitly using python 2.7
-
-```
-python2.7 -m virtualenv --python=`which python2.7` env && source env/bin/activate && python2.7 -m pip install -U pip && python2.7 -m pip install -r requirements.txt
-```
-
 ### IPsec: Difficulty connecting through router
 
 Some routers treat IPsec connections specially because older versions of IPsec did not work properly through [NAT](https://en.wikipedia.org/wiki/Network_address_translation). If you're having problems connecting to your AlgoVPN through a specific router using IPsec you might need to change some settings on the router.

install.sh

@@ -27,10 +27,10 @@ installRequirements() {
     build-essential \
     libssl-dev \
     libffi-dev \
-    python-dev \
+    python3-dev \
-    python-pip \
+    python3-pip \
-    python-setuptools \
+    python3-setuptools \
-    python-virtualenv \
+    python3-virtualenv \
     bind9-host \
     jq -y
 }
@@ -39,11 +39,11 @@ getAlgo() {
   [ ! -d "algo" ] && git clone "https://github.com/${REPO_SLUG}" -b "${REPO_BRANCH}" algo
   cd algo
 
-  python -m virtualenv --python="$(command -v python2)" .venv
+  python3 -m virtualenv --python="$(command -v python3)" .venv
   # shellcheck source=/dev/null
   . .venv/bin/activate
-  python -m pip install -U pip virtualenv
+  python3 -m pip install -U pip virtualenv
-  python -m pip install -r requirements.txt
+  python3 -m pip install -r requirements.txt
 }
 
 publicIpFromInterface() {

inventory

@@ -1,2 +1,2 @@
 [local]
-localhost ansible_connection=local ansible_python_interpreter=python
+localhost ansible_connection=local ansible_python_interpreter=python3

main.yml

@@ -25,7 +25,7 @@
         msg: >
           Ansible version is {{ ansible_version.full }}.
           You must update the requirements to use this version of Algo.
-          Try to run python -m pip install -U -r requirements.txt
+          Try to run python3 -m pip install -U -r requirements.txt
 
 - name: Include prompts playbook
   import_playbook: input.yml