Using venv, pyvenv, autoenv on macOS

At some point, you might work on a Python project that requires specific dependencies, such as a machine learning project with an exact PyTorch version. It becomes a necessity to structure your workflow in order to avoid conflicts and iterate quickly across projects.

In this post I will explain the main tools that I use on macOS and show a neat trick in order to switch virtual environments automatically.

Why Use Virtual Environments?

The ability to replicate environments not only makes onboarding easier but also minimizes the « works on my machine » issue. To accomplish our goal, we will need three tools: pyenv, venv, and autoenv.

Pyenv

Pyenv is an incredible tool that allows you to switch between multiple versions of Python. You can even search and install Python versions and set local and global versions.

Venv

Venv is a built-in and simple method for creating isolated Python environments. While pyenv (through pyenv-virtualenv) could be used for isolating projects you should use venv.

Autoenv

Autoenv is magical tool that just makes using virtual environments seamless and uses .env and .env.leave files to activate and deactivate environments.

Setting up the tools

The tools require homebrew or another package manager and the ability to modify your shell.

We first need to install pyenv:

1
brew install pyenv

Then, we need to append the following lines to our .zshrc file:

1
if command -v pyenv 1>/dev/null 2>&1; then eval "$(pyenv init -)"; fi

You can refer to the documentation but the minimal commands are:

1
pyenv versions
2
pyenv install your_python_version
3
pyenv global your_python_version
4
pyenv local your_python_version

Finally we need autoenv:

1
brew install autoenv

Then executing the following in your zsh shell:

1
printf '%s\n' "source $(brew --prefix autoenv)/activate.sh" >> "${ZDOTDIR:-$HOME}/.zprofile"

Before using the tool you should read the documentation and activate the « AUTOENV_ENABLE_LEAVE » option by setting it to any non empty string.

Practical workflow

Create or clone your project:

1
mkdir exllama && cd exllama
2
git clone https://github.com/turboderp/exllama

Set with pyenv the local python version needed:

1
pyenv local 3.10.10

Create the virtual environment:

1
python3 -m venv venv

Add the .env and .env.leave and approve the autoenv changes:

1
# .env file for autoenv
2
# It looks quite cryptic but it's to preserve the virtual environment state across sub folders
3
venv_dir="venv"
4
currentvenv=""
5

6
# Function to traverse up the directory structure to find the parent directory containing venv_folder
7
get_project_root() {
8
    local current_dir="$PWD"
9
    while [[ "$current_dir" != "" && ! -d "$current_dir/$venv_dir" ]]; do
10
        current_dir=${current_dir%/*}
11
    done
12
    echo "$current_dir"
13
}
14

15
root_dir=$(get_project_root)
16

17
if [[ -z "$root_dir" || ! -d "$root_dir/$venv_dir" ]]; then
18
    echo "Unable to find the virtual environment folder."
19
    return
20
fi
21

22
if [[ $VIRTUAL_ENV != "" ]]; then
23
    # Strip out the path and just leave the env name
24
    currentvenv="${VIRTUAL_ENV##*/}"
25
fi
26

27
if [[ "$currentvenv" != "$venv_dir" ]]; then
28
    python_version=$(python --version 2>&1)
29
    echo "Switching to environment: $venv_dir | $python_version"
30
    # Source the activation script
31
    source "$root_dir/$venv_dir/bin/activate"
32
fi

1
# .env.leave for autoenv
2
deactivate

Conclusion

You are now ready to start your development with a clean and isolated environment!

If you found this guide useful you can also check the previous ones about Setting up macOS for development and AI Homelab: A guide into hardware to software considerations.