Basic Python Dependencies Management

Published in

Stackademic

11 min readNov 21, 2023

In my previous blog post, I introduced some basic glossaries and provided an overview of our series on Python dependencies management. In this post, I will delve deeper into basic Python dependencies management, which is essential knowledge for any Python developer.

Pip

Pip is the entry point and the basic thing in order to start managing Python packages. Pip is simply a package manager tool that helps in managing and installing Python packages from different sources. By default, it will try to install packages from the Python Package Index (PyPI) unless otherwise specified. Starting from Python 3.4, the pip is included as part of the Python installation.

Pip can be installed in different ways:

Part of Python installation (starting from Python 3.4)
Part of a virtual environment
Using ensurepip module
Using get-pip.py script

~ pip --help
Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  inspect                     Inspect the python environment.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  cache                       Inspect and manage pip's wheel cache.
  index                       Inspect information available from package indexes.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  debug                       Show information useful for debugging.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --debug                     Let unhandled exceptions propagate outside the main subroutine, instead of logging them to stderr.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  --require-virtualenv        Allow pip to only run in a virtual environment; exit with an error otherwise.
  --python <python>           Run pip with the specified Python interpreter.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and
                              CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --no-input                  Disable prompting for input.
  --proxy <proxy>             Specify a proxy in the form scheme://[user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to PEM-encoded CA certificate bundle. If provided, overrides the default. See 'SSL Certificate
                              Verification' in pip documentation for more information.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with
                              --no-index.
  --no-color                  Suppress colored output.
  --no-python-version-warning
                              Silence deprecation warnings for upcoming unsupported Pythons.
  --use-feature <feature>     Enable new functionality, that may be backward incompatible.
  --use-deprecated <feature>  Enable deprecated functionality, that will be removed in the future.

The output above shows that pip is already installed as part of the Python installation. If pip is not installed for some reason, developers can still install pip using ensurepip and this will be useful, especially for offline environments that use at least Python 3.4.

python -m ensurepip --upgrade

Moreover, there is still another option to download the script get-pip.py and install pip locally.

~ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2544k  100 2544k    0     0  6161k      0 --:--:-- --:--:-- --:--:-- 6251k

pip supports a lot of commands and options. However, we are not going to cover all of them in this blog as we mainly going to focus on the most important ones.

Pip install

This command is used to install packages from the index registry or from any sources that store Python packages.

~ pip install requests==2.31.0
Collecting requests==2.31.0
  Obtaining dependency information for requests==2.31.0 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.pyenv/versions/3.11.2/lib/python3.11/site-packages (from requests==2.31.0) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in ./.pyenv/versions/3.11.2/lib/python3.11/site-packages (from requests==2.31.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.pyenv/versions/3.11.2/lib/python3.11/site-packages (from requests==2.31.0) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.pyenv/versions/3.11.2/lib/python3.11/site-packages (from requests==2.31.0) (2023.7.22)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Installing collected packages: requests
Successfully installed requests-2.31.0

If we do not tell the pip where to install Python packages, it will install them by default from PyPI. In some cases, PyPI may not be a good option — e.g. in offline environments or environments that depend on their own index registry for security reasons.

--index-url option can be overridden as part of the pip install command as the following:

pip install requests==2.31.0 --index-url <YOUR_OWN_URL>

or even, the --index-url can be added to the top of the requirements.txt to avoid typing the --index-url every time.

--index-url <YOUR_OWN_URL>
alembic==1.12.0

As we mentioned, there could be a case where we want to install Python packages in an offline environment where a request to the internet is not permitted to download your packages. pip already supports downloading packaging from local directories.

pip install requests --no-index --find-links file:///your-path/to/packages

pip install also provided a lot of options that we are not going to cover in this blog as our goal is to focus on the basic options needed to install Python packages.

Pip uninstall

This command just reverses the operation of the pip install and uninstall packages that were installed before.

➜ pip uninstall --help

Usage:
  pip uninstall [options] <package> ...
  pip uninstall [options] -r <requirements file> ...

Uninstalling packages can be done by either specifying the list of packages or even a list of requirements files

➜ pip uninstall -y requests
Found existing installation: requests 2.31.0
Uninstalling requests-2.31.0:
  Successfully uninstalled requests-2.31.0

➜ pip uninstall -y -r requirements.txt
Found existing installation: requests 2.31.0
Uninstalling requests-2.31.0:
  Successfully uninstalled requests-2.31.0

Pip freeze

This command helps to output installed packages in the requirements format.

Note --all include other packages like setuptools, distribute, pip, wheel

➜  ~ pip freeze --all
black=cs=23.7.0
certifi==2023.7.22
charset-normalizer==3.2.0
click==8.1.7
confluent-kafka==2.2.0
hj-domain-schemas==1.5.1
idna==3.4
isort==5.12.0
mypy-extensions==1.0.0
packaging==23.1
pathspec==0.11.2
pip==23.2.1
platformdirs==3.10.0
protobuf==4.24.0
requests==2.31.0
setuptools==65.5.0
urllib3==2.0.4

Pip list

This command, lists installed packages, including editables using different formats and various options. The output format is the columns (default)

➜  ~ pip list
Package            Version
------------------ ---------
black              23.7.0
certifi            2023.7.22
charset-normalizer 3.2.0
click              8.1.7
confluent-kafka    2.2.0
hj-domain-schemas  1.5.1
idna               3.4
isort              5.12.0
mypy-extensions    1.0.0
packaging          23.1
pathspec           0.11.2
pip                23.2.1
platformdirs       3.10.0
protobuf           4.24.0
requests           2.31.0
setuptools         65.5.0
urllib3            2.0.4

Pip show

This command helps to display information about one or more installed packages

➜  pip show black isort
Name: black
Version: 23.7.0
Summary: The uncompromising code formatter.
Home-page:
Author:
Author-email: Łukasz Langa <lukasz@langa.pl>
License: MIT
Location: /Users/mabuaisha/.pyenv/versions/3.11.2/lib/python3.11/site-packages
Requires: click, mypy-extensions, packaging, pathspec, platformdirs
Required-by:
---
Name: isort
Version: 5.12.0
Summary: A Python utility / library to sort Python imports.
Home-page: https://pycqa.github.io/isort/
Author: Timothy Crosley
Author-email: timothy.crosley@gmail.com
License: MIT
Location: /Users/mabuaisha/.pyenv/versions/3.11.2/lib/python3.11/site-packages
Requires:
Required-by:

Virtualenv

As we mentioned before the concept of a virtual environment helps the installation of the packages in an isolated environment instead of using the global Python environment.

virtualenv is a tool that creates isolated Python environments that help in the following:

Make development easier especially when we work with multiple projects using different package versions
Isolation gives the power to have the same package installed on different environment with different packages
Avoid using a single environment (global Python environment) for multiple projects which adds overhead when switching between projects
Avoid the conflicting requirements between packages.

virtualenv does not support out-of-the-box switching and managing different Python versions. It only focuses on creating isolated environments. It can be installed simply using pip and start using it without any problem to create virtual environments.

Creating virtual environments is simple as all you need to do is:

Make sure that you have virtualenv installed
Create a virtual environment by specifying the name and the location
Activate the virtual environment in order to start managing the packages you want to modify in your environment

➜  pip install virtualenv
Collecting virtualenv
  Obtaining dependency information for virtualenv from https://files.pythonhosted.org/packages/4e/8b/f0d3a468c0186c603217a6656ea4f49259630e8ed99558501d92f6ff7dc3/virtualenv-20.24.5-py3-none-any.whl.metadata
  Downloading virtualenv-20.24.5-py3-none-any.whl.metadata (4.5 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv)
  Obtaining dependency information for distlib<1,>=0.3.7 from https://files.pythonhosted.org/packages/43/a0/9ba967fdbd55293bacfc1507f58e316f740a3b231fc00e3d86dc39bc185a/distlib-0.3.7-py2.py3-none-any.whl.metadata
  Downloading distlib-0.3.7-py2.py3-none-any.whl.metadata (5.1 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv)
  Obtaining dependency information for filelock<4,>=3.12.2 from https://files.pythonhosted.org/packages/5e/5d/97afbafd9d584ff1b45fcb354a479a3609bd97f912f8f1f6c563cb1fae21/filelock-3.12.4-py3-none-any.whl.metadata
  Downloading filelock-3.12.4-py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: platformdirs<4,>=3.9.1 in /Users/mabuaisha/.pyenv/versions/3.11.2/lib/python3.11/site-packages (from virtualenv) (3.10.0)
Downloading virtualenv-20.24.5-py3-none-any.whl (3.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 4.2 MB/s eta 0:00:00
Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 kB 324.1 kB/s eta 0:00:00
Downloading filelock-3.12.4-py3-none-any.whl (11 kB)
Installing collected packages: distlib, filelock, virtualenv
Successfully installed distlib-0.3.7 filelock-3.12.4 virtualenv-20.24.5

➜  ~ virtualenv test_env
created virtual environment CPython3.11.2.final.0-64 in 1696ms
  creator CPython3Posix(dest=/Users/mabuaisha/test_env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mabuaisha/Library/Application Support/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.0, wheel==0.41.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator➜  ~ source test_env/bin/activate(test_env) ➜  ~

virtualenv has two main phases:

Discover the Python interpreter to create the desired virtual environment. By default, it will be the Python interpreter where virtualenv running from. The option-p/--python can be specified in order to choose which interpreter to create an environment from. So if we have multiple Python versions installed in our system, virtualenv can choose from which Python version we need to create our virtualenv (There are other tools like pyenv that helps to achieve this goal as well)
Create the virtual environment at the desired location. By default, it will be created in the current directory where you run the command.

Running the above command will create a directory with some bin and lib sub-directories.

(test_env) ➜  test_env ls -lhtra
drwxr-xr-x@  3 mabuaisha  staff    96B Oct 15 12:18 lib
drwxr-xr-x@ 19 mabuaisha  staff   608B Oct 15 12:18 bin

The bin directory contains activate, pip, and links to the Python interpreter.

(test_env) ➜  bin ls -lh
-rw-r--r--@ 1 mabuaisha  staff   2.2K Oct 15 12:18 activate
-rw-r--r--@ 1 mabuaisha  staff   1.5K Oct 15 12:18 activate.csh
-rw-r--r--@ 1 mabuaisha  staff   3.0K Oct 15 12:18 activate.fish
-rw-r--r--@ 1 mabuaisha  staff   2.7K Oct 15 12:18 activate.nu
-rw-r--r--@ 1 mabuaisha  staff   1.6K Oct 15 12:18 activate.ps1
-rw-r--r--@ 1 mabuaisha  staff   1.3K Oct 15 12:18 activate_this.py
-rwxr-xr-x@ 1 mabuaisha  staff   241B Oct 15 12:18 pip
-rwxr-xr-x@ 1 mabuaisha  staff   241B Oct 15 12:18 pip-3.11
-rwxr-xr-x@ 1 mabuaisha  staff   241B Oct 15 12:18 pip3
-rwxr-xr-x@ 1 mabuaisha  staff   241B Oct 15 12:18 pip3.11
lrwxr-xr-x@ 1 mabuaisha  staff    54B Oct 15 12:18 python -> /Users/mabuaisha/.pyenv/versions/3.11.2/bin/python3.11
lrwxr-xr-x@ 1 mabuaisha  staff     6B Oct 15 12:18 python3 -> python
lrwxr-xr-x@ 1 mabuaisha  staff     6B Oct 15 12:18 python3.11 -> python
-rwxr-xr-x@ 1 mabuaisha  staff   228B Oct 15 12:18 wheel
-rwxr-xr-x@ 1 mabuaisha  staff   228B Oct 15 12:18 wheel-3.11
-rwxr-xr-x@ 1 mabuaisha  staff   228B Oct 15 12:18 wheel3
-rwxr-xr-x@ 1 mabuaisha  staff   228B Oct 15 12:18 wheel3.11

On, the other hand, the lib, contains the site-packages which represents the installed packages

(test_env) ➜  site-packages pwd
/Users/mabuaisha/test_env/lib/python3.11/site-packages
(test_env) ➜  site-packages ls -lhtra
drwxr-xr-x@  3 mabuaisha  staff    96B Oct 15 12:18 ..
-rw-r--r--@  1 mabuaisha  staff    18B Oct 15 12:18 _virtualenv.pth
-rw-r--r--@  1 mabuaisha  staff   4.2K Oct 15 12:18 _virtualenv.py
-rw-r--r--@  1 mabuaisha  staff     0B Oct 15 12:18 setuptools-68.2.0.virtualenv
-rw-r--r--@  1 mabuaisha  staff   151B Oct 15 12:18 distutils-precedence.pth
-rw-r--r--@  1 mabuaisha  staff     0B Oct 15 12:18 pip-23.2.1.virtualenv
drwxr-xr-x@  9 mabuaisha  staff   288B Oct 15 12:18 setuptools-68.2.0.dist-info
drwxr-xr-x@ 12 mabuaisha  staff   384B Oct 15 12:18 wheel
-rw-r--r--@  1 mabuaisha  staff     0B Oct 15 12:18 wheel-0.41.2.virtualenv
drwxr-xr-x@  8 mabuaisha  staff   256B Oct 15 12:18 wheel-0.41.2.dist-info
drwxr-xr-x@ 49 mabuaisha  staff   1.5K Oct 15 12:18 setuptools
drwxr-xr-x@  5 mabuaisha  staff   160B Oct 15 12:18 pkg_resources
drwxr-xr-x@ 10 mabuaisha  staff   320B Oct 15 12:18 pip-23.2.1.dist-info
drwxr-xr-x@  5 mabuaisha  staff   160B Oct 15 12:35 _distutils_hack
drwxr-xr-x@  9 mabuaisha  staff   288B Oct 15 12:35 pip
drwxr-xr-x@ 12 mabuaisha  staff   384B Oct 15 12:36 platformdirs
drwxr-xr-x@  7 mabuaisha  staff   224B Oct 15 12:36 platformdirs-3.11.0.dist-info
drwxr-xr-x@ 11 mabuaisha  staff   352B Oct 15 12:36 pathspec
drwxr-xr-x@  7 mabuaisha  staff   224B Oct 15 12:36 pathspec-0.11.2.dist-info
drwxr-xr-x@ 18 mabuaisha  staff   576B Oct 15 12:36 packaging
drwxr-xr-x@  9 mabuaisha  staff   288B Oct 15 12:36 packaging-23.2.dist-info
-rw-r--r--@  1 mabuaisha  staff   6.1K Oct 15 12:36 mypy_extensions.py
drwxr-xr-x@  8 mabuaisha  staff   256B Oct 15 12:36 mypy_extensions-1.0.0.dist-info
drwxr-xr-x@ 20 mabuaisha  staff   640B Oct 15 12:36 click
drwxr-xr-x@  8 mabuaisha  staff   256B Oct 15 12:36 click-8.1.7.dist-info
-rwxr-xr-x@  1 mabuaisha  staff   6.7M Oct 15 12:36 2ec0e72aa72355e6eccf__mypyc.cpython-311-darwin.so
-rw-r--r--@  1 mabuaisha  staff    19B Oct 15 12:36 _black_version.py
drwxr-xr-x@ 33 mabuaisha  staff   1.0K Oct 15 12:36 .
drwxr-xr-x@  5 mabuaisha  staff   160B Oct 15 12:36 __pycache__
drwxr-xr-x@ 42 mabuaisha  staff   1.3K Oct 15 12:36 black
drwxr-xr-x@  6 mabuaisha  staff   192B Oct 15 12:36 blackd
drwxr-xr-x@ 13 mabuaisha  staff   416B Oct 15 12:36 blib2to3
drwxr-xr-x@  9 mabuaisha  staff   288B Oct 15 12:36 black-23.9.1.dist-info

The exit from the active virtual enviroment can be simply done by running deactivate command.

Venv

Starting from Python 3.3, a subset of the virtualenv has been integrated into the standard library under a new module called venv module. venv supports creating lightweight “virtual environments” without having the full features that virtualenv support.

➜  ~ python -m venv --help
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
            ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.positional arguments:
  ENV_DIR               A directory to create the environment in.options:
  -h, --help            show this help message and exit
  --system-site-packages
                        Give the virtual environment access to the system site-packages dir.
  --symlinks            Try to use symlinks rather than copies, when symlinks are not the default for the platform.
  --copies              Try to use copies rather than symlinks, even when symlinks are the default for the platform.
  --clear               Delete the contents of the environment directory if it already exists, before environment creation.
  --upgrade             Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place.
  --without-pip         Skips installing or upgrading pip in the virtual environment (pip is bootstrapped by default)
  --prompt PROMPT       Provides an alternative prompt prefix for this environment.
  --upgrade-deps        Upgrade core dependencies: pip setuptools to the latest version in PyPI

Simply, a virtual environment can be created natively using Python if you have Python ≥=3.3

➜  ~ python -m venv venv
➜  ~ source venv/bin/activate
(venv) ➜  ~

Summary

In this blog post, we have discussed various tools for managing Python dependencies. These tools are crucial for developing Python applications. However, they have their limitations and cannot solve problems like deterministic builds. We will discuss that in more detail in our next blog post.

Stackademic

Thank you for reading until the end. Before you go:

Please consider clapping and following the writer! 👏
Follow us on Twitter(X), LinkedIn, and YouTube.
Visit Stackademic.com to find out more about how we are democratizing free programming education around the world.

Basic Python Dependencies Management

Pip

Virtualenv

Venv

Summary

Stackademic

Written by Mohammed AbuAisha