Advanced Python Dependencies Management

Published in

Stackademic

22 min readNov 21, 2023

In my previous blog post, I discussed the fundamental tools that can be used for managing Python dependencies. However, tools such as pip, virtualenv, or venv have certain limitations, such as ensuring deterministic builds. In this post, I will delve into more detail about some advanced tools that are considered important in order to overcome issues like this.

Pip Tools

Making your Python environment’s build deterministic is important. You can easily achieve this by pinning your dependencies to the stable versions in requirements.txt.

So instead of letting pip install the latest version always, we need to make sure to specify the desired version to avoid any conflicts with other dependencies, unstable behavior of the latest version,…etc. For example, if we have the Django framework pinned to 4.2.6

Django==4.2.6

Django depends on other packages like asgiref >= 3.7.0 & sqlparse >= 0.3.1 . By default, pip installs the latest version of both packages, which may not be desirable.

Manually adding all the sub-dependencies to the requirements.txt file can be a solution to the problem. However, it can become particularly challenging when your project has multiple dependencies, and each of these dependencies has further sub-dependencies. Updating the files manually under such circumstances would require a significant amount of effort and time..

(venv) ➜  venv pip freeze
asgiref==3.7.2
Django==4.2.6
sqlparse==0.4.4

In addition, pip alone is unable to handle dependency resolution. Dependency resolution comes into play when your main dependencies share sub-dependencies with varying versions. Using pip is insufficient to solve this problem, and will result in failure while attempting to install the dependencies. To illustrate, consider the following scenario:

package_x
package_y

package_x depends on the sub-package called package_z >= 2.0.0 However, package_y also depends on the same sub-package but with a different version package_z <= 3.0.0 . Using pip to install the above packages will fail as pip is not smart enough to resolve these dependencies

pip will try to install package_x and fulfill its requirements and install the latest version of package_z which could be 4.0.0
When it comes to install the package_y which has package_z <= 3.0.0 it cannot install that because package_z already installed with the latest version 4.0.0

I haven’t talked about pip-tools yet, but I need to explain why we may need to use them to address the previous issues.

pip-tools is a set of command line tools that helps to build predictable and deterministic dependencies for your Python application and synchronize your changes to your virtual environment. pip-tools ships with two tools which are pip-compile & pip-sync .

pip-compile

This tool helps to generate and compile your requirements.txt files from different sources. Mainly, pip-compile helps in solving the issues we mentioned before including the dependency resolution. pip-compile supports compiling your dependencies from the following sources:

pyproject.toml
setup.cfg
setup.py
requirements.in

We are going to focus on requirements.insince it is the most common one. pip-tools can be simply installed using pip as the following:

(env) ➜  ~ pip install pip-tools
Collecting pip-tools
  Downloading pip_tools-7.3.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━ 57.4/57.4 kB 1.3 MB/s eta 0:00:00
Collecting build
  Downloading build-1.0.3-py3-none-any.whl (18 kB)
Collecting click>=8
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Requirement already satisfied: pip>=22.2 in ./env/lib/python3.11/site-packages (from pip-tools) (22.3.1)
Requirement already satisfied: setuptools in ./env/lib/python3.11/site-packages (from pip-tools) (65.5.0)
Collecting wheel
  Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
Collecting packaging>=19.0
  Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting pyproject_hooks
  Downloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Installing collected packages: wheel, pyproject_hooks, packaging, click, build, pip-tools
Successfully installed build-1.0.3 click-8.1.7 packaging-23.2 pip-tools-7.3.0 pyproject_hooks-1.0.0 wheel-0.41.2

All you need to do is to define the source file that needs to be compiled. We can compile our source easily into deterministic dependencies using pip-compile

(env) ➜  pip-tools pip-compile -o requirements.txt requirements.in
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
asgiref==3.7.2
    # via django
django==4.2.6
    # via -r requirements.in
sqlparse==0.4.4
    # via django

The above command compiles the requiremets.in into dependencies that also include the sub-dependencies for Django

Some notes to consider:

By default pip-compile will try to read from requirements.in if not specified in the current directory
By default pip-compile will compile the results into requirements.txt in the current directory if not specified
You can use -q/--quiet to avoid less output
You can use -U/--upgrade option in order to force upgrade all dependencies to their latest versions.

pip-sync

It helps to sync all of your dependencies into the active virtual environment as it detects the changes and tries to always make the compiled requirements.txt match the virtual environment. This means that pip-sync can install/uninstall dependencies based on the state of the compiled dependencies files.

(env) ➜  pip-tools pip-sync
Collecting asgiref==3.7.2
  Using cached asgiref-3.7.2-py3-none-any.whl (24 kB)
Collecting django==4.2.6
  Using cached Django-4.2.6-py3-none-any.whl (8.0 MB)
Collecting sqlparse==0.4.4
  Using cached sqlparse-0.4.4-py3-none-any.whl (41 kB)
Installing collected packages: sqlparse, asgiref, django
Successfully installed asgiref-3.7.2 django-4.2.6 sqlparse-0.4.4

(env) ➜  pip-tools pip freeze
asgiref==3.7.2
Django==4.2.6
sqlparse==0.4.4

Pipenv

Pipenv is a Python management tool that simplifies the process of managing dependencies by using only one tool instead of using multiple tools separately. In addition to that, pipenv helps in solving the common issues that we explained before and even provides some other cool features that we are going to explain in a bit.

Installation

Simply, pipenv can be installed using different options and among these options is pip

pip install --user pipenv

➜ pipenv --help
Usage: pipenv [OPTIONS] COMMAND [ARGS]...

Options:
  --where                         Output project home information.
  --venv                          Output virtualenv information.
  --py                            Output Python interpreter information.
  --envs                          Output Environment Variable options.
  --rm                            Remove the virtualenv.
  --bare                          Minimal output.
  --man                           Display manpage.
  --support                       Output diagnostic information for use in
                                  GitHub issues.
  --site-packages / --no-site-packages
                                  Enable site-packages for the virtualenv.
                                  [env var: PIPENV_SITE_PACKAGES]
  --python TEXT                   Specify which version of Python virtualenv
                                  should use.
  --clear                         Clears caches (pipenv, pip).  [env var:
                                  PIPENV_CLEAR]
  -q, --quiet                     Quiet mode.
  -v, --verbose                   Verbose mode.
  --pypi-mirror TEXT              Specify a PyPI mirror.
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.
Usage Examples:
   Create a new project using Python 3.7, specifically:
   $ pipenv --python 3.7   Remove project virtualenv (inferred from current directory):
   $ pipenv --rm   Install all dependencies for a project (including dev):
   $ pipenv install --dev   Create a lockfile containing pre-releases:
   $ pipenv lock --pre   Show a graph of your installed dependencies:
   $ pipenv graph   Check your installed dependencies for security vulnerabilities:
   $ pipenv check   Install a local setup.py into your virtual environment/Pipfile:
   $ pipenv install -e .   Use a lower-level pip command:
   $ pipenv run pip freezeCommands:
  check         Checks for PyUp Safety security vulnerabilities and against
                PEP 508 markers provided in Pipfile.
  clean         Uninstalls all packages not specified in Pipfile.lock.
  graph         Displays currently-installed dependency graph information.
  install       Installs provided packages and adds them to Pipfile, or (if no
                packages are given), installs all packages from Pipfile.
  lock          Generates Pipfile.lock.
  open          View a given module in your editor.
  requirements  Generate a requirements.txt from Pipfile.lock.
  run           Spawns a command installed into the virtualenv.
  scripts       Lists scripts in current environment config.
  shell         Spawns a shell within the virtualenv.
  sync          Installs all packages specified in Pipfile.lock.
  uninstall     Uninstalls a provided package and removes it from Pipfile.
  update        Runs lock, then sync.
  upgrade       Resolves provided packages and adds them to Pipfile, or (if no
                packages are given), merges results to Pipfile.lock
  verify        Verify the hash in Pipfile.lock is up-to-date.

pipenv provides a lot of options when it comes to manage dependencies. These are the top features that pipenv provide:

Predictable and deterministic build.
Create a virtual environment automatically in the standard location.
Create Pipfile & Pipfile.lock automatically.
Update Pipfile when adding/removing packages.
Loads the .env file if it exists (This is useful for testing purposes)
Install the required Python version automatically. (When pyenv is installed)

Pipfile

The Pipfile is the format the pipenv used to manage the dependencies. Mainly, two files exist in order to handle the dependencies: Pipfile & Pipfile.lock . The Pipfile and its sister is a new standard that comes as a replacement for the requirement.txt . The pypy is the one that introduced this format.

Pipfile is the main file that contains the primary dependencies and is used as a source to generate the Pipfile.lock which represents all dependencies including the sub-dependencies to build a predictable and deterministic environment.

Pipenv Install

Let us imagine that we want to install requests==2.30.0 using pipenv

➜  pipenv install requests==2.30.0
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
Creating a virtualenv for this project...
Pipfile: /Users/mabuaisha/test/Pipfile
Using default python from /Users/mabuaisha/.local/pipx/venvs/pipenv/bin/python (3.11.2) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.11.2.final.0-64 in 313ms
  creator CPython3Posix(dest=/Users/mabuaisha/.local/share/virtualenvs/test-CfuChXcD, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mabuaisha/Library/Application Support/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /Users/mabuaisha/.local/share/virtualenvs/test-CfuChXcD
Creating a Pipfile for this project...
Installing requests==2.30.0...
Resolving requests==2.30.0...
Added requests to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (d49ddfed48295f4e1ec62cfa5bdd6c43789b6415d94931a84c88d7ecff073c99)!
Installing dependencies from Pipfile.lock (073c99)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.s

✔ Successfully created virtual environment!
Virtualenv location: /Users/mabuaisha/.local/share/virtualenvs/test-CfuChXcD
Creating a Pipfile for this project...
Installing requests==2.30.0...
Resolving requests==2.30.0...
Added requests to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (d49ddfed48295f4e1ec62cfa5bdd6c43789b6415d94931a84c88d7ecff073c99)!
Installing dependencies from Pipfile.lock (073c99)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.s

Automatically create virtualenv under ~/.local/share/virtualenvs/test-CfuChXcD
Automatically create Pipfile if it does not exist
Automatically create Pipfile.lock if it does not exist
Automatically add the requests to the Pipfile and resolve all the sub-dependencies and dump them into Pipfile.lock

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.30.0"

[dev-packages]

[requires]
python_version = "3.11"

The above file represents the content of the Pipfile . The Pipfile.lock contains detailed information about the packages and sub-packages. Moreover, it contains the sha256 hashes of each downloaded package to make sure to get the same packages on any network as specified in the lock file, even on untrusted networks. An example of how the Pipfile.lock looks like can be found below.

{
    "_meta": {
        "hash": {
            "sha256": "d49ddfed48295f4e1ec62cfa5bdd6c43789b6415d94931a84c88d7ecff073c99"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082",
                "sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2023.7.22"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:02673e456dc5ab13659f85196c534dc596d4ef260e4d86e856c3b2773ce09843",
                "sha256:02af06682e3590ab952599fbadac535ede5d60d78848e555aa58d0c0abbde786",
                "sha256:03680bb39035fbcffe828eae9c3f8afc0428c91d38e7d61aa992ef7a59fb120e",
                "sha256:0570d21da019941634a531444364f2482e8db0b3425fcd5ac0c36565a64142c8",
                "sha256:09c77f964f351a7369cc343911e0df63e762e42bac24cd7d18525961c81754f4",
                "sha256:0d3d5b7db9ed8a2b11a774db2bbea7ba1884430a205dbd54a32d61d7c2a190fa",
                "sha256:1063da2c85b95f2d1a430f1c33b55c9c17ffaf5e612e10aeaad641c55a9e2b9d",
                "sha256:12ebea541c44fdc88ccb794a13fe861cc5e35d64ed689513a5c03d05b53b7c82",
                "sha256:153e7b6e724761741e0974fc4dcd406d35ba70b92bfe3fedcb497226c93b9da7",
                "sha256:15b26ddf78d57f1d143bdf32e820fd8935d36abe8a25eb9ec0b5a71c82eb3895",
                "sha256:1872d01ac8c618a8da634e232f24793883d6e456a66593135aeafe3784b0848d",
                "sha256:187d18082694a29005ba2944c882344b6748d5be69e3a89bf3cc9d878e548d5a",
                "sha256:1b2919306936ac6efb3aed1fbf81039f7087ddadb3160882a57ee2ff74fd2382",
                "sha256:232ac332403e37e4a03d209a3f92ed9071f7d3dbda70e2a5e9cff1c4ba9f0678",
                "sha256:23e8565ab7ff33218530bc817922fae827420f143479b753104ab801145b1d5b",
                "sha256:24817cb02cbef7cd499f7c9a2735286b4782bd47a5b3516a0e84c50eab44b98e",
                "sha256:249c6470a2b60935bafd1d1d13cd613f8cd8388d53461c67397ee6a0f5dce741",
                "sha256:24a91a981f185721542a0b7c92e9054b7ab4fea0508a795846bc5b0abf8118d4",
                "sha256:2502dd2a736c879c0f0d3e2161e74d9907231e25d35794584b1ca5284e43f596",
                "sha256:250c9eb0f4600361dd80d46112213dff2286231d92d3e52af1e5a6083d10cad9",
                "sha256:278c296c6f96fa686d74eb449ea1697f3c03dc28b75f873b65b5201806346a69",
                "sha256:2935ffc78db9645cb2086c2f8f4cfd23d9b73cc0dc80334bc30aac6f03f68f8c",
                "sha256:2f4a0033ce9a76e391542c182f0d48d084855b5fcba5010f707c8e8c34663d77",
                "sha256:30a85aed0b864ac88309b7d94be09f6046c834ef60762a8833b660139cfbad13",
                "sha256:380c4bde80bce25c6e4f77b19386f5ec9db230df9f2f2ac1e5ad7af2caa70459",
                "sha256:3ae38d325b512f63f8da31f826e6cb6c367336f95e418137286ba362925c877e",
                "sha256:3b447982ad46348c02cb90d230b75ac34e9886273df3a93eec0539308a6296d7",
                "sha256:3debd1150027933210c2fc321527c2299118aa929c2f5a0a80ab6953e3bd1908",
                "sha256:4162918ef3098851fcd8a628bf9b6a98d10c380725df9e04caf5ca6dd48c847a",
                "sha256:468d2a840567b13a590e67dd276c570f8de00ed767ecc611994c301d0f8c014f",
                "sha256:4cc152c5dd831641e995764f9f0b6589519f6f5123258ccaca8c6d34572fefa8",
                "sha256:542da1178c1c6af8873e143910e2269add130a299c9106eef2594e15dae5e482",
                "sha256:557b21a44ceac6c6b9773bc65aa1b4cc3e248a5ad2f5b914b91579a32e22204d",
                "sha256:5707a746c6083a3a74b46b3a631d78d129edab06195a92a8ece755aac25a3f3d",
                "sha256:588245972aca710b5b68802c8cad9edaa98589b1b42ad2b53accd6910dad3545",
                "sha256:5adf257bd58c1b8632046bbe43ee38c04e1038e9d37de9c57a94d6bd6ce5da34",
                "sha256:619d1c96099be5823db34fe89e2582b336b5b074a7f47f819d6b3a57ff7bdb86",
                "sha256:63563193aec44bce707e0c5ca64ff69fa72ed7cf34ce6e11d5127555756fd2f6",
                "sha256:67b8cc9574bb518ec76dc8e705d4c39ae78bb96237cb533edac149352c1f39fe",
                "sha256:6a685067d05e46641d5d1623d7c7fdf15a357546cbb2f71b0ebde91b175ffc3e",
                "sha256:70f1d09c0d7748b73290b29219e854b3207aea922f839437870d8cc2168e31cc",
                "sha256:750b446b2ffce1739e8578576092179160f6d26bd5e23eb1789c4d64d5af7dc7",
                "sha256:7966951325782121e67c81299a031f4c115615e68046f79b85856b86ebffc4cd",
                "sha256:7b8b8bf1189b3ba9b8de5c8db4d541b406611a71a955bbbd7385bbc45fcb786c",
                "sha256:7f5d10bae5d78e4551b7be7a9b29643a95aded9d0f602aa2ba584f0388e7a557",
                "sha256:805dfea4ca10411a5296bcc75638017215a93ffb584c9e344731eef0dcfb026a",
                "sha256:81bf654678e575403736b85ba3a7867e31c2c30a69bc57fe88e3ace52fb17b89",
                "sha256:82eb849f085624f6a607538ee7b83a6d8126df6d2f7d3b319cb837b289123078",
                "sha256:85a32721ddde63c9df9ebb0d2045b9691d9750cb139c161c80e500d210f5e26e",
                "sha256:86d1f65ac145e2c9ed71d8ffb1905e9bba3a91ae29ba55b4c46ae6fc31d7c0d4",
                "sha256:86f63face3a527284f7bb8a9d4f78988e3c06823f7bea2bd6f0e0e9298ca0403",
                "sha256:8eaf82f0eccd1505cf39a45a6bd0a8cf1c70dcfc30dba338207a969d91b965c0",
                "sha256:93aa7eef6ee71c629b51ef873991d6911b906d7312c6e8e99790c0f33c576f89",
                "sha256:96c2b49eb6a72c0e4991d62406e365d87067ca14c1a729a870d22354e6f68115",
                "sha256:9cf3126b85822c4e53aa28c7ec9869b924d6fcfb76e77a45c44b83d91afd74f9",
                "sha256:9fe359b2e3a7729010060fbca442ca225280c16e923b37db0e955ac2a2b72a05",
                "sha256:a0ac5e7015a5920cfce654c06618ec40c33e12801711da6b4258af59a8eff00a",
                "sha256:a3f93dab657839dfa61025056606600a11d0b696d79386f974e459a3fbc568ec",
                "sha256:a4b71f4d1765639372a3b32d2638197f5cd5221b19531f9245fcc9ee62d38f56",
                "sha256:aae32c93e0f64469f74ccc730a7cb21c7610af3a775157e50bbd38f816536b38",
                "sha256:aaf7b34c5bc56b38c931a54f7952f1ff0ae77a2e82496583b247f7c969eb1479",
                "sha256:abecce40dfebbfa6abf8e324e1860092eeca6f7375c8c4e655a8afb61af58f2c",
                "sha256:abf0d9f45ea5fb95051c8bfe43cb40cda383772f7e5023a83cc481ca2604d74e",
                "sha256:ac71b2977fb90c35d41c9453116e283fac47bb9096ad917b8819ca8b943abecd",
                "sha256:ada214c6fa40f8d800e575de6b91a40d0548139e5dc457d2ebb61470abf50186",
                "sha256:b09719a17a2301178fac4470d54b1680b18a5048b481cb8890e1ef820cb80455",
                "sha256:b1121de0e9d6e6ca08289583d7491e7fcb18a439305b34a30b20d8215922d43c",
                "sha256:b3b2316b25644b23b54a6f6401074cebcecd1244c0b8e80111c9a3f1c8e83d65",
                "sha256:b3d9b48ee6e3967b7901c052b670c7dda6deb812c309439adaffdec55c6d7b78",
                "sha256:b5bcf60a228acae568e9911f410f9d9e0d43197d030ae5799e20dca8df588287",
                "sha256:b8f3307af845803fb0b060ab76cf6dd3a13adc15b6b451f54281d25911eb92df",
                "sha256:c2af80fb58f0f24b3f3adcb9148e6203fa67dd3f61c4af146ecad033024dde43",
                "sha256:c350354efb159b8767a6244c166f66e67506e06c8924ed74669b2c70bc8735b1",
                "sha256:c5a74c359b2d47d26cdbbc7845e9662d6b08a1e915eb015d044729e92e7050b7",
                "sha256:c71f16da1ed8949774ef79f4a0260d28b83b3a50c6576f8f4f0288d109777989",
                "sha256:d47ecf253780c90ee181d4d871cd655a789da937454045b17b5798da9393901a",
                "sha256:d7eff0f27edc5afa9e405f7165f85a6d782d308f3b6b9d96016c010597958e63",
                "sha256:d97d85fa63f315a8bdaba2af9a6a686e0eceab77b3089af45133252618e70884",
                "sha256:db756e48f9c5c607b5e33dd36b1d5872d0422e960145b08ab0ec7fd420e9d649",
                "sha256:dc45229747b67ffc441b3de2f3ae5e62877a282ea828a5bdb67883c4ee4a8810",
                "sha256:e0fc42822278451bc13a2e8626cf2218ba570f27856b536e00cfa53099724828",
                "sha256:e39c7eb31e3f5b1f88caff88bcff1b7f8334975b46f6ac6e9fc725d829bc35d4",
                "sha256:e46cd37076971c1040fc8c41273a8b3e2c624ce4f2be3f5dfcb7a430c1d3acc2",
                "sha256:e5c1502d4ace69a179305abb3f0bb6141cbe4714bc9b31d427329a95acfc8bdd",
                "sha256:edfe077ab09442d4ef3c52cb1f9dab89bff02f4524afc0acf2d46be17dc479f5",
                "sha256:effe5406c9bd748a871dbcaf3ac69167c38d72db8c9baf3ff954c344f31c4cbe",
                "sha256:f0d1e3732768fecb052d90d62b220af62ead5748ac51ef61e7b32c266cac9293",
                "sha256:f5969baeaea61c97efa706b9b107dcba02784b1601c74ac84f2a532ea079403e",
                "sha256:f8888e31e3a85943743f8fc15e71536bda1c81d5aa36d014a3c0c44481d7db6e",
                "sha256:fc52b79d83a3fe3a360902d3f5d79073a993597d48114c29485e9431092905d8"
            ],
            "markers": "python_full_version >= '3.7.0'",
            "version": "==3.3.0"
        },
        "idna": {
            "hashes": [
                "sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4",
                "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.4"
        },
        "requests": {
            "hashes": [
                "sha256:10e94cc4f3121ee6da529d358cdaeaff2f1c409cd377dbc72b825852f2f7e294",
                "sha256:239d7d4458afcb28a692cdd298d87542235f4ca8d36d03a15bfc128a6559a2f4"
            ],
            "index": "pypi",
            "markers": "python_version >= '3.7'",
            "version": "==2.30.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:c97dfde1f7bd43a71c8d2a58e369e9b2bf692d1334ea9f9cae55add7d0dd0f84",
                "sha256:fdb6d215c776278489906c2f8916e6e7d4f5a9b602ccbcfdf7f016fc8da0596e"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==2.0.7"
        }
    },
    "develop": {}
}

Pipenv Graph

Pipenv provides the option to display the dependencies using the graph command.

➜  pipenv graph
requests==2.30.0
├── certifi [required: >=2017.4.17, installed: 2023.7.22]
├── charset-normalizer [required: >=2,<4, installed: 3.3.0]
├── idna [required: >=2.5,<4, installed: 3.4]
└── urllib3 [required: >=1.21.1,<3, installed: 2.0.7]

Pipenv Shell

This command will activate the virtual environment if it exists and create a new one if it does not exist.

➜  pipenv shell
Launching subshell in virtual environment...
 . /Users/mabuaisha/.local/share/virtualenvs/test-CfuChXcD/bin/activate
➜  test  . /Users/mabuaisha/.local/share/virtualenvs/test-CfuChXcD/bin/activate
(test) ➜

Deactivate the virtual environment can be done by typing exit .

Pipenv Upgrade

In case we want to upgrade one of our primary packages in Pipfile we can do that easily without manually updating the Pipfile or touch the Pipfile.lock

(test) ➜  pipenv upgrade requests==2.31.0
Building requirements...
Resolving dependencies...
✔ Success!
Building requirements...
Resolving dependencies...
✔ Success!

The above command will update both Pipfile & Pipfile.lock

Pipenv Lock

This command will generate a new Pipfile.lock if it does not exist. If the file already exists, then pipenv try to sync it Pipfile to make both of them compatible and up-to-date.

Some Notes about the locking:

Never let your CI/CD run any lock files. The locked file should be committed as part of the PR
Whenever we install/uninstall any packages, the lock file will be updated automatically
The Pipfile.lock should be the file used in order to install packages

Pipenv Sync

In case you want to install the packages from the Pipfile.lock which I think in my opinion is the option that you should use always, then pipenv sync is the one to go with.

Let us imagine you join a new team and would like to test the project and install all the packages from Pipfile.lock .

(test) ➜  pipenv sync
Installing dependencies from Pipfile.lock (121ad4)...
All dependencies are now up-to-date!

Pipenv Requirements

Pipenv supports also generating requirements.txt fromPipfile.lock in case you want still to use the old format.

pipenv requirements > requirements.txt

-- requirements.txt
-i https://pypi.org/simple
blinker==1.6.3; python_version >= '3.7'
certifi==2023.7.22; python_version >= '3.6'
charset-normalizer==3.3.0; python_full_version >= '3.7.0'
click==8.1.7; python_version >= '3.7'
flask==3.0.0; python_version >= '3.8'
idna==3.4; python_version >= '3.5'
itsdangerous==2.1.2; python_version >= '3.7'
jinja2==3.1.2; python_version >= '3.7'
markupsafe==2.1.3; python_version >= '3.7'
requests==2.31.0; python_version >= '3.7'
urllib3==2.0.7; python_version >= '3.7'
werkzeug==3.0.0; python_version >= '3.8'

Poetry

Poetry is another fancy tool similar to pipenv that help to simplify the development process and solve the common issues related to the dependencies resolutions. Poetry’s capabilities include building wheels packages and publishing them and also supports plugins in case you want to extend Poetry’s functionality.

Installation

Poetry can be installed using different ways:

Using Official Installer
Using pipx.
Manually using pip.

Using an installer is the simplest approach by downloading the binaries and installing them using Python

curl -sSL https://install.python-poetry.org | python3 -


Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/Users/mabuaisha/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.6.1): Installing Poetry
Installing Poetry (1.6.1): Done

Poetry (1.6.1) is installed now. Great!

You can test that everything is set up by executing:

`poetry --version`

Make sure that the Poetry command is available, run the following command

➜  ~ poetry --version
Poetry (version 1.6.1)

Poetry supports more options than the pipenv

➜  ~ poetry list
Poetry (version 1.6.1)

Usage:
  command [options] [arguments]

Options:
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.
      --ansi                 Force ANSI output.
      --no-ansi              Disable ANSI output.
  -n, --no-interaction       Do not ask any interactive question.
      --no-plugins           Disables plugins.
      --no-cache             Disables Poetry source caches.
  -C, --directory=DIRECTORY  The working directory for the Poetry command (defaults to the current working directory).
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug.

Available commands:
  about              Shows information about Poetry.
  add                Adds a new dependency to pyproject.toml.
  build              Builds a package, as a tarball and a wheel by default.
  check              Validates the content of the pyproject.toml file and its consistency with the poetry.lock file.
  config             Manages configuration settings.
  export             Exports the lock file to alternative formats.
  help               Displays help for a command.
  init               Creates a basic pyproject.toml file in the current directory.
  install            Installs the project dependencies.
  list               Lists commands.
  lock               Locks the project dependencies.
  new                Creates a new Python project at <path>.
  publish            Publishes a package to a remote repository.
  remove             Removes a package from the project dependencies.
  run                Runs a command in the appropriate environment.
  search             Searches for packages on remote repositories.
  shell              Spawns a shell within the virtual environment.
  show               Shows information about packages.
  update             Update the dependencies as according to the pyproject.toml file.
  version            Shows the version of the project or bumps it when a valid bump rule is provided.

 cache
  cache clear        Clears a Poetry cache by name.
  cache list         List Poetry's caches.

 debug
  debug info         Shows debug information.
  debug resolve      Debugs dependency resolution.

 env
  env info           Displays information about the current environment.
  env list           Lists all virtualenvs associated with the current project.
  env remove         Remove virtual environments associated with the project.
  env use            Activates or creates a new virtualenv for the current project.

 self
  self add           Add additional packages to Poetry's runtime environment.
  self install       Install locked packages (incl. addons) required by this Poetry installation.
  self lock          Lock the Poetry installation's system requirements.
  self remove        Remove additional packages from Poetry's runtime environment.
  self show          Show packages from Poetry's runtime environment.
  self show plugins  Shows information about the currently installed plugins.
  self update        Updates Poetry to the latest version.

 source
  source add         Add source configuration for project.
  source remove      Remove source configured for the project.
  source show        Show information about sources configured for the project.

Project Setup

Poetry supports a concept called project acting as a boilerplate for your Python application and creating all the necessary files for managing dependencies.

Poetry supports creating a new project or initialising from the existing directory.

➜  ~ poetry new poetry-app
Created package poetry_app in poetry-app

The above command sets a new project by creating a directory that contains all the files and directories for Poetry.

poetry-app
├── pyproject.toml
├── README.md
├── poetry_app
│   └── __init__.py
└── tests
    └── __init__.py

The pyproject.toml is the main file that is responsible for managing the project and its dependencies (Similar to the Pipfile but with extra capabilities).

[tool.poetry]
name = "poetry-app"
version = "0.1.0"
description = ""
authors = ["Mohammed AbuAisha <mabuaisha@outlook.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

By default, poetry automatically creates a package name that matches the name of the project but it normalizes the package names by transforming the dash (-) of your project name into an underscore (_) as specified in the example above (project name poetry-app , package name poetry_app ).

poetry gives the ability to change this behavior by one of the two options:

Using --name option when creating a new project
Using the srclayout for the project.

Choosing the src layout will create sub-directories under src directory

➜  ~ poetry new --src poetry-app
Created package poetry_app in poetry-app

poetry-app/
│
├── src/
│   │
│   └── poetry_app/
│       └── __init__.py
│
├── tests/
│   ├── __init__.py
│   
│
├── README.rst
└── pyproject.toml

[tool.poetry]
name = "poetry-app"
version = "0.1.0"
description = ""
authors = ["Mohammed AbuAisha <mohammed.abuaisha@hotjar.com>"]
readme = "README.md"
packages = [{include = "poetry_app", from = "src"}]

[tool.poetry.dependencies]
python = "^3.11"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The configuration for the pyproject.toml was updated, where we can see that packages now refers to the poetry_app under src .

Note: Mainly the packages configuration is important for the packages and modules to be included in the final distribution

Someone could ask, what about the pre-existing project? how to configure it using poetry ?. Simply, poetry provide an option for that using init command.

cd pre-existing-project
poetry init

poetry will interactively create and configure the pyproject.toml

Poetry Dependencies

New dependencies can be added easily from the poetry command line using add the command.

➜  poetry-app poetry add requests
Using version ^2.31.0 for requests

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 5 installs, 0 updates, 0 removals

  • Installing certifi (2023.7.22)
  • Installing charset-normalizer (3.3.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.7)
  • Installing requests (2.31.0)

Writing lock file

The above command will do the following:

Update pyproject.toml by adding the dependencies
Update the lock file poetry.lock
Create a virtual environment if it does not exist
Install all dependencies inside the created/existing virtual environment associated with the current project

[[package]]
name = "certifi"
version = "2023.7.22"
description = "Python package for providing Mozilla's CA Bundle."
optional = false
python-versions = ">=3.6"
files = [
    {file = "certifi-2023.7.22-py3-none-any.whl", hash = "sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"},
    {file = "certifi-2023.7.22.tar.gz", hash = "sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082"},
]

[[package]]
name = "charset-normalizer"
version = "3.3.0"
description = "The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet."
optional = false
python-versions = ">=3.7.0"
files = [
    {file = "charset-normalizer-3.3.0.tar.gz", hash = "sha256:63563193aec44bce707e0c5ca64ff69fa72ed7cf34ce6e11d5127555756fd2f6"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:effe5406c9bd748a871dbcaf3ac69167c38d72db8c9baf3ff954c344f31c4cbe"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4162918ef3098851fcd8a628bf9b6a98d10c380725df9e04caf5ca6dd48c847a"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0570d21da019941634a531444364f2482e8db0b3425fcd5ac0c36565a64142c8"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5707a746c6083a3a74b46b3a631d78d129edab06195a92a8ece755aac25a3f3d"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:278c296c6f96fa686d74eb449ea1697f3c03dc28b75f873b65b5201806346a69"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a4b71f4d1765639372a3b32d2638197f5cd5221b19531f9245fcc9ee62d38f56"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f5969baeaea61c97efa706b9b107dcba02784b1601c74ac84f2a532ea079403e"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a3f93dab657839dfa61025056606600a11d0b696d79386f974e459a3fbc568ec"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:db756e48f9c5c607b5e33dd36b1d5872d0422e960145b08ab0ec7fd420e9d649"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:232ac332403e37e4a03d209a3f92ed9071f7d3dbda70e2a5e9cff1c4ba9f0678"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:e5c1502d4ace69a179305abb3f0bb6141cbe4714bc9b31d427329a95acfc8bdd"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-musllinux_1_1_s390x.whl", hash = "sha256:2502dd2a736c879c0f0d3e2161e74d9907231e25d35794584b1ca5284e43f596"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:23e8565ab7ff33218530bc817922fae827420f143479b753104ab801145b1d5b"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-win32.whl", hash = "sha256:1872d01ac8c618a8da634e232f24793883d6e456a66593135aeafe3784b0848d"},
    {file = "charset_normalizer-3.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:557b21a44ceac6c6b9773bc65aa1b4cc3e248a5ad2f5b914b91579a32e22204d"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:d7eff0f27edc5afa9e405f7165f85a6d782d308f3b6b9d96016c010597958e63"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6a685067d05e46641d5d1623d7c7fdf15a357546cbb2f71b0ebde91b175ffc3e"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0d3d5b7db9ed8a2b11a774db2bbea7ba1884430a205dbd54a32d61d7c2a190fa"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2935ffc78db9645cb2086c2f8f4cfd23d9b73cc0dc80334bc30aac6f03f68f8c"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9fe359b2e3a7729010060fbca442ca225280c16e923b37db0e955ac2a2b72a05"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:380c4bde80bce25c6e4f77b19386f5ec9db230df9f2f2ac1e5ad7af2caa70459"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f0d1e3732768fecb052d90d62b220af62ead5748ac51ef61e7b32c266cac9293"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1b2919306936ac6efb3aed1fbf81039f7087ddadb3160882a57ee2ff74fd2382"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:f8888e31e3a85943743f8fc15e71536bda1c81d5aa36d014a3c0c44481d7db6e"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:82eb849f085624f6a607538ee7b83a6d8126df6d2f7d3b319cb837b289123078"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:7b8b8bf1189b3ba9b8de5c8db4d541b406611a71a955bbbd7385bbc45fcb786c"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:5adf257bd58c1b8632046bbe43ee38c04e1038e9d37de9c57a94d6bd6ce5da34"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c350354efb159b8767a6244c166f66e67506e06c8924ed74669b2c70bc8735b1"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-win32.whl", hash = "sha256:02af06682e3590ab952599fbadac535ede5d60d78848e555aa58d0c0abbde786"},
    {file = "charset_normalizer-3.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:86d1f65ac145e2c9ed71d8ffb1905e9bba3a91ae29ba55b4c46ae6fc31d7c0d4"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:3b447982ad46348c02cb90d230b75ac34e9886273df3a93eec0539308a6296d7"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:abf0d9f45ea5fb95051c8bfe43cb40cda383772f7e5023a83cc481ca2604d74e"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b09719a17a2301178fac4470d54b1680b18a5048b481cb8890e1ef820cb80455"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b3d9b48ee6e3967b7901c052b670c7dda6deb812c309439adaffdec55c6d7b78"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:edfe077ab09442d4ef3c52cb1f9dab89bff02f4524afc0acf2d46be17dc479f5"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3debd1150027933210c2fc321527c2299118aa929c2f5a0a80ab6953e3bd1908"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86f63face3a527284f7bb8a9d4f78988e3c06823f7bea2bd6f0e0e9298ca0403"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:24817cb02cbef7cd499f7c9a2735286b4782bd47a5b3516a0e84c50eab44b98e"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:c71f16da1ed8949774ef79f4a0260d28b83b3a50c6576f8f4f0288d109777989"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:9cf3126b85822c4e53aa28c7ec9869b924d6fcfb76e77a45c44b83d91afd74f9"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:b3b2316b25644b23b54a6f6401074cebcecd1244c0b8e80111c9a3f1c8e83d65"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-musllinux_1_1_s390x.whl", hash = "sha256:03680bb39035fbcffe828eae9c3f8afc0428c91d38e7d61aa992ef7a59fb120e"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:4cc152c5dd831641e995764f9f0b6589519f6f5123258ccaca8c6d34572fefa8"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-win32.whl", hash = "sha256:b8f3307af845803fb0b060ab76cf6dd3a13adc15b6b451f54281d25911eb92df"},
    {file = "charset_normalizer-3.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:8eaf82f0eccd1505cf39a45a6bd0a8cf1c70dcfc30dba338207a969d91b965c0"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:dc45229747b67ffc441b3de2f3ae5e62877a282ea828a5bdb67883c4ee4a8810"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2f4a0033ce9a76e391542c182f0d48d084855b5fcba5010f707c8e8c34663d77"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ada214c6fa40f8d800e575de6b91a40d0548139e5dc457d2ebb61470abf50186"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b1121de0e9d6e6ca08289583d7491e7fcb18a439305b34a30b20d8215922d43c"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1063da2c85b95f2d1a430f1c33b55c9c17ffaf5e612e10aeaad641c55a9e2b9d"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:70f1d09c0d7748b73290b29219e854b3207aea922f839437870d8cc2168e31cc"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:250c9eb0f4600361dd80d46112213dff2286231d92d3e52af1e5a6083d10cad9"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:750b446b2ffce1739e8578576092179160f6d26bd5e23eb1789c4d64d5af7dc7"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-musllinux_1_1_ppc64le.whl", hash = "sha256:fc52b79d83a3fe3a360902d3f5d79073a993597d48114c29485e9431092905d8"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-musllinux_1_1_s390x.whl", hash = "sha256:588245972aca710b5b68802c8cad9edaa98589b1b42ad2b53accd6910dad3545"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:e39c7eb31e3f5b1f88caff88bcff1b7f8334975b46f6ac6e9fc725d829bc35d4"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-win32.whl", hash = "sha256:abecce40dfebbfa6abf8e324e1860092eeca6f7375c8c4e655a8afb61af58f2c"},
    {file = "charset_normalizer-3.3.0-cp37-cp37m-win_amd64.whl", hash = "sha256:24a91a981f185721542a0b7c92e9054b7ab4fea0508a795846bc5b0abf8118d4"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:67b8cc9574bb518ec76dc8e705d4c39ae78bb96237cb533edac149352c1f39fe"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:ac71b2977fb90c35d41c9453116e283fac47bb9096ad917b8819ca8b943abecd"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:3ae38d325b512f63f8da31f826e6cb6c367336f95e418137286ba362925c877e"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:542da1178c1c6af8873e143910e2269add130a299c9106eef2594e15dae5e482"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:30a85aed0b864ac88309b7d94be09f6046c834ef60762a8833b660139cfbad13"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:aae32c93e0f64469f74ccc730a7cb21c7610af3a775157e50bbd38f816536b38"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:15b26ddf78d57f1d143bdf32e820fd8935d36abe8a25eb9ec0b5a71c82eb3895"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7f5d10bae5d78e4551b7be7a9b29643a95aded9d0f602aa2ba584f0388e7a557"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:249c6470a2b60935bafd1d1d13cd613f8cd8388d53461c67397ee6a0f5dce741"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:c5a74c359b2d47d26cdbbc7845e9662d6b08a1e915eb015d044729e92e7050b7"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-musllinux_1_1_ppc64le.whl", hash = "sha256:b5bcf60a228acae568e9911f410f9d9e0d43197d030ae5799e20dca8df588287"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-musllinux_1_1_s390x.whl", hash = "sha256:187d18082694a29005ba2944c882344b6748d5be69e3a89bf3cc9d878e548d5a"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:81bf654678e575403736b85ba3a7867e31c2c30a69bc57fe88e3ace52fb17b89"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-win32.whl", hash = "sha256:85a32721ddde63c9df9ebb0d2045b9691d9750cb139c161c80e500d210f5e26e"},
    {file = "charset_normalizer-3.3.0-cp38-cp38-win_amd64.whl", hash = "sha256:468d2a840567b13a590e67dd276c570f8de00ed767ecc611994c301d0f8c014f"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:e0fc42822278451bc13a2e8626cf2218ba570f27856b536e00cfa53099724828"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:09c77f964f351a7369cc343911e0df63e762e42bac24cd7d18525961c81754f4"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:12ebea541c44fdc88ccb794a13fe861cc5e35d64ed689513a5c03d05b53b7c82"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:805dfea4ca10411a5296bcc75638017215a93ffb584c9e344731eef0dcfb026a"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:96c2b49eb6a72c0e4991d62406e365d87067ca14c1a729a870d22354e6f68115"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:aaf7b34c5bc56b38c931a54f7952f1ff0ae77a2e82496583b247f7c969eb1479"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:619d1c96099be5823db34fe89e2582b336b5b074a7f47f819d6b3a57ff7bdb86"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a0ac5e7015a5920cfce654c06618ec40c33e12801711da6b4258af59a8eff00a"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:93aa7eef6ee71c629b51ef873991d6911b906d7312c6e8e99790c0f33c576f89"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:7966951325782121e67c81299a031f4c115615e68046f79b85856b86ebffc4cd"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_ppc64le.whl", hash = "sha256:02673e456dc5ab13659f85196c534dc596d4ef260e4d86e856c3b2773ce09843"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_s390x.whl", hash = "sha256:c2af80fb58f0f24b3f3adcb9148e6203fa67dd3f61c4af146ecad033024dde43"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:153e7b6e724761741e0974fc4dcd406d35ba70b92bfe3fedcb497226c93b9da7"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-win32.whl", hash = "sha256:d47ecf253780c90ee181d4d871cd655a789da937454045b17b5798da9393901a"},
    {file = "charset_normalizer-3.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:d97d85fa63f315a8bdaba2af9a6a686e0eceab77b3089af45133252618e70884"},
    {file = "charset_normalizer-3.3.0-py3-none-any.whl", hash = "sha256:e46cd37076971c1040fc8c41273a8b3e2c624ce4f2be3f5dfcb7a430c1d3acc2"},
]

[[package]]
name = "idna"
version = "3.4"
description = "Internationalized Domain Names in Applications (IDNA)"
optional = false
python-versions = ">=3.5"
files = [
    {file = "idna-3.4-py3-none-any.whl", hash = "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"},
    {file = "idna-3.4.tar.gz", hash = "sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4"},
]

[[package]]
name = "requests"
version = "2.31.0"
description = "Python HTTP for Humans."
optional = false
python-versions = ">=3.7"
files = [
    {file = "requests-2.31.0-py3-none-any.whl", hash = "sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f"},
    {file = "requests-2.31.0.tar.gz", hash = "sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1"},
]

[package.dependencies]
certifi = ">=2017.4.17"
charset-normalizer = ">=2,<4"
idna = ">=2.5,<4"
urllib3 = ">=1.21.1,<3"

[package.extras]
socks = ["PySocks (>=1.5.6,!=1.5.7)"]
use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]

[[package]]
name = "urllib3"
version = "2.0.7"
description = "HTTP library with thread-safe connection pooling, file post, and more."
optional = false
python-versions = ">=3.7"
files = [
    {file = "urllib3-2.0.7-py3-none-any.whl", hash = "sha256:fdb6d215c776278489906c2f8916e6e7d4f5a9b602ccbcfdf7f016fc8da0596e"},
    {file = "urllib3-2.0.7.tar.gz", hash = "sha256:c97dfde1f7bd43a71c8d2a58e369e9b2bf692d1334ea9f9cae55add7d0dd0f84"},
]

[package.extras]
brotli = ["brotli (>=1.0.9)", "brotlicffi (>=0.8.0)"]
secure = ["certifi", "cryptography (>=1.9)", "idna (>=2.0.0)", "pyopenssl (>=17.1.0)", "urllib3-secure-extra"]
socks = ["pysocks (>=1.5.6,!=1.5.7,<2.0)"]
zstd = ["zstandard (>=0.18.0)"]

[metadata]
lock-version = "2.0"
python-versions = "^3.11"
content-hash = "a5780ef8e06df616beb6eb67292099db49b8fe658fcbf22940e5e1af96a7c14e"

Moreover, removing any package will also remove it from pyproject.toml , poetry.lock and virtual environment

Poetry Lock

We mentioned before that poetry.lockwill be updated automatically when you add a package from the command line. However, if someone updates the pyproject.toml manually by editing the file and adding a new package under tool-poetry-dependencies the following things must be taken into consideration:

Avoid using poetry install in this case because poetry.lock will be out of sync with the pyproject.toml
Make sure to run poetry lock in order to resolve all the dependencies and make poetry.lock compatible with pyproject.toml

For example, if we add flask == "^2.2.3" to the pyproject.toml and run poetry install you will get a warning

➜  poetry-app poetry install
Installing dependencies from lock file
Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.

Because poetry-app depends on flask (^2.3.3) which doesn't match any versions, version solving failed.

In order to solve the above, you must run poetry lock

➜  poetry-app poetry lock
Updating dependencies
Resolving dependencies... (2.5s)

Writing lock file

So after locking the dependencies, you can easily install them without any problem. Moreover, --no-update is another option that can be used with poetry lock in case you do not want to update the dependencies that are already in the poetry.lock file

➜  poetry-app poetry install
Installing dependencies from lock file

Package operations: 7 installs, 1 update, 0 removals

  • Installing markupsafe (2.1.3)
  • Installing blinker (1.6.3)
  • Updating charset-normalizer (3.3.0 -> 3.3.1)
  • Installing click (8.1.7)
  • Installing itsdangerous (2.1.2)
  • Installing jinja2 (3.1.2)
  • Installing werkzeug (3.0.1)
  • Installing flask (2.3.3)

Installing the current project: poetry-app (0.1.0)

Poetry Update

Packages also can be updated using poetry update <PACKAGE_1> <PACKAGE_2> ... This command will update the specified packages by updating both pyproject.toml + poetry.lock

➜  poetry-app poetry update flask
Updating dependencies
Resolving dependencies... (0.3s)

Poetry Requirements

Someone could argue, what about if I want to generate or use requirements.txt ?. Poetry supports both reading and writing to requirements.txt

// This will read from poetry.lock and dump to requirements.txt
➜  poetry-app poetry export -o requirements.txt

The other option is that if you already have a project that uses requirements.txt , you can still add these packages using poetry add

For example, if you have the following requirements.txt and you want to add it to the poetry ecosystem, then this can be done easily

ipython==8.12.2
sphinx==4.1.2
diagrams==0.23.3

➜  poetry-app poetry add `cat requirements.txt`

Updating dependencies
Resolving dependencies... (2.5s)

Package operations: 19 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing asttokens (2.4.1)
  • Installing executing (2.0.0)
  • Installing parso (0.8.3)
  • Installing ptyprocess (0.7.0)
  • Installing pure-eval (0.2.2)
  • Installing traitlets (5.12.0)
  • Installing wcwidth (0.2.8)
  • Installing appnope (0.1.3)
  • Installing backcall (0.2.0)
  • Installing decorator (5.1.1)
  • Installing jedi (0.19.1)
  • Installing matplotlib-inline (0.1.6)
  • Installing pexpect (4.8.0)
  • Installing pickleshare (0.7.5)
  • Installing prompt-toolkit (3.0.39)
  • Installing pygments (2.16.1)
  • Installing stack-data (0.6.3)
  • Installing ipython (8.12.2)

Writing lock file

Poetry Shell

Poetry supports activating the virtual environment and provides the ability to run any command from the shell directly without the need to specify the binaries.

➜  poetry-app poetry shell
Spawning shell within /Users/mabuaisha/Library/Caches/pypoetry/virtualenvs/poetry-app-7KqqTxz1-py3.11
➜  poetry-app emulate bash -c '. /Users/mabuaisha/Library/Caches/pypoetry/virtualenvs/poetry-app-7KqqTxz1-py3.11/bin/activate
'
(poetry-app-py3.11) ➜  poetry-app

Exiting from this shell can also be done using exit command

Poetry Config

Poetry also supports listing some configuration values like cache-dir and other settings.

➜  poetry-app poetry config --list
cache-dir = "/Users/mabuaisha/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/mabuaisha/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
➜  poetry-app

Poetry Env

You can get information about virtual environments and manage them using env commands support by poetry.

➜  poetry-app poetry env

The command "env" does not exist.

Did you mean one of these?
    env use
    env info
    env list
    env remove

➜  poetry-app poetry env  info

Virtualenv
Python:         3.11.2
Implementation: CPython
Path:           /Users/mabuaisha/Library/Caches/pypoetry/virtualenvs/poetry-app-7KqqTxz1-py3.11
Executable:     /Users/mabuaisha/Library/Caches/pypoetry/virtualenvs/poetry-app-7KqqTxz1-py3.11/bin/python
Valid:          True

System
Platform:   darwin
OS:         posix
Python:     3.11.2
Path:       /Users/mabuaisha/.pyenv/versions/3.11.2
Executable: /Users/mabuaisha/.pyenv/versions/3.11.2/bin/python3.11

➜  poetry-app poetry env list
poetry-app-7KqqTxz1-py3.11 (Activated)

Summary

In this blog, We discussed different tools for managing Python dependencies and versions. Choosing between these tools depends on your needs. However, I would recommend that when developing any Python application consider having a deterministic build and dependencies to avoid any unexpected behavior and these can be achieved using either pip-compile/pipenv/poetry .

Choosing between these tools depends on your needs and preferences but I found that pip-compile could be enough to solve this major issue.

Check the next blog if you want to know more about how to manage different Python versions and running Python-based applications.

Stackademic

Thank you for reading until the end. Before you go:

Please consider clapping and following the writer! 👏
Follow us on Twitter(X), LinkedIn, and YouTube.
Visit Stackademic.com to find out more about how we are democratizing free programming education around the world.

Advanced Python Dependencies Management

Pip Tools

pip-compile

pip-sync

Pipenv

Poetry

Summary

Stackademic

Written by Mohammed AbuAisha