Importing python packages installed with pipx
I want to use the panflute package in some Python code, like this: import panflute. I don’t want to start a virtual environment every time I run that code, if I can avoid it. (It’s a Pandoc filter, obviously.)
I tried $ sudo apt install python3-panflute, but there is no such package. I then tried using pipx. This installs panflute, but I get a "ModuleNotFoundError: No module named ‘panflute’ " error.
Am I wrong to think that installing via pipx should make the package importable as a module? Did I miss a step?
I am using Ubuntu 23.04.
pipx vs. pip
pipx(Install and Run Python Applications in Isolated Environments) is not an equal alternative to pip(The Python Package Installer) … It uses pip and works differently too … It isolates the apps/packages it downloads … So, they are not made available to system Python outside that isolated environment … It creates a separate virtual environment for each app it installs … Its main purpose, however, is to make Python apps run-able as regular commands from the command-line in the console/terminal and not to make them available(importable) to your Python interpreter.
However, this(using the pip Python3 module):
python3 -m pip install -U panflute
run either system-wide(not advised) or inside a virtual environment(preferred) should install and update that package so it could be imported by the interpreter.
Ubuntu 23.04+
That said, you are using Ubuntu 23.04(later releases included) on which system-wide Python packages installation is prohibited(by default) via Python package managers like pip and handed over to the release package managers like APT to comply with PEP 668 – Marking Python base environments as “externally managed” … You should see an error message like this when trying to use pip(other Python package managers included) to install a system-wide(i.e. outside a virtual environment) package:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
However, the suggested package naming in:
To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
is merely a suggestion and doesn’t necessarily mean that the package has been already made available in the Ubuntu official repositories … Therefore, you should confirm its availability first with something like:
$ apt search python3-panflute
Sorting... Done
Full Text Search... Done
Unfortunately, as you can see that package is not available yet(at least from the official repositories).
Workarounds
Your best choice to install Python3 packages that are not yet available through APT(usually under the name python3-packagename), is to do that from inside a Python3 virtual environment that you create and use the method to install with pip described above.
Creating and using a virtual environment is not really that difficult be it in scripts or in the terminal … Please see for example:
- How do I install a clean version of Python in addition to my system Python? (i.e., doesn’t come with a bunch of unremovable packages)
- Using /usr/bin/env python3 to point to virtual environment in 22.04
- How to launch gnome-terminal and activate a python virtual environment?
Note: There is the other option to pass both these two option --user --break-system-packages to pip which will force the package install … This although limited to user environment, but is not considered safe and might still do what it says will do i.e. --break-system-packages so not advised either.
Am I wrong to think that installing via pipx should make the package importable as a module?
Yes, it was wrong to think so. (I know this sounds harsh and I’m not intending to be rude, I just want to give you a clear and direct answer.) The whole point of pipx is to let you install Python applications – that is, Python programs that you can run by typing a command on the command line, or in any of the other usual ways you might run a program – without allowing those packages to be imported from other code.
Honestly, I would strongly suggest you just use a virtual environment. I know you said you don’t want to, but whatever reason you think you have for avoiding virtual environments, I highly doubt it’s the best way to do whatever it is you want to do. If you’re developing a piece of software, you may find it useful to use tox to manage virtual environments for you. (BTW I’d suggest using pipx to install tox.)
Am I wrong to think that installing via pipx should make the package importable as a module? Did I miss a step?
pipx is for installing applications, not for managing development environments. It actually creates a virtual environment under the hood to support this use case, which is why it has no impact on the rest of the system. You need different tools if you’re trying to do development work.
I don’t want to start a virtual environment every time I run that code, if I can avoid it.
You can’t avoid it with Python unless the packages are available through you system package manager. There might be a PPA somewhere that includes panflute, but my guess would be probably not.
That said, handling this does not have to be difficult, there are plenty of ways you can handle this mostly transparently. The ‘classic’ approach is to just write a short shell script that runs the program in the virtual environment you need it to be run in (this is quite literally a three-line script, and the first line is just the mandatory #!/bin/sh to get the kernel to recognize it as a shell script).
Alternatively, look into poetry, which handles most of this for you without issue and also adds some significantly improved dependency management tooling on top of what you would get with pip. You can install python3-poetry in Ubuntu 22.04 and later by running the following command:
sudo apt install python3-poetry