~ 8 min read

October 2023 - Packaging a Python Program for PyPI: Tools, Tips + GitLab CI Config

Written by Brie Carranza

A collection of things I learned while publishing a Python package to PyPI in 2023.

TL;DR | The tools that I recommend are bump2version and cookiecutter (with the cookiecutter-pylibrary template in particular) and the tips are “use Test PyPI” and ”🤷‍♀️ maybe there’s no need to sign your PyPI package”.

🌊 Hello, world!

Last month, I announced the 🚀 release of v1.0 of pastebin-bisque, a Python program I’ve been working on since 2020. The v1.0 release included publishing pastebin-bisque on PyPI. I wanted to take a moment to write about some of the most helpful tools and notes from my experience.

This is intended to be a quick blog post but here’s an even quicker outline:

  • 🥠 The power of cookiecutter
    • AKA how to not write annoying or boring code and skip straight to the stuff you wanted to write in the first place
  • 🚀 Umm, bump2version is awesome
    • A super useful tool for managing the process of releasing a new version of a package
  • 🧪 Use Test PyPI (test.pypi.org)
    • To make sure your package looks right before going live
  • 🖋 On package signing
    • …and why I declined
  • 💚 Plus: the full .gitlab-ci.yml that I use to
    • Run tests against Python 3.8-3.11 with tox
    • Build the package
    • Upload the package to GitLab Container Registry, Test PyPI and PyPI
      • Authenticate to (Test) PyPI with a token
    • Install and run the package that is published to the registries above

🖼 The Big Picture

A few words on the big picture, for context:

  • In early 2020, I started pastebin-bisque with this commit on February 26.
    • To use it, you did pip install -r requirements.txt and then python main.py after cloning the git repo.
  • In 2022, pastebin-bisque started getting some attention and I wanted to make the program easier to use and easier to develop and contribute to.
    • Plus, I always wanted to make a proper Python package on PyPI. (I didn’t want to do it just to do it; I wanted to have a good reason and I finally had one.)
  • In September 2023, I refactored the code so that it could be installed via pip install pastebin-bisque and executed with pastebin-bisque.

This post is a quick(ish) list of the stuff I learned and the tools that were most helpful for me during this process. I’m excluding the obvious stuff that is better explained elsewhere, like git and tox. I assume that the reader has some familiarity with writing Python.

🥠 The power of cookiecutter

I used cookiecutter to save ⏱ time. Specifically, I wanted to get to the exciting part and I didn’t want to write (boring) skeleton code.

So, cookiecutter is:

A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C projects.

After inspecting several options, the specific cookiecutter template that I used is cookiecutter-pylibrary and it took care a bunch of stuff including:

  • tox for testing pastebin-bisque against multiple versions of Python
  • Documentation with Sphinx — ready for use with ReadTheDocs
  • Support for bump2version (keep reading…)

The easiest way to demonstrate this might be to take a look at the output of tree after generating the project with cookiecutter.

cookiecutter gh:ionelmc/cookiecutter-pylibrary

The cookiecutter command prompts one to answer quite a few questions. After accepting the defaults, the directory structure looks like this:

# tree python-cutestcat 
python-cutestcat
├── AUTHORS.rst
├── CHANGELOG.rst
├── CONTRIBUTING.rst
├── LICENSE
├── MANIFEST.in
├── README.rst
├── ci
│   ├── bootstrap.py
│   ├── requirements.txt
│   └── templates
├── docs
│   ├── authors.rst
│   ├── changelog.rst
│   ├── conf.py
│   ├── contributing.rst
│   ├── index.rst
│   ├── installation.rst
│   ├── readme.rst
│   ├── reference
│   │   ├── cutestcat.rst
│   │   └── index.rst
│   ├── requirements.txt
│   ├── spelling_wordlist.txt
│   └── usage.rst
├── pyproject.toml
├── pytest.ini
├── setup.py
├── src
│   └── cutestcat
│       ├── __init__.py
│       ├── __main__.py
│       └── cli.py
├── tests
│   └── test_cutestcat.py
└── tox.ini

8 directories, 28 files

😎 Cool! The emphasis was on editing src/cutestcat/cli.py to start to add my code. Once the cookiecutter invocation was complete, I had a pastebin-bisque executable. I started bringing the imports and functions from main.py into src/pastebin_bisque/cli.py until the functionality of the original program was restored.

Effectively, this is the loop I used while actively developing:

while true ; do 
    vim src/cutestcat/cli.py ; 
    rm -f $(which cutestcat) && python setup.py install && cutestcat ; 
done

The rm -f $(which cutestcat) && python setup.py install && cutestcat loop removes the old executable, creates a new one based on my changes and runs it. I stayed in that loop until things worked the way I wanted or until I needed to take a break.

🚀 Umm, bump2version is awesome

The cookiecutter prompts asked about tbump or bump2version. I’d been interested in bump2version already anyway so I chose it and it’s really nice. I would definitely recommend further adjusting .bumpversion.cfg to customize things the way you’d like.

You’d then use one of these commands depending on what kind of version you’d like to release:

  • bump2version major
  • bump2version minor
  • bump2version patch

If you followed along in the earlier section and want to test the release process, you might find this loop helpful:

pre-commit run --all-files &&  \
    git commit -am"✨ Use dependencies from PyPi" && \
    bump2version patch && git push --tags

The .bumpversion.cfg file contains the current_version and has information about where to replace the current version with the new version in your project. You’ll see setup.py and docs/conf.py and files like that in .bumpversion.cfg.

🚀 Releasing a new major version becomes:

  1. bump2version major && git push --tags
  2. Go watch the pipeline!
  3. ▶️ Press Play to publish the package

The ”🚀 deploy to 🎊 PyPi” job that publishes the package to PyPI is a manual job so it requires me to push Play when the earlier steps are complete.

🧪 Use TestPyPI (test.pypi.org))

Since this was the first Python package I’d published, this was the first time I had use for something like TestPyPI. Given the permanence of releases on PyPI, it’s very nice to have a testing environment that mimics the real thing pretty well. You can totally see my tests.

The docs on 📚 Using TestPyPI are pretty great but I’ll say a little more.

✍️ On PyPI and TestPyPI

You can use our friend twine to upload to TestPyPI as I do in the 🚀 deploy to 🧪 TestPyPi CI job (after python -m build succeeds):

TWINE_USERNAME=__token__ TWINE_REPOSITORY_URL=https://test.pypi.org/legacy/ \
    twine upload --non-interactive --comment "🦄 hello world" \
    --skip-existing --password $TESTPYPI_TOKEN  dist/*
  • TWINE_USERNAME: You’ll want to set TWINE_USERNAME to __token__ if you are using an API token to authenticate.
  • TESTPYPI_TOKEN: You’ll also need to set TESTPYPI_TOKEN to the value of the API token.

A note about using TestPyPI: do not expect that every package on PyPI is also on TestPyPI. In my GitLab CI config, I do this to install my package from TestPyPI while getting the dependencies from PyPI:

pip install -i https://test.pypi.org/simple/ pastebin-bisque \
    --extra-index-url https://pypi.org/simple

Another tip: don’t trust the namespaces to have the same ownership between PyPI and TestPyPI. In other words: the person who authored the package at test.pypi.org/project/whatever may not be the person who authored the package at pypi.org/project/whatever.

This doesn’t mean much but as of this writing:

  • 🧪 TestPyPI:
    • 180,202 projects
    • 964,359 releases
    • 1,993,081 files
    • 171,379 users
  • 🚀 PyPI:
    • 487,007 projects
    • 4,962,866 releases
    • 9,315,735 files
    • 748,166 users

…it does mean that there are a few hundred thousand projects that are not on TestPyPI. What’s the angle? If I wanted to target Python developers, I would look for projects that are not on Test PyPI. You want something that is used often enough but isn’t super high profile. A high profile package is likely to already be on TestPyPI or for its sudden appearance on TestPyPI to be noticeable. This is a more subtle, less scattershot approach to the kinds of things written up in articles like 10 malicious PyPI packages found stealing developer’s credentials. 🤷‍♀️ It probably would not be worth it but one might get “lucky”. 😉

🖋 On package signing and why I declined

When I looked into signing the package I was building, I ultimately decided against proceeding. While I don’t hold all of the views in the linked articles, this summarizes my thinking and is followed by a more extensive reading list:

TL;DR: A large number of PGP signatures on PyPI can’t be correlated to any well-known PGP key and, of the signatures that can be correlated, many are generated from weak keys or malformed certificates.

source: PGP signatures on PyPI: worse than useless

📚 Reading List

I wound up declining to sign the pastebin-bisque package. The time required would not be worth the benefit at this point in time.

🛸 Extra: Full .gitlab-ci.yml for building and deploying Python package to GitLab Package Registry, TestPyPI and PyPI

Want to see the .gitlab-ci.yml that I am using for building, testing and releasing the pastebin-bisque package to GitLab’s Package Registry, TestPyPI and to PyPI?

Here’s what these looked like in v1.0.0:

🐈‍⬛ What’s next?

As the 10th Hacktoberfest comes to an end, I’ll be releasing v1.0.1 of pastebin-bisque and I’m super excited about the new contributors whose work will be included in that release: 😻 thank you!