~ 8 min read

October 2023 - Packaging a Python Program for PyPI: Tools, Tips + GitLab CI Config

Written by Brie Carranza

A collection of things I learned while publishing a Python package to PyPI in 2023.

TL;DR | The tools that I recommend are bump2version and cookiecutter (with the cookiecutter-pylibrary template in particular) and the tips are β€œuse Test PyPI” and β€πŸ€·β€β™€οΈ maybe there’s no need to sign your PyPI package”.

🌊 Hello, world!

Last month, I announced the πŸš€ release of v1.0 of pastebin-bisque, a Python program I’ve been working on since 2020. The v1.0 release included publishing pastebin-bisque on PyPI. I wanted to take a moment to write about some of the most helpful tools and notes from my experience.

This is intended to be a quick blog post but here’s an even quicker outline:

  • πŸ₯  The power of cookiecutter
    • AKA how to not write annoying or boring code and skip straight to the stuff you wanted to write in the first place
  • πŸš€ Umm, bump2version is awesome
    • A super useful tool for managing the process of releasing a new version of a package
  • πŸ§ͺ Use Test PyPI (test.pypi.org)
    • To make sure your package looks right before going live
  • πŸ–‹ On package signing
    • …and why I declined
  • πŸ’š Plus: the full .gitlab-ci.yml that I use to
    • Run tests against Python 3.8-3.11 with tox
    • Build the package
    • Upload the package to GitLab Container Registry, Test PyPI and PyPI
      • Authenticate to (Test) PyPI with a token
    • Install and run the package that is published to the registries above

πŸ–Ό The Big Picture

A few words on the big picture, for context:

  • In early 2020, I started pastebin-bisque with this commit on February 26.
    • To use it, you did pip install -r requirements.txt and then python main.py after cloning the git repo.
  • In 2022, pastebin-bisque started getting some attention and I wanted to make the program easier to use and easier to develop and contribute to.
    • Plus, I always wanted to make a proper Python package on PyPI. (I didn’t want to do it just to do it; I wanted to have a good reason and I finally had one.)
  • In September 2023, I refactored the code so that it could be installed via pip install pastebin-bisque and executed with pastebin-bisque.

This post is a quick(ish) list of the stuff I learned and the tools that were most helpful for me during this process. I’m excluding the obvious stuff that is better explained elsewhere, like git and tox. I assume that the reader has some familiarity with writing Python.

πŸ₯  The power of cookiecutter

I used cookiecutter to save ⏱ time. Specifically, I wanted to get to the exciting part and I didn’t want to write (boring) skeleton code.

So, cookiecutter is:

A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C projects.

After inspecting several options, the specific cookiecutter template that I used is cookiecutter-pylibrary and it took care a bunch of stuff including:

  • tox for testing pastebin-bisque against multiple versions of Python
  • Documentation with Sphinx β€” ready for use with ReadTheDocs
  • Support for bump2version (keep reading…)

The easiest way to demonstrate this might be to take a look at the output of tree after generating the project with cookiecutter.

cookiecutter gh:ionelmc/cookiecutter-pylibrary

The cookiecutter command prompts one to answer quite a few questions. After accepting the defaults, the directory structure looks like this:

# tree python-cutestcat 
β”œβ”€β”€ AUTHORS.rst
β”œβ”€β”€ CHANGELOG.rst
β”œβ”€β”€ CONTRIBUTING.rst
β”œβ”€β”€ LICENSE
β”œβ”€β”€ MANIFEST.in
β”œβ”€β”€ README.rst
β”œβ”€β”€ ci
β”‚   β”œβ”€β”€ bootstrap.py
β”‚   β”œβ”€β”€ requirements.txt
β”‚   └── templates
β”œβ”€β”€ docs
β”‚   β”œβ”€β”€ authors.rst
β”‚   β”œβ”€β”€ changelog.rst
β”‚   β”œβ”€β”€ conf.py
β”‚   β”œβ”€β”€ contributing.rst
β”‚   β”œβ”€β”€ index.rst
β”‚   β”œβ”€β”€ installation.rst
β”‚   β”œβ”€β”€ readme.rst
β”‚   β”œβ”€β”€ reference
β”‚   β”‚   β”œβ”€β”€ cutestcat.rst
β”‚   β”‚   └── index.rst
β”‚   β”œβ”€β”€ requirements.txt
β”‚   β”œβ”€β”€ spelling_wordlist.txt
β”‚   └── usage.rst
β”œβ”€β”€ pyproject.toml
β”œβ”€β”€ pytest.ini
β”œβ”€β”€ setup.py
β”œβ”€β”€ src
β”‚   └── cutestcat
β”‚       β”œβ”€β”€ __init__.py
β”‚       β”œβ”€β”€ __main__.py
β”‚       └── cli.py
β”œβ”€β”€ tests
β”‚   └── test_cutestcat.py
└── tox.ini

8 directories, 28 files

😎 Cool! The emphasis was on editing src/cutestcat/cli.py to start to add my code. Once the cookiecutter invocation was complete, I had a pastebin-bisque executable. I started bringing the imports and functions from main.py into src/pastebin_bisque/cli.py until the functionality of the original program was restored.

Effectively, this is the loop I used while actively developing:

while true ; do 
    vim src/cutestcat/cli.py ; 
    rm -f $(which cutestcat) && python setup.py install && cutestcat ; 

The rm -f $(which cutestcat) && python setup.py install && cutestcat loop removes the old executable, creates a new one based on my changes and runs it. I stayed in that loop until things worked the way I wanted or until I needed to take a break.

πŸš€ Umm, bump2version is awesome

The cookiecutter prompts asked about tbump or bump2version. I’d been interested in bump2version already anyway so I chose it and it’s really nice. I would definitely recommend further adjusting .bumpversion.cfg to customize things the way you’d like.

You’d then use one of these commands depending on what kind of version you’d like to release:

  • bump2version major
  • bump2version minor
  • bump2version patch

If you followed along in the earlier section and want to test the release process, you might find this loop helpful:

pre-commit run --all-files &&  \
    git commit -am"✨ Use dependencies from PyPi" && \
    bump2version patch && git push --tags

The .bumpversion.cfg file contains the current_version and has information about where to replace the current version with the new version in your project. You’ll see setup.py and docs/conf.py and files like that in .bumpversion.cfg.

πŸš€ Releasing a new major version becomes:

  1. bump2version major && git push --tags
  2. Go watch the pipeline!
  3. ▢️ Press Play to publish the package

The β€πŸš€ deploy to 🎊 PyPi” job that publishes the package to PyPI is a manual job so it requires me to push Play when the earlier steps are complete.

πŸ§ͺ Use TestPyPI (test.pypi.org))

Since this was the first Python package I’d published, this was the first time I had use for something like TestPyPI. Given the permanence of releases on PyPI, it’s very nice to have a testing environment that mimics the real thing pretty well. You can totally see my tests.

The docs on πŸ“š Using TestPyPI are pretty great but I’ll say a little more.

✍️ On PyPI and TestPyPI

You can use our friend twine to upload to TestPyPI as I do in the πŸš€ deploy to πŸ§ͺ TestPyPi CI job (after python -m build succeeds):

TWINE_USERNAME=__token__ TWINE_REPOSITORY_URL=https://test.pypi.org/legacy/ \
    twine upload --non-interactive --comment "πŸ¦„ hello world" \
    --skip-existing --password $TESTPYPI_TOKEN  dist/*
  • TWINE_USERNAME: You’ll want to set TWINE_USERNAME to __token__ if you are using an API token to authenticate.
  • TESTPYPI_TOKEN: You’ll also need to set TESTPYPI_TOKEN to the value of the API token.

A note about using TestPyPI: do not expect that every package on PyPI is also on TestPyPI. In my GitLab CI config, I do this to install my package from TestPyPI while getting the dependencies from PyPI:

pip install -i https://test.pypi.org/simple/ pastebin-bisque \
    --extra-index-url https://pypi.org/simple

Another tip: don’t trust the namespaces to have the same ownership between PyPI and TestPyPI. In other words: the person who authored the package at test.pypi.org/project/whatever may not be the person who authored the package at pypi.org/project/whatever.

This doesn’t mean much but as of this writing:

  • πŸ§ͺ TestPyPI:
    • 180,202 projects
    • 964,359 releases
    • 1,993,081 files
    • 171,379 users
  • πŸš€ PyPI:
    • 487,007 projects
    • 4,962,866 releases
    • 9,315,735 files
    • 748,166 users

…it does mean that there are a few hundred thousand projects that are not on TestPyPI. What’s the angle? If I wanted to target Python developers, I would look for projects that are not on Test PyPI. You want something that is used often enough but isn’t super high profile. A high profile package is likely to already be on TestPyPI or for its sudden appearance on TestPyPI to be noticeable. This is a more subtle, less scattershot approach to the kinds of things written up in articles like 10 malicious PyPI packages found stealing developer’s credentials. πŸ€·β€β™€οΈ It probably would not be worth it but one might get β€œlucky”. πŸ˜‰

πŸ–‹ On package signing and why I declined

When I looked into signing the package I was building, I ultimately decided against proceeding. While I don’t hold all of the views in the linked articles, this summarizes my thinking and is followed by a more extensive reading list:

TL;DR: A large number of PGP signatures on PyPI can’t be correlated to any well-known PGP key and, of the signatures that can be correlated, many are generated from weak keys or malformed certificates.

source: PGP signatures on PyPI: worse than useless

πŸ“š Reading List

I wound up declining to sign the pastebin-bisque package. The time required would not be worth the benefit at this point in time.

πŸ›Έ Extra: Full .gitlab-ci.yml for building and deploying Python package to GitLab Package Registry, TestPyPI and PyPI

Want to see the .gitlab-ci.yml that I am using for building, testing and releasing the pastebin-bisque package to GitLab’s Package Registry, TestPyPI and to PyPI?

Here’s what these looked like in v1.0.0:

  • 🦊 See the .gitlab-ci.yml for v1 of pastebin-bisque on GitLab.
  • πŸ“Š See the .gitlab-ci.yml for v1 of pastebin-bisque on Sourcegraph.

πŸˆβ€β¬› What’s next?

As the 10th Hacktoberfest comes to an end, I’ll be releasing v1.0.1 of pastebin-bisque and I’m super excited about the new contributors whose work will be included in that release: 😻 thank you!