TL;DR | The tools that I recommend are bump2version and cookiecutter (with the cookiecutter-pylibrary template in particular) and the tips are “use Test PyPI” and ”🤷♀️ maybe there’s no need to sign your PyPI package”.
🌊 Hello, world!
Last month, I announced the 🚀 release of
pastebin-bisque, a Python program I’ve been working on since 2020. The
v1.0 release included publishing
pastebin-bisque on PyPI. I wanted to take a moment to write about some of the most helpful tools and notes from my experience.
This is intended to be a quick blog post but here’s an even quicker outline:
- 🥠 The power of
- AKA how to not write annoying or boring code and skip straight to the stuff you wanted to write in the first place
- 🚀 Umm,
- A super useful tool for managing the process of releasing a new version of a package
- 🧪 Use Test PyPI (
- To make sure your package looks right before going live
- 🖋 On package signing
- …and why I declined
- 💚 Plus: the full
.gitlab-ci.ymlthat I use to
- Run tests against Python 3.8-3.11 with
- Build the package
- Upload the package to GitLab Container Registry, Test PyPI and PyPI
- Authenticate to (Test) PyPI with a token
- Install and run the package that is published to the registries above
- Run tests against Python 3.8-3.11 with
🖼 The Big Picture
A few words on the big picture, for context:
- In early 2020, I started pastebin-bisque with this commit on February 26.
- To use it, you did
pip install -r requirements.txtand then
python main.pyafter cloning the
- To use it, you did
- In 2022,
pastebin-bisquestarted getting some attention and I wanted to make the program easier to use and easier to develop and contribute to.
- Plus, I always wanted to make a proper Python package on PyPI. (I didn’t want to do it just to do it; I wanted to have a good reason and I finally had one.)
- In September 2023, I refactored the code so that it could be installed via
pip install pastebin-bisqueand executed with
This post is a quick(ish) list of the stuff I learned and the tools that were most helpful for me during this process. I’m excluding the obvious stuff that is better explained elsewhere, like
tox. I assume that the reader has some familiarity with writing Python.
🥠 The power of
cookiecutter to save ⏱ time. Specifically, I wanted to get to the exciting part and I didn’t want to write (boring) skeleton code.
A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C projects.
pastebin-bisqueagainst multiple versions of Python
- Documentation with Sphinx — ready for use with ReadTheDocs
- Support for
The easiest way to demonstrate this might be to take a look at the output of
tree after generating the project with
cookiecutter command prompts one to answer quite a few questions. After accepting the defaults, the directory structure looks like this:
# tree python-cutestcat python-cutestcat ├── AUTHORS.rst ├── CHANGELOG.rst ├── CONTRIBUTING.rst ├── LICENSE ├── MANIFEST.in ├── README.rst ├── ci │ ├── bootstrap.py │ ├── requirements.txt │ └── templates ├── docs │ ├── authors.rst │ ├── changelog.rst │ ├── conf.py │ ├── contributing.rst │ ├── index.rst │ ├── installation.rst │ ├── readme.rst │ ├── reference │ │ ├── cutestcat.rst │ │ └── index.rst │ ├── requirements.txt │ ├── spelling_wordlist.txt │ └── usage.rst ├── pyproject.toml ├── pytest.ini ├── setup.py ├── src │ └── cutestcat │ ├── __init__.py │ ├── __main__.py │ └── cli.py ├── tests │ └── test_cutestcat.py └── tox.ini 8 directories, 28 files
😎 Cool! The emphasis was on editing
src/cutestcat/cli.py to start to add my code. Once the
cookiecutter invocation was complete, I had a
pastebin-bisque executable. I started bringing the imports and functions from
src/pastebin_bisque/cli.py until the functionality of the original program was restored.
Effectively, this is the loop I used while actively developing:
while true ; do vim src/cutestcat/cli.py ; rm -f $(which cutestcat) && python setup.py install && cutestcat ; done
rm -f $(which cutestcat) && python setup.py install && cutestcat loop removes the old executable, creates a new one based on my changes and runs it. I stayed in that loop until things worked the way I wanted or until I needed to take a break.
bump2version is awesome
cookiecutter prompts asked about
bump2version. I’d been interested in
bump2version already anyway so I chose it and it’s really nice. I would definitely recommend further adjusting
.bumpversion.cfg to customize things the way you’d like.
You’d then use one of these commands depending on what kind of version you’d like to release:
If you followed along in the earlier section and want to test the release process, you might find this loop helpful:
pre-commit run --all-files && \ git commit -am"✨ Use dependencies from PyPi" && \ bump2version patch && git push --tags
.bumpversion.cfg file contains the
current_version and has information about where to replace the current version with the new version in your project. You’ll see
docs/conf.py and files like that in
🚀 Releasing a new major version becomes:
bump2version major && git push --tags
- Go watch the pipeline!
- ▶️ Press Play to publish the package
🧪 Use TestPyPI (
Since this was the first Python package I’d published, this was the first time I had use for something like TestPyPI. Given the permanence of releases on PyPI, it’s very nice to have a testing environment that mimics the real thing pretty well. You can totally see my tests.
The docs on 📚 Using TestPyPI are pretty great but I’ll say a little more.
✍️ On PyPI and TestPyPI
TWINE_USERNAME=__token__ TWINE_REPOSITORY_URL=https://test.pypi.org/legacy/ \ twine upload --non-interactive --comment "🦄 hello world" \ --skip-existing --password $TESTPYPI_TOKEN dist/*
TWINE_USERNAME: You’ll want to set
__token__if you are using an API token to authenticate.
TESTPYPI_TOKEN: You’ll also need to set
TESTPYPI_TOKENto the value of the API token.
A note about using TestPyPI: do not expect that every package on PyPI is also on TestPyPI. In my GitLab CI config, I do this to install my package from TestPyPI while getting the dependencies from PyPI:
pip install -i https://test.pypi.org/simple/ pastebin-bisque \ --extra-index-url https://pypi.org/simple
Another tip: don’t trust the namespaces to have the same ownership between PyPI and TestPyPI. In other words: the person who authored the package at
test.pypi.org/project/whatever may not be the person who authored the package at
This doesn’t mean much but as of this writing:
- 🧪 TestPyPI:
- 180,202 projects
- 964,359 releases
- 1,993,081 files
- 171,379 users
- 🚀 PyPI:
- 487,007 projects
- 4,962,866 releases
- 9,315,735 files
- 748,166 users
…it does mean that there are a few hundred thousand projects that are not on TestPyPI. What’s the angle? If I wanted to target Python developers, I would look for projects that are not on Test PyPI. You want something that is used often enough but isn’t super high profile. A high profile package is likely to already be on TestPyPI or for its sudden appearance on TestPyPI to be noticeable. This is a more subtle, less scattershot approach to the kinds of things written up in articles like 10 malicious PyPI packages found stealing developer’s credentials. 🤷♀️ It probably would not be worth it but one might get “lucky”. 😉
🖋 On package signing and why I declined
When I looked into signing the package I was building, I ultimately decided against proceeding. While I don’t hold all of the views in the linked articles, this summarizes my thinking and is followed by a more extensive reading list:
TL;DR: A large number of PGP signatures on PyPI can’t be correlated to any well-known PGP key and, of the signatures that can be correlated, many are generated from weak keys or malformed certificates.
📚 Reading List
- PyPI and gpg signed packages
- Package signing in PIP - It works, in a roundabout sort of way
- PGP signatures on PyPI: worse than useless
- Source: woodruffw/pypi-pgp-statistics
- GPG signing - how does that really work with PyPI?
- Remove GPG references from publishing tutorial
I wound up declining to sign the
pastebin-bisque package. The time required would not be worth the benefit at this point in time.
🛸 Extra: Full
.gitlab-ci.yml for building and deploying Python package to GitLab Package Registry, TestPyPI and PyPI
Want to see the
.gitlab-ci.yml that I am using for building, testing and releasing the
pastebin-bisque package to GitLab’s Package Registry, TestPyPI and to PyPI?
Here’s what these looked like in
- 🦊 See the .gitlab-ci.yml for
- 📊 See the .gitlab-ci.yml for
🐈⬛ What’s next?
As the 10th Hacktoberfest comes to an end, I’ll be releasing
pastebin-bisque and I’m super excited about the new contributors whose work will be included in that release: 😻 thank you!