Publishing Python Packages: Test, share, and automate your projects

by Dane Hillard

Overview

Create masterful, maintainable Python packages! This book includes pro tips for design, automation, testing, deployment, and even release as an open source project!

In Publishing Python Packages you will learn how to:

Build extensions and console script commands
Use tox to automate packaging, installing, and testing
Build a continuous integration pipeline using GitHub Actions
Improve code quality and reduce manual review using black, mypy, and flake8
Create published documentation for your packages
Keep packages up to date with pyupgrade and Dependabot
Foster an open source community using GitHub features

Publishing Python Packages teaches you how to easily share your Python code with your team and the outside world. Learn a repeatable and highly automated process for package maintenance that’s based on the best practices, tools, and standards of Python packaging. This book walks you through creating a complete package, including a C extension, and guides you all the way to publishing on the Python Package Index. Whether you’re entirely new to Python packaging or looking for optimal ways to maintain and scale your packages, this fast-paced and engaging guide is for you.

Foreword by David Beazley.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Successful Python packages install easily, run flawlessly, and stay reliably up to date. Publishing perfect Python packages requires a rigorous process that supports systematic testing and review, along with excellent documentation. Fortunately, the Python ecosystem includes tools and techniques to automate package creation and publishing.

About the book
Publishing Python Packages presents a practical process for sharing Python code in an automated and scalable way. Get hands-on experience with the latest packaging tools, and learn the ins and outs of package testing and continuous integration. You’ll even get pro tips for setting up a maintainable open source project, including licensing, documentation, and nurturing a community of contributors.

What's inside

Build extensions and console script commands
Improve code quality with automated review and testing
Create excellent documentation
Keep packages up to date with pyupgrade and Dependabot

About the reader
For intermediate Python programmers.

About the author
Dane Hillard has spent the majority of his development career using Python to build web applications.

Table of Contents
PART 1 FOUNDATIONS
1 The what and why of Python packages
2 Preparing for package development
3 The anatomy of a minimal Python package
PART 2 CREATING A VIABLE PACKAGE
4 Handling package dependencies, entry points, and extensions
5 Building and maintaining a test suite
6 Automating code quality tooling
PART 3 GOING PUBLIC
7 Automating work through continuous integration
8 Authoring and maintaining documentation
9 Making a package evergreen
10 Scaling and solidifying your practices
11 Building a community

Product Details

ISBN-13:	9781617299919
Publisher:	Manning
Publication date:	01/10/2023
Pages:	248
Product dimensions:	7.38(w) x 9.25(h) x 0.60(d)

About the Author

Dane Hillard has spent the majority of his development career using Python to build web applications.

Table of Contents

Foreword xi

Preface xiii

Acknowledgments xv

About this book xvii

About the author xxi

About the cover illustration xxii

Part 1 Foundations 1

1 The what and why of Python packages 3

1.1 What is a package, anyway? 4

Standardizing packaging for automation 5

The contents of a distribution package 6

The challenges of sharing software 7

1.2 How packaging helps you 7

Enforcing cohesion and encapsulation through packaging 8

Promoting clear ownership of code 9

Decoupling implementation from usage 10

Filling roles by composing small packages 12

2 Preparing for package development 15

2.1 Managing Python virtual environments 16

Creating virtual environments with venv 18

3 The anatomy of a minimal Python package 21

3.1 The Python build workflow 22

Parts of the Python build system 22

3.2 Authoring package metadata 26

Required core metadata 26

Optional core metadata 28

Specifying a license 31

3.3 Controlling source code and file discovery 33

3.4 Including non-Python files in a package 35

Part 2 Creating a viable package 39

4 Handling package dependencies, entry points, and extensions 41

4.1 A package for calculating vehicle drift 41

4.2 Creating a C extension for Python 44

Creating C extension source code 44

Integrating Cython into a Python package build 45

Installing and profiling your C extension 47

Build targets for binary wheel distributions 48

Specifying required Python versions 50

4.3 Offering command-line tools from a Python package 50

Creating commands with Setuptools entry points 50

4.4 Specifying dependencies for Python packages 53

4.5 Answer to exercises 54

5 Building and maintaining a test suite 56

5.1 Integrating a testing setup 57

The pytest testing framework 57

Adding test coverage measurement 59

Increasing test coverage 65

5.2 Addressing testing tedium 69

Addressing repetitive, data-driven tests 69

Addressing frequent package installation 70

Configuring test environments 74

Tips for quicker and safer testing 76

5.3 Answers to exercises 80

6 Automating code quality tooling 82

6.1 The true power of tox environments 83

Creating nondefault tox environments 83

Managing dependencies across tox environments 85

6.2 Analyzing type safety 88

Creating a tox environm.ent for type checking 90

Configuring mypy 91

6.3 Creating a tox environment for code formatting 93

Configuring Hack 95

6.4 Creating a tox environment for linting 96

Configuring flake8 97

6.5 Answers to exercises 99

Part 3 Going Public 101

7 Automating work through continuous integration 103

7.1 The continuous integration workflow 104

7.2 Continuous integration with GitHub Actions 105

A high-level GitHub Actions workflow 106

Understanding GitHub Actions terminology 106

Starting a GitHub Actions workflow configuration 109

7.3 Converting manual tasks to GitHub Actions 112

Running a job multiple times with a build matrix 114

Building Python package distributions for a variety of platforms 117

7.4 Publishing a package 119

8 Authoring and maintaining documentation 128

8.1 Some quick philosophy on documentation 129

8.2 Starting your documentation with Sphinx 130

Automating documentation refresh during development 135

Automating extraction of code documentation 135

8.3 Publishing documentation to Read the Docs 143

Configuring Read the Docs 149

8.4 Documentation best practices 153

What to document 154

Prefer linking over repetition 154

Use consistent, empathetic language 155

Avoid assumptions and create context 156

Create visual interest and coherent structure 156

Powering up your documentation 156

9 Making a package evergreen 158

9.1 Choosing a package-versioning strategy 159

Direct and indirect dependencies 159

Python dependency specifiers and dependency hell 162

Semantic versioning and calendar versioning 164

9.2 Getting the most out of GitHub 166

The GitHub de-pendency graph 166

Mitigating security vulnerabilities with Dependabot 168

9.3 Thresholding test coverage 172

9.4 Updating Python syntax using pyupgrade 173

9.5 Reducing rework using pre-commit hooks 174

9.6 Answers to exercises 176

Part 4 The long haul 177

10 Scaling and solidifying your practices 179

10.1 Creating a project template for future packages 180

Creating a cookiecutter configuration 180

Extracting a cookiecutter template from an existing project 185

10.2 Using namespace packages 188

Converting an existing package to a namespace package 191

10.3 Scaling packaging in your organization 191

Private package repository servers 192

11 Building a community 195

11.1 Your README needs to make a value proposition 196

11.2 Provide supporting documentation for different user types 197

11.3 Establish, provide, and enforce a code of conduct 199

11.4 Conveying the project's road map, status, and changes 201

Using GitHub projects for kanban management 201

Use GitHub labels to track status for individual tasks 201

Track high-level changes in a log 203

11.5 Gather consistent information with issue templates 205

11.6 Go forth 207

Appendix A Installing asdf and python-launcher 209

Appendix B Installing pipx, build, tox, pre-commit, and cookiecutter 215

Index 219

Editorial Reviews

'Takes you through the understanding of what packaging is into all the details of how it works and demonstrates each step with a clear example. I learned a lot.' Mike Baran

'Narrative is succinct and convincing. The ideas and principles, covered in the book come from the author's deep expertise.' Kevin Etienne

'The type of gem you would expect to see many books about.' Jose Apablaza

'This book definitely occupies an unfilled need.' Eric Chiang

From the Publisher