The Book: Second Edition

Preface

Effective Python Book Cover

The Python programming language has unique strengths and charms that can be hard to grasp. Many programmers familiar with other languages often approach Python from a limited mindset instead of embracing its full expressivity. Some programmers go too far in the other direction, overusing Python features that can cause big problems later.

This second edition book (to be published by Pearson Addison-Wesley in mid-November 2019) provides insight into the Pythonic way of writing programs: the best way to use Python. It builds on a fundamental understanding of the language that I assume you already have. Novice programmers will learn the best practices of Python’s capabilities. Experienced programmers will learn how to embrace the strangeness of a new tool with confidence.

My goal is to prepare you to make a big impact with Python.

Buy the Book on Amazon Buy the DRM-free eBook

Already Have the Book?

Visit the GitHub project to see all of the code snippets from the book in one place. Run and modify the example code yourself to confirm your understanding. You can also report any errors you’ve found.

For future updates about the book, related videos, translations, conference presentations, and more, choose one of these ways to stay in touch:

More Info

What This Book Covers

Each chapter in Effective Python: Second Edition contains a broad but related set of items. Feel free to jump between all 90 items and follow your interest. Each item contains concise and specific guidance explaining how you can write Python programs more effectively. Items include advice on what to do, what to avoid, how to strike the right balance, and why this is the best choice. Items reference each other to make it easier to fill in the gaps as you read.

This second edition is focused exclusively on Python 3, up to and including version 3.8. It contains 30+ completely new items with additional best practices. Most of the original items from the first edition have been revised and included, but many have undergone substantial updates. For some items my advice has completely changed between the two editions of the book due best practices evolving as Python has matured.

If, for whatever reason, you’re still primarily using Python 2, despite its end-of-life on January 1st, 2020, the previous edition of the book (and its translations) may be more useful to you.

Chapter 1: Pythonic Thinking

The Python community has come to use the adjective Pythonic to describe code that follows a particular style. The idioms of Python have emerged over time through experience using the language and working with others. This chapter covers the best way to do the most common things in Python.

  1. Know Which Version of Python You’re Using
  2. Follow the PEP 8 Style Guide
  3. Know the Differences Between bytes and str
  4. Prefer Interpolated F-Strings Over C-style Format Strings and str.format
  5. Write Helper Functions Instead of Complex Expressions
  6. Prefer Multiple Assignment Unpacking Over Indexing
  7. Prefer enumerate Over range
  8. Use zip to Process Iterators in Parallel
  9. Avoid else Blocks After for and while Loops
  10. Prevent Repetition with Assignment Expressions

Chapter 2: Lists and Dictionaries

In Python, the most common way to organize information is in a sequence of values stored in a list. A lists natural complement is the dict that stores lookup keys mapped to corresponding values. This chapter covers how to build programs with these versatile building blocks.

  1. Know How to Slice Sequences
  2. Avoid Striding and Slicing in a Single Expression
  3. Prefer Catch-All Unpacking Over Slicing
  4. Sort by Complex Criteria Using the key Parameter
  5. Be Cautious When Relying on dict Insertion Ordering
  6. Prefer get Over in and KeyError to Handle Missing Dictionary Keys
  7. Prefer defaultdict Over setdefault to Handle Missing Items in Internal State
  8. Know How to Construct Key-Dependent Default Values with __missing__

Chapter 3: Functions

Functions in Python have a variety of extra features that make a programmer’s life easier. Some are similar to capabilities in other programming languages, but many are unique to Python. This chapter covers how to use functions to clarify intention, promote reuse, and reduce bugs.

  1. Never Unpack More Than Three Variables When Functions Return Multiple Values
  2. Prefer Raising Exceptions to Returning None
  3. Know How Closures Interact with Variable Scope
  4. Reduce Visual Noise with Variable Positional Arguments
  5. Provide Optional Behavior with Keyword Arguments
  6. Use None and Docstrings to Specify Dynamic Default Arguments
  7. Enforce Clarity with Keyword-Only and Position-Only Arguments
  8. Define Function Decorators with functools.wraps

Chapter 4: Comprehensions and Generators

Python has special syntax for quickly iterating through lists, dictionaries, and sets to generate derivative data structures. It also allows for a stream of iterable values to be incrementally returned by a function. This chapter covers how these features can provide better performance, reduced memory usage, and improved readability.

  1. Use Comprehensions Instead of map and filter
  2. Avoid More Than Two Control Subexpressions in Comprehensions
  3. Avoid Repeated Work in Comprehensions by Using Assignment Expressions
  4. Consider Generators Instead of Returning Lists
  5. Be Defensive When Iterating Over Arguments
  6. Consider Generator Expressions for Large List Comprehensions
  7. Compose Multiple Generators with yield from
  8. Avoid Injecting Data into Generators with send
  9. Avoid Causing State Transitions in Generators with throw
  10. Consider itertools for Working with Iterators and Generators

Chapter 5: Classes and Interfaces

Python is an object-oriented language. Getting things done in Python often requires writing new classes and defining how they interact through their interfaces and hierarchies. This chapter covers how to use classes to express your intended behaviors with objects.

  1. Compose Classes Instead of Nesting Many Levels of Built-in Types
  2. Accept Functions Instead of Classes for Simple Interfaces
  3. Use @classmethod Polymorphism to Construct Objects Generically
  4. Initialize Parent Classes with super
  5. Consider Composing Functionality with Mix-in Classes
  6. Prefer Public Attributes Over Private Ones
  7. Inherit from collections.abc for Custom Container Types

Chapter 6: Metaclasses and Attributes

Metaclasses and dynamic attributes are powerful Python features. However, they also enable you to implement extremely bizarre and unexpected behaviors. This chapter covers the common idioms for using these mechanisms to ensure that you follow the rule of least surprise.

  1. Use Plain Attributes Instead of Setter and Getter Methods
  2. Consider @property Instead of Refactoring Attributes
  3. Use Descriptors for Reusable @property Methods
  4. Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes
  5. Validate Subclasses with __init_subclass__
  6. Register Class Existence with __init_subclass__
  7. Annotate Class Attributes with __set_name__
  8. Prefer Class Decorators Over Metaclasses for Composable Class Extensions

Chapter 7: Concurrency and Parallelism

Python makes it easy to write concurrent programs that do many different things seemingly at the same time. Python can also be used to do parallel work through system calls, subprocesses, and C extensions. This chapter covers how to best utilize Python in these subtly different situations.

  1. Use subprocess to Manage Child Processes
  2. Use Threads for Blocking I/O, Avoid for Parallelism
  3. Use Lock to Prevent Data Races in Threads
  4. Use Queue to Coordinate Work Between Threads
  5. Know How to Recognize When Concurrency Is Necessary
  6. Avoid Creating New Thread Instances for On-demand Fan-out
  7. Understand How Using Queue for Concurrency Requires Refactoring
  8. Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency
  9. Achieve Highly Concurrent I/O with Coroutines
  10. Know How to Port Threaded I/O to asyncio
  11. Mix Threads and Coroutines to Ease the Transition to asyncio
  12. Avoid Blocking the asyncio Event Loop to Maximize Responsiveness
  13. Consider concurrent.futures for True Parallelism

Chapter 8: Robustness and Performance

Python has built-in features and modules that aid in hardening your programs so they are dependable. Python also includes tools to help you achieve higher performance with minimal effort. This chapter covers how to use Python to optimize your programs to maximize their reliability and efficiency in production.

  1. Take Advantage of Each Block in try/except/else/finally
  2. Consider contextlib and with Statements for Reusable try/finally Behavior
  3. Use datetime Instead of time for Local Clocks
  4. Make pickle Reliable with copyreg
  5. Use decimal When Precision Is Paramount
  6. Profile Before Optimizing
  7. Prefer deque for Producer–Consumer Queues
  8. Consider Searching Sorted Sequences with bisect
  9. Know How to Use heapq for Priority Queues
  10. Consider memoryview and bytearray for Zero-Copy Interactions with bytes

Chapter 9: Testing and Debugging

You should always test your code, regardless of what language it’s written in. However, Python’s dynamic features can increase the risk of runtime errors in unique ways. Luckily, they also make it easier to write tests and diagnose malfunctioning programs. This chapter covers Python’s built-in tools for testing and debugging.

  1. Use repr Strings for Debugging Output
  2. Verify Related Behaviors in TestCase Subclasses
  3. Isolate Tests from Each Other with setUp, tearDown, setUpModule, and tearDownModule
  4. Use Mocks to Test Code with Complex Dependencies
  5. Encapsulate Dependencies to Facilitate Mocking and Testing
  6. Consider Interactive Debugging with pdb
  7. Use tracemalloc to Understand Memory Usage and Leaks

Chapter 10: Collaboration

Collaborating on Python programs requires you to be deliberate about how you write your code. Even if you’re working alone, you’ll want to understand how to use modules written by others. This chapter covers the standard tools and best practices that enable people to work together on Python programs.

  1. Know Where to Find Community-Built Modules
  2. Use Virtual Environments for Isolated and Reproducible Dependencies
  3. Write Docstrings for Every Function, Class, and Module
  4. Use Packages to Organize Modules and Provide Stable APIs
  5. Consider Module-Scoped Code to Configure Deployment Environments
  6. Define a Root Exception to Insulate Callers from APIs
  7. Know How to Break Circular Dependencies
  8. Consider warnings to Refactor and Migrate Usage
  9. Consider Static Analysis via typing to Obviate Bugs

About the Author

Brett Slatkin

Brett Slatkin

Brett Slatkin is a principal software engineer at Google. He is the technical co-founder of Google Surveys, the co-creator of the PubSubHubbub protocol, and he launched Google’s first cloud computing product (App Engine). Fourteen years ago, he cut his teeth using Python to manage Google’s enormous fleet of servers.

Outside of his day job, he likes to play piano and surf (both poorly). He also enjoys writing about programming-related topics on his personal website. He earned his B.S. in computer engineering from Columbia University in the City of New York. He lives in San Francisco.

Updates

Item 10: Prevent Repetition with Assignment Expressions

Sun 02 February 2020

An assignment expression—also known as the walrus operator—is a new syntax introduced in Python 3.8 to solve a long-standing problem with the language that can cause code duplication. Whereas normal assignment statements are written a = b and pronounced “a equals b”, these assignments are written a := b and pronounced “a walrus b” (because := looks like a pair of eyeballs and tusks). Continue reading »

Item 51: Prefer Class Decorators Over Metaclasses for Composable Class Extensions

Wed 18 December 2019

Although metaclasses allow you to customize class creation in multiple ways (see Item 48: “Validate Subclasses with __init_subclass__” and Item 49: “Register Class Existence with __init_subclass__”), they still fall short of handling every situation that may arise. Continue reading »

Digital Versions of the 2nd Edition are Now Available

Wed 06 November 2019

You can immediately read the second edition of Effective Python as a DRM-free eBook, on Kindle, on Google Play, and on O’Reilly Online Learning.

Item 74: Consider memoryview and bytearray for Zero-Copy Interactions with bytes

Tue 22 October 2019

Though Python isn’t able to parallelize CPU-bound computation without extra effort (see Item 64: “Consider concurrent.futures for True Parallelism”), it is able to support high-throughput, parallel I/O in a variety of ways (see Item 53: “Use Threads for Blocking I/O, Avoid for Parallelism” and Item 60: “Achieve Highly Concurrent I/O with Coroutines” for details). That said, it’s surprisingly easy to use these I/O tools the wrong way and reach the conclusion that the language is too slow for even I/O-bound workloads. Continue reading »

Preorder the Second Edition

Mon 21 October 2019

Effective Python: Second Edition is now available for preorder. Follow this link to buy your copy in advance. It will ship in mid-November (2019) once the book has finished printing and is stocked in the warehouse. Digital editions will become available when the physical book ships or sooner.