Planet Python

Last update: October 28, 2026 07:49 PM UTC

October 28, 2026

Rodrigo Girão Serrão

4 upgrades to the Python standard library in 3.15

Learn about 4 of the most exciting updates made to the Python standard library in 3.15.

This article highlights 4 of the most exciting updates made to the Python standard library in 3.15:

• new object_hook in json.load: together with frozendict this unlocks a very powerful use case
• new module math.integer: new module that introduces integer arithmetic functions
• new re.prefixmatch function: clarifies some confusion that typically arises when using re.match
• new asyncio.TaskGroup.cancel function: ergonomic way of cancelling all the tasks in a task group

15 days of Python 3.15

To celebrate the release of Python 3.15, over the next 15 business days I'll be writing about a new 3.15 feature every day. Explained clearly and with examples so you don't have to sift through the changelog.

Subscribe to receive this free email series:

If you want, you can also check the schedule of the upcoming emails.

New object_hook in json.load

The function json.load now has a parameter object_hook, which can be set to a callable. When set, JSON objects that are decoded into dictionaries are passed to object_hook and replaced by their result.

For example, suppose you have the following JSON data in a file data.json:

{
  "name": {
    "first": "Rodrigo",
    "last": "Girão Serrão"
  },
  "interests": [
    "Python",
    "Rust",
    "AI"
  ]
}

Define an object hook and use it when loading the data:

import json

def object_hook(d):
    print(f"Inside object_hook with {d = }.")
    return d

with open("data.json", "r") as f:
    data = json.load(f, object_hook=object_hook)

Running this code produces two lines of output:

Inside object_hook with d = {'first': 'Rodrigo', 'last': 'Girão Serrão'}.
Inside object_hook with d = {'name': {'first': 'Rodrigo', 'last': 'Girão...

October 28, 2026 11:10 AM UTC

Python GUIs

Passing Arguments to Threaded Functions in PyQt6 — Understanding the difference between passing a function and calling a function when using QThreadPool and QRunnable

I'm running a long-running function in a separate thread using QThreadPool and a QRunnable worker. When I pass the function without arguments, it runs correctly in the background. But when I try to pass arguments, the GUI freezes until the function finishes. What am I doing wrong?

This is a very common stumbling block when working with threads in PyQt6, and the good news is that it has a simple fix. The problem comes down to the difference between passing a function and calling a function — two things that look similar in Python but behave very differently.

The Setup

Let's say you have a basic app with a main window, a push button, and a status bar. When the button is pressed, you want to start a long-running task on a background thread. That task should emit a signal to update the status bar, then carry on with its work.

Here's a minimal version of this pattern using QThreadPool and QRunnable, based on the approach covered in our Multithreading PyQt6 applications with QThreadPool tutorial:

import sys
import time

from PyQt6.QtCore import QObject, QRunnable, QThreadPool, pyqtSignal, pyqtSlot
from PyQt6.QtWidgets import QApplication, QMainWindow, QPushButton


class WorkerSignals(QObject):
    status = pyqtSignal(str)


class Worker(QRunnable):
    def __init__(self, fn, *args, **kwargs):
        super().__init__()
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    @pyqtSlot()
    def run(self):
        self.fn(*self.args, **self.kwargs)


class Model(QObject):
    def __init__(self):
        super().__init__()
        self.signals = WorkerSignals()

    def do_this(self):
        self.signals.status.emit("Working on it...")
        time.sleep(5)  # Simulating a long-running task
        self.signals.status.emit("Done!")


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()

        self.model = Model()
        self.model.signals.status.connect(self.update_status_bar)

        button = QPushButton("Run Task")
        button.pressed.connect(self.call_do_this)
        self.setCentralWidget(button)

        self.threadpool = QThreadPool()

    def call_do_this(self):
        worker = Worker(self.model.do_this)
        self.threadpool.start(worker)

    def update_status_bar(self, msg):
        self.statusBar().showMessage(msg)


app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec()

This works perfectly. The function runs on a background thread, the status bar updates, and the GUI stays responsive.

Where Things Go Wrong

Now suppose do_this needs to accept a message as an argument. You might naturally try this:

def call_do_this(self):
    worker = Worker(self.model.do_this("This is a message"))
    self.threadpool.start(worker)

Suddenly the GUI freezes for the entire duration of the task. The status bar doesn't update until everything is finished. What happened?

Passing a Function vs. Calling a Function

Look closely at what's being passed to Worker:

# This CALLS the function immediately and passes its return value
Worker(self.model.do_this("This is a message"))

# This PASSES the function itself, along with the argument separately
Worker(self.model.do_this, "This is a message")

When you write self.model.do_this("This is a message") with parentheses and arguments, Python executes that function right then and there, on the main thread. The function runs to completion — including the long-running part — and then its return value (which is None, since the function doesn't explicitly return anything) gets passed to Worker.

So Worker receives None as its fn parameter. The actual work already happened on the main thread, blocking the GUI the whole time.

When you write self.model.do_this without parentheses, you're passing a reference to the function itself. The Worker then calls it later, inside run(), which executes on a background thread. The arguments "This is a message" are passed separately and stored until run() is called.

The Fix

Pass the function and its arguments as separate items to Worker:

def call_do_this(self):
    worker = Worker(self.model.do_this, "This is a message")
    self.threadpool.start(worker)

The Worker.__init__ method is set up to handle exactly this pattern using *args and **kwargs:

class Worker(QRunnable):
    def __init__(self, fn, *args, **kwargs):
        super().__init__()
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    @pyqtSlot()
    def run(self):
        self.fn(*self.args, **self.kwargs)

The function reference is stored in self.fn, and any additional positional or keyword arguments are captured and stored in self.args and self.kwargs. When run() is called on the worker thread, it calls the function with those stored arguments.

Complete Working Example

Here's a full working example that demonstrates passing arguments to a threaded function, with proper signal handling for status updates:

import sys
import time

from PyQt6.QtCore import QObject, QRunnable, QThreadPool, pyqtSignal, pyqtSlot
from PyQt6.QtWidgets import (
    QApplication,
    QLabel,
    QMainWindow,
    QPushButton,
    QVBoxLayout,
    QWidget,
)


class WorkerSignals(QObject):
    """Signals available from a running worker thread."""
    status = pyqtSignal(str)
    finished = pyqtSignal()


class Worker(QRunnable):
    """Worker thread using QRunnable to run a function in the background."""

    def __init__(self, fn, *args, **kwargs):
        super().__init__()
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    @pyqtSlot()
    def run(self):
        self.fn(*self.args, **self.kwargs)


class Model(QObject):
    """Holds the application logic."""

    def __init__(self):
        super().__init__()
        self.signals = WorkerSignals()

    def do_this(self, msg):
        self.signals.status.emit(msg)
        # Simulate a long-running task
        for i in range(5):
            time.sleep(1)
            self.signals.status.emit(f"Processing step {i + 1} of 5...")
        self.signals.status.emit("Task complete!")
        self.signals.finished.emit()


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Thread Arguments Example")

        self.model = Model()
        self.model.signals.status.connect(self.update_status_bar)
        self.model.signals.finished.connect(self.task_finished)

        self.label = QLabel("Press the button to start a background task.")

        self.button = QPushButton("Run Task")
        self.button.pressed.connect(self.call_do_this)

        layout = QVBoxLayout()
        layout.addWidget(self.label)
        layout.addWidget(self.button)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

        self.threadpool = QThreadPool()

    def call_do_this(self):
        self.button.setEnabled(False)
        # Pass the function and its argument separately
        worker = Worker(self.model.do_this, "Starting task...")
        self.threadpool.start(worker)

    def update_status_bar(self, msg):
        self.statusBar().showMessage(msg)
        self.label.setText(msg)

    def task_finished(self):
        self.button.setEnabled(True)


app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec()

When you run this and click the button, you'll see the status bar and label update in real time as the background task progresses. The GUI stays responsive throughout because the work is happening on a separate thread.

Quick Reference

The key thing to remember is to pass the function by name, and pass the arguments separately after it. This keeps the function from running until the worker thread is ready.

If you're new to building PyQt6 applications and want to understand the fundamentals first, check out our guide to creating your first window with PyQt6.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

October 28, 2026 06:00 AM UTC

Getting a Value from a Model Using Row-Column Indices — How to retrieve data from a Qt model using row and column integers

How do I retrieve a value from a QAbstractTableModel using a pair of integers (row and column), for example when clicking a button, rather than clicking on a cell in a QTableView?

When you're working with QTableView and a custom model, you'll often want to access data in the model programmatically — maybe in response to a button click, or as part of some background logic. You might think you can just pass the row and column numbers directly to the model's data() method, but that's not quite how Qt's model/view system works.

Understanding QModelIndex

Qt's model/view framework uses a special object called a QModelIndex to refer to a specific cell in a model. The data() method on a model doesn't accept raw row and column integers directly. Instead, it expects a QModelIndex as its first argument.

To get data from a model using row and column integers, you first create a QModelIndex from those integers using the model's .index() method. Then you pass that index to .data().

Here's the pattern:

index = self.model.index(0, 0)
value = self.model.data(index, Qt.ItemDataRole.DisplayRole)

That's it. The .index(row, column) method creates the QModelIndex for you, and then .data() uses that index to look up the value.

The second argument, Qt.ItemDataRole.DisplayRole, tells the model which role you want — in this case, the display text. If your model returns strings for DisplayRole, then value will be a string.

A complete working example

Let's put this together in a small application with a QTableView and a QPushButton. Clicking the button retrieves the value at row 0, column 0 from the model and prints it:

import sys

from PyQt6.QtCore import Qt, QAbstractTableModel
from PyQt6.QtWidgets import (
    QApplication,
    QMainWindow,
    QPushButton,
    QTableView,
    QVBoxLayout,
    QWidget,
)

class TableModel(QAbstractTableModel):
    def __init__(self, data):
        super().__init__()
        self._data = data

    def data(self, index, role):
        if role == Qt.ItemDataRole.DisplayRole:
            return self._data[index.row()][index.column()]

    def rowCount(self, index):
        return len(self._data)

    def columnCount(self, index):
        return len(self._data[0])

class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()

        self.setWindowTitle("Model Data Access Example")

        data = [
            ["Apple", "Red", "Sweet"],
            ["Banana", "Yellow", "Sweet"],
            ["Broccoli", "Green", "Bitter"],
        ]

        self.model = TableModel(data)

        self.table = QTableView()
        self.table.setModel(self.model)

        self.button = QPushButton("Get value at row 0, column 0")
        self.button.clicked.connect(self.get_value)

        layout = QVBoxLayout()
        layout.addWidget(self.table)
        layout.addWidget(self.button)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

    def get_value(self):
        index = self.model.index(0, 0)
        value = self.model.data(index, Qt.ItemDataRole.DisplayRole)
        print(f"Value at row 0, column 0: {value}")

app = QApplication(sys.argv)
window = MainWindow()
window.show()
sys.exit(app.exec())

When you run this and click the button, you'll see Value at row 0, column 0: Apple printed to the console.

Accessing any cell

You can use any valid row and column values. For example, to get the value at row 2, column 1:

index = self.model.index(2, 1)
value = self.model.data(index, Qt.ItemDataRole.DisplayRole)
print(value)  # Output: Green

Just make sure the row and column are within the bounds of your data — otherwise the index won't be valid and you will get None back.

Summary

To retrieve data from a Qt table model using row and column integers:

1. Create a QModelIndex using self.model.index(row, column).
2. Pass that index to self.model.data(index, Qt.ItemDataRole.DisplayRole).

This two-step pattern — create the index, then query the data — is how Qt's model/view system is designed to work. Once you're comfortable with it, you can use the same approach to access data from any model, whether it's backed by a list, a NumPy array, or a Pandas DataFrame.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

October 28, 2026 06:00 AM UTC

"Michael Kennedy's Thoughts on Technology"

Chameleon templates, modernized: chameleon-rs

tl;dr; chameleon-rs is my Rust implementation of Chameleon for Python web apps. It runs existing templates unchanged, with faster rendering, async support, and support for free-threaded Python.

The Chameleon HTML (template) language is one of my favorites. It was the default in Pyramid when I used to use that framework for my projects. I’ve since created fastapi-chameleon and chameleon-flask to allow you to easily use Chameleon on these more modern web frameworks.

Why modernize Chameleon for free-threaded Python?

But the world has changed. With official support for free-threaded Python in Python 3.14 and continued evolution in 3.15, we need to start expecting more from our web apps. Is the Chameleon package updated for multi-threaded Python? Not at all. In fact, the repository shows support only up to Python 3.13 and a failing build. It’s not keeping up with the times.

And there are many more features that belong in modern templating languages for Python apps. Partial template support for excellent libraries like HTMX. Faster performance. Markdown support. And much more.

I wanted more performance, parallelism, and modern features from all the major templating languages like Chameleon, Jinja2, or Django templates. Amazingly, they have all failed to deliver. That changes today (at least for Chameleon, for now).

What is chameleon-rs?

I reimplemented the Chameleon template language almost 100% in Rust. The core goals were compatibility, performance, modernization, and thread safety.

Existing templates and code run unchanged, with byte-identical output, several times faster. It also adds what a modern Python web app expects:

• Async rendering: keeps the event loop free while awaiting data.
• Built-in framework integrations: Flask/Quart, FastAPI, Starlette, Robyn, and Litestar.
• Template features: partials and Markdown.
• Free-threaded Python support: thread-safe rendering without the GIL.

No runtime dependencies.

How much faster is chameleon-rs?

Measured result: In my web app, switching to chameleon-rs cut median response times by roughly 20% across the site.

chameleon-rs compiles a template to a native program and runs it in Rust. The template’s own work (the loops, conditions, escaping, macro and slot calls, partials and output assembly) happens without Python bytecode. Python only runs where the template calls into your code: an attribute lookup, a method call, a helper.

You can see our synthetic benchmarks on the performance page of the docs. But the graph below is much more convincing. It shows the median response time when running on traditional Chameleon (the package) or chameleon-rs.

This is not “templates render 20% faster”. This is the entire web app, with user authentication, significant database work, logging, ecommerce, etc., seeing roughly 20% lower response times by using chameleon-rs!

How do I switch from Chameleon to chameleon-rs?

chameleon-rs runs code written for Chameleon unchanged. There are two ways to switch: install the compatibility alias and change nothing else, or change your imports.

One line: compat.install()

Add this at the very top of your app’s entry point (the module your server imports, or __main__), before anything imports Chameleon:

import chameleon_rs.compat
chameleon_rs.compat.install()

Or adopt more of the API and change the imports. Every public name is available from chameleon_rs directly:

# Before
from chameleon import PageTemplate, PageTemplateFile, PageTemplateLoader
import chameleon_partials
import chameleon_flask

# After
from chameleon_rs import PageTemplate, PageTemplateFile, PageTemplateLoader
import chameleon_rs.partials as chameleon_partials
import chameleon_rs.flask as chameleon_flask

What modern features does chameleon-rs add?

Performance itself is a feature. Compatibility is a feature. But why stop there? I added partials support, Markdown support, and asyncio support directly inside the template, as well as automatic event loop support for any async def view function. That’s the modernization part. And it was achieved by making chameleon-rs a superset of what Chameleon offered before, not by changing it.

Built-in template partials

A partial is a template file rendered inside another template, with its own small model. Use them for anything you repeat: a card in a listing, a form field, a table row, a fragment you also return on its own to HTMX (see HTMX).

chameleon-rs has partials built in. Every template can call render_partial, with nothing to register or pass in:

<div tal:repeat="video videos">
  ${render_partial('shared/partials/video_card.pt', video=video, show_date=True)}
</div>

Markdown inside Chameleon templates

chameleon-rs can convert Markdown to HTML while it renders, with a markdown: expression:

<article>${markdown: episode.notes}</article>
<div tal:content="markdown: comment.text"></div>

It’s opt-in: Chameleon has nothing like it, so you turn it on with configure() and use it in new templates. The converter is built into chameleon-rs (comrak, in Rust), so there’s no Markdown package to install. And because it entirely happens in Rust, it’s roughly 26x faster than mistune with the same extensions.

Turning it on

import chameleon_rs
from chameleon_rs import PageTemplate

chameleon_rs.configure(markdown=True)

page = PageTemplate('<article>${markdown: notes}</article>')
print(page(notes='# Show notes\n\nWe talked about **templates** and `Rust`.'))

Async rendering with render_async()

Chameleon renders synchronously. In an async web app (FastAPI, Starlette, Quart, Litestar, Robyn) that means a render runs on the event loop, and nothing else in that worker runs until it finishes. chameleon-rs keeps render() exactly as it was and adds render_async(), which renders on the event loop without holding it while it waits for data.

The framework integrations use it for you: an async def view decorated with @template renders with render_async. See the async rendering guide for the details and opt-in extras.

Thread safety for free-threaded Python

chameleon-rs has been tested extensively to be thread-safe. I have repeatedly profiled and optimized it to share as little data and take as few locks as possible. So when you’re ready to move to a truly multi-threaded (no GIL) web framework, your template library won’t hold you back.

How can I test chameleon-rs before switching?

If this sounds interesting, you can use shadow mode to try it out. Shadow mode renders a sample of real traffic with both engines (existing Chameleon and chameleon-rs) and reports any difference, while Chameleon keeps serving. Your users won’t see a difference, and you can check that the sampled requests get exactly the same output.

October 28, 2026 03:01 AM UTC

October 27, 2026

Talk Python to Me

#567: Personal AI Agents for Pythonistas

The most popular Python project you've probably never heard of has 252,000 GitHub stars and merged over 1,800 pull requests in a single week. It's Hermes, a self-hosted AI agent you program just by talking to it. And it never clocks out. It runs 24/7 on a box in your house, wakes up every 15 minutes, keeps your YouTube channel in sync, and has your morning briefing ready before you're up. Mine started as a few config files. A few months later, it's 16,000 files of skills, memories, and tools it built for itself, all while I wasn't watching. Jeff Triplett joins me to explore this honestly weird but kind of amazing world. And yes, Hermes has a few words for you, too.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/talk-python-membership-inner-circle'>Talk Python Membership</a><br> <a href='https://talkpython.fm/talk-python-membership-inner-circle'>Talk Python Membership</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading mb-4">Links from the show</h2> <div><strong>Talk Python Tech Talks</strong>: <a href="https://talkpython.fm/meetups" target="_blank" >talkpython.fm/meetups</a><br/> <br/> <strong>Hermes</strong>: <a href="https://github.com/NousResearch/hermes-agent?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Ollama cloud models</strong>: <a href="https://ollama.com/search?c=cloud&amp;featured_on=talkpython" target="_blank" >ollama.com</a><br/> <strong>https\://hermes-agent.nousresearch.com</strong>: <a href="https://hermes-agent.nousresearch.com?featured_on=talkpython" target="_blank" >hermes-agent.nousresearch.com</a><br/> <strong>Hermes Web UI</strong>: <a href="https://get-hermes.ai/setup/?featured_on=talkpython" target="_blank" >get-hermes.ai</a><br/> <strong>built my own UI</strong>: <a href="https://blobs.talkpython.fm/hermes-iris-example.webp?cache_id=a338c4" target="_blank" >blobs.talkpython.fm</a><br/> <strong>the Hermes Dashboard</strong>: <a href="https://blobs.talkpython.fm/hermes-dashboard.png?cache_id=931bf5" target="_blank" >blobs.talkpython.fm</a><br/> <strong>the audio from the chat example</strong>: <a href="https://blobs.talkpython.fm/hermes-greets-talk-python-listeners.mp3?cache_id=9c1669" target="_blank" >blobs.talkpython.fm</a><br/> <strong>the CLI</strong>: <a href="https://blobs.talkpython.fm/hermes-cli.webp?cache_id=4f8b2c" target="_blank" >blobs.talkpython.fm</a><br/> <strong>Tailscale</strong>: <a href="https://tailscale.com?featured_on=talkpython" target="_blank" >tailscale.com</a><br/> <strong>Hindsight</strong>: <a href="https://hindsight.vectorize.io?featured_on=talkpython" target="_blank" >hindsight.vectorize.io</a><br/> <strong>trmnl display</strong>: <a href="https://trmnl.com?featured_on=talkpython" target="_blank" >trmnl.com</a><br/> <strong>JEV</strong>: <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev?featured_on=talkpython" target="_blank" >typesafe.ai</a><br/> <strong>Home Assistant Green</strong>: <a href="https://www.home-assistant.io/green/?featured_on=talkpython" target="_blank" >www.home-assistant.io</a><br/> <strong>QMD - Query Markup Documents</strong>: <a href="https://github.com/tobi/qmd?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>OrangePi</strong>: <a href="http://www.orangepi.org?featured_on=talkpython" target="_blank" >www.orangepi.org</a><br/> <br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=ZfV5G5EsJa8" target="_blank" >youtube.com</a><br/> <strong>Episode #567 deep-dive</strong>: <a href="https://talkpython.fm/episodes/show/567/personal-ai-agents-for-pythonistas#takeaways-anchor" target="_blank" >talkpython.fm/567</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/567/personal-ai-agents-for-pythonistas" target="_blank" >talkpython.fm</a><br/> <br/> <strong>Theme Song: Developer Rap</strong><br/> <strong>🥁 Served in a Flask 🎸</strong>: <a href="https://talkpython.fm/flasksong" target="_blank" >talkpython.fm/flasksong</a><br/> <br/> <strong>---== Don't be a stranger ==---</strong><br/> <strong>YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" ><i class="fa-brands fa-youtube"></i> youtube.com/@talkpython</a><br/> <br/> <strong>Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm</a><br/> <strong>Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i> @talkpython@fosstodon.org</a><br/> <strong>X.com</strong>: <a href="https://x.com/talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @talkpython</a><br/> <br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i> @mkennedy@fosstodon.org</a><br/> <strong>Michael on X.com</strong>: <a href="https://x.com/mkennedy?featured_on=talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @mkennedy</a><br/></div>

October 27, 2026 10:27 PM UTC

PyCoder’s Weekly

Issue #758: Security, Optimization, itertools, and More (2026-10-27)

#758 – OCTOBER 27, 2026
View in Browser »

Current vs. Historical Memory in Python Agents

Your Python agent stores a preference, the user changes it, and vector search can return both versions. The fix is linking the new memory to the old one, so the old one is marked historical and stays available. Explore the walkthrough and notebook
ORACLE sponsor

Articles & Tutorials

Unblock Your Coding Agent’s Web Access

Ask Claude Code or Cursor to fetch a page and it often comes back blocked, half-rendered, or buried in markup. Zyte MCP, from the maintainers of Scrapy, connects your agent to Zyte API for rendered pages, clean Markdown, and structured data in the same session. It’s hosted, connects with one command Connect Zyte MCP in one command →
ZYTE sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #758.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 27, 2026 07:30 PM UTC

Python Bytes

#502 Introducing chameleon-rs

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong>PHP wasn’t the only problem last week</strong></li> <li><strong><a href="https://mkennedy.codes/docs/chameleon-rs/?featured_on=pythonbytes">chameleon-rs</a></strong></li> <li><strong><a href="https://astral.sh/blog/uv-ty-integration?featured_on=pythonbytes">Bringing uv and ty together</a></strong></li> <li><strong><a href="https://github.com/whitphx/stlite?featured_on=pythonbytes">Stlite runs Streamlit apps entirely in the browser</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=V7m5MAhp4qE' style='font-weight: bold;' data-umami-event="Livestream-Past" data-umami-event-episode="502">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy">Mastodon</a> / <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/mkennedy?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/mkennedy/?featured_on=pythonbytes">LinkedIn</a></li> <li>Calvin: <a href="https://sixfeetup.social/@calvin?featured_on=pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/calvinhp.com?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/calvinhp?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/calvinhp/?featured_on=pythonbytes">LinkedIn</a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/pythonbytes.fm">BlueSky</a> / <a href="https://x.com/PythonBytes?featured_on=pythonbytes">X</a> Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Tuesday at 7am PT</strong>. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</li> </ul> <p><strong>Calvin #1: PHP wasn’t the only problem last week</strong></p> <ul> <li>Three security incidents in one week, each at a different layer: CA trust (eMudhra), the release pipeline (LiteLLM) and a C decoder (Brotli). All three have fixes released.</li> <li>eMudhra CA: the keys of a compromised eMudhra intermediate CA are now distrusted by CPython. 3.15.1, 3.14.9 and 3.13.17 shipped Oct 21. 3.15.2, 3.14.10, 3.13.18, 3.12.16 and 3.11.18 followed Oct 26 (CVE-2026-115557) and also fix a tarfile filter bypass. Python's post says pip operations are safe even before pip is upgraded.</li> <li>eMudhra, tooling: pip 26.2.2 bundles certifi 2026.10.19, which removes the root. uv 0.13.4 (GHSA-hq85-5cpj-mr62) and 0.13.5 (rustls-webpki 0.103.16) are the uv fixes. After upgrading uv, run uv python upgrade. Existing venvs need pip upgraded by hand, and the 3.9 and 3.10 builds still bundle a pip that trusts the root. <ul> <li>Anyone holding those intermediate keys can mint certificates for any domain that an unpatched client will accept as valid.</li> <li>That enables man-in-the-middle attacks: reading or altering HTTPS traffic without a certificate warning.</li> <li>The risk is highest where the attacker can sit on the network path, such as hostile Wi-Fi, a compromised router or ISP, or a cloud network position.</li> <li>For Python specifically, the exposed traffic would be package downloads from PyPI, API calls with credentials or tokens, and webhooks. A tampered package download is the worst case.</li> <li>Containers and old venvs that nobody rebuilds are where this lingers. uv says its Python 3.9 and 3.10 builds still bundle a pip that trusts the root, and existing venvs don't pick up the fix automatically.</li> </ul></li> <li>LiteLLM 1.107.0: malicious wheels hit PyPI on Oct 21. All seven wheels were live from about 11:40 to 13:23 UTC, with roughly 49k downloads. The Docker images were also infected. It's the second LiteLLM compromise this year, after 1.82.7 and 1.82.8 in March.</li> <li>LiteLLM payload: cook.pth runs on every Python start, so nothing has to import litellm. It steals SSH, cloud, browser, CI and AI-tool credentials, and an hourly loader runs remote code. The clean pin is litellm==1.106.4. Kill the loader first, rebuild rather than repair, then rotate everything the host could reach.</li> <li>LiteLLM root cause: a master key committed to a public repo on May 27 was never revoked. On Oct 10 it yielded an org owner's GitHub CLI token. The attacker then changed the trusted release workflow to build from a lookalike Berri-AI/litellm repo. BerriAI's first explanation (tokens in logs) was wrong, and the Oct 23 report supersedes it.</li> <li>Broccoli2Shell: CVE-2026-111497 is an out-of-bounds write in Google's Brotli C decoder. It affects the Python Brotli and brotlicffi packages 1.0.3 through 1.2.0(.x). Upgrade to Brotli 1.2.1 or brotlicffi 1.2.1.0. The CVE was published Oct 22 and rated 9.2 Critical by GitHub.</li> <li>Broccoli2Shell workaround: if you can't upgrade, uninstall both packages. urllib3, requests and aiohttp then fall back to gzip and deflate. Also reject Content-Encoding: br request bodies at the proxy or WAF. <strong>Sources</strong></li> <li>https://blog.python.org/2026/10/python-3152-31410-31318-31216-31118/</li> <li>https://www.stepsecurity.io/blog/litellm-1-107-0-compromised-pypi-backdoored-wheels-credential-stealer</li> <li>https://broccoli2shell.com/</li> </ul> <p><strong>Michael #2: <a href="https://mkennedy.codes/docs/chameleon-rs/?featured_on=pythonbytes">chameleon-rs</a></strong></p> <ul> <li><strong>Chameleon templates</strong>, much faster than before (up to 70x faster on start)</li> <li><strong>Modernized</strong>: A Rust engine that runs your Chameleon templates unchanged with async rendering, partials (think HTMX), Markdown</li> <li><strong>Built-in framework support</strong>: Flask, Quart, FastAPI, Starlette, Pyramid, Robyn and Litestar integrations.</li> <li><strong>A <a href="https://mkennedy.codes/docs/chameleon-rs/user-guide/switching-from-chameleon.html?featured_on=pythonbytes">compatibility shim</a></strong> for testing out quickly: <code>chameleon_rs.compat.install()</code></li> <li><strong>AI agent support</strong> via docs (every doc page has a .md extension variant, eg <a href="https://mkennedy.codes/docs/chameleon-rs/user-guide/fastapi.md?featured_on=pythonbytes">FastAPI docs</a>) and <a href="https://mkennedy.codes/docs/chameleon-rs/skills.html?featured_on=pythonbytes">the skill</a> is installable or downloadable.</li> <li><strong>Byte-for-byte identical</strong> on every page for <a href="http://talkpython.fm?featured_on=pythonbytes">talkpython.fm</a> and <a href="http://training.talkpython.fm?featured_on=pythonbytes">training.talkpython.fm</a></li> <li>Passes 100% of the unit tests of the <a href="https://pypi.org/project/Chameleon/?featured_on=pythonbytes">original chameleon project</a>. See <a href="https://mkennedy.codes/docs/chameleon-rs/user-guide/compatibility.html?featured_on=pythonbytes">compatibility page</a>.</li> <li>Natively supports my <a href="https://mkennedy.codes/docs/chameleon-flask/?featured_on=pythonbytes">chameleon-flask</a>, <a href="https://mkennedy.codes/docs/fastapi-chameleon/?featured_on=pythonbytes">fastapi-chameleon</a>, and <a href="https://mkennedy.codes/docs/chameleon-partials/?featured_on=pythonbytes">chameleon-partials</a> libraries.</li> <li>Swapping Talk Python Training to chameleon-rs <a href="https://blobs.pythonbytes.fm/chameleon-vs-chameleon-rs-perf-chart.webp?cache_id=c63b7e">made it 20% fast</a> (end-to-end, not just template rendering)!</li> </ul> <p><strong>Calvin #3: <a href="https://astral.sh/blog/uv-ty-integration?featured_on=pythonbytes">Bringing uv and ty together</a></strong></p> <ul> <li>Status: Astral (Sara Sklenka, Oct 13) announced "an experimental integration between uv and ty, available in preview." They ask for feedback, and say "in parallel, we're continuing toward ty's stable release."</li> <li>uv check: a new uv command that prepares the project environment, installs dependencies and runs ty. Flags are --package [HTML_REMOVED], --all-packages and --script <a href="http://example.py/?featured_on=pythonbytes">example.py</a>. Ruff support is planned, so lint and type check would run in one command.</li> <li>Lockfile pinning: uv check uses the ty version recorded in your lockfile. Developers and CI therefore share one ty version without separate pins.</li> <li>PEP 723 scripts: ty reads inline script metadata and uses uv to build an environment per script. Each script is checked against its own declared dependencies, even when they differ from the project's. In the editor, saving a script syncs it with uv.</li> <li>Enabling it: running uv check turns it on automatically. In VS Code, install the astral-sh.ty extension and set "ty.experimental.useUv": "on". On the command line, run TY_UV=scripts ty check.</li> <li>Opt-in: the post says automatic environment setup "is currently opt-in" because installing dependencies can execute third-party code. Astral plans to enable script support by default over time, and automatic setup for standard projects.</li> <li>Under the hood: it's powered by uv workspace metadata. uv 0.13.6 (Oct 26) changed that preview output to nest each module's owners in an object, so anything parsing the old flat structure breaks.</li> <li>Versions: it needs uv 0.12.3+ and ty 0.0.76+. Current releases are past both (uv 0.13.6, ty 0.0.92 on Oct 26), though I couldn't confirm 0.0.76's exact date.</li> <li>Next: Astral says it's working on checks that combine uv's dependency knowledge with ty's code analysis, such as detecting unused dependencies.</li> <li>Also credit to listener Sky Kasko. :)</li> </ul> <p><strong>Michael #4: <a href="https://github.com/whitphx/stlite?featured_on=pythonbytes">Stlite runs Streamlit apps entirely in the browser</a></strong></p> <p><a href="https://streamlit.io?featured_on=pythonbytes">Streamlit</a> is a very unique way to create data-driven dashboards. <a href="https://github.com/whitphx/stlite?featured_on=pythonbytes">Stlite</a> is a WebAssembly port of Streamlit built on Pyodide, so a Streamlit app runs completely in the visitor's browser with no Python server behind it. It ships as three packages (@stlite/browser for web pages, @stlite/desktop for offline desktop apps, @stlite/react for embedding in React), plus Stlite Sharing, a hosted editor for trying apps online. The project has been active since 2022 and is now at 0.85.1 with sponsorship from Streamlit/Snowflake and TestMu AI.</p> <ul> <li>A Streamlit app becomes static files you can host anywhere, the same story JupyterLite tells for notebooks.</li> <li>Deployment shrinks to a script tag or a single HTML file; the server only serves static assets and never runs Python.</li> <li>It patches requests and urllib through pyodide-http and supports pyfetch, so most existing HTTP code keeps working.</li> <li>Real limitations: time.sleep is a no-op in favor of asyncio.sleep, asyncio.run breaks, and packages with unported binary extensions cannot install.</li> <li>Files default to ephemeral memory, but IDBFS mountpoints persist data across reloads in the browser's IndexedDB.</li> <li>A SharedWorker mode lets multiple apps share one Python runtime to cut memory, at the cost of shared state between apps.</li> <li>The @stlite/desktop bundler turns an app into an offline desktop executable, demonstrated converting Streamlit apps to .exe files.</li> </ul> <p><strong>Extras</strong></p> <p>Calvin:</p> <ul> <li><a href="https://realpython.com/podcasts/rpp/316/?featured_on=pythonbytes">Real Python Podcast #316</a> with Seth Larson on Python security</li> <li><a href="https://pyfound.blogspot.com/2026/10/wikis-relaunched.html?featured_on=pythonbytes">**PSF wikis</a> relaunched on GitHub Pages** Michael:</li> <li><a href="https://www.bleepingcomputer.com/news/security/php-ships-emergency-patches-for-critical-php2shell-flaws/?featured_on=pythonbytes">PHP ships emergency patches for critical PHP2Shell flaws</a></li> <li><a href="https://github.com/openai/php2shell-checker?featured_on=pythonbytes">OpenAI released a checker</a></li> <li><a href="https://talkpython.fm/meetups?featured_on=pythonbytes">Talk Python Next Tech Talk: Talk Python in Production(v2)</a></li> <li><a href="https://0ver.org/#notable-zerover-projects">Notable ZeroVer Projects</a> via Vincent Poulailleau</li> <li><a href="https://docs.python.org/release/3.15.2/whatsnew/changelog.html?featured_on=pythonbytes">Python 3.15.2 is out</a></li> </ul> <p><strong>Joke: <a href="https://x.com/PR0GRAMMERHUM0R/status/2056495137933394100?featured_on=pythonbytes">captcha: too real!</a></strong></p>

October 27, 2026 05:39 PM UTC

Rodrigo Girão Serrão

Typing improvements in Python 3.15

This article goes over the major improvements made to the typing module in Python 3.15.

Since static type hinting is becoming more and more popular in the Python world, every Python release brings with it major improvements to the world of typing. This article goes over the main improvements made to typing in Python 3.15 like the improvements to TypedDict, the addition of typing.TypeForm, or the addition of the decorator typing.disjoint_base.

15 days of Python 3.15

To celebrate the release of Python 3.15, over the next 15 business days I'll be writing about a new 3.15 feature every day. Explained clearly and with examples so you don't have to sift through the changelog.

Subscribe to receive this free email series:

If you want, you can also check the schedule of the upcoming emails.

New class parameters for TypedDict

Python 3.15 added two class parameters to TypedDict: extra_items and closed. The gist of it is that extra_items allows specifying that your dictionary may have other, unknown keys whose values have the specified type:

from typing import TypedDict

class Movie(TypedDict, extra_items=str):
    title: str
    year: int

The definition above means that a variable of type Movie might have other keys, as long as their values are assignable to the type str:

movie: Movie = {
    "title": "Forrest Gump",
    "year": 1994,
    "genre": "Comedy / Romance",
}

The extra item "genre": "Comedy / Romance" is allowed because the value is a string and extra_items=str.

To completely disallow extra items, regardless of their type, you can set closed=True as a class parameter.

There's an article dedicated to the improvements made to TypedDict in Python 3.15 that...

October 27, 2026 10:54 AM UTC

PyPy

The Shift of Theseus — Toy Bit Provenance Abstract Domain

The Knownbits Toy abstract domain that I wrote about earlier is great for all kinds of bit manipulation rewrites. However, it is not particularly good at recognizing where bits move around in a sequence of instructions. It has a notion of "this bit is 0 or 1 or unknown" but no notion of "this bit is the same as this other bit". This prevents it from optimizing something like this:

x = ...
a = x & 0x0f
b = x << 60
c = b >> 60
result = a & c

Here a and c compute the same value, so it can be optimized to result = a. It is possible to do this optimization with local peephole rewrite rules too, but doing that requires adding a lot of these rules, and if one is missing we might miss collapsing a long sequence of operations.

In this post I want to talk about a more complicated abstract domain that is great at optimizing this kind of bit movement. It's a "bit provenance" domain. For every SSA variable in a trace, it computes where each bit from the variable comes from (or whether it's constant 0 or 1). In particular it can track if a bit in one variable must be the same as the bit of another variable at another position.

For the example above, the domain would store the following provenance information. Nothing is known about x, so its bits refer to themselves. Using 64-bit integers and a logical right shift, we get this table (most significant bits first):

Then we can directly conclude that result is the same value as a.

In the rest of the post I'll describe how the domain values store this information, how the transfer functions work and how we can use the abstract domain to optimize.

Contents

• Implementing the Domain

• Implementing Transfer Functions

• Implementing Bitwise Binary Transfer Functions

• Extracting Simpler Operations

  • Constants

  • Identity

  • Reusing Earlier Values

  • Simple Shifts

  • More Rewrites from the Same Rules

• Evaluation with Pydrofoil and PyPy

• Related work

• Conclusion

• Acknowledgements

Implementing the Domain

We'll start with reusing the toy optimizer from the previous posts, in the fold for completeness’ sake.

from dataclasses import dataclass

INTEGER_WIDTH = 64
INT_MASK = 2 ** INTEGER_WIDTH - 1

class Value:
    def find(self):
        raise NotImplementedError("abstract")


@dataclass(eq=False)
class Operation(Value):
    name : str
    args : list[Value]

    forwarded : Value | None = None

    def find(self) -> Value:
        op = self
        while isinstance(op, Operation):
            next = op.forwarded
            if next is None:
                return op
            op = next
        return op

    def arg(self, index):
        return self.args[index].find()

    def make_equal_to(self, value : Value):
        self.find().forwarded = value


@dataclass(eq=False)
class Constant(Value):
    value : object

    def find(self):
        return self


class Block(list):
    def __getattr__(self, opname):
        def wraparg(arg):
            if not isinstance(arg, Value):
                arg = Constant(arg)
            return arg
        def make_op(*args):
            op = Operation(opname,
                [wraparg(arg) for arg in args])
            self.append(op)
            return op
        return make_op


def bb_to_str(l : Block, varprefix : str = "var"):
    def arg_to_str(arg : Value):
        if isinstance(arg, Constant):
            return str(arg.value)
        else:
            return varnames[arg]

    varnames = {}
    res = []
    for index, op in enumerate(l):
        # give the operation a name used while
        # printing:
        var =  f"{varprefix}{index}"
        varnames[op] = var
        arguments = ", ".join(
            arg_to_str(op.arg(i))
                for i in range(len(op.args))
        )
        strop = f"{var} = {op.name}({arguments})"
        res.append(strop)
    return "\n".join(res)

def dce(bb: Block) -> Block:
    pure = {"getarg", "int_and", "int_or", "int_xor", "int_invert",
            "int_lshift", "int_rshift"}
    used = set()
    kept = []
    for op in reversed(bb):
        if op not in used and op.name in pure:
            continue
        kept.append(op)
        for i in range(len(op.args)):
            arg = op.arg(i)
            if isinstance(arg, Operation):
                used.add(arg)
    return Block(reversed(kept))


def test_dce():
    bb = Block()
    x = bb.getarg(0)
    live = bb.int_or(x, 1)
    dead = bb.int_xor(live, 7)
    bb.int_and(dead, 15)
    forwarded = bb.int_and(x, 0)
    forwarded.make_equal_to(Constant(0))
    bb.dummy(live)
    bb.dummy(forwarded)

    assert bb_to_str(dce(bb), "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_or(optvar0, 1)
optvar2 = dummy(optvar1)
optvar3 = dummy(0)"""

(The dce function was adapted from the Prospero blog post.)

Then we can start implementing the domain values like this:

@dataclass(frozen=True)
class SourceBit:
    source: Operation
    index: int
    inverted: bool = False


@dataclass(frozen=True)
class BitProvenance:
    # least significant bit first
    bits: tuple[int | SourceBit, ...]

    @staticmethod
    def make(bits):
        bits = tuple(bits)
        if len(bits) < INTEGER_WIDTH:
            # zero-extend, mainly for tests
            bits = bits + (0, ) * (INTEGER_WIDTH - len(bits))
        assert len(bits) == INTEGER_WIDTH
        return BitProvenance(bits)

    @staticmethod
    def from_constant(value):
        return BitProvenance(tuple((value >> i) & 1 for i in range(INTEGER_WIDTH)))

    @staticmethod
    def from_source(source: Operation):
        # nothing is known about this SSA value
        return BitProvenance(tuple(SourceBit(source, i) for i in range(INTEGER_WIDTH)))

    def get_constant(self) -> int | None:
        res = 0
        shift = 0
        for bit in self.bits:
            if isinstance(bit, SourceBit):
                return None
            res |= bit << shift
            shift += 1
        return res

Every SourceBit refers to a bit of an SSA Operation. For an SSA value that we know nothing about, we use the from_source factory method, which returns something of the form:

x = ... # x[63] x[62] x[61] x[60] x[59] ... x[ 3] x[ 2] x[ 1] x[0]

For a constant, the from_constant factory returns an abstract value where all the bits are concretely known values. Constants are simply 0 or 1 and need no source at all.

The inverted field of a SourceBit specifies whether the bit is a direct copy of the source bit, or whether the bit is flipped.

Let's make sure we got the bit twiddling in from_constant and get_constant right by writing a Hypothesis test:

from hypothesis import given, strategies

ints = strategies.integers(0, 2**64-1)

@given(ints)
def test_hypothesis_from_constant_get_constant_roundtrip(val):
    assert BitProvenance.from_constant(val).get_constant() == val

Implementing Transfer Functions

The simplest transfer function to implement for this domain is for ~, bit invert. It inverts constant bits and flips the inverted field of any SourceBit. Here's a test:

def test_abstract_invert():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.make([1, 0, SourceBit(source, 1, True), SourceBit(source, 0)])
    res = x.abstract_invert()
    assert res == BitProvenance.make((0, 1, SourceBit(source, 1), SourceBit(source, 0, True)) + (1, ) * 60)

The implementation looks like this (the reason for the result argument will become clear a bit further down):

def invert_bit(bit):
    if isinstance(bit, int):
        return bit ^ 1
    return SourceBit(bit.source, bit.index, not bit.inverted)


class BitProvenance:
    ...

    def abstract_invert(self, result=None):
        return BitProvenance.make(tuple(invert_bit(bit) for bit in self.bits))

The shift implementations are also quite simple as long as the shift amounts are constant, they just move bits around. If the shifts aren't constant, we just know nothing about the result.

If we "know nothing", we want to use BitProvenance.from_source for the shift operation. Therefore the transfer functions need to get the shift Operation as an extra argument.

def test_lshift():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.from_source(source) # no info

    # x << 7, the seven high bits get cut off
    seven = BitProvenance.from_constant(7)
    result = Operation("int_lshift", [source, Constant(7)])
    assert x.abstract_lshift(seven, result).bits == (0, ) * 7 + x.bits[:-7]

    # x << 65, constant 0
    big = BitProvenance.from_constant(65)
    result = Operation("int_lshift", [source, Constant(65)])
    assert x.abstract_lshift(big, result).get_constant() == 0

    # x << y, we know nothing
    amount = Operation("getarg", [Constant(1)])
    result = Operation("int_lshift", [source, amount])
    assert x.abstract_lshift(BitProvenance.from_source(amount), result) == BitProvenance.from_source(result)


@given(ints, strategies.integers(0, 100))
def test_hypothesis_lshift_on_constants(val, shift):
    absval = BitProvenance.from_constant(val)
    absshift = BitProvenance.from_constant(shift)
    result = Operation("int_lshift", [Constant(val), Constant(shift)])
    assert absval.abstract_lshift(absshift, result).get_constant() == (val << shift) & INT_MASK


class BitProvenance:
    ...

    def abstract_lshift(self, other, result: Operation):
        const = other.get_constant()
        if const is None:
            return BitProvenance.from_source(result)
        if const >= INTEGER_WIDTH:
            # make oversized shift amounts return 0 for simplicity
            return BitProvenance.from_constant(0)
        if const == 0:
            return self
        return BitProvenance.make((0, ) * const + self.bits[:-const])

Logical right shift is pretty similar, except we append the zeros at the other end:

def test_rshift():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.from_source(source) # no info

    # x >> 7, the seven low bits get cut off
    seven = BitProvenance.from_constant(7)
    result = Operation("int_rshift", [source, Constant(7)])
    assert x.abstract_rshift(seven, result).bits == x.bits[7:] + (0, ) * 7

    # x >> 65, constant 0
    big = BitProvenance.from_constant(65)
    result = Operation("int_rshift", [source, Constant(65)])
    assert x.abstract_rshift(big, result).get_constant() == 0

    # x >> y, we know nothing
    amount = Operation("getarg", [Constant(1)])
    result = Operation("int_rshift", [source, amount])
    assert x.abstract_rshift(BitProvenance.from_source(amount), result) == BitProvenance.from_source(result)


@given(ints, strategies.integers(0, 100))
def test_hypothesis_rshift_on_constants(val, shift):
    absval = BitProvenance.from_constant(val)
    absshift = BitProvenance.from_constant(shift)
    result = Operation("int_rshift", [Constant(val), Constant(shift)])
    assert absval.abstract_rshift(absshift, result).get_constant() == val >> shift


class BitProvenance:
    ...

    def abstract_rshift(self, other, result: Operation):
        const = other.get_constant()
        if const is None:
            return BitProvenance.from_source(result)
        if const >= INTEGER_WIDTH:
            return BitProvenance.from_constant(0)
        if const == 0:
            return self
        return BitProvenance.make(self.bits[const:] + (0, ) * const)

Implementing Bitwise Binary Transfer Functions

Next are the abstract binary transfer functions and, or, xor. We can think about the behaviour on individual bits first. A few cases are simple (e.g. a & 0 == 0) but some need reasoning about sources.

def and_bits(a, b):
    # None means the result cannot be represented as a constant or source
    # bit
    if a == 0:
        return 0
    if b == 0:
        return 0
    if a == 1:
        return b
    if b == 1:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a & a == a
        return a
    if a == invert_bit(b):
        # a & not a == 0
        return 0
    return None


def test_and_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert and_bits(a, 0) == and_bits(0, a) == 0
    assert and_bits(a, 1) == and_bits(1, a) == a
    assert and_bits(a, a) == a
    assert and_bits(a, invert_bit(a)) == 0

    # positions disagree
    assert and_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert and_bits(a, SourceBit(y, 3)) is None
    assert and_bits(a, SourceBit(y, 3, True)) is None

When and_bits returns None for output bit i, we can use SourceBit(result, i): we don't know how that bit relates to the inputs. The other output bits keep their own provenance, even if they refer to different operations.

Now we can combine and_bits calls for all the bits of a BitProvenance.

def test_and():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x & 0 == 0
    result = Operation("int_and", [x, Constant(0)])
    assert absx.abstract_and(BitProvenance.from_constant(0), result).get_constant() == 0

    # x & -1 == x
    result = Operation("int_and", [x, Constant(INT_MASK)])
    assert absx.abstract_and(BitProvenance.from_constant(INT_MASK), result) == absx

    # x & x == x
    result = Operation("int_and", [x, x])
    assert absx.abstract_and(absx, result) == absx

    # x & y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_and", [x, y])
    assert absx.abstract_and(absy, result) == BitProvenance.from_source(result)

    # x & ~x == 0
    result = Operation("int_and", [x, Operation("int_invert", [x])])
    assert absx.abstract_and(absx.abstract_invert(), result).get_constant() == 0


@given(ints, ints)
def test_hypothesis_and_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_and", [Constant(a), Constant(b)])
    assert abs_a.abstract_and(abs_b, result).get_constant() == a & b


class BitProvenance:
    ...

    def abstract_and(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = and_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

or is very similar:

def or_bits(a, b):
    if a == 1:
        return 1
    if b == 1:
        return 1
    if a == 0:
        return b
    if b == 0:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a | a == a
        return a
    if a == invert_bit(b):
        # a | not a == 1
        return 1
    return None

def test_or_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert or_bits(a, 1) == or_bits(1, a) == 1
    assert or_bits(a, 0) == or_bits(0, a) == a
    assert or_bits(a, a) == a
    assert or_bits(a, invert_bit(a)) == 1

    # positions disagree
    assert or_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert or_bits(a, SourceBit(y, 3)) is None
    assert or_bits(a, SourceBit(y, 3, True)) is None


def test_or():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x | 0 == x
    result = Operation("int_or", [x, Constant(0)])
    assert absx.abstract_or(BitProvenance.from_constant(0), result) == absx

    # x | -1 == -1
    result = Operation("int_or", [x, Constant(INT_MASK)])
    assert absx.abstract_or(BitProvenance.from_constant(INT_MASK), result).get_constant() == INT_MASK

    # x | x == x
    result = Operation("int_or", [x, x])
    assert absx.abstract_or(absx, result) == absx

    # x | y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_or", [x, y])
    assert absx.abstract_or(absy, result) == BitProvenance.from_source(result)

    # x | ~x == -1
    result = Operation("int_or", [x, Operation("int_invert", [x])])
    assert absx.abstract_or(absx.abstract_invert(), result).get_constant() == INT_MASK


@given(ints, ints)
def test_hypothesis_or_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_or", [Constant(a), Constant(b)])
    assert abs_a.abstract_or(abs_b, result).get_constant() == a | b


class BitProvenance:
    ...

    def abstract_or(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = or_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

And finally here is xor:

def xor_bits(a, b):
    if a == 1:
        # x ^ 1 == ~x
        return invert_bit(b)
    if b == 1:
        return invert_bit(a)
    if a == 0:
        # x ^ 0 == x
        return b
    if b == 0:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a ^ a == 0
        return 0
    if a == invert_bit(b):
        # a ^ not a == 1
        return 1
    return None

def test_xor_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert xor_bits(a, 1) == xor_bits(1, a) == invert_bit(a)
    assert xor_bits(a, 0) == xor_bits(0, a) == a
    assert xor_bits(a, a) == 0
    assert xor_bits(a, invert_bit(a)) == 1

    # positions disagree
    assert xor_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert xor_bits(a, SourceBit(y, 3)) is None
    assert xor_bits(a, SourceBit(y, 3, True)) is None


def test_xor():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x ^ 0 == x
    result = Operation("int_xor", [x, Constant(0)])
    assert absx.abstract_xor(BitProvenance.from_constant(0), result) == absx

    # x ^ -1 == ~x
    result = Operation("int_xor", [x, Constant(INT_MASK)])
    assert absx.abstract_xor(BitProvenance.from_constant(INT_MASK), result) == absx.abstract_invert()

    # x ^ x == 0
    result = Operation("int_xor", [x, x])
    assert absx.abstract_xor(absx, result).get_constant() == 0

    # x ^ y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_xor", [x, y])
    assert absx.abstract_xor(absy, result) == BitProvenance.from_source(result)

    # x ^ ~x == -1
    result = Operation("int_xor", [x, Operation("int_invert", [x])])
    assert absx.abstract_xor(absx.abstract_invert(), result).get_constant() == INT_MASK


@given(ints, ints)
def test_hypothesis_xor_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_xor", [Constant(a), Constant(b)])
    assert abs_a.abstract_xor(abs_b, result).get_constant() == a ^ b


class BitProvenance:
    ...

    def abstract_xor(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = xor_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

Extracting Simpler Operations

Now that we have the analysis, we need to find out how to use the information during optimization. We'll discuss four ways of doing that, from easy to more complicated.

Constants

The simplest thing to do is to exploit the fact that provenance can also identify constant results.

def test_constant_via_provenance():
    bb = Block()
    x = bb.getarg(0)
    a = bb.int_and(x, 0x0f)
    b = bb.int_or(x, 0xf0)
    c = bb.int_and(b, 0x0f)
    # we compute result = (x & 0x0f) ^ ((x | 0xf0) & 0x0f)
    # ((x | 0xf0) & 0x0f) can be rewritten to
    # (x & 0x0f) | (0xf0 & 0x0f)
    # which is x & 0x0f
    # this means result == (x & 0x0f) ^ (x & 0x0f) == 0
    result = bb.int_xor(a, c)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = dummy(0)"""

To make this test pass, we can do something very similar to what we did in the KnownBits post. We go along a basic block, apply the transfer functions, and if a result is constant we can replace the operation with that constant.

def unknown_transfer_functions(*abstract_args, result):
    return BitProvenance.from_source(result)

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Identity

The next optimization we can add is that sometimes complicated bit reshuffling leads to a result that was already computed earlier in the block. We can start with a simple test that splits a value into low and high bits and recombines them:

def test_identity_by_recombining_bits():
    # split and then recombine x
    # lo = x & 0x0f
    # hi = (x >> 4) << 4
    # result = lo | hi
    bb = Block()
    x = bb.getarg(0)
    lo = bb.int_and(x, 0x0f)
    shifted = bb.int_rshift(x, 4)
    hi = bb.int_lshift(shifted, 4)
    result = bb.int_or(lo, hi)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""

To help implement this, we can add a method to check whether a provenance is identical to that of an earlier result. The idea is that if the bits of a BitProvenance come from bits 0, 1, 2, ..., INTEGER_WIDTH - 1 of the same source, without inversion, then the operation we are looking at computes the same result as the source too.

def test_provenance_get_identical_source():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.make([1, 0, SourceBit(source, 1, True), SourceBit(source, 0)])
    assert x.get_identical_source() is None # not the same value as source

    # trivially, a BitProvenance where we know nothing is a valid source for itself
    res = BitProvenance.from_source(source)
    assert res.get_identical_source() is source


class BitProvenance:
    ...
    def get_identical_source(self) -> Operation | None:
        bit0 = self.bits[0]
        if isinstance(bit0, int):
            return None # a constant bit cannot match an unconstrained source bit
        source = bit0.source
        # are all the bits the same as in the unconstrained source?
        if self == BitProvenance.from_source(source):
            return source
        return None

Now we can use that to recognize identities when optimizing:

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        source = abstract_res.get_identical_source()
        if source is not None and source is not op:
            op.make_equal_to(source)
            continue
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Reusing Earlier Values

The motivating example from the start of the blog post is next. It doesn't quite work with the idea from the previous subsection. Here it is in test form:

def test_identity_via_provenance():
    bb = Block()
    x = bb.getarg(0)
    a = bb.int_and(x, 0x0f)
    b = bb.int_lshift(x, 60)
    c = bb.int_rshift(b, 60)
    # result is same as a, but the bits come from x
    result = bb.int_and(a, c)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 15)
optvar2 = dummy(optvar1)"""

We will use a hashmap to store all the provenance patterns we have seen in the basic block so far.

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance
    seen_provenances = {} # dict mapping BitProvenance to Value

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # check whether we've seen the provenance before, if yes, reuse the op
        source = seen_provenances.get(abstract_res)
        if source is not None:
            op.make_equal_to(source)
            continue
        seen_provenances[abstract_res] = op
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Simple Shifts

In addition to using existing values, we can sometimes emit a new shift operation of some earlier value, to get fewer operations than how the value was actually computed. Here's an example:

def test_extract_rshift_from_source():
    bb = Block()
    x = bb.getarg(0)
    # lo = (x & 0xf)
    lo = bb.int_and(x, 0x0f)
    # cleared = x ^ (x & 0xf)
    cleared = bb.int_xor(x, lo)
    # result = cleared >> 4
    result = bb.int_rshift(cleared, 4)
    bb.dummy(result)

    # the xor clears only the bits that the shift discards
    # the result's provenance matches a direct x >> 4
    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_rshift(optvar0, 4)
optvar2 = dummy(optvar1)"""

To implement this, we add a new method BitProvenance.recognize which detects whether a BitProvenance instance could be represented as a shift of the source.

def test_recognize_rshift():
    x = Operation("getarg", [Constant(0)])
    # low bits come from x[4], x[5], ..., x[63]; the high four are zero
    bits = tuple(SourceBit(x, i) for i in range(4, INTEGER_WIDTH))
    shifted = BitProvenance.make(bits + (0, 0, 0, 0))
    assert shifted.recognize() == ("int_rshift", x, 4)

    # the same alignment is not enough if a high bit is one
    not_a_shift = BitProvenance.make(bits + (0, 0, 0, 1))
    assert not_a_shift.recognize() is None

    # the first bit suggests >> 4, but the next bit comes from x[6], not x[5]
    misplaced = bits[:1] + (SourceBit(x, 6),) + bits[2:]
    not_a_shift = BitProvenance.make(misplaced + (0, 0, 0, 0))
    assert not_a_shift.recognize() is None


@given(strategies.integers(1, 63))
def test_hypothesis_recognize_shift(shift):
    x = Operation("getarg", [Constant(0)])
    absval = BitProvenance.from_source(x)
    shift_amount = BitProvenance.from_constant(shift)
    # check that lshifts are recognized
    result = Operation("int_lshift", [x, Constant(shift)])
    res = absval.abstract_lshift(shift_amount, result).recognize()
    assert res == ("int_lshift", x, shift)

    # check that rshifts are recognized
    result = Operation("int_rshift", [x, Constant(shift)])
    res = absval.abstract_rshift(shift_amount, result).recognize()
    assert res == ("int_rshift", x, shift)

class BitProvenance:
    ...
    def recognize(self):
        # return None or a triple (opname, source, shift amount), where
        # opname is "int_rshift" or "int_lshift"
        bits = [(i, b) for i, b in enumerate(self.bits) if isinstance(b, SourceBit)]
        if not bits:
            return None
        # try to guess the shift amount
        i, b = bits[0]
        delta = i - b.index
        source = b.source
        original = BitProvenance.from_source(source)
        # use the transfer functions to check whether it really is that shift
        if delta >= 0:
            expected = original.abstract_lshift(BitProvenance.from_constant(delta), None)
            # does the shift of the source match self?
            if self == expected:
                return ("int_lshift", source, delta)
        else:
            expected = original.abstract_rshift(BitProvenance.from_constant(-delta), None)
            # does the shift of the source match self?
            if self == expected:
                return ("int_rshift", source, -delta)
        return None

Using the abstract transfer functions in recognize has the nice effect that we only specify what lshift and rshift do in a single place.

Now we can start using this in simplify:

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance
    seen_provenances = {} # dict mapping BitProvenance to Value

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # check whether we've seen the provenance before, if yes, reuse the op
        source = seen_provenances.get(abstract_res)
        if source is not None:
            op.make_equal_to(source)
            continue

        # now try to synthesize a shift with the same provenance
        pattern = abstract_res.recognize()
        if pattern is not None:
            name, source_op, shift_amount = pattern
            if source_op is not op:
                assert shift_amount > 0 # otherwise one of the earlier rules would have caught it
                new_op = getattr(opt_bb, name)(source_op, Constant(shift_amount))
                abstract_values[new_op] = abstract_res
                seen_provenances[abstract_res] = new_op
                op.make_equal_to(new_op)
                continue
        # otherwise emit the op
        seen_provenances[abstract_res] = op
        opt_bb.append(op)
    return opt_bb

That's the whole optimizer. We propagate where each bit comes from, then look for a cheaper way to produce the resulting provenance. Constant folding, reuse of earlier values, and shift recognition all use the same information. None of them needs to know which sequence of operations produced it.

More Rewrites from the Same Rules

Now we can look at a few other rewrites that fall out of this design. Here are some further passing tests:

def test_xor_masks_cancel():
    bb = Block()
    x = bb.getarg(0)
    # each low-byte bit is flipped twice: 0x55 ^ 0xff ^ 0xaa == 0
    # therefore (((x ^ 0x55) ^ 0xff) ^ 0xaa) == x
    # by constant reassociation
    a = bb.int_xor(x, 0x55)
    b = bb.int_xor(a, 0xff)
    result = bb.int_xor(b, 0xaa)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""


def test_recover_low_bits_after_packing_two_sources():
    bb = Block()
    x = bb.getarg(0)
    y = bb.getarg(1)
    lo = bb.int_and(x, 0xff)
    hi = bb.int_lshift(y, 8)
    packed = bb.int_or(lo, hi)
    # masking the packed value recovers the earlier low byte of x
    result = bb.int_and(packed, 0xff)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 255)
optvar2 = dummy(optvar1)"""


def test_reuse_mask_via_de_morgan():
    bb = Block()
    x = bb.getarg(0)
    masked = bb.int_and(x, 0x0f)
    inverted = bb.int_invert(x)
    high_ones = bb.int_or(inverted, INT_MASK ^ 0x0f)
    # ~(~x | ~0x0f) == x & 0x0f, which we already computed
    result = bb.int_invert(high_ones)
    bb.dummy(masked, result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 15)
optvar2 = dummy(optvar1, optvar1)"""

def test_byte_swap_twice():
    def byte_swap(bb, value):
        result = 0
        for byte_index in range(8):
            byte = bb.int_and(bb.int_rshift(value, byte_index * 8), 0xff)
            moved = bb.int_lshift(byte, (7 - byte_index) * 8)
            result = bb.int_or(result, moved)
        return result

    bb = Block()
    x = bb.getarg(0)
    # little-endian host to network byte order, then back to host order
    network_order = byte_swap(bb, x)
    result = byte_swap(bb, network_order)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""

Here's an even more complicated test. It reverses the bits of the argument, then shifts left by 7, then reverses the bits of the result again. The input block has 63 operations, and dce(simplify(bb)) reduces it to just three. The bit reversal below is adapted from Hacker's Delight, section 7-1.

def test_reverse_lshift_reverse_becomes_rshift():
    # Hacker's Delight, section 7-1: swap adjacent bits, then pairs,
    # nibbles, bytes, and larger fields. adapted from 32 to 64 bits
    def reverse_bits(bb, value):
        for shift, mask in [
            (1,  0x5555555555555555),
            (2,  0x3333333333333333),
            (4,  0x0f0f0f0f0f0f0f0f),
            (8,  0x00ff00ff00ff00ff),
            (16, 0x0000ffff0000ffff),
            (32, 0x00000000ffffffff),
        ]:
            # value = ((value & mask) << shift) | ((value >> shift) & mask)
            left = bb.int_lshift(bb.int_and(value, mask), shift)
            right = bb.int_and(bb.int_rshift(value, shift), mask)
            value = bb.int_or(left, right)
        return value

    bb = Block()
    x = bb.getarg(0)
    reversed_once = reverse_bits(bb, x)
    shifted = bb.int_lshift(reversed_once, 7)
    result = reverse_bits(bb, shifted)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    # reversing before and after a left shift turns it into a right shift
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_rshift(optvar0, 7)
optvar2 = dummy(optvar1)"""

What I like about all of these is how quite varied operation rewrites just fall out from the bit provenance approach, without needing complicated peephole rewrite patterns.

Evaluation with Pydrofoil and PyPy

To evaluate the bit provenance optimizer, I wrote a program that applies it to PyPy JIT traces. I didn't fully build it into the PyPy JIT because I was quite skeptical that it would help. Instead, I wrote an external optimization pass that parses RPython JIT logs, applies the optimization and then produces statistics. This uses an extended prototype that supports additional operations such as arithmetic right shifts and sign extension, and can extract shifts of earlier computed values as well as shifts of the original source.

I did this experiment for two different RPython projects, for PyPy (running the translator, that bootstraps PyPy to C code) and Pydrofoil (RISC-V simulation based on Sail, booting the Linux kernel). I picked the latter because Pydrofoil does a lot of integer operations, whereas most Python code does very little.

For both of these programs I wanted to know how well the bit provenance optimizer can optimize the traces compared to the integer optimizations of the PyPy JIT. I took the unoptimized traces, removed everything but integer operations and performed dead code elimination. To do that, the integer results of non-pure operations are turned into unconstrained arguments of the trace. Integer values used by removed operations are retained as outputs. The remaining integer operations are our baseline traces for this experiment.

The following table shows the results. The first column shows the number of unoptimized integer operations. The second column shows how many operations are left after running just bit provenance optimizations. The third column shows how many operations are left after the RPython JIT integer optimizations run. The last column shows how many operations are left after we run first the RPython JIT optimizer and then bit provenance optimizations. All three paths include dead code elimination after optimization.

In parentheses are the percentages of how many of the unoptimized operations are removed by the corresponding path.

The 'RPython then Provenance' column shows that, for the PyPy workload, provenance removes no additional operations after the RPython integer optimization passes in this experiment. For Pydrofoil it does remove more operations, but we will have to investigate whether this benefit carries over to the full JIT optimization pipeline.

However, the comparison between the 'Provenance' and 'RPython' column for Pydrofoil shows that the design of the optimizer is quite good! The small provenance optimizer achieves nearly as much reduction in integer operation count as the complicated and well-tuned integer optimization passes of the RPython JIT.

Related work

LLVM has a very similar bit provenance tracking optimization as part of the Hexagon backend. It also tracks 0 bits, 1 bits or references to specific bits from other SSA operations (it doesn't track negation). It is split into the BitTracker analysis and the HexagonBitSimplify optimization pass.

LLVM also has collectBitParts (source). Rather than running a separate forward analysis, it recursively walks SSA operations following provenance (with memoization and limited recursion depth for performance). Then there is a check whether the resulting provenance describes a byte swap or bit reversal.

In GCC this tracking happens on a byte level to be able to recognize byte swaps in the byte-swap recognizer. It also tracks memory addresses and offsets to recognize wider loads assembled from separate byte loads, possibly with a byte swap. However, there's also a recent (as of writing unmerged) patch series to extend this to bit granularity.

Conclusion

The interesting result here isn't that we beat an existing optimizer. It's that a handful of transfer functions lets us discover rewrites across arbitrarily long chains of bit operations, without writing a rule for each combination. The evaluation suggests that existing integer optimizations already cover much of this stuff. However, bit provenance gives us a compact way to describe a whole family of simplifications. It's quite cool to write an optimizer that can undo a bit reversal without ever knowing what a bit reversal is.

The entire code can be found in this gist.

Acknowledgements

Thanks to Joe Groff for the fantastic title. Thanks to Can Joshua Lehmann, Andy Wingo, Matti Picus, and Nico Rittinghaus for feedback on the draft, and to Max Bernstein for encouragement.

October 27, 2026 06:04 AM UTC

Graham Dumpleton

AI slop is the new TLDR

It is hard to get through a day online at the moment without someone dismissing something as "AI slop". A pull request, a library, a whole project, all written off with two words. When the thing being dismissed is code, I have started to notice that the label usually arrives well before anyone could have actually read the code in question.

That has led me to a slightly uncomfortable conclusion. In a lot of cases, calling AI generated code slop is just a new way of saying TLDR (too long, didn't read). It isn't really a measure of whether the code is good or bad. It says more about the willingness of the person saying it to sit down and learn what the code does.

There is even slang emerging for the general version of this. Posts that are obviously AI generated are now getting the response AI;DR, as in "AI, didn't read". At least that is honest about what is going on. Slop dresses the same reaction up as a judgement of quality.

When slop is a fair call

I should say up front that some AI generated code really is slop. Open source maintainers in particular are being buried under pull requests and bug reports where the person submitting them clearly has no idea what they are submitting. Generating that stuff costs the submitter almost nothing, while reviewing it costs the maintainer real time, and that imbalance is a legitimate thing to be angry about.

Look closely at those cases though and the failure is the same one. The person who generated the code didn't read it either. They asked for something, got something back, and passed it on without understanding it. So the TLDR problem exists on both sides. There is the person who produces code without reading it, and there is the person who dismisses code without reading it. Neither has actually engaged with what is in front of them.

One working definition I came across describes slop as code you can't work with, that is, a codebase that nobody understands well enough to work with effectively. I think that is about right, but notice what the definition hinges on. It is about understanding, and understanding is not purely a property of the code. It also depends on whether anyone has put in the effort to acquire it.

A simple test I now apply to myself is to ask whether I would have said the same thing if a new colleague had written the code. If the answer is that I would have sat down, read through it, and given proper feedback, then reaching for "slop" instead is a sign I am taking a shortcut.

Every day is onboarding day

So why would people take that shortcut? I don't think it is simply laziness, or at least not only laziness. I think it is fatigue, and it is worth being honest about where that fatigue comes from.

Think back to the first few weeks of starting at a new organisation. You arrive and there is an existing body of software that you didn't write and don't understand. Before you can do anything meaningful you have to learn what it does, then how it does it, and then why it was built that way. It is a steep learning curve for anything you didn't already know, and it is mentally exhausting. Most people I know come home from their first week in a new job completely wiped out.

Working with AI generated code can feel a lot like that, except it doesn't stop after a few weeks. Every day there is another pile of code you didn't write, built on decisions you didn't personally make. In some respects every day becomes your first day.

It has always been harder to read code than to write it. When you write code yourself, the understanding builds up as you go, almost as a side effect. When you read someone else's code, you have to reconstruct that understanding after the fact, from whatever clues they left behind. With AI doing much of the writing, more of the job has shifted to the harder of those two activities.

There is also an old observation from the study of automation that applies here. When you automate the routine part of a job, the human is left with the part that is harder, which is monitoring what the automation is doing and stepping in when it goes wrong. And because the human is no longer doing the routine work, the hands-on skill needed to step in can quietly fade away. Supervising an AI that writes code is a textbook case of this.

The gap that builds up has been given a name, comprehension debt, meaning the growing difference between how much code exists in a system and how much of it any human actually understands. Like any debt it accumulates quietly, until the day something breaks and nobody knows where to start.

Faced with all that, it is much easier to throw your hands up and declare the whole thing slop than to admit that you can't keep up, or that you haven't been willing to put in the time to learn it properly. I get the temptation. I just don't think it is an honest answer.

Agentic engineering and the conductor

What all this points to is that what it means to be a software developer has changed. The term that seems to be sticking for the new way of working is agentic engineering, the idea being that you are mostly no longer writing the code directly, but orchestrating AI agents that do, and providing the oversight.

The analogy I keep coming back to is a conductor of an orchestra. A conductor isn't playing every instrument, but no one would suggest a conductor can get away without a deep grounding in music theory. They have to read the full score, know what each instrument is capable of, hear when a section is out of tune, and know how to fix it. For a developer working with AI, the equivalent of music theory is the fundamentals of software development, architecture and design. You may not be writing every line, but you still need to know what good looks like and recognise when something is wrong.

The analogy does break down in one telling way though. A real orchestra rehearses the same piece over and over before a performance, and the conductor builds up an intimate knowledge of it. With AI the orchestra is effectively playing a new piece every day, with no rehearsal. Which brings us right back to the onboarding problem.

It is also worth noting that reviewing and steering code you didn't write was always part of the job of a senior or lead engineer. People generally grew into that role over many years. What has changed is that everyone is now being put in that position, from day one, often without the experience to back it up. That raises an awkward question about how people new to the industry are meant to learn the music theory if they never get to play an instrument.

I struggle with this too

I don't want to give the impression I am lecturing from on high here. I now lean on AI quite heavily for some new projects, and I struggle to keep up with the constant flow of new code and new information as much as anyone.

I have tried different approaches. For a while I used Obsidian as a knowledge base, a sort of second brain, to track design decisions and how things fitted together. For some projects that worked well. For others I got so deep into the design, and then into directing the AI to write the code, that going back to get it all into Obsidian felt like more work than it was worth. The end result was that I was back to flying by the seat of my pants and trying to keep it all in my head.

Looking back, I think the reason it failed is that the note taking was a separate chore. Anything that sits outside the actual work is the first thing to get dropped when you are in the flow of things. Whatever approach works has to be a by-product of something you would be doing anyway.

Having the AI teach me what it built

The approach that has worked best for me so far, at least for projects that I expect other people to use, is to have the AI generate workshops.

The process goes something like this. I have the AI build the software, and keep directing it until what it has produced matches the idea I had in my head. At that point I have it generate a set of hands-on workshops teaching how to use it, which run on my Educates training platform or in the JupyterLab workshop extension I wrote. Then I sit down and work through those workshops myself.

In other words, I deliberately put myself through the new employee onboarding. Rather than trying to absorb the code in bits and pieces as it flies past, I get a structured path through what the software does and how to use it, at a pace where I can actually take it in.

What I didn't expect was how good this is at exposing problems. When you work through a workshop as a user, you very quickly see where the design falls short, where the experience is clumsy, or where things just don't work. The AI also contributes here. If, when generating the workshop, I tell it to highlight problems, it will report when it couldn't get something to work, when it ran into what looks like a bug, or when the documentation doesn't match what the code actually does. So the AI generating the workshop is acting as a QA and user experience tester, and so am I when I work through it.

Because the workshops are interactive and the steps are actually executed, they also behave a bit like tests. If a step in the workshop fails, that is a real problem with the software, not just a typo in some documentation nobody will read.

And then there is the secondary benefit, which is that the workshops exist afterwards for other people. A number of them are available in the workshops area of this site. My hope is that they can also encourage people to get past the mental block of TLDR, or AI slop, or whatever they want to call it, and be prepared to sit down and learn something properly. That seems more important than ever at a time when most people will feel it is easier to just ask an AI to do it for them.

Still working it out

I still don't have a workflow I would call good for using AI and keeping on top of what it is doing. I regard myself as fairly conservative when it comes to tools. I don't go chasing every new fad, because to be frank that usually ends up being a complete waste of time. I suspect most people are similar, and adapting is a slow, incremental process of personal experimentation to see what works for them.

The AI tools themselves, including the tooling from the AI vendors, have certainly made it easier to do a lot of this without reaching for other tools. It is still a matter of finding what works for you, though, and right now things are moving too quickly for anything to settle.

One principle I am trying to hold myself to in the meantime is to not let the AI get too far ahead of my own understanding. A new employee is given time to ramp up, and nobody expects them to be onboarding forever. If the code is arriving faster than I can understand it, the answer is to slow down and make time to learn it, not to stop reading and call it slop.

Hopefully one day I will have it sorted out. Fingers crossed.

October 27, 2026 12:00 AM UTC

October 26, 2026

"Michiel's Blog"

Python on my wrist

I have a PineTime. It is a cheap, open, hackable smartwatch from Pine64.

It’s not an Apple Watch

When people see it, they think: that looks like a smartwatch, so it must be a smartwatch. Like an Apple Watch. It’s not! An Apple Watch costs around 700 USD, and this one 25. Think of it as a programmable Casio with a color display, Bluetooth, a step counter and a heart rate monitor.

It really does not have much CPU or memory: a 64 MHz ARM Cortex-M4, 64 KB of RAM (yes, kilobytes), 512 KB of flash for the firmware and 4 MB of external flash for files. After the Bluetooth stack and MicroPython itself have had their share, about 12 KB of RAM is left for the apps. Everything on this watch has to be small.

The screen is fine, though: 240×240 pixels on 1.3 inches, about 260 pixels per inch. So if anything on it looks blocky in this post, that’s on purpose. However, you can’t have an always-on display: the battery would run out too soon. Apple Watches once had the same limitation, too!

The watch features an app for aforementioned watch monitor and step counter, and you can send notifications to it via Bluetooth.

How I got here

I bought my PineTime in March 2023, for 26.99 USD. It ships with InfiniTime. That’s fine. But InfiniTime is written in C++, and I run a Python group. So I preferred wasp-os, a MicroPython-based watch OS. However, even in 2023 that project was already quite dormant. It was running a 2019 version of MicroPython and was quite hard to build. Recently its maintainer added a note that wasp-os is looking for new maintainers.

Reviving the OS

Then I found my watch in a drawer again, and thought it would be nice to work on it a bit. So I took it upon myself to get wasp-os into shape again. Besides moving to mainline MicroPython (more on that below), this is what changed:

• It builds again. The Python tooling is managed with uv, so setting up is one command, and the full build works again, including the bootloader bits.
• Faster flashing. Updating the firmware over Bluetooth now uses bleak instead of the deprecated gatttool, and takes about three and a half minutes instead of over eight. When it fails, it tells you why, instead of hiding the errors, as it did before.
• More memory. The filesystem on the external flash now uses the chip’s real 4 KB blocks, which got rid of a permanent 4 KB buffer. Free RAM at boot went from 7.1 KB to 12.4 KB. On this watch, that’s a lot!
• It recovers by itself. A watchdog now resets the watch when it hangs, instead of leaving it dead with Bluetooth still up.
• No more buzzing. A swipe could leave the vibration motor buzzing until you reset the watch. That’s fixed, and the motor is driven right now.
• Swipes work reliably. The touch controller now reports when you lift your finger, and in Settings you can swipe along a slider to change it.
• It survives a flat battery. After my holiday, where I did not bring the watch, I put it on the charger, it booted and the watch said INIT HARDWARE FAILED. A chip had kept the I2C bus stuck. Now it unsticks the bus at boot, shows the actual error if something still fails, and the clock starts at the build date instead of somewhere in 2020.

Mainline MicroPython

wasp-os ran on a fork of MicroPython from 2019, with patches for the watch. Porting it to mainline MicroPython also took a few changes in MicroPython itself, so I submitted three pull requests upstream: frozen native code, a fix for PWM allocation and an ADC sample time option. All three got merged, so for the first time wasp-os runs on mainline MicroPython, and an up-to-date version too. No fork needed! The changes are targeted for MicroPython 1.30, which is planned for December.

Two new watch faces

Of course I wanted my own watch faces. The first one is simple: the existing week clock, but in Dutch, because I’m Dutch. Dutch abbreviates weekdays with two letters, so it says ma 26 okt 2026. Remember that the memory on this watch is tiny, so there’s no room for a full multilingual OS. But you can just translate a watch face, build it, and put your localized version on the watch. You’ll see a picture of it on top of this post.

The second one is more fun: an 8-bit style clock. The digits come from a tiny 6×7 pixel font, and every font pixel is drawn as a block of 7×7 screen pixels for the time and 3×3 for the rest. That gives the retro look, and it needs no font data at all. Below the time and the ISO 8601 date (YYYY-MM-DD, obviously), it shows the Unix epoch, counting every second.

The Guido app

And then, as a finishing touch and to emphasize the Python angle, I created the Guido app. It shows a blocky portrait of Guido van Rossum, the creator of Python. Swipe, and you get the Python logo, then the MicroPython logo, and then the logo of our own Python Leiden meetup. One more swipe and you’re back at the clock.

The portrait is 80×80 pixels in 16 colors, drawn three times larger. That fills the 240×240 screen and gives it a nice retro look. The logos are drawn sharp, at full resolution.

Flash is tight on this watch. The firmware has about 330 KB to work with, and a single full-screen picture as raw pixels takes 112 KB, a third of that. So I made a small image format of up to 16 colors per picture plus run-length encoding, with a decoder of about 40 lines that draws them straight to the display. All four pictures together now take under 12 KB. Still, that’s too much for something this silly to include by default, so the app is optional: you have to add it to your build yourself.

The original photo is by Kushal Das, taken at PyCon US 2024 and licensed CC BY-SA 4.0, so the blocky version is shared under the same license.

Coming up: the PineTime Pro

Pine64 is working on a successor: the PineTime Pro. It gets a 410×502 AMOLED screen, GPS, a dual-core Cortex-M33 and a digital crown. It’s still early, there’s no release date yet, and the original PineTime keeps being made. But that would be a much bigger playground for MicroPython. Guido in AMOLED!

Try it

Everything is in the pinetime-upstream-micropython branch of my wasp-os fork.

Discussion on Hacker News

October 26, 2026 10:00 PM UTC

HoloViz

Lumen 1.4.0 Release

October 26, 2026 07:39 PM UTC

Django Weblog

Natalia Bidart - Planned short-term leave

After more than three years as a Django Fellow, I will be taking a three-month break from Fellow duties during November, December, and January. This time will allow me to properly rest, reset, and recover from some accumulated fatigue so I can return with renewed energy.

I'm deeply grateful to the DSF Board for its support of this arrangement, and to my Django Fellow colleagues and the members of the various teams and working groups who will help provide continuity during my leave.

I look forward to resuming my Fellow duties in February. Super excited to be release manager for Django 2028!

October 26, 2026 05:00 PM UTC

Rodrigo Girão Serrão

TypedDict with extra items

Learn how to define a TypedDict that accepts extra items of a given type.

A TypedDict is a type constructor that allows creating specialised types for dictionaries that have a fixed set of string keys:

from typing import TypedDict

class Movie(TypedDict):
    title: str
    year: int

The snippet above creates a new type Movie that represents dictionaries with the key "title" and a string value, and the key "year" and an integer value:

forrest_gump = {"title": "Forrest Gump", "year": 1994}

In Python 3.15, TypedDict was improved to allow specifying whether or not the dictionary may have other keys and what must be the type of those other items when they're allowed.

15 days of Python 3.15

To celebrate the release of Python 3.15, over the next 15 business days I'll be writing about a new 3.15 feature every day. Explained clearly and with examples so you don't have to sift through the changelog.

Subscribe to receive this free email series:

If you want, you can also check the schedule of the upcoming emails.

Allowing extra items of a given type

When creating a TypedDict, you may specify the class parameter extra_items. When extra_items is set, your TypedDict may have additional keys of the type specified:

from typing import TypedDict

class Movie(TypedDict, extra_items=str):
    title: str
    year: int

This snippet of code creates a TypedDict with two keys that allows extra string items, as specified by extra_items=str:

# ✅ Type checks
m1: Movie = {"title": "Forrest Gump", "year": 1994, "genre": "Comedy/Romance"}

The assignment to m1 is type safe. While the key "genre" is not specified in the definition of Movie, the corresponding item is a...

October 26, 2026 04:53 PM UTC

The Python Coding Stack

A New Price Without Breaking the Old One • The O in SOLID

The potion shop has a special price for apprentices. They receive ten per cent off a potion’s full price, including any rare-ingredient surcharge.

We’ve visited this shop before. In The Potion Shop That Does Too Much • The S in SOLID, its PotionShop class handled several jobs, from checking stock and recording sales to sending messages by owl. But you don’t need the code in that article to follow along in this one.

This time, we’ll stay at the same shop and focus on one part of its work: pricing. Before adding anything new, let’s look at the apprentice rule the shop already has.

For a potion with a base price of £100 and a £20 surcharge, the calculation is:

£100 + £20 = £120
10% off £120 = £108

Let’s look at how this is implemented.

Start with the Apprentice Price

The shop’s PotionPricing class knows how to calculate the apprentice price. Its .price() method accepts a customer type because that is part of each quote. But apprentices are the only special-price customers the shop supports so far.

You can follow along by putting this code in pricing.py:

All code blocks are available in text format at the end of this article • #1

You’ll get this output when you run the code:

108.00

The apprentice discount applies after adding the surcharge. That’s the rule for apprentice discounts.

A Few Months Later…

A few months pass. A different programmer is asked to add a price for members of the local guild. Guild members should receive five per cent off the base price, while the surcharge stays fixed. For the same potion, their price should be £115.

The new programmer sees that both rules involve a percentage discount and decides to avoid repeating logic by using one shared calculation.

Add the Guild Price to the Same Class

He adds the guild rule to PotionPricing and generalises the calculation:

#2

Run pricing.py again:

110.00
115.00

The guild price is right, but what happened to the apprentice price? It was £108. Now it is £110.

Apprentices are getting a worse deal now, and it’s the fault of the new programmer. But should the original programmer share some of the blame, too? Yes. Let’s see why.

The new calculation discounts only the £100 base price, then adds the £20 surcharge. That is the guild’s rule, not the apprentice rule.

The new programmer did not have to make this mistake. A careful change could have preserved the £108 price.

But adding the guild pricing meant making changes to code that already produced the correct calculation for the apprentice price. Because both rules were handled in one place, the change had an opportunity to disturb behaviour the shop had been getting right for months.

We could correct the if branches and get both prices right. But each new customer rule would still send us back into PotionPricing, where a future change could affect behaviour that already works.

Modifying existing code always carries risks. We should avoid this whenever possible. But we don’t want to prevent extensions to the code. This is the key to what we’re trying to achieve here. And that’s where the Open-Closed Principle, or OCP, comes in. This concept gives us a safer way to extend code without modifying what already works, as you’ll see in the next section.

Subscribe now

Let Each Pricing Rule Handle Its Own Price

Instead of asking one class to choose a rule, we can represent each rule separately. First, make an ApprenticePricing class and a quote() function that delegates to whichever pricing object it receives:

#3

Run pricing.py:

108.00

The apprentice price is back to £108. The quote() function does not need to know how the policy calculates its price. It only needs an object with a .price() method.

Now add a class for the guild rule. Leave ApprenticePricing and quote() as they are, and add GuildPricing:

#4

Run the file once more:

108.00
115.00

The first price is still £108, and the guild price is £115. Adding the guild rule did not require us to change the apprentice calculation or the quote() function. We added a new policy that works with the existing quote() interface.

This is the safety gain here. Adding guild pricing does not send the programmer back to edit the code that produces the apprentice price. The new policy could still contain a mistake, but it is much harder for that mistake to change the established apprentice calculation by accident.

Where OCP Appears in This Example

The Open–Closed Principle says that software should be open to extension but closed to modification. Here, quote() is open to different pricing policies: it can use any object with the right .price() method. When we add guild pricing, we extend the program with GuildPricing instead of modifying ApprenticePricing or teaching quote() about a new customer type.

This is one way to apply OCP. The policy objects use a strategy-style design with Python’s duck typing. They do not need a shared parent class.

However, it’s possible to use abstract base classes or protocols to implement code using the OCP, too. Read And Now You Know Your ABC for an example of abstract base classes. For protocols, see the Python documentation for `typing.Protocol`.

Classes are not required by OCP, and separating responsibilities alone does not make a design open for extension. The important part is the boundary that lets us add a rule without changing the stable code that already handles another rule.

That boundary does not make bugs impossible. GuildPricing could still calculate the wrong amount, or a caller could choose the wrong policy. OCP narrows which existing code needs to change when behaviour is added. It does not guarantee every new change is correct.

This extra structure is useful when pricing rules really do vary and new ones are likely. If the shop is unlikely to add more tiers, the separate classes may cost more to understand than they save. The useful question is not “How can we prevent all modification?” It is “Which part is likely to vary, and can we add that variation without disturbing behaviour that already works?”

This is the second article in a series on the SOLID principles. I’ll look at what each principle means, but more importantly, why it matters and what can go wrong when we ignore it.

Code in this article uses Python 3.15.

Do you want to keep up with how Python programming and the art of making things are changing in this rapidly changing world? Then don’t miss out on the exclusive content for premium members here on The Python Coding Stack.

Subscribe now

I’ll be sharing more about how my programming, my daily work, and how everything has changed over the recent months.

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com.

Appendix: Code Blocks

Code Block #1

from decimal import Decimal

class PotionPricing:
    def price(self, customer_type, base_price, surcharge):
        if customer_type == "apprentice":
            subtotal = base_price + surcharge
            return (subtotal * Decimal("0.90")).quantize(Decimal("0.01"))
        raise ValueError(f"Unknown customer type: {customer_type}")

base_price = Decimal("100.00")
surcharge = Decimal("20.00")
pricing = PotionPricing()
print(pricing.price("apprentice", base_price, surcharge))

Code Block #2

from decimal import Decimal

class PotionPricing:
    def price(self, customer_type, base_price, surcharge):
        if customer_type == "apprentice":
            discount = Decimal("0.10")
        elif customer_type == "guild":
            discount = Decimal("0.05")
        else:
            raise ValueError(f"Unknown customer type: {customer_type}")

        discounted_base = base_price * (Decimal("1.00") - discount)
        return (discounted_base + surcharge).quantize(Decimal("0.01"))

base_price = Decimal("100.00")
surcharge = Decimal("20.00")
pricing = PotionPricing()
print(pricing.price("apprentice", base_price, surcharge))
print(pricing.price("guild", base_price, surcharge))

Code Block #3

from decimal import Decimal

class ApprenticePricing:
    def price(self, base_price, surcharge):
        subtotal = base_price + surcharge
        return (subtotal * Decimal("0.90")).quantize(Decimal("0.01"))

def quote(base_price, surcharge, policy):
    return policy.price(base_price, surcharge)

base_price = Decimal("100.00")
surcharge = Decimal("20.00")
apprentice_pricing = ApprenticePricing()
print(quote(base_price, surcharge, apprentice_pricing))

Code Block #4

from decimal import Decimal

class ApprenticePricing:
    def price(self, base_price, surcharge):
        subtotal = base_price + surcharge
        return (subtotal * Decimal("0.90")).quantize(Decimal("0.01"))

class GuildPricing:
    def price(self, base_price, surcharge):
        discounted_base = base_price * Decimal("0.95")
        return (discounted_base + surcharge).quantize(Decimal("0.01"))

def quote(base_price, surcharge, policy):
    return policy.price(base_price, surcharge)

base_price = Decimal("100.00")
surcharge = Decimal("20.00")
apprentice_pricing = ApprenticePricing()
guild_pricing = GuildPricing()
print(quote(base_price, surcharge, apprentice_pricing))
print(quote(base_price, surcharge, guild_pricing))

stephengruppetta.com

October 26, 2026 10:40 AM UTC

Python Insider

Python 3.15.2, 3.14.10, 3.13.18, 3.12.16 and 3.11.18 are out

Five expedited security releases to distrust the compromised eMudhra intermediate CA keys.

October 26, 2026 12:00 AM UTC

Stein Magnus Jodal

Introduction to barcode parsing

Barcodes are everywhere.

They’re so omnipresent that we easily take them for granted, and rarely think about their existence and how they really work. Meanwhile, barcodes are extremely important for all kinds of logistics, the foundation of modern society.

This post is an introduction to barcode parsing, and to all the corner cases that make it a bad idea to write your own barcode parser.

October 26, 2026 12:00 AM UTC

October 25, 2026

The No Title® Tech Blog

Just updated - Optimize Images v2.2.0

Optimize Images 2.2.0 has just been released, and it makes JPEG files even smaller by default, adds an optional SSIM-based quality check, and now keeps embedded colour profiles such as the Display P3 profile of iPhone photos. It also requires Pillow 13.0.0 for important security fixes, and makes batches and watch mode more reliable.

October 25, 2026 09:40 PM UTC

October 24, 2026

Go Deh

Finding Flaws in Technical Documentation

An idea

I spent some time on a project and had finished updating the tutorials and reference documentation I'd created as Markdown .md files. Then I suddenly thought:

Why not use the AI to test the documentation?

So I asked ChatGPT if it could create an isolated new chat that read just the project's .md documentation files and then tried to use the tool.

Putting it to the test

My current chat session said it could, created a new chat, and expanded my sentence into an initial prompt that included a lot more detail on what to do and what it could not do.

It came back to me and said it would run and monitor the new chat and report back on any documentation deficiencies it found.

That led to several points, some valid and some not, that I then addressed.

One example was that the documentation didn't explicitly state that the tool was intended for Linux use. It included setup details for Linux and WSL, but that hadn't stopped the AI from trying to work out how to run it in a Windows environment.

An obvious assumption on my part, perhaps, but one worth making explicit.

A new, (to me), way to review documentation

I like the idea of getting AI to act as someone new to a project, trying to get something done and reporting on any problems encountered along the way.

It's a way of testing the documentation by actually trying to follow it, rather than just asking AI to review the wording.

Of course, the AI won't always get it right. Some of the points it raised weren't valid, so you still need to use your own judgement when deciding what to change.

Documentation for humans and AI

You still have to make documentation approachable for any user, without requiring them to use AI. But I'd expect more of my target audience to use AI to help them work with projects.

That means documentation has to serve two purposes: it needs to be usable by people reading it directly, but also clear enough for an AI assistant to use when helping someone get things done.

I'm not suggesting we should write documentation just for AI. But if AI is going to be another way people interact with our projects, it makes sense to consider how well our docs support that.

Perhaps we should be testing documentation in much the same way we test code: by seeing whether someone, or something, can actually follow it and accomplish the intended task.

Something worth trying on your next project?

October 24, 2026 01:59 PM UTC

LernerPython blog, from Reuven Lerner

Pandas MultiIndex: Select rows by the outer level with loc

To retrieve all rows from a multi-indexed Python Pandas data frame or series based on the outermost index, use “loc”:

(
  pd
 .read_parquet('/Users/reuven/Data/nyc_taxi_2025-11.parquet')
 .groupby(['passenger_count', 'VendorID'])['total_amount'].mean()
 .loc[1.0]
)

The post Pandas MultiIndex: Select rows by the outer level with loc appeared first on LernerPython.

October 24, 2026 06:00 AM UTC

October 23, 2026

LernerPython blog, from Reuven Lerner

Pandas offset aliases for resample: ME, MS, and other codes

Using resample in Python Pandas? You’ll want a list of “offset aliases,” the codes that let you specify units.

Some are surprising, such as “ME” and “MS” for “month end” and “month start”.

They’re at: https://pandas.pydata.org/docs/user_guide/timeseries.html#offset-aliases

The post Pandas offset aliases for resample: ME, MS, and other codes appeared first on LernerPython.

October 23, 2026 06:00 AM UTC

October 22, 2026

Daniel Roy Greenfeld

TIL: How to extract subtitles from video

Recently I've been making more and more videos for my youtube channel. I like videos with subtitles, and decided to add them to my own efforts. This is what I do:

This command assumes you have uv and ffmpeg installed.

uvx --python 3.14 --from openai-whisper whisper "name-of-video.mp4" \
  --model small \
  --language English \
  --task transcribe \
  --output_format srt \
  --output_dir captions \
  --fp16 False