skip to navigation
skip to content

Planet Python

Last update: November 15, 2026 09:47 PM UTC

November 15, 2026

Speed Matters

zar-rs

ZAR: A New Archive Format with a Python API

Have you ever been bored because of slow compression/decompression tools? Because speed matters, I’ve developed a new archive format with very fast tools.

ZAR is a new archive format, not just another Python wrapper around ZIP or TAR. It is designed to combine fast parallel Zstandard compression with features that the familiar formats don’t offer together: content deduplication, versioned commits, historical restores, and recovery checks.

The format is implemented in Rust and comes with a native Python API. Create archives quickly, keep earlier file versions in the same archive, and restore part of the archive without unpacking everything.

And Python is just one way to use it: the same Rust library also powers a command-line tool, a desktop archive manager, and bindings for many other languages.

What makes the format different?

Most archive formats are great at packaging files. ZAR is built to manage changes to those files over time, too. Each update can add a commit while preserving earlier states, so you can inspect or restore history without keeping a separate full archive for every version.

Capability ZAR ZIP tar.gz 7z
Parallel, block-based Zstandard compression Yes No (Deflate) No (single-threaded gzip) No (typical LZMA workflows)
Commit-based updates and historical states Yes Appendable, not versioned No No
Within-commit content deduplication Yes No No Solid blocks only
Random access to a file Yes Yes No Limited by solid blocks
Per-block checksums and recovery scan past damaged regions Yes CRC; no recovery scan No Limited
Authenticated encryption with per-commit passwords Yes Varies No (requires a wrapper) AES-CBC

These are not just checkboxes: commits let you update an archive without discarding the old state, deduplication avoids storing duplicate content within a create or add operation, and block framing lets ZAR compress in parallel and verify individual blocks. The result is a format for people who want both quick archive operations and useful history.

Why Python developers should take a look

🚀 Fast compression, without leaving Python

The Python package, pyzarlib, is a PyO3 extension for the Rust archive library. ZAR splits data into blocks and compresses them in parallel with Zstandard. Long-running Rust operations release Python’s GIL, so other Python threads can keep working while an archive operation is running.

In one unencrypted, hot-cache benchmark on a Linux Core i7-10700, ZAR compressed a 525 MiB Linux source tree in 0.86 seconds. In the same run, ZIP took 11.01 seconds and 7z took 24.91 seconds. ZAR’s archive was 115.5 MiB—smaller than ZIP’s 151.1 MiB, though not as small as 7z’s 81.1 MiB. If you need a much smaller archive, 7z may be the better fit; if you want a compelling balance of compression time and size, ZAR is worth a try.

ZAR archive benchmark results on a Linux Core i7-10700 One hot-cache run on a Linux Core i7-10700; results vary by machine, storage, and data.

The full benchmark report includes extraction times, cold-cache runs, and details about the test setup.

🐍 Python API — and a CLI for Python workflows

Create, update, inspect, verify, and extract archives directly from Python. Inputs accept pathlib.Path as well as strings:

from pathlib import Path

import pyzarlib

archive = Path("backup.zar")

created = pyzarlib.create_archive(
    archive,
    [Path("src"), Path("README.md")],
)
print(f"Created with {len(created['warnings'])} warning(s)")

# Add a new commit. Earlier file versions remain in the archive history.
pyzarlib.add_archive(archive, [Path("README.md")])

# Store and read custom archive-level metadata.
pyzarlib.update_custom_metadata(
    archive,
    set={"project": "example", "revision": 1},
)
metadata = pyzarlib.read_custom_metadata(archive)
print(metadata["project"], metadata["revision"])

for entry in pyzarlib.list_archive(archive):
    print(entry["path"], entry["type"])

check = pyzarlib.check_archive(archive)
print(f"Verified {check['commits']} commit(s)")

# Preview a restore before writing files.
plan = pyzarlib.extract_archive(
    archive,
    "restored",
    dry_run=True,
)
print(f"Would extract {len(plan['files'])} files")

Custom metadata values can be strings, signed integers, or finite floats. Updating metadata appends a metadata-only commit; keys beginning with zar. are reserved. Metadata updates require an unencrypted archive.

To append an encrypted commit to the archive above and read its file back:

from getpass import getpass

password = getpass("Commit password: ").encode("utf-8")
pyzarlib.add_archive(
    archive,
    [Path("secrets.txt")],
    password=password,
    encryption_scope="header-and-data",
)

contents = pyzarlib.extract_archive_to_memory(archive, password=password)
print(f"Read {len(contents['secrets.txt'])} bytes")

Only the new commit is encrypted; earlier commits remain unchanged. Without the password, commits with encrypted headers are skipped, so a listing or extraction can show only part of the archive. For commits with different passwords, pass a passwords mapping from commit IDs to password bytes.

Need a terminal workflow? The package also provides zarc-py, a Python CLI for creating, adding, removing, joining, undoing, checking, listing, inspecting, and extracting archives.

The package supports Python 3.11 and newer, with CI coverage for CPython 3.11–3.15 and free-threaded CPython 3.14t and 3.15t. Native work releases the GIL, and the extension is tested with concurrent Python calls.

🗜️ Compression controls and familiar archive inputs

Choose a Zstandard compression level from -7 to 22 and a block size from 64 KiB to 32 MiB, or let ZAR select a block size automatically. Default compression level is 3. Files with identical content can be deduplicated within a create or add operation. Some already-compressed formats and media are stored without an extra compression pass to improve compression speed.

You don’t have to start with a ZAR archive. Python create_archive and add_archive can import entries from ZIP, 7z, RAR, ISO, TAR, TAR.GZ, TGZ, and GZ files, preserving their internal paths. Encrypted ZIP/7z/RAR inputs and some special archive entries are not supported; see the project documentation for the precise limits.

More than a compressed file

ZAR archives are built around commits:

That makes ZAR useful for more than a one-off download bundle: it can serve as a compact, inspectable history of project snapshots or application data.

One Rust core, many ways to use it

The shared archive library is available through:

Language or runtime Integration
Rust Native zarlib crate
Python PyO3 pyzarlib extension
C and C++ czarlib C ABI and cxxzarlib
Go gozarlib through cgo
Java FFM and JNI bindings
C# P/Invoke binding
WASI WebAssembly Component; Node.js integration is available

You can use the zarc command-line tool, manage archives in the zarg desktop application, or connect an AI client to the local zarmcp server. The companion zcopy tool transfers file trees over a network using compressed, optionally encrypted ZAR batches. Try it out. You’ll be impressed how fast you can transfer big file trees over the network without creating (temporary) archives.

Installation

Build the Python extension from the project source with Maturin:

python -m pip install maturin
cd pyzarlib
maturin develop --release

Then import pyzarlib in your Python code. The repository includes Python installation and API documentation, examples, and tests.

Get started

If you’re a Python developer who wants quicker archives without giving up a comfortable Python API, try ZAR on a representative dataset. The benchmark includes machine and method details so you can compare it against your own workload—not just trust a headline number.

Explore the source, complete documentation, and benchmark reports:

https://github.com/brmmm3/zar-rs

Licensed under the MIT or Apache-2.0 License.

November 15, 2026 07:00 AM UTC

Graham Dumpleton

Muru: working with Kubernetes, not around it

The last post was about muructl serve as the main process of a container: running the workers it is told to fit the container, answering probes on a port of its own, shutting down cleanly on SIGTERM, and running with a read-only root under any user ID. None of that cares what runs the container. Most of these containers end up on Kubernetes though, and Kubernetes has its own way of doing several things a Python web server would otherwise need help with. Rolling out new versions, spreading traffic, scaling, running one-off tasks, all of that is done by the platform already.

So the aim with Muru on Kubernetes is to fit in with what Kubernetes provides rather than duplicate it, and to fill the gaps where what Kubernetes provides doesn't quite reach the application.

The Deployment

The image is built the same way as in the last post, with a small WSGI application which has a page that waits as if on a database. The Deployment follows the one in the Muru documentation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels: {app: shop}
  template:
    metadata:
      labels: {app: shop}
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: shop
          image: muru-blog-k8s:1
          imagePullPolicy: IfNotPresent
          env:
            - {name: MURU_SERVE_PROCESSES, value: "2"}
            - {name: MURU_SERVE_THREADS, value: "4"}
            - {name: MURU_SERVE_SHUTDOWN_DELAY, value: "5s"}
            - {name: MURU_SERVE_SHUTDOWN_BUDGET, value: "25s"}
            - {name: MURU_SERVE_LOG_FORMAT, value: "json"}
            - {name: SHOP_DATABASE, value: "postgresql://db.shop/shop"}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
            - name: POD_NAMESPACE
              valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
            - name: NODE_NAME
              valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
          ports:
            - {name: http, containerPort: 8080}
            - {name: admin, containerPort: 9090}
          startupProbe:
            httpGet: {path: /startupz, port: admin}
            periodSeconds: 2
            failureThreshold: 60
          livenessProbe:
            httpGet: {path: /livez, port: admin}
          readinessProbe:
            httpGet: {path: /readyz, port: admin}
            periodSeconds: 2
            failureThreshold: 1
          resources:
            requests: {cpu: 500m, memory: 512Mi}
            limits: {cpu: "2", memory: 1Gi}
          resizePolicy:
            - {resourceName: cpu, restartPolicy: NotRequired}
            - {resourceName: memory, restartPolicy: NotRequired}
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: run, mountPath: /run/muru}
            - {name: tmp, mountPath: /tmp}
      volumes:
        - name: run
          emptyDir: {sizeLimit: 2Gi}
        - name: tmp
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: shop
spec:
  selector: {app: shop}
  ports:
    - {name: http, port: 80, targetPort: http}

Much of this is what the last post covered, now expressed as Kubernetes configuration. All the settings come from environment variables. The probes go to the admin port, which the Service doesn't expose. The pod, namespace and node names come in through the Downward API so they end up in every JSON log record. I labelled the namespace to enforce the restricted Pod Security Standard and the pods were admitted without any warnings, as Muru only needs /run/muru and /tmp to be writable.

As in the last post, the number of processes and the threads in each are set explicitly, chosen to go with the pod's resources. With a CPU limit of 2 and a memory limit of 1Gi, that is two workers of four threads:

container limits: 2 CPUs from cpu.max 200000 100000, memory.max 1.0GB with 128MB of headroom
...
serving app:application on http://0.0.0.0:8080 with 2 workers of 4 threads

If the limits change, so should these. The memory limit is also what the memory guard from the last post works from, so a worker which grows is replaced before the pod is out of memory, with nothing more to set. The parts which are particular to Kubernetes are the shutdown delay, the readiness probe's failureThreshold and the absence of any preStop hook, and those are what the rest of this post is about, along with autoscaling and migrations.

Stopping without a preStop hook

When a pod is deleted, as happens to each old pod during a rolling update, two things start at the same time. The kubelet on the node begins shutting the pod down, which means sending SIGTERM to process 1 in each container. Separately, the control plane marks the pod's endpoint in the Service's EndpointSlices as terminating. Every kube-proxy, and any ingress controller, then has to notice that and stop sending new connections to the pod. Nothing makes the second wait for the first, so for a short while after SIGTERM a pod can still be handed new connections.

A server which closes its listening socket as soon as it gets SIGTERM, which is what gunicorn and uvicorn both do, will refuse those connections. The usual fix is a preStop hook which sleeps for a few seconds. The kubelet runs the hook before sending SIGTERM, so the server goes on serving during the sleep while the endpoint changes spread through the cluster. For a long time that meant an exec hook running sleep, which needs a sleep binary in the image. There is now also a sleep action, stable since Kubernetes 1.34, which the kubelet runs itself. Either way, it is something you have to know to add, and during the sleep the server has no idea it is about to be stopped. Neither gunicorn nor uvicorn has an option for this. Gunicorn's --graceful-timeout and uvicorn's --timeout-graceful-shutdown only govern what happens after the signal, by which time the listener is already closed.

Muru does this itself, with --shutdown-delay, set here through MURU_SERVE_SHUTDOWN_DELAY. On SIGTERM, /readyz starts failing at once. For the length of the delay the listener stays open and new connections are still served, but each response carries Connection: close, and idle connections are closed, so clients reconnect and land on another pod. After the delay, the listener closes, the requests in flight finish and the container exits, all within the budget, which is set below terminationGracePeriodSeconds.

To see what difference it makes, I ran a small Python client in a pod in the same namespace. It used eight threads, each on its own keep-alive connection to the Service, asking for a page which takes 50 milliseconds, for 40 seconds. Five seconds in I ran kubectl rollout restart deploy/shop. With the five second delay:

responses: {200: 6106}
errors: {} first at (s): {}
responses with Connection: close, by pod: {'shop-7f8f5b489f-bbnt5': 13, 'shop-7f8f5b489f-hl9zb': 5}

Every request succeeded, and the 18 responses which carried Connection: close were the client's connections being moved off the two old pods as each was stopped. Watching the EndpointSlice at the same time showed the first old pod being marked terminating:

shop-64df649bb5-x4hwq ready=true serving=true terminating=false
shop-7f8f5b489f-bbnt5 ready=false serving=true terminating=true
shop-7f8f5b489f-hl9zb ready=true serving=true terminating=false

That happened at 09:03:43.350. The last request that pod's access log recorded was at 09:03:43.455, and its host process didn't log receiving SIGTERM, which is when the workers start stopping, until 09:03:48.353, five seconds later. On a single node cluster, the endpoint changes reached kube-proxy within a fraction of a second.

So is the delay needed at all on a cluster this small? I set it to zero and ran the same test:

responses: {200: 6100}
errors: {'ConnectionRefusedError': 4} first at (s): {'ConnectionRefusedError': 11.4}
responses with Connection: close, by pod: {'shop-55885b469d-rxk2k': 2, 'shop-55885b469d-hftl9': 7}

Four requests failed because their connections were refused. I repeated both tests: four refused again with no delay, none again with the delay. Even when the endpoints catch up in a fraction of a second, there is a window where kube-proxy still routes new connections to a pod whose listener has closed. On a larger cluster, with an ingress controller in front, that window only gets longer, which is why the Muru documentation starts with five seconds.

Under Docker or Compose the delay defaults to zero, since a stopped container stops getting connections straight away. Under Kubernetes it is a single environment variable in the Deployment, and the pod spec has no preStop hook.

Resizing and restarting in place

As of Kubernetes 1.35 it is a stable feature that a running container's CPU and memory can be changed in place, through the pod's resize subresource, which a VerticalPodAutoscaler can also use. With the resizePolicy in the Deployment above, restartPolicy: NotRequired for both, no restart is needed, as the kubelet just changes the container's cgroup limits. Muru reads them again at its next memory check, which a pod I gave three CPUs and 1.5Gi showed in its log:

container limits changed: cpu.max from 200000 100000 to 300000 100000, memory.max from 1.0GB to 1.5GB

The memory guard then works from the new limit, and the warnings Muru gives at start are checked again, so a resize which leaves the workers short of CPUs or memory says so. What doesn't change is the number of processes and threads, which are configuration. Giving a pod more CPU without more workers to use it gains little, and giving it less memory only has the memory guard replace workers sooner. A change to the resources which ought to change the number of workers is better made to the Deployment, with the MURU_SERVE_* settings changed alongside the limits, so it rolls out as new pods, which the shutdown delay above already covers.

A resize did turn out to be a handy way of showing what happens when a container is restarted without its pod going anywhere, as also happens after a failed liveness probe. Setting restartPolicy: RestartContainer for memory in the resizePolicy instead has the kubelet restart the container for a resize. I did that and, with the client opening a new connection for each request, changed the memory limit of one pod. The container restarted in the same pod, on the same node with the same address.

The endpoint went to not ready 1.5 seconds after the resize, and nothing failed:

responses: {200: 4448}
errors: {} first at (s): {}
responses with Connection: close, by pod: {'shop-567fc9d9c7-46lpz': 59}

The reason nothing failed is the failureThreshold: 1 on the readiness probe. With a rolling update the pod is deleted, so its endpoint is marked terminating at once. A container restarting in a pod which isn't going anywhere doesn't get that. The only thing taking the pod out of the Service is the readiness probe, and by default a readiness probe has to fail three times in a row. Every two seconds, three failures take up to six seconds, which is longer than the five second delay. Taking the setting out and repeating the same resize under the same load shows what happens:

responses: {200: 4448}
errors: {'ConnectionRefusedError': 10} first at (s): {'ConnectionRefusedError': 10.0}
responses with Connection: close, by pod: {'shop-5dfdbd5847-nz5bh': 180}

The resize went in at 09:28:23.665. Muru started its shutdown delay, the kubelet's readiness probe got a 503 from 09:28:25, and Muru sent Connection: close on 180 responses. But the EndpointSlice still listed the pod as ready until 09:28:29.062, by which time the old container had exited.

A threshold of one would be a risky setting for a server which answers readiness from the same threads that serve requests, since one slow probe under load would take the pod out of the Service. With Muru, /readyz is answered by Muru itself and not by a worker, and as the last post showed it answers in a couple of milliseconds with every worker busy, so a failure means the pod really is stopping or has no ready workers.

Autoscaling on busy slots, not CPU

A HorizontalPodAutoscaler usually scales on CPU. For a Python web application which spends most of its time waiting on a database, that is close to useless. Every thread can be waiting, with requests queuing, while the CPU sits almost idle.

Muru publishes the measure that actually matters, muru_capacity_utilisation, the share of worker slots in use averaged over 30 seconds, on the admin port's /metrics. To get that to an autoscaler takes Prometheus to scrape it and the Prometheus adapter to present it as a pods metric through the custom metrics API. The pods got the usual scrape annotations:

template:
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9090"

and the adapter got a rule for the one metric:

rules:
  default: false
  custom:
    - seriesQuery: 'muru_capacity_utilisation{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: namespace}
          pod: {resource: pod}
      metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

The autoscaler aims to keep the average across pods at 70 percent:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: shop}
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric: {name: muru_capacity_utilisation}
        target: {type: AverageValue, averageValue: 700m}

For load, the client ran 24 threads for four minutes, each asking over and over for a page which waits half a second as if on a database. Each request was on a new connection, so the Service spread them across however many pods there were, much as an ingress controller balances each request. Two pods have 16 slots between them, two workers of four threads each, so 24 requests at a time can't all be served at once. Checking every ten seconds:

NAME   REFERENCE         TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
shop   Deployment/shop   0/700m      2         10        2          40s
shop   Deployment/shop   391m/700m   2         10        2          50s
shop   Deployment/shop   853m/700m   2         10        2          61s
shop   Deployment/shop   839m/700m   2         10        3          81s
shop   Deployment/shop   754m/700m   2         10        4          91s
shop   Deployment/shop   732m/700m   2         10        4          2m1s
shop   Deployment/shop   726m/700m   2         10        4          3m2s

The autoscaler's events:

New size: 3; reason: pods metric muru_capacity_utilisation above target
New size: 4; reason: pods metric muru_capacity_utilisation above target

Within a minute of the load starting, the Deployment had gone from two pods to four. It then stayed at four, because 24 requests over 32 slots is 0.75, and the autoscaler ignores anything within its default tolerance of 10 percent of the target. While all that was happening, kubectl top showed the pods using next to no CPU:

shop-5dfdbd5847-d5ds9   10m   39Mi
shop-5dfdbd5847-nz5bh   10m   39Mi
shop-5dfdbd5847-pjmzj   11m   40Mi
shop-5dfdbd5847-pmfdt   10m   39Mi

That is about 2 percent of the 500m each pod requests. An autoscaler targeting 70 percent CPU would have left the Deployment at two pods, with a third of the requests waiting for a slot. When the load stopped, the metric dropped to zero, and the autoscaler took the Deployment back to two pods about five minutes later, its default stabilisation window for scaling in.

The other figure worth scaling on is how long requests wait for a slot, which is the queue phase of Muru's request duration histogram. Asking Prometheus for the 95th percentile ninety seconds into the run, with its two minute window covering the time at two pods:

histogram_quantile(0.95, sum by (le) (rate(muru_request_duration_seconds_bucket{namespace="muru-blog",phase="queue"}[2m])))

gave 0.457 seconds. Unlike the utilisation, this is a single figure for the whole Deployment rather than one for each pod, so it can't be given to the autoscaler in the same way. It can instead be set up as an external metric, through a separate rule in the Prometheus adapter, or by using KEDA in place of the adapter. KEDA is a project which manages a HorizontalPodAutoscaler for you and feeds it from sources such as a Prometheus query, so a query like this one can be given to it directly, with a threshold of, say, 100 milliseconds. Either way, the Deployment would then scale on how long users wait rather than on how busy the workers are.

Neither gunicorn nor uvicorn exposes anything like this. You can get at request latency through a Prometheus client in the application, but how many slots are busy and how long a request waited for one are things only the server knows.

Migrations as a Job

In the releases post, migrations ran from the new release, with muructl command run shop@r4 migrate, while the old release kept serving. On Kubernetes the new release is a new image, and the natural place to run something once from it before the Deployment changes is a Job. The project declares the command as before:

[tool.muru.commands.migrate]
description = "Apply database migrations"
run = ["python", "migrate.py"]
timeout = "10m"

and the Job runs it from the new image:

apiVersion: batch/v1
kind: Job
metadata:
  name: shop-migrate-2-0
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: migrate
          image: muru-blog-k8s:2
          imagePullPolicy: IfNotPresent
          command: ["muructl", "command", "run", "migrate"]
          env:
            - {name: SHOP_DATABASE, value: "postgresql://db.shop/shop"}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: {drop: ["ALL"]}

With no muructl serve running in the Job's pod, muructl command run runs the command itself, using the image's interpreter, the project's environment and the command's timeout. The Deployment is then only updated once the Job has completed:

kubectl apply -f migrate-job-2.yaml
kubectl wait --for=condition=Complete job/shop-migrate-2-0 --timeout=120s
kubectl set image deploy/shop shop=muru-blog-k8s:2

job.batch/shop-migrate-2-0 created
job.batch/shop-migrate-2-0 condition met
deployment.apps/shop image updated

The migration's output is in the Job's logs:

connecting to postgresql://db.shop/shop
applied 0007_order_notes
applied 0008_order_notes_index

A failed migration fails the Job, since muructl exits with the command's status. Running the migration in a Job of its own without the database setting:

SHOP_DATABASE is not set
muru: the command exited with status 1 after 8ms

NAME             STATUS   COMPLETIONS   DURATION   AGE
shop-migrate-2   Failed   0/1           3s         3s

A kubectl wait for that Job would never see it complete, so a deploy script running the three commands above would stop there rather than update the Deployment. With backoffLimit: 0 the Job doesn't try again, which for a migration that has failed halfway is usually what you want.

You could of course write command: ["python", "migrate.py"] in the Job and get much the same result with gunicorn in the image. What the declared command buys is that the Job, a kubectl exec into a running pod and the daemon from the releases post all run the migration the same way, from the one definition in the project, with its timeout.

Looking inside a pod

Everything the last post did with docker exec works through kubectl exec, since muructl in the container finds muructl serve through its run directory. With a request stuck in a page which sleeps for eight seconds:

kubectl exec deploy/shop -- muructl requests

RELEASE              GROUP        QUEUED  QUEUED-OLDEST  THREADS  THREAD-OLDEST
shop@r1              default      0       -              1/8      1.9s

kubectl exec deploy/shop -- muructl debug stacks

muru-app-2 in worker 1:
  serving GET /slow?seconds=8 for 1.9 s (request 01M6TNH4EM7EES4KX4129GEBM8)
  File "/opt/muru/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/app/app.py", line 18, in application
    time.sleep(seconds)

Who is allowed to do this is then up to RBAC, like any other kubectl exec.

What's next

Very little in this post needed Muru to do anything Kubernetes-specific. The shutdown delay is one setting, the workers are set to go with the pod's resources, the autoscaler reads a metric Muru already publishes, and the migration is a Job running a command the project already declares.

The autoscaling here works from outside, with Kubernetes adding whole pods once the workers are busy. The next post, scaling from what the workers report, goes back to the daemon, where Muru decides itself how many workers a release needs, as a scaling policy also has it do within a pod, and where, because it knows why requests are waiting, it can tell whether more workers will help or whether something in the release needs changing.

November 15, 2026 12:00 AM UTC

Bob Belderbos

Give Your AI Reviewer the Spec, Not Just the Diff

I review everything Claude Code writes, but at the current speed increase, it's easier to miss things. So after my review, a second, independent reviewer checks the work, and it gets more than the diff.

Here's how I've set it up.

Claude Opus generates, I review, Codex does a second review

The idea here is to use a different model/tool (in my case Codex) to keep bias out. But instead of a pure diff, which is already helpful, the reviewer also needs to know what the aim was.

For serious work (one-off scripts can get away with less), I start each feature with a grillme session to come up with a concise and well-scoped spec. I updated my variant of this skill to write the resulting spec to a local specs folder (goal, decisions (including rejected options), in/out of scope, acceptance). This way the reviewer has much richer context about the diff it's reviewing.

This is the spec template my grillme skill writes to specs/<slug>.md, kept under ~40 lines:

# <Feature>

## Goal
## Decisions
## Scope
## Acceptance
## Out of scope

If you're new to skills, check out how I use Claude Code skills for structured AI collaboration.

This was also a succinct comment from somebody on X when I posted about this last week:

"a reviewer with zero context from the writing session, otherwise it's grading its own homework. Do you give the reviewer the spec, or only the diff?"

So the answer is both!

A tight CLAUDE.md (touch only what the task needs, grep callers before changing a signature, ruff → ty → pytest) stops the simple mistakes before review.

So my flow for more serious work: grillme → spec → Opus implements → I review (!), then Codex for a second, independent and well-scoped review.

Recent example: Codex raised one P2. Each input passed validation on its own, but their product overflowed the DB column. Unlikely in practice for that app, but cross-field constraints like this are easy to miss (I've seen it in 1:1 coaching more than once).

The diff tells the reviewer what changed. The spec tells it whether that was the point.

November 15, 2026 12:00 AM UTC

November 14, 2026

Graham Dumpleton

Muru: a better tenant in a container

The last two posts, on releases and canaries, were all about the Muru daemon running more than one release of an application side by side and moving traffic between them. When the application runs in a container, Muru deliberately does none of that. A container image is already a fixed release of an application, and starting new containers, moving traffic to them and putting the old ones back is the job of whatever runs the containers. Doing the same thing again inside the container would only get in its way.

So in a container Muru runs as muructl serve, which serves the single application held in the image. The edge, the controller, the host and its workers are all still there, as in development mode, but releases and traffic policies aren't. Instead, it concentrates on being a good main process for a container, which mostly means handling things people otherwise end up doing by hand when using gunicorn or uvicorn.

An image

The image for this post holds a small WSGI application with a few pages which misbehave in different ways, which I will get to later. The Dockerfile follows the pattern from the Muru documentation:

FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
RUN UV_TOOL_BIN_DIR=/usr/local/bin uv tool install muructl
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN muructl build --deployment --deps-only --no-runtime --target /app/.venv
COPY . .
RUN muructl build --deployment --target /app/.venv --runtime-dir /opt/muru

FROM python:3.13-slim
COPY --from=build /usr/local/bin/muructl /usr/local/bin/muructl
COPY --from=build /app /app
COPY --from=build /opt/muru /opt/muru
WORKDIR /app
RUN useradd --uid 10001 app && install -d -o 10001 -g 0 -m 0770 /run/muru
USER 10001
EXPOSE 8080 9090
CMD ["muructl", "serve"]

muructl is a single executable, so the final image copies just the binary. The two muructl build steps install the application's environment from uv.lock. The first installs only the dependencies, so that its layer can be reused until they change. Each says what it is running:

muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode --no-install-project
...
muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode
muru: built /app/.venv on Python 3.13.18 (cp313)
muru: fetched the runtime library muru-1.0.0-cp313-cp313-manylinux_2_28_aarch64.whl from PyPI into /opt/muru

The last line is the part which is specific to Muru. The runtime library which runs inside the workers has to match both the version of muructl and the interpreter, so it is fetched when the image is built. It is kept outside the application's environment, so the application doesn't need to list Muru as a dependency.

The important thing is that this is the only point at which anything is fetched or installed. When the container starts, muructl serve never builds an environment, downloads a Python or fetches the runtime library, and anything it runs gets UV_OFFLINE=1 and UV_PYTHON_DOWNLOADS=never in its environment. If something is missing, it stops with a message naming the build step which would have provided it. For example, pointing it at the wrong runtime directory gives:

muru: error: no runtime library for /app/.venv/bin/python (cp313): build the image with muructl build --runtime-dir /nowhere, or add muru 1.0.0 to the environment

That matters more than it might seem. It isn't unusual to find images where an entrypoint script pip installs something, or where a missing package is quietly downloaded at start. That works until the network is unavailable or the package index changes, at which point two containers started from the same image are no longer the same.

Telling it how many workers

Starting the image with limits of two CPUs and a gigabyte of memory, with no swap, on a Docker host with ten CPUs, I gave it the number of worker processes and the threads in each:

docker run --name shop --cpus 2 --memory 1g --memory-swap 1g \
  -e MURU_SERVE_PROCESSES=2 -e MURU_SERVE_THREADS=4 -p 8080:8080 -p 9090:9090 shop

The output says what it found and what it is running:

muru: container limits: 2 CPUs from cpu.max 200000 100000, memory.max 1.0GB with 128MB of headroom
...
muru.host[12]: preloaded app:application in 1 ms
muru.host[12]: app:application is served as WSGI
muru.host[12]: worker 0 started as pid 15
muru.host[12]: worker 1 started as pid 16
...
muru: serving app:application on http://0.0.0.0:8080 with 2 workers of 4 threads
...
muru: the host holds 34MB once the application is imported

These settings should always be supplied, chosen to suit the resources given to the container. If they are left out, Muru runs a single worker. Two processes for two CPUs suits an application whose requests mostly use the CPU, while one whose requests spend most of their time waiting on a database can use more threads in each. The memory has to fit as well. Two workers, along with the host which imported the application, leave plenty of room in a gigabyte, but an application whose workers each need 750MB would have to run a single process in a container this size, or be given a bigger container. The other examples in this post start the container with the same settings, which I leave out to keep the commands short.

Muru doesn't try to choose these numbers itself, because they can't be reliably worked out from inside the container. Asking Python how many CPUs it has, in the same container, gives:

docker run --rm --cpus 2 shop python3 -c "import os; print(os.cpu_count(), os.process_cpu_count())"

10 10

A CPU limit set with --cpus, or resources.limits.cpu in Kubernetes, is a CFS quota in the cgroup's cpu.max file, which neither of those functions reads. Gunicorn's default is a single worker unless WEB_CONCURRENCY says otherwise, and uvicorn's is the same. Gunicorn's documentation suggests (2 x CPU cores) + 1, and the image the Gunicorn project publishes uses that formula by default, working out the count with nproc, which also says 10 here, so it would start 21 workers in a container allowed two CPUs. On a large Kubernetes node, with a pod limited to a couple of CPUs, it is far worse. Even a count which got the CPUs right would say nothing about how much memory each worker needs, or whether the requests use much CPU at all.

Muru does read the limits from the cgroup, as the first line of its output showed, but only to check the settings against them. Asking for four workers in the same container still starts four, but with a warning:

muru: warning: 4 workers is more than the 2 CPUs cpu.max allows; see https://docs.muruhq.dev/en/latest/howto/containers.html#container-limits

It also warns if the workers wouldn't fit in memory once each holds its own copy of what the host imported. A forked Python process tends to end up with such a copy over time, as it gradually stops sharing memory with the process it was forked from.

Running short of memory

A Python process tends to hang on to memory it has used, whether through a cache, a leak or one unusually large request. In a container, the usual outcome of this is that the container runs out of memory and the kernel kills a process outright.

When the container has a memory limit, Muru gets in first. Every few seconds it checks how much memory is left under the limit. If that drops below a margin it keeps in reserve, which it calls the headroom, it takes the largest worker out of service and replaces it with a fresh one. The old worker finishes the requests it already has before it goes, so nobody sees an error, and its memory is given back. Nothing needs to be set for this beyond the container's memory limit.

To see it in action, I used a page of the application which holds on to another 10MB each time it is requested, and asked for it once a second:

muru.host[12]: worker 0 (pid 15) the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; replacing it
muru.host[12]: worker 0 (pid 15) stops before its replacement starts: 101MB is left under the memory limit, less than its size
muru.host[12]: worker 0 started as pid 38
muru.host[12]: worker.recycled: worker 0 (pid 15) replaced by pid 38: the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 0 (pid 38) is ready
...
muru.host[12]: worker 1 (pid 16) the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; replacing it
muru.host[12]: worker 1 (pid 16) stops before its replacement starts: 100MB is left under the memory limit, less than its size
muru.host[12]: worker 1 started as pid 49
muru.host[12]: worker.recycled: worker 1 (pid 16) replaced by pid 49: the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 1 (pid 49) is ready

Both workers had been growing at the same rate, so when the container first ran short, worker 0 was only just the larger of the two, at 424MB. Replacing it gave that memory back, which left room for worker 1 to keep growing. By the time the container ran short again, worker 1 was at 645MB, and it was replaced in turn.

Normally Muru starts the new worker before the old one leaves, so there is never a gap. Here less memory was left in the container than the old worker was using, so rather than risk two large workers at once, the old worker stopped first, finishing the requests it already had, and its replacement started once it had gone. The other worker carried on serving in the meantime, and all 150 requests succeeded.

Without this, the first anyone knows of a worker growing is when the kernel's out-of-memory killer picks a process in the container to kill, and that can just as easily be one in the middle of a perfectly good request. Neither gunicorn nor uvicorn keeps any watch on memory. The best either offers is restarting a worker after a set number of requests, however much memory it is using.

Muru calls this its memory guard. Running muructl status inside the container shows the limits, how much memory is used and left, and what the guard last did:

docker exec shop muructl status

...
cpu      2 CPUs from cpu.max
memory   memory.max 1.0GB, 402MB used, 622MB left, headroom 128MB
         workers 329MB, host 34MB
guard    watching; it last replaced shop@r1 worker 1 (pid 16) at 07:31:18 UTC

The guard looks at the container as a whole. If a single worker should never grow beyond a certain size, however much room is left, it can also be given a hard memory limit, such as MURU_SERVE_MEMORY=hard_limit=300MB. A worker which goes over that limit is replaced in the same way. The value is chosen to suit the application, and should be low enough that all the workers together still fit in the container.

Memory is only checked every five seconds, so if it grows by more than the headroom between checks, the out-of-memory killer can still strike first. A request which allocated a gigabyte all at once showed what happens then:

502 Bad Gateway
reason: worker_crashed
request id: 01M6TF11TNSGS7R9AW4G704V4X

The worker sent no response head.

muru.host[12]: worker.oom_killed: worker 0 (pid 15) was killed by the kernel for memory, restarting
muru.host[12]: worker 0 started as pid 47
muru.host[12]: worker 0 (pid 47) is ready

Each worker raises its own oom_score_adj as it starts, which makes the workers the processes the kernel picks first. So it was the worker which was killed, rather than the host or muructl, and the host started a new one. The kill is also reported for what it was, as a worker.oom_killed event and in the muru_worker_exits_total metric with the reason oom_killed, rather than leaving a worker which mysteriously went away.

Probes on their own port

The output when it started also included this line:

muru: admin port on http://0.0.0.0:9090/

Port 9090 is separate from the application's port and answers the questions a container platform asks. /livez says whether the process is alive, /readyz whether enough workers are ready to take requests, and /startupz whether the application has finished loading. /metrics has the metrics in Prometheus format.

Gunicorn and uvicorn have nothing like this. The usual answer is to add a /health view to the application, which means the health check is served by the same workers as everything else. When the application is slow, perhaps because its database is, every worker is busy. The health check then queues behind real requests and times out, and the platform restarts containers which were doing nothing wrong, just when they are needed most.

To check that Muru doesn't suffer from this, I sent twelve requests to a page which takes six seconds to answer, more than the two workers with four threads each could handle at once, and then queried the admin port:

livez 200 0.001336s
readyz 200 0.001337s

The metrics confirmed every slot was busy, with requests queuing:

muru_queue_depth{app="shop",release="r1",group="default"} 4
muru_slots_busy{app="shop",release="r1",group="default",side="threads"} 8
muru_slots_total{app="shop",release="r1",group="default",side="threads"} 8

The probes are answered by muructl itself, from what it knows about the host and its workers, without a request having to go through a worker. A container whose workers are all busy is still alive, so it isn't killed for being busy. Having a port of their own also keeps the probes and metrics out of reach of the application's Ingress or load balancer.

An image without curl, such as one based on the slim Python images, can use muructl itself for a health check:

HEALTHCHECK --interval=5s CMD ["muructl", "probe", "ready"]

Process 1

Looking at the processes inside the container (by reading /proc, since the slim image has no ps), each of Muru's processes has set a title saying what it is:

    1     0 S  muructl serve
    7     1 S  muru serve shop
   11     7 S  muru relay shop@r1
   12     7 S  muru host shop@r1
   15    12 S  muru worker shop@r1 default/0
   16    12 S  muru worker shop@r1 default/1

Process 1 is muructl acting as an init, in the way tini or docker run --init would. It forwards signals to the real muructl serve, which is its child, and collects any process in the container which exits, so nothing is left as a zombie. Process 1 in a container has to do this, because orphaned processes are handed to it, and if it never collects them, zombies fill up the process table. Below muructl serve are the host and its two workers, and a log relay which passes on what they write.

The application has a page which starts a report by running a shell command, which leaves a process running in the background. That happens in real applications more often than anyone would like. Just after I requested the page three times, muructl status --processes --all, where --all adds the processes Muru didn't start itself, showed them:

docker exec shop muructl status --processes --all

edge  3.0 req/s  0 err/s  p50 2.1 ms  p95 4.6 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 53B/s

PID   ROLE           WHAT                         UP    CPU      RSS   REQ/S   BUSY   QUEUE      P95   GIL/LAG
7     serve          controller and edge, :8080   5s   2.3%   19.5MB
12    └─ host        app:application (r1)         5s   0.0%   34.7MB     3.0    0/8       0   4.6 ms
11       ├─ relay    output to muructl            5s   0.0%   11.4MB
15       ├─ worker   worker 0                     5s   0.0%   28.7MB     2.0    0/4           4.8 ms
16       ├─ worker   worker 1                     5s   0.0%   28.9MB     1.0    0/4           2.4 ms
54       ├─ other    sleep 1                      1s           1.5MB
56       ├─ other    sleep 1                      1s           1.5MB
58       └─ other    sleep 1                      1s           1.5MB

The orphaned sleep processes were taken over by the host, which collects any process its workers leave behind, and a couple of seconds later they were gone, with no zombies. Gunicorn does collect any child that exits, including ones it didn't start. Uvicorn doesn't, so when run as process 1 it leaves a zombie behind for each one. Neither project's documentation says anything about tini or --init. An image which uses its own init can turn off Muru's by setting MURU_NO_INIT=1.

Stopping in order

When docker stop or Kubernetes wants a container gone, it sends SIGTERM, waits, and then sends SIGKILL. Docker waits ten seconds and Kubernetes thirty by default. For a server, what happens in between decides whether anyone sees an error.

I started four requests to the page which takes three seconds, and half a second later ran docker stop, while checking /readyz every 300 milliseconds:

readyz 200
readyz 503
readyz 503
readyz 503
readyz 503

The four requests all completed:

 200 3.004430s
 200 3.004411s
 200 3.004890s
 200 3.004973s

and docker stop returned after 2.76 seconds, with the container's exit status 0. On SIGTERM, readiness starts failing at once, so nothing new is sent to the container. The listener then closes, the requests in flight finish, and the workers stop. All of that has to happen within a budget, eight seconds unless --shutdown-budget changes it. That is less than the ten seconds docker stop allows, so anything still running at the end is stopped by Muru rather than killed by Docker.

Compare that with the defaults elsewhere. Gunicorn's graceful_timeout is 30 seconds, so under docker stop a slow request is cut off by SIGKILL at ten seconds and gunicorn never gets to finish shutting down. Uvicorn has no graceful shutdown timeout by default, so it waits as long as it takes, which under Docker means until the SIGKILL arrives. With either, the fix is to remember to set a timeout which fits within the platform's.

Kubernetes adds a wrinkle, as a pod can keep receiving new connections for a few seconds after SIGTERM, while the endpoints catch up. That is what --shutdown-delay is for, and I will come back to it in the next post.

A read-only root and any user

A security conscious platform will run the container with a read-only root filesystem, and OpenShift will run it as an arbitrary user ID in group 0, with no entry for it in /etc/passwd. Everything muructl serve writes at run time goes under one directory, /run/muru. With a read-only root and nothing mounted there, it stops at once and says what it needs:

docker run --rm --read-only --user 123456:0 shop

muru: error: the run directory /run/muru cannot be written (Read-only file system (os error 30)); with a read-only root filesystem give it a writable volume, a tmpfs or an emptyDir mounted at /run/muru, or name another with --run-dir

Giving it what it asks for:

docker run --read-only --tmpfs /run/muru --tmpfs /tmp --user 123456:0 ... -p 8080:8080 -p 9090:9090 shop

it runs as normal under a user ID which doesn't exist in the image:

hello from pid 15

Gunicorn in the same situation fails at start, with No usable temporary directory found, as it writes a heartbeat file for each worker to the temporary directory. That can be fixed with a tmpfs on /tmp, or by pointing --worker-tmp-dir at /dev/shm, but you have to know to do it.

Settings in layers

Every setting of muructl serve can be given as a command line flag. An image can also set defaults in the project's pyproject.toml, under [tool.muru.serve], and a deployment can override any of them with an environment variable named after the flag. --print-config shows the settings in force and where each came from, and works inside a running container too:

docker run --name shop -e MURU_SERVE_PROCESSES=3 -e MURU_SERVE_LOG_FORMAT=json ... shop
docker exec shop muructl serve --print-config

processes = 3                     # MURU_SERVE_PROCESSES
...
log_format = "json"              # MURU_SERVE_LOG_FORMAT

--config, or MURU_SERVE_CONFIG, names a settings file to read instead of the one in the image, such as a file mounted from a ConfigMap.

Logs a collector can use

With the JSON log format, every line is a JSON object. When the environment has POD_NAME, POD_NAMESPACE and NODE_NAME, which the Kubernetes Downward API can provide, each line also says where it came from. I set those by hand for this run. More useful is what happens to a traceback. The application has a page which raises an exception, and the error shows up as a single record, with the whole traceback in its exception field, alongside the request, release and worker it happened in:

{"time":"2026-11-07T07:37:52.296Z","kind":"app","source":"muru.worker[0]","app":"shop","exception":"Traceback (most recent call last):\n  File \"/app/app.py\", line 25, in application\n    raise RuntimeError(\"something went wrong\")\nRuntimeError: something went wrong","group":"default","level":"ERROR","logger":"muru","message":"the application raised an exception","release":"r1","request_id":"01M6TF2DXFEHAEQZN1C1JEXGBE","worker":0,"pod":"shop-7d4b9c-x2k8p","namespace":"shop","node":"worker-2"}

Anyone who has had to piece a Python traceback back together from a log system which treated each line as a separate entry will appreciate that.

Inside the container

Finally, the commands from looking inside the workers work inside the container through docker exec, talking to muructl serve over its admin socket. With a slow request under way:

docker exec shop muructl debug stacks

muru-app-0 in worker 0:
  serving GET /slow?seconds=4 for 1.2 s (request 01M6TF2SRXPXTHSAPPDG6MVNJQ)
  File "/opt/muru/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/app/app.py", line 15, in application
    time.sleep(seconds)

What's next

None of this needs anything outside the image. There is no init to add, no health check view to write, no timeouts to line up with the platform's and no temporary directory to remember. What it does need to be told is how many workers of how many threads to run, chosen to fit the resources the container is given.

Most containers end up on Kubernetes though, and there are a few more things there where Muru can work with what the platform provides rather than around it, such as the shutdown delay in place of the usual preStop hook, autoscaling on how busy the workers really are rather than on CPU, and running migrations as a Job from the new image. That is the subject of the next post, working with Kubernetes, not around it.

November 14, 2026 12:00 AM UTC

November 13, 2026

James Bennett

Boring Python: dependency management

This post has been updated since it was originally written. The most recent update was in November 2026; a summary of what changed is at the end.

This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.

So before I go any further, I want to be absolutely clear on what I mean by “boring”: I don’t mean “reliable” or “bug-free” or “no incidents”. While there is some overlap, and some of the things I’ll be recommending can help to reduce bugs, I also want to be clear: there will be bugs. There will be incidents where a feature or maybe an entire service is down. “Boring”, to me, is about the sources of those incidents. It’s difficult enough to manage your own code and the bugs and other problems that will inevitably pop up in it from time to time; you don’t want to compound that by having bugs or other surprises coming from the tools and processes you use to build, manage, and deploy it. So when I call something “boring”, I mean it’s unlikely to add another source of bugs and nasty surprises that cause a pager to go off at 2AM; the pager will, of course, eventually go off at 2AM, but when it does you’ll be able to feel reasonably confident (if that’s the right word for such a situation) that the source of it was something in your own code that you can diagnose and fix.

And so I’m planning several posts exploring different aspects of making Python development boring (in this sense). But for this first installment, I’ll be talking about one of the internet’s favorite topics: managing dependencies.

If you’re just interested in the end result, the process I recommend is pretty simple and can be found in the section titled “Putting it all together”. The rest of this post exists to explain, in as much detail as I can manage, all the things going on behind those simple-looking recommendations, and why I make those specific recommendations.

Python packaging: a brief overview

I’ve written about this before, but to make sure it’s clear I want to do a quick refresher.

As I see it, there are three basic aspects to “packaging”, and one of the difficult things about “Python packaging” discussions is lack of clarity around which are being discussed. They are:

  1. Given some working code, define and produce from it a distributable artifact.
  2. Given that someone has produced a distributable artifact, use it to cause the corresponding code to appear, in working form, somewhere else.
  3. Given the existence of potentially many separate projects or versions of projects each with their own independent and potentially-conflicting sets of dependencies, make it possible to work on and run more than one at a time, on the same system.

Most of the actual remaining complexity in “Python packaging” today is concentrated in item (1), and even then only really kicks in if you’re packaging extensions written in non-Python languages.

A lot of the perceived complexity of “Python packaging” is due to its long history, evolving expectations over the course of that history, and the resulting pile of strata of different things people have come up with to try to suit the expectations of different eras.

Much of the public grumbling and complaining about “Python packaging” these days is due to the lack of a single, standard first-party, all-in-one high-level tool to handle all aspects of packaging, and the proliferation of third-party tools to try to fill that niche.

Finally, most of the meaningful work in Python packaging, for quite a while now, has been the unglamorous, underappreciated slog of people on mailing lists and discussion forums patiently working out what Python packaging really should look like and thoroughly specifying/standardizing it, but with the added thrill of all that having to occur in a way that doesn’t break things for the vast number of people relying on Python and its huge package ecosystem for their day-to-day work.

To the extent that we now have a bunch of shiny fancy flashy third-party all-in-one packaging tools, it’s because of that heroic unsung work that’s been quietly going on for years. The fancy new tools get to be so fancy and flashy in large part because modern Python packaging has such strong foundations (and also because, unlike the pre-existing tools, the new ones don’t have to maintain compatibility with literal decades of historical features and configuration options, all of which have been relied on, at some point, by someone).

If I went into detail on everything I’d wind up with several posts’ worth of material, so let me just pick out a few things that I think are the most crucial underpinnings of modern Python packaging:

Boring Python packaging: a philosophy

When I wrote the first version of this post in 2022, I focused on defining a workflow that relied as much as possible on standard first-party (as in, included with Python itself) tooling. I recommended exactly one third-party tool, and only as an optional convenience; the functionality it provided could also be obtained from chaining together invocations of first-party tooling.

That focus on first-party tooling was deliberate. As I wrote at the time:

[The] core default tools I’ve mentioned have all been around and stable for a long time, in software terms, at least — they’re all over a decade old, and their end-user interfaces evolve incredibly slowly, when they evolve at all. They’re reliable. They’re well-understood. That’s exactly what I want and exactly what I recommend.

This is in line with what I said at the start of this post about wanting things to be boring. And my stance hasn’t changed, but advances in the foundational standards of Python packaging do let me be a bit more flexible. So although I’m still going to be focused on first-party tools I’ll now point out a couple of places where I think it’s safe to use some of the hip and trendy new third-party tools if you want the added conveniences they offer.

Also, I should note here that my use case for Python is deploying networked (primarily web) services, on servers. If you primarily use Python for, say, machine learning or data science, you are likely to already be a happy user of a completely different world of tools optimized for those use cases (in particular, for the much greater amount of non-Python code wrapped by/accessed through Python in those fields and for their different patterns of code reuse and sharing). Please continue using them! They suit your use case really well, and replacing them with my preferred tools and workflow would probably be a regression for you.

Now, let’s get started.

Lock it up

The standard Python package manager is pip. And my eventual goal is to invoke pip install to, well, do what it says: install a bunch of Python packages. But how to tell pip which packages to install? There are a couple of options:

Historically, the second option was referred to as a “requirements file”, and by convention was generally named requirements.txt. Now that PEP 751 has defined a standard Python lock file format, and now that pip supports producing and consuming that standard lock file format, that’s the format I recommend.

This prompts a question, though: since dependencies could already be specified as part of the metadata in pyproject.toml, why did we need another file format to record them? By way of an answer, consider a web application written using the Django web framework. You might think it’s simple enough to just declare Django as a dependency in your pyproject.toml file, like so:

[project]
dependencies = ["Django"]

But if I create a new virtual environment and run pip install Django inside it, three packages get installed:

$ pip install Django
Collecting Django
  Using cached django-6.1-py3-none-any.whl.metadata (3.9 kB)
Collecting asgiref>=3.9.1 (from Django)
  Using cached asgiref-3.12.1-py3-none-any.whl.metadata (9.4 kB)
Collecting sqlparse>=0.5.0 (from Django)
  Using cached sqlparse-0.6.0-py3-none-any.whl.metadata (6.0 kB)
Using cached django-6.1-py3-none-any.whl (8.4 MB)
Using cached asgiref-3.12.1-py3-none-any.whl (25 kB)
Using cached sqlparse-0.6.0-py3-none-any.whl (50 kB)
Installing collected packages: sqlparse, asgiref, Django
Successfully installed Django-6.1 asgiref-3.12.1 sqlparse-0.6.0

And which versions of those packages I get will depend on when I run pip install. Right now I get Django 6.1, but Django does frequent bug fix releases, so if I re-run that pip install in the future I’ll get Django 6.1.1, and further in the future I’ll get Django 6.1.2, and so on. If I test my application locally with the set of packages I installed today, and then deploy it to a server later using pip install to fetch dependencies, the deployment will very likely get a different set of packages than I used during development, which is a potential source of problems.

So the key difference is that the pyproject.toml file typically specifies a list of direct dependencies, often with broad version constraints or none at all, while a lock file specifies an entire environment, with all direct and transitive dependencies resolved and pinned to exact versions along with the expected checksums of the packages, to be reproduced as exactly as possible.

There are two main approaches to generating a lock file:

Managing your local environment and producing your lock file is one of the areas where I think it’s generally OK to use a third-party tool. If something goes wrong with it, it might temporarily mess with your ability to get work done on your own computer, but that won’t break your actual deployed application.

The hip and trendy third-party all-in-one tool these days is uv, which has its own tool-specific lock file format (uv.lock), but also has a uv export command which can export to the standard pylock.toml format.

My own personal preference, and the tool I use on my own projects, is PDM, which has a similar feature set to uv and can export to pylock.toml from its own lock format, but also can just manage a project directly in a pylock.toml file.

As long as the tool you choose is one you and your team are all happy with, I think you can pick anything that can produce the standard lock file format and be OK. Also, you should probably automate the process of exporting the lock file, to ensure developers don’t have to remember how to manually do it. I personally like to include a Makefile in most projects, with a target specifically to refresh the lock file (make lock, for example), but the idea is portable to a lot of task-automation tools, so it’s easy to make it work with whatever your team likes to use.

I also want to point out here that if you’re deploying services written in Python, you can build them as distributable packages and then install those packages to automatically pull their dependencies, rather than using a lock file. I just don’t recommend this, for a few reasons:

Getting testy

In addition to direct dependencies to run an application, it’s fairly common to have additional dependencies needed to run a test suite, linters, or other development and quality-control tasks. But there’s no need for a production deployment to have all those extra dependencies installed, and in general the less stuff you put in your production environment the fewer things you have that can go wrong, so I generally like to install those only when the test suite (or other task requiring extra dependencies) is actually going to be run.

The standard solution for this is dependency groups, which were first defined for library-style package dependencies but are also part of the lock file specification. And as of pip version 26.3, selecting groups when installing from a lock file is now supported. Popular third-party package-manager frontends also seem to have broad support now, in late 2026, for specifying dependency groups and exporting them to a standard lock file, so you can manage multiple logical groups of dependencies with ease.

The general consensus among the third-party tools seems to be to automatically support a group named dev for dependencies used only in development/testing, so my recommendation is to go with that convention rather than manually creating your own group with a different name.

Here’s an example with PDM of how you could set up a project to use Django as the web framework in production, and the pytest-django library for testing:

pdm add Django
pdm add --dev pytest-django

And then you can export to a standard lock file like so:

pdm export --format pylock -o pylock.toml

The lock file format is pretty readable, so you can look inside and see that it specifies multiple groups (one named default and one named dev) and tracks which dependencies belong to which group(s). The exact commands for adding packages to and exporting a “dev” group with other tools will vary, but should generally look pretty similar to what’s above.

Using the right invocations

Before putting all this together, it’s worth covering one more detail: how to invoke the tools. This may seem a bit silly, since they all provide executable entry points. Just run pip install, right?

But there’s a potential issue here: in a moment I’m going to recommend creating a Python virtual environment, which opens up the possibility of multiple Python environments coexisting on the same machine. This is certainly a useful feature, but it comes with a new concern, which is how to ensure you’re using and running things in the environment you expect to be using.

For example, one way to run into trouble with multiple Python environments is to have one environment’s package directory be first on your $PYTHONPATH (which controls Python import locations) while a different one’s bin/ directory is first on your general $PATH (which is where executable scripts will be found). If you just run pip install you’ll get the second one’s instance of pip, which may not be at all what you want.

This is why the official Python packaging guides, and official documentation for tools like pip, always use a different approach: they tell you to run python -m pip instead of pip, and python -m venv instead of a standalone script like virtualenv. The -m flag allows a Python module to be run directly like a script (as long as it’s been written to provide an entry point for this, which pip and venv both have), and can prevent a lot of potential hard-to-debug issues that can accidentally result from things like manually hacking around with paths, by ensuring you’re getting the version of pip or venv that actually goes with the Python environment you invoked.

And to guarantee that you get the Python environment you actually want, you can specify the full path. For example, many base Python container images will put the Python interpreter in /usr/local/bin, so invoking /usr/local/bin/python instead of just python ensures you get that interpreter.

There’s also one more thing I like to do that isn’t (currently) in some of the popular guides, and that’s invoking Python with the -I flag. This runs Python in “isolated” mode, which removes some of the automatic implicit directories from the import path and also ignores environment variables like PYTHONPATH. Once again, this reduces the number of things that can go wrong (for example, with -I the current working directory won’t be implicitly added to the import path, so you can’t accidentally depend on it being importable). So whenever you can invoke Python in isolated mode, I generally recommend that you do.

Putting it all together

That was a lot of explanation for what ends up being, ultimately, a pretty simple process to actually use. So now let’s finally take a look at it.

First things first: always work in, and always deploy in, a virtual environment. Even if you think you don’t need one. In fact, especially if you don’t think you need one. This may seem like strange advice if you’re already using a container or other virtual machine, since you’re probably thinking that provides all the isolation you’ll need. But virtual environments don’t cost you anything to create, and if you ever do end up with multiple Python interpreters—which is easy to accidentally do, if you use a base system with a purpose-built Python and then install a system package that turns out to depend on the distro’s own Python, for example—using one from the start will help to save you from potentially having a pager go off one night when suddenly the wrong Python is being invoked.

Virtual environments also provide a useful “Python environment” artifact that can be copied between stages of a container build, and many tools automatically recognize and can work with them. And ever since PEP 668 started to be adopted by operating-system vendors, many “system” Python installations will require you to use a virtual environment in order to install packages with pip (and only allow you to interact with the “system” Python environment through the system’s own package manager). So use a virtual environment, even if you’re working in a container or other VM.

When you’re working in a local directory on your own computer, you can invoke the correct Python version with -m venv to create a virtual environment and install things into it, but again this is an area where third-party package tools can be useful, because they’ll manage this for you automatically. If you do create your local virtual environment manually, you should almost certainly put it inside your project in a subdirectory named .venv, because that’s already the common unofficial convention supported by a lot of IDEs and other tools, and likely to become the official standard convention once PEP 832 finalizes.

In a container, I think the choice that’s most in line with Linux filesystem hierarchy standards is to put a Python virtual environment somewhere under /opt. Generally I like to create an /opt/venvs in a base “build” stage, which can populate multiple virtual environments with different package sets to be copied into later stages for tasks like CI or deployment.

So, taking as an example current Debian stable (as I write this, Debian 13 “trixie”) and the most recent Python (as I write this, Python 3.15), let’s see what this actually looks like. This example assumes you’re producing a standard Python lock file as described above:

# syntax=docker/dockerfile:1

ARG DISTRO="slim-trixie"
ARG PYTHON_VERSION="3.15"

FROM python:${PYTHON_VERSION}-${DISTRO} AS dependencies

# Turn off pip upgrade reminders, since we're about to upgrade it
# anyway, and also specify the root cache directory pip under which
# will create its package cache.
ENV PIP_DISABLE_VERSION_CHECK=1 \
    XDG_CACHE_HOME=/var/cache

RUN <<END
mkdir -p /opt/venvs/deploy
mkdir -p /opt/venvs/tests
# Ensure the pip cache directory exists.
mkdir -p ${XDG_CACHE_HOME}/pip
/usr/local/bin/python -Im venv /opt/venvs/deploy
/usr/local/bin/python -Im venv /opt/venvs/tests
END

COPY pylock.toml  /opt/venvs

The next step is to ensure pip is present at the latest version in each virtual environment, and invoke it to install packages.

At this point you might be wondering: if I was willing to recommend using a third-party packaging frontend earlier for local development use and producing lock files, why am I insisting on pip here? And the answer is this is one of the places where I think using boring standard default tools really matters. As I said above, if something goes wrong with your fancy third-party package tool in a local development setup, it just causes a problem for that local development setup. If it goes wrong in your production build/deploy process, it breaks all your builds and deployments until you resolve the issue. I want to avoid that, so I stick to pip here, because even if it’s not as fancy as some of the newer third-party tools, in my experience it’s much less likely to be a source of unexpected issues than the newer third-party alternatives.

Also, I’m not going to go into a full explanation of an ideal Python application Dockerfile here, but the example below does at least use a Docker cache mount to store the downloaded packages, so that subsequent rebuilds only have to download new or changed packages, and can pull everything else from cache.

The pip invocation adds a few flags, mostly to ensure maximum safety and reproducibility:

If you’re using GitHub Actions as your CI/CD, Python core developer Brett Cannon has written a reusable action which automatically invokes pip with these flags.

But continuing with the example of a Dockerfile containing a “dependencies” stage, here’s what it looks like. Note the difference between the two pip install invocations, though. Asking pip to install from a lock file with -r pylock.toml will only install the default dependency group, and for the deployment dependencies, that’s the right behavior, but for the test dependencies it isn’t, so the test dependencies use the --group flag to explicitly select the set of groups to install (both default and dev, in this case).

# Install test dependencies first since they're likely to be a
# superset of the deployment dependencies and will populate the
# package cache.
RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/tests/bin/python -Im pip install --upgrade pip
/opt/venvs/tests/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    --group /opt/venvs/pylock.toml:default \
    --group /opt/venvs/pylock.toml:dev
END

RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/deploy/bin/python -Im pip install --upgrade pip
/opt/venvs/deploy/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    -r /opt/venvs/pylock.toml
END

Now you can pull the installed packages into later stages of your build by copying the virtual environment. For example, in a deployment stage:

FROM python:${PYTHON_VERSION}-${DISTRO} AS deploy

COPY --from=dependencies /opt/venvs/deploy /opt/venvs/deploy

# Do the rest of your deployment stage setup here: copying in
# your application source code, setting the entry point, etc.

Once again, I’m not going to go into a ton of detail here on other things that make a good Python Dockerfile. If you want to learn more about that, I’d recommend reading Itamar Turner-Trauring’s articles on Python and Docker or Hynek Schlawack’s guide to Python and Docker. Both are regularly updated, and they’ll both give you a solid education in how to containerize Python.

Be up-to-date, but stay cool

The only thing still missing here is how to handle updates as new versions of your dependencies are released. For security updates this is crucial, but it’s also important as a general practice. If you make dependency updates a regular, routine part of your development process that’s easy for developers to do, then it’ll also be routine and easy to apply critical updates when they appear. And by applying updates as they come, a couple dependencies at a time, you avoid building up a huge backlog of deferred updates that will make a critical issue even more difficult to address when one inevitably occurs.

And luckily, all of the popular third-party package management tools have straightforward commands you can run to identify and update outdated packages. But there is one wrinkle: you probably don’t want to always eagerly accept most routine dependency updates as soon as they’re released, since that can expose you to potential security issues if a dependency has been compromised but nobody’s noticed it yet. The solution to this is dependency cooldowns: when upgrading, only consider packages which have had at least a bit of time for security researchers and scanners to take a look. The typical recommended window is three days; anything newer than that should be excluded from your package updates unless it’s a critical security patch.

At this point, most popular tools in the Python ecosystem support dependency cooldowns. For example:

You also can configure third-party dependency update bots to apply updates for you as they appear. GitHub’s Dependabot automatically applies a three-day cooldown for everything except critical security updates, and so does the Renovate update bot.

Whether to have a dependency update be a periodic manual task for a developer or something to automate via a tool like Dependabot is up to you (though I personally prefer automation). The important thing is you do it and commit to it, on a cadence that lets you take in updates only one or a few at a time, rather than falling behind and needing to update dozens of packages all in one go.

And that’s a wrap

As promised, that was a lot of words for what’s really a pretty simple set of recommendations. Unfortunately there’s a lot of complexity—necessary complexity, of a sort that pops up in any software packaging ecosystem—lurking in this topic, and explaining all that is what drives up the word count.

But hopefully you now understand how to do “boring” Python dependency management, relying primarily on standard first-party tooling. Even if you don’t want to adopt my recommendations, I’d like to think that learning what they are and why I make them is helpful to you. For me, these recommendations are the result of over a decade of work, across multiple employers, to develop a dependency management workflow that keeps things up-to-date with minimal risk of causing pagers to go off.

Meanwhile I’ve got some ideas for further “boring Python” articles, but those will have to wait for another day.

Changelog

The first version of this post was written and published in May 2022. A major revision was written during late August 2026 and published in early September 2026. The key changes between those versions were:

A second update was written and published in November 2026. This was a smaller change, in response to pip 26.3 adding support for selecting and installing specific dependency groups from a lock file. Previously, the lack of this support led me to recommend the use of multiple lock files to track separate deployment/test/etc. dependencies; now, with dependency groups supported in pip, I can recommend the simpler approach of a single lock file containing multiple dependency groups, and instruct pip to install the correct group(s) for a given use case.

November 13, 2026 08:23 PM UTC

Robin Wilson

I released Cloud-Native Environment Agency DEM data!

At the FOSS4G UK 2026 conference in October, I gave a talk announcing that I’d re-released the Environment Agency DEM data for England in cloud-native formats. This was covering some work I did for my client Rebalance Earth and that we’ve decided to release openly.

file

If you already know what I’m talking about then you can find the data on Source Cooperative as COGs with STAC indexes. The README on that page will explain how to access the data, and you can access an interactive webmap of the data too. If that didn’t make full sense to you, then read on…

I’m not going to go into full details here, as sometime soon the video recording of my talk at the conference should be released, and I can link to that here. In the meantime, my slides are available and may help with understanding some of what I explain below.

So, basically, the Environment Agency (EA) collect LiDAR data across England and process it to produce Digital Elevation Model (DEMs). DEMs are rasters where every pixel contains a height value, and the EA produce two types: a Digital Terrain Model which is just the height of the terrain, and a Digital Surface Model which includes the height of everything on top of the ground – like buildings, trees and so on.

The EA provide access to this data under the Open Government License, but their web interface for getting hold of the data is a bit of a pain – you can’t download much data at once. You can ask for bulk access to the data, but even then you have to work out exactly what OS 5x5km grid squares you want, and the whole thing is a pain.

As the data is released under the OGL, people are allowed to re-release processed versions of it under the same license. So, I converted the files to modern cloud-native formats, specifically Cloud Optimized GeoTIFF with a Spatio-Temporal Asset Catalog. These formats store the data efficiently, allowing small parts of the image to be easily downloaded, as well as ‘zoomed-out’ lower-resolution versions of the data.

The upshot of all of this is that you can use a few lines of Python or a GDAL command-line to download and mosaic the DEM over any area you choose, and you can also display the data easily on a web map. The latter is quite impressive – you can see an example webmap here, which does live rescaling in the browser, along with hillshading and colour blending, producing a far nicer output than the EA’s viewer – the example below is the DSM over Bristol (the location of next year’s FOSS4G international conference!)

I’ll leave the detailed documentation on how to use this to the README file in the data repository, but a few quick examples will show how easy it is to use.

You can use a command like this to download and mosaic all of the DSM data in a particular bounding box:

gdal raster clip \
  --bbox 438000,108000,446000,116000 \
  GTI:/vsicurl/https://data.source.coop/rebalance-earth/ea-dem/ea-lidar-fz-dsm-1m-items.parquet output.tif

You can also do similarly in Python using the lazycogs library:

import lazycogs

da = lazycogs.open(
            “items.parquet”,
            bbox=lonlat_to_bng_bbox(bbox_lonlat),
            crs="EPSG:27700", 
            resolution=1.0, 
            chunks=chunks)

da = da.squeeze(["band", "time"], drop=True)

which gives you a nice xarray DataArray to play with.

The talk went down very well, and was described by some people as the "highlight of the conference" (including Alastair Graham on the Geomob Podcast). The data has also been downloaded quite a lot since the conference – there have been over 1 million individual downloads, with over 4TB of data transferred. Given that each tile of DEM data is only around 80MB in size then that’s the equivalent of half a million full tiles (the whole of England is only around 5,500 tiles).

I’m quite chuffed with this – it’s something I’ve wanted to do for ages but never quite got round to, until we found that my client Rebalance Earth needed something like this, and decided we could release it openly. I want to thank Rebalance Earth for paying me to work on this (alongside lots of other stuff) and covering my costs of attending the conference to present it.

November 13, 2026 07:13 PM UTC

Mike C. Fletcher

Tim Peters will be missed

Tim spent a very long time in the early days of Python List teaching me all sorts of things I didn't know from my father's scattershot teaching as a child. He was generous with his time, full of humour and willing to teach some random kid on the internet when he had lots of other important things he could be doing.

Spending ages with him and Christian trying to optimise some random function to the Nth degree to work around Python's quirks is one of my fondest memories from that era of my life.

November 13, 2026 05:28 PM UTC

PyOpenGL 4.0.0b1 Released

So we seem to have reached mostly-stable state for the PyOpenGL 4.0.0 release. There's no other planned major changes to the core library, so unless we see significant regressions we'll likely release 4.0.0 final over the next few weeks.

Changes since the 4.0.0a5 release last mentioned here:

Bug/crash fixes:

* OSMesa -- a dropped OffscreenContext cleans up and prevents further writes
* Buffers -- a memoryview upload sent the wrapper instead of the data; this is fixed
* Buffers -- use-after-free in the buffers format handler is fixed
* Offsets -- an integer where a const void* is wanted is treated as a buffer offset (regression)
* Arrays -- bare numbers are accepted as input types again (regression)
* Arrays -- an input array is checked against the length count
* Dispatch -- a function derived from a C entry point keeps its own customisations
* Dispatch -- define() applies the annotation table to every command
* Dispatch -- context handles now compares equal to themselves
* Bindings -- glGetUniformIndices takes names and returns indices
* Bindings -- GL 3.0 pixel formats (GL_RG and similar) are sized without importing their extension module
* Bindings -- A wrapper that names a parameter its entry point lacks is handled
* OpenGL.GLES2 imports with error checking switched off (regression)
* Errors -- error messages are bounded size for all wrappers
* Errors -- NullFunctionError says which of three things went wrong
* Errors -- a platform plugin or format handler that will not import now reports which one and why
* Errors -- a config flag set too late says so rather than silently failing to activate
* Python 3.15 -- no longer reads ctypes.__version__ and 3.15 is added to CI/tox

VBO Helper

* References and aliases -- VBO attributes hold their own references
* VBO and VBOOffset are accepted by every array alias
* vbo.VBO is a class to type checkers
* Context sharing -- dispatch.share_context, gives contexts that share objects one identity

Platform Support

* Tk GLFrame for macOS (Aqua), working on Apple silicon (only tested in CI however)
* Tk Contexts take a swapInterval
* OSMesa now runs on Windows
* EGL -- the gbm render node is opened read-write
* Windows -- a GL library beside the interpreter is preferred to the system one (ANGLE support)
* Windows -- user-installed freeglut is found by its own name
* Windows -- opengl32 is loaded from Windows rather than from PATH
* Windows -- freeglut and GLE binaries left the main wheel. pip install PyOpenGL[glut] brings them back
* Namespaces -- a platform-specific API says which platform serves it

Packaging and CI

* Wheels -- arm64, aarch64 and Intel Mac wheels are built
* PyOpenGL_accelerate now has the same version as PyOpenGL, run-time checks it
* sdist no longer ships Cython output
* Test and CI setup -- the suite runs on OSMesa, and CI runs it there
* macOS test suite that never ran is not counted as a failing suite.

Tests, tooling and docs

* Tests -- a large batch of added tests. Many are tickets from the github tracker
* Lint -- ruff cleanups, typing cleanups
* Docs -- lots of updates, particularly around how we're generating them

November 13, 2026 04:11 PM UTC

PyCharm

What’s Fixed and Improved in PyCharm 2026.3

PyCharm 2026.3 is here! Alongside major features like JetBrains Air, workspaces, Python 3.15 lazy imports, and the redesigned interpreter widget, this release includes 303 fixes and improvements to everyday Python development.

From smarter code completion to more reliable environments and debugging, here are some of the changes worth knowing about—including fixes for issues reported more than a decade ago.

Smarter code insight and type inference

Better support for functools.partial

functools.partial lets you create a new callable by supplying some of a function’s arguments in advance. Previously, PyCharm could struggle to infer the resulting signature, leading to missing parameter information and incorrect warnings.

PyCharm now understands the remaining parameters, checks their types, and recognizes partial objects as valid Callable instances. In the example below, PyCharm correctly identifies the remaining parameter and flags an incompatible argument.

PyCharm understands the remaining parameters of a functools.partial object.

This resolves several long-standing issues with completion, parameter information, and type checking (PY-35363, PY-33618, PY-25698, PY-36385).

Fewer false positives for generics and collections

PyCharm now provides more accurate type inference for generic classes, bounded and constrained type variables, overloaded constructors, and generic __getattr__ methods. For example, calling a method on a bounded type variable now preserves its return type instead of falling back to Any (PY-89863).

Collection inference has improved, too. Nested lists, optional values, dictionary literals, and tuple unpacking produce fewer false warnings (PY-89564, PY-91999). There are also improvements to Literal, Enum, and TypedDict, along with support for PEP 747’s TypeForm[T] and intersection types through ty_extensions.Intersection (PY-90287, PY-62524, PY-89043, PY-87024).

Clearer type warnings and suppression comments

When PyCharm finds a type mismatch, its tooltips now provide clearer explanations, improved formatting, and links to the relevant types. Call and control-flow analysis also handles more cases involving optional callables, bound methods, and paths that never return, reducing false warnings (PY-90555, PY-83644).

When a warning is intentional, you have more control over suppressing it. PyCharm now supports shorter codes in # type: ignore[...] comments and introduces # pycharm: ignore[...] for individual PyCharm inspections. There’s also a Suppress with ‘type: ignore’ comment action, and unknown suppression codes are flagged to catch typos (PY-90787, PY-92114).

PyCharm 2026.3 introduces a new action to suppress individual type warnings using # type: ignore[...] comments.

Testing and developer tools

Better completion for pytest and mocks

Mocks and fixtures can be tricky for an IDE to understand. PyCharm now provides completion for the pytest-mock mocker fixture, navigation for mocker.patch() targets, and completion based on classes passed to Mock(spec=...) or Mock(spec_set=...).

from unittest.mock import Mock

class Client:
    def fetch(self, url: str) -> str:
        return "OK"

client = Mock(spec=Client)
client.fetch("https://example.com")

PyCharm 2026.3 provides completion for methods defined in classes passed to Mock(spec=...).

PyCharm also flags invalid unittest.mock.patch() targets and improves completion and documentation for pytest fixtures (PY-36260, PY-32282).

For projects with custom test infrastructure, you can configure a unittest runner script, such as Django’s tests/runtests.py. If it uses unittest.TextTestRunner, test results appear in PyCharm’s test tree (PY-26068).

More capable type-checker and Ruff integrations

Zuban, a mypy-compatible type checker and language server, joins ty, Pyrefly, and basedpyright as a supported external tool. Its integration provides diagnostics, completion, hover information, inlay hints, and quick fixes. Pyrefly also receives fixes for custom import roots and missing-package installation (PY-85009, PY-89805).

With Ruff, Optimize Imports now removes unused imports and organizes them. Ruff fixes can run automatically on save or before a commit, and the new Apply Ruff Fixes action works across a selected scope, including an entire project. Configuration discovery is also more reliable in multi-root, network, and WSL projects (PY-85534, PY-86352).

Apply Ruff fixes to an individual file, custom scope, or entire project directly from PyCharm.

Django, FastAPI, and SQLAlchemy

Django: You can create run configurations for Django 6.0 tasks, and TextChoices members defined with value-and-label tuples no longer trigger false type errors (PY-86368, PY-92177).

In Django REST framework, rest_framework.reverse.reverse() now offers URL-name completion and navigation, just like Django’s built-in reverse() (PY-78900).

PyCharm 2026.3 provides URL-name completion for Django REST framework’s reverse() function.

FastAPI: Alongside the new interactive dependency graphs, path parameters supplied through dependencies no longer cause incorrect warnings about missing endpoint parameters (PY-88935).

SQLAlchemy: Expanded ORM support improves completion, navigation, and type inference for declarative models. Fixes also cover attributes added through relationship() and columns that use SQLAlchemy’s ENUM type (PY-79736, PY-29212).

Editing and refactoring

Several small fixes make editing more predictable. Pressing Enter after a comment introducing a block now inserts the correct indentation, while moving lines in tab-indented files no longer causes false Unexpected indent warnings (PY-15693, PY-78251).

A new Align consecutive assignments formatter option lines up assignment operators. You can enable it under Settings | Editor | Code Style | Python | Wrapping and Braces. The formatter also does a better job preserving configured blank lines (PY-34006).

Refactoring is safer, too. Inline Function preserves fallback behavior for omitted optional arguments, while Implement abstract methods generates valid implementations of asynchronous abstract methods (PY-81983, PY-53074).

More reliable environments and faster startup

conda: Environment variables configured using conda env config vars set or activate.d scripts are now available when you run scripts, tests, and the Python console from PyCharm. Previously, variables could work in a terminal but be missing inside the IDE—particularly frustrating for projects using GDAL, QGIS, PyMC, or compiler toolchains (PY-55405, PY-38751).

uv: Workspaces using a shared root .venv retain their interpreter after upgrading from PyCharm 2026.1. Projects that use uv pip without a pyproject.toml now use the correct installation command, creating a uv environment no longer requires uv init, and new environments can optionally access system site-packages (PY-91108, PY-85589).

Performance: Python scripts start faster thanks to reduced helper overhead. In particular, simply having matplotlib installed no longer adds approximately 100 milliseconds to the startup of an otherwise empty script (PY-92211).

Debugging improvements

The debugpy debugger mode now supports Set Value and Jump to Cursor. Other improvements include sorting variables by type, displaying plots in the Plots tool window, accepting interactive input with terminal emulation, and using Drop into debugger for failed tests.

In the example below, we’ve paused execution and changed discount from 0.10 to 0.25, allowing us to test a different result without restarting the program.

Changing a variable’s value while debugging in debugpy mode.

Debugging also works more reliably for tests in WSL, applications launched with fastapi dev, and long-running IDE sessions on macOS (PY-90157, PY-90281, PY-90588).

Elsewhere in the debugger, dictionary keys appear in insertion order, Python 3.15’s frozendict can be expanded, and the debug console supports a configurable starting script (PY-75697, PY-21419).

Easier Jupyter notebook workflows

Find now searches cell outputs as well as source code, so you can locate a result without scrolling through the notebook. You can also move several selected cells together or use Extract to Python File to copy them into a regular Python file. A bug that caused scripts embedded in HTML outputs to execute twice has been fixed, too (PY-84634, PY-89385).

Thank you for your feedback

Not every meaningful improvement is a headline feature. Sometimes it’s a false warning disappearing, completion working where it didn’t before, or a debugger behaving as expected. Some fixes in this release trace back to issues reported more than a decade ago.

Download PyCharm 2026.3 and try these improvements in your projects. And if PyCharm gets something wrong, report it in YouTrack. Your reports and votes help us decide what to improve next.

Thanks for using PyCharm, and happy coding!

November 13, 2026 02:28 PM UTC

Python Software Foundation

Python is for everyone: Join in the PSF year-end fundraiser & membership drive!

The Python Software Foundation (PSF) is the charitable organization behind Python, dedicated to advancing, supporting, and protecting the Python programming language and the community that sustains it. That mission is more than words to us; it’s the community we show up for every day. Our tiny but mighty team works hard to deliver the projects, services, and programs that allow Python to be the thriving, independent, community-driven language it is today. Some of what the PSF does includes producing PyCon US, hosting the Python Package Index (PyPI), supporting Developers in Residence, maintaining critical community infrastructure, and more.

Python is for gathering, teaching, learning, playing, researching, exploring, creating, building, working, sustaining. The list goes on and on and on! Support this year’s fundraiser with your donations and memberships to help the PSF, the Python community, and the language stay strong and sustainable. Because Python is for everyone, thanks to you. 

 

There are two direct ways to join through donate.python.org: 

>>> Donate or Become a Member Today! <<<

 
  

If you already donated and/or you’re already a member, you can:

 

Your donations and support:

 

PSF highlights from 2026:

November 13, 2026 08:41 AM UTC

Graham Dumpleton

Muru: canaries without a service mesh

In releases, not restarts I showed the Muru daemon deploying a new release beside the one serving, and only moving traffic across once the new release was ready. That protects you from a release which can't start. It does nothing for a release which starts perfectly well and then fails some of the requests it is given, which in my experience is the more common sort of bad deploy. A health check passes, the deploy reports success, and the first you hear about it is from the people using the site.

The usual answer is a canary. The new release is given a small share of the traffic first, is compared with the release already serving, and only gets more if it holds up. Automating that is normally something added around the server rather than something the server does. On Kubernetes it means something like Argo Rollouts or Flagger, typically with a service mesh or an ingress controller doing the traffic split and Prometheus queries deciding whether the new version is healthy. Off Kubernetes it tends to mean load balancer weights and someone watching a dashboard. Gunicorn, uvicorn and the other Python servers have nothing to say about it, since none of them has any notion of two versions of an application being compared.

Under the Muru daemon, the edge already splits traffic between releases and already counts what each release does with the requests it is given, so the daemon can run the canary itself.

A release with a bug

Since the last post the shop has gained a page for each order, deployed as release r5 the way that post showed. The next change adds prices to those pages:

PRICES = {"teapot": 45.00, "kettle": 60.00}


def order(db, order_id):
    row = db.execute("SELECT item, quantity FROM orders WHERE id = ?", (order_id,)).fetchone()
    if row is None:
        return None
    item, quantity = row
    total = quantity * PRICES[item]
    return f"Order {order_id}: **{quantity} x {item}** for ${total:.2f}"

Someone forgot about mugs. The order pages for teapots and kettles work, the home page works, and the page for the one order of a mug raises a KeyError. The release starts without a problem, so a plain deploy would put it in front of everyone.

Instead it is deployed with --hold, which starts the release but leaves it out of traffic:

muructl deploy shop releases/v6 --hold

muru: release shop@r6 is ready and held: not serving

In the last post I used the Muru-Preview header to reach a held release, and doing that here would find the bug for anyone who thought to check the right page:

curl -H 'Muru-Preview: r6' http://127.0.0.1:8080/orders/1
curl -H 'Muru-Preview: r6' http://127.0.0.1:8080/orders/3

<p>Order 1: <strong>1 x teapot</strong> for $45.00, served by shop@r6</p>
500 Internal Server Error
reason: app_error
request id: 01M6TE1RN0DZ6YEXWKNV4W5AKN

The application raised an exception.

The point of a canary is not having to know which page to check.

Running a canary

A canary is judged on real requests, so to stand in for people using the site I ran a small script with four clients, each requesting one of the home page and the three order pages at random, as fast as the shop would answer them, for 25 seconds. The script doesn't keep cookies, which matters below. With it running:

muructl rollout canary shop@r6 --steps 10,50 --hold 30s
muructl rollout status shop --wait

muru: rollout 1 of shop@r6 started against r5: step 1 of 2: 10% for 30s
muru: rollout 1 running step 1 of 2: 10% for 30s
muru: rollout 1 rolled_back: gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)

The plan was to give the new release 10% of the requests for 30 seconds, then 50% for another 30, and then all of them. It never got past the first step. The events show how quickly it was decided:

    44 18:20:02 notice   rollout.started      shop@r6                  rollout 1 against r5: 10% for 30s, 50% for 30s
    45 18:20:07 warning  rollout.gate_failed  shop@r6                  step 1 at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)
    46 18:20:07 warning  rollout.rolled_back  shop@r6                  rollout 1: gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%); the candidate is kept for inspection

Five seconds after the canary started, the candidate was out of traffic again. Over the run the script got this back:

{200: 6618, 500: 27}

so 27 requests failed out of 6,645. A plain deploy would have failed a quarter of the order page requests for everyone until somebody noticed and rolled back.

What decided it is what Muru calls a gate. The edge keeps counters for every release, of the requests it answered, those answered with a 5xx status, and histograms of how long requests took overall and how long they waited for a worker. During a step the daemon reads those for the candidate and for the baseline, the release serving everything else, over the same window since the step began. Once the candidate has served 100 requests in the step, the gates are checked every second, and a failing gate rolls back straight away rather than waiting for the hold to end.

Importantly, a gate only fails if the candidate is both over its limit and worse than the baseline. If the database goes away and both releases start failing, that isn't the candidate's fault, and it isn't rolled back for it. The candidate is also kept running out of traffic, so its log can still be looked at, which muructl logs shop@r6 shows has the traceback:

2026-11-07T07:20:07.756Z muru.worker[0]:   File "/Users/graham/shop/releases/v6/app.py", line 22, in order
2026-11-07T07:20:07.756Z muru.worker[0]:     total = quantity * PRICES[item]
2026-11-07T07:20:07.756Z muru.worker[0]:                        ~~~~~~^^^^^^
2026-11-07T07:20:07.756Z muru.worker[0]: KeyError: 'mug'

A release that passes

The next release adds the price of a mug. This time I deployed it held as before, but started the canary with shorter steps, and asked for it to wait for my approval before each step after the first:

muructl deploy shop releases/v7 --hold
muructl rollout canary shop@r7 --steps 10,50 --hold 20s --approve-each-step

muru: rollout 2 of shop@r7 started against r5: step 1 of 2: 10% for 20s

While the candidate had its 10%, I wanted to see who was getting it. The edge doesn't pick a release at random for each request. It gives each new client an anonymous id in a muru_id cookie, and hashes that to decide which release the client gets. Because my load script keeps no cookies, each of its requests counts as a new client, which is why it saw both releases. A browser keeps the cookie, so to see what that means I used curl with a cookie jar as 20 separate clients, each making five requests to the home page:

client 1: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 2: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
...
client 7: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 8: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 9: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
...
client 16: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 17: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 18: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 19: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 20: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5

Four of the 20 clients landed on the new release, and each of them stayed on it for every request, as did everyone else on the old one. Someone clicking around the site doesn't flip between versions from one page to the next. The buckets are also laid out so that a client in the canary at 10% is still in it at 50%.

After 20 seconds with the gates passing, the rollout stopped and waited:

muructl rollout status shop

APP              ROLLOUT  CANDIDATE    BASELINE     STATE              TRAFFIC  MESSAGE
shop             1        r6           r5           rolled_back        0%       gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)
shop             2        r7           r5           awaiting_approval  10%      step 2 of 2: 50% for 20s awaits approval: muructl rollout resume shop

Approving the next step and following the rollout to the end:

muructl rollout resume shop
muructl rollout status shop --wait

muru: rollout 2 running: step 2 of 2: 50% for 20s
muru: rollout 2 running step 2 of 2: 50% for 20s
muru: rollout 2 promoted: every step passed its gates

The last step passing promoted the new release to take all of the traffic, with the old one drained and kept warm, so muructl rollback shop would still put things back at once. This time the load script got nothing but 200 responses, 20,421 of them.

When a worker gets stuck

Errors are the easy case, as they show up as 5xx responses. Some bugs never produce a response at all. Here is a release where the order page takes a lock it already holds, but only for kettles:

prices_lock = threading.Lock()


def order(db, order_id):
    ...
    with prices_lock:
        if item == "kettle":
            prices_lock.acquire()  # a bug: takes the lock it already holds
        total = quantity * PRICES[item]

The first request for a kettle blocks forever, and every request after it which wants the lock blocks behind it. Under gunicorn's sync workers the master would eventually kill the stuck worker for not answering its heartbeat, but a threaded worker keeps answering its heartbeat while its request threads are stuck, and nothing outside the server would connect any of that to the deploy which caused it.

Muru has a hard timeout for this, which replaces a worker when one of its requests has run too long. It is off by default, so I gave this release one of three seconds when deploying it, then started a canary as before:

muructl deploy shop releases/v8 --hold --timeout hard=3s
muructl rollout canary shop@r8 --steps 10,50 --hold 30s

muru: rollout 3 of shop@r8 started against r7: step 1 of 2: 10% for 30s
muru: rollout 3 running step 1 of 2: 10% for 30s
muru: rollout 3 rolled_back: gate failed at 10%: 2 worker crashes and timeout kills, over the limit of 0 (baseline 0)

This gate doesn't wait for 100 requests. A worker which crashes or is killed by a timeout counts against the candidate as soon as it happens, and by default one more than the baseline had in the step is enough to roll it back. The events say which workers were killed, and where they were stuck, which came from the stacks the host gathered from inside the worker before replacing it:

    70 20:53:23 warning  worker.timeout_killed shop@r8                  worker 0 (pid 82673) replaced by pid 83393: a request has run for 3.8 s, past the hard timeout of 3 s; at app.py line 27 in order, called from app.py line 39 in application, called from _pool.py line 30 in serve_jobs
    72 20:53:23 warning  worker.timeout_killed shop@r8                  worker 1 (pid 82674) replaced by pid 83395: a request has run for 3.6 s, past the hard timeout of 3 s; at app.py line 27 in order, called from app.py line 39 in application, called from _pool.py line 30 in serve_jobs
    75 20:53:24 warning  rollout.gate_failed  shop@r8                  step 1 at 10%: 2 worker crashes and timeout kills, over the limit of 0 (baseline 0)

Line 27 of app.py is the second acquire(). The release log has the full stack of every thread in each worker as well, much like the muructl debug stacks output from looking inside the workers.

Looking at the timestamps, the rollback was about as quick as with the errors. The hard timeout fired about four seconds into the canary, and the decision to replace a stuck worker is what counted against the candidate, so it was out of traffic within a second. The stuck workers themselves were given their drain grace and went about 15 seconds later, by which time the candidate had no traffic left to lose. The load script ended up with two 504 responses and three requests it gave up on after five seconds, out of about 9,500. That is a long way short of every kettle order hanging for everyone.

There are more gates than the two I have shown. --max-p95 and --max-p99 judge how long the candidate's requests take, --max-queue-wait judges how long they wait for a worker thread, and --probe names URLs the daemon requests from both releases itself every five seconds, with one failing on the candidate alone enough to roll it back.

Sticky clients and previews

The canary above uses the anonymous muru_id cookie to keep each client on one release. That can be changed per route, to hash a header such as a user id set by a proxy in front, a cookie of the application's own, or the client's address. When the canary ends, though, a client on the losing side is moved across on its next request. That is usually fine, but not for an application where a session started on one release shouldn't be continued on another. For that a route can be made sticky, so the edge sets a cookie naming the release a client was given and keeps honouring it for a grace period after that release leaves the route, before it drains.

The preview header used earlier also has a cookie form, muru_preview=r7, which makes it easy to try a held release in a browser, and a route can give a release a share of zero, so it is reachable only by preview, for a dark launch.

What's next

Everything in this post and the last relied on the daemon being able to run more than one release of an application and move traffic between them. When the application is in a container, Muru deliberately does none of that, since there the container is the release, and starting, replacing and rolling back containers is the orchestrator's job. Next I will look at what muructl serve does instead as a container's main process, and why that makes it a better tenant than the usual gunicorn or uvicorn command.

November 13, 2026 12:00 AM UTC

November 12, 2026

Luke Plant

Why I’m calling your contribution “slop”

You have probably arrived at this web page because you have attempted to contribute to an Open Source software project, probably via a Pull Request and I, or perhaps someone else, has called your efforts “slop” and promptly closed/ignored it.

I might also use this word to describe emails or other communication received which are of a similar nature.

This post is addressed to you, and is my attempt to explain why I’ve called it “slop”. I don’t want to repeat myself on this subject, and I also don’t want to leave you totally puzzled, upset and embittered, so I’m writing it once, and just once.

By “slop”, I’m referring especially to AI slop. It is, however, entirely possible that you didn’t use AI, but I can’t tell that, and your code or communication seems to share the same quality problems that AI slop has.

The old way

It would be nice if I could give more detailed feedback about why exactly I think your contribution matches the term “slop”. In the past, when I got a pull request that wasn’t up to scratch from a new contributor, I would try to take the time to give detailed feedback, to effectively train them.

This approach was based on a few assumptions:

Unfortunately, use of AI in software has massively eroded all of the above. The human involvement in creating the PR may have been almost zero. Writing is no longer proof of work. Many people will not benefit at all from the detailed feedback I give, because often they just give up. Basically, the balance that Open Source relied has been destroyed by the slopocalypse, which means that I have to allocate my time in a very different way to the past.

If you are a budding Open Source contributor trying to do the right thing, trying to learn, maybe using AI or maybe not using it all but still getting lumped with those who are, this sucks for you. I’m sorry for that, and for the world that has been lost, but I’m afraid I can’t fix it, and this post is the best I can do.

I’m also sorry for the curtness of my interactions with you, and the cynical tone you might pick up from this post, but again it is a matter of necessity to avoid the burn-out that many others have suffered, and to ensure I can give a good account for how I’ve used my life (Romans 14:12).

What’s wrong with your contribution

It might be that you have used an AI to generate something, or you’ve worked hard on it yourself, and you think it looks pretty good, but it is still labelled slop. Below are some clues as to why that might be. You will have to work through these yourself and see which apply to yours.

The best summary I can give for what I’m looking for in a good quality code contribution is the following:

Every single line, word, token and character should have been written with the understanding that another human being will need to read it, understand it and potentially maintain it indefinitely. This covers:

Of course, the code must actually work (that’s step zero, the computer has to understand it too), and in addition, it must be for the benefit of all current and future users, which means it mustn’t recklessly break anything, for example, and must be designed with long term needs in mind.

In short, the contribution should treat the recipients (reviewers and maintainer) as human beings whose time is extremely valuable, and the project as something valuable that needs to be protected.

Here are some of the most common problems I see with AI generated stuff:

At the end of this, you might think something like this:

Wow, this is really hard! It would probably be faster to start from scratch and write every word myself. That way, I’m only going to write the necessary stuff, because it will take too much effort to write unnecessary things. It will also force me to think about whether something really is necessary.

Then, if something looks unnecessary but actually isn’t I might even need to justify why it is needed!

If I end up actually needing a lot of text, I’m going to have to structure it extremely well and design my whole approach to this work to ensure that people’s eyes don’t just glaze over. But, I really need to avoid writing a lot of text if I possible can…

Yep, exactly, all that is true. It was always true of good coding. Doing the above is simply what is necessary when you treat the recipient as a human. A high quality PR was always a lot of work – you have to do all the work needed to make the maintainer’s job easy, because you are asking them to do something for you.

The only thing that has got harder about contributing to an Open Source project is that I’m now not telling you exactly what is wrong, you’ll have to figure that bit out for yourself, and I’m now much more likely to simply put something in the slop bin.

Your own use of AI may well have made things harder for you:

To gain empathy with the recipient, you are indeed better off starting from scratch and literally typing everything in, even if you use some of the insights from the AI.

In saying this, don’t think I’m 100% against AI. I have successfully used AI to help debug things that I might never have been able to work out otherwise, and I’ve used agents in very controlled ways where I’ve been able to judge their output and the trade-offs they involve. But, for many of the reasons above, I virtually never use them to write actual code or text if I need another human to read it.

Well, that’s the end. You better believe that I typed every letter myself.

November 12, 2026 11:58 AM UTC

LernerPython blog, from Reuven Lerner

Announcing: Vibe to valid

Vibe coding is fun: You can go from an idea to a working app in a weekend.

But if you don’t have a background in software, how can you be sure your app is secure? Tested? Backed up? Ready for real customers, or for an investor’s technical review?

Today I’m launching Vibe to Valid, an audit service for vibe-coded Python apps.

Here’s how it works:

  1. You share your repo.
  2. I review it and send you a plain-English report: what’s wrong, what’s right, and what to fix first.
  3. If you want, I fix the problems, and set up your AI coding tools so you can keep vibe coding with the AI on a short leash.

There’s also a due diligence audit for investors and acquirers, and a walkthrough session for your team. I’ve been a Python trainer and consultant since 1995, and Claude Code is a core part of my workflow. But every finding is reviewed and explained by me. You’re getting an experienced human, not an automated scan.

Learn more at https://VibeToValid.com, where you can also book a free intro call.

The post Announcing: Vibe to valid appeared first on LernerPython.

November 12, 2026 10:01 AM UTC

PyPy

20 Years of Pyrolog

On November 12, 2006 I did the first surviving commit of my RPython based Prolog implementation Pyrolog into the PyPy repository. I had been working on it since spring 2006 already. In that year we were finishing up the first PyPy EU research project and I was finishing up my Bachelor studies at Uni Düsseldorf. Since it's now 20 years later I wanted to write a personal retrospective post on this project. Pyrolog never had any real users, but it still led to some interesting research and influenced a number of other RPython projects.

Contents

Context

As part of the EU project we had promised lots of things, including implementing some constraint programming in PyPy. For this strand of the work I learned Prolog by reading Clocksin and Mellish's Programming Prolog [CM2003] and later the Art of Prolog [SS1994].

In March 2006, the PyPy project partner Logilab organized a sprint in UCLouvain in Louvain-la-Neuve (Belgium), with Peter van Roy and some of the researchers working on Gecode. The plan was to add some kind of unification and constraint solving to PyPy. The Logilab people were quite inspired by the Van Roy/Haridi book [VH2004] and I later studied his Prolog implementation survey paper [VanRoy1994].

In 2006 I also attended my advisor Michael Leuschel's Prolog lectures in Düsseldorf. During that I started working on a Prolog interpreter in RPython, later called Pyrolog. Much of the early commit history is lost unfortunately, it didn't make the Subversion->Mercurial jump correctly and I don't know where the codespeak archives are now.

2006/2007 The first year

Pyrolog's first incarnation had a fairly straightforward implementation of unification in object-oriented RPython code. Logical variables stored an index, with their bindings stored in a global array at that index.

Memory layout of the interpreter

Pyrolog Memory Layout from [Bolz2007]

That version represented choice points by having exception handlers on the interpreter level, and backtracking via throwing exceptions, catching them further up. It used a continuation-based architecture for storing the calls left to do. Those success continuations were stored on the heap and continuously allocated and consumed. This approach was taken from Paul Tarau's BinProlog ([TB1990], [Tarau1992]) and also based on some other Prolog papers [Lindgren1994] [Neumerkel1995].

To deal with deep recursion that Prolog programs typically do, that version of Pyrolog relied on RPython's stackless transformation [BR2007D071], which would unwind the host stack (i.e. the C stack) and store its information on the host heap if host stack overflow was getting close. When storing C stack frames on the heap in this way, we also did some amount of tail call optimization. We were super into this idea at the time. However, it turned out not to be too practical, making the binary twice as large and slower, so eventually we removed it again.

I also experimented with using the first RPython JIT on that version of Pyrolog. That JIT was a partial-evaluation-based method JIT and managed to speed up some small Python programs. I don't think it did very much for Pyrolog though.

The performance of that version of Pyrolog was quite disappointing. In my B.Sc. thesis [Bolz2007] I did some measurements against SWI and Sicstus Prolog. Sicstus was often an order of magnitude faster than Pyrolog at this point.

In the thesis I also implemented a way to call into Prolog from Python code and had a case study where I implemented a Tic-Tac-Toe GUI in Python, with the game logic in Prolog.

Shortly after finishing my thesis, I changed the variable representation: instead of indexing into a global binding array, each variable object stored its own binding. The exception-based backtracking architecture stayed in place for another two years.

2008/2009 JIT developments on the PyPy side

After the EU project finished we started working on various new architectural ideas for the RPython meta-JIT. Armin Rigo, me and some others wrote two new prototypes. We weren't happy with them, so in 2008 Armin and I started experimenting with a prototype for a new trace-based JIT written in Prolog. We used Prolog because its pattern matching and ease of meta-programming allowed us to experiment more easily with different designs.

Towards the end of 2008 and throughout 2009 Armin, Maciej Fijałkowski, me and others started porting the design back to RPython and called it pyjitpl5 (the "pl" in the name stands for "Prolog"). This is still the base of the current PyPy meta-JIT compiler and became the subject of the second (smaller) PyPy EU project in the Eurostars program.

We eventually published a paper about the Prolog version of the JIT [BLR2010] at LOPSTR 2009.

2009-2010 New Pyrolog Architecture: More Continuations

After working mostly on PyPy and the new JIT in 2008 and 2009 I started working on Pyrolog again over the 2009 Christmas break. Since the stackless transformation wasn't compatible with the new JIT I had to switch to a different architecture, still used today. The first version of the Pyrolog architecture had already stored success continuations on the heap. The goal of this refactoring was to also store the failure continuations as reified RPython objects.

# the simplified main interpreter loop

def interpret(sc, fc, trail):
    while not sc.is_done():
        try:
            sc, fc, trail = sc.activate(fc, trail)
        except UnificationFailed:
            sc, fc, trail = fc.fail(trail)

This had the effect that the host stack use is completely flat when interpreting Prolog calls. The state needed to continue with Prolog execution is always stored in continuation objects on the heap. The interpreter is therefore dealing with two continuations, one activated upon success, and one on failure.

Additionally, there is a (segmented) trail object which stores the variable bindings that need to be undone when backtracking. Failure continuations have a reference to the part of the trail they need to backtrack to, when activated.

The downside of this approach is that the interpreter continuously has to allocate heap data for all the success and failure continuations. This puts a lot of pressure on the RPython GC. Also, term unification still uses the host stack, so unifying very deep terms can still be a problem and lead to implementation-level recursion errors.

JIT support

The real goal of the refactoring was to make the interpreter more JIT-friendly. Since the interpreter doesn't use recursion for Prolog calls any more, the loops at the Prolog level (expressed as recursion) can be traced by the JIT. All the continuous allocations of continuations can be optimized by the allocation removal of the RPython JIT. If all this works correctly, the performance of Pyrolog becomes very good, often beating Sicstus by a significant factor on suitable microbenchmarks.

/images/2026-pyrolog-vmgen.svg

Pyrolog VM Generation vs. Runtime from [BLS2010]

Unfortunately this is mainly the case for code with very predictable control flow, ideally with nice loops. Since the main appeal of Prolog is to do backtracking search, control flow is typically not very predictable for interesting more Prolog-like programs. As soon as the control flow becomes too complicated and unpredictable, the continuations survive beyond the region that the JIT optimizes and have to be allocated on the heap again. Therefore Pyrolog has quite extreme performance cliffs.

We wrote a paper about this version of Pyrolog for PPDP 2010 [BLS2010] (together with Michael Leuschel and my later colleague David Schneider). That conference was super interesting and important for me. I met Paul Tarau (who invented BinProlog, one of the inspirations of Pyrolog. He later gave an excellent Prolog talk at the virtual machine summer school in 2016), Olivier Danvy (whose paper [BD2004] I should have cited. Genuinely one of the nicest people), Kenichi Asai (who had worked on partial evaluation and reflection) and several others. At that point during my PhD I was feeling relatively demotivated, and my chats with these researchers were incredibly encouraging.

https://www3.risc.jku.at/conferences/ppdp2010/photos/img_8487.jpg

Group photo at PPDP 2010

On the bus trip to Gmunden, where the conference dinner took place, somebody explained how to implement unification of cyclic terms (i.e. rational trees) to me. Unfortunately I am not quite sure who it was any more, I think it might have been Thierry Martinez. This led to me implementing support for cyclic term unification. I still like that trick a lot and tried to explain it to Phil Zucker earlier this year. I hope to blog about it eventually. For a fun use case see my blog post on rational tree unification for dfa minimization.

2011/2012 Widening Prolog support

During 2011 and 2012 I supervised several Bachelor/Master theses that worked on adding features to Pyrolog. In the process we added support for Prolog modules, DCGs, attributed variables, freeze and when. All these were done by my excellent students Sven Hager and René Bartelmus. This left the fundamental execution model intact (with some small additions). I also spent some time improving the error messages of Pyrolog to for example show backtraces on Prolog exceptions.

Pyrolog also served as an early test case for David Schneider's ARM JIT backend. In January 2011, he could already run Pyrolog microbenchmarks with the JIT on a BeagleBoard, while the backend still needed debugging before running more complex Python benchmarks.

In 2012 I mostly wrote my PhD thesis [Bolz2012]. It contains a chapter on Pyrolog based on the PPDP 2010 paper. That chapter argues that the fact that Pyrolog works relatively well demonstrates that the RPython meta-JIT can be used for quite a wide variety of languages.

There was also another chapter that compared how a meta-tracing JIT works with more classical partial evaluation approaches. The code of that was again written in Prolog and originally presented here on the blog.

At this point I left Düsseldorf. In the winter semester 2012/13 I spent half a year in Berlin at Hasso-Plattner-Institute with my PhD co-advisor Robert Hirschfeld as a postdoc. In 2013 I started working with Laurie Tratt at King's College.

At this point the main Pyrolog story kind of ends. There were only some small improvements happening in the succeeding years. However, Pyrolog ideas led to followup research in other projects, and I want to discuss some of them.

2012/2013 Compressed Terms

One experiment that I had tried to get working in Pyrolog in 2010 and 2011, but ended up never merging, was compressed terms. The idea comes from the observation that certain patterns of terms occur repeatedly. For example an association list has terms of the form [A/B|C] repeated. Instead of allocating separate list and pair objects for every entry, the plan was to share a description of that structure, the "shape", and store each entry's keys and values in a flat storage data structure. This would make every entry have less indirection and use fewer words of memory.

However, I had never managed to get that working in Prolog, because variable bindings can create these patterns after the fact. A field that initially contained an unbound variable could later refer to a whole structured term, so the opportunities for compression changed after construction, and again on backtracking.

When I was at HPI for six months in winter 2012/2013 I worked with Tobias Pape to implement the same compression idea for a simpler functional language called lamb, later Theseus. Tobias pushed this approach much further in the following years [PBH2015], [PBH2017]. My hope was to eventually understand the problem well enough to be able to port it back to Pyrolog. But that part never happened. We did eventually try those ideas in Pycket though.

2013 Unipycation

In 2013 I started working in London. The goal of our research project there was "language composition". Edd Barrett, Laurie Tratt and I implemented combined RPython implementations of PyPy plus another language, with a way to integrate the two involved languages. The first pair we tried was combining PyPy and Pyrolog in a project called Unipycation [BBT2013]. The project made it possible to call into Prolog from Python and vice versa. The RPython meta-JIT can optimize across this language boundary and even perform cross-language inlining. Unfortunately we never found super compelling use-cases for this combination.

2013 Pycket

Also in 2013 I started working on Pycket, an RPython implementation of Racket, together with Sam Tobin-Hochstadt and Jeremy Siek. Later Tobias Pape and Spenser Bauman also joined. The Pycket architecture was indirectly inspired by the Pyrolog continuation approach. We implemented the CEK machine relatively directly in RPython. This also used a flat host stack and it made implementing things like call/cc relatively straightforward [Pycket2015].

One of the great new ideas we adopted in Pycket was "false loop filtering", originally from IBM Research [HWISN2011]. This takes calling context into account when tracing function calls and prevents the tracing JIT from recognizing "false loops", which would never run for more than one iteration. I always wanted to port this idea back to Pyrolog and implemented a prototype, but never really polished it enough to be able to merge it.

Another implementation technique in Pycket was to make continuation objects much smaller, to make the case of them escaping to the heap less bad. This approach did make it back to Pyrolog.

2022-2024 RPython implementation of Effekt

I left King's College in 2016 and have been back at Uni Düsseldorf (half-time) since 2017. In 2022 I started working with Marcial Gaißert and Jonathan Brachthäuser on an RPython implementation of Effekt and related languages. This was an implementation of a JIT for effect handlers in RPython [GBB2025]. Again, it uses the approach of having reified stack frames/continuations which can then be manipulated by the interpreter, adding effect handlers on top of it.

2026 Git Conversion

This fall, I got an email by a researcher about Pyrolog and realized that it was 20 years old this year. Subsequently I merged various strands of the surviving mercurial history into a new Github repo, fixed some bugs and implemented some convenience features.

/images/2026-pyrolog-screenshot.png

Screenshot of Pyrolog

Conclusion

/images/2026-pyrolog-activity-by-month-authors.svg

commits per month on the main branch, by author

Pyrolog has always been a much more researchy project than PyPy itself. While PyPy tries to be an open source project and "product", Pyrolog never had any users and even I never used it for anything beyond small demos, some research prototypes, and benchmarking. While some quite interesting research came out of Pyrolog, I think having some real applications running on top of it would have given me a much more concrete goal to improve it towards.

Implementation wise, Pyrolog inherited some of the problems of PyPy. In PyPy we were quite strongly focused on peak performance in the first few years, leading to performance cliffs. Pyrolog also had those in even stronger form, due to the constant allocations of the interpreter.

Some very interesting interaction between Pyrolog, various Prolog JIT prototypes and the PyPy/RPython meta-JIT emerged. Pyrolog forced the RPython JIT to support quite different execution models which made it applicable to other languages than just object-oriented imperative ones.

I still have no concrete use case for Pyrolog so it's kind of unlikely that I will work much more on it right now. But my development on this project was always quite bursty, so I am bound to return to it at some point in the future.

I would like to end with thanking all the contributors, my other collaborators and the long line of Prolog researchers that inspired and encouraged me.

Bibliography

[TB1990]

P. Tarau, M. Boyer. Elementary logic programs. PLILP 1990, LNCS 456, pp. 159–173. DOI.

[Tarau1992]

Paul Tarau. BinProlog: a continuation passing style Prolog engine. PLILP 1992, LNCS 631, pp. 479–480. Poster.

[SS1994]

Leon S. Sterling, Ehud Y. Shapiro. The Art of Prolog: Advanced Programming Techniques. 2nd edition, MIT Press, 1994. Publisher page.

[Lindgren1994]

Thomas Lindgren. A Continuation-Passing Style for Prolog. International Logic Programming Symposium, 1994, pp. 603–617.

[VanRoy1994]

Peter Van Roy. 1983–1993: The Wonder Years of Sequential Prolog Implementation. Journal of Logic Programming 19–20, 1994, pp. 385–441. DOI; author-hosted technical report PDF (DEC Paris Research Laboratory report 36, December 1993).

[Neumerkel1995]

Ulrich Neumerkel. Continuation Prolog: A new intermediary language for WAM and BinWAM code generation. Post-ILPS 1995 Workshop on Implementation of Logic Programming Languages, 1995. PDF.

[CM2003]

William F. Clocksin, Christopher S. Mellish. Programming in Prolog: Using the ISO Standard. 5th edition, Springer, 2003. DOI.

[VH2004]

Peter Van Roy, Seif Haridi. Concepts, Techniques, and Models of Computer Programming. MIT Press, 2004. ISBN 9780262220699. Publisher page; author's book page.

[BD2004]

Dariusz Biernacki, Olivier Danvy. From Interpreter to Logic Engine by Defunctionalization. LOPSTR 2003, revised selected papers, published 2004, pp. 143–159. DOI, PDF.

[BR2007D071]

CF Bolz, Armin Rigo. Support for massive parallelism, optimisation results, practical usages and approaches for translation aspects. PyPy Consortium, EU deliverable D07.1, 28 February 2007. Listed by PyPy as Massive Parallelism and Translation Aspects. Official report index; report PDF.

[Bolz2007] (1,2)

CF Bolz. A Prolog Interpreter in Python. Bachelor thesis, Heinrich Heine University Düsseldorf, 2007. PDF.

[BLR2010]

CF Bolz, Michael Leuschel, Armin Rigo. Towards Just-In-Time Partial Evaluation of Prolog. LOPSTR 2009, revised selected papers, LNCS 6037, 2010, pp. 158–172. DOI; author manuscript.

[BLS2010] (1,2)

CF Bolz, Michael Leuschel, David Schneider. Towards a Jitting VM for Prolog Execution. PPDP 2010, pp. 99–108. DOI; author manuscript.

[HWISN2011]

Hiroshige Hayashizaki, Peng Wu, Hiroshi Inoue, Mauricio J. Serrano, Toshio Nakatani. Improving the Performance of Trace-based Systems by False Loop Filtering. ASPLOS XVI, 2011, pp. 405–418. DOI; IBM Research publication page.

[Bolz2012]

CF Bolz. Meta-Tracing Just-in-Time Compilation for RPython. PhD thesis, Heinrich-Heine-Universität Düsseldorf, September 2012. Author's thesis page; PDF; university repository.

[BBT2013]

Edd Barrett, CF Bolz, Laurence Tratt. Unipycation: A Case Study in Cross-Language Tracing. VMIL 2013, pp. 31–40. DOI; author manuscript.

[Pycket2015]

Spenser Bauman, CF Bolz, Robert Hirschfeld, Vasily Kirilichev, Tobias Pape, Jeremy G. Siek, Sam Tobin-Hochstadt. Pycket: A Tracing JIT for a Functional Language. ICFP 2015, pp. 22–34. DOI; author manuscript.

[PBH2015]

Tobias Pape, CF Bolz, Robert Hirschfeld. Adaptive Just-in-time Value Class Optimization: Transparent Data Structure Inlining for Fast Execution. SAC 2015. DOI; PDF.

[PBH2017]

Tobias Pape, CF Bolz, Robert Hirschfeld. Adaptive just-in-time value class optimization for lowering memory consumption and improving execution time performance. Science of Computer Programming 140, 2017, pp. 17–29. Published online in 2016. DOI; PDF.

[GBB2025]

Marcial Gaißert, CF Bolz-Tereick, Jonathan Immanuel Brachthäuser. Tracing Just-in-Time Compilation for Effects and Handlers. Proceedings of the ACM on Programming Languages 9 (OOPSLA2), Article 307, October 2025. DOI; author publication page; technical report.

November 12, 2026 08:16 AM UTC

Tryton News

Release 0.3.0 of tryton-filestore-cellar

We are proud to announce the release of tryton-filestore-cellar version 0.3.0.

tryton-filestore-cellar is Tryton’s filestore implementation based on the Cellar storage service.

In addition to bug fixes, this release contains the following improvements:

The package is available at tryton-filestore-cellar · PyPI

1 post - 1 participant

Read full topic

November 12, 2026 07:00 AM UTC

Graham Dumpleton

Muru: releases, not restarts

With the previous post I finished with development mode. Something I kept coming back to in those posts was that when muructl run reloads after a change, it isn't restarting anything. It starts a new release beside the one serving, and only moves traffic across once the new release is ready. In how it fits together I said the same mechanism is what the daemon uses for deploying an application to a server. Time to show that.

This is where Muru departs most from how a Python web application is usually deployed. With gunicorn, deploying new code typically means putting the new code in place and sending the master process a HUP signal. Going by its own description, that reloads the configuration, starts new worker processes and gracefully shuts down the old ones, with the new workers loading the application code afresh unless preload_app is on, in which case the old code stays. A USR2 signal goes further, starting a second master and set of workers from the same command line beside the first, with the old master left around so a change can be backed out by sending more signals. Either way, there is one copy of the code on disk, both sets of workers share the one listening socket while they overlap, and anything more than that, such as keeping the previous version running so you can switch straight back to it, is left to a load balancer and a deploy script you write yourself. I can hardly point the finger, as mod_wsgi was no better, with a deploy there meaning a restart of Apache, or a similar mix of signals and gymnastics to get a daemon process to reload against different code.

Under the Muru daemon, each deploy is a release of its own, started beside the one serving, given traffic only once it is ready, and kept warm afterwards in case you want it back.

Starting a daemon

The daemon runs in the background as your own user, or under systemd as a service. For trying things out, starting it as your own user is enough:

muructl daemon start

muru: daemon started (pid 31018), serving on http://127.0.0.1:8080, log /Users/graham/Library/Logs/muru/daemon.log

muructl status --processes shows the processes the daemon is looking after, which for now is only the edge it has started, listening on port 8080. The daemon's own row says whose daemon it is, here one run as my own user, and where it keeps its state:

muructl status --processes

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE      WHAT                                                    USER     UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon    user daemon, state ~/Library/Application Support/muru   graham   1s   0.0%   32.5MB
31019   └─ edge   listening on 127.0.0.1:8080                             graham   1s   0.0%   26.0MB       0

Nothing has been deployed yet, so the edge answers every request with a 503 and says why:

503 Service Unavailable
reason: no_workers
request id: 01M6TDFVFPFWF4JKE4TJ1G86E2

No worker is available to handle the request.

A release is a directory

What gets deployed is a source directory. For this post the application is a small shop, kept in a directory per version under releases, as a deploy script might lay out checkouts of each tag. Each version is its own uv project with a pyproject.toml and a uv.lock, and the first version has no dependencies at all:

[project]
name = "shop"
version = "1.0"
requires-python = ">=3.15"
dependencies = []

[tool.muru]
entrypoint = "app:application"

[tool.muru.commands.migrate]
description = "Apply database migrations"
run = ["python", "migrate.py"]
exclusive = true

The application in app.py reports how many orders are in an SQLite database, and which release answered, using muru.context():

import os
import sqlite3

import muru

DATABASE = os.environ["SHOP_DATABASE"]


def application(environ, start_response):
    where = muru.context()
    with sqlite3.connect(DATABASE) as db:
        (count,) = db.execute("SELECT COUNT(*) FROM orders").fetchone()
    body = f"{count} orders, served by {where.app}@{where.release}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body]

The migrate command declared in pyproject.toml runs a small script which applies any SQL files in a migrations directory that haven't been applied yet. I will come back to commands later, as they matter once two releases share a database.

The first deploy names the application and the source directory, and gives the settings the release needs:

muructl deploy shop releases/v1 --processes 2 --env SHOP_DATABASE=/Users/graham/shop/data/shop.db

muru: release shop@r1 is ready and serving

The database starts out empty, so before anything else its tables need creating, by running the declared command with the release:

muructl command run shop migrate

applied 0001_orders.sql

after which the shop answers:

3 orders, served by shop@r1

There is a fair bit hidden behind that first deploy, and muructl events tells the story:

     2 18:10:12 info     release.created      shop@r1                  from /Users/graham/shop/releases/v1
     3 18:10:12 info     release.building     shop@r1                  from /Users/graham/shop/releases/v1/uv.lock
     4 18:10:12 info     release.built        shop@r1                  /Users/graham/Library/Application Support/muru/envs/shop/r1 in 0.1s from uv.lock 8c788a3a641a
     5 18:10:12 info     release.state_changed shop@r1                  fetched the runtime library muru-1.0.0-cp315-cp315t-macosx_11_0_arm64.whl from PyPI, once
     6 18:10:13 info     release.ready        shop@r1                  2 workers ready in 0.7 s
     7 18:10:13 notice   traffic.switched     shop@r1                  traffic moved to the release
     8 18:10:13 info     command.started      shop@r1                  migrate: started (pid 31050)
     9 18:10:13 info     command.finished     shop@r1                  migrate: exited with status 0 after 60ms

Because the source directory has a uv.lock and no .venv, the daemon built a virtual environment for the release from the lock file, in a directory belonging to that release alone. The application doesn't need to depend on Muru, as the daemon adds its own runtime library to the environment. Only once both workers were ready did traffic move to the release.

Asking for the processes again shows what the deploy added:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                    USER     UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon         user daemon, state ~/Library/Application Support/muru   graham   2s   0.0%   38.9MB
31019   ├─ edge        listening on 127.0.0.1:8080                             graham   2s   0.0%   26.5MB       0
31045   └─ host        shop@r1 serving                                         graham   0s   0.0%   54.5MB            0/8       0
31044      ├─ relay    shop@r1 output to apps/shop/r1.log                      graham   0s   0.0%   17.5MB
31047      ├─ worker   group default, worker 0                                 graham   0s   0.0%   20.3MB            0/4
31048      └─ worker   group default, worker 1                                 graham   0s   0.0%   19.5MB            0/4

The release is a host process, which imported the application, with the two workers forked from it, each with four threads. The host's row also says what part the release has in routing, which for now is that it is serving. Beside the host is a log relay, which takes what the host and its workers write and puts it in the release's own log, so that a log which is slow to write never holds up a request. The relay is shown under the host it serves, although it is the daemon which starts it. The columns on the right, for requests per second, busy threads, queueing, response times and time spent waiting on the GIL, stay mostly empty until there is some traffic.

That each release gets an environment of its own is more important than it might look. If releases shared one virtual environment, installing a new dependency for the next release would change what the running release sees underneath it. Giving each release its own means a release really is fixed once started, which is what makes the rest of this work.

A release that can't start

The second version of the shop renders its page with the markdown package, so where the first version returned plain text, it returns the count in bold as a line of HTML. Whoever made the change imported the package but forgot to add it to the project's dependencies, so it isn't in uv.lock. Deploying it:

muructl deploy shop releases/v2

Error: the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'

The site carries on as if nothing happened, still answering from the first release:

3 orders, served by shop@r1

muructl release list shows the new release as failed beside the one still serving:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
shop@r2                  failed              - (2x4)              4.7MB /Users/graham/shop/releases/v2
                         the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'
shop@r1                  ready      yes      2 (2x4)              4.7MB /Users/graham/shop/releases/v1

and the release's own log, from muructl logs shop@r2, has the same reason in context, after the lines from building its environment:

2026-11-07T07:10:13.431Z [build] Creating virtual environment at: /Users/graham/Library/Application Support/muru/envs/shop/r2
2026-11-07T07:10:13.478Z [build] Bytecode compiled 0 files in 42ms
2026-11-07T07:10:13.886Z muru.host[31069]: cannot load app:application: ModuleNotFoundError: No module named 'markdown'

The lines of a release's log carry the full time in UTC, as is usual for a log file, whereas muructl events gives the time of each event on the local clock.

Nor did the failed release leave anything running, with muructl status --processes showing the same processes as before the deploy. Had the new code been put in place over the old and the server reloaded, the new workers would have been trying to import the broken code from where the working code used to be. Here, the release that couldn't start cost nothing but the attempt.

Moving traffic without refusing requests

The third version adds markdown to the project properly, with uv add markdown, so it ends up in the lock file.

Each deploy after the first takes what it isn't told from the release before it, so the two processes and the SHOP_DATABASE variable given to the first deploy carry through. A deploy with --dry-run shows what the deploy would do, with where each setting would come from, and changes nothing. Trimmed down a bit, that looks like:

muructl deploy shop releases/v3 --dry-run

muru: plan (nothing changed): release shop@r3, app:application 2x4 from /Users/graham/shop/releases/v3
  entrypoint = app:application          ([tool.muru])
  python = /Users/graham/.local/bin/python3.15t  (the previous release)
  build.when = auto                     (default)
  ...
  processes = 2                         (the previous release)
  threads = 4                           (the previous release)
  ...
  env SHOP_DATABASE (the previous release)
  ...
  settings nothing else gives come from shop@r1; --fresh takes none
  an environment would be built from the source's uv.lock
  the release would take over from shop@r1, which would stay warm for 1800s with 1 worker to a group

Environment variables carry across by name, which is why SHOP_DATABASE doesn't need to be given again. Where you would rather a deploy be described entirely by its own command line, --fresh ignores the previous release.

To see whether the switch refuses anything, I ran a small script which requests the page every 10 milliseconds, prints a line whenever the response changes, and counts the status codes it got back. With it running, I deployed the third version:

muructl deploy shop releases/v3

muru: release shop@r3 is ready and serving

The output of the script was:

18:10:14 200 3 orders, served by shop@r1
18:10:16 200 <p><strong>3 orders</strong>, served by shop@r3</p>
{200: 695}

Every one of the 695 requests got a 200, with the responses changing over from one release to the other without a gap. The events show the order things happened in:

    14 18:10:15 info     release.created      shop@r3                  from /Users/graham/shop/releases/v3
    15 18:10:15 info     release.building     shop@r3                  from /Users/graham/shop/releases/v3/uv.lock
    16 18:10:16 info     release.built        shop@r3                  /Users/graham/Library/Application Support/muru/envs/shop/r3 in 0.2s from uv.lock 62c319f2cb1f
    17 18:10:16 info     release.ready        shop@r3                  2 workers ready in 0.6 s
    18 18:10:16 notice   traffic.switched     shop@r3                  traffic moved to the release
    19 18:10:16 info     release.draining     shop@r1                  draining
    20 18:10:16 info     release.retained     shop@r1                  kept warm for 1800s with 1 worker per group
    21 18:10:17 info     worker.exited        shop@r1                  worker 1 (pid 31048) stopped: scaled in

The new release was built and started while the old one carried on serving. Traffic moved only once the new release was ready, and the old release then drained, finishing the requests it already had. What happened next to the old release is the interesting part. It wasn't stopped, but scaled in to a single worker and kept warm for 30 minutes, which muructl release list notes against it:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
shop@r3                  ready      yes      2 (2x4)              6.4MB /Users/graham/shop/releases/v3
shop@r2                  failed              - (2x4)              4.7MB /Users/graham/shop/releases/v2
                         the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'
shop@r1                  retained            1 (2x4)              4.7MB /Users/graham/shop/releases/v1
                         warm until 07:40 UTC; a rollback to it is instant

The switch can also be watched as it happens, by leaving muructl status --processes --watch running in another terminal during the deploy. The third release's host, log relay and workers appear beside those of the first, the requests per second move across from one to the other, and the first then drops to a single worker.

Rolling back is a routing change

Say the new page turns out to be a mistake. With the script running again:

muructl rollback shop

muru: rolled back to shop@r1 (ready)

18:10:24 200 <p><strong>3 orders</strong>, served by shop@r3</p>
18:10:26 200 3 orders, served by shop@r1
{200: 363}

The command took a fifth of a second. Nothing was deployed, built or imported. The retained release still had its host and a worker running, with the application already loaded, so rolling back was only a matter of telling the edge to send requests there instead. The events show what followed:

    22 18:10:26 notice   traffic.switched     shop@r1                  traffic moved to the release
    23 18:10:26 notice   app.rolled-back      shop                     rolled back to r1
    24 18:10:26 info     release.draining     shop@r3                  draining
    25 18:10:26 info     release.retained     shop@r3                  kept warm for 1800s with 1 worker per group
    26 18:10:26 info     worker.started       shop@r1                  worker 1 started as pid 31226
    27 18:10:26 info     worker.ready         shop@r1                  worker 1 (pid 31226) is ready
    28 18:10:27 info     worker.exited        shop@r3                  worker 1 (pid 31122) stopped: scaled in

Once it had traffic again, the first release grew back to its two workers by forking from its host, which still had the application imported, as described in import once, fork many. The processes show this, with the first release's second worker only a few seconds old under a host which has been running all along, and the third release, now the retained one, down to the one worker it is kept warm with:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                    USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon         user daemon, state ~/Library/Application Support/muru   graham   17s   0.1%   40.7MB
31019   ├─ edge        listening on 127.0.0.1:8080                             graham   17s   0.0%   29.7MB       0
31045   ├─ host        shop@r1 serving                                         graham   15s   0.0%   54.5MB       0    0/8       0
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log                      graham   15s   0.0%   17.6MB
31047   │  ├─ worker   group default, worker 0                                 graham   15s   0.0%   48.8MB       0    0/4
31226   │  └─ worker   group default, worker 1                                 graham    3s   0.1%   29.4MB       0    0/4
31117   └─ host        shop@r3 retained until 07:40 UTC                        graham   13s   0.1%   56.2MB       0
31116      ├─ relay    shop@r3 output to apps/shop/r3.log                      graham   13s   0.0%   17.5MB
31121      └─ worker   group default, worker 0                                 graham   12s   0.0%   58.1MB

The release rolled back from is now the retained one, so running muructl rollback shop a second time goes forward to it again. Having decided the new page was fine after all, that is what I did, putting the third release back in front of the first:

muructl rollback shop

muru: rolled back to shop@r3 (ready)

The third release grew back to two workers in the same way, and the first went back to being the retained release with one.

Rolling back to a release which has already been stopped, because its 30 minutes are up, also works. The release then has to be started again first, with traffic moving once it is ready. When I tried that with this application, it took about a second rather than a fifth of one, which for a larger application would be however long its import takes.

Restarting the daemon

The other thing which has to happen to a server from time to time is the hosting software itself being restarted or upgraded. As I explained in how it fits together, the daemon is never on the path a request takes, which means it can go away while requests carry on. With the script running once more, I stopped the daemon:

muructl daemon stop

muru: daemon detached; hosts and the edge keep serving

With the daemon gone there is nothing for muructl status --processes to ask, so this time it has to be ps. Each of Muru's processes sets a title saying what it is, so ps is still readable, and shows the edge, the release hosts and their log relays are all still there, now with process 1 as their parent, and the workers still children of their hosts:

31019     1 muru edge user
31044     1 muru relay shop@r1
31045     1 muru host shop@r1
31047 31045 muru worker shop@r1 default/0
31116     1 muru relay shop@r3
31117     1 muru host shop@r3
31121 31117 muru worker shop@r3 default/0
31266 31117 muru worker shop@r3 default/1

Starting the daemon again, it finds them and takes them over:

    36 18:10:31 info     daemon.detached      daemon                   detached; hosts and the edge keep serving
    37 18:10:34 info     daemon.started       daemon                   serving on http://127.0.0.1:8080, admin socket /tmp/muru-501/admin.sock
    38 18:10:34 info     release.adopted      shop@r3                  re-adopted the host (pid 31117)
    39 18:10:34 info     release.adopted      shop@r1                  re-adopted the host (pid 31045)
    40 18:10:34 notice   traffic.switched     shop@r3                  traffic moved to the release

and muructl status --processes has them back in the tree under the new daemon, noting against the edge and each host it re-adopted that the operating system now has process 1 as its parent:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                         USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31301   daemon         user daemon, state ~/Library/Application Support/muru        graham    1s   0.0%   35.8MB
31019   ├─ edge        listening on 127.0.0.1:8080 (re-adopted, OS parent 1)        graham   25s   8.4%   30.2MB       0
31045   ├─ host        shop@r1 retained until 07:40 UTC (re-adopted, OS parent 1)   graham   23s   0.0%   54.5MB
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log                           graham   23s   0.0%   17.6MB
31047   │  └─ worker   group default, worker 0                                      graham   23s   0.0%   48.9MB
31117   └─ host        shop@r3 serving (re-adopted, OS parent 1)                    graham   21s   0.0%   56.2MB            0/8       0          0.1 ms
31116      ├─ relay    shop@r3 output to apps/shop/r3.log                           graham   21s   0.0%   17.5MB
31121      ├─ worker   group default, worker 0                                      graham   20s   5.8%   68.5MB            0/4
31266      └─ worker   group default, worker 1                                      graham    7s   5.3%   63.1MB            0/4

The daemon is a second old, while everything under it has been running since before it was stopped. Meanwhile the script didn't notice anything had happened:

18:10:29 200 <p><strong>3 orders</strong>, served by shop@r3</p>
{200: 703}

Upgrading Muru works along similar lines. muructl daemon upgrade re-executes the daemon from the newly installed muructl and replaces the edge without refusing connections, while the releases keep the runtime library they started with until the next deploy replaces them.

Two releases, one database

Being able to run two releases side by side has a consequence which is easy to overlook. While the new release is starting, and for as long as the old one is retained for a rollback, both are running against the same database. A schema change made for the new release therefore has to keep working with the code of the old one.

The usual answer is to split such a change into steps. The first step only adds things, such as a new table or a new column with a default, which the old code won't notice. Then the release which uses the new schema is deployed. Only once no release that relies on the old schema is serving or retained does a later release remove what is no longer needed. That is general advice, whatever you host with. What Muru adds is a way of running each step with the release it belongs to.

The fourth version of the shop adds a quantity column, in a new file migrations/0002_quantity.sql:

ALTER TABLE orders ADD COLUMN quantity INTEGER NOT NULL DEFAULT 1;

and changes the page to report the number of items as well. If this were deployed like the others, the release would start, be ready, take all the traffic, and then fail every request because the column doesn't exist yet. Instead, it is deployed with --hold, which starts the release but leaves traffic where it is:

muructl deploy shop releases/v4 --hold

muru: release shop@r4 is ready and held: not serving

With the script from earlier still sending requests, the processes show the fourth release fully started and held, with its host, log relay and two workers, while every request is still going to the workers of the third:

edge  37 req/s  0 err/s  p50 1.8 ms  p95 2.7 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 1.9KB/s

PID     ROLE           WHAT                                                         USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE      P95   GIL/LAG
31301   daemon         user daemon, state ~/Library/Application Support/muru        graham    4s   0.0%   38.4MB
31019   ├─ edge        listening on 127.0.0.1:8080 (re-adopted, OS parent 1)        graham   28s   0.0%   30.3MB      37
31045   ├─ host        shop@r1 retained until 07:40 UTC (re-adopted, OS parent 1)   graham   27s   0.0%   54.5MB       0
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log                           graham   27s   0.0%   17.6MB
31047   │  └─ worker   group default, worker 0                                      graham   26s   0.0%   48.9MB
31117   ├─ host        shop@r3 serving (re-adopted, OS parent 1)                    graham   24s   0.0%   56.2MB      37    0/8       0   2.7 ms    0.1 ms
31116   │  ├─ relay    shop@r3 output to apps/shop/r3.log                           graham   24s   0.0%   17.5MB
31121   │  ├─ worker   group default, worker 0                                      graham   24s   0.0%   70.1MB      19    0/4           3.3 ms
31266   │  └─ worker   group default, worker 1                                      graham   11s   0.0%   63.5MB      19    0/4           2.5 ms
31345   └─ host        shop@r4 held                                                 graham    0s   0.0%   56.9MB            0/8       0
31344      ├─ relay    shop@r4 output to apps/shop/r4.log                           graham    0s   0.0%   17.4MB
31346      ├─ worker   group default, worker 0                                      graham    0s   0.0%   20.7MB            0/4
31347      └─ worker   group default, worker 1                                      graham    0s   0.0%   19.8MB            0/4

A held release can still be reached by naming it in a Muru-Preview header. Doing that now shows the problem the migration needs to fix, while ordinary requests are still served by the third release:

curl -H 'Muru-Preview: r4' http://127.0.0.1:8080/

500 Internal Server Error
reason: app_error
request id: 01M6TDGPENFWF4JKE4TJ1G86E2

The application raised an exception.

with muructl logs shop@r4 showing why:

2026-11-07T07:10:40.086Z muru.worker[0]:     (count, items) = db.execute("SELECT COUNT(*), SUM(quantity) FROM orders").fetchone()
2026-11-07T07:10:40.086Z muru.worker[0]:                      ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-11-07T07:10:40.086Z muru.worker[0]: sqlite3.OperationalError: no such column: quantity

The migration is then run with the held release:

muructl command run shop@r4 migrate

applied 0002_quantity.sql

Naming the release matters here. A command runs with the release's own source directory, interpreter, environment variables and secrets, so this ran the fourth release's migrate.py against its own migrations directory, which is the only one with the new file in it. Commands are declared in the project and are part of each release, so the same command name always runs the version of the command which goes with that release's code. With exclusive = true, a second run of the command for the application is refused while one is under way, and muructl command run exits with the command's status, so a deploy script can stop at a failed migration before any traffic moves.

With the migration applied, the held release works, and the third release, which knows nothing about the new column, carries on serving ordinary requests:

<p><strong>3 orders</strong> for 3 items, served by shop@r4</p>
<p><strong>3 orders</strong>, served by shop@r3</p>

All that remains is to move the traffic:

muructl rollout bluegreen shop@r4

muru: release shop@r4 is ready and serving

The third release then becomes the one kept warm for a rollback, and the first, no longer needed for one, is stopped. Because the migration only added a column, rolling back to the third release afterwards still works against the new schema, which I checked by rolling back and forward again. The rollback doesn't touch the schema, which is the reason for doing changes like this in steps.

What's next

With gunicorn and similar servers, all of this is something you build yourself around the server. Here it is what a deploy does.

That said, a deploy only protects you from a release which can't start. The fourth release, deployed without first holding it, would have been ready as far as Muru was concerned, and would then have failed every request it was given. The same goes for a release which starts fine but has a bug in only some of its pages. In the next post I will look at canaries, where a new release is given a small share of the traffic first and is judged against the release already serving, using what the workers report about the requests they handle, before it gets any more.

November 12, 2026 12:00 AM UTC

November 11, 2026

The Python Coding Stack

Use Your Initiative

Subscribe now

stephengruppetta.com

November 11, 2026 04:31 PM UTC

LernerPython blog, from Reuven Lerner

Pandas option_context: Temporarily change display settings

Pandas option_context: Temporarily change display settings

Want to set an option in Python Pandas, but just temporarily? You can use option_context:

with pd.option_context('display.precision', 2):
    print(s/3)

Within the “with” block, the display precision is 2. But outside — before or after — it is the default.

The post Pandas option_context: Temporarily change display settings appeared first on LernerPython.

November 11, 2026 07:00 AM UTC

Python GUIs

Keeping Your Main Window Open After Closing a Dialog in PyQt6 — Why your app closes when you dismiss a dialog, and how to fix it

I'm working with dialogs in PyQt6, and when I press either Accept or Cancel, the entire application closes. How can I prevent this?

When you create and display a dialog (for example, using QDialog.exec()), the dialog runs its own local event loop. Once you click OK or Cancel, the dialog closes and exec() returns a result. At that point, your code continues executing whatever comes after the dialog call.

If there's no main window being shown and no main event loop running, the application simply has nothing left to do — so it exits.

In other words, the dialog is your entire application. Once it's gone, everything shuts down.

The Solution: Run a Main Window with an Event Loop

To keep your application alive after the dialog closes, you need a main window that stays open and a properly running event loop. The dialog should be launched from that main window, so that when the dialog is dismissed, control returns to the main window which is still visible and active.

Here's a minimal complete example that demonstrates the correct setup:

python 
import sys

from PyQt6.QtWidgets import (
    QApplication, QDialog, QDialogButtonBox,
    QMainWindow, QPushButton, QVBoxLayout,
)


class CustomDialog(QDialog):
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setWindowTitle("My Dialog")

        buttons = QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel
        self.button_box = QDialogButtonBox(buttons)
        self.button_box.accepted.connect(self.accept)
        self.button_box.rejected.connect(self.reject)

        layout = QVBoxLayout()
        layout.addWidget(self.button_box)
        self.setLayout(layout)


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Main Window")

        button = QPushButton("Open Dialog")
        button.clicked.connect(self.open_dialog)
        self.setCentralWidget(button)

    def open_dialog(self):
        dialog = CustomDialog(self)
        result = dialog.exec()

        if result == QDialog.DialogCode.Accepted:
            print("User clicked OK")
        else:
            print("User clicked Cancel")


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec())

Let's walk through what's happening here.

The Main Window

The MainWindow class creates a simple window with a single button. When you click that button, it calls open_dialog(), which creates an instance of CustomDialog and displays it using exec().

Once you accept or cancel the dialog, exec() returns and you're back inside the open_dialog method. The main window is still visible and responsive — you can click the button again to reopen the dialog as many times as you like. If you're new to building windows in PyQt6, see our tutorial on creating your first window.

The Event Loop

The block at the bottom of the script is what keeps everything running:

python 
if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec())

QApplication manages the main event loop for your application. Calling app.exec() starts that loop, which keeps the application alive, processing user input, repainting widgets, and handling signals and slots. The application will only exit when you close the main window (or explicitly call app.quit()).

Without this event loop, your script runs top-to-bottom and exits as soon as it reaches the end — which is exactly the behavior you'd see if a dialog was the only thing in your script.

Passing self as Parent

You might have noticed that we pass self when creating the dialog:

python 
dialog = CustomDialog(self)

This sets the main window as the dialog's parent. It means the dialog will appear centered over the main window and will be properly tied to it in terms of window management. It's a good habit to set a parent for your dialogs whenever you can.

Summary

If your app is closing when you dismiss a dialog, make sure you have:

  1. A QApplication instance with its event loop running via app.exec().
  2. A main window that is shown using .show().
  3. Your dialog launched from within that main window, so control returns to it after the dialog is dismissed.

With this structure in place, your dialogs will open and close without affecting the rest of your application. If you want to open additional windows beyond dialogs, take a look at our guide on creating multiple windows in PyQt6.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

November 11, 2026 06:00 AM UTC

Graham Dumpleton

AI as a lifestyle choice

When I wrote about twenty years of mod_wsgi, I mentioned that I was on sabbatical, that AI coding tools had changed what I was prepared to take on, and that I was hoping people would tell me what they think is missing from the current options for hosting Python web applications. Each of those only got a sentence or two in a post which was mostly about Apache, and I want to expand on them here.

I should start by clarifying my situation, or at least try to. I would like to call it retirement. My better half would like to see me still working. Nothing is official either way, and negotiations continue. Some people in my position would buy a caravan and head off around Australia, but I already travelled the outback a lot when I was much younger. What I seem to have chosen to do instead is spend my days building software with AI, and I am having about as much fun creating software as I have ever had.

What the lifestyle looks like

It is only a month since I decided I would start on Muru. There were a few days of planning and thinking before any code got written, and since then I have been working on it seven days a week, days and evenings. Nobody is making me do this. I can't remember the last time something was engaging, fulfilling and fun all at the same time.

Before that sounds too alarming, it doesn't mean I am chained to a desk. AI coding tools have turned out to fit quite well around the rest of life. I can set something going, then head out for a walk, do some gardening, or deal with whatever else needs doing around the house, and come back to see how it went. When I was writing all the code myself, stepping away meant the work stopped. My better half may even be prepared to count that in my favour during the negotiations.

That is what I mean by a lifestyle choice. I said in that earlier post that the coding is the part I find a chore these days, which might seem odd coming from someone who has never really stopped creating software. The making of things was always the point though, not the typing. AI gets rid of the grunt work that can be so frustrating, and leaves me with the ideas and the satisfaction of seeing them work.

Getting away from the meter

It took me a while to realise how much the price of these tools was affecting how I used them. When I was paying $20 a month I was always worried I would run out, and as a result I actually held back from relying on it. Moving up to $200 a month produced the opposite problem. Paying that much, I felt I had to make the most of it, and so would end up using it for a crazy number of hours just to get my money's worth. Neither is a great way to go about something you are meant to be doing for fun.

Right now I don't have that problem, because I have been accepted into the Claude for Open Source Program for a second round. I very much appreciate that, and it has made a huge difference. More than anything it has released me from worrying about the cost, and I feel free to do whatever I want without feeling constrained.

Obviously there is the question of what happens when I go back to having to pay for it, and I don't have a good answer to that yet. For now I am taking as much advantage of it as I can.

Everything starts out as mud

I will acknowledge that AI may not always generate the best code out of the box. To be frank though, neither did I when I started on mod_wsgi. The early versions had plenty wrong with them. I made lots of mistakes and had to go back and do things differently, and it took years of fixing and rethinking to get it to where it is now.

All developers are like this when they start out on something. People seem to forget that as soon as AI is involved, and expect instant perfection of a kind they would never have been able to achieve themselves.

Writing software is an incremental process of improvement, and in some respects can be likened to making a dorodango. This is a Japanese craft where you take ordinary dirt and water and form it into a ball. It starts out as a crude lump of mud. You shape it, let it dry, add layers of fine dust and keep working at the surface, over and over, until you end up with a sphere with a shine on it that looks nothing like what it was made from.

I wrote recently about how calling something "AI slop" is often just a way of saying you didn't read it. Well, slop is more or less what a dorodango starts out as. The insult only sticks if you stop there. The AI may be doing a lot of the work on the surface, but deciding what isn't good enough yet is still down to me, and twenty years of getting things wrong with mod_wsgi turns out to be useful for that. I am prepared to put in the hard work to keep polishing Muru until it is something I would trust, and something you would too.

Tinkering in the back room

I have been programming for over 40 years now, and have been involved in free or open source software for nearly all of that time. Even as I head towards retirement, or whatever we end up calling it, I still want to do more. The Python community has always been the most welcoming community I have ever interacted with, and Muru is my way of giving something back to it.

There are many ways of contributing to a community like that. A lot of people give their time to the PSF and to running Python conferences, and we are all better off for the work they do. It just wouldn't suit me, as bureaucracy and organising things are not where my strengths lie. Back room tinkering and creating software has always been my thing, and I expect always will be.

I also know how badly AI written code being handed to open source projects can go, with maintainers left to deal with what lands on them. The difference here, I hope, is that I am not throwing anything over the fence for someone else to look after. If Muru has problems, they are my problems.

The things you never had time to build

I know how a month old project written largely by AI sounds. Muru is not a weekend of vibe coding that will be forgotten about by Christmas though. The code may be new, but the thinking behind it has been accumulating for twenty years. Muru is my thing for the foreseeable future, and I intend to commit to it and see how far I can take it.

I am not doing this only to see what I can achieve based on my own ideas. Part of the reason for putting it out there is so people can see those ideas and come back with their own, about what they would like to see in a system for hosting Python web applications.

A few people have already left comments on my social media posts about the series, showing an interest in specific things Muru is doing. That has been encouraging, and suggests I am at least on the right track. I would like a lot more of that feedback though, especially about what isn't there yet.

If you work with Python web applications for a living, I suspect you have a list. There will be points of friction in your day to day development work that you would love to see gone. There will be things you need to do in production which are much harder than they should be. Some of it you have wanted to fix for years, but you never had the time, or your employer wasn't interested. I know that list well, as I had one of my own for most of my working life.

That is where my situation is now different. Not having a job, I have the time. Open source has always said "patches welcome", which isn't much use when lack of time is the whole problem. So don't send me a patch, just describe the problem.

If you aren't sure where to start, think about the scripts you have written around your deployments that you wish you didn't need, or the thing you end up explaining to every new person who joins the team. Maybe there is something that keeps getting pushed down the backlog, or a production task you put off because of how painful it is. I can't promise everything will make it in, as an idea needs to fit within the overall system I am trying to develop, but if it does then I am happy to look at it.

In that earlier post I said I wasn't going to point to where Muru lives just yet. It is time to change that. To see what Muru can already do, which is a lot more than the posts so far have covered, have a look through the documentation. The place to tell me what you would like to see is the ideas section of the Muru discussions on GitHub. I am not a huge fan of online discussion forums like this, but there don't seem to be any other good choices these days. If they aren't your thing either, then just leave a comment on one of my social media posts when something sounds like it is what you would use or need. Even that much helps me work out where to put the effort.

Polishing in public

You will have to put up with me posting about Muru every day for a while yet. Sorry about that. The posts are actually part of the polishing process, as are the online interactive workshops I have been creating for it using the workshop extension I wrote for JupyterLab. If I can't write a post which explains something easily, or create a workshop which makes it usable for the people who may want to use it, then its design and implementation were bad to start with. Having to explain it is what shows me where it needs more work.

The ball is still pretty muddy in places. I have the time though, I am having fun, and whatever my better half and I end up deciding to call this, I plan to keep polishing. Fingers crossed it ends up with a shine.

November 11, 2026 12:00 AM UTC

Muru: WSGI and ASGI together

The previous post was about serving HTTPS from a development run. This time I want to get back to something I mentioned in passing when I first talked about Muru, which is that it isn't limited to WSGI, and that WSGI and ASGI can be mixed in the one application.

The case where this matters most is an existing WSGI application which now needs websockets. WSGI has no way of doing websockets, so the usual answer is to stand up a separate ASGI server beside the WSGI one, put a proxy in front to route the websocket paths to it, and then work out how the two halves are going to share things like sessions and configuration. The other answer is to port the whole application to ASGI, which for a large application is rarely something anyone wants to do just to get one feature.

With Muru, neither is necessary. The WSGI application can hand a request over to an ASGI application running in the same worker process.

An existing WSGI application

To show how this works I will use a small Flask application, standing in for an application which has been around for years. The project has Flask installed with uv add flask, and the pyproject.toml file names the entrypoint:

[tool.muru]
entrypoint = "shop.app:app"

The application itself has the existing page, plus a new /chat view where the websocket is to live:

from flask import Flask, request

from shop.chat import chat

app = Flask(__name__)


@app.route("/")
def index():
    return "Welcome to the shop\n"


@app.route("/chat")
def chat_view():
    bridge = request.environ["muru.wsgi_to_asgi"]
    if not bridge.is_websocket:
        return "websocket required\n", 426
    return bridge.handoff(chat)

Every WSGI request Muru passes to an application has a muru.wsgi_to_asgi object in the WSGI environ. This is the bridge across to the ASGI side. It can say whether the request is a websocket upgrade, and its handoff() method takes an ASGI application, not a WSGI one. Calling handoff() marks the request so that, once the WSGI side has returned, Muru carries on with the request by passing it to that ASGI application.

What handoff() returns is there for convenience, and is designed to suit both raw WSGI applications and web frameworks. It is an empty iterable, which is what a WSGI application returns for a response with no body, so a raw WSGI application can return it straight away. The same object can also be called as if it were a WSGI application, in which case it responds with a 200 status and returns itself as an empty body. Between them, these cover what web frameworks will accept from a view. Current versions of Flask take an iterator returned from a view as the body of a response, while older versions instead run a callable returned from a view as a WSGI application. Either way, a Flask view can return what handoff() gives back, as the example above does.

For any other framework, how you return from the handler doesn't matter. Once a request has been handed off, Muru ignores whatever response the WSGI side produces, so all that is needed is to call handoff() and then return whatever response the framework expects.

Websockets as callbacks

What the request is handed off to, chat, has to be an ASGI application. You could write that as a raw ASGI coroutine, but if your application is WSGI, the people working on it are probably more at home with synchronous code. For that Muru provides muru.asgi.websocket.Handler, an ASGI application which turns the events of a websocket into method calls on an object of your own. Here is a minimal chat room in shop/chat.py, where every message received is sent on to everyone connected:

from muru.asgi import websocket


class Chat:
    def __init__(self):
        self.sockets = set()

    def on_connect(self, ws):
        self.sockets.add(ws)

    def on_receive(self, ws, message):
        for peer in list(self.sockets):
            try:
                peer.send(message)
            except websocket.ConnectionClosed:
                self.sockets.discard(peer)

    def on_disconnect(self, ws, code, reason):
        self.sockets.discard(ws)


chat = websocket.Handler(Chat())

The methods are called on the worker's application threads, the same threads the WSGI application uses. Calls for any one socket happen one at a time and in order, while different sockets can be handled at the same time. The important part, which I will come back to, is that a socket sitting there waiting for its next message isn't holding on to a thread while it waits.

Turning handoff on

Start this with muructl run and the ordinary page works, but a websocket client connecting to /chat is turned away. The access log shows why, with the view having answered with a 426 status:

426 GET /chat 19B 0.5ms queue=0.0ms app=0.5ms worker=0 id=01M6JWP0E8W9FQ721K81F9AJ9H

Handing requests to ASGI has to be enabled, as it means the worker needs to run an event loop beside its threads. Without the setting, a websocket upgrade is just an ordinary request, so is_websocket is false and the view turns it away. Enabling it is one more line in pyproject.toml:

[tool.muru]
entrypoint = "shop.app:app"
handoff = "asgi"

Run it again and the start-up output confirms the worker now has an event loop as well as its threads:

muru: serving shop.app:app on http://127.0.0.1:8080 with 1 worker of 4 threads and an event loop of concurrency 1000

To check the chat room really works, I used a small client written with the websockets package, installed into a separate virtual environment, which connects as two users and has each of them say something:

import asyncio

from websockets.asyncio.client import connect

URL = "ws://127.0.0.1:8080/chat"


async def main():
    async with connect(URL) as alice, connect(URL) as bob:
        await alice.send("hello from alice")
        print("alice got:", await alice.recv())
        print("bob got:", await bob.recv())
        await bob.send("hi alice, bob here")
        print("alice got:", await alice.recv())
        print("bob got:", await bob.recv())


asyncio.run(main())

Running this the output is:

alice got: hello from alice
bob got: hello from alice
alice got: hi alice, bob here
bob got: hi alice, bob here

The access log has a line for each websocket when it is accepted, and another when it closes, recording who closed it, how, and how many messages went each way. Both are marked with handoff=asgi so you can tell the request was handed across:

101 GET /chat 0B 1.8ms queue=0.0ms app=0.5ms worker=0 handoff=asgi id=01M6JWPC691F8KMJB3TQH8CKSK
101 GET /chat 0B 0.3ms queue=0.1ms app=0.2ms worker=0 handoff=asgi id=01M6JWPC6B1F8KMJB3TQH8CKSK
101 GET /chat 34B 1.7ms websocket=closed by=client code=1000 received=1 sent=2 worker=0 handoff=asgi id=01M6JWPC6B1F8KMJB3TQH8CKSK
101 GET /chat 34B 2.9ms websocket=closed by=client code=1000 received=1 sent=2 worker=0 handoff=asgi id=01M6JWPC691F8KMJB3TQH8CKSK

Sockets don't hold threads

The reason this isn't just a party trick is what happens to the thread the WSGI view ran on. Once the view hands the request off and returns, the thread goes back to the pool to handle other requests. The websocket now lives on the worker's event loop, and only borrows a thread for the brief time it takes to run one of the callback methods.

That matters, because this worker only has four threads. If each open websocket tied up a thread, the fifth person to open the chat would stop the rest of the site from responding. To check, I changed the client to open fifty websockets to /chat, leave them all open, and then make an ordinary request to the home page:

50 websockets open
GET / answered b'Welcome to the shop\n' in 6.2ms

Fifty open websockets on a worker with four threads, and the rest of the site carries on as normal.

Side by side by path

Handing off in code is the right tool when the decision about which side a request goes to belongs in the application, such as wanting the Flask view to check the user is logged in before accepting the websocket. When the decision is simply about the path, Muru can instead mount an ASGI application beside the WSGI one.

Say newer parts of the shop are being written as async code under /api. Here is a stand-in for that in shop/api.py, where the sleep pretends to be waiting on some other service:

import asyncio
import json


async def app(scope, receive, send):
    if scope["type"] != "http":
        return

    await asyncio.sleep(1)  # stand in for waiting on another service

    body = json.dumps({"root_path": scope["root_path"], "path": scope["path"]}).encode()
    await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"application/json")]})
    await send({"type": "http.response.body", "body": body})

Mounting it at /api is a table in pyproject.toml:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"

On start-up, Muru says what each entrypoint is being served as, and where each mount is:

muru.host[89570]: shop.app:app is served as WSGI
muru.host[89570]: mount api: shop.api:app is served as ASGI
...
muru: mount api serves /api from shop.api:app

Anything under /api now goes to the ASGI application, and everything else to Flask. The ASGI application sees the mount path as its root_path, as the ASGI specification asks:

curl http://127.0.0.1:8080/api/orders

{"root_path": "/api", "path": "/api/orders"}

Both applications run in the same worker processes, imported by the same preload, from the one project.

Requests to the ASGI application run on the worker's event loop, so they aren't held back by the four threads the WSGI application has. Firing off twenty requests to /api at once, each awaiting that one second sleep, they all finished together in 1.06 seconds, rather than queuing up four at a time behind the threads.

Left alone, a mount shares the worker's event loop, and so its concurrency, with everything else running on it, such as the websockets handed off from Flask. If one part of the site shouldn't be able to take all of that, a mount can be given a concurrency of its own:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"
concurrency = 500

Muru confirms this when it starts:

muru: mount api serves /api from shop.api:app with concurrency 500 of its own

There is a similar max_connections setting to put a limit on how many websockets an ASGI mount can hold open.

And back the other way

Handing off works in the other direction too. An ASGI application can hand a request to WSGI code, which is useful when moving an application to ASGI a piece at a time, leaving the parts not yet converted as they are. In the scope given to an ASGI application, Muru's extensions include an asgi_to_wsgi bridge, whose handoff() runs a WSGI application for the rest of the request:

def legacy_report(environ, start_response):
    body = f"report generated by WSGI code for {environ['SCRIPT_NAME']}{environ['PATH_INFO']}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body]


async def app(scope, receive, send):
    if scope["type"] != "http":
        return

    if scope["path"].startswith("/api/reports/"):
        return scope["extensions"]["muru"]["asgi_to_wsgi"].handoff(legacy_report)

    ...

As before, this has to be enabled, and if you forget, the error says exactly what to add:

muru.HandoffNotEnabled: mount api does not enable handing requests to WSGI; set handoff = "wsgi" on the mount

WSGI code needs threads to run on, so as well as setting handoff, the mount here is given two threads of its own:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"
concurrency = 500
handoff = "wsgi"
threads = 2

The start-up output now shows the mount with its own threads as well as its own concurrency:

muru: mount api serves /api from shop.api:app on 2 threads of its own with concurrency 500 of its own

Now a request for a report goes to the old WSGI code, while everything else under /api stays with the ASGI application:

curl http://127.0.0.1:8080/api/reports/daily

report generated by WSGI code for /api/reports/daily

with the access log marking it with handoff=wsgi:

200 GET /api/reports/daily 53B 0.4ms queue=0.0ms app=0.4ms mount=api worker=0 handoff=wsgi id=01M6JWRJ0DQ38194C2WQX2KDCJ

gRPC as well

The same mount mechanism also takes care of the third kind of service I mentioned in the first post, which is gRPC. A gRPC mount doesn't have a path. Instead, calls are recognised by their content type and service name, so gRPC and ordinary HTTP share the same port. The entrypoint is a function which registers the servicers generated by grpcio-tools, just as you would with a grpcio server:

class Greeter(greeter_pb2_grpc.GreeterServicer):
    def SayHello(self, request, context):
        return greeter_pb2.HelloReply(message=f"Hello {request.name}")


def register(server):
    greeter_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)

[tool.muru.mounts.api]
interface = "grpc"
entrypoint = "svc.app:register"

muru.host[91221]: mount api: svc.app:register is served as gRPC
...
muru: mount api serves the gRPC services of svc.app:register

With that, a grpcio client calling SayHello and a browser fetching the home page can talk to the same port of the same development run, and both show up in the one access log:

200 GET / 30B 0.2ms queue=0.0ms app=0.2ms worker=0 id=01M6JWS8HV7NK43EP3VYEA225P
200 POST /greet.Greeter/SayHello 14B 0.9ms queue=0.0ms app=0.8ms worker=0 id=01M6JWS8KT7NK43EP3VYEA225P

One catch is that grpcio doesn't provide wheels for free-threaded Python, so for gRPC I had to use a regular build of Python. There is a lot more to gRPC in Muru, such as streaming calls and gRPC-Web, which I may come back to in a post of its own.

What's next

What I like about all of this is that you don't have to make a big decision up front. An application can start as WSGI, take websockets when it needs them, grow async parts under a path, and move across to ASGI piece by piece if that ever makes sense, all without becoming more than one deployment.

That wraps up what I wanted to show of Muru in development mode. Next it is time to move on to the daemon, starting with deploying an application to a server, where the releases that a reload creates in development mode become deploys that can run side by side, take a share of the traffic, and be rolled back.

November 11, 2026 12:00 AM UTC

Seth Michael Larson

Creating for yourself

Today marks the 7th anniversary of starting this blog. If you've been following along for a while: thank you! 😊

After observing Volunteer Responsibility Amnesty Day in 2025 and stopping most of my volunteer OSS contributions, I really leaned hard into using my creative leisure time "for me". This has helped protect my flame from burn-out and focus on areas I'm excited about, like writing, video-games (retro, modding, ...developing? 👀), and keeping my volunteer OSS focus on mostly a single project: truststore. This year was the first time since 2016 that I was uninvolved in the latest release of urllib3, which felt both deeply rewarding and strange as I read the v2.8.0 release notes.

Writing

This year I published 54 articles to my blog, an average of once per week across a year. This wasn't a specific goal I had in mind, looking through the past year of articles has me thinking "oh, that was this year??". Writing continued to be a focus of my role at the PSF, too, where I wrote 14 articles for Python Insider, 5 for the PSF blog, and 2 for the Python Package Index blog (that's a lot of yapping!)

The piece I'm most proud of from this year is “Food JPEGs in Super Smash Bros. and Kirby Air Riders”. This took tons of research, buying the Japanese Kirby Air Ride, writing some custom software, and I think this work really showed through in the end. This was my most-visited article of the year (mostly from the Hacker News front page).

My under-dog favorite is “Adding Animal Crossing events to your digital calendar” where I got to deep-dive into how Animal Crossing events work, differences between regions and games, the Animal Crossing decompilation project, and bought a physical Animal Crossing calendar too. I learned about the ICS calendar format and which features are supported across different platforms like iOS and Android.

Software

I took a much more fast-and-loose approach to sharing software I've worked on this year, opting to use this blog and Gists instead of each project needing a forge, issue tracker, and to be published to PyPI. I found that this approach let me spend more time on what actually mattered to me: being able to tell the story!

I continued building tools allowing you to extract and archive ROMs from legally owned physical copies of games:

Finally, I created tools that track my play activity across all the platforms I care about. Can't improve what you don't measure, this data comes up in the next section:

Gaming

Ever since 2023 I've been “in my feels” about gaming. I considered myself a gamer, but I wasn't actually playing games. I set a goal to get back into the hobby more seriously, and figure out how I can integrate play back into my week. 2025 was successful already, and 2026 was even more so.

I ended up really experimenting this year by writing about a slice of my experience for four of the games I completed. I don't think I'll want to write about every game I play, but thinking deeply about my feelings while playing the game and especially the time between play sessions has been an interesting lens to look through. Maybe more to come here?

These are all the games that are "completable" that I started and mostly finished this year:

Started Game Play Duration
2026-01-01 The Legend of Zelda: Link's Awakening 12 hours
2026-03-17 The Legend of Zelda: Oracle of Seasons (DNF) 6.5 hours
2026-03-31 Kaeru no Tame ni Kane wa Naru 5 hours
2026-04-13 Sonic Mania 3.5 hours
2026-05-03 Super Mario 3D World + Bowser's Fury 17 hours
2026-05-28 Mina the Hollower (DNF) 5 hours
2026-08-16 Big Walk 6.75 hours
2026-09-11 Paper Mario: The Origami King 20.5 hours
2026-11-04 The Legend of Zelda: Ocarina of Time (WIP!) 6.75 hours
Total ~83 hours

Using the three play activity tracking projects I have data about play activity from the Nintendo Switch, GameCube, and the Delta emulator. This data showed I played 83 hours of games this year compared to 78 in 2025.

I'm ending this year playing “Ocarina of Time” for the first time and oh my is this game incredible! I cannot stop thinking about it having immersed myself completely in the design, discussion, and themes prior to starting. I'm part-way through the Forest Temple and all I can think is that there are 4 more temples to complete, what a game!


Thanks for reading ♥ I would love to hear your thoughts! Contact me via Mastodon, Bluesky, or email. Browse the blog archive. Check out my blogroll.


November 11, 2026 12:00 AM UTC

November 10, 2026

Python Docs Editorial Board

Meeting Minutes: Nov 10, 2026

Meeting Minutes from Python Docs Editorial Board: Nov 10, 2026

November 10, 2026 08:00 PM UTC

PyCoder’s Weekly

Issue #760: Type Checkers, Claude Code Hooks, MarkItDown, and More (2026-11-10)

#760 – NOVEMBER 10, 2026
View in Browser »

The PyCoder’s Weekly Logo

How a Python Type Checker Decides a Variable Is Unused

A variable is unused only if no read resolves to its binding and in Python that read can sit inside a nested function or comprehension. This post walks through how ty, Astral’s Python type checker, finds local bindings nobody reads.
DENYSZHAK.COM • Shared by Denys Zhak

How to Automate Your Workflow With Claude Code Hooks

Learn how to use Claude Code hooks that auto-format your Python code after every edit, enforce your chosen tools, and ping you when a task finishes.
REAL PYTHON

Why Your Python Agent Returns Outdated Memories

alt

A user changes a preference, but vector search still surfaces the old one alongside the new. Link the new memory to the one it replaces and mark the old one historical, so your agent retrieves what’s current and keeps the history. Explore the walkthrough and runnable notebook.
ORACLE sponsor

Converting Documents to Markdown With MarkItDown

Learn how to use MarkItDown to convert Excel files, PDFs, and images into clean Markdown that’s ready for LLMs and AI workflows.
REAL PYTHON course

Quiz: Converting Documents to Markdown With MarkItDown

REAL PYTHON

Django Security Releases Issued: 6.1.3, 6.0.10, and 5.2.19

DJANGO SOFTWARE FOUNDATION

Agentic Coding With Codex CLI: A Live Course for Python Developers

If you pay for ChatGPT Plus, you already have Codex CLI, an agent that runs in your terminal, edits your files, and runs your tests. Over two live half-days on November 14–15, you’ll use it to build a Python CLI app from an empty directory to GitHub, write an AGENTS.md file so it follows your project’s rules, and practice debugging the code it writes. You keep the recordings. Registration closes Friday, November 13 →
REAL PYTHON

Discussions

Pull Request Limits for CPython Repo

PYTHON.ORG

Articles & Tutorials

Soft Delete in Django Without Making a Mess

Soft delete sounds simple: add a deleted_at field and stop calling DELETE. The trouble starts a little later, when deleted rows still show up in queries. This tutorial shows a soft-delete setup that handles those cases without hiding too much magic in the model layer.
STEIN HELSET

Hello Python 3.15, Goodbye 3.10, and Now I Can Use 3.11

Simon covers the latest features in Python 3.15. If you’ve seen that already, the second half of the article is also worth reading: now that 3.10 is end-of-life, what Python 3.11 features can you use everywhere?
SIMON WILLISON

Parsing JSON at Tens of Gigabytes Per Second

This article talks about fastsimdjson, a drop-in replacement for the json module in the standard library. It has been optimized for speed, demonstrated by reading large amounts of JSON GitHub commit data.
DANIEL LEMIRE

Why Python Devs Should Know About the ABI

Why should every Python developer know about the CPython ABI? This week on the show, we speak with Nathan Goldbaum about free-threaded Python and the changes to the Python Application Binary Interface.
REAL PYTHON podcast

Building a Lift Scheduling Algorithm in Python

This article walks you through the logic needed to code a scheduling algorithm for elevators, starting with a single car to a more complex multi-car system.
PIETER CLAERHOUT

Histogram Plotting: NumPy, Matplotlib, pandas & Seaborn

In this tutorial, you’ll be equipped to make production-quality, presentation-ready Python histogram plots with a range of choices and features.
REAL PYTHON

Python Exceptions: An Introduction

Learn what Python exceptions are, how to raise them, how to handle them with try, except, else, and finally, and how to create custom ones.
REAL PYTHON

Python Program Lexical Structure

Dig into the Python program structure: the syntactic elements that make up statements, and how statements group into larger blocks.
REAL PYTHON

Announcing New Security Developer at the PSF

The Python Software Foundation welcomes Brett Smith as an additional security developer.
PYTHON SOFTWARE FOUNDATION

Python’s Binary Wheels and Supply Chain Attacks

Why precompiled Python wheels are a weak spot in the supply chain security.
ARTEM GOLUBIN

Projects & Code

pgcli: Postgres CLI With Autocomple & Syntax Highlighting

GITHUB.COM/DBCLI

django-mcpz: Easy Peasy MCP Servers in Django

GITHUB.COM/ADAMCHAINZ

Ghostman: Postman Alternative Built in Python With Textual

GITHUB.COM/4YK5 • Shared by 4yk5

VizTracer: Visualize Code Debugging/Profiling/Logging

GITHUB.COM/GAOGAOTIANTIAN

django-ses: Django Backend for Amazon’s Simple Email Service

GITHUB.COM/DJANGO-SES

Events

Weekly Real Python Office Hours Q&A (Virtual)

November 11, 2026
REALPYTHON.COM

Python Help

November 12, 2026
MEETUP.COM

Python Leiden User Group

November 12, 2026
PYTHONLEIDEN.NL

Python Atlanta

November 13, 2026
MEETUP.COM

PyCon Chile 2026

November 14 to November 16, 2026
PYCON.CL

PyCon Hong Kong 2026

November 14 to November 16, 2026
PYCON.HK

Wagtail Space 2026

November 18 to November 21, 2026
WAGTAIL.ORG

PyCon Wroclaw 2026

November 20 to November 21, 2026
PYCONWROCLAW.COM

PyCon Ireland 2026

November 21 to November 22, 2026
PYCON.IE

Happy Pythoning!
This was PyCoder’s Weekly Issue #760.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

November 10, 2026 07:30 PM UTC

Python Bytes

#504 Who turned the GIL back on?

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://blog.pypi.org/posts/2026-11-06-ui-updates-complete/?featured_on=pythonbytes">PyPI's redesigned project pages</a> now warn when provenance changes</strong></li> <li><strong><a href="https://mkennedy.codes/posts/catch-the-import-name-the-culprit-keep-threads-free-introducing-gilguard/?featured_on=pythonbytes">gilguard catches the import that turns the GIL back on</a></strong></li> <li><strong><a href="https://pip.pypa.io/en/stable/news/#v26-3">pip 26.3:</a> pylock extras, and pip says plainly that untrusted wheels aren't safe</strong></li> <li><strong><a href="https://heyutils.com/tools/?featured_on=pythonbytes">All developer tools</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=rPczSyhinpc' style='font-weight: bold;' data-umami-event="Livestream-Past" data-umami-event-episode="504">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by <a href="https://pythonbytes.fm/longlink"><strong>LongLink</strong></a>: Open-source platform for building and running custom business applications with Python and FastAPI <strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy">Mastodon</a> / <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/mkennedy?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/mkennedy/?featured_on=pythonbytes">LinkedIn</a></li> <li>Calvin: <a href="https://sixfeetup.social/@calvin?featured_on=pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/calvinhp.com?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/calvinhp?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/calvinhp/?featured_on=pythonbytes">LinkedIn</a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/pythonbytes.fm">BlueSky</a> / <a href="https://x.com/PythonBytes?featured_on=pythonbytes">X</a> Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Tuesday at 7am PT</strong>. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</li> </ul> <p><strong>Calvin #1: <a href="https://blog.pypi.org/posts/2026-11-06-ui-updates-complete/?featured_on=pythonbytes">PyPI's redesigned project pages</a> now warn when provenance changes</strong></p> <ul> <li>Live now on <a href="http://pypi.org/?featured_on=pythonbytes">pypi.org</a> (Nov 6). Announced in July, shaped by user research, funded by OpenSSF and built by Kabu Creative.</li> <li>New Security tab on every project page shows each release's provenance state and per-file attestations, with a link to the transparency log.</li> <li>Two attestation types: PyPI Publish (uploaded by a Trusted Publisher) and SLSA Build Provenance (ties a file to its source commit, build platform and instructions).</li> <li>"No provenance" isn't a malware flag; it means the file's origin can't be cryptographically verified. That's typical for older releases or projects without Trusted Publishing.</li> <li>Four new provenance warnings: Incomplete (e.g. wheels via Trusted Publishing but sdist via API token), Inconsistent (mixed build systems or repos), Change (different repo or workflow than last release), Loss (stopped publishing attestations).</li> <li>Warnings are meant to catch files that may not have come from the maintainers, such as after an account or publishing-workflow takeover. PyPI says a Change of Provenance "can also indicate a compromised pipeline or account takeover," and an unexpected Loss of Provenance should be treated "with skepticism until verified.”</li> <li>Warnings now sit at the top of the page with yanked, archived and pre-release banners. A quarantine banner overrides all of them. Sidebar metadata is labeled "PyPI data" or "Data verified by PyPI."</li> <li>Release files tab gets search, filters, a per-file provenance column and orange icons for "late" files (uploaded more than 14 days after release). Those only appear on releases from before July 2026, when PyPI started rejecting new files after 14 days.</li> </ul> <p><strong>Michael #2: <a href="https://mkennedy.codes/posts/catch-the-import-name-the-culprit-keep-threads-free-introducing-gilguard/?featured_on=pythonbytes">gilguard catches the import that turns the GIL back on</a></strong></p> <p>gilguard catches a nasty free-threading surprise: importing a C extension without declared free-threading support can turn the GIL back on for the rest of the process. Import gilguard first and it raises a detailed error naming the extension, distribution/version, import chain, and the line of your code responsible.</p> <ul> <li>A transitive dependency can undo your move to free-threaded Python. Fail at the offending import instead of discovering lost parallelism in a benchmark.</li> <li>Uses import hooks plus CPython's warning path to catch GIL activation; GILEnabledError inherits BaseException, so even except Exception won't swallow it.</li> <li>python -m gilguard audits installed top-level packages in fresh interpreters. Supports targeted modules and JSON output; exits 1 when it finds an offender for CI.</li> <li>The audit misses extensions loaded lazily on first use. Exercise those paths in tests with gilguard installed early; one process can identify only its first GIL-flipping import.</li> <li>GILGUARD_ACTION=warn reports without stopping; off permits standard builds. Default mode raises on standard Python or an already-enabled GIL. PYTHON_GIL=0 prevents activation, so the guard cannot detect offenders there.</li> <li>Published on <a href="https://pypi.org/project/gilguard/?featured_on=pythonbytes">PyPI</a> as 0.1.0. Install with uv pip install gilguard. Python 3.13+, MIT; no runtime dependencies.</li> </ul> <p><strong>Calvin #3: <a href="https://pip.pypa.io/en/stable/news/#v26-3">pip 26.3:</a> pylock extras, and pip says plainly that untrusted wheels aren't safe</strong></p> <ul> <li>pip 26.3 released Nov 2. Its security policy now says pip "has never considered installing untrusted wheels safe," because Python's import system is dynamic, so pip can't guarantee installing a wheel won't execute third-party code (#14412).</li> <li>The change: pip blocks unexpected imports during wheel installation "to avoid loading code from packages being replaced." It loads what it needs for installing and error reporting up front (PR #14247, merged Nov 2: https://github.com/pypa/pip/pull/14247).</li> <li>CLI vs. library use: from the command line, imports stay blocked until pip exits. Code that calls pip programmatically gets warnings after control returns.</li> <li>Rolled out in steps: pip 26.1 started warning ("Unexpected import of '...' after pip install started. pip 26.3 will enforce this behavior change"), and 26.3 turns it into an error. Tracking issue #13842 collected reports before enforcement: https://github.com/pypa/pip/issues/13842</li> <li>What tripped it in the wild: a <code>.pth</code> startup hook at Pradyun Gedam's workplace that registers an atexit handler to report interpreter usage metrics over HTTP (https://github.com/pypa/pip/issues/13842#issuecomment-4613463933), plus warnings involving <code>lark_oapi</code> and <code>rich._emoji_codes</code>.</li> <li>Escape-hatch debate: the author, Damian Shaw (notatallshaw), suggested appending module names to <code>pip._internal.commands.install._EAGER_IMPORTS</code>. Pradyun objected that this requires importing from <code>pip._internal</code>, that importing pip's install command is expensive (about 689 ms with no .pyc files and a cold filesystem cache), and that pip isn't necessarily installed in every environment. Shaw's reply: it "just moves where the import cost is," and turning the warning into an error is "crucial."</li> <li>Interrupted installs: pip now writes recovery metadata and reports recovery failures and where it kept backups. An interrupted uninstall may need manual recovery, and an interrupted --target install can leave a partial target directory.</li> <li>Also in 26.3: <code>-r pylock.toml[extra1,extra2]</code> for extras, <code>--group pylock.toml:docs</code> for non-default groups, <code>--no-binary</code> with pylock files, and pip refuses to follow an HTTP redirect to a file: URL.</li> </ul> <p><strong>Michael #4: <a href="https://heyutils.com/tools/?featured_on=pythonbytes">All developer tools</a></strong></p> <p>HeyUtils is a collection of 306 free developer utilities in 41 categories, with no account or installation required. The full catalog covers data conversion, APIs, networking, DevOps, frontend work and binary protocols. Python developers get model generation and packaging helpers alongside the everyday formatters and decoders worth bookmarking.</p> <ul> <li>One bookmark for the tiny jobs that interrupt coding: inspect a payload, compare configs, decode a token or figure out a cron schedule.</li> <li>Data tools include JSON/YAML/TOML conversion, CSV joins, JSONPath and JMESPath testers, a jq playground, and a SQLite viewer with read-only queries.</li> <li>Python-specific options go beyond regex: generate dataclasses or Pydantic models from JSON, check PEP 440 versions, and inspect requirements.txt or pyproject.toml dependencies.</li> <li>The collection also covers JWT/OAuth debugging, DNS and TLS checks, Docker Compose, Kubernetes YAML, GitHub Actions, and frontend helpers such as color contrast checks.</li> <li>Most tools process input locally, according to the site. Network tools make requests after you start them and explain what they send; don't assume every page is offline.</li> <li>Generated models need review: sample data cannot describe a complete validation contract, and dataclass type hints do not validate incoming values.</li> </ul> <p><strong>Extras</strong></p> <p>Calvin:</p> <ul> <li><a href="https://github.com/hedgerow-dev/coldframe?featured_on=pythonbytes">Coldframe</a>: a stdlib-only honeypot that poses as a supermarket fridge controller** Michael:</li> <li><a href="https://pyfound.blogspot.com/2026/10/announcing-new-security-developer-2026.html?featured_on=pythonbytes">Announcing new Security Developer at the PSF</a> <ul> <li>Brett Smith is joining the PSF as a Security Developer, a new role funded by Anthropic. He'll work with Seth Larson and Mike Fiedler on vulnerability response and PyPI malware removal, with more time for proactive work as the team's workload grows. <ul> <li>More paid security capacity for the Python and PyPI infrastructure we depend on every day.</li> <li>The PSF reports 3-4x the volume of reports and CVEs for Python and pip compared with a year ago.</li> <li>Malware published and removed from PyPI is up 60% year over year. That's a workload measure, not a claim that 60% more packages are malicious.</li> <li>Planned work includes scaling PyPI's autonomous quarantine program and streamlining how the Python Security Response Team handles reports.</li> <li>Smith brings software development and Linux administration experience, including work at W3C and Software Freedom Conservancy.</li> </ul></li> </ul></li> </ul> <p><strong>Joke: <a href="https://x.com/PR0GRAMMERHUM0R/status/2016768305395040692?featured_on=pythonbytes">No papa</a></strong></p>

November 10, 2026 07:04 PM UTC