skip to navigation
skip to content

Planet Python

Last update: December 06, 2023 09:42 PM UTC

December 06, 2023

Glyph Lefkowitz

Safer, Not Later

Facebook — and by extension, most of Silicon Valley — rightly gets a lot of shit for its old motto, “Move Fast and Break Things”.

As a general principle for living your life, it is obviously terrible advice, and it leads to a lot of the horrific outcomes of Facebook’s business.

I don’t want to be an apologist for Facebook. I also do not want to excuse the worldview that leads to those kinds of outcomes. However, I do want to try to help laypeople understand how software engineers—particularly those situated at the point in history where this motto became popular—actually meant by it. I would like more people in the general public to understand why, to engineers, it was supposed to mean roughly the same thing as Facebook’s newer, goofier-sounding “Move fast with stable infrastructure”.

Move Slow

In the bad old days, circa 2005, two worlds within the software industry were colliding.

The old world was the world of integrated hardware/software companies, like IBM and Apple, and shrink-wrapped software companies like Microsoft and WordPerfect. The new world was software-as-a-service companies like Google, and, yes, Facebook.

In the old world, you delivered software in a physical, shrink-wrapped box, on a yearly release cycle. If you were really aggressive you might ship updates as often as quarterly, but faster than that and your physical shipping infrastructure would not be able to keep pace with new versions. As such, development could proceed in long phases based on those schedules.

In practice what this meant was that in the old world, when development began on a new version, programmers would go absolutely wild adding incredibly buggy, experimental code to see what sorts of things might be possible in a new version, then slowly transition to less coding and more testing, eventually settling into a testing and bug-fixing mode in the last few months before the release.

This is where the idea of “alpha” (development testing) and “beta” (user testing) versions came from. Software in that initial surge of unstable development was extremely likely to malfunction or even crash. Everyone understood that. How could it be otherwise? In an alpha test, the engineers hadn’t even started bug-fixing yet!

In the new world, the idea of a 6-month-long “beta test” was incoherent. If your software was a website, you shipped it to users every time they hit “refresh”. The software was running 24/7, on hardware that you controlled. You could be adding features at every minute of every day. And, now that this was possible, you needed to be adding those features, or your users would get bored and leave for your competitors, who would do it.

But this came along with a new attitude towards quality and reliability. If you needed to ship a feature within 24 hours, you couldn’t write a buggy version that crashed all the time, see how your carefully-selected group of users used it, collect crash reports, fix all the bugs, have a feature-freeze and do nothing but fix bugs for a few months. You needed to be able to ship a stable version of your software on Monday and then have another stable version on Tuesday.

To support this novel sort of development workflow, the industry developed new technologies. I am tempted to tell you about them all. Unit testing, continuous integration servers, error telemetry, system monitoring dashboards, feature flags... this is where a lot of my personal expertise lies. I was very much on the front lines of the “new world” in this conflict, trying to move companies to shorter and shorter development cycles, and move away from the legacy worldview of Big Release Day engineering.

Old habits die hard, though. Most engineers at this point were trained in a world where they had months of continuous quality assurance processes after writing their first rough draft. Such engineers feel understandably nervous about being required to ship their probably-buggy code to paying customers every day. So they would try to slow things down.

Of course, when one is deploying all the time, all other things being equal, it’s easy to ship a show-stopping bug to customers. Organizations would do this, and they’d get burned. And when they’d get burned, they would introduce Processes to slow things down. Some of these would look like:

  1. Let’s keep a special version of our code set aside for testing, and then we’ll test that for a few weeks before sending it to users.
  2. The heads of every department need to sign-off on every deployed version, so everyone needs to spend a day writing up an explanation of their changes.
  3. QA should sign off too, so let’s have an extensive sign-off process where each individual tester does a fills out a sign-off form.

Then there’s my favorite version of this pattern, where management decides that deploys are inherently dangerous, and everyone should probably just stop doing them. It typically proceeds in stages:

  1. Let’s have a deploy freeze, and not deploy on Fridays; don’t want to mess up the weekend debugging an outage.
  2. Actually, let’s extend that freeze for all of December, we don’t want to mess up the holiday shopping season.
  3. Actually why not have the freeze extend into the end of November? Don’t want to mess with Thanksgiving and the Black Friday weekend.
  4. Some of our customers are in India, and Diwali’s also a big deal. Why not extend the freeze from the end of October?
  5. But, come to think of it, we do a fair amount of seasonal sales for Halloween too. How about no deployments from October 10 onward?
  6. You know what, sometimes people like to use our shop for Valentine’s day too. Let’s just never deploy again.

This same anti-pattern can repeat itself with an endlessly proliferating list of “environments”, whose main role ends up being to ensure that no code ever makes it to actual users.

… and break things anyway

As you may have begun to suspect, there are a few problems with this style of software development.

Even back in the bad old days of the 90s when you had to ship disks in boxes, this methodology contained within itself the seeds of its own destruction. As Joel Spolsky memorably put it, Microsoft discovered that this idea that you could introduce a ton of bugs and then just fix them later came along with some massive disadvantages:

The very first version of Microsoft Word for Windows was considered a “death march” project. It took forever. It kept slipping. The whole team was working ridiculous hours, the project was delayed again, and again, and again, and the stress was incredible. [...] The story goes that one programmer, who had to write the code to calculate the height of a line of text, simply wrote “return 12;” and waited for the bug report to come in [...]. The schedule was merely a checklist of features waiting to be turned into bugs. In the post-mortem, this was referred to as “infinite defects methodology”.

Which lead them to what is perhaps the most ironclad law of software engineering:

In general, the longer you wait before fixing a bug, the costlier (in time and money) it is to fix.

A corollary to this is that the longer you wait to discover a bug, the costlier it is to fix.

Some bugs can be found by code review. So you should do code review. Some bugs can be found by automated tests. So you should do automated testing. Some bugs will be found by monitoring dashboards, so you should have monitoring dashboards.

So why not move fast?

But here is where Facebook’s old motto comes in to play. All of those principles above are true, but here are two more things that are true:

  1. No matter how much code review, automated testing, and monitoring you have some bugs will can only be found by users interacting with your software.
  2. No bugs can be found merely by slowing down and putting the deploy off another day.

Once you have made the process of releasing software to users sufficiently safe that the potential damage of any given deployment can be reliably limited, it is always best to release your changes to users as quickly as possible.

More importantly, as an engineer, you will naturally have an inherent fear of breaking things. If you make no changes, you cannot be blamed for whatever goes wrong. Particularly if you grew up in the Old World, there is an ever-present temptation to slow down, to avoid shipping, to hold back your changes, just in case.

You will want to move slow, to avoid breaking things. Better to do nothing, to be useless, than to do harm.

For all its faults as an organization, Facebook did, and does, have some excellent infrastructure to avoid breaking their software systems in response to features being deployed to production. In that sense, they’d already done the work to avoid the “harm” of an individual engineer’s changes. If future work needed to be performed to increase safety, then that work should be done by the infrastructure team to make things safer, not by every other engineer slowing down.

The problem is that slowing down is not actually value neutral. To quote myself here:

If you can’t ship a feature, you can’t fix a bug.

When you slow down just for the sake of slowing down, you create more problems.

The first problem that you create is smashing together far too many changes at once.

You’ve got a development team. Every engineer on that team is adding features at some rate. You want them to be doing that work. Necessarily, they’re all integrating them into the codebase to be deployed whenever the next deployment happens.

If a problem occurs with one of those changes, and you want to quickly know which change caused that problem, ideally you want to compare two versions of the software with the smallest number of changes possible between them. Ideally, every individual change would be released on its own, so you can see differences in behavior between versions which contain one change each, not a gigantic avalanche of changes where any one of hundred different features might be the culprit.

If you slow down for the sake of slowing down, you also create a process that cannot respond to failures of the existing code.

I’ve been writing thus far as if a system in a steady state is inherently fine, and each change carries the possibility of benefit but also the risk of failure. This is not always true. Changes don’t just occur in your software. They can happen in the world as well, and your software needs to be able to respond to them.

Back to that holiday shopping season example from earlier: if your deploy freeze prevents all deployments during the holiday season to prevent breakages, what happens when your small but growing e-commerce site encounters a catastrophic bug that has always been there, but only occurs when you have more than 10,000 concurrent users. The breakage is coming from new, never before seen levels of traffic. The breakage is coming from your success, not your code. You’d better be able to ship a fix for that bug real fast, because your only other option to a fast turn-around bug-fix is shutting down the site entirely.

And if you see this failure for the first time on Black Friday, that is not the moment where you want to suddenly develop a new process for deploying on Friday. The only way to ensure that shipping that fix is easy is to ensure that shipping any fix is easy. That it’s a thing your whole team does quickly, all the time.

The motto “Move Fast And Break Things” caught on with a lot of the rest of Silicon Valley because we are all familiar with this toxic, paralyzing fear.

After we have the safety mechanisms in place to make changes as safe as they can be, we just need to push through it, and accept that things might break, but that’s OK.

Some Important Words are Missing

The motto has an implicit preamble, “Once you have done the work to make broken things safe enough, then you should move fast and break things”.

When you are in a conflict about whether to “go fast” or “go slow”, the motto is not supposed to be telling you that the answer is an unqualified “GOTTA GO FAST”. Rather, it is an exhortation to take a beat and to go through a process of interrogating your motivation for slowing down. There are three possible things that a person saying “slow down” could mean about making a change:

  1. It is broken in a way you already understand. If this is the problem, then you should not make the change, because you know it’s not ready. If you already know it’s broken, then the change simply isn’t done. Finish the work, and ship it to users when it’s finished.
  2. It is risky in a way that you don’t have a way to defend against. As far as you know, the change works, but there’s a risk embedded in it that you don’t have any safety tools to deal with. If this is the issue, then what you should do is pause working on this change, and build the safety first.
  3. It is making you nervous in a way you can’t articulate. If you can’t describe an known defect as in point 1, and you can’t outline an improved safety control as in step 2, then this is the time to let go, accept that you might break something, and move fast.

The implied context for “move fast and break things” is only in that third condition. If you’ve already built all the infrastructure that you can think of to build, and you’ve already fixed all the bugs in the change that you need to fix, any further delay will not serve you, do not have any further delays.

Unfortunately, as you probably already know,

This motto did a lot of good in its appropriate context, at its appropriate time. It’s still a useful heuristic for engineers, if the appropriate context is generally understood within the conversation where it is used.

However, it has clearly been taken to mean a lot of significantly more damaging things.

Purely from an engineering perspective, it has been reasonably successful. It’s less and less common to see people in the industry pushing back against tight deployment cycles. It’s also less common to see the basic safety mechanisms (version control, continuous integration, unit testing) get ignored. And many ex-Facebook engineers have used this motto very clearly under the understanding I’ve described here.

Even in the narrow domain of software engineering it is misused. I’ve seen it used to argue a project didn’t need tests; that a deploy could be forced through a safety process; that users did not need to be informed of a change that could potentially impact them personally.

Outside that domain, it’s far worse. It’s generally understood to mean that no safety mechanisms are required at all, that any change a software company wants to make is inherently justified because it’s OK to “move fast”. You can see this interpretation in the way that it has leaked out of Facebook’s engineering culture and suffused its entire management strategy, blundering through market after market and issue after issue, making catastrophic mistakes, making a perfunctory apology and moving on to the next massive harm.

In the decade since it has been retired as Facebook’s official motto, it has been used to defend some truly horrific abuses within the tech industry. You only need to visit the orange website to see it still being used this way.

Even at its best, “move fast and break things” is an engineering heuristic, it is not an ethical principle. Even within the context I’ve described, it’s only okay to move fast and break things. It is never okay to move fast and harm people.

So, while I do think that it is broadly misunderstood by the public, it’s still not a thing I’d ever say again. Instead, I propose this:

Make it safer, don’t make it later.


Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support me on Patreon as well! I am also available for consulting work if you think your organization could benefit from expertise on topics like “how do I make changes to my codebase, but, like, good ones”.

December 06, 2023 08:01 PM UTC

Python Engineering at Microsoft

Python Linting in Visual Studio Code – Hinting and Linting Video Series

One of the most important parts of writing code is making sure your code is readable. There are so many positive downstream effects of clean code from its ease to maintain and add features, debug subtle programming errors or find uninitialized variables. Some people call this code hygiene. VS Code has linter extension support that enables you develop faster, produce cleaner code and is tweaked to your set up.

How VS Code handles Python linters

Linters are the development tool that is used to make sure your code is formatted consistently across your team. Then add your linter to your requirements-dev.txt— or otherwise named file to store your development only requirements — and pip install -r requirements-dev.txt. The linter development experience can live entirely within VS Code if you’d like. This means you do not need to pip install pylint in your Python environment for example. However in most collaborative programming projects, I prefer to install my linter in my virtual environment (old habits die hard) so if I want to use local terminal features of Pylint in VS Code, I can.

🔥PRO-TIP: Set your default importStrategy importStrategy is a setting on all of our Python linter extensions that defines which linter should be automatically used in VS Code. I like to set it to fromEnvironment in my User level settings so that it automatically uses the linter from my the Python environment for any project I’m working on with a team, but also allow VS Code to default to any Workspace level settings my team is sharing. settings.json

// use the pylint version with extension
"pylint.importStrategy": "useBundled"

// use the pylint version found in requirements
"pylint.importStrategy": "fromEnvironment"

When you start your project, the first thing you will likely do is activate your virtual environment or open up a container to develop in (like Dev Containers). In VS Code, you’ll select your Python interpreter by using shortcut key Ctrl+P to open up the Command Pallette, and select from the dropdown menu which environment you’d like to use for your project. I select the interpreter associated with my project environment.

Screen shot of the Command Pallette dropdown menu. The top of the menu allows the user to search for a name. The first selection is the option to add an interpreter path. The following options are different Python versions VS Code has found on my machine. The last of five options is starred and

There are many packages for code quality. At the time of this post, VS Code and its active extension-contributing community supports flake8, ruff, pylint and the newly released mypy. The Ruff linter was created from the Python tools extension template by Charlie R. Marsh. The VS Code team encourages community contributed packages and you can learn more in the VS Code documentation.

🔥PRO-TIP: VS Code extension works as soon as its installed VS Code linting is automatically enabled once the extension is installed. You no longer need python.linting.enabled set to true under settings.json.

Enable what you want and disable what you don’t

Navigate to the extensions tab in the activity bar to the far left and search for “Pylint.” I often like to enable pre-release so I can get the latest features and report bugs if I come across them, doing my part for the community.

GIF: Enable Pylint Enabling Pylint from the extensions panel

🔥PRO-TIP: Install isort Install an import sorting extension like isort then use the shortcut key Shift+Alt+Oin order to sort your imports quickly.

🔥PRO-TIP: Toggle Problems tab Use Ctrl+Shift+M to toggle open and close the Problems tab in VS Code to access any issues reported by linters.

GIF: Solving my first problems in VS Code for a Wagtail project Open problems tab in VS Code and sort import order of code

You can specify problems to consistently ignore in your projects by adding disable flags to your settings. You can do this either in your settings panel Ctrl+, or with your settings.json Ctrl+P then typing “Settings JSON” in the text bar. The following can be added via your Workspace or User settings depending on your desired scope.

Here’s what it looks like to disable a few doc string problems among other arguments in Pylint. “Args” are always lists in brackets: settings.json

"pylint.arg": [

The same settings work for other linter extensions: settings.json

"flake8.arg": ["--ignore=E24,W504", "--verbose"],
"ruff.args": ["--config=/path/to/pyproject.toml"]

Add arguments to specify pylint settings in both the settings panel and settings.json

Want to dig more into the VS Code Linting documentation?

Keep in touch!

I host The Python Pulse every second Friday of the month 11 AM PT / 2 PM ET / 7 PM UTC

Python Pulse thumbnail - Getting the most out of Python with VS Code and Azure

… or join me on the Microsoft Python Discord

More Reading…

The post Python Linting in Visual Studio Code – Hinting and Linting Video Series appeared first on Python.

December 06, 2023 07:29 PM UTC

Real Python

Build a Hangman Game With Python and PySimpleGUI

Wouldn’t it be cool to build a hangman game with a nice graphical user interface (GUI) in Python? Maybe you’ve built a text-based user interface version of hangman, and now you want to make an even more attractive game. For you as a Python programmer, building a GUI version of this game can be a rewarding challenge that can take you to the next skill level.

Throughout this tutorial, you’ll build the hangman game in Python in a series of steps. The game will have a neat graphical interface based on the PySimpleGUI library.

In this tutorial, you’ll:

  • Create a GUI for your hangman game using PySimpleGUI
  • Implement the logic for the hangman game
  • Connect the game’s logic with its GUI layer
  • Handle the game results and provide other options

To get the most from this tutorial, you should be comfortable with general Python programming, including how to lay out a project. You should also have some experience working with graphical user interfaces, especially those based on PySimpleGUI. You don’t need to have any previous knowledge about game programming, but it would be a plus.

All the code that you’ll write in this tutorial is available for download. You can get it by clicking the link below:

Get Your Code: Click here to download the free Python source code for your PySimpleGUI hangman game.

Demo: Hangman With Python and PySimpleGUI

In this tutorial, you’ll write an implementation of hangman using Python and PySimpleGUI. Once you’ve run all the steps to build the game, then you’ll end up with a graphical user interface (GUI) that will work something like the following:

In this tutorial, you’ll run through several challenges designed to help you solve specific game-related problems. Ultimately, you’ll have a hangman game that works like the demo above.

Project Overview

When creating a GUI version of hangman, you’ll use a GUI framework that can handle a text-based game like hangman while still providing the ability to draw a hanged man. There are several good GUI engines available for Python. While you can use any of them to good effect, you’ll rely on PySimpleGUI in this tutorial.

PySimpleGUI is a wrapper around several well-known GUI frameworks and libraries, such as Tkinter, wxPython, and PyQt. It allows you to design a GUI as data. You’ll use the Tkinter wrapper in this tutorial, but you can explore the wxPython and QT wrappers on PyPI.

While many people know hangman well, it’s helpful to have a formal description of the game. You’ll use this description to resolve programming issues later during development. The game’s description could vary from person to person. In this tutorial, you’ll describe the game as follows:

  • Game setup: The game of hangman is for two or more players, comprising a selecting player and one or more guessing players.
  • Word selection: The selecting player selects a word that the guessing players will try to guess.
    • The selected word is traditionally represented as a series of underscores for each letter in the word.
    • The selecting player also draws a scaffold to hold the hangman illustration.
  • Guessing: The guessing players attempt to guess the word by selecting letters one at a time.
  • Feedback: The selecting player indicates whether each guessed letter appears in the word.
    • If the letter appears, then the selecting player replaces each underscore with the letter as it appears in the word.
    • If the letter doesn’t appear, then the selecting player writes the letter in a list of guessed letters. Then, they draw the next piece of the hanged man. To draw the hanged man, they begin with the head, then the torso, the arms, and the legs for a total of six parts.
  • Winning conditions: The selecting player wins if the hanged man drawing is complete after six incorrect guesses, in which case the game is over. The guessing players win if they guess the word.
    • If the guess is right, the game is over, and the guessing players win.
    • If the guess is wrong, the game continues.

A game in progress is shown below. In this game, the word is hangman:

A game of hangman in progress

In this tutorial, you’ll make a few additional design decisions while writing the hangman game in Python:

  • The game will be between the computer and one human player.
  • The computer will act as the selecting player and will select the word to guess, process human input, and handle all output.
  • The human player is the guessing player, hereafter simply referred to as the player. When the player knows the word, they continue to guess correct letters until the word is complete.

With this basic understanding of the game and some design decisions for the computer version, you can begin creating the game. First, however, you need to be aware of some knowledge requirements.


The project that that you’ll build in this tutorial will require familiarity with general Python programming. You should have basic knowledge of the following topics:

However, if you don’t have all this knowledge yet, then that’s okay! You might learn more by going ahead and giving the project a shot. You can always stop and review the resources linked here if you get stuck.

Read the full article at »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

December 06, 2023 02:00 PM UTC

Sandipan Dey

Non-Negative Matrix Factorization to solve Text Classification and Recommendation Problems

In this blog, we shall discuss on how matrix-factorization-based unsupervised machine learning techniques can be applied to solve problems such as text classification (BBC news docs) and movie recommendation (MovieLens ratings predictions). The problems appeared in the coursera course Unsupervised Learning by the university of Colorado Boulder.More specifically, we shall focus on the NMF (Non-negative … Continue reading Non-Negative Matrix Factorization to solve Text Classification and Recommendation Problems

December 06, 2023 10:39 AM UTC

James Bennett

Understanding virtual environments in Python

This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.

Linking up

I want to talk today about Python virtual environments (or “venvs”), but first I need to cover a bit of background. Suppose you write a program, and it needs access …

Read full entry

December 06, 2023 01:35 AM UTC

Seth Michael Larson

Review of the Security Developer-in-Residence role in 2023

Review of the Security Developer-in-Residence role in 2023


Review of the Security Developer-in-Residence role in 2023

Published 2023-12-06 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the OpenSSF Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!

This report is short on new work due to me being "mostly away" since Thanksgiving. This week I've been working on November and end-of-year reports for this roles' sponsor the Alpha-Omega project and the PSF.

November 2023 update

I've published the Alpha-Omega update for November 2023 which highlights the Python Software Foundation's response to the US Government RFI on open source security, the publication of the guide for becoming a CVE Numbering Authority as an open source project to the OpenSSF blog, and the proposal for adding Software Bill of Materials to the CPython release process.

2023 end-of-year report

Creating these end-of-year reports had me looking back through what I've done, and I'm proud of what I've been able to accomplish since June. The list of highlights includes:

  • Joined the Python Security Response Team as a coordinator for disclosures, remediations, and releases.
  • Authorized the Python Software Foundation as a CVE Numbering Authority covering Python and pip.
  • Authored and published the guide to becoming a CVE Numbering Authority as an open source organization.
  • Created a database for vulnerabilities in the CPython runtime using the Open Source Vulnerability (OSV) format. Populated this database with historical and new vulnerabilities. This database has been integrated into the global OSV database for use by software vulnerability scanners.
  • Documented the CPython release process and identified sources of supply chain risk in the current process.
  • Audited current Sigstore signatures for CPython releases and worked with core developers to fix discrepancies. Updated tools so that signatures would always be consistent.
  • Back-filled the new Sigstore bundle format to old releases so tooling can make use of offline verification and only one format of verification materials.
  • Build reproducibility for CPython's source artifacts.
  • Proposed implementation for moving CPython's source and documentation releases to GitHub Actions as a hardened and SLSA-compliant build platform.
  • Proposed implementation for Software Bill-of-Materials generation and publication as a part of the CPython release process.
  • Helped author the Python Software Foundations first response to a government Request For Information about investing in open source security.
  • Spoke at OpenSSF Day Europe 2023 with Cheuk Ting-Ho, appeared on Talk Python, received news coverage from The Register and The New Stack.
  • Authored 20 weekly report posts on my blog, 6 monthly reports, and 2 end-of-year reports.

None of the above would have been possible without the trust and support of the Python community. The year isn't quite over yet, but I'm already excited for what we can all accomplish next year together. Thanks a million, everyone! 💜

That's all for this week! 👋 If you're interested in more you can read last week's report.

Don't let social media algorithms decide what you want to see.

Get notified of new publications by subscribing to the RSS feed or the email newsletter:

This work is licensed under CC BY-SA 4.0

December 06, 2023 12:00 AM UTC

December 05, 2023

PyCoder’s Weekly

Issue #606 (Dec. 5, 2023)

#606 – DECEMBER 5, 2023
View in Browser »

The PyCoder’s Weekly Logo

Advent of Code: Solving Puzzles With Python

Advent of Code is an online advent calendar that shares new programming puzzles each day from December 1 to the 25. In this Code Conversation, you’ll learn why solving programming puzzles can be beneficial and how you can get started with Advent of Code using Python.

Two Kinds of Threads Pools, and Why You Need Both

This article talks about thread pools and how tuning them correctly can make a difference in the performance of your concurrent code.

The Morning Paper for Hacker News Readers


Want a byte-sized version of Hacker News? Try TLDR’s free daily newsletter. TLDR covers the most interesting tech, startup, and coding stories in just 5 minutes. No sports, politics, or weather. Subscribe for free →
TLDR sponsor

Talking to Notebooks With Jupyter AI

Talk Python interviews David Qiu and they discuss LLM integration with Jupyter notebooks.
KENNEDY & QIU podcast

Welcoming the Supporting Developer in Residence


Django 5.0 Released



Ideas: Support Multiple Divisors in divmod()


Long-Term Vision for a Parallel Programming Model?


The Realest Python 😂


Python Jobs

Senior Python Architect and Tech Lead (Anywhere)

Six Feet Up

Software Engineer - Intern (Summer 2024) (Dallas, TX, USA)

Causeway Capital Management

More Python Jobs >>>

Articles & Tutorials

The Python Rich Package: Unleash the Power of Console Text

Rich is a powerful library for constructing text-based user interfaces (TUIs) using Python. With it, you can make your code more readable by pretty-printing complex data structures, and you can make your app more attractive for your user with colored and formatted text, tables, animations, and more.

Running Python Parallel Applications With Sub Interpreters

Python 3.13 is adding programmatic control over sub-interpreters in Python. They spawn faster than creating a new process, but slower than threads. Learn why and how you can use them in the next release of Python.

Achieve Certified Python Excellence with Pybites! 🥇 🐍


Master Python 🐍 through our 12-week coaching program earning a Python developer certificate at the end. From foundational Python knowledge to sophisticated CI/CD pipelines, deploying your apps using IaC, all through personalized 1:1 coaching, real-world projects, and a supportive community … 🚀🔑 →
PYBITES sponsor

Understanding Linux cp and Implementing It in Python

Dive into the workings of the Linux cp command and learn how to replicate it in Python. This post breaks down the command’s process and shows you how the same could be accomplished using only Python.

Say It Again: Values Not Expressions

Sometimes you can explain a simple thing for the thousandth time, and come away with a deeper understanding yourself. It happened to Ned the other day with Python mutable argument default values.

Divmod for Unit Conversions

Python’s built-in divmod() provides the quotient and remainder of two numbers. This kind of division occurs frequently when calculating unit conversions. This article shows some examples.

Examples of Great URL Design

URLs are everywhere, but some sites spend more time thinking about what they mean and how visitors read them. This blog post covers some of Jim’s favorite examples of good URL designs.

Adding Full Text Search to Django With django-watson

Learn how to supercharge your Django app with full-text search using Django-Watson. Dive deep into Postgres magic and boost search functionality.

Untyped Python: The Python That Was

This post by Armin is an opinion piece about the changes to Python over the years, particularly typing, and how that relates to other languages.

CPython Internals: Understanding the Role of PyObject

Understand how objects are implemented in CPython and how CPython emulates Inheritance and Polymorphism in C using struct embedding.

Why You Should Not Overuse List Comprehensions in Python

List comprehensions in Python are super helpful one-liners. But if overused, they can make your code a pain to maintain. Here’s why.
BALA PRIYA C • Shared by Bala Priya C

Projects & Code

Polars-business: Polars Business Date Arithmetic

GITHUB.COM/MARCOGORELLI • Shared by Marco Gorelli

movis: Video Editing as a Code


filequery: Query CSV, JSON & Parquet With SQL


tracemem: Memory Tracker for Python Session

GITHUB.COM/NYGGUS • Shared by Marcin

django-web-components: Reusable Template Components



PyData Global 2023

December 6 to December 9, 2023

Weekly Real Python Office Hours Q&A (Virtual)

December 6, 2023

Canberra Python Meetup

December 7, 2023

Sydney Python User Group (SyPy)

December 7, 2023

PyCon Thailand 2023

December 15 to December 17, 2023

FlaskCon 2023

December 16 to December 18, 2023

Happy Pythoning!
This was PyCoder’s Weekly Issue #606.
View in Browser »


[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

December 05, 2023 07:30 PM UTC

Real Python

How to Get the Current Time in Python

Getting the current time in Python is a nice starting point for many time-related operations. One very important use case is creating timestamps. In this tutorial, you’ll learn how to get, display, and format the current time with the datetime module.

To effectively use the current time in your Python applications, you’ll add a few tools to your belt. For instance, you’ll learn how to read attributes of the current time, like the year, minutes, or seconds. To make the time more easily readable, you’ll explore options for printing it. You’ll also get to know different formats of time and understand how to deal with time zones.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

December 05, 2023 02:00 PM UTC

Mike Driscoll

Viewing an Animated GIF with Python

Animated GIFs are a fun way to share silent videos on social media. This website has a tutorial to help you learn how to create your own animated GIFs with Python. But what if you wanted to view an animated GIF with Python?

If you’re wondering if Python can present an animated GIF to the viewer, then you’ve found the right tutorial!

You will learn how to display a GIF with Python using the following methods:

Getting Started

The first step is to find an animated GIF you’d like to display. You can get GIFs from Google or Giphy. There are also many applications that you can use to make your own GIFs.

For the purposes of this tutorial, you can use this silly example GIF that shows a “hello world” example from Asciimatics, a Python TUI package:

Asciimatics hello world GIF

Save this file as asicmatics_hello.gif to your computer so you can use it in the examples in this tutorial.

Now that you have an animated GIF, you are ready to learn how to display it using Python!

Viewing the Animated GIF with tkinter

StackOverflow has many suggestions you can use to view animated GIFs with Python. One suggestion was to use tkinter, the Python GUI toolkit included with Python, to view an animated GIF.

The following code is a cleaned up variation on the suggestion from that post:

from tkinter import *
from PIL import Image, ImageTk

class MyLabel(Label):
    def __init__(self, master, filename):
        im =
        seq =  []
            while 1:
       # skip to next frame
        except EOFError:
            pass # we're done

            self.delay =['duration']
        except KeyError:
            self.delay = 100

        first = seq[0].convert('RGBA')
        self.frames = [ImageTk.PhotoImage(first)]

        Label.__init__(self, master, image=self.frames[0])

        temp = seq[0]
        for image in seq[1:]:
            frame = temp.convert('RGBA')

        self.idx = 0

        self.cancel = self.after(self.delay,

    def play(self):
        self.idx += 1
        if self.idx == len(self.frames):
            self.idx = 0
        self.cancel = self.after(self.delay,

class Main():
    def __init__(self):
        root = Tk()
        self.anim = MyLabel(root, 'asciimatics_hello.gif')
        Button(root, text='stop', command=self.stop_it).pack()

    def stop_it(self):

main = Main()

This code will import tkinter and create a Label() widget. The label can contain all kinds of different things, including images. For this example, you will create a list of frames from the GIF and store them in self.frames as instances of ImageTk.PhotoImage.

Then you will call tkinter’s handy after() method to make it play the frames back after a short delay. You use this command to call play() recursively.

When you run this code, you should see the following:

You’ll note that in Tkinter, it seems to default to playing the GIF in black-and-white. You can do some more searching yourself and see if you can find a way to make it display the GIF in color.

Now, let’s try viewing it with PySimpleGUI!

Viewing the Animated GIF with PySimpleGUI

Tkinter isn’t the only GUI toolkit in town. You can also use PySimpleGUI, a wrapper around Tkinter, wxPython, and PyQt. If you’d like to know more, you can check out this brief intro PySimpleGUI.

You will need to install PySimpleGUI since it doesn’t come with Python. You can use pip to do that:

python -m pip install pysimplegui

Now you’re ready to write some code. The following code is based on an example from the PySimpleGUI GitHub page:

import PySimpleGUI as sg
from PIL import Image, ImageTk, ImageSequence

gif_filename = 'asciimatics_hello.gif'

layout = [[sg.Image(key='-IMAGE-')]]

window = sg.Window('Window Title', layout, element_justification='c', margins=(0,0), element_padding=(0,0), finalize=True)

interframe_duration =['duration']

while True:
    for frame in ImageSequence.Iterator(
        event, values =
        if event == sg.WIN_CLOSED:
        window['-IMAGE-'].update(data=ImageTk.PhotoImage(frame) )

This code is a little shorter than the Tkinter example, so that’s pretty neat. Give it a try and see what happens!

Now you’re ready to try out viewing a GIF in Jupyter Notebook!

Viewing the Animated GIF in Jupyter Notebook

Jupyter Notebooks are great ways to share code along with some markup. You can create presentations, code examples, and their output and even widgets using Jupyter.

You can also insert images into Jupyter Notebook cells. That includes animated GIFs too! If you’d like to learn more about Jupyter Notebook, you should check out this introductory tutorial.

You’ll need to install Jupyter to be able to follow along with this part of the tutorial. Fortunately, that’s only a pip-install away:

python -m pip install jupyter

Now that you have Jupyter Notebook installed, open up your terminal or Powershell and run the following command:

jupyter notebook

The next step is to create a new Notebook. Click the New button on the top right in the new web page that should have loaded in your default browser. Look for a URL like this: http://localhost:8888/tree

You will be presented with several choices when you click the New button. You should pick the top choice, which is Notebook. A new browser tab will open, and you may see a pop-up window asking which Kernel to choose. The default will probably be Python 3, which is what you want. Pick that, and you should be good to go!

In the middle of the toolbar along the top of the Jupyter Notebook Window, you will see a drop-down labeled Code. Click on that and change it to Markdown. Now the current cell is n Markdown mode instead of Code mode. That’s great!

Enter the following into your cell:

![MyGif](asciimatics_hello.gif "segment")

Make sure that asciimatics_hello.gif is in the same folder as your new Jupyter Notebook is in. Or you can put the absolute path to the file in there instead.

Now, you need to run the cell to see the animated GIF. You can click the play button in the toolbar or press SHIFT+ENTER. Either way that should run the cell.

At this point, you should see the following:

You did it! That looks great!

Wrapping Up

Python has many different ways to view animated GIFs. You can use the built-in Tkinter GUI package, the PySimpleGUI package, or Jupyter Notebook.

There are probably others you can use as well. Feel free to drop a comment and let us all know what version you like best or a different package that you’ve tried.

The post Viewing an Animated GIF with Python appeared first on Mouse Vs Python.

December 05, 2023 01:35 PM UTC

Python Bytes

#363 DNS Again? It's Always DNS.

<strong>Topics covered in this episode:</strong><br> <ul> <li><a href=""><strong>Fixit 2: Meta’s next-generation auto-fixing linter</strong></a></li> <li><a href=""><strong>FastUI</strong></a></li> <li><a href="">Mail list / newsletter conversation</a></li> <li><strong>CLIs from type hints</strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='' style='font-weight: bold;'>Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href=""><strong>courses at Talk Python Training</strong></a></li> <li><a href=""><strong>The Complete pytest Course</strong></a></li> <li><a href=""><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href=""><strong></strong></a></li> <li>Brian: <a href=""><strong></strong></a></li> <li>Show: <a href=""><strong></strong></a></li> </ul> <p>Join us on YouTube at <a href=""><strong></strong></a> to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too.</p> <p><strong>Michael #1:</strong> <a href=""><strong>Fixit 2: Meta’s next-generation auto-fixing linter</strong></a></p> <ul> <li>via Bart Kappenburg</li> <li>Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter.</li> <li>Fixit provides a highly configurable linting framework with support for auto-fixes, custom “local” lint rules, and hierarchical configuration, built on <a href="">LibCST</a>.</li> <li>Fixit 2 is available today on PyPI.</li> <li>Created by Meta’s Python Language Foundation team — a hybrid team of both PEs and traditional SWEs — helps own and maintain the infrastructure and tooling for Python.</li> <li>Interesting comments on <a href="">this article on Hacker News</a></li> <li>I wonder if ruff format was already a thing when Fixit was adopted, whether it would exist?</li> </ul> <p><strong>Brian #2:</strong> <a href=""><strong>FastUI</strong></a></p> <ul> <li>Samuel Colvin</li> <li>“FastUI is a new way to build web application user interfaces defined by declarative Python code.”</li> <li>MK: Reminds me of the code matches DOM style of Flutter. <a href="">See code samples at the end</a>.</li> </ul> <p><strong>Michael #3:</strong> <a href="">Mail list / newsletter conversation</a></p> <ul> <li>I’ve been tired of Mailchimp for a long time</li> <li>Raising the prices month over month by $100 several months may be the straw</li> <li>But what are the options? <a href="">Lets ask Mastodon</a>: <ul> <li><a href=""></a></li> <li><a href=""></a> [self hosted, open source]</li> <li><a href=""></a> [self/saas, open source]</li> <li><a href=""></a> [self hosted, open source]</li> <li><a href=""></a> [self hosted, open source]</li> <li><a href=""></a></li> <li><a href=""></a> [django]</li> <li><a href=""></a></li> <li><a href=""></a> [use your own bulk emailer (e.g. sendgrid or aws ses)</li> <li><a href=""></a> </li> <li><a href=""></a> [open source]</li> <li><a href=""></a> </li> <li><a href=""></a></li> <li><a href=""></a></li> </ul></li> </ul> <p><strong>Brian #4:</strong> <strong>CLIs from type hints</strong></p> <ul> <li>From Sander76</li> <li><a href=""><strong>Pydantic Argparse</strong></a> “is a Python package built on top of pydantic which provides declarative typed argument parsing using pydantic models.” <ul> <li><a href=""><strong>Clipstick</strong></a> is a “cli-tool based on Pydantic models.”</li> <li><a href=""><strong>tyro</strong></a> “is a tool for generating command-line interfaces and configuration objects in Python.” <ul> <li>tyro includes support for dataclasses and attrs in place of Pydantic</li> </ul></li> </ul></li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href=""><strong>Django 5.0 has been released</strong></a></li> <li><a href=""><strong>vim-keybindings-everywhere-the-ultimate-list</strong></a> - submitted by Paul Barry</li> <li><a href=""><strong>PythonTest</strong></a> (the podcast formerly known as <a href="">Test &amp; Code</a>, to be read in an undertone similar to the way one used to say “The artist formerly known as Prince”) has moved form to <a href=""></a> <ul> <li>Plus more <a href="">guests</a> are listed now. I think I’ve gone backwards from current to episode 182. I tried to get my kid to help out, unsuccessfully. May have to hire someone to help. grrr.</li> </ul></li> </ul> <p>Michael:</p> <ul> <li>Essay: <a href="">Don't Sweat the Ad Blocker Drama</a></li> <li>A story: my project this weekend, unify my over 20 domains to one host</li> </ul> <p><strong>Joke:</strong> <a href="">Honest LinkedIn</a></p>

December 05, 2023 08:00 AM UTC

Read the Docs

Read the Docs newsletter - December 2023

News and updates

You can always see the latest changes to our platforms in our Read the Docs Changelog.

Upcoming changes

Want to follow along with our development progress? View our full roadmap 📍️

Possible issues

Questions? Comments? Ideas for the next newsletter? Contact us!

December 05, 2023 12:00 AM UTC

December 04, 2023

James Bennett

Easy HTTP status codes in Python

This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar for Advent 2023, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post in the series for an introduction.

The most useful test

I could be misremembering, but I think Frank Wiles was the first person I ever heard explain that, for a web application, …

Read full entry

December 04, 2023 08:56 PM UTC

Python Software Foundation

It's time for our annual year-end PSF fundraiser and membership drive 🎉

Support Python in 2023!



There are two ways to join in the drive this year:

Or, heck, why not do both? 🥳

 Your Donations:

  • Invest directly in CPython and PyPI progress
  • Bring the global Python community together
  • Make our community more diverse and robust every year

Let’s take a look back on 2023:

PyCon US - We held our 20th PyCon US, in Salt Lake City and online, which was an exhilarating success! For the online component, PyCon US OX, we added two moderated online hallway tracks (in Spanish and English) and saw a 33% increase in virtual engagement. It was great to see everyone again in 2023, and we’re grateful to all the speakers, volunteers, attendees, and sponsors who made it such a special event.

Security Developer in Residence - Seth Larson joined the PSF earlier this year as our first ever Security Developer-in-Residence. Seth is already well-known to the Python community – he was named a PSF Fellow in 2022 and has already written a lot about Python and security on his blog. This critical role would not be possible without funding from the OpenSSF Alpha-Omega Project.

PyPI Safety & Security Engineer - Mike Fiedler joined the PSF earlier this year as our first ever PyPI Safety & Security Engineer. Mike is already a dedicated member of the Python packaging community – he has been a Python user for some 15 years, maintains and contributes to open source, and became a PyPI Maintainer in 2022. You can see some of what he's achieved for PyPI already on the PyPI blog. This critical role would not be possible without funding from AWS.

Welcome, Marisa and Marie!
- In 2023 we were able to add two new full time staff members to the PSF. Marisa Comacho joined as Community Events Manager and Marie Nordin joined as Community Communications Manager. We are excited to add two full time dedicated staff members to the PSF to support PyCon US, our communications, and the community as a whole.  

CPython Developer in Residence
- Our CPython Developer in Residence, Łukasz Langa, continued to provide trusted support and advancement of the Python language, including oversight for the releases of Python 3.8 and 3.9, adoption of Sigstore, and stewardship of PEP 703 (to name a few of many!). Łukasz also engaged with the community by orchestrating the Python Language Summit and participating in events such as PyCon US 2023, EuroPython, and PyCon Colombia. This critical role would not be possible without funding from Meta.

Authorized as CVE Numbering Authority (CNA) - Being authorized as a CNA is one milestone in the Python Software Foundation's strategy to improve the vulnerability response processes of critical projects in the Python ecosystem. The Python Software Foundation CNA scope covers Python and pip, two projects which are fundamental to the rest of Python ecosystem.

Five new Fiscal Sponsorees
- Welcome to Bandit, BaPya, Twisted, PyOhio, and North Bay Python as new Fiscal Sponsorees of the PSF! The PSF provides 501(c)(3) tax-exempt status to fiscal sponsorees and provides back office support so they can focus on their missions.


Our Thanks:

Thank you for being a part of this drive and of the Python community! Keep an eye on this space and on our social media in the coming weeks for updates on the drive and the PSF 👀

Your support means the world to us. We’re incredibly grateful to be in community with you!

December 04, 2023 04:50 PM UTC

Zato Blog

Smart IoT integrations with Akenza and Python

Smart IoT integrations with Akenza and Python


The Akenza IoT platform, on its own, excels in collecting and managing data from a myriad of IoT devices. However, it is integrations with other systems, such as enterprise resource planning (ERP), customer relationship management (CRM) platforms, workflow management or environmental monitoring tools that enable a complete view of the entire organizational landscape.

Complementing Akenza's capabilities, and enabling the smooth integrations, is the versatility of Python programming. Given how flexible Python is, the language is a natural choice when looking for a bridge between Akenza and the unique requirements of an organization looking to connect its intelligent infrastructure.

This article is about combining the two, Akenza and Python. At the end of it, you will have:

Since WebSocket connections are persistent, their usage enhances the responsiveness of IoT applications which in turn helps to exchange occurs in real-time, thus fostering a dynamic and agile integrated ecosystem.

Python and Akenza WebSocket connections

First, let's have a look at full Python code - to be discussed later.

 # -*- coding: utf-8 -*-

# Zato
from zato.server.service import WSXAdapter

# ###############################################################################################
# ###############################################################################################

if 0:
    from zato.server.generic.api.outconn.wsx.common import OnClosed, \
        OnConnected, OnMessageReceived

# ###############################################################################################
# ###############################################################################################

class DemoAkenza(WSXAdapter):

    # Our name
    name = 'demo.akenza'

    def on_connected(self, ctx:'OnConnected') -> 'None':'Akenza OnConnected -> %s', ctx)

# ###############################################################################################

    def on_message_received(self, ctx:'OnMessageReceived') -> 'None':

        # Confirm what we received'Akenza OnMessageReceived -> %s',

        # This is an indication that we are connected ..
        if['type'] == 'connected':

            # .. for testing purposes, use a fixed asset ID ..
            asset_id:'str' = 'abc123'

            # .. build our subscription message ..
            data = {'type': 'subscribe', 'subscriptions': [{'assetId': asset_id, 'topic': '*'}]}


            # .. if we are here, it means that we received a message other than type "connected".
  'Akenza message (other than "connected") -> %s',

# ##############################################################################################

    def on_closed(self, ctx:'OnClosed') -> 'None':'Akenza OnClosed -> %s', ctx)

# ##############################################################################################
# ##############################################################################################

Now, deploy the code to Zato and create a new outgoing WebSocket connection. Replace the API key with your own and make sure to set the data format to JSON.

Receiving messages from WebSockets

The WebSocket Python services that you author have three methods of interest, each reacting to specific events:

Let's focus on on_message_received, which is where the majority of action takes place. It receives a single parameter of type OnMessageReceived which describes the context of the received message. That is, it is in the "ctx" that you will both the current request as well as a handle to the WebSocket connection through which you can reply to the message.

The two important attributes of the context object are:

Now, the logic from lines 30-40 is clear:

A sample message from Akenza will look like this:

INFO - WebSocketClient -  Akenza message (other than "connected") -> {'type': 'subscribed',
'replyTo': None, 'timeStamp': '2023-11-20T13:32:50.028Z',
'subscriptions': [{'assetId': 'abc123', 'topic': '*', 'tagId': None, 'valid': True}],
'message': None}

How to send messages to WebSockets

An aspect not to be overlooked is communication in the other direction, that is, sending of messages to WebSockets. For instance, you may have services invoked through REST APIs, or perhaps from a scheduler, and their job will be to transform such calls into configuration commands for IoT devices.

Here is the core part of such a service, reusing the same Akenza WebSocket connection:

 # -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# ##############################################################################################
# ##############################################################################################

class DemoAkenzaSend(Service):

    # Our name
    name = 'demo.akenza.send'

    def handle(self) -> 'None':

        # The connection to use
        conn_name = 'Akenza'

        # Get a connection ..
        with self.out.wsx[conn_name].conn.client() as client:

            # .. and send data through it.

# ##############################################################################################
# ##############################################################################################

Note that responses to the messages sent to Akenza will be received using your first service's on_message_received method - WebSockets-based messaging is inherently asynchronous and the channels are independent.

Now, we have a complete picture of real-time, IoT connectivity with Akenza and WebSockets. We are able to establish persistent, responsive connections to assets, we can subscribe to and send messages to devices, and that lets us build intelligent automation and integration architectures that make use of powerful, emerging technologies.

December 04, 2023 04:00 PM UTC

Daniel Roy Greenfeld

TIL: Forcing pip to use virtualenv

Necessary because installing things into your base python causes false positives, true negatives, and other head bangers.

Set this environment variable, preferably in your rc file:

# ~/.zshrc 

Now if I try to use pip outside a virtualenv:

dj-notebook on  main [$] is 📦 v0.6.1 via 🐍 v3.10.6 
❯ pip install ruff 
ERROR: Could not find an activated virtualenv (required).

This TIL is thanks to David Winterbottom.

December 04, 2023 03:30 PM UTC

Real Python

Serialize Your Data With Python

Whether you’re a data scientist crunching big data in a distributed cluster, a back-end engineer building scalable microservices, or a front-end developer consuming web APIs, you should understand data serialization. In this comprehensive guide, you’ll move beyond XML and JSON to explore several data formats that you can use to serialize data in Python. You’ll explore them based on their use cases, learning about their distinct categories.

By the end of this tutorial, you’ll have a deep understanding of the many data interchange formats available. You’ll master the ability to persist and transfer stateful objects, effectively making them immortal and transportable through time and space. Finally, you’ll learn to send executable code over the network, unlocking the potential of remote computation and distributed processing.

In this tutorial, you’ll learn how to:

  • Choose a suitable data serialization format
  • Take snapshots of stateful Python objects
  • Send executable code over the wire for distributed processing
  • Adopt popular data formats for HTTP message payloads
  • Serialize hierarchical, tabular, and other shapes of data
  • Employ schemas for validating and evolving the structure of data

To get the most out of this tutorial, you should have a good understanding of object-oriented programming principles, including classes and data classes, as well as type hinting in Python. Additionally, familiarity with the HTTP protocol and Python web frameworks would be a plus. This knowledge will make it easier for you to follow along with the tutorial.

You can download all the code samples accompanying this tutorial by clicking the link below:

Get Your Code: Click here to download the free sample code that shows you how to serialize your data with Python.

Feel free to skip ahead and focus on the part that interests you the most, or buckle up and get ready to catapult your data management skills to a whole new level!

Get an Overview of Data Serialization

Serialization, also known as marshaling, is the process of translating a piece of data into an interim representation that’s suitable for transmission through a network or persistent storage on a medium like an optical disk. Because the serialized form isn’t useful on its own, you’ll eventually want to restore the original data. The inverse operation, which can occur on a remote machine, is called deserialization or unmarshaling.

Note: Although the terms serialization and marshaling are often used interchangeably, they can have slightly different meanings for different people. In some circles, serialization is only concerned with the translation part, while marshaling is also about moving data from one place to another.

The precise meaning of each term depends on whom you ask. For example, Java programmers tend to use the word marshaling in the context of remote method invocation (RMI). In Python, marshaling refers almost exclusively to the format used for storing the compiled bytecode instructions.

Check out the comparison of serialization and marshaling on Wikipedia for more details.

The name serialization implies that your data, which may be structured as a dense graph of objects in the computer’s memory, becomes a linear sequence—or a series—of bytes. Such a linear representation is perfect to transmit or store. Raw bytes are universally understood by various programming languages, operating systems, and hardware architectures, making it possible to exchange data between otherwise incompatible systems.

When you visit an online store using your web browser, chances are it runs a piece of JavaScript code in the background to communicate with a back-end system. That back end might be implemented in Flask, Django, or FastAPI, which are Python web frameworks. Because JavaScript and Python are two different languages with distinct syntax and data types, they must share information using an interchange format that both sides can understand.

In other words, parties on opposite ends of a digital conversation may deserialize the same piece of information into wildly different internal representations due to their technical constraints and specifications. However, it would still be the same information from a semantic point of view.

Tools like Node.js make it possible to run JavaScript on the back end, including isomorphic JavaScript that can run on both the client and the server in an unmodified form. This eliminates language discrepancies altogether but doesn’t address more subtle nuances, such as big-endian vs little-endian differences in hardware.

Other than that, transporting data from one machine to another still requires converting it into a network-friendly format. Specifically, the format should allow the sender to partition and put the data into network packets, which the receiving machine can later correctly reassemble. Network protocols are fairly low-level, so they deal with streams of bytes rather than high-level data types.

Depending on your use case, you’ll want to pick a data serialization format that offers the best trade-off between its pros and cons. In the next section, you’ll learn about various categories of data formats used in serialization. If you already have prior knowledge about these formats and would like to explore their respective scenarios, then feel free to skip the basic introduction coming up next.

Compare Data Serialization Formats

There are many ways to classify data serialization formats. Some of these categories aren’t mutually exclusive, making certain formats fall under a few of them simultaneously. In this section, you’ll find an overview of the different categories, their trade-offs, and use cases, as well as examples of popular data serialization formats.

Later, you’ll get your hands on some practical applications of these data serialization formats under different programming scenarios. To follow along, download the sample code mentioned in the introduction and install the required dependencies from the included requirements.txt file into an active virtual environment by issuing the following command:

(venv) $ python -m pip install -r requirements.txt

This will install several third-party libraries, frameworks, and tools that will allow you to navigate through the remaining part of this tutorial smoothly.

Textual vs Binary

At the end of the day, all serialized data becomes a stream of bytes regardless of its original shape or form. But some byte values—or their specific arrangement—may correspond to Unicode code points with a meaningful and human-readable representation. Data serialization formats whose syntax consists purely of characters visible to the naked eye are called textual data formats, as opposed to binary data formats meant for machines to read.

The main benefit of a textual data format is that people like you can read serialized messages, make sense of them, and even edit them by hand when needed. In many cases, these data formats are self-explanatory, with descriptive element or attribute names. For example, take a look at this excerpt from the Real Python web feed with information about the latest tutorials and courses published:

Read the full article at »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

December 04, 2023 02:00 PM UTC

Django Weblog

Django 5.0 released

The Django team is happy to announce the release of Django 5.0.

The release notes cover a deluge of exciting new features in detail, but a few highlights are:

You can get Django 5.0 from our downloads page or from the Python Package Index. The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E.

With the release of Django 5.0, Django 4.2 has reached the end of mainstream support. The final minor bug fix release, 4.2.8, was issued today. Django 4.2 is an LTS release and will receive security and data loss fixes until April 2026. All users are encouraged to upgrade before then to continue receiving fixes for security issues.

Django 4.1 has reached the end of extended support. The final security release (4.1.13) was issued on November 1st. All Django 4.1 users are encouraged to upgrade to Django 4.2 or later.

See the downloads page for a table of supported versions and the future release schedule.

December 04, 2023 11:17 AM UTC


Django 5.0 Delight: Unraveling the Newest Features

Hello everyone! As 2023 draws to a close, our reasons for celebration extend well beyond the upcoming holidays and vacation. Exciting developments await in the technology realm, including the unveiling of Python 3.12 and the much-anticipated Django 5.0. This latest release of our Python-based web framework (still the most popular option on the market) signifies […]

December 04, 2023 10:20 AM UTC

Django Weblog

Django bugfix release: 4.2.8

Today we've issued the 4.2.8 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Mariusz Felisiak: 2EF56372BA48CD1B.

December 04, 2023 08:33 AM UTC

James Bennett

A Python/Django Advent calendar

Advent is the liturgical season preceding Christmas in many Christian traditions, and generally begins on a Sunday — often the fourth Sunday before Christmas Day, but it varies depending on the church and the rite, which can put the first Sunday in Advent in either late November or early December.

The concept of an Advent “calendar” which counts down to Christmas, and in which each day has a door or panel which opens to reveal a …

Read full entry

December 04, 2023 01:03 AM UTC

December 03, 2023

TechBeamers Python

44 Python Data Analyst Interview Questions

Here are 44 Python data analytics interview questions focused on Python programming along with their answers. You may like to check these out. Python Data Analyst Interview Questions and Answers. Question: How do you read data from a CSV file in Python? Answer: To read data from a CSV file, you can use the pandas [...]

The post 44 Python Data Analyst Interview Questions appeared first on TechBeamers.

December 03, 2023 08:39 PM UTC

December 01, 2023

Marcos Dione

ikiwiki to nikola: the script

People asked for it:

#! /usr/bin/python3

import argparse
from datetime import datetime
from glob import glob
from os import stat
from os.path import basename, splitext
import re
import sys
import time

footnote_re = re.compile(r'\[(?P<foot_number>\d+)\]')
taglink_re = re.compile(r'\[\[!taglink (?P<tag_name>[^\]]*)\]\]')
image_re = re.compile(r'\[\[!img (?P<path>.*)\]\]')
format_start_re = re.compile(r'^\[\[!format (?P<language>.*) """$')
format_end_re = re.compile(r'^"""\]\]$')

def rewrite_footnotes_line(line, text_block, footnote_block, taglink_block, foot_number):
    new_line = line
    changed = False

    while footnote :=
        # remove the []s
        start = footnote.start('foot_number') - 1
        end = footnote.end('foot_number') + 1

        prefix = new_line[:start]
        postfix = new_line[end:]
        foot_number ='foot_number')

        if text_block:
            new_line = f"{prefix}[^{foot_number}]{postfix}"
        elif footnote_block:
            new_line = f"{prefix}[^{foot_number}]:{postfix}"
            raise ValueError('found a footnote in the taglink_block!')

        changed = True
        if not changed and footnote_block and len(line) > 0:
            # '[^]: ' <-- 5 extra chars
            new_line = f"{' ' * (len(foot_number) + 5)}{line.strip()}"

    return new_line, foot_number

def rewrite_footnotes(src):
    lines = src.splitlines()
    hr_count = len([ line for line in lines if line.startswith('---') ])

    new_lines = []
    text_block = True
    footnote_block = False
    taglink_block = False
    hr_seen = 0
    foot_number = ''

    for line in lines:
        line_length = len(line)

        if line_length > 4 and line[:4] == '    ':
            # it's an inline code block, leave alone

        if line.startswith('---'):
            hr_seen += 1

            # if there is only one hr, then we have text + taglink blocks
            # if there are two or more, it's text + footnote + taglink blocks
            if text_block and hr_count >= 2 and hr_seen == hr_count - 1:
                text_block = False
                footnote_block = True
                # don't keep it
            elif hr_seen == hr_count:
                text_block = False
                footnote_block = False
                taglink_block = True
                # we'll need it later

            new_line, foot_number = rewrite_footnotes_line(line, text_block, footnote_block, taglink_block, foot_number)
        except Exception as e:
            print(f"got `{e}´ for `{line}´.")


    return '\n'.join(new_lines) + '\n'

def rewrite_taglinks(src):
    new_lines = []
    new_tags = []

    for line in src.splitlines():
        if len(line) > 0 and line == '-' * len(line):
            # don't keep it

        tags = taglink_re.findall(line)
        if len(tags) > 0:

    return '\n'.join(new_lines) + '\n', new_tags

def rewrite_images(src):
    new_lines = []

    for line in src.splitlines():
        image =
        if image is not None:
            # get the text before and after the whole directive
            start = image.start(0)
            end = image.end(0)

            prefix = line[:start]
            postfix = line[end:]
            path ='path')

            # the root to which this 'absolute' path points is the website's root
            new_line = f"{prefix}![](/{path}){postfix}"


    return '\n'.join(new_lines) + '\n'

lang_map = dict(

def rewrite_format(src):
    new_lines = []

    for line in src.splitlines():
        start = format_start_re.match(line)
        if start is not None:
            lang ='language')
            # if there's no mapping return the same lang
            new_line = f"```{lang_map.get(lang, lang)}"

        if format_end_re.match(line):


    return '\n'.join(new_lines) + '\n'

def titlify(src):
    words = src.split('-')
    words[0] = words[0].title()

    return ' '.join(words)

def test_offesetify():
    src = -3600
    dst = '+0100'

    assert offsetify(src) == dst

def offsetify(src):
    hours, seconds = divmod(src, 3600)
    # "offsets are always in minutes" sounds like one item in 'things dveloper believe about timezones'
    minutes, _ = divmod(seconds, 60)

    # NOTE: time.timezone returns seconds west of UTC, which is opposite of what usual offsets go
    if src > 0:
        sign = '-'
        sign = '+'

    return f"{sign}{-hours:02d}{minutes:02d}"

def datify(src):
    '''1701288755.377908 -> 2023-11-29 21:12:35 +0100'''

    # BUG: I'm gonna assume current timezone.
    # thanks
    # dto=DT(2023,11,29, 12,13,59, tzinfo=UTC_TZ);  DT.astimezone( dto , getTZ('Europe/Brussels') ) #==> 2023-11-29 13:13:59+01:00
    offset = time.timezone
    dt = datetime.fromtimestamp(src)

    return f"{dt.strftime('%Y-%m-%d %H:%M:%S')} {offsetify(offset)}"

# zoneinfo for some reason doesn't know about CEST, so I'll just hack a mapping here
tzname_to_utc_offset = dict(

month_name_to_number = dict(
    jan= 1,
    ene= 1,
    feb= 2,
    mar= 3,
    apr= 4,
    abr= 4,
    may= 5,
    jun= 6,
    jul= 7,
    aug= 8,
    ago= 8,
    sep= 9,

def dedatify(src):
    #       0         1      2     3      4       5           6     7
    # src=['Posted', 'Sun', '26', 'Aug', '2012', '11:27:16', 'PM', 'CEST']
    month = month_name_to_number[src[3].lower()]
    utc_offset = tzname_to_utc_offset[src[7]]
    h, m, s = [ int(x) for x in src[5].split(':') ]

    if src[6].upper() == 'PM':
        h += 12  # TODO: support 12PM

    return f"{src[4]}-{month:02d}-{int(src[2]):02d} {h:02d}:{m:02d}:{s:02d} {utc_offset}"

def build_meta(filepath, tags, date=None):
    filename = splitext(basename(filepath))[0]
    if date is None:
        mtime = stat(filepath).st_mtime
        date_string = datify(mtime)
        date_string = dedatify(date)

    meta = f""".. title: {titlify(filename)}
.. slug: {filename}
.. date: {date_string}
.. tags: {', '.join(tags)}
.. type: text

    return filename, meta

def import_post(opts):
    src = open(opts.filepath).read()
    mid, tags = rewrite_taglinks(rewrite_footnotes(src))
    dst = rewrite_format(rewrite_images(mid))
    if is None:
        filename, meta = build_meta(opts.filepath, tags)
        filename, meta = build_meta(opts.filepath, tags,

    open(f"posts/{filename}.md", 'w+').write(dst)
    open(f"posts/{filename}.meta", 'w+').write(meta)

def parse_args():
    parser = argparse.ArgumentParser()

    parser.add_argument('filepath', metavar='FILE')
    parser.add_argument('-d', '--date', nargs=8,
                        help='Just pass something like "Posted Wed 12 Sep 2012 08:19:23 PM CEST".')

    return parser.parse_args()

if __name__ == '__main__':
    opts = parse_args()


I removed all the tests, but they all looked like this:

def test_dedatify():
    src = 'Posted Wed 12 Sep 2012 08:19:23 PM CEST'.split()
    dst = '2012-09-12 20:19:23 +0200'
    assert dedatify(src) == dst


December 01, 2023 11:31 PM UTC

Real Python

The Real Python Podcast – Episode #182: Building a Python JSON Parser & Discussing Ideas for PEPs

Have you thought of a way to improve the Python language? How do you share your idea with core developers and start a discussion in the Python community? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

December 01, 2023 12:00 PM UTC

Tryton News

Newsletter December 2023

In the last month we focused on fixing bugs, improving the behaviour of things, speeding-up performance issues and adding new features for you.

Changes for the User

Accounting, Invoicing and Payments

We relaxed the former unique constraint on IBAN account numbers to now allow multiple equal deactivated IBAN account numbers.

When changing the company or party of an invoice, the tax identifiers are now cleared when they are no longer valid.

On payment terms we now display the fields to define the payment term delta in the correct order of application.

It is now possible to shorten or extend the fiscal year as long as all periods are still in its date range.

On refunds we now show the external payment ID from the payment provider.

We now order payable/receivable lines by maturity date or move date.

Parties and CRM

The height of the street widget is now reduced to three lines.

New Releases

We released bug fixes for the currently maintained long term supported series
7.0, 6.0 and the penultimate series 6.8.

Changes for the System Administrator

Tryton now fully supports updating database records from CSV data. The missing piece had been handling the removal of links in xxx2Many fields during the update, which is now done. To unlink or remove existing xxx2Many target records, just exclude them from the CSV data to import. This way the imported data is similar to the stored records in the database.

The Tryton client now cleans-up all temporary files and directories on exit.

Changes for Implementers and Developers

Now it is possible to specify a database statement timeout on RPC calls. The new timeout parameter on RPC calls helps to avoid costly database queries. The default value is 60 sec and can be modified in the configuration.

We included a new policy to require documentation updates for modules when contributing a new feature to an existing module. So far we’ve been working with this rule for a month, which has already improved the documentation for some of the modules.

A new contrib group have been added to the heptapod repository. This includes some tools related to Tryton which provide web integration, filestore integration and even a module to send SMS. We are happy to include more similar projects in the group, feel free to contribute yours!

Authors: @dave @pokoli @udono

1 post - 1 participant

Read full topic

December 01, 2023 08:00 AM UTC

Armin Ronacher

Untyped Python: The Python That Was

A lot has been said about Python typing. If you have been following me on Twitter (or you have the dubious pleasure of working with me), you probably know my skepticism towards Python typing. This stems from the syntax's complexity, the sluggishness of mypy, the overall cumbersome nature of its implementation and awkwardness of interactions with it. I won't dwell on these details today, instead I want to take you on a little journey back to my early experiences with Python. Why? Because I believe the conflict between the intrinsic philosophy of Python and the concept of typing is fundamental and profound, but also not new.

The concept of typed programming languages predates 2015 by a long stretch. They were not invented now. Debates over the necessity of typing are not a recent phenomenon at all. When you wanted to start a new software project, particularly something that resembles a web service you always had a choice of programming language. Back in 2004 when I started diving into programming, there were plenty of languages to chose. The conventional choice was not Python, the obvious choice was not even PHP rather Java. Java was the go-to for serious web application projects, given its typing system and enterprise-grade features. PHP was for toys, Python was nowhere to be found. PHP was popular, but in my circles it was always seen as an entirely ridiculous concept and the idea that someone would build a business on it even more so. I remember in my first year of University the prevalent opinion was that the real world runs on .NET, Java and C++. PHP was ridiculed, Python and Ruby did not appear in conversations and JavaScript on the server was non existent.

Yet here I was, I built stuff in PHP and Python. My choice wasn't driven by an aversion to static typing out of laziness but by the exceptional developer experience these languages offered, to a large part because of the lack of types. There was a stellar developer experience. Yes it did not have intellisense, but all the changes that I did appear on the web instantly. I recall directly modifying live websites via FTP in real time. Later editing web sites straight from vim on the production server. Was it terrible and terrifying? Absolutely. But damn it was productive. I learned a lot from that. They taught me valuable lessons about trade-offs. It was not just me that learned that, an entire generation of developers in those languages learned that our biggest weakness (it not being typed, and i wasn't compiled) was also our biggest strength. It required a bit of restraint and it required a slightly different way of programming, but it was incredibly productive.

There was the world of XPath, there was the world of DTDs, there was the world of SOAP and WSDL. There was the world where the inherent complexity of the system was so great, that you absolutely required an IDE, code generation and compile time tooling. In contrast there was my world. My world had me sitting with Vim, CVS and SVN and a basic Linux box and I was able to build things that I was incredibly proud of. I eventually swapped PHP for Python because it had better trade offs for me. But I will never not recognize what PHP gave me: I learned from it that not everything has to be pretty, it has to solve problems. And it did.

But in the same way with PHP, the total surface area between me and the Python language runtime was tiny. The code I wrote, was transformed by the interpreter into bytecode instructions (which you could even look at!) and evaluated by a tiny loop in the interpreter. The interpreter was Open Source, it was easy to read, and most importantly I was able to poke around in it. Not only was I able to learn more about computers this way, it also made it incredibly easy for me to understand what exactly was going on. Without doubt I was able to understand everything between the code that I wrote, and the code that ran end to end.

Yes, there was no static type checking and intellisense was basically non existing. Companies like Microsoft did not even think that Python was a language yet. But screw it, we were productive! Not only that, we build large software projects. We knew were the tradeoffs were. We had runtime errors flying left and right in production because bad types were passed, but we also had the tools to work with it! I distinctly remember how blown away a colleague from the .NET world was when I showed him some of the tools I had. That after I deployed bad code and it blew up in someone's face, I got an email that not only shows a perfectly readable stack trace, but also a line of source code for the frames. He was even more blown away when I showed him that I had a module that allowed me to attach remotely to the running interpreter and execute Python code on the fly to debug it. The developer experience was built around there being very few layers in the onion.

But hear me out: all the arguments against dynamic languages and dynamic typing systems were already there! Nothing new has been invented, nothing really has changed. We all knew that there was value in typing, and we also all collectively said: screw it. We don't need this, we do duck typing. Let's play this to our advantage.

Here is what has changed: we no longer trust developers as much and we are re-introducing the complexity that we were fighting. Modern Python can at times be impossible to comprehend for a developer. In a way in some areas we are creating the new Java. We became the people we originally displaced. Just that when we are not careful we are on a path to the world's worst Java. We put typing on a language that does not support it, our interpreter is slow, it has a GIL. We need to be careful not to forget that our roots are somewhere else. We should not collectively throw away the benefits we had.

The winds changed, that's undeniable. Other languages have shown that types add value in new and exciting ways. When I had the arguments with folks about Python vs Java typing originally, Java did not even have generics. JavaScript was fighting against its reputation of being an insufferable toy. TypeScript was years away from being created. While nothing new has been invented, some things were popularized. Abstract data types are no longer a toy for researchers. .NET started mixing static and dynamic typing, TypeScript later popularized adding types to languages originally created without them. There are also many more developers in our community who are less likely to understand what made those languages appealing in the first place.

So, where does this leave us? Is this a grumpy me complaining about times gone and how types are ruining everything? Hardly. There's undeniable utility in typing, and there is an element that could lead to greater overall productivity. Yet, the inherent trade-offs remain unchanged, and opting for or against typing should be a choice free from stigma. The core principles of this decision have not altered: types add value and they add cost.

Post script: Python is in a spot now where the time spent for me typing it, does not pay dividends. TypeScript on the other hand tilts more towards productivity for me. Python could very well reach that point. I will revisit this.

December 01, 2023 12:00 AM UTC