skip to navigation
skip to content

Planet Python

Last update: August 01, 2021 01:40 AM UTC

July 31, 2021


Łukasz Langa

Weekly Report 2021, July 26 - August 1

This week I tried to drop the open pull request count below 1,400. This was a grueling task and I barely succeeded.

July 31, 2021 11:38 AM UTC


Zero to Mastery

Python Monthly 💻🐍 July 2021

20th issue of Python Monthly! Read by 20,000+ Python developers every month. This monthly Python newsletter is focused on keeping you up to date with the industry and keeping your skills sharp, without wasting your valuable time.

July 31, 2021 10:00 AM UTC

July 30, 2021


Mike Driscoll

Pre-Order Automating Excel with Python

My 10th Python book is called Automating Excel with Python: Processing Spreadsheets with OpenPyXL. It only has 11 more days to go on Kickstarter where you can get an exclusive t-shirt!

I also have the eBook available for pre-order on Gumroad. No matter whether you purchase a copy on Kickstarter or Gumroad, you will be getting an early version of the book plus all the updates that I ever make to the book.

You can get some sample chapters here so you can try out the book before you commit to purchasing a copy!

Automating Excel with Python

In this book, you will learn how to use Python to do the following:

The post Pre-Order Automating Excel with Python appeared first on Mouse Vs Python.

July 30, 2021 07:00 PM UTC


Stack Abuse

How to Join/Concatenate/Append Strings in Python

Introduction

In this short tutorial, we'll take a look at how to concatenate strings in Python, through a few different approaches.

It's worth noting that strings in Python are immutable - a string object in memory cannot be changed once created:

newString = "Some new string"

If you'd like to change this string in any way - under the hood, a new string with those changes is created. The same applies to concatenating strings - a new object must be created in memory.

String Concatenation and String Appending

Concatenation of strings refers to joining two or more strings together, as if links in a chain. You can concatenate in any order, such as concatenating str1 between str2 and str3.

Appending strings refers to appending one or more strings to the end of another string.

In some cases, these terms are absolutely interchangeable. Appending strings is the same operation as concatenating strings at the end of another. Let's begin with the simplest way of concatenating/appending two (or more) strings.

Concatenate or Append Strings with The + Operator

In Python, a string is a list of characters, which means that the + operator can be used to add their constituent elements together in a new list:

title = "Prof. "
name = "Richard Feynman"

result = title + name
print(result)

This results in:

Prof. Richard Feynman

This operator doesn't limit the amount of strings which can be added together, so you can easily join a large number of strings:

string1 = "Concatenating"
string2 = "strings"
string3 = "in Python"
string4 = "is easy!"

print(string1 + string2 + string3 + string4)

Though, if your aim is to construct a sentence from a list of strings such as this, concatenating them manually, and without special characters is both inefficient and produces an unintelligible output:

Concatenatingstringsin Pythonis easy!

It'd be much more sensible to iterate through a list of strings, and add them together with a whitespace between each concatenated string:

strings = ["Concatenating", "strings", "in Python", "is easy!"]

result = ""
for string in strings:
    result += string + " "

print(result)

This results in:

Concatenating strings in Python is easy! 

A shorthand operator you can use to concatenate two strings is +=, just like in the previous example. This saves you from the trouble of having to create a new variable to store the results, as you can reuse one of the existing reference variables to assign to the new object in memory:

string1 = "one"
string2 = "two"

string1 += string2
print(string1) # Output: onetwo

Limitation of the + Operator

The main limitation of the + operator is the fact that you can't mix-and-match types. You can't add an integer to a list of characters, for instance:

print("some string" + 2)

Many languages, such as JavaScript and Java pick up on this, and would automatically convert the integer into a string (match the types) and perform the concatenation, though, Python would throw a TypeError:

TypeError: can only concatenate str (not "int") to str

However, there is an easy workaround - you can use Python's built-in str() function which converts compatibile data types into a string. Let's add some integers to our strings list and box all of the items with a str() in case there are non-string elements:

strings = ["Concatenating", "strings", "in Python", "is easy!", 5, 2]

result = ""
for string in strings:
    result += str(string) + " "

print(result)

This results in:

Concatenating strings in Python is easy! 5 2 

Concatenate or Append Strings with The * Operator

If you want to create a new string by replicating a string n amount of times and appending it, you can achieve this with the * operator:

string = "w"

print(string * 3) # Output: www

This can be even more useful when combined with the + operator:

print("w"*3 + "." + "stackabuse.com")

Which results in:

www.stackabuse.com

Concatenate or Append Strings with The % Operator

Again, concatenation doesn't necessarily mean we're adding a string to the end. Using the % operator, we can perform string interpolation. By adding % in a string as a marker, we can replace the markers with concrete strings later on:

string = "This %s is a %s string" % ("here", "random")

print(string)

This should output:

This here is a random string

Similarly enough, you can use other markers for other data types:

string = "This is a string%d" % (1)
print(string) # Output: This is a string
string = "This string starts with a %c" % ('T')
print(string) # Output: This string starts with a T
string = "There is a %f percent chance that you'll learn string concatenation in Python after reading this article" % (99.99)
print(string) # Output filled 99,990000 in %f place

Note: If you wish to explicitly mark how many digits should the number be rounded to (say 2), you can achieve it with: %.2f.

If you'd like to read more about formatting strings in Python and the different ways to do it, read our Guide to Formatting Strings in Python 3's f-Strings.

Concatenating Strings With the join() Method

The join() method takes an iterable as an argument and returns a string created by joining the elements of that iterable. It's worth noting that these have to be strings - each element is not inherently converted using str(), unlike our own method from before.

Additionally, a separator is used to define the separator between the joined strings, and it's the base string we call join() on:

my_list = ["1", "2", "3", "4"] # List - iterable
string_from_list = "-".join(my_list) # The separator is "-"

print(string_from_list)

This should output:

1-2-3-4

In a lot of cases, the separator is just a whitespace, so you'll commonly be seeing:

" ".join(iterable)

Implementing a Custom join() Method

Since the built-in join() method might behave a bit differently than you might've expected, let's implement our own join() method with an adjustable separator.

We'll want it to be able to handle 2D lists as well, so if a list contains another list within itself - it's flattened to a 1-dimensional list, before joined:

import itertools

def join(iterable, separator):
    # Empty string to hold result
    result = ""
    # Flatten 2D potentially 2D list into 1D
    iterable = list(itertools.chain(*iterable))
    # Concatenate strings with separator
    for string in iterable:
        result += str(string) + separator
    return result
    

string = join(['a', 'b', 'c'], ' ')
string2 = join(['a', ['b', 'c'], 'd'], ' ')

print(string)
print(string2)

This results in:

a b c 
a b c d 

Concatenating Strings Using Space

A simple way to concatenate strings, usually only when you're printing them is to leverage the space bar. This approach is commonly used only for printing, since assigning it to an object in memory is easy, but awkward:

print("Concat" " strings" " using Space")

If you'd like to avoid using whitespaces in the strings, you can add commas (,) between each element:

print("Concat", "strings", "using Space")

Both of these result in:

Concat strings using Space

If you'd like to assign them to a variable, you're free to do so, and they'll automatically be concatenated into a single string:

string = "this " "is " "a " "big " "string"
print(type(string))
print(string)

This results in:

<class 'str'>
this is a big string

You can even do multi-line strings. To achieve this, we add a \ to the end of each line to let Python know there's more than one line:

multi_line_string = "this " "string " \
			   	  "is coded in " "three lines " \
    			  "but printed in one"
			   	  
print(multi_line_string)

This results in:

this string is coded in three lines but printed in one

Though, this approach is fiddly and awkward, and others are preferred.

Note: The same effect cannot be achieved with variables, only string literals:

string1 = "one string"
string2 = "two string"

final_string = string1 string2

This results in:

File "<string>", line 4
    final_string = string1 string2
                           ^
SyntaxError: invalid syntax

Conclusion

Joining/Appending/Concatenating Strings in Python is fairly simple and like everything related to Python, there are many ways to do it. It just comes down to your needs and preferences.

In this short tutorial, we've taken a look at some of the ways to concatenate strings

July 30, 2021 12:30 PM UTC


Real Python

The Real Python Podcast – Episode #71: Start Using a Debugger With Your Python Code

Are you still sprinkling print statements throughout your code while writing it? Print statements are often clunky and offer only a limited view of the state of your code. Have you thought there must be a better way? This week on the show, we have Nina Zakharenko to discuss her conference talk titled "Goodbye Print, Hello Debugger."


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

July 30, 2021 12:00 PM UTC


PyBites

Are you overwhelmed by tutorial paralysis?

Tutorial Paralysis, information overload, hoarding and never completing online courses (How large are your Udemy and Coursera libraries?)

This is the pain we’re increasingly hearing about in conversations with developers and it’s more common than you may realise.

It’s real, it’s crippling and it can make you feel like you’re going nowhere.

Even if you do manage to complete a course, is it what you needed? What next? Another course? Then what?

The reality is that nothing beats actual coding and implementationThis is where the real learning and growth happens.

It may be easier said than done, but the best time to do it is yesterday.

Tutorial Paralysis can be just as prohibiting as these signs!Photo by Sangga Rima Roman Selia on Unsplash

That’s why we recommend that anybody stuck in tutorial purgatory do the following:

1. Don’t be hard on yourself. It happens. What matters is what you do next. Also tutorials in and by themselves are not bad, they give you a good understanding of fundamental concepts, what matters is how you apply what you read.

2. Pause and reflect on what your actual goal is. Without this clear vision on what you want it’s easy to waste your time consuming content that really gets you nowhere. (Is that Blockchain tutorial really going to get you where you need to be?)

3. Make a plan on precisely what you need to do to reach that goal. Our free 6 Key Ingredients of a Successful Pythonista training (particularly day 5) can help you with this.

4. Follow the plan. Consistently. Show up every day, even when you don’t feel like it. Once you’re done, we want to see your goal and your plan. 


We’d love to hear what plan you came up with so comment below or shoot us a message. If you want help book a free assessment call using the link below.

And lastly it can be hard to go at it alone, we have a vibrant Python community a.k.a. The PyBites Slack that, at the time of this writing, has 2,700 members! You can join here and start building valuable relationships. See you there…

July 30, 2021 11:22 AM UTC


Talk Python to Me

#327: Little Automation Tools in Python

You've heard me talk to wide cast of people building amazing things with Python. Some of them are building bio-reactors to remove carbon from the air with AI and Python. Others are optimizing aerodynamics and race strategy at the highest levels of automobile racing. <br/> <br/> This episode is different. Rather than seeing how far we can push Python to the edges of technology, we are diving in to the tiny Python applications that might never be released publicly and yet can transform our day to day lives with simple automation on an individual level. <br/> <br/> We have 4 great guests with us here today: Rivers Cuomo, Jay Miller, Kim van Wyk, and Rusti Gregory. They will each share a couple of apps and the underlying packages they used to build them. I know this will be a super motivational episode for many of you. I hope that after listening, you'll transform something tedious and error-prone in your live to an instantaneous button click solution with Python.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Panelists</b><br/> <b>Rivers Cuomo</b>: <a href="https://twitter.com/RiversCuomo" target="_blank" rel="noopener">@RiversCuomo</a><br/> <b>Jay Miller</b>: <a href="https://twitter.com/kjaymiller" target="_blank" rel="noopener">@kjaymiller</a><br/> <b>Kim van Wyk</b>: <a href="https://twitter.com/kim_vanwyk" target="_blank" rel="noopener">@kim_vanwyk</a><br/> <b>Rusti Gregory</b>: <a href="https://talkpython.fm/episodes/show/194/learning-and-teaching-python-in-a-vacuum" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>DiversityOrgs.Tech</b>: <a href="https://diversityorgs.tech/?q=portland&size=n_20_n" target="_blank" rel="noopener">diversityorgs.tech</a><br/> <b>MP3TAG App</b>: <a href="https://www.mp3tag.de/en/index.html" target="_blank" rel="noopener">mp3tag.de</a><br/> <b>Screenshot from Rivers' demo clip selection app</b>: <a href="https://www.youtube.com/watch?v=DJVzQuaahSg" target="_blank" rel="noopener">Timestamp @ youtube.com</a><br/> <b>pywinauto</b>: <a href="https://pywinauto.readthedocs.io/en/latest/" target="_blank" rel="noopener">pywinauto.readthedocs.io</a><br/> <b>pyperclip</b>: <a href="https://pypi.org/project/pyperclip/" target="_blank" rel="noopener">pypi.org</a><br/> <b>ffmpeg</b>: <a href="https://ffmpeg.org/ffprobe.html#json" target="_blank" rel="noopener">ffmpeg.org</a><br/> <b>selenium</b>: <a href="https://selenium-python.readthedocs.io/" target="_blank" rel="noopener">selenium-python.readthedocs.io</a><br/> <b>Github actions</b>: <a href="https://github.com/marketplace?type=actions" target="_blank" rel="noopener">github.com/marketplace</a><br/> <b>RUMPS</b>: <a href="https://pypi.org/project/rumps/" target="_blank" rel="noopener">pypi.org</a><br/> <b>py2app</b>: <a href="https://pypi.org/project/py2app/" target="_blank" rel="noopener">pypi.org</a><br/> <b>PyMuPDF</b>: <a href="https://pypi.org/project/PyMuPDF/" target="_blank" rel="noopener">pypi.org</a><br/> <b>Gooey</b>: <a href="https://pypi.org/project/Gooey/" target="_blank" rel="noopener">pypi.org</a><br/> <b>conduit podcast</b>: <a href="https://www.relay.fm/conduit" target="_blank" rel="noopener">relay.fm</a><br/> <b>feedparser</b>: <a href="https://feedparser.readthedocs.io/en/latest/introduction.html" target="_blank" rel="noopener">feedparser.readthedocs.io</a><br/> <b>awesome-python audio recommendations</b>: <a href="https://awesome-python.com/#audio" target="_blank" rel="noopener">awesome-python.com</a><br/> <b>foxdot live coding</b>: <a href="https://toplap.org/foxdot-live-coding-with-python-and-supercollider/" target="_blank" rel="noopener">toplap.org</a><br/> <b>spotipy</b>: <a href="https://pypi.org/project/spotipy/" target="_blank" rel="noopener">pypi.org</a><br/> <b>pipx</b>: <a href="https://github.com/pypa/pipx" target="_blank" rel="noopener">github.com</a><br/> <b>eyed3</b>: <a href="https://eyed3.readthedocs.io/en/latest/" target="_blank" rel="noopener">eyed3.readthedocs.io</a><br/> <b>Youtube Live Stream</b>: <a href="https://www.youtube.com/watch?v=DJVzQuaahSg" target="_blank" rel="noopener">youtube.com</a><br/> <br/> <b>Episodes referenced at the beginning:</b><br/> <b>Bio-reactor</b>: <a href="https://talkpython.fm/episodes/show/250/capture-over-400x-c02-as-trees-with-ai-and-python" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>Nascar</b>: <a href="https://talkpython.fm/episodes/show/281/python-in-car-racing" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>F1</b>: <a href="https://talkpython.fm/episodes/show/296/python-in-f1-racing" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>Episode transcripts</b>: <a href="/episodes/transcript/327/little-automation-tools-in-python" target="_blank" rel="noopener">talkpython.fm</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://talkpython.fm/linode'>Linode</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a><br> <a href='https://talkpython.fm/assemblyai'>AssemblyAI</a>

July 30, 2021 08:00 AM UTC


Python Bytes

#244 vendorizing your Python podcast

<p><strong>Watch the live stream:</strong></p> <a href='https://www.youtube.com/watch?v=5Tm5tr3prPs' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>About the show</strong></p> <p>Sponsored by <strong>us:</strong></p> <ul> <li>Check out the <a href="https://training.talkpython.fm/courses/all"><strong>courses over at Talk Python</strong></a></li> <li>And <a href="https://pythontest.com/pytest-book/"><strong>Brian’s book too</strong></a>!</li> </ul> <p>Special guest: <strong><a href="https://twitter.com/blbraner">Brandon Braner</a></strong></p> <p><strong>Brain #1:</strong> <a href="https://pip.pypa.io/en/stable/topics/configuration/#environment-variables"><strong>pip Environmental Variables</strong></a></p> <ul> <li>The problem with snakes on a plane → no internet</li> <li>Situation: <ul> <li>want to work on some code on a plane</li> <li>project set up to use tox, which creates venvs, and pip installs dependencies <strong>from pypi.org.</strong></li> <li>but… no internet</li> </ul></li> <li>Preflight work: <ul> <li>run tox with an internet connection</li> <li>copy everything from all of the “site-packages” directories under “.tox” into a “wheels” directory or whatever you want to call it.</li> <li>set environmental variables:</li> <li>PIP_FIND_LINKS=file:/Users/okken/wheels</li> <li>PIP_NO_INDEX=1</li> <li>Try this out first BEFORE getting on the plane, to make sure you didn’t miss anything.</li> </ul></li> <li>In flight: <ul> <li>tox works fine now, using local dir as index</li> </ul></li> <li>Thanks <a href="https://twitter.com/pganssle/status/1420807532076060677?s=20">Paul Ganssle</a> for helping with this.</li> <li>All command line flags for pip are available as env variables: <ul> <li>“pip’s command line options can be set with environment variables using the format <code>PIP_[HTML_REMOVED]</code> . Dashes (<code>-</code>) have to be replaced with underscores (<code>_</code>).”</li> </ul></li> </ul> <p><strong>Michael #2: Extra, Extra, 6x Extra, hear all about it</strong></p> <ul> <li><a href="https://www.youtube.com/watch?v=lon-dEXfY2I"><strong>Michael’s Pydantic talk</strong></a></li> <li><strong><a href="https://talkpython.fm/episodes/show/327/little-automation-tools-in-python">Little Automation Tools in Python episode</a></strong></li> <li><a href="https://amzn.to/3BJ05z0"><strong>A Day in Code- Python: Learn to Code in Python through an Illustrated Story Released</strong></a> by Shari Eskenas</li> <li><a href="https://www.tabnine.com/"><strong>TabNine AI complete</strong></a></li> <li><a href="https://twitter.com/RhetTurnbull/status/1420037464366882818"><strong>macOS photos follow up</strong></a></li> <li><a href="https://github.com/mikeckennedy/jinja_partials"><strong>jinja-partials</strong></a></li> <li><a href="https://github.com/mikeckennedy/chameleon_partials"><strong>chameleon-partials</strong></a></li> <li><a href="https://github.com/mikeckennedy/fastapi-chameleon"><strong>fastapi-chameleon</strong></a></li> </ul> <p><strong>Brandon</strong> <strong>#3:</strong> <a href="https://www.kaggle.com/thirty-days-of-ml"><strong>Kaggle 30 Days of Machine Learning</strong></a> </p> <ul> <li>What is <a href="https://www.kaggle.com/">Kaggle</a> <ol> <li>Find and publish data sets</li> <li>Explore and build models in a web-based data-science environment(Jupyter Notebooks)</li> <li>Work with other data scientists and machine learning engineers</li> <li><strong>Enter competitions</strong> to solve data science challenges.</li> </ol></li> <li>Starts August 2nd, 2021</li> <li>Has an introduction to Python and covers basic and intermediate machine learning concepts</li> <li>Get some completion certificates</li> <li>Kaggle competition at the end with teams of 3</li> </ul> <p><strong>Brian #4:</strong> <a href="https://docs.github.com/en/actions/guides/building-and-testing-python"><strong>Building and testing Python with GitHub Actions</strong></a></p> <ul> <li>GitHub Docs</li> <li>This is an incredible resource</li> <li>I suggest starting with <a href="https://docs.github.com/en/actions/guides/building-and-testing-python#running-tests-with-tox"><strong>Running tests with tox</strong></a>, as it’s a super simple setup.</li> <li>And, you can test all of the tox environments locally.</li> </ul> <pre><code> name: Python package on: [push, pull_request] jobs: build: runs-on: ubuntu-latest strategy: matrix: python: [3.7, 3.8, 3.9, 3.10-dev] steps: - uses: actions/checkout@v2 - name: Setup Python uses: actions/setup-python@v2 with: python-version: ${{ matrix.python }} - name: Install Tox and any other packages run: pip install tox - name: Run Tox run: tox -e py </code></pre> <p><strong>Michael #5:</strong> <a href="https://github.com/mwilliamson/python-vendorize"><strong>python-vendorize</strong></a></p> <ul> <li>via Patrick Park</li> <li>Vendoring a dependency means basically copying it into your package or your app rather than installing and importing it externally.</li> <li>Here’s a simple way to do that.</li> <li><code>python-vendorize</code> allows pure-Python dependencies to be vendorized</li> <li><code>vendorize.toml</code> might look something like:</li> </ul> <pre><code> target = "your_app/_vendor" packages = [ "six", ] </code></pre> <ul> <li>run <code>python-vendorize</code> in the same directory as <code>vendorize.toml</code></li> <li>In <code>your_app.py</code>, <code>six</code> can be imported from <code>_vendor</code>:</li> </ul> <pre><code> from ._vendor import six </code></pre> <p><strong>Brandon</strong> <strong>#6:</strong> <a href="https://supabase.io"><strong>Supabase</strong></a> <strong>and the new</strong> <a href="https://github.com/supabase/supabase-py"><strong>Python library</strong></a></p> <ul> <li>Supabase is an open source Firebase Alternative</li> <li>Postgres Database, Authentication, instant APIs, realtime subscriptions and Storage</li> <li>uses: <ul> <li><a href="https://postgrest.org/en/stable/">https://postgrest.org/en/stable/</a> for rest api access to your database</li> <li>GoTrue from Netlify for handling user authentication <a href="https://github.com/netlify/gotrue">https://github.com/netlify/gotrue</a></li> <li>Realtime client built in Elixir to listen to Postgres changes. <a href="https://github.com/supabase/realtime">https://github.com/supabase/realtime</a></li> <li>Supabase local so you don’t need internet to develop your project <a href="https://supabase.io/docs/guides/local-development">https://supabase.io/docs/guides/local-development</a></li> </ul></li> </ul> <p><strong>Extras</strong></p> <ul> <li><a href="https://simonwillison.net/2021/Jul/28/baked-data/"><strong>The Baked Data architectural pattern</strong></a> <ul> <li>Simon Willison</li> <li>“Baked Data: bundling a read-only copy of your data alongside the code for your application, as part of the same deployment”</li> </ul></li> <li><strong><a href="https://www.released.sh/">Released service - released.sh</a></strong></li> </ul> <p><strong>Joke</strong> </p> <p><a href="https://devhumor.com/media/monkeyuser-circle-of-life"><strong>The (tech) circle of life</strong></a></p>

July 30, 2021 08:00 AM UTC

July 29, 2021


Python for Beginners

Convert a List to String in Python

Python strings are one of the most commonly used data types. Whereas, Python lists are the most commonly used data structures. In this article, we will try to convert a list to a string using different functions in python. We will use string methods like join() and functions like map() and str() to convert a list into string.

List to string using String Concatenation

The easiest way to convert a list to a string in python is by using for loop and string concatenation. To convert a list to a string, we can simply convert each element of the list to a string. Then we can concatenate them to form a string.

To perform the conversion, first we will take an empty string. After that, we will start adding each element of the list to the empty string after converting them to string. In this process, we will also add spaces between each element. This can be done as follows.

myList = ["PFB", 11.2, 11, "Python"]
print("The List is:",myList)
myString = ""
for elem in myList:
    myString = myString + str(elem) + " "

print("Output String is:")
print(myString.rstrip())

Output:


The List is: ['PFB', 11.2, 11, 'Python']
Output String is:
PFB 11.2 11 Python

In the above example, we can see that an extra space has been added at the end of the string. We can use the rstrip() method to remove the extra space from the string. 

An alternative way to convert a list to string in python is to use the join() method. The join() method is used to create a string from a list of strings.

The join() method is invoked on a separator which is used to separate the elements of the list in the string. The list of strings is given as input to the join() method and It returns a string created by the elements of the list.

We will create an empty string to which all the elements of the list will be concatenated. To create the string, we will take each element of the list one by one and will convert it into string. Then, we will create a list of  strings using the string made by previous elements of the list and the current element. We will create the new string from the current string and the previously made string using the join() method by taking a space character as a separator and invoking the join() method on it. We will do this process on all the elements of the list using a for loop till the complete string is formed. This can be seen in the following example.

myList = ["PFB", 11.2, 11, "Python"]
print("The List is:", myList)
myString = ""
for elem in myList:
    myString = " ".join([myString, str(elem)])

print("Output String is:")
print(myString.lstrip())

Output:

The List is: ['PFB', 11.2, 11, 'Python']
Output String is:
PFB 11.2 11 Python

In the above method, an extra space is added at the left of the output string which has to be removed using the lstrip() method. To avoid this,Instead of applying str() function on every element of the list, we can use map() function  to convert each element of the list into a string so that we can perform the string concatenation using join() method to get the output string using a single statement.

The map() function takes as input a function and an iterable as argument and executes the function on each element of the iterable object and returns  the output map object which can be converted into a list.

To convert the elements of the list to strings, we will pass the str() function and the input list to the map() method. After that, we can create a string from a list of strings using the join() method as follows.

myList = ["PFB", 11.2, 11, "Python"]
print("The List is:", myList)
strList = list(map(str, myList))
myString = " ".join(strList)

print("Output String is:")
print(myString.lstrip())

Output:

The List is: ['PFB', 11.2, 11, 'Python']
Output String is:
PFB 11.2 11 Python

Using List Comprehension to convert a list to a string

We can use list comprehension to convert a list into a string. For this, We will convert each element of the list to string using list comprehension and str() function and then we can concatenate them using the join() method as follows.

myList = ["PFB", 11.2, 11, "Python"]
print("The List is:", myList)
strList = [str(i) for i in myList]
myString = " ".join(strList)

print("Output String is:", myString)

Output:

The List is: ['PFB', 11.2, 11, 'Python']
Output String is: PFB 11.2 11 Python

Alternatively, we can use the map() function to convert the elements of the list to string. Then we will create a new list from the map object created from the map() function using list comprehension and will give it as an input to the join() method to create a string from the list as follows.

myList = ["PFB", 11.2, 11, "Python"]
print("The List is:", myList)
myString = " ".join([i for i in map(str, myList)])
print("Output String is:", myString)

Output:

The List is: ['PFB', 11.2, 11, 'Python']
Output String is: PFB 11.2 11 Python

Conclusion

In this article, we have used the str(), map() and join() functions to convert a list to string. We have also seen how to use list comprehension for this task. To know how to convert a string into a list, read this article on string split operation in python.

The post Convert a List to String in Python appeared first on PythonForBeginners.com.

July 29, 2021 01:57 PM UTC


Twisted Matrix Labs

Twisted 21.7.0 Released

 On behalf of the Twisted contributors I announce the final release of Twisted 21.7.0

This is mostly a bugfix release.

Python 3.5 is no longer a supported platform.
The minimum supported platform is Python 3.6.7.

The notable features are:
  • Python 3.10 beta is now a supported platform and should be ready for the final 3.10 release.
  • twisted.web.template.renderElement() now accepts any IRequest implementer instead of only twisted.web.server.Request. Add type hints to twisted.web.template. (#10184)
  • Type hinting was added to twisted.internet.defer, making this the first release  of Twisted where you might reasonably be able to use mypy without your own custom stub files. (#10017)
The full release notes are available at

    https://github.com/twisted/twisted/releases/tag/twisted-21.7.0

Documentation is available at

    https://docs.twistedmatrix.com/en/stable/

Wheels for the release candidate are available on PyPI

    https://pypi.org/project/Twisted/21.7.0/

    python -m pip install Twisted==21.7.0

Many thanks to everyone who had a part in Twisted - the supporters of the Twisted Software Foundation, the developers, and all the people testing and building great things with Twisted!

Enjoy the release

-Adi Roiban

July 29, 2021 01:51 AM UTC

July 28, 2021


Lucas Cimon

Bonnes pratiques Gitlab CI

Logo Gitlab

À oui.sncf, je travaille au sein d'une équipe en charge de l'usine logicielle, qui administre depuis des années une instance Gitlab self-hosted.

Cet article contient quelques-unes de nos recommandations à l'intention des utilisateurs de notre Gitlab, ayant pour but à la fois améliorer les performances de leurs pipelines …

Permalink

July 28, 2021 08:10 PM UTC


PyCharm

Introducing PyCharm 2021.2!

This release comes with Python 3.10 support, auto-reload for browser HTML preview, and more!

Before we start the overview of the major PyCharm 2021.2 features, we have some important information to make you aware of.

Complete_current_statement

First of all, we have good news for our Asian users. Starting with this version, you can enjoy a fully localized UI in Chinese, Korean, or Japanese. Localization is available as a non-bundled language pack plugin, which can be easily installed in your IDE.

Secondly, please note that we are planning to end support for several packages. Namely for mako, buildout, web2py. This change will be introduced later this year, starting from PyCharm 2021.3.

DOWNLOAD PYCHARM 2021.2

Python 3.10

Structural Pattern Matching is coming in Python 3.10. PyCharm provides a number of key features to help you adjust to its arrival, like the Unused local symbols and Unreachable code inspections, smart code completion and syntax highlighting for the match and case keywords, and the Complete Current Statement action.

Complete_current_statement

In Python 3.10 you will be able to use int | str for union types instead of Union[int, str]. This functionality is already available in earlier versions of Python through the use of from __future__ import annotations.

Union_types

PyCharm provides overall code insight for the new syntax, including intention actions and information in the Quick Documentation. PyCharm now supports type inference for isinstance and issubclass arguments with the new syntax type.

Python console support in collaborative mode

Working in the Python console is now supported on both the client machine and the host side. During a collaborative coding session, guests can review the code while the host runs it in the interactive console. Support for the Python console in Code With Me comes complete with all the usual PyCharm features, such as on-the-fly syntax highlighting with inspections and code completion.

Python_console

DOWNLOAD PYCHARM 2021.2

Databases [Pro only]

Code completion for fields and operators in the MongoDB console

Now PyCharm can complete fields, nested fields, and fields inside aggregation expressions, as well as query operators, projection operators, update operators, and aggregation stages in the MongoDB console.
mongodb

Context live templates from the data editor with SQL scripts action

Context live templates now work directly from the data editor. If you’re working with a table and you wish to query it, you can easily do so with the help of the SQL scripts action!

live_templates_sql

DOWNLOAD PYCHARM 2021.2

Frontend development [Pro only]

Reload pages in browser on save

PyCharm lets you preview HTML files in a browser using the built-in web server. Now it is able to automatically update the pages in a browser as you edit and save your HTML, CSS, and JavaScript files. To get started, open an HTML file in the editor, hover over the corner of the editor, and click on the icon for the browser you want to use – all browsers are supported.

html_preview

Onboarding Tour in the Features Trainer plugin

You can now take the new Onboarding Tour in the IDE Features Trainer plugin.
With this 7-minute tour, you can get familiar with the PyCharm user interface and learn how to code in Python with smart assistance! You can start the tour from the Welcome screen or from the PyCharm main menu: Help | Learn IDE Features.

onboarding_tour

DOWNLOAD PYCHARM 2021.2

Testing

Test runner auto-detection

PyCharm auto-detects when there is a test runner installed on your Python interpreter and uses it to run tests. If no specific test runner is installed, PyCharm uses unittest. Still, you always have an option to explicitly specify the required test runner in the project settings. To explicitly set the required test runner in the project settings, press ⌘, to open the IDE settings and select Tools | Python Integrated Tools, and then select the target test runner from the Default test runner list.

Integrate with TestRail

The Test Management plugin now includes support for Python and is available for PyCharm Community. The Test Management plugin helps keep your unit tests in sync with test cases in TMS. You can load test cases from TestRail and link them with existing and new unit tests.
When you copy and paste a test case to a Python file a copy of the unit test will be pasted. You can customize the unit test template and share it with the team. With the plugin you can find test cases that are not yet automated (not linked with unit tests) and obsolete tests.

test-runner

Generate Test Data

Being able to insert a randomly generated email, name, or phone number is important when developing unit tests. As a part of our Test Automation Kit the new Test Data plugin plugin brings a lot of useful actions that help to generate random data. Use the Generate menu (Cmd+N) to see all available options. If you need a specific format, feel free to create your own custom data format based on regular expression or Velocity templates. All custom data types are available in bulk mode and can be shared with your team.

test-runner

DOWNLOAD PYCHARM 2021.2

In this blog post we covered only a few major features of PyCharm 2021.2. To learn about all the other new features in PyCharm 2021.2 visit our What’s New page.

July 28, 2021 04:37 PM UTC


Python for Beginners

Convert Integer to String in Python

Python strings are one of the most used data types while doing data analysis for operations like pattern matching. In this article, we will use different ways to convert an integer to string in python.

Convert integer to string using str() function

The easiest way to convert an integer to string is to use the str() function. The str() function takes the integer as input and returns its string representation as follows.

myInt = 1117
myStr = str(myInt)
print("The integer myInt is:", myInt)
print("The string myStr is:", myStr)

Output:

The integer myInt is: 1117
The string myStr is: 1117

We can check the type of input variable and output variable to confirm if the integer has been  converted to a string or not. To do this, we will use the type() function. The type function takes a python object as input and returns the data type of the input object as follows.

myInt = 1117
myStr = str(myInt)
print("The data type of myInt is:", type(myInt))
print("The data type of myStr is:", type(myStr))

Output:

The data type of myInt is: <class 'int'>
The data type of myStr is: <class 'str'>

Convert integer to string using string formatting

String formatting is a method to insert a variable or another string to a predefined string. We can also use string formatting to convert an integer to string. We will use the “%s” operator as well as the format() method to convert an integer to string in this article.

The “%s” operator is used to format a value inside a string. It is generally used to avoid string concatenation. But, we can use this operator to convert an integer to a string. For this, first we will create an empty string and put a %s placeholder in the empty string. After that, we can specify the integer which has to be converted into string. During execution of the program, the python interpreter will convert the integer into string as seen in the following example.

myInt = 1117
myStr = "%s" % myInt
print("myInt is:",myInt)
print("The data type of myInt is:", type(myInt))
print("myStr is:",myStr)
print("The data type of myStr is:", type(myStr))

Output:

myInt is: 1117
The data type of myInt is: <class 'int'>
myStr is: 1117
The data type of myStr is: <class 'str'>

In place of the “%s” operator, we can also use format() method to perform the conversion. For this we can put a {} placeholder in an empty string. After that, we can invoke the format method on the empty string with the integer given as the input to the format() method. This will convert the integer into string as follows.

myInt = 1117
myStr = "{}".format(myInt)
print("myInt is:",myInt)
print("The data type of myInt is:", type(myInt))
print("myStr is:",myStr)
print("The data type of myStr is:", type(myStr))

Output:

myInt is: 1117
The data type of myInt is: <class 'int'>
myStr is: 1117
The data type of myStr is: <class 'str'>

Conversion using F-strings

F strings are used to embed a value or an expression into a string. We can also use f strings to convert an integer into a string. 

The syntax for using f strings is similar to that of format() method. The only difference is that we can put the variables directly into the placeholders. This makes the code more readable. To put an integer variable n into a string, we simply put n into the {} placeholder as follows.

f"This is a string containing {n}"

To convert an integer to string using f strings, we will declare an empty string with only a single placeholder for the integer. In this way, at runtime, the integer will be converted to string. This can be seen in the following example.

myInt = 1117
myStr = f"{myInt}"
print("myInt is:",myInt)
print("The data type of myInt is:", type(myInt))
print("myStr is:",myStr)
print("The data type of myStr is:", type(myStr))

Output:

myInt is: 1117
The data type of myInt is: <class 'int'>
myStr is: 1117
The data type of myStr is: <class 'str'>

Conclusion

In this article, we have seen different ways to convert an integer into a string in python. We have used in-built str() method, string formatting as well as f-strings. To read more about strings, you can read this article on python string split operation.We can also write the programs used in this article with exception handling using python try except to make the programs more robust and handle errors in a systematic way.  

The post Convert Integer to String in Python appeared first on PythonForBeginners.com.

July 28, 2021 02:39 PM UTC


Real Python

Python and REST APIs: Interacting With Web Services

There’s an amazing amount of data available on the Web. Many web services, like YouTube and GitHub, make their data accessible to third-party applications through an application programming interface (API). One of the most popular ways to build APIs is the REST architecture style. Python provides some great tools not only to get data from REST APIs but also to build your own Python REST APIs.

In this tutorial, you’ll learn:

  • What REST architecture is
  • How REST APIs provide access to web data
  • How to consume data from REST APIs using the requests library
  • What steps to take to build a REST API
  • What some popular Python tools are for building REST APIs

By using Python and REST APIs, you can retrieve, parse, update, and manipulate the data provided by any web service you’re interested in.

Free Bonus: Click here to download a copy of the "REST API Examples" Guide and get a hands-on introduction to Python + REST API principles with actionable examples.

REST Architecture

REST stands for representational state transfer and is a software architecture style that defines a pattern for client and server communications over a network. REST provides a set of constraints for software architecture to promote performance, scalability, simplicity, and reliability in the system.

REST defines the following architectural constraints:

  • Stateless: The server won’t maintain any state between requests from the client.
  • Client-server: The client and server must be decoupled from each other, allowing each to develop independently.
  • Cacheable: The data retrieved from the server should be cacheable either by the client or by the server.
  • Uniform interface: The server will provide a uniform interface for accessing resources without defining their representation.
  • Layered system: The client may access the resources on the server indirectly through other layers such as a proxy or load balancer.
  • Code on demand (optional): The server may transfer code to the client that it can run, such as JavaScript for a single-page application.

Note, REST is not a specification but a set of guidelines on how to architect a network-connected software system.

REST APIs and Web Services

A REST web service is any web service that adheres to REST architecture constraints. These web services expose their data to the outside world through an API. REST APIs provide access to web service data through public web URLs.

For example, here’s one of the URLs for GitHub’s REST API:

https://api.github.com/users/<username>

This URL allows you to access information about a specific GitHub user. You access data from a REST API by sending an HTTP request to a specific URL and processing the response.

HTTP Methods

REST APIs listen for HTTP methods like GET, POST, and DELETE to know which operations to perform on the web service’s resources. A resource is any data available in the web service that can be accessed and manipulated with HTTP requests to the REST API. The HTTP method tells the API which action to perform on the resource.

While there are many HTTP methods, the five methods listed below are the most commonly used with REST APIs:

HTTP method Description
GET Retrieve an existing resource.
POST Create a new resource.
PUT Update an existing resource.
PATCH Partially update an existing resource.
DELETE Delete a resource.

A REST API client application can use these five HTTP methods to manage the state of resources in the web service.

Status Codes

Once a REST API receives and processes an HTTP request, it will return an HTTP response. Included in this response is an HTTP status code. This code provides information about the results of the request. An application sending requests to the API can check the status code and perform actions based on the result. These actions could include handling errors or displaying a success message to a user.

Below is a list of the most common status codes returned by REST APIs:

Code Meaning Description
200 OK The requested action was successful.
201 Created A new resource was created.
202 Accepted The request was received, but no modification has been made yet.
204 No Content The request was successful, but the response has no content.
400 Bad Request The request was malformed.
401 Unauthorized The client is not authorized to perform the requested action.
404 Not Found The requested resource was not found.
415 Unsupported Media Type The request data format is not supported by the server.
422 Unprocessable Entity The request data was properly formatted but contained invalid or missing data.
500 Internal Server Error The server threw an error when processing the request.

These ten status codes represent only a small subset of the available HTTP status codes. Status codes are numbered based on the category of the result:

Read the full article at https://realpython.com/api-integration-in-python/ »


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

July 28, 2021 02:01 PM UTC


Stack Abuse

Observer Design Pattern in Python

Introduction

Software Design Patterns help accelerate the development process by providing a reusable blueprint for your code to solve a particular problem. We follow Design Patterns to write generalized, reusable, and readable code that could be easily understood by others familiar with the patterns we've applied.

They encapsulate cumulative experience of software engineers solving the same problems, and represent solutions to common design-related issuees.

There are different classifications of design patterns depending on which class of problems they solve - among which the Observer Design Pattern belongs to the Behavioral Pattern class.

This class of patterns determines how objects communicate with each other. In this guide, you will learn everything you need to know about the Observer Design Pattern and understand how we can use it to solve certain problems efficiently.

Observer Design Pattern

The Observer Design Pattern deals with One-to-Many relationships and utilizes events to let subscribed entities know about changes in an observable.

The source of these events is called the subject or observable which sends events as streams. The observers or sinks can subscribe to the observable to obtain the events. The observable keeps track of the list of observers and notifies them of the changes when the state of the observable changes.

Architecture Diagram of Observer Design Pattern - shows a news feed as an observable and multiple readers as observers subscribed to the news feed

This functionality has many implications and implementations, and similar functionality is all around you. It's an extremely simple, yet very effective and wide-spread pattern.

A similar implementation of this design pattern is seen in generating feeds on your social platforms - the Pub/Sub (Publisher/Subscriber) Model/Pattern. When a content publisher publishes their posts, the subscribers get notified of the content. A similar analogy may be people looking out for a flare signal or a firework for a certain event, and reacting (or not) depending on their specific roles.

Does that mean that the Observer Design Pattern and Publish/Subscribe Pattern are the same?

Previously, both patterns were synonymous. Nowadays, each pattern has distinct traits that make them two separate patterns.

The following are the major differences between the Observer Pattern and the Pub/Sub Pattern:

One of the best ways to get a feel for this pattern is to implement it, let's implement it in Python!

Implementation

A basic implementation requires two classes - an Observable and an Observer. The Observer class is initialized with an object as an argument. The object is none other than an Observable to keep track of, to which it is subscribed upon creation.

The class also has a notify() function which triggers a reaction and acknowledges the receival of a notification/event from the observable:

class Observer:

    def __init__(self, observable):
        observable.subscribe(self)

    def notify(
        self,
        observable,
        *args,
        **kwargs
        ):
        print ('Got', args, kwargs, 'From', observable)

The Observable class is initialized with an empty list to hold the Observer instances. It also has functions such as subscribe() to add an observer, notify_observers() to call the notify() function on each observer, and unsubscribe() to remove the observer from the list:

class Observable:

    def __init__(self):
        self._observers = []

    def subscribe(self, observer):
        self._observers.append(observer)

    def notify_observers(self, *args, **kwargs):
        for obs in self._observers:
            obs.notify(self, *args, **kwargs)

    def unsubscribe(self, observer):
        self._observers.remove(observer)

Plugging in all of the above-mentioned components, let's write some code that sets up an observer and observable and sends messages, which triggers a reaction:

# observer_pattern.py

"""
Demonstrating the Observer pattern implementation
"""
# Initializing the subject
subject = Observable()

# Initializing twp observers with the subject object
observer1 = Observer(subject)
observer2 = Observer(subject)

# The following message will be notified to 2 observers
subject.notify_observers('This is the 1st broadcast',
                         kw='From the Observer')
subject.unsubscribe(observer2)

# The following message will be notified to just 1 observer since
# the observer has been unsubscribed
subject.notify_observers('This is the 2nd broadcast',
                         kw='From the Observer')

Notice that we also unsubscribe an observer before publishing the second message. This will lead to the message being printed only once instead of twice on the second attempt, as it is received by only one subscriber.

Running this code will result in:

$ python observer_pattern.py
Got ('This is the 1st broadcast',) {'kw': 'From the Observer'} From <__main__.Observable object at 0x7f6c50d2fb50>
Got ('This is the 1st broadcast',) {'kw': 'From the Observer'} From <__main__.Observable object at 0x7f6c50d2fb50>
Got ('This is the 2nd broadcast',) {'kw': 'From the Observer'} From <__main__.Observable object at 0x7f6c50d2fb50>

As you can see, the observable can directly interact with the observers and vice versa. The observable will be in interaction with the observer as long as the observer is subscribed to the observable's subscription list.

Pros & Cons

With the implementation in place, the pros and cons of this design pattern can be compared as follows:

Pros:

Cons:

To mitigate most of the cons, a message queue was introduced in between the observer and the observable to overcome all these problems, which led to devising the Pub/Sub pattern - a variation of the Observer Pattern.

Conclusion

This guide covered the the Observer Pattern, how it can be implemented, and compares its pros and cons.

It's interesting to note that the Observer Pattern is one of the Behavioral Patterns that has led to many of the features we use today, such as RSS feeds, social media feeds, etc.

By being introduced to the nuances of the Design Patterns, it is easier to build the functionality from the ground up. And of course, knowing different Design Patterns allows you to build the best solution for different types of problems.

July 28, 2021 12:30 PM UTC


John Ludhi/nbshare.io

How To Calculate Stocks Support And Resistance Using Clustering

How To Calculate Stocks Support And Resistance Using Clustering

In this notebook, I will show you how to calculate Stocks Support and Resistance using different clustering techniques.

Stock Data - I have stocks data in mongo DB. You can also get this data from Yahoo Finance for free.

MongoDB Python Setup

In [1]:
import pymongo
from pymongo import MongoClient
client_remote = MongoClient('mongodb://localhost:27017')
db_remote = client_remote['stocktdb']
collection_remote = db_remote.stock_data

Get Stock Data From MongoDB

I will do this analysis using last 60 days of Google data.

In [2]:
mobj = collection_remote.find({'ticker':'GOOGL'}).sort([('_id',pymongo.DESCENDING)]).limit(60)

Prepare the Data for Data Analysis

I will be using Pandas and Numpy for the data manipulation. Let us first get the data from Mongo Cursor object to Python list.

In [3]:
prices = []
for doc in mobj:
    prices.append(doc['high'])

Stocks Support and Resistance Using K-Means Clustering

In [4]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import AgglomerativeClustering

For K means clustering, we need to get the data in to Numpy array format.

In [5]:
X = np.array(prices)

For K means clustering, K which means number of clusters is very important. We can find the optimal K using the Knee plot as shown below.

In [6]:
from sklearn.cluster import KMeans
import numpy as np
from kneed import KneeLocator
    
sum_of_sq_distances = []
K = range(1,10)
for k in K:
    km = KMeans(n_clusters=k)
    km = km.fit(X.reshape(-1,1))
    sum_of_sq_distances.append(km.inertia_)
kn = KneeLocator(K, sum_of_sq_distances,S=1.0, curve="convex", direction="decreasing")
kn.plot_knee()

Let us check the value of K using kn.knee

In [7]:
kn.knee
Out[7]:
3
In [8]:
kmeans = KMeans(n_clusters= kn.knee).fit(X.reshape(-1,1))
c = kmeans.predict(X.reshape(-1,1))
min_and_max = []
for i in range(kn.knee):
    min_and_max.append([-np.inf,np.inf])
for i in range(len(X)):
    cluster = c[i]
    if X[i] > min_and_max[cluster][0]:
        min_and_max[cluster][0] = X[i]
    if X[i] < min_and_max[cluster][1]:
        min_and_max[cluster][1] = X[i]

Let us check the min and max values of our clusters.

In [9]:
min_and_max
Out[9]:
[[2461.9099, 2365.55], [2687.98, 2508.0801], [2357.02, 2239.4399]]

There are 3 clusters shown above, every cluster has max and min value.

At the writing of this notebook, Google stock price is 2687.98 (high of day) which happens to be 52 week high as well. Therefore based on the above clusters, we can say that 2687.98 is the resistance and next support level is 2508.0801. The next levels of support are 2461.9099, 2365.55 2357.02, 2239.4399.

Remember these support and resistances will change depending upon the range of data and value of Clustering parameter K.

Stocks Support and Resistance Using Agglomerative Clustering

In [10]:
mobj = collection_remote.find({'ticker':'GOOGL'}).sort([('_id',pymongo.DESCENDING)]).limit(60)
prices = []
for doc in mobj:
    prices.append(doc['high'])

Another approach that can be used is Agglomerative Clustering which is hierarchical clustering.

Agglomerative clustering is a bottoms up approach which merges child clusters to find out the big clusters of data.

I have found Aggloerative to be useful on stocks rolling data.

Let us create a rolling data of 20 days each for both calculating max and min values.

In [11]:
df = pd.DataFrame(prices)
max = df.rolling(20).max()
max.rename(columns={0: "price"},inplace=True)
min = df.rolling(20).min()
min.rename(columns={0: "price"},inplace=True)

Below step is required to prepare the data in two column format.

In [12]:
maxdf = pd.concat([max,pd.Series(np.zeros(len(max))+1)],axis = 1)
mindf = pd.concat([min,pd.Series(np.zeros(len(min))+-1)],axis = 1)    
maxdf.drop_duplicates('price',inplace = True)
mindf.drop_duplicates('price',inplace = True)

Let us use n_clusters =3 value for our number of clusters.

In [13]:
F = maxdf.append(mindf).sort_index()
F = F[ F[0] != F[0].shift() ].dropna()

# Create [x,y] array where y is always 1
X = np.concatenate((F.price.values.reshape(-1,1),
                    (np.zeros(len(F))+1).reshape(-1,1)), axis = 1 )

cluster = AgglomerativeClustering(n_clusters=3,
          affinity='euclidean', linkage='ward')
cluster.fit_predict(X)
F['clusters'] = cluster.labels_

F2 = F.loc[F.groupby('clusters')['price'].idxmax()]

# Plotit
fig, axis = plt.subplots()
for row in F2.itertuples():

    axis.axhline( y = row.price, 
            color = 'green', ls = 'dashed' )

axis.plot( F.index.values, F.price.values )
plt.show()

Let us plot our clusters now. As shown below, there are 2 clusters found. If we take in to account the todays closing price of Google which is 2638.00, we can say that 2687.98 is the resistance and 2357.02 is the support.

In [14]:
F2
Out[14]:
price 0 clusters
29 2561.1499 1.0 0
19 2687.9800 1.0 1
19 2445.1399 -1.0 0
37 2357.0200 -1.0 2

One thing to notice here is that, there are only 2 clusters at price 2357.02 which is not that many. To see if we can find more number of clusters either we have to increase our number of price points in our source data or increase the number of clusters, or make our rolling window smaller.

Let us increase the number of clusters to 5 and see what happens.

In [15]:
F = maxdf.append(mindf).sort_index()
F = F[ F[0] != F[0].shift() ].dropna()

# Create [x,y] array where y is always 1
X = np.concatenate((F.price.values.reshape(-1,1),
                    (np.zeros(len(F))+1).reshape(-1,1)), axis = 1 )

cluster = AgglomerativeClustering(n_clusters=5,
          affinity='euclidean', linkage='ward')
cluster.fit_predict(X)
F['clusters'] = cluster.labels_

F2 = F.loc[F.groupby('clusters')['price'].idxmax()]

# Plotit
fig, axis = plt.subplots()
for row in F2.itertuples():

    axis.axhline( y = row.price, 
            color = 'green', ls = 'dashed' )

axis.plot( F.index.values, F.price.values )
plt.show()
In [16]:
F2
Out[16]:
price 0 clusters
36 2399.0300 -1.0 0
36 2461.9099 1.0 0
29 2561.1499 1.0 1
37 2357.0200 -1.0 2
19 2687.9800 1.0 3
19 2445.1399 -1.0 0
51 2239.4399 -1.0 4
51 2436.0000 1.0 0

Ok this time around we got more number of clusters at price 2239.43 which is quite far from todays closing price of 2638. However the resistance number looks good of 2687.98 based on 3 clusters.

Let us make our rolling window smaller. Instead of 20 days let us make it 10 days.

In [17]:
df = pd.DataFrame(prices)
max = df.rolling(10).max()
max.rename(columns={0: "price"},inplace=True)
min = df.rolling(10).min()
min.rename(columns={0: "price"},inplace=True)
maxdf = pd.concat([max,pd.Series(np.zeros(len(max))+1)],axis = 1)
mindf = pd.concat([min,pd.Series(np.zeros(len(min))+-1)],axis = 1)    
maxdf.drop_duplicates('price',inplace = True)
mindf.drop_duplicates('price',inplace = True)
F = maxdf.append(mindf).sort_index()
F = F[ F[0] != F[0].shift() ].dropna()

# Create [x,y] array where y is always 1
X = np.concatenate((F.price.values.reshape(-1,1),
                    (np.zeros(len(F))+1).reshape(-1,1)), axis = 1 )

cluster = AgglomerativeClustering(n_clusters=5,
          affinity='euclidean', linkage='ward')
cluster.fit_predict(X)
F['clusters'] = cluster.labels_

F2 = F.loc[F.groupby('clusters')['price'].idxmax()]

# Plotit
fig, axis = plt.subplots()
for row in F2.itertuples():

    axis.axhline( y = row.price, 
            color = 'green', ls = 'dashed' )

axis.plot( F.index.values, F.price.values )
plt.show()
In [18]:
F2
Out[18]:
price 0 clusters
45 2318.8899 -1.0 0
45 2399.0300 1.0 4
19 2561.1499 1.0 1
26 2432.0601 -1.0 2
26 2461.9099 1.0 2
9 2687.9800 1.0 3
33 2412.8799 -1.0 4
33 2455.5100 1.0 2

Ok this data looks much better. We got a Google resistance around 2687.98 and support around 2399.03 and 2412.8799 which is quite close to say that support is around 2400.

July 28, 2021 04:38 AM UTC


Podcast.__init__

Taking Aim At The Legacy Of SQL With The Preql Relational Language

SQL has gone through many cycles of popularity and disfavor. Despite its longevity it is objectively challenging to work with in a collaborative and composable manner. In order to address these shortcomings and build a new interface for your database oriented workloads Erez Shinan created Preql. It is based on the same relational algebra that inspired SQL, but brings in more robust computer science principles to make it more manageable as you scale in complexity. In this episode he shares his motivation for creating the Preql project, how he has used Python to develop a new language for interacting with database engines, and the challenges of taking on the legacy of SQL as an individual.

Summary

SQL has gone through many cycles of popularity and disfavor. Despite its longevity it is objectively challenging to work with in a collaborative and composable manner. In order to address these shortcomings and build a new interface for your database oriented workloads Erez Shinan created Preql. It is based on the same relational algebra that inspired SQL, but brings in more robust computer science principles to make it more manageable as you scale in complexity. In this episode he shares his motivation for creating the Preql project, how he has used Python to develop a new language for interacting with database engines, and the challenges of taking on the legacy of SQL as an individual.

Announcements

  • Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Your host as usual is Tobias Macey and today I’m interviewing Erez Shinan about Preql, an interpreted, relational programming language, that specializes in database queries

Interview

  • Introductions
  • How did you get introduced to Python?
  • Can you describe what Preql is and the story behind it?
    • What are goals and target use cases for the project?
  • There have been numerous projects that aim to make SQL more maintainable and composable. What is it about the language and syntax that makes it so challenging?
    • How does Preql approach this problem that is different from other efforts? (e.g. ORMs, dbt-style Jinja, PyPika)
  • How did you approach the design of the syntax to make it familiar to people who know SQL?
  • Can you describe how Preql is implemented?
    • How has the design and architecture changed or evolved since you began working on it?
  • What is a typical workflow for someone using Preql to build a library of analytical queries?
  • Beyond strict compilation to SQL, what are some of the other features that you have incorporated into Preql?
    • How does a Preql program get executed against a target database, particularly when using capabilities that can’t be directly translated to SQL?
  • ** What are the main difficulties / challenges of compiling to SQL ?
  • What are some of the features or use cases that are not immediately obvious or prone to be overlooked that you think are worth mentioning?
  • What are the most interesting, innovative, or unexpected ways that you have seen Preql used?
  • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Preql?
  • When is Preql the wrong choice?
  • What do you have planned for the future of Preql?

Keep In Touch

Picks

Closing Announcements

  • Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at pythonpodcast.com/chat

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

July 28, 2021 02:54 AM UTC

July 27, 2021


PyCoder’s Weekly

Issue #483 (July 27, 2021)

#483 – JULY 27, 2021
View in Browser »

The PyCoder’s Weekly Logo


Python’s collections: A Buffet of Specialized Data Types

Python has a number of useful data types beyond the built-in lists, tuples, dicts, and sets. In this tutorial, you’ll learn all about the series of specialized container data types in the collections module from the Python standard library. Learning the collections module is a great way to level up your Python programming knowledge!
REAL PYTHON

Moving SciPy to the Meson Build System

In accordance with PEP 632, distutils will be deprecated in Python 3.10 and in Python 3.12 it will be removed. This posed a big problem for SciPy, since it’s build system depends on NumPy’s distutils module — an extension of Python’s built-in distutils. The SciPy maintainers set out to find a new build system and settled on Meson, which solves a number of build issues and even scores a 4x speed-up on build times!
RALF GOMMERS

An Immersive Virtual Office For Your Team

alt

What if you could run into people at the office…while you work remotely? Gather makes it possible. Claim a desk, chat by the water cooler, and hold important meetings in an interactive space. Free up to 25 users →
GATHER sponsor

The Unknown Features of Python’s Operator Module

Have you heard of Python’s operator module? It includes functions for common operators, including mathematical operators like +, -, *, and /, and well as operators for getting items from dictionaries and accessing object attributes. The operator module might seem confusing at first glance. This article explores why it exists, what benefits you get from it, and when it makes sense to use it in your code.
MARTIN HEINZ • Shared by Martin Heinz

Python Software Foundation Fellow Members for Q2 2021

PYTHON SOFTWARE FOUNDATION

Discussions

Do Not Use Mutable Objects as Default Arguments in Python

Ah, mutable default arguments. If you haven’t come across them in your Python journey yet, this Reddit thread is full of examples of Python developers that lost time debugging functions with a mutable default argument. Avoiding mutable defaults is good advice in general, but blanket absolute statements are dangerous. To counter this discussion, here’s a Twitter thread exploring some genuine use cases.
REDDIT

Python Jobs

Backend Software Engineer (Remote)

Catalpa International

Python Developer (Remote)

Tessian

Software Developer (Remote)

Univention GmbH

Backend Software Engineer (Washington, D.C., USA)

Quorum

Senior Cloud Platform Engineer (Berlin, Germany)

Apheris

Software Engineer (Remote)

Close

Backend Software Engineer (Remote)

Tessian

Python Web Developer (Remote)

Premiere Digital Services

More Python Jobs >>>

Articles & Tutorials

The Pandas DataFrame: Working With Data Efficiently

In this tutorial, you’ll get started with Pandas DataFrames, which are powerful and widely used two-dimensional data structures. You’ll learn how to perform basic operations with data, handle missing values, work with time-series data, and visualize data from a Pandas DataFrame.
REAL PYTHON course

Testing the Diff

How do you write good tests? It’s a challenging problem because there are a number of layers involved. You need to write good test code that follows best practices. But you also need to write tests that aren’t fragile and make sure tests really test what you expect them to. This article gives an example of a seemingly harmless test that turns out to be pretty fragile. You’ll learn one technique for making a test more robust called “testing the diff.”
FILIPE XIMENES • Shared by Filipe Ximenes

Try Scout Free for 14-days, no CC Needed, and see why Developers Call Scout their Best Friend

alt

Scout uses tracing logic to tie bottlenecks to source code so devs can get back to building great products instead of wasting time fixing performance issues. Our real-time alerting and streamlined dashboards provide the data insights necessary for any developer to become a performance pro →
SCOUT APM sponsor

Beautiful Ideas in Programming: Generators and Continuations

This article explores how generators, which you can create in Python using the yield keyword in a function, are special cases of a more powerful construct called “continuations.” The author compares Python’s generator syntax to creating continuations in Scheme, a modern LISP dialect. The idea isn’t so much to learn how to write generators and continuations, but rather to explore how the ideas are related and to appreciate the elegance and beauty of both concepts.
HSIN-HAO YU

What Can You Do With Python and Counting Objects Using Counter

How is Python being used today, and what can you do with the language? Do you want to develop software, dive into data science and math, automate parts of your job and digital life, or work with electronics? This week on the Real Python Podcast, David Amos is back, and he’s brought another batch of PyCoder’s Weekly articles and projects.
REAL PYTHON podcast

Why Tracing Might Replace (Almost) All Logging

Lightstep CEO and Co-founder Ben Sigelman shares his vision for the future of distributed tracing.
LIGHTSTEP sponsor

Efficient Pagination in Django and Postgres

Pagination breaks up results from a query into chunks called “pages” so that only a few results are returned to the user at a time. In many cases, using native SQL tools like LIMIT and OFFSET can get you up and running with pagination quickly and work fairly well. But on datasets with millions of rows, this method breaks down. This helpful article shows you three methods for efficiently handling pagination using Django and Postgres.
RYAN WESTERBERG • Shared by Manuel Weiss

Your First Steps With Django: Set Up a Django Project

This tutorial provides a walkthrough and a reference for starting a Django project and app. You can use it as a quick setup guide for any future Django project and tutorial you’ll work on.
REAL PYTHON

Projects & Code

qmsolve: A Module for Solving and Visualizing the Schrödinger Equation

GITHUB.COM/QUANTUM-VISUALIZATIONS

jupyter-text2code: A Proof-of-Concept Jupyter Extension Which Converts English Queries Into Python Code

GITHUB.COM/DEEPKLARITY

dlib: A C++/Python Toolkit for Making Real World Machine Learning and Data Analysis Applications

GITHUB.COM/DAVISKING

aptus: Mandelbrot Set Explorer and Renderer

GITHUB.COM/NEDBAT

hook-slinger: A Generic Service to Send, Retry, and Manage Webhooks

GITHUB.COM/REDNAFI • Shared by Redowan Delowar

Events

Real Python Office Hours (Virtual)

July 28, 2021
REALPYTHON.COM

EuroPython 2021 (Virtual)

July 26 – August 1, 2021
EUROPYTHON.EU

PyOhio 2021 (Virtual)

July 31 — August 1, 2021
PYOHIO.ORG

PyCon India 2021 (Virtual)

September 17 – 20, 2021
PYCON.ORG


Happy Pythoning!
This was PyCoder’s Weekly Issue #483.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

July 27, 2021 07:30 PM UTC


Real Python

The Pandas DataFrame: Working With Data Efficiently

The Pandas DataFrame is a structure that contains two-dimensional data and its corresponding labels. DataFrames are widely used in data science, machine learning, scientific computing, and many other data-intensive fields.

DataFrames are similar to SQL tables or the spreadsheets that you work with in Excel or Calc. In many cases, DataFrames are faster, easier to use, and more powerful than tables or spreadsheets because they’re an integral part of the Python and NumPy ecosystems.

In this course, you’ll learn:


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

July 27, 2021 02:00 PM UTC


Peter Bengtsson

How to install Python Poetry in GitHub Actions in MUCH faster way

We use Poetry in a GitHub project. There's a pyproject.toml file (and a poetry.lock file) which with the help of the executable poetry gets you a very reliable Python environment. The only problem is that adding the poetry executable is slow. Like 10+ seconds slow. It might seem silly but in the project I'm working on, that 10+s delay is the slowest part of a GitHub Action workflow which needs to be fast because it's trying to post a comment on a pull request as soon as it possibly can.

Installing poetry being the slowest partt
First I tried caching $(pip cache dir) so that the underlying python -v pip install virtualenv -t $tmp_dir that install-poetry.py does would get a boost from avoiding network. The difference was negligible. I also didn't want to get too weird by overriding how the install-poetry.py works or even make my own hacky copy. I like being able to just rely on the snok/install-poetry GitHub Action to do its thing (and its future thing).

The solution was to cache the whole $HOME/.local directory. It's as simple as this:

- name: Load cached $HOME/.local
  uses: actions/cache@v2.1.6
  with:
    path: ~/.local
    key: dotlocal-${{ runner.os }}-${{ hashFiles('.github/workflows/pr-deployer.yml') }}

The key is important. If you do copy-n-paste this block of YAML to speed up your GitHub Action, please remember to replace .github/workflows/pr-deployer.yml with the name of your .yml file that uses this. It's important because otherwise, the cache might be overzealously hot when you make a change like:

       - name: Install Python poetry
-        uses: snok/install-poetry@v1.1.6
+        uses: snok/install-poetry@v1.1.7
         with:

...for example.

Now, thankfully install-poetry.py (which is the recommended way to install poetry by the way) can notice that it's already been created and so it can omit a bunch of work. The result of this is as follows:

A fast install poetry

From 10+ seconds to 2 seconds. And what's neat is that the optimization is very "unintrusive" because it doesn't mess with how the snok/install-poetry workflow works.

But wait, there's more!

If you dig up our code where we use poetry you might find that it does a bunch of other caching too. In particular, it caches .venv it creates too. That's relevant but ultimately unrelated. It basically caches the generated virtualenv from the poetry install command. It works like this:

- name: Load cached venv
  id: cached-poetry-dependencies
  uses: actions/cache@v2.1.6
  with:
    path: deployer/.venv
    key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}-${{ hashFiles('.github/workflows/pr-deployer.yml') }}

...

- name: Install deployer
  run: |
    cd deployer
    poetry install
  if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'

In this example, deployer is just the name of the directory, in the repository root, where we have all the Python code and the pyproject.toml etc. If you have yours at the root of the project you can just do: run: poetry install and in the caching step change it to: path: .venv.

Now, you get a really powerful complete caching strategy. When the caches are hot (i.e. no changes to the .yml, poetry.lock, or pyproject.toml files) you get the executable (so you can do poetry run ...) and all its dependencies in roughly 2 seconds. That'll be hard to beat!

July 27, 2021 01:50 PM UTC


Mike Driscoll

Creating Spreadsheets with OpenPyXL and Python

Reading Excel spreadsheets is all well and good. However, you also need to be able to create or edit a spreadsheet. The focus of this chapter will be on learning how to do that! OpenPyXL lets you create Microsoft Excel spreadsheets with a minimum of fuss.

Creating Excel spreadsheets using Python allows you to generate a new type of report that your users will use. For example, you might receive your data from a client in the form of JSON or XML. These data formats are not something that most accountants or business people are used to reading.

Once you learn how to create Excel spreadsheets with Python, you can leverage that knowledge to transform other data into Excel spreadsheets. This knowledge also allows you to do the reverse, taking in an Excel spreadsheet and output a different format, such as JSON or XML.

In this article, you will learn how to use OpenPyXL to do the following:

Let’s get started by creating a brand new spreadsheet!

Editor’s note: This article is based on a chapter from the book: Automating Excel with Python. You can order a copy on Gumroad or Kickstarter.

Creating a Spreadsheet

Creating an empty spreadsheet using OpenPyXL doesn’t take much code. Open up your Python editor and create a new file. Name it creating_spreadsheet.py.

Now add the following code to your file:

# creating_spreadsheet.py

from openpyxl import Workbook


def create_workbook(path):
    workbook = Workbook()
    workbook.save(path)


if __name__ == "__main__":
    create_workbook("hello.xlsx")

The critical piece here is that you need to import the Workbook class. This class allows you to instantiate a workbook object that you can then save. All this code does is create the file that you pass to it and save it.

Your new spreadsheet will look like this:

An empty Excel spreadsheet

Now you’re ready to learn how to add some data to the cells in your spreadsheet.

Writing to a Spreadsheet

When writing data in a spreadsheet, you need to get the “sheet” object. You learned how to do that in the previous chapter using workbook.active, which gives you the active or currently visible sheet. You could also explicitly tell OpenPyXL which sheet you want to access by passing it a sheet title.

For this example, you will create another new program and then use the active sheet. Open up a new file and name it adding_data.py. Now add this code to your file:

# adding_data.py

from openpyxl import Workbook


def create_workbook(path):
    workbook = Workbook()
    sheet = workbook.active
    sheet["A1"] = "Hello"
    sheet["A2"] = "from"
    sheet["A3"] = "OpenPyXL"
    workbook.save(path)


if __name__ == "__main__":
    create_workbook("hello.xlsx")

This code will overwrite the previous example’s Excel spreadsheet. After you create the Workbook() object, you grab the active Worksheet. Then you add text strings to the cells: A1, A2, and A3. The last step is to save your new spreadsheet.

When you run this code, your new spreadsheet will look like this:

Hello World Excel Spreadsheet

 

You can use this technique to write data to any cell in your spreadsheet.

Now let’s find out how to add and remove a worksheet!

Adding and Removing Sheets

Adding a worksheet to a workbook happens automatically when you create a new Workbook. The Worksheet will be named “Sheet” by default. If you want, you can set the name of the sheet yourself.

To see how this works, create a new file named creating_sheet_title.py and add the following code:

# creating_sheet_title.py

from openpyxl import Workbook


def create_sheets(path):
    workbook = Workbook()
    sheet = workbook.active
    sheet.title = "Hello"
    sheet2 = workbook.create_sheet(title="World")
    workbook.save(path)


if __name__ == "__main__":
    create_sheets("hello_sheets.xlsx")

Here you create the Workbook and then grab the active Worksheet. You can then set the Worksheet’s title using the title attribute. The following line of code adds a new worksheet to the Workbook by calling create_sheet().

The create_sheet() method takes two parameters: title and index. The title attribute gives a title to the Worksheet. The index tells the Workbook where to insert the Worksheet, from left to right. If you specify zero, your Worksheet gets inserted at the beginning.

If you run this code, your new spreadsheet will look like this:

Creating Multiple Worksheets

Sometimes you will need to delete a worksheet. Perhaps that sheet no longer has valid information, or it was created by accident.

To see how to delete a worksheet, create another new file and name it delete_sheets.py. Then add this code:

# delete_sheets.py

import openpyxl


def create_worksheets(path):
    workbook = openpyxl.Workbook()
    workbook.create_sheet()
    print(workbook.sheetnames)
    # Insert a worksheet
    workbook.create_sheet(index=1, title="Second sheet")
    print(workbook.sheetnames)
    del workbook["Second sheet"]
    print(workbook.sheetnames)
    workbook.save(path)


if __name__ == "__main__":
    create_worksheets("del_sheets.xlsx")

In this example, you create two new sheets. The first Worksheet has no title specified, so it defaults to “Sheet1”. You supply a title to the second sheet, and then you print out all the current worksheet titles.

Next, you use Python’s del keyword to delete the Worksheet’s name from the workbook, which removes the sheet. Then you print out the current worksheet titles again.

Here is the output from running the code:

['Sheet', 'Sheet1']
['Sheet', 'Second sheet', 'Sheet1']
['Sheet', 'Sheet1']

The first Worksheet gets created automatically when you instantiate the Workbook. The Worksheet is named “Sheet”. Then you make “Sheet1”. Lastly, you create “Second sheet”, but you insert it at position 1, which tells the Workbook to shift ‘Sheet1’ to the right by one position.

You can see from the output above how the worksheets are ordered before and after you add and delete the “Second sheet”.

Now let’s learn about inserting and removing rows and columns!

Inserting and Deleting Rows and Columns

The OpenPyXL package provides you with several methods that you can use to insert or delete rows and columns. These methods are a part of the Worksheet object.

You will learn about the following four methods:

Each of these methods can take these two arguments:

You can use the insert methods to insert rows or columns at the specified index.

Open up a new file and name it insert_demo.py. Then enter this code in your new file:

# insert_demo.py

from openpyxl import Workbook


def inserting_cols_rows(path):
    workbook = Workbook()
    sheet = workbook.active
    sheet["A1"] = "Hello"
    sheet["A2"] = "from"
    sheet["A3"] = "OpenPyXL"
    # insert a column before A
    sheet.insert_cols(idx=1)
    # insert 2 rows starting on the second row
    sheet.insert_rows(idx=2, amount=2)
    workbook.save(path)


if __name__ == "__main__":
    inserting_cols_rows("inserting.xlsx")

Here you create another new Spreadsheet. In this case, you add text to the first three cells in the “A” column. Then you insert one column at index one. That means you inserted a single column before “A”, which causes the cells in column “A” to shift to column “B”.

Next, you insert two new rows starting at index two. This code will insert two rows between the first and second rows.

You can see how this changes things by taking a look at the following screenshot:

Inserting rows and columns

Try changing the indexes or number of rows and columns that you want to insert and see how it works.

You will also need to delete columns and rows from time to time. To do that you will use .delete_rows() and .delete_cols().

Open up a new file and name it delete_demo.py. Then add this code:

# delete_demo.py

from openpyxl import Workbook


def deleting_cols_rows(path):
    workbook = Workbook()
    sheet = workbook.active
    sheet["A1"] = "Hello"
    sheet["B1"] = "from"
    sheet["C1"] = "OpenPyXL"
    sheet["A2"] = "row 2"
    sheet["A3"] = "row 3"
    sheet["A4"] = "row 4"
    # Delete column A
    sheet.delete_cols(idx=1)
    # delete 2 rows starting on the second row
    sheet.delete_rows(idx=2, amount=2)
    workbook.save(path)


if __name__ == "__main__":
    deleting_cols_rows("deleting.xlsx")

In this example, you add text to six different cells. Four of those cells are in column “A”. Then you use delete_cols() to delete column “A”! That means you got rid of four values. Next, you delete two rows, starting at row number two.

When you run this code, your result should look like this:

Deleting Rows and Columns

Try editing the index or amount values to get familiar with deleting rows and columns.

Now you are ready to learn about editing a spreadsheet’s values!

Editing Cell Data

You can use OpenPyXL to change the values in a pre-existing Excel spreadsheet. You can do that by specifying the cell you want to change and then setting it to a new value.

For this example, you will use the inserting.xlsx file you created in the previous section. Now create a new Python file named editing_demo.py. Then add the following code:

# editing_demo.py

from openpyxl import load_workbook


def edit(path, data):
    workbook = load_workbook(filename=path)
    sheet = workbook.active
    for cell in data:
        current_value = sheet[cell].value
        sheet[cell] = data[cell]
        print(f'Changing {cell} from {current_value} to {data[cell]}')
    workbook.save(path)

if __name__ == "__main__":
    data = {"B1": "Hi", "B5": "Python"}
    edit("inserting.xlsx", data)

This code loads up the Excel file that you created in the previous section. It then loops over each value in the data dictionary that you passed to the edit() function. You get the current value for the cell using a key in the dictionary. Then you change that cell’s value to match the value in the dictionary.

To make it more obvious what is going on, you print out the old and new values of the cell.

When you run this code, you will see the following output:

Changing B1 from Hello to Hi
Changing B5 from OpenPyXL to Python

Open up the new version of the inserting.xlsx file, and it should now look like this:

Editing Cells

Here you can see how the cell values have changed to match the one specified in the data dictionary.

Now you can move on and learn how to create merged cells!

Creating Merged Cells

A merged cell is where two or more cells get merged into one. To set a MergedCell’s value, you have to use the top-left-most cell. For example, if you merge “A2:E2”, you would set the value of cell “A2” for the merged cells.

To see how this works in practice, create a file called merged_cells.py and add this code to it:

# merged_cells.py

from openpyxl import Workbook
from openpyxl.styles import Alignment


def create_merged_cells(path, value):
    workbook = Workbook()
    sheet = workbook.active
    sheet.merge_cells("A2:E2")
    top_left_cell = sheet["A2"]
    top_left_cell.alignment = Alignment(horizontal="center",
                                        vertical="center")
    sheet["A2"] = value
    workbook.save(path)


if __name__ == "__main__":
    create_merged_cells("merged.xlsx", "Hello World")

OpenPyXL has many ways to style cells. In this example, you import Alignment from openpyxl.styles. You will learn more about styles and formatting cells in a later chapter.

Here you merge the cells “A2:E2” and set the alignment to the center of the cell. Then you set the value of “A2” to a string that you passed to the create_merged_cells() function.

When you run this example, your new Excel spreadsheet will look like this:

Merged Cells

To get some hands-on experience, change the range of cells you want to merge and try it with and without the alignment set.

Now you are ready to learn about folding columns or rows!

Folding Rows and Columns

Microsoft Excel supports the folding of rows and columns. The term “folding” is also called “hiding” or creating an “outline”. The rows or columns that get folded can be unfolded (or expanded) to make them visible again. You can use this functionality to make a spreadsheet briefer. For example, you might want to only show the sub-totals or the results of equations rather than all of the data at once.

OpenPyXL supports folding too. To see how this works, create a new file named folding.py and enter the following code:

# folding.py

import openpyxl


def folding(path, rows=None, cols=None, hidden=True):
    workbook = openpyxl.Workbook()
    sheet = workbook.active

    if rows:
        begin_row, end_row = rows
        sheet.row_dimensions.group(begin_row, end_row, hidden=hidden)

    if cols:
        begin_col, end_col = cols
        sheet.column_dimensions.group(begin_col, end_col, hidden=hidden)

    workbook.save(path)


if __name__ == "__main__":
    folding("folded.xlsx", rows=(1, 5), cols=("C", "F"))

Your folding() function accepts a tuple of rows or columns or both. You can tell OpenPyXL whether or not you want those rows and columns to be hidden, or folded. In this example, you fold rows 1-5 and columns C-F. To cause the folding to occur, you need to call sheet.row_dimensions.group().

When you run this code, your spreadsheet will look like this:

Folding Cells

You can see in this spreadsheet that some of the rows and columns are folded or hidden. There is a “+” symbol next to row 6 and another “+” symbol above column “G”. If you click on either of those buttons, it will expand the folded rows or columns.

Give this code a try. You can also experiment with different row or column ranges.

Now you are ready to learn how to freeze a pane!

Freezing Panes

Microsoft Excel allows you to freeze panes. What that means is that you can freeze one or more columns or rows. One popular use case is to freeze a row of headers so that the headers are always visible while scrolling through a lot of data.

OpenPyXL provides a freeze_panes attribute on the Worksheet object that you can set. You need to select a cell below and to the right of the columns that you want to freeze. For example, if you want to freeze the first row in your spreadsheet, then you would select cell at “A2” to apply the freeze to that row.

You can see how this works by writing some code. Open up a new file and name it freezing_panes.py. Then enter the following into it:

# freezing_panes.py

from openpyxl import Workbook


def freeze(path, row_to_freeze):
    workbook = Workbook()
    sheet = workbook.active
    sheet.title = "Freeze"
    sheet.freeze_panes = row_to_freeze
    headers = ["Name", "Address", "State", "Zip"]
    sheet["A1"] = headers[0]
    sheet["B1"] = headers[1]
    sheet["C1"] = headers[2]
    sheet["D1"] = headers[3]
    data = [dict(zip(headers, ("Mike", "123 Storm Dr", "IA", "50000"))),
            dict(zip(headers, ("Ted", "555 Tornado Alley", "OK", "90000")))]
    row = 2
    for d in data:
        sheet[f'A{row}'] = d["Name"]
        sheet[f'B{row}'] = d["Address"]
        sheet[f'C{row}'] = d["State"]
        sheet[f'D{row}'] = d["Zip"]
        row += 1
    workbook.save(path)


if __name__ == "__main__":
    freeze("freeze.xlsx", row_to_freeze="A2")

Here you create a new Workbook and set the active sheet’s title to “Freeze”. Then you set the freeze_panes attribute to “A2”. The rest of the code in the function adds a couple of rows of data to the Worksheet.

When you run this code, the spreadsheet that you create will look like this:

Freeze Panes

Try scrolling down through some rows in the spreadsheet. The top row should always remain visible because it has been “frozen”.

Wrapping Up

You can use OpenPyXL not only to create an Excel spreadsheet, but modify a pre-existing one. In this chapter, you learned how to do the following:

Give the examples in this chapter a try. Then modify them a bit to see what else you can do on your own.

The post Creating Spreadsheets with OpenPyXL and Python appeared first on Mouse Vs Python.

July 27, 2021 01:06 PM UTC


Stack Abuse

How to Get the Max Element of a Pandas DataFrame - Rows, Columns, Entire DataFrame

How to Get the Max Element of a Pandas DataFrame

A DataFrame is a data structure that represents a special kind of two-dimensional array, built on top of multiple Series objects. These are the central data structures of Pandas - an extremely popular and powerful data analysis framework for Python.

If you're not already familiar with DataFrames and how they work, read our Guide to DataFrames.

DataFrames have the ability to give a name to rows and/or columns, and in a sense, represent tables.

Let's import Pandas and create a DataFrame from a dictionary:

import pandas as pd

df_data = {
    "column1": [24, 9, 20, 24],
    "column2": [17, 16, 201, 16]
}

df = pd.DataFrame(df_data) 
print(df)

Pandas has a great integration with Python and we can easily create DataFrames from dictionaries. The df we've constructed now contains the columns and their respective values:

   column1  column2
0       24       17
1        9       16
2       20      201
3       24       16

Each column has a list of elements, and we can search for the maximum element of each column, each row or the entire DataFrame.

Find Maximum Element in Pandas DataFrame's Column

To find the maximum element of each column, we call the max() method of the DataFrame class, which returns a Series of column names and their largest values:

max_elements = df.max()
print(max_elements)

This will give us the max value for each column of our df, as expected:

column1     24
column2    201
dtype: int64

However, to find the max() element of a single column, you first isolate it and call the max() method on that specific Series:

max_element = df['column1'].max()
print(max_element)
24

Find Maximum Element in Pandas DataFrame's Row

Finding the max element of each DataFrame row relies on the max() method as well, but we set the axis argument to 1.

The default value for the axis argument is 0. If the axis equals to 0, the max() method will find the max element of each column. On the other hand, if the axis equals to 1, the max() will find the max element of each row.

max_elements = df.max(axis=1)
print(max_elements)

This will give us the max value for each row of our df, as expected:

0     24
1     16
2    201
3     24
dtype: int64

Alternatively, if you'd like to search through a specific row, you can access it via iloc[]:

print(df)

for row in df.index:
    print(f'Max element of row {row} is:', max(df.iloc[row]))

We've printed the df for reference to make it easier to verify the results, and obtained the max() element of each row, obtained through iloc[]:

column1  column2
0       24       17
1        9       16
2       20      201
3       24       16

Max element of row 0 is: 24
Max element of row 1 is: 16
Max element of row 2 is: 201
Max element of row 3 is: 24

Find Maximum Element in Entire Pandas DataFrame

Finally, we can take a look at how to find the max element in an entire DataFrame.

Based on what we've previously seen, this should be pretty simple. We'll just use the built-in max() method and pass it one of two previously created lists of max elements - either for all rows or all columns. These are two facets of the same data, so the same result is guaranteed.

This should give us a single highest value in the entire df:

max_by_columns = df.max()
max_by_rows = df.max(axis=1)

df_max = max(max_by_columns)
print("Max element based on the list of columns: ", df_max)

df_max2 = max(max_by_rows)
print("Max element based on the list of rows: ", df_max2)

This will output:

Max element based on the list of columns:  201
Max element based on the list of rows:  201

This is both expected and correct! The max element of a list of max elements of each row should be the same as the max element of a list of max elements of each column and both of them should be the same as the max element of the entire DataFrame.

Conclusion

In this short tutorial, we've taken a look at how to find the maximum element of a Pandas DataFrame, for columns, rows and the entire DataFrame instance.

July 27, 2021 12:30 PM UTC


PyBites

PyBites Website 2.0 is here

Hey, what just happened to the pybit.es site?!

After 4.5 years of content updates, products and growth, the site just got outdated, unmanageable and even confusing. The pain was real and we needed a fix!

The site was first created using Pelican, a (Python) static site generator + GitHub pages and we were happy with it. It’s great technology and it supported our growth as a blog, team and business.

However, it was hard to keep the design professional / responsive and made it difficult to work with multiple authors.

A simple refactoring could not do this time. It was time for a complete rewrite and redesign of the site.

So we hired a professional web developer and he did an amazing job!

Screenshot 2021 07 27 at 14.01.00How the PyBites website has evolved over the years.

A few features we want to highlight:

We hope you like the new design and as always we’re happy to receive any feedback to keep improving it.

We’d also love to hear what we can do more of to serve you well on your Python journey.

Thanks for being a part of PyBites.

PS. by the way, Pelican is still an amazing open source tool we recommend to get your site up and running fast (see our training here).

July 27, 2021 12:25 PM UTC


Python for Beginners

When to Use Comments vs. Docstrings in Python

Adding comments to your program will help other programmers understand your work. Without comments, it can be difficult to understand other people’s code. This is why it’s important to know when to use comments vs. docstrings as a Python programmer.

Knowing when to use comments and how to use them can be confusing. Do you know the difference between a docstring and a string comment? 

In this tutorial, we’ll cover best practices for writing comments in Python. We’ll also show you how to use docstrings and explain how they differ from ordinary comments. Hopefully we can demystify the process of documenting Python code.

By the end of this post, you should have a firm grasp of when to use comments vs. docstrings in Python.

Using Python Block Comments

Making comments in Python is done in a variety of ways. This can be a little confusing for beginners. Understanding the difference between block comments and string comments will bring some clarity to the situation.

In Python, most comments are written using block comments. Block comments are preceded by a # sign. Python will ignore these statements when the program runs. Use block comments to provide short, explanatory notes.

Example 1: How to write a Block Comment in Python

# this is a Python block comment 

It’s also possible to write a comment at the end of a Python statement. These comments are called inline comments. Inline comments are great for explaining non-obvious code.

Example 2: Inline Comments in Python

offset = width + 1 # compensate for the screen border

Inline comments can be extremely useful, but it’s best to use them sparingly. Less is more is a good rule to follow here.

Using Python String Comments

Block comments only take up a single line. If you need a comment that spans multiple lines, you’ll need to use more block comments. If you have a really long comment, or need to comment out a block of code, you can use a string comment. String comments are written using quotation marks.

Example 3: How to write a String Comment in Python

"""
This is an example of a string comment.
String comments can be many lines long.

Python will ignore them when you run the program.

"""

With string comments, there’s no limit on how long your comment can be. But there is a need for caution. String comments can be mistaken for docstrings if you put them in the wrong place. 

For example, a string comment following a function will be interpreted as a docstring. For this reason, some Python programmers don’t use string comments at all.

Is it wrong to use Python string comments?

Some programmers frown on using strings as Python comments. Because they can be confused with docstrings, some Python coders even argue it’s best to avoid them altogether.

But the official words is that using string comments is perfectly fine. Even Guido Van Rossum, the creator of Python, has okayed the technique.

When to Use Python Comments

Use comments to leave notes about the code for others, and for yourself. Comments have several uses, but they are generally made for the developers.

Unlike docstrings, comments will not be turned into documentation. There are no rules for writing comments, only guidelines. If you look around, you’ll find varying opinions about when and when not to make comments.

In general, most programmers will tell you that comments are necessary in the following cases:

Here are some other tips to keep in mind when you’re making comments:

What is a Python Docstring?

Docstrings stand for Documents Strings and they provide a way of documenting a Python program. Using a docstring, programs can provide a description of the functions, classes, and modules that other programmers might want to use.

Using docstrings, Python developers can provide a simple explanation of what a function or class is used for. Without such documentation, it would be very difficult—if not impossible—to learn new Python modules.

Docstrings can also be used to generate API’s (Application Programming Interfaces). API’s are used to send data across the web. To use an API, it’s necessary to consult the documentation and learn how it works. Docstrings make generating this documentation easier by allowing programs to include it directly in the code.

How to Use Docstrings in Python

Now that we’ve made a distinction between docstrings vs. comments in Python, let’s take a closer look at how docstrings work and how to use them.

How to declare a docstring: Create a new docstring using triple quotation marks. For this reason, docstrings can be confused with string comments.  

How to Access a Docstring: Docstrings can be read using the __doc__ attribute.

What Should a Docstring Look Like?

Unlike comments, docstrings have some formal requirements. To begin with, every docstring begins with a short summary of the function or class it’s related to. 

The summary should be clear and concise. It’s important that programmers immediately grasp what a function does. This, however, is sometimes easier said than done, but should always be the goal.

Further documentation can be provided below the summary, following a blank line. The details of the docstring should serve as a quick reference for functions and arguments.

Documenting Python Functions

When documenting functions in Python, be sure to include descriptions of the function’s parameters. Also, give a detailed explanation of the return value of the function, if it has one.

Example 4: How to document a function in Python

def get_neighbors(some_list, index):
    """Returns the neighbors of a given index in a list.

        Parameters:
            some_list (list): Any basic list.
            index (int): Position of an item in some_list.

        Returns:
            neighbors (list): A list of the elements beside the index in some_list.

    """
    neighbors = []

    if index - 1 >= 0:
        neighbors.append(some_list[index-1])
    if index < len(some_list):
        neighbors.append(some_list[index+1])

    return neighbors

some_list = [8,7,"car","banana",10]

print(get_neighbors(some_list, 2))

Output

[7, 'banana']

Accessing a Document String in Python

To access a docstring in Python, use the __doc__ attribute. This is a special attribute that will retrieve the docstring of a Python function or class. We can use the __doc__ attribute to print a docstring from the command line.

Example 5: Accessing a docstring with the __doc__ attribute

print(get_neighbors.__doc__)

Output

Returns the neighbors of a given index in a list.

        Parameters:
            some_list (list): Any basic list.
            index (int): Position of an item in some_list.

        Returns:
            neighbors (list): A list of the neighboring elements of the index in some_list.

Documenting a Python Class with Docstrings

When documenting classes in Python, it’s important to include a summary of what the class represents as well as an explanation of the classes’ attributes and methods.

Each method in the class will also require a docstring. Below, you’ll find an example of a blog post class. Python docstrings are used to provide a detailed description of the class and it’s methods.

Example 5: How to document a class in Python

class BlogPost:
    """
    A generic blog post model.

    ...

    Attributes
    ----------
    author: str
        who wrote the blog post
    title: str
        the title of the blog post
    content: str
        the body content of the blog post

    Methods
    ----------
    description():
        Prints the author, title, and content of the blog post

    """

    def __init__(self, author, title, content):
        """Constructs all the necessary attributes of the blog post object.

        Parameters
        ----------
            author: str
                who wrote the blog post
            title: str
                the title of the blog post
            content: str
                the body content of the blog post
        """

        self.author = author
        self.title = title
        self.content = content

    def description(self):
        """Prints the author, title, and content of the blog post.

        Returns
        -------
        None
        """

        print(f'{self.author}\n{self.title}\n{self.content}')

Programmers differ when it comes to the styling choices used to write docstrings. With a little research, you can find many style guides. How you choose to format your docstring will depend on many factors, the least of which may be personal choice. Keep in mind who your documentation is aimed at.

Summary

Knowing when to use comments vs. docstrings in Python is crucial to building robust, complex programs. Learning how and when to make comments will not only aid you in collaborative efforts, it will also save you time.

Unlike string comments, docstrings are meant to be read by the public. They provide an insight into how to use functions, classes, and modules. This makes it easier for other programmers to learn how to use the module in their own projects.

A quick recap on comments vs docstrings:

Related Posts

The post When to Use Comments vs. Docstrings in Python appeared first on PythonForBeginners.com.

July 27, 2021 12:00 PM UTC


Python⇒Speed

Saving memory with Pandas 1.3's new string dtype

When you’re loading many strings into Pandas, you’re going to use a lot of memory. If you have only a limited number of strings, you can save memory with categoricals, but that’s only helpful in a limited number of situations.

With Pandas 1.3, there’s a new option that can save memory on large number of strings as well, simply by changing to a new column type. Let’s see how.

Read more...

July 27, 2021 12:00 AM UTC