This chapter will look at some of the central types of Python objects. We’ll look at working with different kinds of numbers, working with strings, and using tuples. These are the simplest kinds of data that Python works with. In later chapters, we’ll look at data structures built on these foundations.
While these recipes start with a beginner’s level of understanding of Python 3.12, they also provide some deeper background for those familiar with the language. In particular, we’ll look at some details of how numbers are represented internally, because this can help when confronted with more advanced numerical programming problems. This will help us distinguish the uses cases for the rich variety of numeric types.
We’ll also look at the two different division operators. These have distinct use cases, and we’ll look at one kind of algorithm that demands truncated division.
When working with strings, there are several common operations that are important. We’ll explore some of the differences between bytes—as used by our OS files—and strings used to represent Unicode text. We’ll look at how we can exploit the full power of the Unicode character set.
In this chapter, we’ll show the recipes as if we’re working from the >>> prompt in interactive Python. This is the prompt that’s provided when running python from the command line or using the Python console in many Integrated Development Environment (IDE) tools. This is sometimes called the read-evaluate-print loop (REPL). In later chapters, we’ll change the style so it looks more like a script file. One goal of this chapter is to encourage interactive exploration because it’s a great way to learn the language.
We’ll cover these recipes to introduce basic Python data types:
Choosing between float, decimal, and fraction
Choosing between true division and floor division
String parsing with regular expressions
Building complicated strings with f-strings
Building complicated strings from lists of strings
Using the Unicode characters that aren’t on our keyboards
Encoding strings – creating ASCII and UTF-8 bytes
Decoding bytes – how to get proper characters from some bytes
Using tuples of items
Using NamedTuples to simplify item access in tuples
We’ll start with numbers, work our way through strings, and end up working with simple combinations of objects in the form of tuples and NamedTuple objects.
Python offers several ways to work with rational numbers and approximations of irrational numbers. We have three basic choices:
Float
Decimal
Fraction
When we have choices, it helps to have some criteria for making a selection.
1.1.1 Getting ready
There are three general cases for expressions that involve numbers beyond integers, which are:
Currency: Dollars, cents, euros, and so on. Currency generally has a fixed number of decimal places and rounding rules to properly quantize results.
Rational Numbers or Fractions: When we scale a recipe that serves eight, for example, down to five people, we’re doing fractional math using a scaling factor of
.Floating Point: This includes all other kinds of calculations. This also includes irrational numbers, like π, root extraction, and logarithms.
.When we have one of the first two cases, we should avoid floating-point numbers.
1.1.2 How to do it...
We’ll look at each of the three cases separately.
Doing currency calculations
When working with currency, we should always use the decimal module. If we try to use the values of Python’s built-in float type, we can run into problems with the rounding and truncation of numbers:
To work with currency, import the Decimal class from the decimal module:
>>> from decimal import DecimalWe need to create Decimal objects from strings or integers. In this case, we want 7.25%, which is
. We can compute the value using Decimal objects:>>> tax_rate = Decimal(’7.25’)/Decimal(100) >>> purchase_amount = Decimal(’2.95’) >>> tax_rate * purchase_amount Decimal(’0.213875’)We could also use Decimal(’0.0725’) instead of doing the division explicitly.
To round to the nearest penny, create a penny object:
>>> penny = Decimal(’0.01’)Quantize the result using the penny object:
>>> total_amount = purchase_amount + tax_rate * purchase_amount >>> total_amount.quantize(penny) Decimal(’3.16’)
. We can compute the value using This uses the default rounding rule of ROUND_HALF_EVEN. The Decimal module offers other rounding variations. We might, for example, do something like this:
>>> import decimal >>> total_amount.quantize(penny, decimal.ROUND_UP) Decimal(’3.17’)This shows the consequences of using a different rounding rule.
Fraction calculations
When we’re doing calculations that have exact fraction values, we can use the fractions module to create rational numbers. In this example, we want to scale a recipe for eight down to five people, using 
of each ingredient. When the recipe calls for 2
cups of rice, what does that scale down to?
To work with fractions, we’ll do this:
Import the Fraction class from the fractions module:
>>> from fractions import FractionCreate Fraction objects from strings, integers, or pairs of integers. We created one fraction from a string, ’2.5’. We created the second fraction from a floating-point expression, 5/8. This only works when the denominator is a power of 2:
>>> sugar_cups = Fraction(’2.5’) >>> scale_factor = Fraction(5/8) >>> sugar_cups * scale_factor Fraction(25, 16)
We can see that we’ll use almost a cup and a half of rice to scale the recipe for five people instead of eight. While float values will often be useful for rational fractions, they may not be exact unless the denominator is a power of two.
Floating-point approximations
Python’s built-in float type can represent a wide variety of values. The trade-off here is that a float value is often an approximation. There may be a small discrepancy that reveals the differences between the implementation of float and the mathematical ideal of an irrational number:
To work with float, we often need to round values to make them look sensible. It’s important to recognize that all float calculations are an approximation:
>>> (19/155)*(155/19) 0.9999999999999999Mathematically, the value should be 1. Because of the approximations used, the computed result isn’t exactly 1. We can use round(answer,3) to round to three digits, creating a value that’s more useful:
>>> answer = (19/155)*(155/19) >>> round(answer, 3) 1.0
Approximations have a very important consequence.
Don’t compare floating-point values for exact equality.
Code that uses an exact == test between floating-point numbers has the potential to cause problems when two approximations differ by a single bit.
The float approximation rules come from the IEEE, and are not a unique feature of Python. Numerous programming languages work with float approximations and have identical behavior.
1.1.3 How it works...
For these numeric types, Python offers a variety of operators: +, -, *, /, //, %, and **. These are for addition, subtraction, multiplication, true division, truncated division, modulo, and raising to a power, respectively. We’ll look at the two division operators, / and //, in the Choosing between true division and floor division recipe.
Python will do some conversions between the various numeric types. We can mix int and float values; the integers will be promoted to floating-point to provide the most accurate answer possible. Similarly, we can mix int and Fraction as well as mixing int and Decimal. Note that we cannot casually mix Decimal with float or Fraction; an explicit conversion function will be required.
It’s important to note that float values are approximations. The Python syntax allows us to write floating-point values using base 10 digits; however, that’s not how values are represented internally.
We can write the value 8.066 × 1067 like this in Python:
>>> 8.066e+67 8.066e+67The actual value used internally will involve a binary approximation of the decimal value we wrote. The internal value for this example is this:
>>> (6737037547376141/(2**53))*(2**226) 8.066e+67The numerator is a big number, 6737037547376141. The denominator is always 253. This is why values can get truncated.
We can use the math.frexp() function to see these internal details of a number:
>>> import math >>> math.frexp(8.066E+67) (0.7479614202861186, 226)The two parts are called the mantissa (or significand) and the exponent. If we multiply the mantissa by 253, we always get a whole number, which is the numerator of the binary fraction.
Unlike the built-in float, a Fraction is an exact ratio of two integer values. We can create ratios that involve integers with a very large number of digits. We’re not limited by a fixed denominator.
A Decimal value, similarly, is based on a very large integer value, as well as a scaling factor to determine where the decimal place goes. These numbers can be huge and won’t suffer from peculiar representation issues.
1.1.4 There’s more...
The Python math module contains several specialized functions for working with floating-point values. This module includes common elementary functions such as square root, logarithms, and various trigonometry functions. It also has some other functions such as gamma, factorial, and the Gaussian error function.
The math module includes several functions that can help us do more accurate floating-point calculations. For example, the math.fsum() function will compute a floating-point sum more carefully than the built-in sum() function. It’s less susceptible to approximation issues.
We can also make use of the math.isclose() function to compare two floating-point values, an expression, and a literal 1.0, to see if they’re nearly equal:
>>> (19/155)*(155/19) == 1.0 False >>> math.isclose((19/155)*(155/19), 1.0) TrueThis function provides us with a way to compare two floating-point numbers meaningfully for near-equality.
Python also offers complex numbers. A complex number has a real and an imaginary part. In Python, we write 3.14+2.78j to represent the complex number 3.14 + 2.78
. Python will comfortably convert between float and complex. We have the usual group of operators available for complex numbers.
To support complex numbers, there’s the cmath package. The cmath.sqrt() function, for example, will return a complex value rather than raise an exception when extracting the square root of a negative number. Here’s an example:
>>> math.sqrt(-2) Traceback (most recent call last): ... ValueError: math domain error >>> import cmath >>> cmath.sqrt(-2) 1.4142135623730951jThis module is helpful when working with complex numbers.
1.1.5 See also
We’ll talk more about floating-point numbers and fractions in the Choosing between true division and floor division recipe.
Python offers us two kinds of division operators. What are they, and how do we know which one to use? We’ll also look at the Python division rules and how they apply to integer values.
1.2.1 Getting ready
There are several general cases for division:
A div-mod pair: We want both parts – the quotient and the remainder. The name refers to the division and modulo operations combined together. We can summarize the quotient and remainder as q,r = (⌊
⌋,a mod b).We often use this when converting values from one base into another. When we convert seconds into hours, minutes, and seconds, we’ll be doing a div-mod kind of division. We don’t want the exact number of hours; we want a truncated number of hours, and the remainder will be converted into minutes and seconds.
The true value: This is a typical floating-point value; it will be a good approximation to the quotient. For example, if we’re computing an average of several measurements, we usually expect the result to be floating-point, even if the input values are all integers.
A rational fraction value: This is often necessary when working in American units of feet, inches, and cups. For this, we should be using the Fraction class. When we divide Fraction objects, we always get exact answers.
We need to decide which of these cases apply, so we know which division operator to use.
1.2.2 How to do it...
We’ll look at these three cases separately.
Doing floor division
When we are doing the div-mod kind of calculations, we might use the floor division operator, //, and the modulo operator, %. The expression a%b gives us the remainder from an integer division of a//b. Or, we might use the divmod() built-in function to compute both at once:
We’ll divide the number of seconds by 3,600 to get the value of hours. The modulo, or remainder in division, computed with the % operator, can be converted separately into minutes and seconds:
>>> total_seconds = 7385 >>> hours = total_seconds // 3600 >>> remaining_seconds = total_seconds % 3600Next, we’ll divide the number of seconds by 60 to get minutes; the remainder is the number of seconds less than 60:
>>> minutes = remaining_seconds // 60 >>> seconds = remaining_seconds % 60 >>> hours, minutes, seconds (2, 3, 5)
Here’s the alternative, using the divmod() function to compute quotient and modulo together:
Compute quotient and remainder at the same time:
>>> total_seconds = 7385 >>> hours, remaining_seconds = divmod(total_seconds, 3600)Compute quotient and remainder again:
>>> minutes, seconds = divmod(remaining_seconds, 60) >>> hours, minutes, seconds (2, 3, 5)
Doing true division
Performing a true division calculation gives a floating-point approximation as the result. For example, about how many hours is 7,385 seconds? Here’s 
using the true division operator:
>>> total_seconds = 7385 >>> hours = total_seconds / 3600 >>> round(hours, 4) 2.0514We provided two integer values, but got a floating-point exact result. Consistent with our previous recipe, when using floating-point values, we rounded the result to avoid having to look at tiny error digits.
Rational fraction calculations
We can do division using Fraction objects and integers. This forces the result to be a mathematically exact rational number:
Create at least one Fraction value:
>>> from fractions import Fraction >>> total_seconds = Fraction(7385)Use the Fraction value in a calculation. Any integer will be promoted to a Fraction:
>>> hours = total_seconds / 3600 >>> hours Fraction(1477, 720)The denominator of 720 doesn’t seem too meaningful. Working with fractions like this requires a bit of finesse to find useful denominators that makes sense to people. Otherwise, converting to a floating-point value can be useful.
If necessary, convert the exact Fraction into a floating-point approximation:
>>> round(float(hours), 4) 2.0514
First, we created a Fraction object for the total number of seconds. When we do arithmetic on fractions, Python will promote any integers to Fraction objects; this promotion means that the math is done as precisely as possible.
1.2.3 How it works...
Python has two division operators:
The / true division operator produces a true, floating-point result. It does this even when the two operands are integers. This is an unusual operator in this respect. All other operators preserve the type of the data. The true division operation – when applied to integers – produces a float result.
The // truncated division operator always produces a truncated result. For two integer operands, this is the truncated quotient. When floating-point operands are used, this is a truncated floating-point result:
>>> 7358.0 // 3600.0 2.0
1.2.4 See also
For more on the choice between floating-point and fractions, see the Choosing between float, decimal, and fraction recipe.
See PEP-238.
How do we decompose a complex string? What if we have complex, tricky punctuation? Or—worse yet—what if we don’t have punctuation, but have to rely on patterns of digits to locate meaningful information?
1.3.1 Getting ready
The easiest way to decompose a complex string is by generalizing the string into a pattern and then writing a regular expression that describes that pattern.
There are limits to the patterns that regular expressions can describe. When we’re confronted with deeply nested documents in a language like HTML, XML, or JSON, we often run into problems and be prohibited from using regular expressions.
The re module contains all of the various classes and functions we need to create and use regular expressions.
Let’s say that we want to decompose text from a recipe website. Each line looks like this:
>>> ingredient = "Kumquat: 2 cups"We want to separate the ingredient from the measurements.
1.3.2 How to do it...
To write and use regular expressions, we often do this:
Generalize the example. In our case, we have something that we can generalize as:
(ingredient words): (amount digits) (unit words)We’ve replaced literal text with a two-part summary: what it means and how it’s represented. For example, ingredient is represented as words, while amount is represented as digits. Import the re module:
>>> import reRewrite the pattern into regular expression (RE) notation:
>>> pattern_text = r’([\w\s]+):\s+(\d+)\s+(\w+)’We’ve replaced representation hints such as ingredient words, a mixture of letters and spaces, with [\w\s]+. We’ve replaced amount digits with \d+. And we’ve replaced single spaces with \s+ to allow one or more spaces to be used as punctuation. We’ve left the colon in place because, in regular expression notation, a colon matches itself.
For each of the fields of data, we’ve used () to capture the data matching the pattern. We didn’t capture the colon or the spaces because we don’t need the punctuation characters.
REs typically use a lot of \ characters. To make this work out nicely in Python, we almost always use raw strings. The r’ tells Python not to look at the \ characters and not to replace them with special characters that aren’t on our keyboards.
Compile the pattern:
>>> pattern = re.compile(pattern_text)Match the pattern against the input text. If the input matches the pattern, we’ll get a match object that shows details of the substring that matched:
>>> match = pattern.match(ingredient) >>> match is None False >>> match.groups() (’Kumquat’, ’2’, ’cups’)Extract the named groups of characters from the match object:
>>> match.group(1) ’Kumquat’ >>> match.group(2) ’2’ >>> match.group(3) ’cups’
Each group is identified by the order of the capture () portions of the regular expression. This gives us a tuple of the different fields captured from the string. We’ll return to the use of the tuple data structure in the Using tuples of items recipe. This can be confusing in more complex regular expressions; there is a way to provide a name, instead of the numeric position, to identify a capture group.
1.3.3 How it works...
There are a lot of different kinds of string patterns that we can describe with regular expressions.
We’ve shown a number of character classes:
\w matches any alphanumeric character (a to z, A to Z, 0 to 9).
\d matches any decimal digit.
\s matches any space or tab character.
These classes also have inverses:
\W matches any character that’s not a letter or a digit.
\D matches any character that’s not a digit.
\S matches any character that’s not some kind of space or tab.
Many characters match themselves. Some characters, however, have a special meaning, and we have to use \ to escape from that special meaning:
We saw that + as a suffix means to match one or more of the preceding patterns. \d+ matches one or more digits. To match an ordinary +, we need to use \+.
We also have * as a suffix, which matches zero or more of the preceding patterns. \w* matches zero or more characters. To match a *, we need to use \*.
We have ? as a suffix, which matches zero or one of the preceding expressions. This character is used in other places, and has a different meaning in the other context. We’ll see it used in ?P<name>...)|,whereitisinside\verb|)— to define special properties for the grouping.
The . character matches any single character. To match a . specifically, we need to use \..
We can create our own unique sets of characters using [] to enclose the elements of the set. We might have something like this:
(?P<name>\w+)\s*[=:]\s*(?P<value>.*)This has a \w+ to match any number of alphanumeric characters. This will be collected into a group called name. It uses \s* to match an optional sequence of spaces. It matches any character in the set [=:]. Exactly one of the two characters in this set must be present. It uses \s* again to match an optional sequence of spaces. Finally, it uses .* to match everything else in the string. This is collected into a group named value.
We can use this to parse strings, like this:
size = 12 weight: 14By being flexible with the punctuation, we can make a program easier to use. We’ll tolerate any number of spaces, and either an = or a : as a separator.
1.3.4 There’s more...
A long regular expression can be awkward to read. We have a clever Pythonic trick for presenting an expression in a way that’s much easier to read:
>>> ingredient_pattern = re.compile( ... r’(?P<ingredient>[\w\s]+):\s+’ # name of the ingredient up to the ":" ... r’(?P<amount>\d+)\s+’ # amount, all digits up to a space ... r’(?P<unit>\w+)’ # units, alphanumeric characters ... )This leverages three syntax rules:
A statement isn’t finished until the () characters match.
Adjacent string literals are silently concatenated into a single long string.
Anything between # and the end of the line is a comment, and is ignored.
We’ve put Python comments after the important clauses in our regular expression. This can help us understand what we did, and perhaps help us diagnose problems later.
We can also use the regular expression’s ”verbose” mode to add gratuitous whitespace and comments inside a regular expression string. To do this, we must use re.X as an option when compiling a regular expression to make whitespace and comments possible. This revised syntax looks like this:
>>> ingredient_pattern_x = re.compile(r’’’ ... (?P<ingredient>[\w\s]+):\s+ # name of the ingredient up to the ":" ... (?P<amount>\d+)\s+ # amount, all digits up to a space ... (?P<unit>\w+) # units, alphanumeric characters ... ’’’, re.X)We can either break the pattern up into separate string components, or make use of extended syntax to make the regular expression more readable. The benefit of providing names shows up when we use the groupdict() method of the match object to extract parsed values by the name associated with the pattern being captured.
1.3.5 See also
The Decoding bytes – how to get proper characters from some bytes recipe.
There are many books on regular expressions and Python regular expressions in particular, like Mastering Python Regular Expressions https://www.packtpub.com/application-development/mastering-python-regular-expressions.
Creating complex strings is, in many ways, the polar opposite of parsing a complex string. We generally use a template with substitution rules to put data into a more complex format.
1.4.1 Getting ready
Let’s say we have pieces of data that we need to turn into a nicely formatted message. We might have data that includes the following:
>>> id = "IAD" >>> location = "Dulles Intl Airport" >>> max_temp = 32 >>> min_temp = 13 >>> precipitation = 0.4And we’d like a line that looks like this:
IAD : Dulles Intl Airport : 32 / 13 / 0.401.4.2 How to do it...
Create an f-string for the result, replacing all of the data items with placeholders. Inside each placeholder, put a variable name (or an expression.) Note that the string uses the prefix of f’. This prefix creates a sophisticated string object where values are interpolated into the template when the string is used:
f’{id} : {location} : {max_temp} / {min_temp} / {precipitation}’For each name or expression, an optional data type can be appended to the names in the template string. The basic data type codes are:
s for string
d for decimal number
f for floating-point number
It would look like this:
f’{id:s} : {location:s} : {max_temp:d} / {min_temp:d} / {precipitation:f}’Because the book’s margins are narrow, the string has been broken to
fit on the page. It’s a single (very wide) line of code.
Add length information where required. Length is not always required, and in some cases, it’s not even desirable. In this example, though, the length information ensures that each message has a consistent format. For strings and decimal numbers, prefix the format with the length like this: 19s or 3d. For floating-point numbers, use a two-part prefix like 5.2f to specify the total length of five characters, with two to the right of the decimal point. Here’s the whole format:
>>> f’{id:3s} : {location:19s} : {max_temp:3d} / {min_temp:3d} / {precipitation:5.2f}’ ’IAD : Dulles Intl Airport : 32 / 13 / 0.40’
1.4.3 How it works...
F-strings can do a lot of relatively sophisticated string assembly by interpolating data into a template. There are a number of conversions available.
We’ve seen three of the formatting conversions—s, d, f—but there are many others. Details can be found in the Formatted string literals section of the Python Standard Library: https://docs.python.org/3/reference/lexical_analysis.html\#formatted-string-literals.
Here are some of the format conversions we might use:
b is for binary, base 2.
c is for Unicode character. The value must be a number, which is converted into a character. Often, we use hexadecimal numbers for these characters, so you might want to try values such as 0x2661 through 0x2666 to see interesting Unicode glyphs.
d is for decimal numbers.
E and e are for scientific notations. 6.626E-34 or 6.626e-34, depending on which E or e character is used.
F and f are for floating-point. For not a number, the f format shows lowercase nan; the F format shows uppercase NAN.
G and g are for general use. This switches automatically between E and F (or e and f) to keep the output in the given sized field. For a format of 20.5G, up to 20-digit numbers will be displayed using F formatting. Larger numbers will use E formatting.
n is for locale-specific decimal numbers. This will insert , or . characters, depending on the current locale settings. The default locale may not have 1,000 separators defined. For more information, see the locale module.
o is for octal, base 8.
s is for string.
X and x are for hexadecimal, base 16. The digits include uppercase A-F and lowercase a-f, depending on which X or x format character is used.
% is for percentage. The number is multiplied by 100 and the output includes a % character.
We have a number of prefixes we can use for these different types. The most common one is the length. We might use {name:5d} to put in a 5-digit number. There are several prefixes for the preceding types:
Fill and alignment: We can specify a specific filler character (space is the default) and an alignment. Numbers are generally aligned to the right and strings to the left. We can change that using <, >, or ^. This forces left alignment, right alignment, or centering, respectively. There’s a peculiar = alignment that’s used to put padding after a leading sign.
Sign: The default rule is a leading negative sign where needed. We can use + to put a sign on all numbers, - to put a sign only on negative numbers, and a space to use a space instead of a plus for positive numbers. In scientific output, we often use {value:5.3f}. The space makes sure that room is left for the sign, ensuring that all the decimal points line up nicely.
Alternate form: We can use the # to get an alternate form. We might have something like {0:#x}, {0:#o}, or {0:#b} to get a prefix on hexadecimal, octal, or binary values. With a prefix, the numbers will look like 0xnnn, 0onnn, or 0bnnn. The default is to omit the two-character prefix.
Leading zero: We can include 0 to get leading zeros to fill in the front of a number. Something like {code:08x} will produce a hexadecimal value with leading zeroes to pad it out to eight characters.
Width and precision: For integer values and strings, we only provide the width. For floating-point values, we often provide width.precision.
There are some times when we won’t use a {name:format} specification. Sometimes, we’ll need to use a {name!conversion} specification. There are only three conversions available:
{name!r} shows the representation that would be produced by repr(name).
{name!s} shows the string value that would be produced by str(name); this is the default behavior if you don’t specify any conversion. Using !s explicitly lets you add string-type format specifiers.
{name!a} shows the ASCII value that would be produced by ascii(name).
Additionally, there’s a handy debugging format specifier available. We can include a trailing equals sign, =, to get a handy dump of a variable or expression. The following example uses both forms:
>>> value = 2**12-1 >>> f’{value=} {2**7+1=}’ ’value=4095 2**7+1=129’
The f-string showed the value of the variable named value and the result of an expression, 2**7+1.
In Chapter7, we’ll leverage the idea of the {name!r} format specification to simplify displaying information about related objects.
1.4.4 There’s more...
The f-string processing relies on the string format() method. We can leverage this method and the related format_map() method for cases where we have more complex data structures.
Looking forward to Chapter5, we might have a dictionary where the keys are simple strings that fit with the format_map() rules:
>>> data = dict( ... id=id, location=location, max_temp=max_temp, ... min_temp=min_temp, precipitation=precipitation ... ) >>> ’{id:3s} : {location:19s} : {max_temp:3d} / {min_temp:3d} / {precipitation:5.2f}’.format_map(data) ’IAD : Dulles Intl Airport : 32 / 13 / 0.40’We’ve created a dictionary object, data, that contains a number of values with keys that are valid Python identifiers: id, location, max_temp, min_temp, and precipitation. We can then use this dictionary with the format_map() method to extract values from the dictionary using the keys.
Note that the formatting template here is not an f-string. It doesn’t have the f" prefix. Instead of using the automatic formatting features of an f-string, we’ve done the interpolation ”the hard way” using the format_map() method of an f-string.
1.4.5 See also
More details can be found in the Formatted string literals section of the Python Standard Library: https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals.
How can we make complicated changes to an immutable string? Can we assemble a string from individual characters?
In most cases, the recipes we’ve already seen give us a number of tools for creating and modifying strings. There are yet more ways in which we can tackle the string manipulation problem. In this recipe, we’ll look at using a list object as a way to decompose and rebuild a string. This will dovetail with some of the recipes in Chapter4.
1.5.1 Getting ready
Here’s a string that we’d like to rearrange:
>>> title = "Recipe 5: Rewriting an Immutable String"We’d like to do two transformations:
Remove the part before :.
Replace the punctuation with _ and make all the characters lowercase.
We’ll make use of the string module:
>>> from string import whitespace, punctuationThis has two important constants:
string.whitespace lists all of the whitespace characters that are also part of ASCII, including space and tab.
string.punctuation lists punctuation marks that are also part of ASCII. Unicode has a large domain of punctuation marks. This is a widely used subset.
1.5.2 How to do it...
We can work with a string exploded into a list. We’ll look at lists in more depth in Chapter4:
Explode the string into a list object:
>>> title_list = list(title)Find the partition character. The index() method for a list has the same semantics as the index() method has for a string. It locates the position with the given value:
>>> colon_position = title_list.index(’:’)Delete the characters that are no longer needed. The del statement can remove items from a list. Unlike strings, lists are mutable data structures:
>>> del title_list[:colon_position+1]Replace punctuation by stepping through each position. In this case, we’ll use a for statement to visit every index in the string:
>>> for position in range(len(title_list)): ... if title_list[position] in whitespace+punctuation: ... title_list[position]= ’_’The expression range(len(title_list)) generates all of the values between 0 and len(title_list)-1. This assures us that the value of position will be each value index in the list. Join the list of characters to create a new string. It seems a little odd to use a zero-length string, ’’, as a separator when concatenating strings together. However, it works perfectly:
>>> title = ’’.join(title_list) >>> title ’_Rewriting_an_Immutable_String’
We assigned the resulting string back to the original variable. The original string object, which had been referred to by that variable, is no longer needed: it’s automatically removed from memory (this is known as garbage collection). The new string object replaces the value of the variable.
1.5.3 How it works...
This is a change in representation trick. Since a string is immutable, we can’t update it. We can, however, convert it into a mutable form; in this case, a list. We can make whatever changes are required to the mutable list object. When we’re done, we can change the representation from a list back to a string and replace the original value of the variable.
Lists provide some features that strings don’t have. Conversely, strings provide a number of features lists don’t have. As an example, we can’t convert a list into lowercase the way we can convert a string.
There’s an important trade-off here:
Strings are immutable, which makes them very fast. Strings are focused on Unicode characters. When we look at mappings and sets, we can use strings as keys for mappings and items in sets because the value is immutable.
Lists are mutable. Operations are slower. Lists can hold any kind of item. We can’t use a list as a key for a mapping or an item in a set because the list value could change.
Strings and lists are both specialized kinds of sequences. Consequently, they have a number of common features. The basic item indexing and slicing features are shared. Similarly, a list uses the same kind of negative index values that a string does: the expression list[-1] is the last item in a list object.
We’ll return to mutable data structures in Chapter4.
1.5.4 See also
Sometimes, we need to build a string, and then convert it into bytes. See the Encoding strings – creating ASCII and UTF-8 bytes recipe for how we can do this.
Other times, we’ll need to convert bytes into a string. See the Decoding bytes – how to get proper characters from some bytes recipe for more information.
A big keyboard might have almost 100 individual keys. Often, fewer than 50 of these keys are letters, numbers, and punctuation. At least a dozen are function keys that do things other than simply insert letters into a document. Some of the keys are different kinds of modifiers that are meant to be used in conjunction with another key—for example, we might have Shift, Ctrl, Option, and Command.
Most operating systems will accept simple key combinations that create about 100 or so characters. More elaborate key combinations may create another 100 or so less popular characters. This isn’t even close to covering the vast domain of characters from the world’s alphabets. And there are icons, emojis, and dingbats galore in our computer fonts. How do we get to all of those glyphs?
1.6.1 Getting ready
Python works in Unicode. There are thousands of individual Unicode characters available.
We can see all the available characters at https://en.wikipedia.org/wiki/List_of_Unicode_characters, as well as at http://www.unicode.org/charts/.
We’ll need the Unicode character number. We may also want the Unicode character name.
A given font on our computer may not be designed to provide glyphs for all of those characters. In particular, Windows computer fonts may have trouble displaying some of these characters. Using the following Windows command to change to code page 65001 is sometimes necessary:
chcp 65001Linux and macOS rarely have problems with Unicode characters.
1.6.2 How to do it...
Python uses escape sequences to extend the ordinary characters we can type to cover the vast space of Unicode characters. Each escape sequence starts with a \ character. The next character tells us exactly which of the Unicode characters to create. Locate the character that’s needed. Get the name or the number. The numbers are always given as hexadecimal, base 16. Websites describing Unicode often write the character as U+2680. The name might be DIEFACE-1. Use \unnnn with up to a four-digit number, nnnn. Or, use \N{name} with the spelled-out name. If the number is more than four digits, use \Unnnnnnnn with the number padded out to exactly eight digits:
>>> ’You Rolled \u2680’ ’You Rolled ’ >>> ’You drew \U0001F000’ ’You drew ’ >>> ’Discard \N{MAHJONG TILE RED DRAGON}’ ’Discard ’Yes, we can include a wide variety of characters in Python output. To place a \ in the string without the following characters being part of an escape sequence, we need to use \\. For example, we might need this for Windows file paths.
1.6.3 How it works...
Python uses Unicode internally. The 128 or so characters we can type directly using the keyboard all have handy internal Unicode numbers.
When we write:
’HELLO’Python treats it as shorthand for this:
’\u0048\u0045\u004c\u004c\u004f’Once we get beyond the characters on our keyboards, the remaining thousands of characters are identified only by their number.
When the string is being compiled by Python, \uxxxx, \Uxxxxxxxx, and \N{name} are all replaced by the proper Unicode character. If we have something syntactically wrong—for example, \N{name with no closing }—we’ll get an immediate error from Python’s internal syntax checking.
Regular expressions use a lot of \ characters and that we specifically do not want Python’s normal compiler to touch them; we used the r’ prefix on a regular expression string to prevent \ from being treated as an escape and possibly converted into something else. To use the full domain of Unicode characters, we cannot avoid using \ as an escape.
What if we need to use Unicode in a regular expression? We’ll need to use \\ all over the place in the regular expression. We might see something like this: ’\\w+[\u2680\u2681\u2682\u2683\u2684\u2685]\\d+’.
We couldn’t use the r’ prefix on the string because we needed to have the Unicode escapes processed. This forced us to use \\ for elements of the regular expression. We used \uxxxx for the Unicode characters that are part of the pattern. Python’s internal compiler will replace \uxxxx with Unicode characters and \\w will become the required \w internally.
When we look at a string at the >>> prompt, Python will display the string in its canonical form. Python prefers to display strings with ’ as a delimiter, using " when the string contains a ’. We can use either ’ or " for a string delimiter when writing code. Python doesn’t generally display raw strings; instead, it puts all of the necessary escape sequences back into the string:
>>> r"\w+" ’\\w+’We provided a string in raw form. Python displayed it in canonical form.
1.6.4 See also
In the Encoding strings – creating ASCII and UTF-8 bytes and the Decoding bytes – how to get proper characters from some bytes recipes, we’ll look at how Unicode characters are converted into sequences of bytes so we can write them to a file. We’ll look at how bytes from a file (or downloaded from a website) are turned into Unicode characters so they can be processed.
If you’re interested in history, you can read up on ASCII and EBCDIC and other old-fashioned character codes here: http://www.unicode.org/charts/.
Our computer files are bytes. When we upload or download from the internet, the communication works in bytes. A byte only has 256 distinct values. Our Python characters are Unicode. There are a lot more than 256 Unicode characters.
How do we map Unicode characters to bytes to write to a file or for transmission?
1.7.1 Getting ready
Historically, a character occupied 1 byte. Python leverages the old ASCII encoding scheme for bytes; this sometimes leads to confusion between bytes and text strings of Unicode characters.
Unicode characters are encoded into sequences of bytes. There are a number of standardized encodings and a number of non-standard encodings.
Plus, there also are some encodings that only work for a small subset of Unicode characters. We try to avoid these, but there are some situations where we’ll need to use a subset encoding scheme.
Unless we have a really good reason not to, we almost always use UTF-8 encoding for Unicode characters. Its main advantage is that it’s a compact representation of the Latin alphabet, which is used for English and a number of European languages.
Sometimes, an internet protocol requires ASCII characters. This is a special case that requires some care because the ASCII encoding can only handle a small subset of Unicode characters.
1.7.2 How to do it...
Python will generally use our OS’s default encoding for files and internet traffic. The details are unique to each OS:
We can make a general setting using the PYTHONIOENCODING environment variable. We set this outside of Python to ensure that a particular encoding is used everywhere. When using Linux or macOS, use the shell’s export statement to set the environment variable. For Windows, use the set command, or the PowerShell Set-Item cmdlet. For Linux, it looks like this:
(cookbook3) % export PYTHONIOENCODING=UTF-8Run Python:
(cookbook3) % pythonWe sometimes need to make specific settings when we open a file inside our script. We’ll return to this topic in Chapter11. Open the file with a given encoding. Read or write Unicode characters to the file:
>>> with open(’some_file.txt’, ’w’, encoding=’utf-8’) as output: ... print(’You drew \U0001F000’, file=output) >>> with open(’some_file.txt’, ’r’, encoding=’utf-8’) as input: ... text = input.read() >>> text ’You drew ’
We can also manually encode characters, in the rare case that we need to open a file in bytes mode; if we use a mode of wb, we’ll also need to use manual encoding of each string:
>>> string_bytes = ’You drew \U0001F000’.encode(’utf-8’) >>> string_bytes b’You drew \xf0\x9f\x80\x80’We can see that a sequence of bytes (\xf0\x9f\x80\x80) was used to encode a single Unicode character, U+1F000, 
.
1.7.3 How it works...
Unicode defines a number of encoding schemes. While UTF-8 is the most popular, there is also UTF-16 and UTF-32. The number is the typical number of bits per character. A file with 1,000 characters encoded in UTF-32 would be 4,000 8-bit bytes. A file with 1,000 characters encoded in UTF-8 could be as few as 1,000 bytes, depending on the exact mix of characters. In UTF-8 encoding, characters with Unicode numbers above U+007F require multiple bytes.
Various OSes have their own coding schemes. macOS files can be encoded in Mac Roman or Latin-1. Windows files might use CP1252 encoding.
The point with all of these schemes is to have a sequence of bytes that can be mapped to a Unicode character and—going the other way—a way to map each Unicode character to one or more bytes. Ideally, all of the Unicode characters are accounted for. Pragmatically, some of these coding schemes are incomplete.
The historical form of ASCII encoding can only represent about 100 of the Unicode characters as bytes. It’s easy to create a string that cannot be encoded using the ASCII scheme.
Here’s what the error looks like:
>>> ’You drew \U0001F000’.encode(’ascii’) Traceback (most recent call last): ... UnicodeEncodeError: ’ascii’ codec can’t encode character ’\U0001f000’ in position 9: ordinal not in range(128We may see this kind of error when we accidentally open a file with an encoding that’s not the widely used standard of UTF-8. When we see this kind of error, we’ll need to change our processing to select the encoding actually used to create the file. It’s almost impossible to guess what encoding was used, so some research may be required to locate metadata about the file that states the encoding.
Bytes are often displayed using printable characters. We’ll see b’hello’ as shorthand for a five-byte value. The letters are chosen using the old ASCII encoding scheme, where byte values from 0x20 to 0x7F will be shown as characters, and outside this range, more complex-looking escapes will be used.
This use of characters to represent byte values can be confusing. The prefix of b’ is our hint that we’re looking at bytes, not proper Unicode characters.
1.7.4 See also
There are a number of ways to build strings of data. See the Building complicated strings with f-strings and the Building complicated strings from lists of strings recipes for examples of creating complex strings. The idea is that we might have an application that builds a complex string, and then we encode it into bytes.
For more information on UTF-8 encoding, see https://en.wikipedia.org/wiki/UTF-8.
For general information on Unicode encodings, see http://unicode.org/faq/utf_bom.html.
How can we work with files that aren’t properly encoded? What do we do with files written in ASCII encoding?
A download from the internet is almost always in bytes—not characters. How do we decode the characters from that stream of bytes?
Also, when we use the subprocess module, the results of an OS command are in bytes. How can we recover proper characters?
Much of this is also relevant to the material in Chapter11. We’ve included this recipe here because it’s the inverse of the previous recipe, Encoding strings – creating ASCII and UTF-8 bytes.
1.8.1 Getting ready
Let’s say we’re interested in offshore marine weather forecasts. Perhaps this is because we are departing the Chesapeake Bay for the Caribbean.
Are there any special warnings coming from the National Weather Services office in Wakefield, Virginia?
Here’s the link: https://forecast.weather.gov/product.php?site=AKQ&product=SMW&issuedby=AKQ.
We can download this with Python’s urllib module:
>>> import urllib.request >>> warnings_uri = ( ... ’https://forecast.weather.gov/’ ... ’product.php?site=AKQ&product=SMW&issuedby=AKQ’ ... ) >>> with urllib.request.urlopen(warnings_uri) as source: ... forecast_text = source.read()Note that we’ve enclosed the URI string in () and broken it into two separate string literals. Python will concatenate these two adjacent literals into a single string. We’ll look at this in some depth in Chapter2.
As an alterative, we can use programs like curl or wget to get this. At the OS Terminal prompt, we might run the following (long) command:
(cookbook3) % curl ’https://forecast.weather.gov/product.php?site=AKQ&product=SMW&issuedby=AKQ’ -o AKQ.htmlTypesetting this book tends to break the command onto many lines. It’s really one very long line.
The code repository includes a sample file, ch01/TextProductsforSMWIssued byAKQ.html.
The forecast_text value is a stream of bytes. It’s not a proper string. We can tell because it starts like this:
>>> forecast_text[:80] b’<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/x’The data goes on for a while, providing details from the web page. Because the displayed value starts with b’, it’s bytes, not proper Unicode characters. It was probably encoded with UTF-8, which means some characters could have weird-looking \xnn escape sequences instead of proper characters. We want to have the proper characters.
While this data has many easy-to-read characters, the b’ prefix shows that it’s a collection of byte values, not proper text. Generally, a bytes object behaves somewhat like a string object. Sometimes, we can work with bytes directly. Most of the time, we’ll want to decode the bytes and create proper Unicode characters from them.
1.8.2 How to do it...
Determine the coding scheme if possible. In order to decode bytes to create proper Unicode characters, we need to know what encoding scheme was used. When we read XML documents, there’s a big hint provided within the document:
<?xml version="1.0" encoding="UTF-8"?>When browsing web pages, there’s often a header containing this information:
Content-Type: text/html; charset=ISO-8859-4Sometimes, an HTML page may include this as part of the header:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">In other cases, we’re left to guess. In the case of US weather data, a good first guess is UTF-8. Another good guess is ISO-8859-1. In some cases, the guess will depend on the language.
The codecs — Codec registry and base classes section of the Python Standard Library lists the standard encodings available. Decode the data:
>>> document = forecast_text.decode("UTF-8") >>> document[:80] ’<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/x’The b’ prefix is no longer used to show that these are bytes. We’ve created a proper string of Unicode characters from the stream of bytes.
If this step fails with an exception, we guessed wrong about the encoding. We need to try another encoding in order to parse the resulting document.
Since this is an HTML document, we should use Beautiful Soup to extract the data. See http://www.crummy.com/software/BeautifulSoup/.
We can, however, extract one nugget of information from this document without completely parsing the HTML:
>>> import re >>> content_pattern = re.compile(r"// CONTENT STARTS(.*?)// CONTENT ENDS", re.MULTILINE | re.DOTALL) >>> content_pattern.search(document) <re.Match object; span=(8530, 9113), match=’// CONTENT STARTS HERE -->\n\n<span style="font-s>This tells us what we need to know: there are no warnings at this time. This doesn’t mean smooth sailing, but it does mean that there aren’t any major weather systems that could cause catastrophes.
1.8.3 How it works...
See the Encoding strings – creating ASCII and UTF-8 bytes recipe for more information on Unicode and the different ways that Unicode characters can be encoded into streams of bytes.
At the foundation of the OS, files and network connections are built up from bytes. It’s our software that decodes the bytes to discover the content. It might be characters, images, or sounds. In some cases, the default assumptions are wrong and we need to do our own decoding.
1.8.4 See also
Once we’ve recovered the string data, we have a number of ways of parsing or rewriting it. See the String parsing with regular expressions recipe for examples of parsing a complex string.
For more information on encodings, see https://en.wikipedia.org/wiki/UTF-8 and http://unicode.org/faq/utf_bom.html.
What’s the best way to represent simple (x,y) and (r,g,b) groups of values? How can we keep things that are pairs, such as latitude and longitude, together?
1.9.1 Getting ready
In the String parsing with regular expressions recipe, we skipped over an interesting data structure.
We had data that looked like this:
>>> ingredient = "Kumquat: 2 cups"We parsed this into meaningful data using a regular expression, like this:
>>> import re >>> ingredient_pattern = re.compile(r’(?P<ingredient>\w+):\s+(?P<amount>\d+)\s+(?P<unit>\w+)’) >>> match = ingredient_pattern.match(ingredient) >>> match.groups() (’Kumquat’, ’2’, ’cups’)The result is a tuple object with three pieces of data. There are lots of places where this kind of grouped data can come in handy.
1.9.2 How to do it...
We’ll look at two aspects to this: putting things into tuples and getting things out of tuples.
Creating tuples
There are lots of places where Python creates tuples of data for us. In the Getting ready section of the String parsing with regular expressions recipe, we showed you how a regular expression match object will create a tuple of text that was parsed from a string.
We can create our own tuples, too. Here are the steps:
Enclose the data in ().
Separate the items with ,.
>>> from fractions import Fraction >>> my_data = (’Rice’, Fraction(1/4), ’cups’)
There’s an important special case for the one-tuple, or singleton. We have to include the , even when there’s only one item in the tuple:
>>> one_tuple = (’item’, ) >>> len(one_tuple) 1The () characters aren’t always required. There are a few times where we can omit them. It’s not a good idea to omit them.
It’s the comma that creates a tuple of values. This means we can see funny things when we have an extra comma:
>>> 355, (355,)The comma after 355 turns the value into a singleton tuple.
We can also create a tuple by conversion from another sequence. For example, tuple([355]) creates a singleton tuple from a singleton list.
Extracting items from a tuple
The idea of a tuple is to be a container with a number of items that’s fixed by the problem domain: for example, for (red, green, blue) color numbers, the number of items is always three.
In our example, we’ve got an ingredient, and amount, and units. This must be a three-item collection. We can look at the individual items in two ways:
By index position; that is, positions are numbered starting with zero from the left:
>>> my_data[1] Fraction(1, 4)Using multiple assignment:
>>> ingredient, amount, unit = my_data >>> ingredient ’Rice’ >>> unit ’cups’
Tuples—like strings—are immutable. We can’t change the individual items inside a tuple. We use tuples when we want to keep the data together.
1.9.3 How it works...
Tuples are one example of the more general Sequence class. We can do a few things with sequences.
Here’s an example tuple that we can work with:
>>> t = (’Kumquat’, ’2’, ’cups’)Here are some operations we can perform on this tuple:
How many items in t?
>>> len(t) 3How many times does a particular value appear in t?
>>> t.count(’2’) 1Which position has a particular value?
>>> t.index(’cups’) 2 >>> t[2] ’cups’When an item doesn’t exist, we’ll get an exception:
>>> t.index(’Rice’) Traceback (most recent call last): ... ValueError: tuple.index(x): x not in tupleDoes a particular value exist?
>>> ’Rice’ in t False
1.9.4 There’s more...
A tuple, like a string, is a sequence of items. In the case of a string, it’s a sequence of characters. In the case of a tuple, it’s a sequence of many things. Because they’re both sequences, they have some common features. We’ve noted that we can pluck out individual items by their index position. We can use the index() method to locate the position of an item.
The similarities end there. A string has many methods it can use to create a new string that’s a transformation of a string, plus methods to parse strings, plus methods to determine the content of the strings. A tuple doesn’t have any of these bonus features. It’s—perhaps—the simplest possible data structure.
1.9.5 See also
We looked at one other sequence, the list, in the Building complicated strings from lists of strings recipe.
We’ll also look at sequences in Chapter4.
When we worked with tuples, we had to remember the positions as numbers. When we use a (r,g,b) tuple to represent a color, can we use ”red” instead of zero, ”green” instead of 1, and ”blue” instead of 2?
1.10.1 Getting ready
Let’s continue looking at items in recipes. The regular expression for parsing the string had three attributes: ingredient, amount, and unit. We used the following pattern with names for the various substrings:
r’(?P<ingredient>\w+):\s+(?P<amount>\d+)\s+(?P<unit>\w+)’)The resulting data tuple looked like this:
>>> item = match.groups() >>> item (’Kumquat’, ’2’, ’cups’)While the matching between ingredient, amount, and unit is pretty clear, using something like the following isn’t ideal. What does 1 mean? Is it really the quantity?
>>> from fractions import Fraction >>> Fraction(item[1]) Fraction(2, 1)We want to define tuples with names, as well as positions.
1.10.2 How to do it...
We’ll use the NamedTuple class definition from the typing package:
>>> from typing import NamedTupleWith this base class definition, we can define our own unique tuples, with names for the items:
>>> class Ingredient(NamedTuple): ... ingredient: str ... amount: str ... unit: strNow, we can create an instance of this unique kind of tuple by using the classname:
>>> item_2 = Ingredient(’Kumquat’, ’2’, ’cups’)When we want a value from the tuple, we can use a name instead of the position:
>>> Fraction(item_2.amount) Fraction(2, 1) >>> f"Use {item_2.amount} {item_2.unit} fresh {item_2.ingredient}" ’Use 2 cups fresh Kumquat’
1.10.3 How it works...
The NamedTuple class definition introduces a core concept from Chapter7. We’ve extended the base class definition to add unique features for our application. In this case, we’ve named the three attributes each Ingredient tuple must contain.
Because a subclass of NamedTuple class is a tuple, the order of the attribute names is fixed. We can use a reference like the expression item_2[0] as well as the expression item_2.ingredient. Both names refer to the item in index 0 of the tuple, item_2.
The core tuple types can be called ”anonymous tuples” or maybe ”index-only tuples.” This can help to distinguish them from the more sophisticated ”named tuples” introduced through the typing module.
Tuples are very useful as tiny containers of closely related data. Using the NamedTuple class definition makes them even easier to work with.
1.10.4 There’s more...
We can have a mixed collection of values in a tuple or a named tuple. We need to perform conversion before we can build the tuple. It’s important to remember that a tuple cannot ever be changed. It’s an immutable object, similar in many ways to the way strings and numbers are immutable.
For example, we might want to work with amounts that are exact fractions. Here’s a more sophisticated definition:
>>> from typing import NamedTuple >>> from fractions import Fraction >>> class IngredientF(NamedTuple): ... ingredient: str ... amount: Fraction ... unit: strThese objects require some care to create. If we’re using a bunch of strings, we can’t simply build this object from three string values; we need to convert the amount into a Fraction instance. Here’s an example of creating an item using a Fraction conversion:
>>> item_3 = IngredientF(’Kumquat’, Fraction(’2’), ’cups’)This tuple has a more useful value for the amount of each ingredient. We can now do mathematical operations on the amounts:
>>> f’{item_3.ingredient} doubled: {item_3.amount * 2}’ ’Kumquat doubled: 4’It’s very handy to explicitly state the data type within the NamedTuple class definition. It turns out Python doesn’t use the type information directly. Other tools, for example, mypy, can check the type hints in a NamedTuple against the operations in the rest of the code to be sure they agree.
1.10.5 See also
We’ll look at class definitions in Chapter7.
Join our community Discord space
Join our Python Discord workspace to discuss and find out more about the book: https://packt.link/dHrHU
