modern python cookbook - programmer books · 2019. 6. 12. · modern python cookbook the latest in...

Modern Python Cookbook

The latest in modern Python recipes for the busy modernprogrammer

Steven F. Lott

BIRMINGHAM - MUMBAI

Modern Python Cookbook

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, ortransmitted in any form or by any means, without the prior written permission of thepublisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of theinformation presented. However, the information contained in this book is sold withoutwarranty, either express or implied. Neither the author, nor Packt Publishing, and itsdealers and distributors will be held liable for any damages caused or alleged to be causeddirectly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of thecompanies and products mentioned in this book by the appropriate use of capitals.However, Packt Publishing cannot guarantee the accuracy of this information.

First published: November 2016

Production reference: 1211116

Published by Packt Publishing Ltd.Livery Place35 Livery StreetBirmingham B3 2PB, UK.

ISBN 978-1-78646-925-0

www.packtpub.com

http://www.packtpub.com

Credits

Author Steven F. Lott

Copy Editor Safis Editing

Reviewers Sanjeev Kumar JaiswalDr. Vahid Mirjalili

Project Coordinator Suzanne Coutinho

Commissioning Editor Kunal Parikh

Proofreader Safis Editing

Acquisition Editor Sonali Vernekar

Indexer Tejal Daruwale Soni

Content Development Editor Zeeyan Pinheiro

Graphics Kirk D'Penha

Technical Editors Pratish ShettyAbhishek Sharma

Production Coordinator Aparna Bhagat

About the AuthorSteven F. Lott has been programming since the 70s, when computers were large, expensive,and rare. As a contract software developer and architect, he has worked on hundreds ofprojects, from very small to very large. He's been using Python to solve business problemsfor over 10 years.

He’s currently leveraging Python to implement microservices and ETL pipelines. His othertitles with Packt Publishing include Python Essentials, Mastering Object-Oriented Python,Functional Python Programming, and Python for Secret Agents.

Steven is currently a technomad who lives in various places on the east coast of the U.S. Histechnology blog is h t t p ://s l o t t - s o f t w a r e a r c h i t e c t . b l o g s p o t . c o m and his LinkedInaddress is h t t p s ://w w w . l i n k e d i n . c o m /i n /s t e v e n - l o t t - 029835.

http://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttp://slott-softwarearchitect.blogspot.comhttps://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835https://www.linkedin.com/in/steven-lott-029835

About the ReviewersSanjeev Jaiswal is a computer graduate with 7 years of industrial experience in webdevelopment and cyber security. He basically uses Perl, Python, and GNU/Linux for hisday-to-day activities. He is currently working on projects involving penetration testing,source code review, and security design and implementations.

He is very much interested in web and cloud security. You can follow him on Twitter at@aliencoders and on GitHub at h t t p s ://g i t h u b . c o m /j a s s i c s .

He has written Instant PageSpeed Optimization and co-authored Learning Django WebDevelopment for Packt Publishing. He has reviewed more than 5 books for Packt Publishingand looks forward to authoring or reviewing more books for Packt Publishing and otherpublishers.

Vahid Mirjalili is a software engineer and data scientist, currently working towards hisPhD study in Computer Science at Michigan State University. His research at the i-PRoBE(integrated pattern recognition and biometrics) lab involves attribute classification of faceimages from large image datasets.

Furthermore, he teaches Python programming as well as computing concepts for dataanalysis and databases. Owing to his specialty in data mining, he is very interested inpredictive modeling and getting insights from data. He is also a Python developer and likesto contribute to the open source community.

Moreover, he enjoys making tutorials for different directions of data science and computeralgorithms, which can be found in his GitHub repository at h t t p ://g i t h u b . c o m /m i r j a l i l/D a t a S c i e n c e .

https://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttps://github.com/jassicshttp://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.http://github.com/mirjalil/DataScience.

www.PacktPub.comFor support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF andePub files available? You can upgrade to the eBook version at www.PacktPub.com and as aprint book customer, you are entitled to a discount on the eBook copy. Get in touch with usat [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for arange of free newsletters and receive exclusive discounts and offers on Packt books andeBooks.

h t t p s ://w w w . p a c k t p u b . c o m /m a p t

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packtbooks and video courses, as well as industry-leading tools to help you plan your personaldevelopment and advance your career.

Why subscribe?Fully searchable across every book published by PacktCopy and paste, print, and bookmark contentOn demand and accessible via a web browser

http://www.PacktPub.comhttp://www.PacktPub.comhttp://www.PacktPub.comhttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapthttps://www.packtpub.com/mapt

Table of ContentsPreface 1Chapter 1: Numbers, Strings, and Tuples 6

Introduction 6Creating meaningful names and using variables 7

Getting ready 7How to do it… 8

Choosing names wisely 8Assigning names to objects 10

How it works… 11There's more… 12See also 12

Working with large and small integers 13Getting ready 13How to do it… 13How it works… 14There's more… 16See also 17

Choosing between float, decimal, and fraction 17Getting ready 17How to do it… 18

Doing currency calculations 18Fraction calculations 19Floating-point approximations 20Converting numbers from one type to another 21


Choosing between true division and floor division 25Getting ready 25How to do it… 25

Doing floor division 26Doing true division 26Rational fraction calculations 27

How it works… 27See also 28

Rewriting an immutable string 28

[ ii ]


Slicing a piece of a string 29Updating a string with a replacement 30Making a string all lowercase 31Removing extra punctuation marks 31


String parsing with regular expressions 34Getting ready 34How to do it… 34How it works… 36There's more… 37See also 37

Building complex strings with “template”.format() 38Getting ready 38How to do it… 38How it works… 40There's more… 42See also 42

Building complex strings from lists of characters 43Getting ready 43How to do it… 44How it works… 44There's more 45See also 46

Using the Unicode characters that aren't on our keyboards 46Getting ready 46How to do it… 47How it works… 47See also 48

Encoding strings – creating ASCII and UTF-8 bytes 49Getting ready 49How to do it… 49How it works… 50See also 51

Decoding bytes – how to get proper characters from some bytes 52Getting ready 52How to do it.. 53

[ iii ]

How it works… 54See also 55

Using tuples of items 55Getting ready 55How to do it… 55

Creating tuples 56Extracting items from a tuple 56

How it works… 57There's more 58See also… 58

Chapter 2: Statements and Syntax 59Introduction 59Writing Python script and module files – syntax basics 60

Getting ready 60How to do it… 61How it works… 63There's more… 64See also 65

Writing long lines of code 65Getting ready 66How to do it… 66

Using backslash to break a long statement into logical lines 67Using the () characters to break a long statement into sensible pieces 67Using string literal concatenation 68Assigning intermediate results to separate variables 69


Including descriptions and documentation 71Getting ready 71How to do it… 72

Writing docstrings for scripts 72Writing docstrings for library modules 74


Writing better RST markup in docstrings 76Getting ready 76How to do it… 77How it works… 78

[ iv ]

There's more… 79Using directives 81Using inline markup 81

See also 82Designing complex if…elif chains 82


Designing a while statement which terminates properly 87Getting ready 87How to do it… 88How it works… 91See also 92

Avoiding a potential problem with break statements 93Getting ready 93How to do it… 94How it works… 95There's more… 96See also 96

Leveraging the exception matching rules 97Getting ready 98How to do it… 99How it works… 100There's more… 100See also 101

Avoiding a potential problem with an except: clause 102Getting ready 102How to do it… 102How it works… 103See also 103

Chaining exceptions with the raise from statement 104Getting ready 104How to do it… 104How it works… 105There's more… 106See also 107

Managing a context using the with statement 107Getting ready 107

[ v ]

How to do it… 108How it works… 109There's more… 110See also 110

Chapter 3: Function Definitions 111Introduction 111Designing functions with optional parameters 112


Particular to General Design 113General to Particular design 115


Using super flexible keyword parameters 118Getting ready 119How to do it… 119How it works… 121There's more… 121See also 122

Forcing keyword-only arguments with the * separator 122Getting ready 123How to do it… 124How it works… 125There's more… 125See also 126

Writing explicit types on function parameters 126Getting ready 127How to do it… 128How it works… 129There's more… 131See also 132

Picking an order for parameters based on partial functions 132Getting ready 132How to do it… 134

Wrapping a function 134Creating a partial function with keyword parameters 135Creating a partial function with positional parameters 135

How it works… 136There's more… 137

[ vi ]

See also 137Writing clear documentation strings with RST markup 137


Designing recursive functions around Python's stack limits 142Getting ready 143How to do it… 144How it works… 145There's more… 146See also 148

Writing reusable scripts with the script library switch 148Getting ready 148How to do it… 149How it works… 151There's more… 151See also 152

Chapter 4: Built-in Data Structures – list, set, dict 153Introduction 153Choosing a data structure 154


Building lists – literals, appending, and comprehensions 159Getting ready 159How to do it… 160

Building a list with the append() method 160Writing a list comprehension 161Using the list function on a generator expression 162


Other ways to extend a list 164See also 165

Slicing and dicing a list 165Getting ready 165

[ vii ]


Deleting from a list – deleting, removing, popping, and filtering 170Getting ready 170How to do it… 171

Deleting items from a list 171The remove() method 172The pop() method 173The filter() function 173


Reversing a copy of a list 176Getting ready 176How to do it… 178How it works… 179See also 179

Using set methods and operators 179Getting ready 181How to do it… 182How it works… 183There's more… 184See also 185

Removing items from a set – remove(), pop(), and difference 186Getting ready 186How to do it… 187How it works… 188There's more… 188See also 189

Creating dictionaries – inserting and updating 189Getting ready 190How to do it… 191How it works… 192There's more… 192See also 194

Removing from dictionaries – the pop() method and the del statement 194Getting ready 195How to do it… 196

[ viii ]


Controlling the order of dict keys 198Getting ready 198How to do it… 199How it works… 200There's more… 200See also 201

Handling dictionaries and sets in doctest examples 201Getting ready 201How to do it… 202How it works… 203There's more… 203

Understanding variables, references, and assignment 204How to do it… 204How it works… 205There's more… 206See also 207

Making shallow and deep copies of objects 207Getting ready 208How to do it… 209How it works… 210See also 211

Avoiding mutable default values for function parameters 211Getting ready 212How to do it… 213How it works… 214There's more… 215See also 216

Chapter 5: User Inputs and Outputs 217Introduction 217Using features of the print() function 218


Using input() and getpass() for user input 223

[ ix ]

Getting ready 224How to do it… 225How it works… 227There's more… 228

Input string parsing 228Interaction via the cmd module 229

See also 229Debugging with “format”.format_map(vars()) 230


Using argparse to get command-line input 232Getting ready 233How to do it… 235How it works… 237There's more… 238See also 240

Using cmd for creating command-line applications 240Getting ready 242How to do it… 243How it works… 246There's more… 247See also 248

Using the OS environment settings 248Getting ready 249How to do it… 250How it works… 251There's more… 252See also 252

Chapter 6: Basics of Classes and Objects 253Introduction 253Using a class to encapsulate data and processing 254


[ x ]

Designing classes with lots of processing 260Getting ready 260How to do it… 263How it works… 265There's more… 266See also 268

Designing classes with little unique processing 268Getting ready 268How to do it… 269

Stateless objects 269Stateful objects with a new class 270Stateful objects using an existing class 271


Optimizing small objects with __slots__ 273Getting ready 273How to do it… 274How it works… 276There's more… 276See also 277

Using more sophisticated collections 277Getting ready 277How to do it… 278How it works… 280There's more… 281See also 282

Extending a collection – a list that does statistics 282Getting ready 282How to do it… 282How it works… 284There's more… 285See also 286

Using properties for lazy attributes 286Getting ready… 286How to do it… 288How it works… 290There's more… 291See also… 291

Using settable properties to update eager attributes 292

[ xi ]


Initialization 296Calculation 297

See also 298Chapter 7: More Advanced Class Design 299

Introduction 299Choosing between inheritance and extension – the is-a question 300


Wrapping – aggregation and composition 303Extending – inheritance 305


Separating concerns via multiple inheritance 309Getting ready 309How to do it… 310How it works… 312There's more… 313See also 315

Leveraging Python's duck typing 315Getting ready 316How to do it… 316How it works… 318There's more… 319See also 319

Managing global and singleton objects 320Getting ready 320How to do it… 321

Module global variable 321Class-level static variable 323


Using more complex structures – maps of lists 325Getting ready 326How to do it… 326How it works… 327

[ xii ]

There's more… 328See also 328

Creating a class that has orderable objects 329Getting ready 329How to do it… 330How it works… 332There's more… 333See also 334

Defining an ordered collection 335Getting ready 336How to do it… 337How it works… 339There's more… 340See also 342

Deleting from a list of mappings 342Getting ready 343How to do it… 345How it works… 346There's more… 347See also 348

Chapter 8: Functional and Reactive Programming Features 349Introduction 349Writing generator functions with the yield statement 351


Using stacked generator expressions 359Getting ready 360How to do it… 361How it works… 364There's more… 366

Namespace instead of list 367See also 369

Applying transformations to a collection 369Getting ready… 370How to do it… 372How it works… 373

[ xiii ]

There's more… 373See also… 374

Picking a subset – three ways to filter 374Getting ready… 375How to do it… 377How it works… 378There's more… 379See also… 380

Summarizing a collection – how to reduce 380Getting ready 381How to do it… 383How it works… 384There's more… 384

Maxima and minima 385Potential for abuse 386

Combining map and reduce transformations 386Getting ready 387How to do it… 388How it works… 392There's more… 393See also 393

Implementing “there exists” processing 394Getting ready 394How to do it… 395How it works… 396There's more… 397

The itertools module 398Creating a partial function 399


Using functools.partial() 403Creating a lambda object 403


Simplifying complex algorithms with immutable data structures 406Getting ready 407How to do it… 410How it works… 411There's more… 412

Writing recursive generator functions with the yield from statement 413

[ xiv ]


Chapter 9: Input/Output, Physical Format, and Logical Layout 420Introduction 420Using pathlib to work with filenames 422


Making the output filename by changing the input suffix 424Making a number of sibling output files with distinct names 425Creating a directory and a number of files 426Comparing file dates to see which is newer 427Removing a file 428Finding all files that match a given pattern 429


Reading and writing files with context managers 432Getting ready 432How to do it… 434How it works… 435There's more… 436See also 436

Replacing a file while preserving the previous version 437Getting ready 437How to do it… 438How it works… 440There's more… 440See also 441

Reading delimited files with the CSV module 441Getting ready 442How to do it… 443How it works… 444There's more… 445See also 447

Reading complex formats using regular expressions 447Getting ready 447How to do it… 449

[ xv ]

Defining the parse function 449Using the parse function 450


Reading JSON documents 453Getting ready 454How to do it… 456How it works… 457There's more… 457

Serializing a complex data structure 458Deserializing a complex data structure 460

See also 461Reading XML documents 461


Reading HTML documents 468Getting ready 469How to do it… 471How it works… 472There's more… 474See also 475

Upgrading CSV from DictReader to namedtuple reader 475Getting ready 475How to do it… 476How it works… 478There's more… 478See also 480

Upgrading CSV from a DictReader to a namespace reader 481Getting ready 481How to do it… 482How it works… 483There's more… 483See also 485

Using multiple contexts for reading and writing files 485Getting ready 485How to do it… 487

[ xvi ]


Chapter 10: Statistical Programming and Linear Regression 492Introduction 492Using the built-in statistics library 493


Average of values in a Counter 502Getting ready 502How to do it… 504How it works… 505There's more… 505See also 507

Computing the coefficient of a correlation 507Getting ready 508How to do it… 509How it works… 511There's more… 512

Computing regression parameters 512Getting ready 513How to do it… 514How it works… 516There's more… 516

Computing an autocorrelation 517Getting ready 517How to do it… 521How it works… 523There's more… 523

Long-term model 524See also 525

Confirming that the data is random – the null hypothesis 525Getting ready 525How to do it… 527How it works… 530There's more… 531See also 532

[ xvii ]

Locating outliers 532Getting ready 533How to do it… 534How it works… 536There's more… 537See also 539

Analyzing many variables in one pass 539Getting ready 540How to do it… 542How it works… 544There's more… 544

Using map() 547See also 547

Chapter 11: Testing 548Introduction 548Using docstrings for testing 549


Writing examples for stateless functions 552Writing examples for stateful objects 552


Testing functions that raise exceptions 557Getting ready 557How to do it… 558How it works… 559There's more… 560See also 561

Handling common doctest issues 561Getting ready 562How to do it… 564

Writing doctest examples for mapping or set values 564Writing doctest examples for floating-point values 565


Creating separate test modules and packages 568Getting ready 568How to do it… 570

[ xviii ]


Some other assertions 574Separate tests directory 575

See also 576Combining unittest and doctest tests 576


Testing things that involve dates or times 580Getting ready 581How to do it… 582How it works… 584There's more… 585See also 587

Testing things that involve randomness 587Getting ready 587How to do it… 588How it works… 589There's more… 590See also 591

Mocking external resources 591Getting ready 592

Creating an entry document in the entrylog collection 593Seeing a typical response 595Client class for database access 596

How to do it… 598How it works… 601

Creating a context manager 601Creating a dynamic, stateful test 602Mocking a complex object 603Using the load_tests protocol 604


Chapter 12: Web Services 606Introduction 606Implementing web services with WSGI 609

Getting ready 609

[ xix ]


Using the Flask framework for RESTful APIs 622Getting ready 622How to do it… 623How it works… 626There's more… 627See also 629

Parsing the query string in a request 629Getting ready 631How to do it… 631How it works… 633There's more… 634See also 636

Making REST requests with urllib 636Getting ready 636How to do it… 638How it works… 641There's more… 642

The OpenAPI (Swagger) specification 643Adding Swagger to the server 643

See also 645Parsing the URL path 645


Server 647Client 650

How it works… 652Deck slicing 653Client side 654

There's more… 655Providing a Swagger specification 655Using a Swagger specification 658

See also 658Parsing a JSON request 658


Swagger specification 660Server 663

[ xx ]

Client 664How it works… 666There's more… 667

Location header 668Additional resources 668Query for a specific player 669Exception handling 670

See also 671Implementing authentication for web services 671

Getting ready 673Configuring SSL 673Users and credentials 674Flask view function decorator 676

How to do it… 676Defining the User class 677Defining a view decorator 680Creating the server 681Creating an example client 683


Creating a command-line interface 689Building the Authentication header 689

See also 690Chapter 13: Application Integration 691

Introduction 691Finding configuration files 692

Getting ready 693Why so many choices? 694


Using YAML for configuration files 702Getting ready 702How to do it… 703How it works… 704There's more… 706See also 709

Using Python for configuration files 709Getting ready 710How to do it… 711

[ xxi ]


Using class-as-namespace for configuration 713Getting ready 714How to do it… 715How it works… 716There's more… 716

Configuration representation 718See also 719

Designing scripts for composition 720Getting ready 720How to do it… 722How it works… 724There's more… 726

Designing as a class hierarchy 727See also 728

Using logging for control and audit output 729Getting ready 729How to do it… 733How it works… 737There's more… 739

Combining two applications into one 740Getting ready 740How to do it… 742How it works… 743There's more… 744

Refactoring 744Concurrency 745Logging 747

See also 748Combining many applications using the Command design pattern 748


Managing arguments and configuration in composite applications 753Getting ready 754How to do it… 755

[ xxii ]

How it works… 757The Command design pattern 758


Wrapping and combining CLI applications 760Getting ready 760How to do it… 761How it works… 762There's more… 763

Unit test 764See also 767

Wrapping a program and checking the output 767Getting ready 767How to do it… 768How it works… 770There's more… 771See also 772

Controlling complex sequences of steps 772Getting ready 772How to do it… 773How it works… 775There's more… 776

Building conditional processing 776See also 777

Index 778

PrefacePython is the preferred choice of developers, engineers, data scientists, and hobbyistseverywhere. It is a great scripting language that can power your applications and providegreat speed, safety, and scalability. By exposing Python as a series of simple recipes, youcan gain insights into specific language features in a particular context. Having a tangiblecontext helps make the language or standard library feature easier to understand.

This book takes a recipe-based approach, where each recipe addresses specific problemsand issues.

What this book coversChapter 1, Numbers, Strings, and Tuples, will look at the different kinds of numbers, workwith strings, use tuples, and use the essential built-in types in Python. We will also exploitthe full power of the Unicode character set.

Chapter 2, Statements and Syntax, will cover some basics of creating script files first. Thenwe’ll move on to looking at some of the complex statements, including if, while, for, try,with, and raise.

Chapter 3, Function Definitions, will look at a number of function definition techniques.We’ll also look at the Python 3.5 typing module and see how we can create more formalannotations for our functions.

Chapter 4, Built-in Data Structures – list, set, dict, will look at an overview of the variousstructures that are available and what problems they solve. From there, we can look at lists,dictionaries, and sets in detail, and also look at some more advanced topics related to howPython handles references to objects.

Chapter 5, User Inputs and Outputs, will explain how to use the different features of theprint() function. We'll also look at the different functions used to provide user input.

Chapter 6, Basics of Classes and Objects, will create classes that implement a number ofstatistical formulae.

Preface

[ 2 ]

Chapter 7, More Advanced Class Design, will dive a little more deeply into Python classes.We will combine some features we have previously learned about to create moresophisticated objects.

Chapter 8, Functional and Reactive Programming Features, provides us with methods towriting small, expressive functions that perform the required data transformations. Movingahead, you will learn about the idea of reactive programming, that is, having processingrules that are evaluated when the inputs become available or change.

Chapter 9, Input/Output, Physical Format, Logical Layout, will work with different fileformats such as JSON, XML, and HTML.

Chapter 10, Statistical Programming and Linear Regression, will look at some basic statisticalcalculations that we can do with Python’s built-in libraries and data structures. We’ll look atthe questions of correlation, randomness, and the null hypothesis.

Chapter 11, Testing, will give us a detailed description of the different testing frameworksused in Python.

Chapter 12, Web Services, will look at a number of recipes for creating RESTful web servicesand also serving static or dynamic content.

Chapter 13, Application Integration, will look at ways that we can design applications thatcan be composed to create larger, more sophisticated composite applications. We’ll alsolook at the complications that can arise from composite applications and the need tocentralize some features, such as command-line parsing.

What you need for this bookAll you need to follow through the examples in this book is a computer running any recentversion of Python. While the examples all use Python 3, they can be adapted to work withPython 2 only a few changes.

Who this book is forThe book is for web developers, programmers, enterprise programmers, engineers, and bigdata scientists. If you are a beginner, Python Cookbook will get you started. If you areexperienced, it will expand your knowledge base. A basic knowledge of programmingwould help.

Preface

[ 3 ]

ConventionsIn this book, you will find a number of text styles that distinguish between different kindsof information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions,pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "We caninclude other contexts through the use of the include directive."

A block of code is set as follows:

if distance is None: distance = rate * time elif rate is None: rate = distance / time elif time is None: time = distance / rate

Any command-line input or output is written as follows:

>>> circumference_diameter_ratio = 355/113>>> target_color_name = 'FireBrick'>>> target_color_rgb = (178, 34, 34)

New terms and important words are shown in bold.

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedbackFeedback from our readers is always welcome. Let us know what you think about thisbook—what you liked or disliked. Reader feedback is important for us as it helps usdevelop titles that you will really get the most out of.

Preface

[ 4 ]

To send us general feedback, simply e-mail [email protected], and mention thebook's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing orcontributing to a book, see our author guide at www.packtpub.com/authors.

Customer supportNow that you are the proud owner of a Packt book, we have a number of things to help youto get the most from your purchase.

Downloading the example codeYou can download the example code files for this book from your account at h t t p ://w w w . pa c k t p u b . c o m . If you purchased this book elsewhere, you can visit h t t p ://w w w . p a c k t p u b . co m /s u p p o r t and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.1.Hover the mouse pointer on the SUPPORT tab at the top.2.Click on Code Downloads & Errata.3.Enter the name of the book in the Search box.4.Select the book for which you're looking to download the code files.5.Choose from the drop-down menu where you purchased this book from.6.Click on Code Download.7.

Once the file is downloaded, please make sure that you unzip or extract the folder using thelatest version of:

WinRAR / 7-Zip for WindowsZipeg / iZip / UnRarX for Mac7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at h t t p s ://g i t h u b . c o m /P a c k t P u b li s h i n g /M o d e r n - P y t h o n - C o o k b o o k . We also have other code bundles from our rich catalogof books and videos available at h t t p s ://g i t h u b . c o m /P a c k t P u b l i s h i n g /. Check them out!

mailto:[email protected]://www.packtpub.com/authorshttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.comhttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttp://www.packtpub.com/supporthttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/Modern-Python-Cookbookhttps://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/https://github.com/PacktPublishing/

Preface

[ 5 ]

ErrataAlthough we have taken every care to ensure the accuracy of our content, mistakes dohappen. If you find a mistake in one of our books—maybe a mistake in the text or thecode—we would be grateful if you could report this to us. By doing so, you can save otherreaders from frustration and help us improve subsequent versions of this book. If you findany errata, please report them by visiting h t t p ://w w w . p a c k t p u b . c o m /s u b m i t - e r r a t a ,selecting your book, clicking on the Errata Submission Form link, and entering the details ofyour errata. Once your errata are verified, your submission will be accepted and the erratawill be uploaded to our website or added to any list of existing errata under the Erratasection of that title.

To view the previously submitted errata, go to h t t p s ://w w w . p a c k t p u b . c o m /b o o k s /c o n t e nt /s u p p o r t and enter the name of the book in the search field. The required information willappear under the Errata section.

PiracyPiracy of copyrighted material on the Internet is an ongoing problem across all media. AtPackt, we take the protection of our copyright and licenses very seriously. If you comeacross any illegal copies of our works in any form on the Internet, please provide us withthe location address or website name immediately so that we can pursue a remedy.

Please contact us at [email protected] with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuablecontent.

QuestionsIf you have a problem with any aspect of this book, you can contact us [email protected], and we will do our best to address the problem.

http://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttp://www.packtpub.com/submit-erratahttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supporthttps://www.packtpub.com/books/content/supportmailto:[email protected]:[email protected]

1Numbers, Strings, and Tuples

We'll cover these recipes to introduce basic Python data types:

Creating meaningful names and using variablesWorking with large and small integersChoosing between float, decimal, and fractionChoosing between true division and floor divisionRewriting an immutable stringString parsing with regular expressionsBuilding complex strings with “template”.format()Building complex strings from lists of charactersUsing the Unicode characters that aren't on our keyboardsEncoding strings – creating ASCII and UTF-8 bytesDecoding bytes – how to get proper characters from some bytesUsing tuples of items

IntroductionThis chapter will look at some central types of Python objects. We'll look at the differentkinds of numbers, working with strings, and using tuples. We'll look at these first becausethey're the simplest kinds of data Python works with. In later chapters, we'll look at datacollections.

Most of these recipes assume a beginner's level of understanding of Python 3. We'll belooking at how we use the essential built-in types available in Python—numbers, strings,and tuples. Python has a rich variety of numbers, and two different division operators, sowe'll need to look closely at the choices available to us.

Numbers, Strings, and Tuples

[ 7 ]

When working with strings, there are several common operations that are important. We'llexplore some of the differences between bytes—as used by our OS files, and Strings—asused by Python. We'll look at how we can exploit the full power of the Unicode characterset.

In this chapter, we'll show the recipes as if we're working from the >>> prompt ininteractive Python. This is sometimes called the read-eval-print loop (REPL). In laterchapters, we'll look more closely at writing script files. The goal is to encourage interactiveexploration because it's a great way to learn the language.

Creating meaningful names and usingvariablesHow can we be sure our programs make sense? One of the core elements of makingexpressive code is to use meaningful names. But what counts as meaningful? In this recipe,we'll review some common rules for creating meaningful Python names.

We'll also look at some of Python's assignment statement variations. We can, for example,assign more than one variable in a single statement.

Getting readyThe core issue when creating a name is to ask ourselves the question what is this thing? Forsoftware, we'd like a name that's descriptive of the object being named. Clearly, a name likex is not very descriptive, it doesn't seem to refer to an actual thing.

Vague, non-descriptive names are distressingly common in some programming. It's nothelpful to others when we use them. A descriptive name helps everyone.

When naming things, it's also important to separate the problem domain—what we'rereally trying to accomplish—from the solution domain. The solution domain consists of thetechnical details of Python, OS, and Internet. Anyone who reads the code can see thesolution; it doesn't require deep explanation. The problem domain, however, can beobscured by technical details. It's our job to make the problem clearly visible. Well-chosennames will help.


[ 8 ]

How to do it…We'll look at names first. Then we'll move on to assignment.

Choosing names wiselyOn a purely technical level, Python names must begin with a letter. They can include anynumber of letters, digits, and the _ character. Python 3 is based on Unicode, so a letter is notlimited to the Latin alphabet. While the A-Z Latin alphabet is commonly used, it's notrequired.

When creating a descriptive variable, we want to create names that are both specific andarticulate the relationships among things in our programs. One widely used technique is tocreate longer names in a style that moves from particular to general.

The steps to choosing a name are as follows:

The last part of the name is a very broad summary of the thing. In a few cases,1.this may be all we need; context will supply the rest. We'll suggest some typicalbroad summary categories later.Use a prefix to narrow this name around your application or problem domain.2.If needed, put more narrow and specialized prefixes on this name to clarify how3.it's distinct from other classes, modules, packages, functions, and other objects.When in doubt about prefixing, remember how domain names work. Think ofmail.google.com—the name flows from particular to general. There's no magicabout the three levels of naming, but it often happens to work out that way.Format the name depending on how it's used in Python. There are three broad4.classes of things we'll put names on, which are shown as follows:

Classes: A class has a name that summarizes the objects that are part ofthe class. These names will (often) use CapitalizedCamelCase. Thefirst letter of a class name is capitalized to emphasize that it's a class,not an instance of the class. A class is often a generic concept, rarely adescription of a tangible thing.Objects: A name for an object usually uses snake_case – all lowercasewith multiple _ characters between words. In Python, this includesvariables, functions, modules, packages, parameters, attributes ofobjects, methods of classes, and almost everything else.Script and module files: These are really the OS resources, as seen byPython. Therefore, a filename should follow the conventions forPython objects, using letters, the _ characters and ending with the .py


[ 9 ]

extension. It's technically possible to have pretty wild and freefilenames. Filenames that don't follow Python rules can be difficult touse as a module or package.

How do we choose the broad category part of a name? The general category depends onwhether we're talking about a thing or a property of a thing. While the world is full ofthings, we can create some board groupings that are helpful. Some of the examples areDocument, Enterprise, Place, Program, Product, Process, Person, Asset, Rule, Condition,Plant, Animal, Mineral, and so on.

We can then narrow these with qualifiers:

FinalStatusDocument ReceivedInventoryItemName

The first example is a class called Document. We've narrowed it slightly by adding a prefixto call it a StatusDocument. We narrowed it even further by calling it aFinalStatusDocument. The second example is a Name that we narrowed by specifyingthat it's a ReceivedInventoryItemName. This example required a four-level name toclarify the class.

An object often has properties or attributes. These have a decomposition based in the kindof information that's being represented. Some examples of terms that should be part of acomplete name are amount, code, identifier, name, text, date, time, datetime, picture, video,sound, graphic, value, rate, percent, measure, and so on.

The idea is to put the narrow, more detailed description first, and the broad kind ofinformation last:

measured_height_value estimated_weight_value scheduled_delivery_date location_code

In the first example, height narrows a more general representation term value. Andmeasured_height_value further narrows this. Given this name, we can expect to seeother variations on height. Similar thinking applies to weight_value, delivery_date andlocation_code. Each of these has a narrowing prefix or two.


[ 10 ]

Some things to avoid:Don't include detailed technical type information using coded prefixes orsuffixes. This is often called Hungarian Notation; we don't usef_measured_height_value where the f is supposed to mean a floating-point. A variable like measured_height_value can be any numeric typeand Python will do all the necessary conversions. The technical decorationdoesn't offer much help to someone reading our code, because the typespecification can be misleading or even incorrect.Don't waste a lot of effort forcing names to look like they belong together.We don't need to make SpadesCardSuit, ClubsCardSuit, and so on.Python has many different kinds of namespaces, including packages,modules, and classes, as well as namespace objects to gather related namestogether. If you combine these names in a CardSuit class, you can useCardSuit.Spades, which uses the class as namespace to separate thesenames from other, similar names.

Assigning names to objectsPython doesn't use static variable definitions. A variable is created when a name is assignedto an object. It's important to think of the objects as central to our processing, and variablesas little more than sticky notes that identify an object. Here's how we use the fundamentalassignment statement:

Create an object. In many of the examples we'll create objects as literals. We'll use1.355 or 113 as literal representations of integer objects in Python. We might use astring like FireBrick or a tuple like (178, 34, 34).Write the following kind of statement: variable = object. Here are some examples:2.

>>> circumference_diameter_ratio = 355/113 >>> target_color_name = 'FireBrick' >>> target_color_rgb = (178, 34, 34)

We've created some objects and assigned them to variables. The first object is the result of acalculation. The next two objects are simple literals. Generally, objects are created byexpressions that involve functions or classes.


[ 11 ]

This basic statement isn't the only kind of assignment. We can assign a single object tomultiple variables using a kind of duplicated assignment like this:

>>> target_color_name = first_color_name = 'FireBrick'

This creates two names for the same string object. We can confirm this by checking theinternal ID values that Python uses:

>>> id(target_color_name) == id(first_color_name)True

This comparison shows us that the internal identifiers for these two objects are the same.

A test for equality uses ==. Simple assignment uses =.

When we look at numbers and collections, we'll see that we can combine assignment withan operator. We can do things like this:

>>> total_count = 0>>> total_count += 5>>> total_count += 6>>> total_count11

We've augmented assignment with an operator. total_count += 5 is the same astotal_count = total_count + 5. This technique has the advantage of being shorter.

How it works…This approach to creating names follows the pattern of using narrow, more specificqualifiers first and the wider, less-specific category last. This follows the commonconvention used for domain names and e-mail addresses.

For example, a domain name like mail.google.com has a specific service, a more generalenterprise, and finally a very general domain. This follows the principle of narrow-to-wider.

As another example, [email protected] starts with a specific destination name, has amore general enterprise, and finally a very general domain. Even the name of destination(PacktPub) is a two-part name with a narrow enterprise name (Packt) followed by a widerindustry (Pub, short for publishing). (We don't agree with those who suggest it stands for PublicHouse.)


[ 12 ]

The assignment statement is the only way to put a name on an object. We noted that we canhave two names for the same underlying object. This isn't too useful right now. But inChapter 4, Built-in Data Structures – list, set, dict we'll see some interesting consequences ofmultiple names for a single object.

There's more…We'll try to show descriptive names in all of the recipes.

We have to grant exceptions to existing software which doesn't follow thispattern. It's often better to be consistent with legacy software than imposenew rules even if the new rules are better.

Almost every example will involve assignment to variables. It's central to stateful object-oriented programming.

We'll look at classes and class names in Chapter 6, Basics of Classes and Objects; we'll look atmodules in Chapter 13, Application Integration.

See alsoThe subject of descriptive naming is a source of ongoing research and discussion. There aretwo aspects—syntax and semantics. The starting point for thoughts on Python syntax is thefamous Python Enhancement Proposal number 8 (PEP-8). This leads to use of CamelCase,and snake_case names.

Also, be sure to do this:

>>> import this

This will provide more insight into Python ideals.

For information on semantics, look at the legacy UDEF and NIEM Namingand Design Rules standards (h t t p ://w w w . o p e n g r o u p . o r g /u d e f i n f o /A b o ut T h e U D E F . p d f ). Additional details are in ISO11179(https://en.wikipedia.org/wiki/ISO/IEC_11179), which talks in detailabout meta-data and naming.

http://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttp://www.opengroup.org/udefinfo/AboutTheUDEF.pdfhttps://en.wikipedia.org/wiki/ISO/IEC_11179


[ 13 ]

Working with large and small integersMany programming languages make a distinction between integers, bytes, and longintegers. Some languages include distinctions for signed versus unsigned integers. How dowe map these concepts to Python?

The easy answer is that we don't. Python handles integers of all sizes in a uniform way.From bytes to immense numbers with hundreds of digits, it's all just integers to Python.

Getting readyImagine you need to calculate something really big. For example, calculate the number ofways to permute the cards in a 52-card deck. The number 52! = 52 × 51 × 50 × … × 2 × 1, is avery, very large number. Can we do this in Python?

How to do it…Don't worry. Really. Python behaves as if it has one universal type of integer, and thiscovers all of the bases from bytes to numbers that fill all of the memory. Here are the stepsto using integers properly:

Write the numbers you need. Here are some smallish numbers: 355, 113. There’s1.no practical upper limit.

Creating a very small value—a single byte—looks like this:2.

>>> 2 2

Or perhaps this, if you want to use base 16:

>>> 0xff 255

In later recipes, we'll look at a sequence of bytes that has only a single value in it:

>>> b'\xfe' b'\xfe'

This isn't—technically—an integer. It has a prefix of b' that shows us it's a 1-bytesequence.


[ 14 ]

Creating a much, much bigger number with a calculation might look like this:3.

>>> 2**2048 323...656

This number has 617 digits. We didn't show all of them.

How it works…Internally, Python uses two kinds of numbers. The conversion between these two isseamless and automatic.

For smallish numbers, Python will generally use 4 or 8 byte integer values. Details areburied in CPython's internals, and depend on the facilities of the C-compiler used to buildPython.

For largish numbers, over sys.maxsize, Python switches to large integer numbers whichare sequences of digits. Digit, in this case, often means a 30-bit value.

How many ways can we permute a standard deck of 52 cards? The answer is 52! ≈ 8 × 1067.Here's how we can compute that large number. We'll use the factorial function in the mathmodule, shown as follows:

>>> import math>>> math.factorial(52)80658175170943878571660636856403766975289505440883277824000000000000

Yes, these giant numbers work perfectly.

The first parts of our calculation of 52! (from 52 × 51 × 50 × ... down to about 42) could beperformed entirely using the smallish integers. After that, the rest of the calculation had toswitch to largish integers. We don't see the switch; we only see the results.

For some of the details on the internals of integers, we can look at this:

>>> import sys>>> import math>>> math.log(sys.maxsize, 2)63.0>>> sys.int_infosys.int_info(bits_per_digit=30, sizeof_digit=4)


[ 15 ]

The sys.maxsize value is the largest of the small integer values. We computed the log tobase 2 to find out how many bits are required for this number.

This tells us that our Python uses 63-bit values for small integers. The range of smallishintegers is from -264… 263– 1. Outside this range, largish integers are used.

The values in sys.int_info tells us that large integers are a sequence of numbers that use30-bit digits, and each of these digits occupies 4 bytes.

A large value like 52! consists of 8 of these 30-bit-sized digits. It can be a little confusing tothink of a digit as requiring 30 bits to represent. Instead of 10 symbols used to representbase 10 numbers, we'd need 2**30 distinct symbols for each digit of these large numbers.

A calculation involving a number of big integer values can consume a fair bit of memory.What about small numbers? How can Python manage to keep track of lots of little numberslike one and zero?

For the commonly used numbers (-5 to 256) Python actually creates a secret pool of objectsto optimize memory management. You can see this when you check the id() value forinteger objects:

>>> id(1)4297537952>>> id(2)4297537984>>> a=1+1>>> id(a)4297537984

We've shown the internal id for the integer 1 and the integer 2. When we calculate a value,the resulting object turns out to be the same integer 2 object that was found in the pool.

When you try this, your id() values may be different. However, every time the value of 2is used, it will be the same object; on the author's laptop, it's id = 4297537984. This saveshaving many, many copies of the 2 object cluttering up memory.

Here's a little trick for seeing exactly how huge a number is:

>>> len(str(2**2048))617

We created a string from a calculated number. Then we asked what the length of the stringwas. The response tells us that the number had 617 digits.


[ 16 ]

There's more…Python offers us a broad set of arithmetic operators: +, -, *, /, //, %, and **. The / and //are for division; we'll look at these in a separate recipe named Choosing between true divisionand floor division. The ** raises a number to a power.

For dealing with individual bits, we have some additional operations. We can use &, ^, |,. These operators work bit-by-bit on the internal binary representations ofintegers. These compute a binary AND, a binary Exclusive OR, Inclusive OR, Left Shift,and Right Shift respectively.

While these will work on very big integers, they don't really make much sense outside theworld of individual bytes. Some binary files and network protocols will involve looking atthe bits within an individual byte of data.

We can play around with these operators by using the bin() function to see what's goingon.

Here's a quick example of what we mean:

>>> xor = 0b0011 ^ 0b0101>>> bin(xor)'0b110'

We've used 0b0011 and 0b0101 as our two strings of bits. This helps to clarify preciselywhat the two numbers have as their binary representation. We applied the exclusive or (^)operator to these two sequences of bits. We used the bin() function to see the result as astring of bits. We can carefully line up the bits to see what the operator did.

We can decompose a byte into portions. Say we want to separate the left-most two bits fromthe other six bits. One way to do this is with bit-fiddling expressions like these:

>>> composite_byte = 0b01101100>>> bottom_6_mask =

modern python cookbook - programmer books · 2019. 6. 12. · modern python cookbook the latest in...

Documents