F
fabs function, 222 Fast Fourier Transform, 275-277 fft function, 275-277 Fedora project Linux , 32 field names in CSV files , 162 figure function, 22-23, 186, 190 filecmp module, 339-340 file compression, 337-339 file formats, 6-7, 104-105, 108-126 binary, 104-105, 109, 117-123 converting image, 289-290 CSV, 6-7, 109-117 header files, 122-123 image, 104 INI files, 123-125 Readme files, 123 selecting, 108 text, 109 XML, 125 FileInput module, 332-333 file manipulation, 333-337 file names, 7,...
Filtering
One of the reasons to transform a time-domain signal to a frequency-domain signal is for the purpose of filtering. A filter is an operation that changes a signal. Much like filters in your kitchen sink, filters let some frequencies pass water , while stopping other frequencies large food remains . Filters are used in a variety of applications, ranging from audio to radar systems. Filters are categorized by their behavior. A filter that lets through low frequencies and stops high frequencies is...
Annotating the Graph
We'd like to add some more information to the GPS graph we'd like to know where we've stopped and where we were speeding. For this we use the function find , which is part of the PyLab package. find returns an array of indices that satisfy the condition, in our case gt STANDING_KMH 10.0 gt SPEEDING_KMH 50.0 gt Istand find v lt STANDING_KMH gt Ispeed find v gt SPEEDING_KMH gt Icruise find v gt STANDING_KMH amp v lt SPEEDING_KMH We also calculate when we're cruising i.e., not speeding nor...
Example Stock Price Charts
Following a convention that stores a short description of the data in the beginning lines of the CSV files can be very useful for annotating a graph or a report associated with the data in the file. To follow along with the example, ensure your directory structure is similar to that presented in Chapter 2 in the section Example Directory Structure for the Book. Your base directory should be Ch4 within Ch4 there should be three subdirectories named src, data, and images. If you wish to use a...
Module difflib
The module difflib provides several objects and functions to help compare lists of strings e.g., text files . Several functions provide a diff result in different formats. These include context_diff , ndiff , and unified_diff . In this section we'll examine the context_ diff fl, f2 , fromfile , tofile function other functions have similar behavior. First we create two files, data filel.txt and data file2.txt, with similar but not identical content, as shown in Listing 10-16. Listing 10-16....
The Pickle Module
Pickle comes in two flavors the Pickle module and the faster C implementation named cPickle. With the better performance of cPickle comes a price you can't subclass the module. Personally, I have not found this limitation an issue, so for me, cPickle is a better choice. To use Pickle, issue import pickle to use cPickle, issue import cPickle. The function cPickle.dump obj, file , protocol serializes an object and writes it to file. The protocol argument can take the values 0 for ASCII the...
Module random
Other than mathematical functions, Python also provides a rich library for random numbers. Random numbers are important in a variety of software applications. In game programming, random numbers are used to change the behavior of elements in the game to make it more interesting or unpredictable. When writing simulations, random numbers are used to generate data that simulates the real world. Random numbers can also be used to answer probability questions, as you'll soon see. The random module...
Image Filtering
Most GUI-based image processing applications come with a bundle of image filters. There's a wide variety of filters available, and different applications group them into different categories. Some of the common filtering categories are blur, enhancement, edge detection, and more. From an image processing standpoint, image filters are known operations that help us achieve a specific effect. For example, I once used the counting objects algorithm presented as an example in this book to try to...
The csvreader Object
To create a csv.reader object, use the following syntax csv.reader f , dialect 'excel' , params . The first parameter, f, is the CSV file, but it can also be any iterable object. The second parameter is the dialect. Since there are no clear definitions of what a CSV file constitutes and there are quite a bit of nuances , you can specify a dialect, which is a set of rules instructing the csv.reader parser how to handle those differences. Furthermore, you can use an existing dialect and override...
Nudge Subplots
In generating subplots of size 2 by 2 for this book, I've noticed that the text for the x-axis of the top subplots clashes with the titles of the lower subplots. To overcome this, I've defined nudge_subplot , a function designed to modify the location of subplots within a figure see Listing A-1 . Listing A-1. Source Listing of nudge_subplot A helper function to move subplots sp_ax subp.get_position sp.set_position sp_ax.x0, sp_ax.y0 dy, sp_ax.x1-sp_ax.x0, sp_ax.y1-sp_ax.y0 To use the function,...
Vector and Matrix Operations
Much like dot , the function vdot returns the dot product of two vectors. So if you're only interested in the value of x in the previous example, you can write gt dot inv M 0 , V 0.50000000000000022 The function inner vl, v2 will perform an inner product, that is, multiply every element in vl with the corresponding element in v2 and then sum them together gt VI array l0, -1.5 gt V2 array l, 2 gt gt gt sum 0 gt gt gt for i in range len Vl sum Vl i V2 i I've implemented an inner product operation...
Fourier Transform
Fourier transform is a linear operation that transforms a function from the time domain to the frequency domain. Much like the sound you hear can be viewed as an amplitude as a function of time, it can also be viewed by its frequency components basses are the low frequencies of audio, for example. The topic of Fourier transforms is quite large and requires some mathematical rigor. I will not be trying to address the topic in depth here instead, I will show how you can use PyLab to perform...
Example Saving and Retrieving Python Session Variables
The example in Listing 10-5 makes use of the cPickle module to write variables of varying data types to file. Listing 10-5. Pickling Several Objects to File fname ' data mysession.pickle' a 3 b A string c 'diet' 10 d eye 3 fout open fname, 'wb' for var in a, b, c, d cPickle.dump var, fout fout.close To pickle objects i.e., serialize them and write them to file, I've used the function cPickle.dump var, file . You can issue subsequent calls to cPickle.dump to store additional values to file as...
Python Integrated Development Environments
An integrated development environment IDE is, simply put, an application that helps programmers write code. Typically an IDE is composed of the language engine Python , an editor, a debugger, documentation, and possibly additional productivity tools. While it is by all means possible to use Python without an IDE, using one will greatly increase your productivity and will enable a faster learning pace. There is a wealth of Python IDEs, and a rather extensive list is provided in the books Python...
Text Annotations
Other than geometrical shapes, ImageDraw also provides text annotation with the function text xy, string gt gt gt from PIL import Image, ImageDraw gt gt gt img Image.new 'L', 160, 160 , 255 gt gt gt draw ImageDraw.Draw img gt gt gt draw.ellipse o, 0, 160, 160 , fill 128 gt gt gt draw.text 80, 80 , 'A long string' gt gt gt img.show Originally, I had intended on having the text centered horizontally. However, the text string has width, so I require a method to calculate the width and height of...
The csv Module
The csv module, which is part of the Python Standard Library, provides simple methods to read and write CSV files. To use the csv module, issue import csv the remaining discussion assumes you've imported the csv module properly. There are two basic objects you'll be working with the csv.reader and the csv.writer. As their names suggest, one is used for reading, while the other is used for writing. The csv.reader object splits a line of text into a list of words also referred to as fields ....
More Integration Methods
Additional integration algorithms are available with the module scipy.integrate. To use this module, issue from scipy import integrate. We'll limit our discussion to the algorithm quad , which uses a Gaussian quadrature to numerically integrate a mathematical function. Unlike previous methods such as trapz , using quad requires supplying a mathematical function and not the x and y vectors. Note I've used the term mathematical function to differentiate this type of function from a generalpurpose...
Example HeartRate Monitor
For the purpose of this example, I'll generate a signal that simulates the data generated from a heart-rate monitor connected to a patient. Please do not use this in any sort of production system it's merely for educational purposes and not meant to truly represent heart signals . The patient walks around, and as a result, two signals are picked up 1 the heart signal and 2 a signal associated with the patient's movement, or what is typically referred to as a movement artifact. Listing 8-8 shows...
Interpolation and Curve Fitting
Interpolation and curve fitting deal with fitting functions to discrete known values. There are several reasons you would want to fit functions to points of data, among which are Fitting a known function to gathered experimental data. This can be helpful in determining other parameters of the experiment. Evaluating the numerical values of functions at additional points other than the given ones . Interpolation allows efficient implementations that are tailor-made to a specific problem. Instead...
Patches
So far we've worked with text and lines, which are both implemented as matplotlib objects. But those two objects at times are not enough. A third object, the patch, allows drawing other types of shapes that don't necessarily fall under the category of a line or text. The way you work with patches is that you assign them to an already existing graph, because in a sense patches are patched on top of a figure. Table 6-6 gives a partial listing of available patches. In this table, the notation xy...
Uses of Polynomials
So why is all this polystuff important The main reason is that you can use polynomials to approximate functions both from gathered data and from analytical functions. And since polynomials only require multiplications and additions, implementing polynomials in an embedded system, for example, is straightforward. Fitting polynomials to data is done using the function polyfit x, y, n . Given a vector of x points and a vector of y points, polyfit x, y, n will return a polynomial of degree n...
Data Structures
Python, being a high-level programming language, also provides additional, more complex, data types, which I refer to as data structures. These include lists, tuples, dictionaries, and sets, to name a few. Data structures make the programming experience a lot more enjoyable. Python documentation does not necessarily differentiate between data types and data structures the way I have. My purpose in this distinction is to split the discussion into two categories simple data types, which you're...
Bar Charts
A favorite of many, bar charts allow quantitative comparison of several values. To use a bar chart, call the function bar left, height , where left is the x coordinates of the bar and height is the bar height. The function bar allows for considerable customization issuing help bar will provide most of the details. For this example, which plots the purchasing power parity GDP of various countries, you'll need the CIA GDP Rank Order file, available from the CIA World Factbook https www.cia. this...
Filter Design
Assuming you know what filter you wish to design, this section will help you do so. Filter design is an advanced topic, and as such this section is meant for those who require a few pointers on designing filters in Python with SciPy. The scipy.signal module includes several functions to help design a filter. The function iirdesign is used for designing an IIR filter. It is quite complete, and it's best to read the online help and follow it through. Other useful IIR design filters include butter...
Using a while Loop
Use this method in conjunction with read to process chunks of the file at a time. Again, this is best suited for larger files and in cases where you don't want to treat a file as a list of lines gt gt gt f open ' data tobuy.txt' gt gt gt while ch 'k' and ch This example reads the file a byte at a time and stops upon encountering the character 'k' or an end-of-file where ch would then evaluate to False.
Magic Square Arrows
In Chapter 7 I presented a figure describing the magic square algorithm. I used matplotlib patch arrows embedded in the algorithm to plot that figure. Listing A-4 is the source code used to generate the diagram. Listing A-4. Magic Square Diagram Creation from pylab import def magic_arrow x, y, top_right, n, c 0 Draws an arrow from point x, y. if top_right top-right arrow mc my_colors c len my_colors ar Arrow x 0.5 d, n-y-0.5 d, 1-2 d, 1-2 d, width 0.2, fc mc, ec mc else down arrow ar Arrow x...
Example Logging Information with a Date and Timestamp
The purpose of this example is to create a log file in accordance with the tip presented previously in the Date and Time section and the ISO time format recommendation see Chapter 4 . We'll use the function localtime , which returns a structtime tuple containing the current time. We then format the current time using the strftime function gt gt gt from time import localtime, strftime gt strftime Y- m- dT H- M- S, localtimeQ '2008-10-23T12-16-41' From now on, I'll assume you either imported the...
Special Functions
The scipy.special module provides a host of special functions that surface usually in higher mathematics and physics. These include Bessel functions, integrals, derivatives, and zeros ofBessel functions Gamma functions and error functions Special polynomials Legendre, Chebyshev and many more. To use the functions, issue the following gt gt gt from scipy import special gt gt gt special.chebyt 2 polyld 2.00000000e 00, -4.44089210e-l6, -1.00000000e 00 For a full account, issue help special .
Signal Processing
Up to this point in the chapter, we've dealt with numerical analysis. Going forward, the topics are related to signal processing. Signal processing is a vast field that deals with signals values that change over time. Popular signal processing algorithms include the processing of sound, such as an equalizer others include algorithms for radars, CAT scanning, and many more. This part of the chapter will cover some of the functionality available with the module scipy.signal and complement the...
More csv Functionality
The csv module allows considerable customization including the use and creation of user-defined dialects. I won't be covering this topic however, I will be covering some parameters and their meaning. The delimiter specifies the field separator. That is, the row is split on an occurrence of the delimiter, provided it is not escaped or quoted. To change the delimiter, add delimiter char as an argument to the csv object char must be a character gt gt gt for line in delimiter ' ' print line '1',...
P
packages, manually installing example , 44 parameters, command-line, 327-333 parse_args method, 329, 331 parsing, date and time, 165-168 pass statement, 4, 86 paste function, 292 patches, 217-220 path names, 127 PATH variable, 59 patterns, regular expression, 173-174 PDF, 184 piecewise linear interpolation, 258-260 plot function, 19-20, 189-193, 214 displaying several graphs in one, 191 GPS location, 18-20 logarithmic, 207-208 matplotlib package, 183-184 plot summary example, 200-201 polar,...
Velocity Plot
We now turn to plotting a graph of the speed. This is a lot simpler gt plot t 0 , t -l , STANDING_KMH, STANDING_KMH , '-g' Standing threshold str STANDING_KMH gt plot t 0 , t -l , SPEEDING_KMH, SPEEDING_KMH , '-r' Speeding threshold str SPEEDING_KMH gt gt gt xlabel 'Time from start of file minutes ' gt gt gt ylabel 'Speed Km H ' We start by opening a different figure with the figure command. We proceed by changing the timescale units to minutes, a value easier for most humans to follow than...
Array Methods and Properties
Arrays are objects and as such have functions called methods and variables called properties. Using IPython see Chapter 2 , you can list an object's methods and properties by using character completion, accessible via the Tab key. Alternatively, you can issue the following gt gt gt m for m in dir numpy.ndarray if not m.startswith '_' 'T', 'all', 'any', 'argmax', 'argmin', 'argsort', 'astype', 'base', 'byteswap', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'ctypes', 'cumprod',...
Piecewise Linear Interpolation
Let's turn back to our half-a-circle example. This time, we'll limit ourselves to a quarter of a circle, that is, positive values of x and y. We start by calculating the y values for x equal to 0, 0.2, . . . , 1. We'll store the results in vectors xp and yp array 0. , 0.2, 0.4, 0.6, 0.8, 1. gt gt gt yp sqrt l-xp 2 We'd like to calculate the values of y for x values equal to 0.1, 0.3, . . . , 0.9 given xp and yp. We'll use the function interp x, xp, yp for this. The function returns the value of...
Module NumPy
NumPy's ndarray object has been the basic building block for a lot of the data processing and visualization scripts presented throughout the book. We now turn to exploring this package and discussing its usage. Note Although used in previous chapters, we have not explicitly seen calls to import NumPy. Nevertheless, we did use NumPy's ndarray object extensively. The reason we have not seen NumPy imports is that we have been using the from pylab import command instead, which imports, among other...
Plotting Several Graphs on One Figure
We use graphs to visualize data and compare it. What's more natural than displaying several graphs in one plot so we can compare results There are two ways to do that in matplotlib. The first one is by adding more vectors to the plot function gt plot t, y, ' ', t, 2 y, 's-' The second method is by calling plot repeatedly. Sometimes you might have only partial data to plot. Say you have vector y, but then you modify it and want to print both the original vector and the newly modified vector....
Dictionaries
Dictionaries are mutable sequences that connect a key with a value. The key must be unique, whereas the value need not be. I like to use a phonebook analogy when I think about dictionaries. Every phone number key has but one entry value associated with it, usually a person however, one person value can have several phones keys . The key and value objects can be most data types, with the exception of some e.g., another dictionary . There are several ways to create a dictionary using the dict...
Part Twinkle Twinkle Little Star
First, we create the stars for our image of the sky at night. The night sky will be composed of white stars and a black background. Since we want the stars to be of varying sizes and shapes, we'll define a function named star that creates a matplotlib patch object see Listing 9-8 . Listing 9-8. A Star Patch, Source of star_patch.py create a star patch object from pylab import def star R, xO, yO, color 'w', N 5, thin 0.5 Returns an N-pointed star of size R at xO, yO matplotlib patch polystar...
Example Extracting Numbers from a Text File
At times, it's useful to be able to extract only the numbers in a text file. This could be for the purpose of creating files based on existing ones, say, for testing purposes. Another scenario might be you have a system that maintains the number of users in a text file, and you'd like to write a script to increment that number. The function presented in Listing 5-13 reads a text file and creates a modified version of the file with all the numbers incremented. For the purpose of this example,...
Working with Text Files
Now that we have the basics covered, that is, reading and writing files and processing strings, it's time to combine the two new skills. This section is presented as a list of examples. The examples can be used for educational purposes, but they can also be used to form the basis of helper functions for text-based data processing. With time, I hope you modify the code presented here to best fit your needs. It is important that you treat these examples for what they are, that is, examples and...
Example Moving Average
On many occasions, filtering is used to smooth a signal. A simple algorithm is that of a moving average. For every two consecutive points, we calculate the average and use that value instead. The points are overlapping, so a result of using the algorithm on the vector l, 2, 0, 2 would be 1.5, 1, l . But why stop at two samples Moving average can be performed on several points, returning the average of those points. In Python, you could write gt gt gt from pylab import gt N 512 gt gt gt x l-exp...
Solving Nonlinear Equations
In Chapter 7 we've talked about Newton's method and used it to draw fractals. Newton's method was used to solve a nonlinear equation. The module scipy.optimize provides us with additional tools to solve nonlinear equations, as well as other optimization routines that will not be discussed here. Of those routines, I'd like to highlight three fsolve f, xo , bisect f, a, b , and newton f, xo . All these functions try to solve the equation f 0, where f is a mathematical function implemented in...
Spline Interpolation
The scipy.interpolate module adds additional interpolation functions. One of these is the spline xp, yp, x interpolation function. Notice that the arguments to the function spline are ordered differently from those of the function interp . Spline interpolation is a piecewise polynomial interpolation that adheres to specific rules to yield smooth results. Let's turn to the previous circle example from scipy.interpolate import spline plot xi, yi, '--', label 'piecewise linear', lw 2 plot xi, ys,...
Example Reading and Writing an Array of Structs
In this example, we'll create a structure containing two data types long and float , write it to file, and then read it using two different methods a structure at a time and the entire file at once. You can follow along by entering the commands interactively at the Python shell. First, we have to import the struct module To illustrate the concept of an array of structs, we'll create a list of rows. Each row is a list of three values a long and two floats, which represent a structure. We'll...
Example Reading and Writing an Array of Binary Values
The Python array data type is an ideal candidate for this sort of binary file handling. The array data type is part of the array module, so to use it, issue the following command To create an array, call the array function with the data type and optional initialization parameters, as follows gt gt gt a array 'h' array of signed shorts, of zero size gt gt gt a gt gt gt b array 'L', 1000, 2000, 3000 array of three unsigned longs gt gt gt c array 'd', range io array of doubles, from 0to9 including...
Example Automatically Reading Yahoo Financial Data
The following discussion is a bit off-topic, but as it is a direct continuation of the previous example, this is probably a logical spot for it. There's an alternative method to manually saving the charts.xls file from NASDAQ. One such option is using the matplotlib.finance module. The two core functions that fetch the data and parse it are fetch_historical_yahoo and parse_yahoo_historical although you could easily parse the data yourself . Another function of interest is the candlestick...
Annotating with Geometrical Shapes
PIL provides us with the ImageDraw object, which allows annotations of an existing image. To import the ImageDraw object, issue from PIL import ImageDraw. To use the ImageDraw object, attach it to an existing image gt gt gt from PIL import Image, ImageDraw gt gt gt img Image.new 'RGB', 200, 300 , 0, 0, 255 gt gt gt draw ImageDraw.Draw img I've created an ImageDraw object named draw and attached it to the image img. Going forward, operations performed with the ImageDraw object will be performed...
Example FFT of a Sampled Cosine Wave
A cosine wave is made of one frequency actually, two frequencies if you include the negative frequency . Let's generate a cosine wave and calculate its frequency using fft , as shown in Listing 8-5. Listing 8-5. Fourier Transform of a Cosine Wave t arange N float N sampled over 1 second subplot 2, 2 f t N xf fft x plot f, abs xf title 'Fourier transform of a cosine wave' I first defined a few parameters N is the number of points in the signal, and f is the frequency of the cosine wave. I then...
Math Functions
Simple arithmetic operations are possible on arrays addition, subtraction, division, multiplication, and exponentiation as well as most math functions available in math and cmath modules albeit now they're implemented as part of NumPy . Example Visualizing Fourier Expansion of a Rectangular Wave The following is an example showing a Fourier expansion of a rectangular wave using a sum of sine waves. Fourier expansion is used in numerous applications ranging from solving differential equations to...
Part Flood Fill and Recursion
We now turn to something completely different recursion. Recursion describes a scenario where a function calls itself. Some known recursion algorithms implement the factorial operation and Fibonacci sequences. We'll use recursion for image processing, specifically to fill an image using a flood-fill algorithm. Flood fill, sometimes also referred to as bucket fill, is an algorithm to fill a closed area of a specific color with a different color. This is a quite common operation in most image...