PyInotify
If you have the privilege of working with GNU Linux platforms, then you will appreciate PyInotify. According to the documentation, it is a Python module for watching filesystem changes. The official project page is here http pyinotify.sourceforge.net. Example 8-9 shows how it could work. Example 8-9. Event-monitoring Pyinotify script from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes class PClose ProcessEvent Processes on close event def __init__ self, path self.path path...
Twisted
Twisted is an event-driven networking framework for Python that can tackle pretty much any type of network-related task you need it to. A comprehensive single solution has a price of complexity. Twisted will begin to make sense after you've used it a few times, but understanding it initially can be difficult. Further, learning Twisted is such a large project that finding a beginning point to solve a specific problem can often be daunting. Despite that, though, we highly recommend that you...
alias
The first feature of a Python Unix shell bridge that we will look at is the alias magic function. With alias you can create an IPython shortcut to execute system commands. To define an alias, simply type alias followed by the system command and any arguments for that command . For example In 1 alias nss netstat -lptn Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all. Active Internet connections only servers Proto Recv-Q...
Scheduling Python Processes
Now that we have covered the gamut of ways to deal with processes in Python, we should talk about ways to schedule these processes. Using good old-fashioned cron is highly suitable for running processes in Python. One of the nice new features of cron in many POSIX systems is the advent of scheduled directories. This is the only way we use cron anymore, as it is very convenient to just drop a python script in one of the four default directories etc cron.daily, etc cron.hour ly, etc cron.monthly,...
Creating a Hello World CommandLine Tool to Distribute
To get started with building packages for almost every nix operating system made, we need something to actually distribute. In the spirit of tradition, we are going to create a simple command-line tool called hello_epm.py. See Example 9-8. Example 9-8. Hello EPM command-line tool usr bin env python def main p optparse.OptionParser p.add_option '--os', '-o', default NIX options, arguments p.parse_args print 'Hello EPM, I like to make packages on s' options.os If we run this tool, we get the...
Scapy
If you like writing network code, you are going to love Scapy. Scapy is an incredibly handy interactive packet manipulation program and library. Scapy can discover networks, perform scans, traceroutes, and probes. There is also excellent documentation available for Scapy. If you like this intro, you should buy the book for even more details on Scapy. The first thing to figure out about Scapy is that, as of this writing, it is kept in a single file. You will need to download the latest copy of...
Help with Magic Functions
As we've already said, IPython is incredibly powerful. One reason for this power is that there is an almost overwhelming number of built-in magic functions. Just what is a magic function The IPython documentation says IPython will treat any line whose first character is a as a special call to a 'magic' function. These allow you to control the behavior of IPython itself, plus a lot of system-type features. They are all prefixed with a character, but parameters are given without parentheses or...
Using SSH Keys NFSMounted Source Directory and CrossPlatform Python to Manage
One way to manage a diverse infrastructure of nix machines is to use a combination of ssh keys, a commonly shared NFS-mounted src directory, and cross-platform Python code. Breaking this process into steps will make it clearer. Step 1 create a public ssh key on the system from which you will manage machines. Note that this can vary by platform. Please consult your operating system documentation or do a man on ssh for details. See Example 8-3. One thing to point out in the example below is that...
Subprocess
With Python 2.4 came subprocess, which takes the place of several older modules os.system, os.spawn, os.popen, and popen2 commands. Subprocess is a revolutionary change for systems administrators and developers who need to deal with processes and shelling out. It is now a one-stop shop for many things dealing with processes and it may eventually include the ability to manage a flock of processes. Subprocess might just be the single most important module for a systems administrator, as it is the...
sh Profile
One IPython concept that we haven't described yet is a profile. A profile is simply a set of configuration data that is loaded when you start IPython. You can customize a number of profiles to make IPython perform in different ways depending on a session's needs. To invoke a specific profile, use the -p command-line option and specify the profile you'd like to use. The sh, or shell, profile is one of the built-in profiles for IPython. The sh profile sets some configuration items so that IPython...
Linux System Administration Language
Being a sysadmin often means that you get thrown to the wolves. Rules, a predictable schedule, or even choice of an operating system is often out of your control. To be even a marginally effective sysadmin nowadays, you need to know it all, and we mean all the operating systems. From Linux, to Solaris, to OS X, to FreeBSD, it needs to be in your toolbelt. Although only time will tell, it does seem as if the proprietary operating systems, such as AIX and HP-UX, won't last forever, but they still...
Using Return Codes with Subprocess
One interesting thing to note about subprocess.call is that you can use return codes to determine the success of your command. If you have experience with programming in C or Bash, you will be quite at home with return codes. The phrases exit code and return code are often used interchangeably to describe the status code of a system process. Every process will have a return code when it exits, and the status of the return code can be used to determine what actions a program should take....
Shell Execute
Another, and possibly easier, way of executing a shell command is to place an exclamation point in front of it In 1 netstat -lptn Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all. Active Internet connections only servers Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 tcp 0 0 You can pass in variables to your shell commands by prefixing them with a dollar sign . For example This listed all bash sessions...
rehash
There is another option for aliasing and or executing shell commands from IPython rehashing. Technically, this is creating an alias for shell commands, but it doesn't really feel like that is what you're doing. The rehash magic function updates the alias table with everything that is on your PATH. You may be asking, What is the alias table When you create an alias, IPython has to map the alias name to the shell command with which you wanted it to be associated. The alias table is where that...
Creating a Custom Bootstrapped Virtual Environment
The release of virtualenv 1.0, which is current as of the writing of this book, includes support to create bootstrap scripts for virtualenv environments. One method of doing that is to call What this does is create a bootstrap script, which is like virtualenv, but with additional features to extend option parsing, adjust_options, and use after_install hooks. Let's go over how easy it is to create a custom bootstrap script that will install virtualenv and a custom set of eggs into a new...
Scripting DSCL or Directory Services Utility
DSCL stands for Directory Services Command Line, and it is a convenient hook into OS X's directory services API. DSCL will let you read, create and delete records, so Python is a natural fit. Example 8-10 shows using IPython to script DSCL to read Open Directory attributes and their values. - Note in the example we read only attributes, but it easy enough to mod ify them as well using the same technique. Example 8-10. Getting user record interactively with DSCL and IPython In 42 import...
Creating Scripts with Scapy
Now that we can build something permanent with Scapy, one interesting thing to show right off the bat is an arping tool. Let's look at a platform-specific arping tool first usr bin env python import subprocess import re import sys def Arping function takes IP Address or Network, returns nested mac ip list Assuming use of arping on Red Hat Linux p subprocess.Popen usr sbin arping -c 2 s ipaddress, shell True, stdout subprocess.PIPE out p.stdout.read result out.split pattern re.compile for item...
EPM Package Manager
Because EPM creates native packages for each operating system, it will need to be installed on each build system. Due to the incredible advances in virtualization in the past few years, it is trivial to get a few build virtual machines set up. I created a small cluster of virtual machines running in the equivalent of Red Hat run level init 3, with a minimal allocation of RAM, to test out the code examples in this book. A coworker and contributor to EPM first introduced me to what EPM can do. I...
Processing Module
So what is this processing module we have hinted at, anyway As of the printing of this book, processing is a package for the Python language which supports the spawning of processes using the API of the standard library's threading module One of the great things about the processing module is that it maps to the threading API, more or less. This means that you don't have to learn a new API to fork processes instead of threads. Visit http pypi.python.org pypi processing to find out more about...
Building an Apache Log Viewer Using Curses
curses is a library that facilitates the creation of interactive text-based applications. Unlike GUI toolkits, curses does not follow an event handling and callback approach. You are responsible for getting input from the user and then doing something with it, whereas in GTK, the widget handles getting input from the user and the toolkit calls a handler function when an event occurs. Another difference between curses and GUI toolkits is that with GUI toolkits you are adding widgets to some...
CrossPlatform Unix Programming in Python
While there are some significant differences between nix operating systems, there is much more in common that not. One way to bring the different versions of nix back together is to write cross-platform tools and libraries that bridge the divide between the operating system differences. One of the most basic ways to accomplish this is to write conditional statements that check for the operating systems, platform, and version in code that you write. Python takes the batteries included philosophy...
Graphical Images
The following two examples will continue the example of parsing an Apache logfile for the client IP address and the number of bytes that were transferred. In the previous section, our example generated a shelve file that contained some information that we want to share with other users. So, now, we will create a chart object from the shelve file to make the data easy to read usr bin env python shelve_file shelve.open 'access.s' items_list i l , i 0 for i in shelve_file.items bytes_sent i 0 for...
OS X Scripting APIs
Often, with OS X it is a requirement for a sysadmin to know a bit about high-level scripting that interacts with the actual UI itself. With OS X Leopard, Python, and Ruby, we're given first-class access to the Scripting Bridge. Refer to this documentation for more information One of the options for accessing the OSA, or Open Scripting Architecture, is py-appscript, which has a project page here Using py-appscript is quite fun and powerful, as it gives Python the ability to interact with the...
pinfo
The pinfo function and related utilities have become such a convenience for us that it's hard to imagine not having them. The pinfo function provides information such as type, base class, namespace, and docstring. If we have a module that contains then we can request information from the module itself In 1 import some_module String Form lt module 'some_module' from 'some_module.py' gt Namespace Interactive We can request information from a class in the module In 3 pinfo some_module.Foo Type...
Solaris Systems Administration
From the late '90s to the early 2000s Solaris was a preferred, Big Iron distribution of Unix. Linux's metioric rise in the early 2000s rapidly cut into Solaris' and Sun was in some real trouble. However, recently, a lot of sysadmins, developers, and enterprises are talking about Sun again. Some of the interesting developments in Sun's future are a 6-month release cycle, just like Ubuntu, with a 18-month support window. It is also copying the single CD approach of Ubuntu as well and ditching the...
IPython and NetSNMP
If you haven't done any SNMP development before, you may have gotten the impression that it is a bit nasty. Well, to be honest, it is. Dealing with SNMP is a bit of a pain, as it involves a very complex protocol, lots of RFCs to read, and a high chance for many things to go wrong. One way to diminish much of the initial pain of getting started with development is to use IPython to explore the SNMP code you will write and to get comfortable with the API. Example 7-1 is a very brief snippet of...
VMware
VMware as, we mentioned earlier, is the current powerhouse in virtualization. Having full control programatically over a virtual machine is obviously the Holy Grail. Luckily, there are several APIs to look at Perl, XML-RPC, Python, and C. As of this writing, some of the Python implementations are somewhat limited, but that could change. The new direction for VMware appears to be in terms of the XML-RPC API. VMware has a few different products with a few different APIs. Some of the products you...
GUI Building Theory
When you write a console utility, you often expect it to run and complete without user intervention. This is definitely the case when scripts are run from cron and at, anyway. But when you write a GUI utility, you expect that a user will have to provide some input in order to make things happen and exercise your utility. Think for a moment about your experiences with GUI applications such as web browsers, email clients, and word processors. You run the application somehow. The application...
daemonizer
Dealing with daemons is a given for anyone who has spent more than a cursory amount of time on Unix. Daemons do everything from handling requests to sending files to a printer such as lpd , fielding HTTP requests, and serving up files such as Apache's httpd . So, what is a daemon A daemon is often thought of as a background task that doesn't have a controlling terminal. If you are familiar with Unix job control, you may think that running a command with an amp at the end of it will make it a...
Extending NetSNMP
As we have discussed earlier, Net-SNMP is installed as an agent on most nix machines. There is a default set of information that an agent can return, but it is also possible to extend an agent on a machine. It is reasonably straightforward to write an agent that collects just about anything and then returns the results via the SNMP protocol. The EXAMPLE.conf file is one of the best sources for information on extending Net-SNMP, and it is included with Net-SNMP. Doing a man on snmpd.conf is also...
Ssh
SSH is an incredibly powerful, widely used protocol. You can also think of it as a tool since the most common implementation includes the same name. SSH allows you to securely connect to a remote server, execute shell commands, transfer files, and forward ports in both directions across the connection. If you have the command-line ssh utility, why would you ever want to script using the SSH protocol The main reason is that using the SSH protocol gives you the full power of SSH combined with the...
Xmlrpc
XML-RPC exchanges a specifically formatted XML document between two processes to perform a remote procedure call. But you don't need to worry about XML part you'll probably never have to know the format of the document that is being exchanged between the two processes. The only thing you really need to know to get started using XML-RPC is that there is an implementation of both the client and the server portions in the Python Standard Library. Two things that might be useful to know are XML-RPC...
Building a Simple PyGTK App
We'll create a simple piece of code which uses the already-mentioned gtk.Window and gtk.Button classes. Following is a simple GUI application that doesn't do anything useful except show some of the basic tenets of GUI programming. Before being able to run this example or write your own PyGTK app, you'll have to install PyGTK. This is pretty simple if you're running a relatively modern Linux distribution. It even looks pretty easy for Windows. If you're running Ubuntu, it should already be...
Enterprise SNMP Integration with Zenoss
Zenoss is a fascinating new option for enterprise SNMP management systems. Not only is Zenoss a completely open source application, it is also written in pure Python. Zenoss is a new breed of enterprise application that is both incredibly powerful and extendable via an XML-RPC or ReST API. For more information on ReST, take a look at RESTful Web Services by Leonard Richardson and Sam Ruby O'Reilly . Finally, if you want to help develop Zenoss, you can contribute patches. For the latest...
Pyrolite Python
Pyro is one framework that alleviates XML-RPC shortcomings. Pyro stands for Python Remote Objects capitalization intentional . It lets you do everything you could do with XML-RPC, but rather than dictifying your objects, it maintains their types when you pass them across. If you do want to use Pyro, you will have to install it separately. It doesn't come with Python. Also be aware that Pyro only works with Python, whereas XML-RPC can work between Python and other languages. Example 5-7 is an...
SNMP Device Control
One of the more interesting things SNMP can do is control a device through the SNMP protocol. Obviously, this creates a significant advantage over using something like Pyexpect to control a router, as it is much more straightforward. For brevity's sake, we will only cover SNMP v1 in the example, but if you are communicating with a device over an insecure network, it should be done via SNMP v3. For this section, it would be good to reference Essential SNMP and Cisco IOS Cookbook by Kevin Dooley...
Building an Apache Log Viewer Using PyGTK
Now that we have covered the basics of GUI building in general and of using PyGTK specifically, the following is an example of building something a little more useful with PyGTK we're going to walk through creating an Apache logfile viewer. The functionality we are going to include in this application is as follows Select and open specified logfile View line number, remote host, status, and bytes sent at a glance Sort loglines by line number, remote host, status, or bytes sent This example...