Admin magazine sample pdf by schmitz

System Management

NEW MAGAZINE!

Smart tools for better networks

Four Live systems on one disc Network & Security

ISSUE 01

ADMIN

ANALYZE AND REPAIR PC SYSTEMS

ADMIN

ADMIN MAGAZINE

Rescue CD ✚ SystemRescueCd ✚ Parted Magic ✚ ✚ Clonezilla Live ✚ Redo Backup ✚

Network & Security

System Management

5Free

Spacewalk Icinga

System Management

Fantastic

Back-up Tools

MySQL Forks Exchange 2010

Spacewalk: Corral your Red Hat Servers Optimizing with SystemTap Mastering Microsoft’s virtual machine manager

Free Backup Tools

Icinga

SystemTap

Oracle Clusters

Teamviewer

Easy remote control

ModSecurity

Protect your Apache web server

Is this nascent network monitor nattier than Nagios?

OpenVZ

Efficient data access with OCFS2

EXCHANGE 2010 What’s new with Microsoft messaging?

MySQL Forks

Teamviewer

A perfect tool for every task

ADMIN Magazine Issue 01

Chef

Windows • Linux • Unix • OpenSolaris WWW.ADMIN-MAGAZINE.COM

£7.99

9 772045 070003

“You even get six-star customer support thrown in." - PC Pro magazine

HALVE YOUR IT COSTS WITH

CLOUD COMPUTING RENT YOUR SERVERS FROM

Scalable, affordable computing instances from virtual machines to performance dedicated servers

We’ve been voted the UK’s best hosting company for 4 years running. Isn’t it time you found out why?

ISO 9001: Quality

www.memset.com 0800 634 9270

ISO 14001: Environmental

ISO 27001: Security

hosting

Welcome to Admin

W E LCO M E

ALL FOR ADMINS Our Admin special edition was so popular we’re back, with a new quarterly magazine that is all for admins. Welcome to the first issue of Admin: Network and Security – a magazine for administrators of heterogeneous networks. In these pages, you’ll learn about tools for configuring, managing, and troubleshooting your networks. You’ll hear about cloud infrastructures, database systems, server plugins, and enterprise management applications. We’ll tune in to security and network protocols, and we’ll show you the latest interoperability techniques. This issue is packed with practical information for real networks. Red Hat published the source code for their popular Network Satellite management tool in 2008, and a community version known as Spacewalk soon followed. Our first article takes you on a walk with Spacewalk. Other special features include a study of Icinga – a GPLed fork of the popular Nagios network monitoring system, as well as an article on the forks and patches of MySQL and a roundup of open source backup tools. Farther in, we’ll show you the OCFS2 filesystem for Oracle clusters. We’ll also look at some virtualization tools, including Microsoft’s vm2008 and OpenVZ – a container-based virtualization alternative built on Linux. We’ll investigate some security and system monitoring tools, and we’ll cook up some recipe’s for fast client configuration with the Chef configuration management app. If you’re an IT professional, and you’re looking for a magazine with detailed, technical articles that are relevant to the real-life world you live and work in, read on. And if you would like to receive future issues of Admin delivered to your door, see page 58 for information on subscribing.

W W W. A D M I N - M AGA Z I N E .CO M

ADMIN 01

Table of Contents

S E RV I C E

ADMIN Network & Security

Features

Tools

Virtualization

Management, monitoring, database forks, and other pertinent glimpses at innovations across the IT landscape.

Save time and simplify your workday with these useful tools for real-world networks.

Still finding your way through the cloud? Keep on course with these cool tools for virtual environments.

Spacewalk Walk on air with this free version of Red Hat's enterprise-ready Satellite management server.

Icinga This is not your father's Nagios fork.

OCFS2 Build a database cluster with Oracleâ&#x20AC;&#x2122;s Cluster Filesystem.

Synergy Too many monitors on your desk? This handy tool lets you control your servers from a single desktop.

MySQL Forks We investigate some invaluable variations on the MySQL theme.

Package Your Scripts We show you how to use Debianâ&#x20AC;&#x2122;s packaging tools to deploy and manage scripts in the cloud.

OpenVZ Container-based virtualization tools like OpenVZ are sometimes more efficient than hypervisor systems. Applications Container Context

Exchange 2010 What's new in Microsoft's email and messaging system? Backup Tools We round up some of the best open source backup utilities.

Resource Container 1

Resource Container 3

Resource Container 2

Applications Host Context OpenVZ Abstraction Layer

Host System Kernel

SystemTap Optimize and troubleshoot your homegrown apps with this powerful profiling tool.

SCVMM 2008 Exploring Microsoft's Service Center Virtual Machine Manager 2008.

BlackHat USA 2010 Learn the latest tricks of network intruders at the BlackHat conference.

ADMIN 01

W W W. A D M I N - M AGA Z I N E .CO M

Table of Contents

86 8

Spacewalk Red Hat released the source code for the Satellite Server management tool in 2008. Spacewalk is a new free version.

ModSecurity Careful admins know new exploits appear every day. Keep the intruders off your pages with this web application firewall for Apache web servers.

S E RV I C E

Rescue CD Toolbox ANALYZE AND REPAIR PC SYSTEMS

Management

Nuts and Bolts

Use these practical apps to extend, simplify, and automate routine admin tasks.

Timely tutorials on fundamental techniques for system administrators.

Teamviewer This popular remote control tool isn't just for Mac and Windows anymore.

Chef This snappy configuration manager lets you roll out Linux systems with a couple of mouse clicks.

PAM The powerful Pluggable Authentication System offers centralized authentication for Unix and Linux systems.

✚ Four fabulous Live Linux systems ✚ SystemRescueCd – A handy collection of advanced rescue utilities ✚ Parted Magic – Partition your hard disk ✚ Clonezilla Live – Clone and restore Linux systems

Sysinternals Get the pulse of your Windows network with this convenient collection of management tools.

ModSecurity How safe is your web server? This powerful Apache extension will help keep intruders from getting control.

Monitoring Daemons Why write a custom script? A few simple shell commands might be all you need to monitor system daemons.

VPNs with SSTP Build a virtual private network with the Secure Sockets Tunneling Protocol (SSTP).

✚ Redo Backup – World‘s easiest backup distro

etails

d See p 6 for full

Service 3 4 6 98

Welcome Table of Contents On the CD Call for Papers

W W W. A D M I N - M AGA Z I N E .CO M

ADMIN 01

On the CD

S E RV I C E

Rescue CD Toolbox

On the CD

CD? DEFECTIVE ill be replaced. s D ective C w Def email to Please send an m. -magazine.co subs@admin

The CD included with this issue lets you boot to four special-purpose Live Linux systems: ■ SystemRescueCd – This versatile rescue distro includes a copious collection of networking and troubleshooting tools, including utilities for accessing and repairing Windows systems. ■ Parted Magic – A little Linux that specializes in formatting, resizing, and recovering partitions. ■ Clonezilla Live – Clone systems for fast and easy backup and restore. The bare-metal backup and recovery approach will bring back your system in a fraction of the time. ■ Redo Backup – Another bare-metal backup tool with an emphasis on simplicity. Place the CD in the drive and reboot your system. (Make sure your computer is configured to boot to the CD drive.) Choose a distro in the handy boot menu. See the box titled “Resources” for links to more information on the tools in our Rescue CD Toolbox. ■

Resources [1] [2] [3] [4]

ADMIN 01

SystemRescueCd: [http://www.sysresccd.org/] Parted Magic: [http://partedmagic.com/] Clonzilla: [http://www.clonezilla.org/] Redo Backup: [http://redobackup.org/]

W W W. A D M I N - M AGA Z I N E .CO M

POWERHOUSE

PERL 11 cool projects! ■ Math Tricks: Solve math problems with Perl ■ Daily Tip: Perl with an SQLite database ■ AJAX: Add dynamic updates to web pages ■ isp-switch: Switch your computer to another ISP if your connection goes down ■ MAC Addresses: Monitor your network for unfamiliar MAC addresses ■ Multimeter: Read and interpret data from an external device ■ Google Chart: Create custom graphs ■ Twitter: Help your scripts send Tweets ■ Webcam: Control a web camera with Perl ■ Perl Gardening: Water your house plants with a little help from Perl ■ GPS: Extract data from a GPS device and plot it on a Yahoo! Map

FREE DVD inside: openSUSE 11.2 Linux Mint 8.0

New to Perl? Perl expert Randal L. Schwartz provides an in-depth introduction to the principles of the versatile Perl language. Then Perlmeister Mike Schilli explains how to speed up and debug your scripts. Also inside: Get hands-on with a collection of some of the Perlmeister’s best columns!

OU! Y R A E N ND NEWSSTA l

NA O pecia S L / R m E o ! P c . E e copy today US azin O g H a order your R so m E d e it x W m li u Supplies are AT lin IND PO INE

ONL R E D R O OR

Spacewalk

F e at u r e s

Managing Linux systems with Spacewalk

Moon Landing As your network grows, managing Linux systems manually becomes time consuming and impractical. Enter Spacewalk: an open source tool that takes the footwork out of network management. By Thorsten Scherf

Spacewalk [1] is the open source derivative of the popular Red Hat Network Satellite Server. Red Hat published the source code for the server in the summer of 2008, and the community has now released version 1.0. The application’s core tasks include RPM package software provisioning, managing configuration files, and kickstart trees, thus supporting the installation of bare-metal systems. The approach that Spacewalk uses is quite simple: Before a system can access Spacewalk’s resources, it

Admin 01

first has to register with the server. Registration can be based either on a username/password combination or an activation key that is pregenerated by the Spacewalk server. After registration, the system appears in the server’s web GUI. If the server has more resources, you can assign them to the system at this point. Resources include software packages or configuration files that are normally organized in channels. A system always has exactly one base channel with optional subchan-

nels. The base channel contains the RPM-based operating system, such as Red Hat Enterprise Linux, Fedora, or CentOS. The subchannels contain additional software packages that are independent of the operating system, such as the Red Hat Cluster Suite or the 389 Directory Server. Spacewalk can clone existing channels and create new channels from scratch. This feature gives you full control of the software stack that you provide via Spacewalk. Configuration channels help you distribute the

w w w. a d m i n - m aga z i n e .co m

Spacewalk

configuration files for the software packages. Spacewalk also keeps older versions of the files to let you roll back to a previous version at any time if the need arises. The software packages or configuration files can be installed either via the target system or centrally in the Spacewalk web front end. To avoid spending too much time on the installation of a large number of systems, you can assign systems to logical groups and apply the installation of a resource to a group. For example, it might make sense to assign all your web servers to a WWW-Server group in Spacewalk. When a new version of the web server software is released, you would simply tell Spacewalk to apply the update to the group, automatically updating all the systems belonging to the group. The installation uses polling by default; in other words, the client systems query the server at a predefined interval (which defaults to four hours) to see if new actions have been defined since the last poll. If so, Spacewalk then runs these actions. As an alternative, you can trigger the installation of software packages and other actions using a push approach. The client system and the Spacewalk server talk to each other constantly using the Jabber protocol. Any new actions you define are immediately run on the client by Spacewalk.

Ground Control Communications are always from the client to the server; this is important with respect to firewall rules. A list of the network ports you need to enable can be found online [2]. Besides software package or configuration file installation, actions can also run arbitrary commands on the individual systems via the Spacewalk server. For example, after creating a new configuration file for your web servers and distributing it to the systems, you need to restart the web server process to parse the new configuration instructions. Instead of logging in to each individual system or using a for loop, simply issue the restart

w w w. a d m i n - m aga z i n e .co m

command centrally on the Spacewalk server. Installing new systems is also quite simple. Spacewalk has the installation files you need in the form of kickstart trees. The installation candidate uses a boot medium such as a CD, a USB stick, or a PXE-capable network card to contact the server. The First-Stage Installer, which is part of the installation medium, defines which server will handle the installation. The remaining installation steps are handled by the Second-Stage Installer, located on the Spacewalk server and transferred to the client system when the installation starts. If you want to automate the installation fully, define the kickstart file location in the boot medium. The kickstart file is a kind of answer file that describes the properties of the installation candidate, such as partitioning, software, language, and firewall settings. Of course, you can create a kickstart file on the Spacewalk server and just include a link to the file on the boot medium. Spacewalk can manage any RPMbased distribution. You even have the option of operating client systems across multiple organizations. Using the web interface, the administrator creates various organizations and assigns a certain number of system entitlements to them. Entitlements are linked to certificates that Spacewalk automatically generates during the installation. You can then add users to the organizations. If a client is registered with a user account from a specific organization, the system is assigned to this organization. When users from the organization logs into the Spacewalk server, they will only see the systems in their own organization. This feature is useful if you manage multiple departments and prefer to manage the systems in the individual departments separately. You just assign them to different organizations, which, of course, you need to create up front.

F e at u r e s

[4], or CentOS [3] Linux. Note that

Spacewalk does need a current Java Runtime Version 1.6.0 or newer. You can use the Open JDK for this; Fedora includes it out of the box. Admins on RHEL or CentOS can retrieve the package via the additional EPEL (Extra Packages for Enterprise Linux) software repository. Besides the Java package, an Oracle 10g database is also required for installing Spacewalk. Oracle XE provides a free version of the database. The developers are currently working hard on implementing support for an open source database after identifying PostgreSQL as the best alternative to Oracle. As of this writing it is hard to say when official support for PostgreSQL will be available, but it makes sense to check the roadmap [5] or the mailing lists [6] at regular intervals.

Oracle XE After installing the repository RPM for your distribution, the first step is to install Oracle Express, which you can download for free [7]. You will need version 10.2.0.1. Besides the database, you also need the oracle‑instantclient‑basic and oracle‑instantclient‑sqlplus, which you can then install with Yum: yum localinstall ‑‑nogpgcheck U oracle‑xe‑univ*.rpm oracle‑instantclient‑basic*.rpm oracle‑instantclient‑sqlplus*.rpm

Before configuring the database, you should make sure that your hostname points to the correct IP address in your /etc/hosts to avoid problems Listing 1: Oracle Listener Configuration cat >> /etc/tnsnames.ora << 'EOF' XE = (DESCRIPTION = (ADDRESS_LIST = ( ADDRESS = (PROTOCOL = TCP)(HOST = localhost) (PORT = 1521)) ) (CONNECT_DATA =

Installation Spacewalk can be installed on Red Hat Enterprise (RHEL) [3], Fedora

(SERVICE_NAME = xe) ) ) EOF

Admin 01

F e at u r e s

Spacewalk

with the Oracle Listener configuration later on. Use the following parameters for the configuration:

the appropriate repository in /etc/ yum.repos.d/. The following command starts the installation:

HTTP port for Oracle Application U

yum install spacewalk‑oracle

Express: 9055 Database listener port: 1521 Password for SYS/SYSTEM: Password Start at boot: y

The default HTTP port for the Oracle Express application (8080) is already occupied by the Tomcat application server, so you need to choose an alternative port to avoid conflicts. To talk to the database, you need to configure the listener in the /etc/ tnsnames.ora file (Listing 1). Now you just need to make a few changes to the database. To do this, log in to the database with sqlplus and create a spacewalk user, to which you could assign a password of spacewalk (Listing 2). The standard configuration of Oracle Express supports a maximum of 40 simultaneous connections, which is not enough for Spacewalk operations. The instructions in Listing 3 change the limit to a maximum of 400 connections. Now you need to restart the database by giving the /sbin/service oracle‑ xe command.

Spacewalk Setup The next step is to install the Spacewalk server. To do so, you need to include the Spacewalk repository as described previously. You should have a spacewalk.repo file that points to

Because this package depends on all the other Spacewalk packages, the package manager will automatically download and install the dependencies in the next step. Then you can configure the application interactively with the setup tool or with the use of an answer file (Listing 4). Pass the file in to the setup tool as follows:

you can set up subchannels for the base channel and assign the subchannels to clients as needed. After doing so, you can use the subchannels to distribute more RPM packages to the systems. The packages can be your own creations or RPMs from other repositories. The easiest approach to setting up a software channel is to use the web interface (Channels | Manage Software Channels | Create; Figure 1). Thanks to the Spacewalk API, you can also script this process [8]. Call the script as follows:

spacewalk‑setup ‑‑disconnected U

create_channel.py ‑‑label=fedora‑12‑i386 U

‑‑answer‑file=answerfile

‑‑name "Fedora 12 32‑bit" U ‑‑summary "32‑bit Fedora 12 channel"

The configuration can take some time to complete as the process sets up the database tables. The setup tool then launches all the required services. You can manually restart using the /usr/sbin/rhn‑satellite tool. To configure the system, launch the Spacewalk web interface via its URL (http://spacewalk.server.tld). Besides contact information, you can also set the password for the Spacewalk administrator here.

Software Channels The next step is to set up an initial software channel for the client systems. When you register a client, you must specify exactly one base channel for the client; it will use this channel to retrieve its operating system packages and their updates. Of course,

In the script, you need to provide the Fully Qualified Domain Name (FQDN) for the Spacewalk server and the user account for creating the channels, such as the Spacewalk administrator account created previously. The Users tab also gives you the option of creating more users with specific privileges (Figure 2). The channel you set up should now be visible in the Channels tab of the web interface but will not contain any software packages. Although you can upload software packages to the server in several ways, the method you choose will depend on whether the packages are available locally (e.g., DVD) or you want to synchronize a remote Yum repository with the Spacewalk server. If you choose the local upload, you can use the

Listing 2: Creating the Spacewalk User sqlplus 'sys@xe as sysdba' SQL> create user spacewalk identified by spacewalk default tablespace users; SQL> grant dba to spacewalk; SQL> quit

Listing 3: Oracle Tuning sqlplus spacewalk/spacewalk@xe SQL> alter system set processes = 400 scope=spfile; SQL> alter system set "_optimizer_filter_pred_pullup" =false scope=spfile; SQL> alter system set "_optimizer_cost_based_ transformation"=off scope=spfile; SQL> quit

Figure 1: The easiest approach to setting up a software channel is to use the web graphical interface.

Admin 01

w w w. a d m i n - m aga z i n e .co m

Spacewalk

Figure 2: Assigning individual users different privileges on the Spacewalk server.

rhnpush tool, which you launch as

follows: rhnpush ‑v ‑‑channel=fedora‑13‑i386 U ‑‑server=http://localhost/APP U ‑‑dir=/path/to/the/packages

To synchronize with a remote software repository, you simply need to specify the URL for the remote repository in the software channel properties in the web interface (Channels | Manage Software Channels | Fedora 12 32-bit). Synchronization can take a while to happen. Your other option here is the spacewalk‑repo‑sync command-line tool that downloads software packages from a Yum repository to your own Spacewalk server. To keep the server up to date, you can use cron to run a script [9] at regular intervals. This script will check your configured software sources and automatically download any new packages. This approach removes the need for manual synchronization. Incidentally, you can use the method discussed here to set up subchannels,

too. Note that any RPM packages you build yourself must be digitally signed. Both the Spacewalk server and the Yum client application will reject unsigned packages by default. Although you can disable this feature, it makes more sense to work with digital signatures for security reasons. The rpm ‑‑resign RPM package command will sign the package for you; you must have GPG keys in place for the RPM tool. The ~/.rpmmacros file tells you the name and location of the key (Listing 5). To allow client systems to verify packages signed with this key, you need to deposit the public key on the Spacewalk server, preferably in /var/www/ html/pub, which any client can access. The following command exports the public key from the GPG keyring: gpg ‑‑armor ‑‑export tscherf@redhat.com > U /var/www/html/pub/rpm‑gpg‑key

To allow the existing client systems to access the software packages you just uploaded, you need to register

F e at u r e s

them with the Spacewalk server. Start by installing the Spacewalk Client Repository RPM on the clients. Fedora 12 systems have a matching RPM [10], as do RHEL5 and CentOS5 [11]. On RHEL and CentOS, you also need to install the RPM for the EPEL repository [12] because the client tool dependencies might not resolve correctly otherwise. The following command installs the Yum file on a 32-bit Fedora 12 system: rpm ‑Uvh http://spacewalk.redhat.com/ U yum/1.0/Fedora/12/i386/spacewalk‑ U client‑repo‑1.0‑2.fc12.noarch.rpm

Then, use Yum to install the client tools: yum install rhn‑client‑tools U rhn‑check rhn‑setup rhnsd m2crypto U yum‑rhn‑plugin

The easiest approach to registering a system on the server is to run the rhnreg_ks tool, which expects a registration key. You need to create the key up front on the Spacewalk server (Systems | Activation Key | Create Key). When you create a key, you can bind various resources to it, such as the Fedora 12 software channel just created here, or various configuration channels, if you have created some (Figure 3). Also, you can assign system groups to the key. Systems that use this key to register are granted access to the associated resources. To do so, specify the key you created during the registration process: E Listing 4: Answer File admin‑email = root@localhost ssl‑set‑org = Tuxgeek Org ssl‑set‑org‑unit = Tuxgeek OU ssl‑set‑city = Essen ssl‑set‑state = NRW ssl‑set‑country = DE ssl‑password = spacewalk ssl‑set‑email = root@localhost ssl‑config‑sslvhost = Y db‑backend=oracle db‑user=spacewalk db‑password=spacewalk db‑sid=xe db‑host=localhost db‑port=1521

Figure 3: Various resources can be bound to the registration key. Systems that use the key are given access to the associated resources.

w w w. a d m i n - m aga z i n e .co m

db‑protocol=TCP enable‑tftp=Y

Admin 01

F e at u r e s

Spacewalk

rhnreg_ks ‑‑serverUrl=U http://spacewalk.server.tld/XMLRPC U ‑‑activationkey=key

If all of this worked correctly, you will see the system in the Systems tab of the server web interface. Viewing the system’s properties should also show you the configured software channel. The easiest approach to checking whether access to the channel is working is to install a package from the channel. If this doesn’t work, one possible issue could be that the client system is not using the Spacewalk server’s CA certificate. The certificate is stored in http://spacewalk.server. tld/pub/ on the server and must be stored in /usr/share/rhn on the client side. The /etc/sysconfig/rhn/ up2date file needs a reference to the certificate. As before, you need to enter the name of the Spacewalk server. You only need to perform these steps on systems you have already installed. Any that you install from scratch via the Spacewalk server are automatically registered with the server as part of the installation process and can thus access the server immediately (Figure 4).

Kickstart Installation To automate the installation of new client systems, you need two pieces of information on the Spacewalk server. One of them is a kickstart file with details of how to install the new system, including partitioning, the software selection, and other settings that you would need to provide for a manual install. The easiest way to create a kickstart file is to select Systems | Kickstart | Profiles in the web front end. After checking out the overview of existing profiles, you can also create a new profile. The kickstart distribution must be specified as part of the profile file. This does not mean the RPM files that belong to the distribution

Figure 4: After completing the registration, the system appears in the Spacewalk server’s web interface.

you want to install, such as Fedora 12, but the basic installation files, like the Anaconda tool. The software repositories you synchronized earlier will not normally provide a kickstart distribution, and this means creating the distribution on the Spacewalk server. Again, just navigate to Systems | Kickstart | Distributions in the web interface and point to the required files. The easiest way to provide the files is to mount an installation CD/DVD for your preferred distribution via the loopback device: mount ‑o loop U /var/iso‑images/Fedora‑23‑i386‑DVD.iso U /var/distro‑trees/Fedora‑12

When you create a Fedora 12 kickstart distribution, you simply point the Spacewalk server to the /var/

distro‑trees/Fedora‑12 directory. If

all of this works out, just point to the distribution you created when you made the kickstart file. When a client system is installed from scratch, it will automatically pick up the right files from this source. Although there are a number of ways to install a Fedora 12 system from scratch, the easiest approach is to point any client PXE requests by your clients to the Spacewalk server with the next‑server command. Thanks to Cobbler [13] integration, the Spacewalk server has a TFTP server and any kickstart profiles that you have set up. To confirm this, you can type cobbler profile list at the command line. When you boot a client system via a PXE-capable network card, you will automatically see a list of the existing

Listing 5: GPG Configuration for RPM cat .rpmmacros %_signature gpg %_gpg_name Thorsten Scherf <tscherf@redhat.com>

Admin 01

Figure 5: The system properties give you a neat option for handling a variety of administrative tasks for a system via the Spacewalk server.

w w w. a d m i n - m aga z i n e .co m

Spacewalk

kickstart profiles. To install the client, simply select the required profile from the list. The client is then automatically registered on the Spacewalk server. Existing systems can easily be reinstalled using: koan ‑‑replace‑self U ‑‑server=Spacewalk‑Server U ‑‑profile=Kickstart‑Profile

This creates an entry in the system’s bootloader menu and automatically selects the entry when the system reboots.

System Management All of the systems registered on the Spacewalk server retrieve their software packages from this source, with no need to access external repositories. This method not only improves your security posture but also saves network bandwidth. With a registered system, you can customize various settings in the System Properties section (Figure 5). For example, you can assign new software or configuration channels, compare the installed software with profiles on other systems, or create snapshots as a backup that you can roll back later. Additionally, you can install new software or distribute configuration files from a centralized location. Thanks to the ability to assign registered systems to groups, you can point and click to do this for a large

number of systems. The rhnsd service on the systems queries the Spacewalk server at predefined intervals to check for new actions, such as software installations. When a system finds an action, it then executes it. If the osad service is enabled on the system, you can even run actions immediately without waiting for the polling interval to elapse. The client and the server then use the Jabber protocol for a continuous exchange. Finally, don’t forget the feature-rich Spacewalk API, which is accessible at http://Servername/rhn/apidoc/index. jsp on the installed server. This tool gives you access to a plethora of functions that are not available in the web interface. The API can be accessed with XMLRPC, which makes it perfect for your own Perl or Python scripts. A Python script [8] for creating a software channel is just one example of accessing the Spacewalk server via the API (Figure 6).

Conclusions Spacewalk gives administrators a very powerful tool for managing largescale Linux landscapes. It facilitates many daily tasks, such as the installation of software updates or uploading of configuration files. Advanced features, such as channel cloning, make it possible to put any software through a quality assurance process

Figure 6: An XML-RPC interface opens up a huge selection of Spacewalk server functions via the programmable API.

w w w. a d m i n - m aga z i n e .co m

F e at u r e s

before rolling it out to your production systems. Thanks to the comprehensive API, many tasks can also be scripted. n

Info [1] Spacewalk project homepage: [https://fedorahosted.org/spacewalk] [2] Spacewalk network ports: [http://magazine.redhat.com/2008/09/ 30/tips‑and‑tricks‑what‑tcpip‑ports‑are‑r equired‑to‑be‑open‑on‑an‑rhn‑satellite‑pr oxy‑or‑client‑system/] [3] RHEL5, CentOS5 Spacewalk Server Repos‑ itory RPM: [http://spacewalk.redhat.com/ yum/1.0/RHEL/5/i386/spacewalk‑repo‑1. 0‑2.el5.noarch.rpm] [4] Fedora12 Spacewalk Server Repository RPM: [http://spacewalk.redhat.com/yum/ 1.0/Fedora/12/i386/spacewalk‑repo‑1.0‑2. fc12.noarch.rpm] [5] Spacewalk Roadmap: [http:// fedorahosted.org/spacewalk/roadmap] [6] Spacewalk mailing list: [http://www.redhat.com/spacewalk/ communicate.html#lists] [7] Oracle XE: [http://www.oracle.com/ technology/software/products/database/ xe/htdocs/102xelinsoft.html] [8] Spacewalk API script for creating a software channel: [http://fedorahosted. org/spacewalk/attachment/wiki/ UploadFedoraContent/create_channel.py] [9] Repository sync: [http://fedorahosted. org/spacewalk/attachment/wiki/ UploadFedoraContent/sync_repos.py] [10] Fedora12 Spacewalk Client Reposi‑ tory RPM: [http://spacewalk.redhat. c om/yum/1.0/Fedora/12/i386/ spacewalk‑client‑repo‑1.0‑2.fc12.noarch. rpm] [11] RHEL5 and CentOS5 Client Repository RPM: [http://spacewalk.redhat.com/yum/ 1.0/RHEL/5/i386/spacewalk‑client‑repo‑1. 0‑2.el5.noarch.rpm] [12] EPEL Repository: [http://download. fedora.redhat.com/pub/epel/5/i386/ epel‑release‑5‑3.noarch.rpm] [13] Cobbler: [https://fedorahosted.org/cobbler/] The Author Thorsten Scherf is a Senior Consultant for Red Hat EMEA. You can meet him as a speaker at conferences. He is also a keen marathon runner whenever time permits.

Admin 01

Icinga

F E AT U R E S

Monitoring network computers with the Icinga Nagios fork

ÂŠ Alterfalter, 123RF.com

Server Observer Icingaâ&#x20AC;&#x2122;s developers grew weary of waiting for updates to the popular Nagios monitoring tool, so they started their own project. By Falko Benthin A server can struggle for many reasons: System resources like the CPU, RAM, or hard disk space could be overloaded, or network services might have crashed. Depending on the applications that run on a server, consequences can be dire â&#x20AC;&#x201C; from irked users to massive financial implications. Therefore, it is more important than ever in a highly networked world to be able to monitor the state of your

server and take action immediately. Of course, you could check every server and service individually, but it is far more convenient to use a monitoring tool like Icinga.

Nagios Fork Icinga [1] is a relatively young project that was forked from Nagios [2] because of disagreements regarding the pace and direction of development.

Icinga delivers improved database connectors (for MySQL, Oracle, and PostgreSQL), a more user-friendly web interface, and an API that lets administrators integrate numerous extensions without complicated modification of the Icinga core. The Icinga developers also seek to reflect community needs more closely and to integrate patches more quickly. The first stable version, 1.0, was released in December 2009, and the version

Listing 1: my_hosts.cfg 19 # Fileserver

01 # Webserver 02 define host{

20 define host{

host_name

webserver

host_name

fileserver

alias

languagecenter

alias

Fileserver

display_name

address

141.20.108.124

display_name

active_checks_enabled

address

192.168.10.127

passive_checks_enabled

active_checks_enabled

max_check_attempts

passive_checks_enabled

check_command

check-host-alive

max_check_attempts

check_interval

check_command

check-host-alive

retry_interval

check_interval

contacts

retry_interval

notification_period

contacts

notification_interval

notification_period

notification_options

notification_interval

}

notification_options

d,u,r

}

ADMIN 01

Server at language center

spz_admin 24x7

Fileserver

admin 24x7

W W W. A D M I N - M AGA Z I N E .CO M

Icinga

F e at u r e s

Table 1: States Option

Status

Server o

Down

Unreachable

Recovered

Services o

Warning

Critical

Recovered

Unknown

counter has risen every couple of months ever since. Icinga comprises three components: the core, the API, and the optional web interface. The core collects system health information generated by plugins and passes it via the IDOMOD interface to the Icinga Data Out Database (IDODB) or the IDO2DB service daemon. The PHP-based API accepts information from the IDODB and displays it in a web-based interface. Additionally, the API facilitates the development of add-ons and plugins. Icinga Web is designed to be a stateof-the-art web interface that is easily customized for administrators to keep an eye on the state of the systems they manage. At the time of writing, Icinga Web was in beta, and it has a couple of bugs that make it difficult to recommend for production use. If you only need to monitor a single host, Icinga is installed easily. Some

Figure 1: If the hosts are healthy, the admin is happy.

distributions offer binaries in their repositories, but if not, or if you prefer to use the latest version, the easy-tounderstand documentation includes a quick-start guide (for the database via libdbi with IDOUtils), which can help you set up the network monitor in next to no time for access at http://â&#x20AC;&#x2039; Server/â&#x20AC;&#x2039;icinga. The challenges come when you want to monitor a larger number of computers.

Icinga can monitor the private services on a computer, including CPU load, RAM, and disk usage, as well as public services like web, SSH, mail, and so on. The lab network environment consists of three computers, one of which acts as the Icinga server; the other two are a web server and a file server that send information to the monitoring server. Because no native approach lets you request information

Listing 2: my_services.cfg (Excerpt) 01 # SERVICE DEFINITIONS

notification_interval

02 d efine service{

notification_options

w,c,u,r

}

host_name

webserver

service_description

HTTP

18 d efine service{

active_checks_enabled

host_name

passive_checks_enabled

service_description

check_command

check_http

active_checks_enabled

max_check_attempts

3 ; how often to perform

passive_checks_enabled

the check before

check_command

check_ssh

Icinga notifies

max_check_attempts

fileserver, webserver SSH

check_interval

retry_interval

check_period

24x7

check_period

24x7

contacts

spz_admin

contacts

notifications_enabled

notification_period

}

w w w. a d m i n - m aga z i n e .co m

1 weekdays

admin 0

Admin 01

Icinga

F e at u r e s

Figure 2: Everything is working, but the NRPE plugin is causing problems.

externally about CPU load, RAM, or disk space usage, you need to install a verbose add-on, such as NRPE [3], on each machine. The remote Icinga server will tell it to execute the plugins on the local machine and

transmit the required information. Icinga sends the system administrator all the information needed and alerts the admin of emergencies. Advanced features that are a genuine help in daily work include groups, redundant

monitoring environments, notification escalation, and check schedules. Icinga differentiates between active and passive checks. Active checks are initiated by the Icinga service and run at times specified by the administrator. For a passive check, an external application does the work and forwards the results to the Icinga server, which is useful if you can’t actively check the computer (e.g., it resides behind a firewall). A large number of plugins [4] already exist for various styles in Nagios and Icinga. But before the first check, the administrator needs to configure the computers and the services to monitor in Icinga. The individual elements involved in a check are referred to as objects in Icinga. Objects include hosts, services, contacts, commands, and time slots. To facilitate daily work, you can group hosts, services, and contacts. The individual objects are defined in CFG files, which reside below Icinga’s etc/objects directory. The network monitor includes a number of sample definitions of various objects that ad-

Listing 3: commands.cfg (Excerpt) 01 # 'notify‑service‑by‑email' command definition 02 define command{ 03

command_name

notify‑service‑by‑email

command_line

/ usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /usr/bin/mail ‑s "**$NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTA

05 T E$ **" $CONTACTEMAIL$ 06

}

07 08 # 'check‑host‑alive' command definition 09 d efine command{ 10

command_name

check‑host‑alive

command_line

$USER1$/check_ping ‑H $HOSTADDRESS$ ‑w 3000.0,80% ‑c 5000.0,100% ‑p

12 5 13

}

Listing 4: timeperiods.cfg (Excerpt) 01 define timeperiod{

timeperiod_name 24x7

alias

24 Hours A Day, 7 Days A Week

sunday

00:00‑24:00

monday

00:00‑24:00

tuesday

00:00‑24:00

wednesday

00:00‑24:00

thursday

00:00‑24:00

friday

saturday

}

Admin 01

13 d efine timeperiod{ 14

timeperiod_name wochentags

alias

Robot Robot

monday

07:00‑17:00

tuesday

07:00‑17:00

wednesday

07:00‑17:00

00:00‑24:00

thursday

07:00‑17:00

00:00‑24:00

friday

07:00‑17:00

}

w w w. a d m i n - m aga z i n e .co m

Icinga

a language center (display_name) and is displayed accordingly in the web interface. To inform the administrator (con‑ tacts) when the server goes down (notification_options), I want Icinga to ping (check_command) the server every 5 minutes (check_interval). If the server is still down 60 minutes (noti‑ fication_interval) after notifying the administrator, I want to send another message. Icinga is capable of deciding whether a host is down or unreachable (see Table 1). However, to determine that

ministrators only need to customize. In principle, you can define multiple objects in a CFG file, but you can just as easily create separate files for each object in a directory below /path‑ to‑Icinga/etc/objects. Lines that start with a hash mark within an object definition are regarded as comments, as is everything within a line to the right of a semicolon.

Defining Hosts and Services Listing 1 provides a sample host definition. The host is the web server at

F e at u r e s

a host is unreachable, you have to define the nodes passed along the route to the host as parents – and this will only work if the routes for outgoing packets are known. The file server definition looks similar. Once the servers are defined, the administrator configures the respective services that Icinga will monitor (Listing 2), along with the matching commands (Listing 3), the intervals (Listing 4), and the stakeholding administrators (Listing 5). The individual configuration files have a similar structure. For each service, you

Listing 5: contacts.cfg (Excerpt) 01 d efine contact{ 02

contact_name

alias

host_notifications_enabled

service_notifications_enabled

host_notification_period

service_notification_period

host_notification_options

service_notification_options

host_notification_commands

service_notification_commands

24x7

}

icingaadmin Falko Benthin 1 1

d,u,r w,u,c,r notify‑host‑by‑email notify‑service‑by‑email root@localhost

Open Source Monitoring Conference 2010

Live at your PC

You can watch high quality speeches about monitoring topics from a Live-Stream on the 6th and 7th of October at your PC. You will be able to see all the slides as well.

Conference Topics: • NSClient++ (Michael Medin) • Clientless Windows Monitoring about WMI with Samba4 (Thomas Sesselmann) • The social seismograph at XING (Dr Johannes Mainusch) • RRDCacheD - how to escape the I/O hell (Sebastian Harl) • Monitoring at Thales Hengelo using Nagios (Pieter van Emmerik)

streaming.linux-magazin.de/en

ss to the including acce

video archive

CONTACT: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail: streaming@linux-magazin.de

Icinga

F e at u r e s

Figure 3: A manual check of commands in commands.cfg reveals the culprit.

need to consider the interval between checks. One useful feature is the ability to define time slots, within which Icinga will perform checks and, if necessary, notify the administrator. Here, time limitations or holidays can be defined. The contact configuration can include email addresses or cell phone numbers, but to integrate each contact with, for example, an Email2SMS gateway or a Text2Speech system (e.g., Festival), you need a matching command. Icinga can use macros, which noticeably simplifies and accelerates many tasks because you can use a single command for multiple hosts and services. Listings 2 and 3 give examples of macros. All services defined for monitoring the file server include a check_nrpe instruction with an exclamation mark. Each exclamation mark can be followed by an argument, which in turn is evaluated by the macros in other definitions. Macros are nested in $ signs. After creating the configuration files and storing them in etc/objects, you still need to tell Icinga by adding a new

Figure 4: Mail dispatched by Icinga is short and to the point.

Icinga with /etc/init.d/icinga re‑ start.

GUI and Messages Icinga works without a graphical interface, but it’s much nicer to have one. The standard interface can’t deny its Nagios ancestry, but it is clear-cut and intuitive. If everything is working, you’ll see a lot of green in the user interface (Figure 1), but if something goes wrong somewhere, the color will change and move closer and closer to red to reflect the status of the hosts or services (Figures 2 and 3). Status messages

are typically linked, so that clicking one takes you to more detailed information. If something is so drastically wrong that a message is necessary, Icinga will check its complex ruleset to see whether it should send a message and, if so, to whom (Figure 4). The filters through which the message passes check the following: whether notifications are required, if the problem occurred at a time when the host and service should be running, if messages should be sent for this service in the current time slot, and what the contacts linked to the service actually want. Each contact can

cfg_file=/ usr/local/icinga/etc/objects/ object.cfg

to the main configuration file, /etc/ icinga.cfg. After doing so, you should verify the configuration, /path‑to‑Icinga/bin/icinga ‑v /path‑to‑Icinga/etc/icinga.cfg;

assuming there are no errors, restart

Admin 01

Figure 5: Icinga Web beta was not entirely convincing. Version 1.0.3 is out now.

w w w. a d m i n - m aga z i n e .co m

Icinga

F e at u r e s

Figure 6: Network overview. If you need to monitor a large number of machines

Figure 7: The alert histogram, another useful gadget Icinga offers, shows peak

and have defined “parents,” you can also visualize the intermediate nodes.

trouble times.

define its own rules to stipulate when it wants to receive messages and for what status. If multiple administrators exist and belong to a single group, Icinga will notify all of them. Again, you can define individual notification periods so that each admin will be responsible for one period.

Interesting Features Icinga contains several interesting features that allow administrators to customize the network monitor to reflect their needs and system environment. For example, you can define distributed monitoring environments. If you need to monitor several hundred or thousand hosts, the Icinga server might conceivably run out of resources because every active check requires system resources. To take some of the load off the main server, Icinga can delegate individual tasks to auxiliary servers which, in turn, forward the results to a central server. Scheduling the checks can also help reduce this load. Instead of running all your active checks in parallel, you can let Icinga stagger them. Another interesting feature is the ability to escalate notifications. Not every administrator can be available and

w w w. a d m i n - m aga z i n e .co m

ready for action 24/7. If the contact that Icinga notifies does not respond within a defined period, Icinga can attempt to establish contact on another channel (e.g., a cell phone instead of email). If this notification fails as well, the case can be escalated to someone higher up the chain of responsibility – the team leader, for example.

Conclusions Icinga is a complex tool that provides valuable services whenever an administrator needs to monitor computers on a network. But don’t expect to be able to set up the network monitor in a couple of minutes of spare time; if all goes well, the installation and configuration will take at least a couple of hours. Once you have battled through the extensive configuration, you can reward yourself with an extended lunch break: If something happens that requires your attention, Icinga will tell you all about it. The traditional web interface is clear cut and packed with information; when this article went to print, however, the new interface wasn’t entirely convincing (Figure 5). The installation was tricky, the documentation required some imagination at times,

and the final results were disappointing. The interface was buggy and very slow under my, admittedly, not very powerful Icinga test server (Via C3, 800MHz, 256MB RAM). As a default, you need a new username and password for Icinga Web. That said, however, the current status does reveal some potential; it makes sense to check how the new interface is developing from time to time. The Icinga kernel is well and comprehensively documented and leaves no questions unanswered. Icinga also offers a plethora of useful gadgets, such as the status map (Figure 6) or the alert histogram (Figure 7), making the job of monitoring hosts less boring – at least initially. The depth of information that Icinga provides is impressive and promises an escape route for avoiding calls from end users. In short, Icinga is a useful tool that makes the administrator’s life more pleasant. n

Info [1] Icinga: [http://www.icinga.org/] [2] Nagios: [http://www.nagios.org/] [3] NRPE: [https://git.icinga.org/] [4] Nagios plugins: [http://sourceforge.net/ projects/nagiosplug/]

Admin 01

F e At u r e s

mysQL Forks and Patches

complete list of patches and notes on how to use them.

Reporting Modern MySQL Forks and Patches

Spoiled for Choice

mysQL is the standard solution for free relational database systems in web applications, but new forks, more storage engines, and patched versions muddy the water. now’s the time to take stock of the most important offerings. By caspar clemens mierau If your MySQL server is too slow, you have various approaches to solving the problem. Besides optimizing queries and indexes, reworking the configuration, and upgrading your hardware, moving to a customized version of the MySQL server can be a good idea. In recent years, so many patches, forks, and new storage engines have been released that it is hard to keep track of them. For hardworking developers and database administrators, this means a change from the simple choice of a standard MySQL distribution.

Little Band-aids Many enhancements for MySQL come from major corporations like Facebook [1] and Google [2], who run their ad services on top of MySQL or from MySQL specialists like Per-

Admin 01

cona [3], whose claim to fame is the “MySQL Performance Blog” [4] (standard reading for anyone interested in MySQL). The patches can be grouped into three categories: (1) reporting enhancements, (2) functional enhancements of the MySQL kernel and database engines, and (3) performance optimizations. In most cases, a combination of patches from all three categories will make the most sense. Moving to a database server that you patched and compiled yourself can be a daunting prospect. Thankfully, projects such as OurDelta [5] offer repositories with meaningfully patched and prebuilt MySQL packages for popular distributions like Debian, Ubuntu, and CentOS/RHEL. The patches I will be looking at in the rest of this article are a cross-section of the current OurDelta versions of the MySQL server; see their website for a

Extended reporting allows administrators to collect more granular information about the MySQL server’s behavior under load. Thus far, slow. log, which offers very little in the line of configuration options, might be your first port of call. However, its utility value is restricted to identifying individual, computationally intensive queries on the basis of the time they use – and non-used indexes. The MicroSlow patch offers new filters for a more targeted search for poorly formulated queries. Thus, it logs queries that are responsible for writing temporary tables to disk, performing complete table scans, or reading a freely defined minimum number of lines in a table. The mysqldumpslow Slow.log statistics tool, which is not very well known but is part of the MySQL standard distribution, has been modified to be able to read and evaluate the extended entries. Aggregated run-time statistics on usage behavior are equally as useful. The UserStats patch extends MySQL by adding statistics for users, clients, tables, and indexes. After enabling data collection in my.cnf or issuing the SQL SET GLOBAL userstat_running = 1 command at run time, four tables in the information_schema, USER_STA‑ TISTICS, CLIENT_STATISTICS, INDEX_ STATISTICS, and TABLE_STATISTICS,

are continually populated with data. The statistics can be accessed via the SHOW command. For example, SHOW TABLE_STATISTICS will give you a table-by-table evaluation of lines read and modified and indexes updated. Direct access to the statistics tables in information_schema is useful because they are accessed as normal tables, and you can target the results to manipulate. Listing 1 shows a query for the five tables with the most frequently read lines. This example taken from Live operations of the Rails-based Moviepilot movie community and the underlying movie database OMDB (both anonymized for

w w w. A d m i n - m AgA z i n e .co m

MySQL Forks and Patches

this evaluation) shows that read access to the images table is particularly frequent. The next step would be for the database user to experiment with code optimization or other changes to the table format to reduce access incidence and save time. The relatively write-intensive movies table has a suspiciously high write-access count and, at the same time, a large number of index updates. Because the statistics can be reset easily with FLUSH TABLE_STATISTICS, interval-based evaluation by means of a Munin plugin that you write yourself, or some similar method, would be your best bet. Retrospectively, you could investigate load peaks in relation to table access and modification.

New Functions New functions in the MySQL kernel give administrators additional options so that maintenance of the MySQL server is more secure and convenient. A typical task is to stop MySQL processes with the KILL command. Under load, you might see a process listed as Idle by SHOW PROCESSLIST start to handle a new query just at the moment you kill it. The “Kill if Idle” patch adds an option to kill a process only if it is doing nothing: KILL IF_ IDLE Process_Id. This saves you the embarrassment of accidentally killing a process while it is handling a query. LVM and ZFS snapshots are commonly regarded as the simplest methods for backing up an InnoDB database on the fly without interrupting operations. If this method is not an option for you and you are forced to rely on a legacy dump file, you need to make sure that the data on your MySQL server do not change while a file is being dumped. The legacy approach to doing this is FLUSH TABLES WITH READ LOCK. However, this might not be sufficient in the InnoDB case because background processes also write to the database. The InnoDB Freeze patch executes SET GLOBAL innodb_disallow_writes = 1

and then freezes all processes that write InnoDB data so you can create a backup. Afterward, SET GLOBAL in‑

w w w. a d m i n - m aga z i n e .co m

nodb_disallow_writes = 0 disables

the freeze.

Performance Enhancements Thanks to its support for transactions and line-based locking, InnoDB has developed into a modern alternative for the now fairly ancient MyISAM engine. Despite performance gains in write access thanks to line-based locking, the overhead for supporting transactions, foreign keys, and other functions (even if you don’t use them) costs valuable CPU cycles and hardware I/O resources. MySQL systems under heavy load thus need a perfectly configured and powerful InnoDB engine. A variety of performance-boosting patches are available for the version of InnoDB that ships with MySQLM; some of them are included in the OurDelta version. One that is worthy of mention is a reworked RW lock that improves locking behavior on multi-processor systems in particular. A description of all the improvements is beyond the scope of this article, but one thing is clear; patches for InnoDB exist that retain compatibility and offer transparent optimization of the engine. Typically, it is difficult to measure the performance gain in an objective way because the performance of the MySQL server will depend to a great extent on the hardware, configuration, and data it uses. The most exhaustive and reliable source is the MySQL Performance Blog [4], which regularly publishes test results. Improvements by selectively installing patches for the legacy InnoDB engine are regarded as a fairly conservative approach. The use of an alternative

F e at u r e s

database engine holds more promise. The InnoDB plugin and Percona XtraDB engines are becoming increasingly widespread.

The InnoDB Plugin The InnoBase InnoDB plugin is an ongoing development of the InnoDB engine that ships with MySQL [6]. Improvements include general optimization of CPU load and I/O access, a faster locking mechanism, extended configuration and reporting options, and optional table compression. MySQL 5.1 introduced the option of unloading the standard engine and replacing it with a different version. Starting with MySQL 5.1.38, MySQL additionally supplies the InnoDB plugin. As MySQL describes this as a release candidate, administrators do need to enable it manually. The official MySQL documentation describes the steps required to do so [7]. To benefit from the combination of distribution updates for the MySQL kernel and the latest functions and optimizations of the InnoDB plugin, it is a good idea to install the latest InnoDB plugin version from the InnoDB website. Ubuntu 10.04 comes with MySQL server version 5.1.41. The /usr/lib/ mysql/plugin/ houses an InnoDB plugin version 1.0.4 that is disabled by default. The InnoDB website has the current version, 1.0.6, which you can download and unpack. Then copy ha_innodb.so to the /usr/lib/mysql/ plugin/ directory. Because Ubuntu uses AppArmor to protect services by default, you need to disable or modify AppArmor to let you load content from the plugin directory by adding

Listing 1: Userstats Patch in Action mysql> select * from information_schema.TABLE_STATISTICS ORDER BY ROWS_READ DESC LIMIT 0,5; +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | TABLE_SCHEMA

| TABLE_NAME

| ROWS_READ

| ROWS_CHANGED | ROWS_CHANGED_X_INDEXES |

+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | moviepilot

| images

| 13138219791 |

14778 |

118224 |

| moviepilot

| events

3957858216 |

59964 |

359784 |

| moviepilot

| comments

2650553183 |

3408 |

20448 |

| moviepilot

| movies

2013076357 |

598505 |

7780565 |

| omdb

| log_entries

1106683022 |

2737 |

5474 |

+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

Admin 01

MySQL Forks and Patches

F e at u r e s

these lines to the /etc/apparmor.d/ usr.sbin.mysqld ruleset: /usr/lib/mysql/plugin/ r, /usr/lib/mysql/plugin/* mr,

Then restart Apparmor with the ser‑ vice apparmor restart command to enable the new rules. In my.cnf, you then need to enable the InnoDB engine and load the InnoDB plugin as shown in Listing 2. The InnoDB plugin documentation contains detailed information on this procedure. The MySQL server error log contains the message shown in Listing 3 after loading the InnoDB plugin. The documentation provides detailed information on the optimizations and new features offered in the InnoDB plugin. One thing that stands out against the rest is the new “BarListing 2: Loading the InnoDB Plugin into my.cnf [mysql] ignore_builtin_innodb plugin_load=innodb=ha_innodb.so; innodb_trx=ha_innodb.so;innodb_locks=ha_innodb.so; innodb_lock_waits=ha_innodb.so; innodb_cmp=ha_innodb.so;innodb_cmp_reset=ha_innodb.so; innodb_cmpmem=ha_innodb.so; innodb_cmpmem_reset=ha_innodb.so

Listing 3: With and Without the InnoDB Plugin MySQL server without InnoDB plugin: InnoDB: Started; log sequence number 0 44233

MySQL server with current InnoDB plugin: InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: highest supported file format is Barracuda. InnoDB Plugin 1.0.6 started; log sequence number 44233

racuda” file format. In contrast to the standard “Antelope” format, it stores the InnoDB tables in a compressed format. Although this could cost additional CPU cycles, it will give you huge I/O performance savings – depending on your data structure – because far fewer operations are required on disk/SSD. To discover whether it is worthwhile changing to Barracuda, you will need to measure performance. Tables with large text and blob fields in particular will benefit from compression. Incidentally, MyISAM has supported “compressed” tables for some time; however, you cannot modify compressed MyISAM tables in ongoing operations.

Percona XtraDB Percona XtraDB [8] takes things one step further than the InnoDB plugin. This storage engine is a merge of the current InnoDB plugin version with additional performance and feature patches. From a codebase point of view, XtraDB is thus the most innovative version of InnoDB. But don’t let the name worry you: XtraDB is an InnoDB engine. The new name simply serves to underline the major differences between it and the version of InnoDB that ships with MySQL. Your easiest approach to installing XtraDB is to resort to the MariaDB packages created by OurDelta. MariaDB [9] itself is a MySQL fork by the well-known MySQL developer Michael “Monty” Widenius. Widenius is working on a transactional alterna-

tive to MyISAM – the new Maria engine. At the same time, the MariaDB fork happily integrates XtraDB as a high-performance update to InnoDB. For the administrator, this means a whole lot more optimizations: a stateof-the-art MySQL version reworked by the MariaDB project and extended to include XtraDB (and Maria), along with additional patches courtesy of the OurDelta project. A conversion from InnoDB to XtraDB tables is not needed because XtraDB replaces the standard InnoDB engine, just as the InnoDB plugin does. Existing or new tables are automatically managed by XtraDB. A downgrade to the InnoDB plugin and the standard InnoDB engine is also possible. To see that XtraDB still refers to itself as “InnoDB,” you can call SHOW ENGINES – also, you will see the other modern engines, such as Maria and PBXT, here. Listing 4 shows the engines on a current MariaDB server. Tests on Live systems demonstrates that migrating to the MariaDB package is unproblematic for the most part. MyISAM tables are left unchanged; InnoDB tables continue to work. However, MySQL-specific configurations in my.cnf are interpreted in a fairly strict manner. sql‑mode=NO_ ENGINE_SUBSTITUTION,TRADITIONAL

will put MySQL in traditional mode, which handles many warnings as errors. For example, a typical Rails-style database migration failed because it did not use completely standardscompliant queries, such as setting default values for text and blob fields.

Listing 4: Engines on a MariaDB Server MariaDB [(none)]> show engines; +‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+ | Engine

| Support | Comment

| Transactions | XA

| Savepoints |

+‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+ | BLACKHOLE

| /dev/null storage engine (anything you write to it disappears) | NO

| NO

| MRG_MYISAM | YES

| YES

| Collection of identical MyISAM tables

| NO

| FEDERATED

| YES

| FederatedX pluggable storage engine

| YES

| NO

| YES

| MARIA

| YES

| Crash‑safe tables with MyISAM heritage

| YES

| NO

| CSV

| YES

| CSV storage engine

| NO

| MEMORY

| YES

| Hash based, stored in memory, useful for temporary tables

| NO

| ARCHIVE

| YES

| Archive storage engine

| NO

| MyISAM

| YES

| Default engine as of MySQL 3.23 with great performance

| NO

| InnoDB

| DEFAULT | Supports transactions, row‑level locking, and foreign keys

| YES

| PBXT

| YES

| NO

| High performance, multi‑versioning transactional engine

+‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+

Admin 01

w w w. a d m i n - m aga z i n e .co m

MySQL Forks and Patches

In non-traditional mode, MySQL ignores the default use and still runs the queries; in traditional mode, the query quits with an error. Thus, it is a good idea in many cases to change the line to sql‑mode=NO_ENGINE_SUB‑ STITUTION and then restart the server. This problem is not specific to MariaDB, simply a very restrictive configuration in line with the MySQL standard. Additionally, it makes sense to check whether programs compiled against Libmysqlclient-dev need to be recompiled against the current Libmariadbclient-device. In a Rails environment, this will affect Mysql-Gem. Besides the benefits of the InnoDB plugin described thus far, XtraDB also offers a considerable performance boost, as a variety of benchmarks with various setups proves [10]. Numerous additional functions make life easier for developers and administrators – and don’t forget the ability to

write out the InnoDB buffer to disk. If you do need to restart the MySQL or MariaDB server, the InnoDB engine loses its valuable buffer pool in RAM. Depending on your configuration and the application, the pool can be several gigabytes and might be filled in the course of hours. Storing the buffer pool before quitting and loading it again after restarting will save valuable warmup time, which you would notice as slow response on the part of the database server. Listing 5 shows the commands and returns for storing and loading the buffer pool.

Drizzle

F e at u r e s

on the code of the not-yet-released MySQL 6.0 and mainly pursues the goals of removing unnecessary functions and reducing complexity. The developers really have been radical in the features they eradicated: E Listing 5: Storing and Loading the Buffer Pool // storing the buffer pool MariaDB [(none)]> select * from information_schema.XTRADB_ADMIN_COMMAND /*!XTRA_LRU_DUMP*/; +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | result_message

+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | XTRA_LRU_DUMP was succeeded. | +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ // loading buffer pool and MariaDB [(none)]> select * from information_schema.

At this point, I’ll take a quick look at a new development, Drizzle [11]. According to the project, the fork is a return to the original MySQL values: simplicity, reliability, and performance. Drizzle was originally based

XTRADB_ADMIN_COMMAND /*!XTRA_LRU_RESTORE*/; +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | result_message

+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+ | XTRA_LRU_RESTORE was succeeded. | +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

Make the most of your Data Center Watch our video archiv of the Open Source Data Center Conference – each session complete with slides. Hot Topics • High Availability • Computer Clusters • Load Balancing • Configuration Management • Security Management

streaming.linux-magazin.de/en

w w w. a d m i n - m aga z i n e .co m Admin 01 23 Contact: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail: streaming@linux-magazin.de

MySQL Forks and Patches

F e at u r e s

MySQL Community Edition

MySQL 5.0

Percona, Google, and other patches

MySQL 5.1

MariaDB 5.1

InnoDB plugin

MySQL 6.0

Percona XtraDB

Drizzle

Maria Percona, Google, and other patches

PBXT BlitzDB FederatedX

OurDelta MySQL 5.0

OurDelta MariaDB 5.1

perfect solution for their requirements profile, this does involve a considerable amount of overhead. At the same time, the installation of the server is no more complex than using distribution packages thanks to prepacketized software. Work is already in progress on integrating modern forks into distribution repositories [14]. The reward for all this effort will ideally be a noticeably faster database server that helps reduce hardware and development costs. n

Figure 1: The MySQL development community now has many forks.

storage engines such as Federated and Merged have been removed; others, such as CSV and MyISAM, have been demoted to temporary engines. Modern engines such as XtraDB are maintained in a separate branch. The standard engine for Drizzle is InnoDB. However, this does not mean that data dumped from a classical MySQL server with InnoDB tables can be integrated without problems, because Drizzle has also eradicated many field types, such as TINYINT, TINYTEXT, and YEAR. Migrating to Drizzle thus means architectural changes to your database design. Although a change from TINYINT to INT could simply mean searching and replacing occurrences in a dump file, the lack of a YEAR field can have a more serious effect on existing applications. A generic solution for the migration does not exist. On a more positive note, Drizzle offers totally new replication mechanisms. One feature that stands out is the ability to perform rabbit replication to NoSQL databases such as Voldemort [12] or services such as Memcached [13]; thus, you would be able to provision a variety of back ends automatically from a central location. As a state-of-the-art, high-performance, non-transactional database engine, the Drizzle project is working on BlitzDB, which will be positioned as an alternative to MyISAM.

Conclusions The community’s response to the lethargic integration of patches into the

Admin 01

Community Edition of MySQL server is to launch new and active projects (Figure 1). Existing MySQL 5.0 installations can be replaced easily by the OurDelta MySQL 5.0 build, which accelerates the server, thanks to performance patches, and offers advanced reporting functionality so the administrator can plan further steps on the basis of run-time statistics. Installations of version 5.1 can benefit from the latest optimizations by installing the current InnoDB plugin – ideally without needing to rebuilding. Migrating to the state-of-the-art MariaDB, which outperforms the InnoDB plugin in performance tests, turns out to be more effective. Luckily, MariaDB is packetized by the OurDelta project, which also adds a number of additional patches. The Drizzle database, with its simplified variant of InnoDB, is still at a very early stage. The Maria engine also represents a possible fast future alternative to the classical combination of MyISAM/InnoDB; however, you need to perform extensive checks before using it. Both projects’ engines require architectural changes to the database system and the program code that accesses it, in contrast to the InnoDB plugin, XtraDB, and popular MySQL patches. Administrators and developers are put in an ambivalent situation: Although it has become inevitable for administrators to concern themselves with alternative MySQL patches, engines, and forks and, ideally, to deploy benchmarks to discover the

Info [1] MySQL on Facebook: [https://launchpad.net/mysqlatfacebook] [2] Google patches: [http://code.google. com/p/google‑mysql‑tools/wiki/ Mysql5Patches] [3] Percona patches: [http://www.percona. com/docs/wiki/patches:start] [4] MySQL performance blog: [http://www. mysqlperformanceblog.com/] [5] Patches from OurDelta: [http://ourdelta.org/patches] [6] InnoDB plugin: [http://www.innodb.com/ products/innodb_plugin/features/] [7] Installing the InnoDB plugin: [http://dev.mysql.com/doc/refman/5.1/en/ innodb.html] [8] Percona XtraDB: [http://www.percona. com/docs/wiki/percona‑xtradb:start] [9] MariaDB: [http://askmonty.org/wiki/MariaDB] [10] Benchmarks: InnoDB plugin and XtraDB vs. InnoDB: [http://www.mysqlperformanceblog.com/ 2010/01/13/innodb‑innodb‑plugin‑vs‑xtrad b‑on‑fast‑storage/] [11] Drizzle: [http://drizzle.org/] [12] Project Voldemort: [http://project‑voldemort.com] [13] “Memcached” by Tim Schürmann, Linux Magazine, November 2009, pg. 28 [14] MariaDB in Ubuntu: [https://wiki.ubuntu. com/Lucid‑MariaDB‑Inclusion] The Author Caspar Clemens Mierau’s “Screenage” project provides consultancy services to Rails and PHP portals such as moviepilot.de, omdb.org, and Artfacts.Net. Caspar Clemens works as a free‑ lance author and is collecting literature for his thesis on development environments.

w w w. a d m i n - m aga z i n e .co m

Exchange 2010

F e at u r e s

Microsoft Exchange 2010: The Highlights

Message Exchange For years, Exchange has been the standard in-house server solution for all messaging tasks on Windows. This article introduces the highlights of the new Exchange version 2010. By Björn Bürstinghaus Exchange 2010 [1] is the latest generation of the Microsoft email server, and it comes with a whole bunch of new and useful functions for administrators and users alike [2]. The version is even interesting for companies currently using Exchange 2007: Besides an archiving function, the new Exchange also integrates an intelligent SMS gateway architecture, thus removing the need for expensive third-party add-ons.

Improved Management for Admins Many functions that posed tedious Exchange Management Shell tasks for administrators in Exchange 2007 have now been integrated into the console. For example, you can create new certificates or view the current crop of certificates at the console, without needing to compose your own PowerShell request (Figure 1). This version of Exchange also offers a console view of the number of licenses used in the company. This new transparency allows administrators to identify cases in which a com-

w w w. a d m i n - m aga z i n e .co m

pany is using more licenses than it has purchased. Giving users permission to create distribution groups and their members is also new; this task can be handled in the new, web-based Exchange Control Panel (ECP). In ECP, users can modify their own Active Directory information, such as cellphone numbers or addresses, without needing to contact IT to do so. In this way, ECP will probably make a major contribution to reducing IT costs in the enterprise. Whereas Exchange 2007 restricted bulk changes to the PowerShell, Exchange 2010 finally lets administrators make bulk changes in the management console, so you can run a task simultaneously against multiple mailboxes, which wasn’t possible in previous versions (Figure 2). The new transport cache prevents email messages transmitted via SMTP from being deleted until the downstream node confirms that they have been forwarded successfully. In other words, if a hub transport server in your company goes down, the transport cache will retain the messages until the server becomes available

again or the transport rules are modified.

New Features from the User’s Point of View Exchange 2010 lets end users search multiple mailboxes and send the results as a PST export to another person. The Unified Messaging function integrates a voicemail function for each user in Exchange and supports speech-to-text conversion, making it possible to display a voicemail as well as listen to the attached audio file. The ability to display messages as conversations both in Microsoft Office Outlook 2010 and in the Outlook Web App gives users a clearer view of their email folders. Users do not need to check the sent items for email they sent to the same person on the same subject, which can save time. The Outlook Wep App’s premium functions were previously restricted to Internet Explorer. Exchange 2010 lifts this restriction. The new Outlook Web App now supports premium functions in Mozilla Firefox and Apple Safari (Figure 3). E

Admin 01

F e at u r e s

Exchange 2010

G Figure 2: Bulk Edit lets admins make changes to multiple recipients. F Figure 1: Wizard for creating new certificates

Another new feature is the approach to migrating mailboxes. Previously, a user’s mailbox was switched offline during migration to another mailbox database, which meant delays of a few minutes to several hours. Now, while the mailbox is in transit, the user can stay online. Exchange copies the total content of the mailbox and then synchronizes any changes that occurred during the migration.

Archive Mailbox Centralized archiving of email previously relied on third-party add-ons. Again, Exchange 2010 puts an end to this. In addition to a normal mail-

box, the administrator can create an archive for each member of staff, although this is a premium feature and does require an Exchange Premium Client Access License for each user (Figure 4). Once enabled, the archive is automatically displayed in the Outlook 2010 and the Outlook Web App folder navigation, in addition to the normal mailbox. This solution allows administrators to create an Exchange solution quickly that lets end users restore messages they deleted from their mailboxes without having to call the help desk. All you need for this is the journaling function, the enterprise-wide policies to use for automatically synchroniz-

ing or moving email from the mailbox to the archive mailbox, and a matching retention policy that prevents deletion. Exchange 2010 currently does not offer the same feature set as archiving solutions by third parties such as MailStore Server or GFI Mail Archiver; however, the functionality that is currently available does relieve the burden on mailboxes, and it puts an end to the “PST Hell” that some administrators rightly complained of.

Text Messages If a company’s employees use smartphones with Windows Mobile 6.5,

Figure 4: The mailbox configuration now supports Figure 3: The Outlook Web App implements a mail client in the web browser.

Admin 01

Archiving for email.

w w w. a d m i n - m aga z i n e .co m

they can now synchronize text messages with their Exchange mailbox, along with email messages, contacts, appointments, and tasks. Exchange previously did not offer an option for sending or receiving text messages (unless you relied on a third-party product), and SMS gateways were the only option for integrating text message functionality. Unfortunately, this meant that the SMS gateway number was the same for the whole company, rather than having extensions provided for individual users. Exchange 2010, in combination with the Windows Mobile 6.5 platform, uses the smartphone as the SMS gateway and thus supports composing or reading personalized text messages with Outlook 2010 or Outlook Web App. When you send a text message via Exchange, the message is forwarded by the server via an ActiveSync interface to the Smartphone, which then uses the employee’s cellphone number to send the message; this approach is far more flexible than using a centralized gateway.

Improved Availability The 24/7 availability of services plays an important role in many large corporations that need permanent availability across multiple time zones because of globalization. Exchange 2010 takes this one step further than the previous versions: Windows Cluster lets you group up to 16 mailbox servers in a cluster, which means databases can be replicated on up to 15 additional servers.

w w w. A d m i n - m AgA z i n e .co m

Also, it provides an option of integrating the logfiles for a database at a later stage. You can refer to the help [3] or forum [4] websites for additional information.

Is It Worthwhile? If you are planning to invest in an email archiving solution or an SMS gateway, you might consider moving to Exchange 2010, instead, because it already integrates many of these functions. The new version also improves and facilitates the experience of integrating subsidiaries, so a change could also help reduce your company’s IT costs. The test version of Exchange supports virtualization, thus giving those who are interested in migrating a chance to familiarize themselves with the new functions and options before they invest in new hardware. ■

Info [1] Exchange Server 2010 product page: [http://technet.microsoft. com/en-us/exchange/dd203064. aspx] [2] Exchange Server 2010 new features: [http://technet. microsoft.com/en-us/library/ dd298136.aspx] [3] Exchange Server 2010 help: [http://technet.microsoft.com/ en-us/library/bb124558.aspx] [4] Exchange Server 2010 Forum: [http://social.technet. microsoft.com/Forums/en-US/ exchange2010/threads] The Author Björn Bürstinghaus is a systems administrator with simyo GmbH in Düsseldorf, Germany. In his leisure time, he runs Björn’s Windows blog, a blog on Microsoft Windows topics. You can find his blog at [http://blog. buerstinghaus.net] (in German).

Admin 01

F e at u r e s

Backup Software

Backup tools to save you time and money

Safety Net One small oversight can cost you hours of extra work and your company thousands of dollars. Here are a few backup tools to help you recover gracefully. By James Mohr Computers are of little value if they are not doing what they’re supposed to, whether they’ve stopped working or they’re not configured correctly. Redundant systems or standby machines are common methods of quickly getting back to business, but they are of little help if the problem is caused by incorrect configuration and those configuration files get copied to the standby machines. Sometimes, the only solution is to restore from backup. Backing up all of your data every day is not always the best approach. The amount of time and the amount of storage space required can be limiting factors. On workstations with only a handful of configuration files and few data files, it might be enough to store these files on external media (such as USB drives). Then, if the system should crash, it could be simpler to reinstall and restore the little data you have. Doing a full system restore often takes longer. Because each system is different, no single solution is ideal. Even if you find a product that has all the features you could imagine, the time and effort needed to administer the software might be restrictive. Therefore, knowing the most significant aspects of backups and how these can be

Admin 01

addressed in the software are key issues when deciding which product to implement (see also the “Support” box).

How Strong Is Your Parachute? A simple copy of your data to a local machine with rsync started from a cron job once a day is an effective way to back up data when everything comes from a single directory. This method is too simple for businesses that have much more and many different kinds of data – sometimes so much data that it is impractical to back it all up every day. In such cases, you need to make decisions about what to back up and when (see the “Backup Alternatives” box). In terms of what to back up and how often, files increase in importance on the basis of how difficult they are to recreate. The most important files are your “data,” such as database files, word-processing documents, spreadsheets, media files, email, and so forth. Such files would be very difficult to recreate from scratch, so they must be protected. Configuration information for system software, such as Apache or your email server, typically changes frequently, but these

files still need to be backed up. Sometimes, reinstalling is not practical if you have to install a lot of patches, or it might be impossible if you have software that is licensed for a particular machine and you cannot simply reinstall the OS without obtaining a new license key. How often you should back up your files is the next thing to determine. For example, database files could be backed up to remote machines every 15 minutes, even if the files are on cluster machines with a hardware RAID. Local machines could be backed up twice a day: The first time to copy the backup from the previous day to an external hard disk, and the second time to create a new full backup. Then, once a month, the most important data could be burned to DVD (see also the “Incremental vs. Differential” box).

What Color Is your Parachute? Individuals and many businesses are not concerned with storing various versions of files over long periods of time. However, in some cases, it might be necessary to store months or even years worth of backups, and the only effective means to store them is on removable media (e.g., a tape drive). Further requirements to store backups off-site often compound the problem. Although archives can be kept on external hard disks, it becomes cumbersome when you get into terabytes of data. In deciding which type of backup medium you need to use, you have to consider many things. For example, you need to consider not only the total amount of data but also how many machines you need to back up. Part of this involves the ability to distinguish quickly between backups from different machines. Moving an external hard disk between two machines might be easiest, but with 20 machines, you should definitely consider a centralized system. Here, too, you must consider the speed at which you can make a backup and possibly recover your

w w w. a d m i n - m aga z i n e .co m

Backup Software

data. One company I worked for had so much data it took more than a day to back up all of the machines. Thus, full backups were done over the weekend, with incrementals in between. Also, in a business environment with dozens of machines, trying to figure out exactly where the specific version of the data resides increases the recovery time considerably. Finally, you must also consider the cost. Although you might be tempted to get a larger single drive because it is less expensive than two drives that are only half as big, being able to switch between two drives (or more) adds an extra level of safety if one fails. Furthermore, you could potentially take one home every night. If you are writing to tape, an extra tape drive also increases safety; it can also speed up backups and recovery.

Which Tape? Some companies remove all of the tapes after the backup is completed and store them in a fireproof safe or somewhere off-site. This means that when doing incremental backups, the most recent copy of a specific file might be on any one of a dozen tapes. Naturally, the question becomes, “Which tape?” (see also the “Whose Data” box). To solve this problem, the backup software must be able keep track of which version of Incremental vs. Differential Because of the amount of data, businesses frequently have a two-tiered backup scheme. Once a week, a full backup is done (of every single file); on subsequent days, backups are done of only those files that have changed. This approach is referred to as an incremental backup. Although it saves media, it potentially takes more time to recover. With this method, you first need to restore the full backup and, depending on which files have changed, you might need to access every single incremental backup. One alternative is a differential backup, which stores only files that have been changed since the last full backup. This has the advantage of saving time compared with an incremental backup, because you need to restore from, at most, two backups.

w w w. a d m i n - m aga z i n e .co m

F e at u r e s

Backup Alternatives If you are running Linux and your software repositories are configured properly, a number of backup applications are available through your respective installation tool (e.g., YaST, Synaptic). In fact, I found more than two dozen products that have defined themselves in one way or another as a backup tool (not counting those explicitly for backing up databases). Here are a few important questions to ask about your backup software: n Is your hardware supported? n How does the software deal with database backups? n Can you do a directed recovery (i.e., to a different directory)? n Can the software verify the data after a backup and restore? n Can the software write to multiple volumes? n Do you really need all of the features? n Can the software do a backup of a remote system?

which file is stored where (i.e., which tape or disk). Once a software product has reached this level, it will typically also be able to manage multiple versions of a given file. Sometimes you will need to make monthly or even yearly backups, which are then stored for longer periods of time. (This setup is common when you have sensitive data like credit card or bank information.) To prevent the software from overwriting tapes that it shouldn’t, you should be able to define a “recycle time” that specifies the minimum amount of time before the media can be reused. Because not all backups are the same and not all companies are the same, you should consider the ability of the software to be configured to your needs. If you have enough time and space, software that can only do a full backup might be sufficient. On the other hand, you might want to be able to pick and choose just specific directories, even when doing a “full” backup. Support One consideration that is often overlooked is the amount of support available for your product. Commercial support might be necessary if implementing the backup solution for a company. However, the amount of free support (forums, mailing lists) can be an issue. When considering open source software of any kind for a business, I always suggest taking a good look at the product’s website. If the product has not been updated in three years, you might want to look elsewhere. If forums have few posts and most are unanswered, you likely won’t get your questions answered either.

Many of the products I looked at have the ability to define “profiles” (or use a similar term). For example, you define a Linux MySQL profile, assign it to a subset of your machines, and the backup software automatically knows which directories to include and which to ignore. The Apache profile, for example, has a different set of directories. This might also include a pre-command that is run immediately before the backup, then a postcommand that is run immediately afterward.

Storage How is the backup information stored? Does the backup software have its own internal format or does it use a database such as my SQL? The more systems you back up, the more you need a product that indexes which files are saved and where they are saved as well. Unless you are simply doing a complete backup every night to one destination for one machine (i.e., one tape or remote diWhose Data? One important aspect is the ability to write data from different sources to specific media. For example, where I work, each customer is assigned specific tapes (often referred to as a “pool”). With the use of labels written to the tape, the software can tell which tape belongs to which pool, so that data from different environments is not mixed. This scheme is very useful if, for example, one customer wants weekly backups stored off-site and another customer frequently requests the backup tapes to load them into a local test system.

Admin 01

F e at u r e s

Backup Software

rectory), finding the right location for a given file can be a nightmare. Even if you are dealing with just a few systems, administration of the backups can become a burden. This leads into the question of how easy it is to recover your data. Can you easily find files from a specific date if there are multiple copies? How easy is it to restore individual files? What about all files changed on a specific date? Depending on your business, you might have legal obligations in terms of how long you are required to keep certain kinds of data. In some cases, it might be a matter of weeks; in other cases, it can be 10 years or longer. Can you recover data from that long ago? Even if it’s not required by law, having long-term backups is a good idea. If you accidentally delete something and don’t notice it has happened for a period longer than your backup cycle, you will probably never get your data back. How easy is it for your backup software to make full backups at the end of each month – for example, to ensure that the media does not get overwritten?

Scheduling If your situation prevents you from doing complete backups all the time, consider how easy it is to schedule them. Can you ensure that a complete backup is done every weekend, for example? Also, you need to consider the scheduling options for the respective tool. Can it start backups automatically? Is it dependent on some command? Is it simply a GUI for an existing tool, and all the operations need to be started manually? Just because a particular operating system has no client does not mean you are out of luck: You can mount filesystems using Samba or NFS and then back up the files.

rsync Sometimes you do not need to look farther than your own backyard. Rsync is available for all Linux distributions, all major Unix versions, Mac

Admin 01

OS X, and Windows. With a handful of machines, configuring rsync by hand might be a viable solution. If you prefer a graphical interface, several different graphical interfaces are available. In fact, many different applications rely on it to do the backup. The rsync tool can be used to copy files either from a local machine to a remote machine or the other way around. A number of features also make rsync a useful tool for synchronizing directories (which is part of its name). For example, rsync can ignore files that have not been changed since the last backup, and it can delete files on the target system that no longer exist on the source. If you don’t want existing files to be overwritten but still want all of the files to be copied, you can tell rsync to add a suffix to files that already exist on the target. The ability to specify files and directories to include or exclude is very useful when doing backups. This can be done by full name or with wildcards, and rsync allows you to specify a file that it reads to determine what to include or exclude. When determining whether a file is a new version or not, rsync can look at the size and modification date, but it can also compare a checksum of the files. A “batch mode” can be used to update multiple destinations from a single source machine. For example, changes to the configuration files can be propagated to all of your machines without having to specify the change files for each target. Rsync also has a GUI, Grsync [Figure 1].

luckyBackup At first, I was hesitant to go into details about luckyBackup [1], because it is still a 0.X version and it has a somewhat “amateurish” appearance. However, my skepticism quickly faded as I began working with it. luckyBackup is very easy to use and provides a surprising number of options. Despite its simplicity, luckyBackup had the distinction of winning third place in the 2009 SourceForge Community Choice Awards as a “Best New Project.”

The repository I used had version 0.33, so I downloaded and installed that (although v0.4.1 is current). The source code is available, but various Linux distributions have compiled packages. Describing itself as a backup and synchronization tool, luckyBackup uses rsync, to which it passes various configuration options. It provides the ability to pass any option to rsync, if necessary. Although it’s not a clientserver application, all it needs is an rsync connection to back up data from a remote system. When you define which files and directories to back up, you create a “profile” that is stored under the user’s home directory. Profiles can be imported and exported, so it is possible to create backup templates that are copied to remote machines. (You still need the luckyBackup binary to run the commands.) Each profile contains one or more tasks, each with a specific source and target directory, and includes the configuration options you select [Figure 2]. Thus, it is possible to have different options for different directories (tasks), all for a single machine (profile). Within a profile, the tool makes it easy to define a restore task on the basis of a given backup task. Essentially, this is the reverse of what you defined for the backup task, but it is very straightforward to change options for the restores, such as restoring to a different directory. Scheduling of the backup profiles is done by cron, but the tool provides a simple interface. The cron parameters are selected in the GUI; you click a button, and the job is submitted to cron. A console, or command-line mode, allows you to manage and configure your backups, even when a GUI is not available, such as when connecting via ssh. Because the profiles are stored in the user’s home directory, it would be possible for users to create their own profile and make their own backups. Although I would not recommend it for large companies (no insult in-

w w w. a d m i n - m aga z i n e .co m

Backup Software

tended), luckyBackup does provide a basic set of features that can satisfy home users and small companies.

Amanda Initially developed internally at the University of Maryland, the Advanced Maryland Automatic Network Disk Archiver (Amanda) [2] is one of the most widely used open source backup tools. The software development is “sponsored” by the company Zmanda [3], which provides an “enterprise” version of Amanda that you can purchase from the Zmanda website. The server only runs on Linux and Solaris (including OpenSolaris), but Mac OS X and various Windows versions also have clients. The documentation describes Amanda has having been designed to work in “moderately sized computer centers.” This and other parts of the product description seem to indicate the free, community version might have problems with larger computer centers. Perhaps this is one reason for selling an “enterprise” version. The latest version is 3.1.1, which came out in June 2010, but it just provided bug fixes. Version 3.1.0 was released in May 2010. Amanda stores the index of the files and their locations in a text file. This naturally has the potential to slow

Figure 1: Grsync – A simple front end to rsync.

w w w. a d m i n - m aga z i n e .co m

down searches when you need to recover specific files. However, the commercial version uses MySQL to store the information. Backups from multiple machines can be configured to run in parallel, even if you only have one tape drive. Data are written to a “holding disk” and from there go onto tape. Data are written with the use of standard (“built-in”) tools like tar, which means data can be recovered independently from Amanda. Proprietary tools typically have a proprietary format, which often means you cannot access your data if the server is down. Scheduling is also done with a local tool: cron. Commands are simply started at the desired time with the respective configuration file as an argument. Amanda supports the concept of “virtual tapes,” which are stored on your disk. These can be of any size smaller than the physical hard disk. This approach is useful for splitting up your files into small enough chunks to be written to DVD, or even CD. Backups are defined by “levels,” with 0 level indicating a full backup; the subsequent levels are backups of the changes made since the last n - 1 or less. The wiki indicates that Amanda’s scheduling mechanism uses these levels to implement a strategy

F e at u r e s

of “optimization” in your backups. Although optimization can be useful in many situations, the explanation is somewhat vague about how this is accomplished – and vague descriptions of how a system makes decisions on its own always annoy me. One important caveat is that Amanda was developed with a particular environment in mind, and it is possible (if not likely) that you will need to jump through hoops to get it to behave the way you want it to. The default should always be to trust the administrator, in my opinion. If the admin wants to configure it a certain way, the product shouldn’t think it knows better. For example, you should determine whether the scheduling mechanism is doing full backups at times other than when you expect or even want. In many cases, large data centers do full backups on the weekend when there is less traffic and not simply “every five days.” If your installation has sudden spikes in data, Amanda might think it knows better and change the schedule. Although such situations can be addressed by tweaking the system, I have a bad feeling when software has the potential for doing something unexpected. After all, as a sys admin, I was hired to think, not simply to push buttons. To make things easier

Figure 2: luckyBackup profile configuration.

Admin 01

F e at u r e s

Backup Software

in this regard, Zmanda recommends their commercial enterprise product. Although Amanda has been around for years and is used by many organizations, I was left with a bad taste in my mouth. Much of the information on their website was outdated, and many links went to the commercial Zmanda website, where you could purchase their products. Additionally, a page with the wish list and planned features is as old as 2004. Although a note states that the page is old, there is no mention of why the page is still online or any explanation of what items are still valid. Half of the pages on the administration table of contents (last updated in 2007) simply list the title with no link to another page. Also, I must admit I was shocked when I read the “Zmanda Contributor License Agreement.” Amanda is an open source tool, which is freely available to everyone. However, in the agreement “you assign and transfer the copyrights of your contribution to Zmanda.” In return, you receive a broad license to use and distribute your contribution. Translated, this means you give up your copyright and not simply give Zmanda the right to use it, which also means Zmanda is free to add your changes to their commercial product and make money off of it – and all you get is a T-shirt!

Areca Backup Sitting in the middle of the features spectrum and somewhat less well known is Areca Backup [4]. Running from either a GUI [Figure 3] or a command-line interface, Areca provides a simple design and a wide range of features. The documentation says it runs on all operating systems with Java 1.4.2 or later, but only Linux and Windows packages are available for download. Installing it on my Ubuntu systems was no problem, and I could not find any references to limitations with specific distributions or other operating systems. Areca is not a client-server application, but rather a local filesystem backup. The Areca website explicitly

Admin 01

states that it cannot do filesystem or disk images, nor can it write to CDs or DVDs. Backups can be stored on remote machines with FTP or FTPS, and you can back up from remotely mounted filesystems, but with no remote agent. Areca provides no scheduler, so it expects you to use some other “task-scheduling software” (e.g., cron) to start your backup automatically. In my opinion, the interface is not as intuitive as others, and it uses terminology that is different from other backup tools, making for slower progress at the beginning. For example, the configuration directory is called a “workspace” and a collection of configurations (which can be started at once) is a “group,” as opposed to a collection of machines. Areca provides three “modes,” which determine how the files are saved: standard, delta, and image. The standard mode is more or less an incremental backup, storing all new files and those modified since the last backup. The delta mode stores the modified parts of files. The image mode is explicitly not a disk image; basically, it is a snapshot that stores a unique archive of all your files with each backup. The standard backups (differential, incremental, or full) determine which files to include. The GUI provides two views of your backups. The physical view lists the archives created by a given target.

The logical view is a consolidated view of the files and directories in the archive. Areca is able to trigger pre- and postactions, like sending a backup report by email, launching shell scripts before or after your backup, and so forth. It also provides a number of variables, such as the archive and computer name, which you can pass to a script. Additionally, you can define multiple scripts and specify when they should run. For example, you can start one script when the backup is successful but start a different one if it returns errors. Areca provides a number of interesting options when creating backups. It allows you to compress the individual files as well as create a single compressed file. To avoid problems with very large files, you can configure the backup to split the compressed archive into files of a given size. Also, you can encrypt the archives with either AES 128 or AES 256. One aspect I liked was the ability to drop directories from the file explorer directly into Areca. The Areca forum has relatively low traffic, but posts are fairly current. However, I did see a number of recent posts remain unanswered for a month or longer. The wiki is pretty limited, so you should probably look through the user documentation, which I found to be very extensive and easy to understand.

Figure 4: Bacula admin tool.

w w w. a d m i n - m aga z i n e .co m

Backup Software

Two “wizards” also ease the creation of backups. The Backup Shortcut wizard simplifies the process of creating the necessary Areca commands, which are then stored in a script that you can execute from the command line or with cron. The Backup Strategy wizard generates a script containing a set of backup commands to implement a specific strategy for the given target. For example, you can create a backup every day for a week, a weekly backup for three weeks, and a monthly backup for six months.

Bacula The Backup Dracula “comes by night and sucks the vital essence from your computers.” Despite this somewhat cheesy tag line, Bacula [5] is an amazing product. Although it’s a newer product than Amanda, I definitely think it surpasses Amanda in both features and quality. To be honest, the setup is not the pointand-click type that you get with other products, but that is not really to be expected considering the range of features Bacula offers. Although Bacula uses local tools to do the backup, it is a true clientserver product with five major components that use authenticated communication: Director, Console, File, Storage, and Catalog. These elements are deployed individually on the ba-

sis of the function of the respective machine. The Director supervises all backup, restore, and other operations, including scheduling backup jobs. Backup jobs can start simultaneously as well as on a priority basis. The Director also provides the centralized control and administration and is responsible for maintaining the file catalog. The Console is used for interaction with the Bacula director and is available as a GUI or command-line tool. The File component is also referred to as the client program, which is the software that is installed on the machines to be backed up. As its name implies, the Storage component is responsible for the storage and recovery of data to and from the physical media. It receives instructions from the Director and then transfers data to or from a file daemon as appropriate. It then updates the catalog by sending file location information to the Director. The Catalog is responsible for maintaining the file indexes and volume database, allowing the user to locate and restore files quickly. The Catalog maintains a record of not only the files but also the jobs run. Currently, Bacula supports MySQL, PostgreSQL, and SQLite. As of this writing, the Directory and Storage daemons on Windows are not directly supported by Bacula, although they are reported to work.

F e at u r e s

One interesting aspect of Bacula is the built-in Python interpreter for scripting that can be used, for example, before starting a job, on errors, when the job ends, and so on. Additionally, you can create a rescue CD for a “bare metal” recovery, which avoids the necessity of reinstalling your system manually and then recovering your data. This process is supported by a “bootstrap file” that contains a compact form of Bacula commands, thus allowing you to store your system without having access to the Catalog. The basic unit is called a “job,” which consists of one client and one set of files, the level of backup, what is being done (backing up, migrating, restoring), and so forth. Bacula supports the concept of a “media pool,” which is a set of volumes (i.e., disk, tape). With labeled volumes, it can easily match the external labels on the medium (e.g., tape) as well as prevent accidental overwriting of that medium. It also supports backing up to a single medium from multiple clients, even if they are on different operating systems. The Bacula website is not as fancy as Amanda, but I found it more useful because the details about how the program works are much more accessible, and the information is more up to date.

The Right Fit Although I only skimmed the surfaces of these products, this article should give you a good idea what is possible in a backup application. Naturally, each product has many more features than I looked at, so if any of these products piqued your interest, take a look at the website to see everything that product has to offer. n

Figure 3: Areca backup.

w w w. a d m i n - m aga z i n e .co m

Info [1] luckyBackup: [http://luckybackup.sourceforge.net] [2] Amanda: [http://www.amanda.org] [3] Zmanda: [http://www.zmanda.com] [4] Areca Backup: [http://www.areca‑backup.org] [5] Bacula: [http://www.bacula.org]

Admin 01

F e at u r e s

BlackHat 2010

BlackHat USA 2010

Learning from the Best The latest and greatest security issues By Kurt Seifried I’ve been to BlackHat twice now, and both times I have taken the same lesson home: If you think things are getting better in the field of computer security, you’re probably wrong. Over the years, progress has been made identifying bug types – currently the CWE lists 668 weaknesses in 120 categories – and some progress has been made with projects to identify and remove them systematically (e.g., OpenBSD has had remarkable success). However, you then come to the BlackHat conference and see a presentation like “HTTPS Can Byte Me,” in which Robert Hansen and Josh Sokol disclosed 24 vulnerabilities (Figure 1) that can compromise the integrity and security of SSLencrypted web traffic [2]. The problem is not so much a failing within SSL, but unless you’re taking extreme measures to protect network traffic against analysis (e.g., padding traffic out, introducing time delays, etc.), chances are, attackers will be able to glean information even if they can’t read the traffic directly. Also, consider the case of the wellmeaning web browser that attempts to be helpful. I guess people hate typ-

Figure 1: Final HTTPS slide – 24 issues in all.

Admin 01

ing in personal information, so almost all browsers support “auto-complete,” which automatically fills out form fields (e.g., name, address, and credit card number). Unfortunately, this feature can be abused by attackers (imagine that), allowing them to steal personal information saved within your web browser if you visit a web page. Using JavaScript, they can set it up so you don’t even have to type anything in – combined with a hidden IFRAME, you might never realize that it happened. The security talks are especially worrying – the ones in which researchers don’t find new vulnerabilities but simply quantify existing ones. In the case of SSL certificates, they scanned the Internet and found 1.2 million SSL-enabled websites [3] [4]. Among the problems found were certificates for reserved addresses (e.g., 192.168.1.2, a reserved IP address used by multiple sites) that never should have been allowed. Also, they found 50 percent of servers configured to allow SSLv2 (known to be insecure for 14 years). Now, I’m not a glass half empty kind of guy, but seeing 50 percent of servers configured insecurely is a bit depressing (which is probably why most security people buy beer in pitchers, not glasses). BlackHat isn’t an unending stream of bad news, however. Many of the presentations not only present problems but also discuss the solutions. The perfect example was a presenta-

tion called “Lifting the Fog” [5], in which Marco Slaviero scanned for memcached (a memory-caching program widely used to speed up web-based applications). He found many memcached servers open to the world, and by using two poorly documented commands, stat detail on (which enables debugging) and stats cachedump (which lists all the key names), he was able to retrieve all the items stored in the memcached server. And by “all” I mean everything; according to his presentation, he retrieved 136TB of data from 229 memcached servers. The good news is that securing your memcached is simple: Firewall it so that only local trusted systems can connect to it (and if you must use it over the Internet, set up a VPN to connect systems to it). This solution is not magical, but it drives home the point that you need to test and verify security measures using tools like Nmap [6] (which is how he found all the memcached instances). So, if you need an excuse (well, a work-related excuse) to go to Las Vegas, BlackHat, and Defcon afterward, they’re not only a lot of fun, but very educational. My only complaint is that with 10 tracks, chances are you’ll have to choose between two or more interesting talks, which is definitely a glass half full type of problem. n

Info [1] Common Weakness Enumeration: [http://cwe.mitre.org/] [2] HTTPS Can Byte Me: [https://media.blackhat.com/bh‑us‑10/ whitepapers/Hansen_Sokol/Blackhat‑USA ‑2010‑Hansen‑Sokol‑HTTPS‑Can‑Byte‑Me ‑wp.pdf] [3] SSL Observatory: [http://www.eff.org/observatory] [4] Internet SSL Survey 2010: [http://blog.ivanristic.com/Qualys_SSL_ Labs‑State_of_SSL_2010‑v1.6.pdf] [5] Lifting the Fog: [https://media.blackhat.com/bh‑us‑10/ presentations/Slaviero/BlackHat‑USA‑2010 ‑Slaviero‑Lifting‑the‑Fog‑slides.pdf] [6] “Nmap scripting” by Eric Amberg, Linux Magazine, February 2008, pg. 68

w w w. a d m i n - m aga z i n e .co m

Backup & Disaster Recovery

Georgetown University Chooses SEP sesam Backup Solution The Solution - SEP sesam Mike Yandrischovitz, Data Systems and Security Manager for the business school, consulted with other members of the user community and discovered SEP sesam. After contacting SEP, he downloaded and installed the software. In less than two hours, Mike, along with SEP assistance, was able to get backups for McDonough’s most critical applications. The decision to move from the old vendor was still difficult. Business school staff had invested a great deal financially and even more in time and lost productivity. Nevertheless it was decided to make a change to SEP sesam.

The Situation Georgetown University’s McDonough School of Business (MSB), one of the most renowned business schools in the United States, called SEP sesam to replace their under-performing backup software. MSB is experiencing a period of strong program growth. The MSB Technology Canter has been tasked with keeping their IT systems up to date and required a state of the art backup system to ensure continuity of operations. After a brief discussion, SEP was able to analyze current problems and provide reliable backups for critical data. During the course of using the old solution, administrators were continually asked to reconfigure and restart their backup systems even though changes were not being made to the environment or the network infrastructure.

The Challenge MSB’s new data and applications services requirements had outstripped its legacy backup software. The old solution was not flexible enough to meet new demands without continuous monitoring. The old system continually failed during overnight backup tasks. Each error and failed backup required a lengthy call to vendor tech support and often required custom code changes and hot patches. The situation finally became untenable when the software could not work with a newly purchased EMC DL3D1500 Disk Library. This final straw initiated an active search for a better and more effective backup solution.

Backup & Disaster Recovery For more information visit: www.sepsoftware.com | www.sepusa.com

“James Delmonico, at SEP, had us up and running in hours. I was getting substantial ‘heat’ from our user community because our backup solution was unstable. We were not able to get reliable backups using the old backup software. Thanks to the SEP sesam solution we had reliable backups almost immediately and restores of critical data for our customers’ everyday requirements were fast, easy and accurate. We are now a great fan of the software and the team at SEP,” said Yandrishovitz. “SEP engineers were even instrumental in using the SEP software to help diagnose a configuration problem with our new SAN. Isolating the problem, we were able to pinpoint an issue with the SAN Switch. The vendor reconfigured the switch and now, with the new software and new hardware, our backups are completed within time windows previously considered unattainable.” According to John Carpenter, McDonough Chief Technology Officer, “SEP sesam and their helpful engineers took a major worry off our plates. The new implementation has performed better than we expected. Our staff can now go home on time and I’vesaved the cost of acquisition of SEP sesam by returning scheduled overtime back to the operating budget. Backups that used to take a whole weekend are now complete in under eight hours.”

Results “Implementing SEP sesam has been truly instrumental in easing our workload and providing a quality backup solution for all of our customers. The implementation has allowed us to use other equipment including our hundred-slot ADIC tape library, which was not available to us when using the previous solution. The time we spend working on backup related issues has been reduced by a factor of 90%. The acquisition cost for SEP sesam was less than our annual maintenance fee for the old back up solution. Call us one satisfied customer,” stated Carpenter.

OCFS2

To o l s

A simple approach to the OCFS2 cluster filesystem

Divide and Conquer The vanilla kernel includes two cluster filesystems: OCFS2 has been around since 2.6.16 and is thus senior to GFS2. Although OCFS2 is non-trivial under the hood, it is fairly simple to deploy. By Udo Seidel Wherever two or more computers need to access the same set of data, Linux and Unix systems will have multiple competing approaches. (For an overview of the various technologies, see the “Shared Filesystems” box.) In this article, I take a close look at OCFS2, the Oracle Cluster File System shared disk filesystem [1]. As the name suggests, this filesystem is mainly suitable for cluster setups with multiple servers.

Admin 01

Before you can set up a cluster filesystem based on shared disks, you need to look out for a couple of things. First, the administrator needs to establish the basic framework of a cluster, including stipulating the computers that belong to the cluster, how to access it via TCP/IP, and the cluster name. In OCFS2’s case, a single ASCII file is all it takes (Listing 1). The second task to tackle with a cluster filesystem is that of controlled and

orderly access to the data with the use of file locking to avoid conflict situations. In OCFS2’s case, the Distributed Lock Manager (DLM) prevents filesystem inconsistencies. Initializing the OCFS2 cluster automatically launches DLM, so you don’t need to configure this separately. However, the best file locking is worthless if the computer writing to the filesystem goes haywire. The only way to prevent computers from writing is fenc-

w w w. a d m i n - m aga z i n e .co m

OCFS2

To o l s

ing. OCFS2 is fairly simplistic in its approach and only uses self-fencing. If a node notices that it is no longer cleanly integrated with the cluster, it throws a kernel panic and locks itself out. Just like the DLM, self-fencing in OCFS2 does not require a separate configuration. Once the cluster configuration is complete and has been distributed to all the nodes, the brunt of the work has been done for a functional OCFS2. Things are seemingly quite simple at this point: Start the cluster, create OCFS2 if needed, mount the filesystem, and you’re done.

Getting Started As I mentioned earlier, OCFS2 is a cluster filesystem based on shared disks. The range of technologies that can provide a shared disk spans expensive SAN over Fibre Channel, from iSCSI to low-budget DRBD [7]. In this article, I will use iSCSI and NDAS (Network Direct Attached Storage). The second ingredient in the OCFS2 setup is computers with an OCFS2-capable operating system. The best choices here are Oracle’s Enterprise Linux, SUSE Linux Enterprise Server, openSUSE, Red Hat Enterprise Linux, and Fedora. The software suite for OCFS2 comprises the ocfs2-tools and ocfs2console packages and the ocfs2‑`uname ‑r` kernel modules. Typing ocfs2con‑ sole launches a graphical interface in which you can create the cluster configuration and distribute it over the nodes involved (Figure 1). However, you can just as easily do this with vi and scp. Table 1 lists the actions the graphical front end supports and the equivalent command-line tools. After creating the cluster configuration, /etc/init.d/o2cb online launches the subsystem (Listing 2). The init script loads the kernel modules and sets a couple of defaults for the heartbeat and fencing. Once the OCFS2 framework is running, the administrator can create the cluster filesystem. In the simplest case, you can use mkfs.ocfs2 devicefile (Listing 3) to do this. The

w w w. a d m i n - m aga z i n e .co m

Figure 1: Cluster configuration with the ocfs2console GUI tool.

man page for mkfs.ocfs provides a full list of options, the most important of which are covered by Table 2. Once you have created the filesystem, you need to mount it. The mount command works much like that on unclustered filesystems (Figure 2). When mounting and unmounting

OCFS2 volumes, you can expect a short delay: During mounting, the executing machine needs to register with the DLM. In a similar fashion, the DLM resolves any existing locks or manages them on the remaining systems in case of a umount. The documentation points to various options

Table 1: Directories in a Repository Filesystem Function Mount Unmount Create Repair Repair Change name Maximum number of nodes

GUI Menu Mount Unmount Format Check Repair Change Label Edit Node Slot Count

CLI Tool mount.ocfs2 umount mkfs.ocfs2 fsck.ocfs2 fsck.ocfs2 tunefs.ocfs2 tunefs.ocfs2

Table 2: Important Options for mkfs.ocfs2 Option

Purpose

Block size

Cluster size

Label

Maximum number of computers with simultaneous access

Journal options

Filesystem type (optimization for many small files or a few large ones)

Listing 1: /etc/o cfs2/c luster.conf node:

ip_port = 7777 ip_port = 7777

i p_address = 192.168.0.2

ip_address = 192.168.0.1

number = 1

number = 0

name = node1

name = node0 cluster = ocfs2 node:

cluster = ocfs2 cluster: node_count = 2

Admin 01

OCFS2

To o l s

you can set for the mount operation. If OCFS2 detects an error in the data structure, it will default to readonly. In certain situations, a reboot can clear this up. The errors=panic mount option handles this. Another interesting option is commit=seconds. The default value is 5, which means that OCFS2 writes the data out to disk every five seconds. If a crash occurs, a consistent filesystem can be guaranteed – thanks to journaling – and only the work from the last five seconds will be lost. The mount option that specifies the way data are handled Listing 2: Starting the OCFS2 Subsystem # /etc/init.d/o2cb online Loading filesystem "configfs": OK Mounting configfs filesystem at /sys/kernel/config: OK Loading filesystem "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting O2CB cluster ocfs2: OK # # /etc/init.d/o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Not active #

Listing 3: OCFS2 Optimized for Mail Server # mkfs.ocfs2 ‑T mail ‑L data /dev/sda1 mkfs.ocfs2 1.4.2 Cluster stack: classic o2cb Filesystem Type of mail Filesystem label=data

for journaling is also important here. The latest version lets OCFS2 write all the data out to disk before updating the journal. date=writeback forces the predecessor’s mode. Inexperienced OCFS2 admins might wonder why the OCFS2 volume is not available after a reboot despite an entry in /etc/fstab. The init script that comes with the distribution, /etc/init.d/ocfs2 makes the OCFS2 mount resistant to reboots. Once enabled, this script scans /etc/fstab for OCFS2 entries and integrates these filesystems. Just as with ext3/4, the administrator can modify a couple of filesystem properties after mounting the filesystem without destroying data. The tunefs.ocfs2 tool helps with this. If the cluster grows unexpectedly and you want more computers to access OCFS2 at the same time, too small a

value for the N option in mkfs.ocfs2 can become a problem. The tunefs. ocfs2 tool lets you change this in next to no time. The same thing applies to the journal size (Listing 4). Also, you can use this tool to modify the filesystem label and enable or disable certain features (see also Listing 8). Unfortunately, the man page doesn’t tell you which changes are permitted on the fly and which aren’t. Thus, you could experience a tunefs. ocfs2: Trylock failed while opening device "/dev/sda1" message when you try to run some commands on OCFS2.

More Detail As I mentioned earlier, you do not need to preconfigure the cluster heartbeat or fencing. When the cluster stack is initialized, default values are set for both. However, you

Shared Filesystems The shared filesystem family is a fairly colorful bunch. By definition, they all share the ability to grant multiple computers simultaneous access to certain data. The differences are in the way they implement these requirements. On the one hand are network filesystems, in which the most popular representative in the Unix/Linux camp is Network Filesystem (NFS) [2]. NFS is available for more or less any operating system and to all intents and purposes only asks the operating system to provide a TCP/IP stack. The setup is also fairly simple. The Andrew filesystem (AFS) is another network filesystem that is available in a free implementation, OpenAFS [3]. On the other hand are cluster filesystems. Before computers can access “distributed” data,

they first need to enter the cluster. The cluster setup requires additional infrastructure, such as additional I/O cards, cluster software, and, of course, a configuration. Cluster filesystems are also categorized by the way they store data. Those based on shared disks allow multiple computers to read and write to the same medium. I/O is handled via Fibre Channel (“classical SAN”) or TCP/IP (iSCSI). The most popular representatives in the Linux camp here are OCFS2 and the Global Filesystem (GFS2) [4]. Parallel cluster filesystems are a more recent invention. They distribute data over computers in the cluster by striping single files across multiple storage nodes. Lustre [5] and Ceph [6] are popular examples of this technology.

Block size=2048 (bits=11) Cluster size=4096 (bits=12) Volume size=1011675136 (246991 clusters) (493982 blocks) 16 cluster groups (tail covers 8911 clusters, rest cover 15872 clusters) Journal size=67108864 Initial number of node slots: 2 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 0 block(s) Formatting Journals: done Formatting slot map: done Writing lost+found: done mkfs.ocfs2 successful

Admin 01

History OCFS2 is a fairly young filesystem. As the “2” in the name suggests, the current version is an enhancement. Oracle developed the predecessor, OCFS, for use in Real Application Cluster for Oracle databases. The new OCFS2 is designed to fulfill the requirements placed on a mature filesystem capable of storing arbitrary data. POSIX compatibility and the typical – and necessary – performance required for databases were further criteria. After two years of development, the programmers released version 1.0 of OCFS2, and it made its way into the vanilla kernel (2.6.16)

just a year later. Version 1.2 became more widespread, with a great deal of support from various Enterprise Linux distributions. OCFS2 has been available for the major players in the Linux world for some time. This applies to commercial variants, such as SLES, RHEL, or Oracle EL, and to the free Debian, Fedora, and openSUSE systems. Depending on the kernel version, users either get version 1.4, which was released in 2008, or version 1.2, which is two years older. The “OCFS2 Choices” box and Table 3 show you what you need to watch out for.

w w w. a d m i n - m aga z i n e .co m

OCFS2

To o l s

F Figure 2: Unspecta cular: the OCFS2 mount process. E Figure 3: Automatic reboot after 20 sec onds on OCFS2 cluster errors.

can modify the defaults to suit your needs. The easiest approach here is via the /etc/init.d/o2cb configure script, which prompts you for the required values – for example, when the OCFS2 cluster should regard a node or network connection as down. At the same time, you can specify when the cluster stack should try to reconnect and when it should send a keep-alive packet. Apart from the heartbeat timeout, all of these values are given in milliseconds. However, for the heartbeat timeout, you need a little bit of math to determine when the cluster should consider that a computer is down. The value represents the number of two-second iterations plus one for the heartbeat. The default value of 31 is thus equivalent to 60 seconds. On larger networks, you might need to increase all these values to avoid false alarms. If OCFS2 stumbles across a critical error, it switches the filesystem to read-only mode and generates a kernel oops or even a kernel panic.

In production use, you will probably want to remedy this state without in-depth error analysis (i.e., reboot the cluster node). For this to happen, you need to modify the underlying operating system so that it automatically reboots in case of a kernel oops or panic (Figure 3). Your best bet for this on Linux is the /proc filesystem for temporary changes, or sysctl if you want the change to survive a reboot. Just like any other filesystem, OCFS2 has a couple of internal limits you need to take into account when designing your storage. The number of subdirectories in a directory is restricted to 32,000. OCFS2 stores data in clusters of between 4 and 1,024Kb. Because the number of cluster addresses is restricted to 232, the maximum file size is 4PB. This limit is more or less irrelevant because another restriction – the use of JBD journaling – limits the maximum OCFS2 filesystem size to 16TB, which can address a maximum of 232 blocks of 4KB.

An active OCFS2 cluster uses a handful of processes to handle its work (Listing 5). DLM-related tasks are handled by dlm_thread, dlm_reco_thread, and dlm_wq. The ocfs2dc, ocfs2cmt, ocfs2_wq, and ocfs2rec processes are responsible for access to the filesystem. o2net and o2hb‑XXXXXXXXXX handle cluster communications and the heartbeat. All of these processes are started and stopped by init scripts for the cluster framework and OCFS2. OCFS2 stores its management files in the filesystem’s system directory, which is invisible to normal commands such as ls. The debugfs.ocfs2 command lets you make the system directory visible (Figure 4). The objects in the system directory are divided into two groups: global and Listing 4: Maintenance with tunefs.ocfs2 # tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1 NumSlots = 2 # tunefs.ocfs2 ‑N 4 /dev/sda1 # tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1 NumSlots = 4 # # tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1 Label = data # tunefs.ocfs2 ‑L oldata /dev/sda1 # tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1 Label = oldata #

Listing 5: OCFS2 Processes # ps ‑ef|egrep '[d]lm|[o]cf|[o]2' root

3460

0 20:07 ?

00:00:00 [user_dlm]

root

3467

0 20:07 ?

00:00:00 [o2net]

root

3965

0 20:24 ?

00:00:00 [ocfs2_wq]

root

7921

0 22:40 ?

00:00:00

root

7935

0 22:40 ?

00:00:00 [ocfs2dc]

root

7936

0 22:40 ?

00:00:00 [dlm_thread]

root

7937

0 22:40 ?

00:00:00 [dlm_reco_

root

7938

0 22:40 ?

00:00:00 [dlm_wq]

Figure 4: The metadata for OCSFS2 are stored in files that are invisible to the ls command. They can be listed

root

7940

0 22:40 ?

00:00:00 [ocfs2cmt]

with the debugfs.ocfs2 command.

[o2hb‑BD5A574EC8]

thread]

w w w. a d m i n - m aga z i n e .co m

Admin 01

OCFS2

To o l s

local (i.e., node-specific) files. The first of these groups includes global_ inode_alloc, slot_map, heartbeat, and global_bitmap. They have access to each node on the cluster; inconsistencies are prevented by a locking mechanism. The only programs that access global_inode_alloc are those for creating and tuning the filesystem. To increase the number of slots, it Listing 6: Debugging with mounted.ocfs2 # grep ‑i ocfs /proc/mounts |grep ‑v dlm # hostname testvm2.seidelnet.de # tunefs.ocfs2 ‑L olddata /dev/sda1 tunefs.ocfs2: Trylock failed while opening device "/dev/ sda1" # mounted.ocfs2 ‑f Device

Nodes

/dev/sda1

ocfs2

testvm

Listing 7: Restoring Corrupted OCFS2 Superblocks # mount /dev/sda1 /cluster/ mount: you must specify the filesystem type # fsck.ocfs2 /dev/sda1 fsck.ocfs2: Bad magic number in superblock while opening "/dev/sda1" # fsck.ocfs2 ‑r1 /dev/sda1 [RECOVER_BACKUP_SUPERBLOCK] Recover superblock information from backup block#262144? <n> y

is necessary to create further nodespecific system files.

What Else? When you make plans to install OCFS2, you need to know which version you will putting on the new machine. Although the filesystem itself – that is, the structure on the medium – is downward compatible, mixed operations with OCFS2 v1.2 and OCFS2 v1.4 are not supported. The network protocol is to blame for this. The developers enabled a tag in the active protocol version so that future OCFS2 versions would be downward compatible through the network stack. This comes at the price of incompatibility with v1.2. Otherwise, administrators have a certain degree of flexibility when mounting OCFS2 media. OCFS2 v1.4 computers will understand the data structure of v1.2 and mount them without any trouble. This even works the other way around: If the OCFS2 v1.4 volume does not use the newer features in this version, you can use an OCFS2 v1.2 computer to access the data.

Checking OCFS2 filesystem in /dev/sda1: label:

backup

uuid:

31 18 de 29 69 f3 4d 95 a0 99 a7

23 ab 27 f5 04 number of blocks:

367486

bytes per block:

4096

number of clusters: 367486 bytes per cluster:

4096

max slots:

/dev/sda1 is clean.

It will be checked after 20

additional mounts.

Debugging A filesystem has a number of potential issues, and the added degree of Table 3: New Features in OCFS2 v1.4 Feature

Description

Ordered journal mode

OCFS2 writes data before metadata.

Flexible allocation

OCFS2 now supports sparse files – that is, gaps in files. Additionally, preallocation of extents is possible.

Inline data

OCFS2 stores the data from small files directly in the inode and not in extents.

Clustered flock()

The flock() system call is cluster capable.

Listing 8: Enabling/Disabling OCFS2 v1.4 Features # tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1

complexity in a cluster filesystem doesn’t help. From the viewpoint of OCFS2, things can go wrong in three different layers – the filesystem structure on the disk, the cluster configuration, or the cluster infrastructure – or even a combination of the three. The cluster infrastructure includes the network stack for the heartbeat, cluster communications, and possibly media access. Problems with Fibre Channel (FC) and iSCSI also belong to this group. For problems with the cluster infrastructure, you can troubleshoot just as you would for a normal network, FC, or iSCSI problems. Problems can also occur if the cluster configuration is not identical on all nodes. Armed with vi, scp, and md5sum, you can check this and resolve the problem. The alternative – assuming the cluster infrastructure is up and running – is to synchronize the cluster configuration on all of your computers by updating the configuration with ocfs 2console. It can be useful to take the problematic OCFS2 volume offline – that is, to unmount it and restart the cluster service on all of your computers by giving the /etc/init.d/o2cb restart command. You can even switch the filesystem to a kind of single-user mode with tunefs.ocfs2.

Incompatibel: sparse inline‑data # tunefs.ocfs2 ‑‑fs‑features=nosparse /dev/sda1 # tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1 Incompatibel: inline‑data # tunefs.ocfs2 ‑‑fs‑features=noinline‑data /dev/sda1 # tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1 Incompatibel: None # # tunefs.ocfs2 ‑‑fs‑features=sparse,inline‑data /dev/ sda1 # tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1 Incompatibel: sparse inline‑data #

Admin 01

OCFS2 Choices Administrators will basically come across two versions of OCFS2: version 1.2 or 1.4. As regards the data structure on disk, the two versions are compatible; however, this does mean doing without the newer features in v1.4. The documentation lists 10 significant differences between versions 1.2 and 1.4. Table 3 lists the most interesting of these. No matter which version you decide on, you should always watch for a couple of things.

The mkfs.ocfs2 supplied with version 1.4 automatically enables all the new features, thus effectively preventing OCFS v1.2 machines from accessing the filesystem. To change this, use tunefs.ocfs2 to disable the new functions (Listing 8). An easier approach is to create the filesystem with the ‑‑fs‑feature‑level=max‑compat

option set. tunefs.ocfs2 will help you migrate from version 1.2 to 1.4.

w w w. a d m i n - m aga z i n e .co m

To do this, you need to change the mount type from cluster to local. After doing so, only a single computer can mount the filesystem, and it doesn’t need the cluster stack to do so. In all of these actions, you need to be aware that the filesystem can be mounted by more than one computer. Certain actions that involve, say, tunefs.ocfs2, will not work if another computer accesses the filesystem at the same time. The example in Listing 6 shows the user attempting to modify the label. This process fails, although the filesystem is offline (on this computer). In this case, mounted.ocfs2 will help: It checks the OCFS2 header to identify the computer that is online with the filesystem. The most important filesystem structure data are contained in the superblock. Just like other Linux filesystems, OCFS2 creates backup copies of the superblock; however, the approach the OCFS2 developers took is slightly unusual. OCFS2 creates a maximum of six copies at non-configurable offsets: 1, 4, 16, 64, and 256GB and 1TB. Needless to say, OCFS2 volumes smaller than 1GB (!) don’t have a copy of the superblock. To be fair, mkfs.ocfs2 does tell you this when you generate the filesystem. You need to watch out for the Writing backup superblock: ... line. A neat side effect of these static backup superblocks is that you can reference them by number during a filesystem check. The example in Listing 7 shows a damaged primary superblock that is preventing mounting and a simple fsck.ocfs2 from

w w w. a d m i n - m aga z i n e .co m

working. The first backup makes it possible to restore.

Basically, Yes, but … On the whole, it is easy to set up an OCFS2 cluster. The software is available for a number of Linux distributions. Because OCFS2 works just as well with iSCSI and Fibre Channel, the hardware side is not too difficult either. Setting up the cluster framework is a fairly simple task that you can handle with simple tools like Vi. Although OCFS2 doesn’t include sophisticated fencing technologies, in contrast to other cluster filesystems, fencing is not necessary in many areas. The lack of a cluster-capable volume manager makes it easier for the user to become immersed in the world of OCFS2. Because OCFS2 is simpler and less complex than other cluster filesystems, it is well worth investigating. n

Info [1] OCFS2: [http://oss.oracle.com/ projects/ocfs2/] [2] First NFS RFC: [http://tools.ietf. org/html/rfc1094] [3] OpenAFS: [http://www.openafs.org/] [4] GFS: [http://sources.redhat. com/cluster/gfs/] [5] Lustre: [http://wiki.lustre.org] [6] Ceph: [http://ceph.newdream.net/] [7] DRBD: [http://www.drbd.org/] The Author Udo Seidel is a teacher of math and physics and has been an avid supporter of Linux since 1996. After completing his PhD, he worked as a Linux/Unix trainer, system administrator, and senior solutions engineer. He now works as the head of a Linux/Unix team for Amadeus Data Processing GmbH in Erding, Germany.

Linux Pros read

LINUX PRO Enjoy a rich blend of tutorials, reviews, international news, and practical solutions for the technical reader.

Subscribe now to receive:

3 issues + 3 DVDs for only

3.00!

www.linuxpromagazine.com/trial

Admin 01

Synergy

©Kheng Ho Toh, 123RF.com

To o l s

Controlling multiple systems simultaneously with Synergy

Side Effect

The many approaches to managing remote computers include VNC, Nomachine, and SSH. Synergy is a clever tool that does a bit of lateral thinking and connects multiple PCs to create a virtual desktop. By Florian Effenberger To operate Synergy, you need at least two PCs, each with its own operating system, monitor, and functional network card. The software supports Windows 95 through Windows 7, Mac OS X as of version 10.2, and Linux with the current X server. Prebuilt packages for Windows and Mac OS X are available from the Synergy homepage [1]. An RPM file is available for Linux and can be installed on most popular distributions with tools such as Alien [2], if needed. Some distributions also offer prebuilt packages; for example, Ubuntu Universe contains a package called Synergy.

Test Case The administrator’s workplace comprises a large desktop system running Ubuntu and a small notebook running Vista on the right. To avoid constantly switching between keyboards,

Admin 01

the administrator has decided to use Synergy. The admin will work mainly on the large PC, which is the Ubuntu system. In Synergy-speak, this is referred to as the control system; the administrator will use the keyboard and mouse on this server. The other devices are clients.

Configuration Before you start using Synergy, you need to configure it by editing the /etc/synergy.conf or ~/.synergy. conf text file. The elementary unit is a screen: Each computer belonging to a group, whether server or client, is a screen with a precisely defined position – just like the display arrangement in a configuration with multiple monitors. For each computer, you need to enter into the configuration file the name of the screen, its aliases, and its position relative to other de-

vices – in both directions. Listing 1 contains an example with comments for the test case. The Synergy homepage documents many additional options [3]. All options in the configuration file should be in lowercase. Also, make sure you use the correct line breaks, because Synergy is fussy about them and will not use the file if they are wrong. After completing all this work, you can launch the Synergy server on Ubuntu as a normal user by typing synergys. The ‑f parameter will prevent Synergy from disappearing into the background. QuickSynergy gives you an even more convenient approach to configuration. On Ubuntu, you can download the package from the Universe repository and launch it with Applications | Tools | QuickSynergy after the install. Unfortunately, the program failed to launch a working server during my test. The Vista client, which I want to control remotely with the Ubuntu system, is even easier to configure. After the installation, you can launch Synergy

w w w. a d m i n - m aga z i n e .co m

Synergy

Figure 1: Synergy as a client on Windows Vista.

directly via the Start menu, and it will come up with a neat graphical interface (Figure 1). To open a connection to the server, select the Use another computer’s shared keyboard and mouse (client) option and enter the name of the computer you require. Power users might also want to configure extras like the Logging Level, AutoStart, and network details.

Connecting the Screens After configuring all the clients, you can simply run synergyc server IP (or click Start on Windows) to combine the screens. The whole thing is unspectacular at first, and you can continue to work on each system in the normal way. However, if you move the mouse on the server beyond the right edge of the screen, the mouse pointer moves onto the Vista desktop just as on a single computer with multiple displays, but here, it moves between two computers with different platforms. Keyboard input also reaches the cliEnabling Universe Ubuntu organizes its software packages into a number of repositories. The Ubuntu Universe repository contains packages that have less comprehensive support and maintenance than others. To use Universe, you first need to enable the corresponding line in the /etc/apt/sources.list file by removing the hash sign. After an apt‑get update, you can install the new packages, including Synergy.

w w w. a d m i n - m aga z i n e .co m

ent while the focus is on its screen (i.e., the mouse cursor is displayed there). But, that’s not all – Synergy also coordinates the clipboard between the two systems. According to the developers, Synergy automatically identifies the correct character set and converts line breaks between operating systems, which is perfect for centralized copying of long text blocks and configuration files. Pressing the Scroll key disables Synergy temporarily if needed. In the configuration file, you can set a number of additional options. Among other things, Synergy can map keys between the server and the client, configure screen areas when you do not want to be able to toggle between screens, and perform certain actions at a key press. Synergy messes up some functions, however: In our lab, the tool failed to synchronize screen savers and failed to lock all the screens centrally. According to the homepage, the Mac OS X variant of the program, in particular, is not as mature as the Linux and Windows versions.

Conclusions Synergy offers an interesting approach to controlling multiple computers centrally without investing in additional hardware. In contrast to legacy approaches, each system keeps its own display. The program is definitely useful for owners of multiple Security Note The authors of Synergy point out on their homepage that Synergy does not provide anything in the line of authentication or encryption [4]. To be on the safe side, you might want to set up an SSH tunnel to encrypt all your data [5].

To o l s

PCs. And the cross-operating system clipboard, which removes the need to copy text files, is really convenient. One item on my Synergy wish list, however, is easier configuration on Linux. n

Info [1] Synergy homepage: [http://synergy-foss.org] [2] Alien: [http://kitenet.net/~joey/code/alien/] [3] Configuration options: [http://synergyfoss.org/pm/projects/synergy/wiki/Setup] [4] Security notes: [http://synergy-foss.org/ pm/projects/synergy/wiki/UserFAQ#QHow-secure-is-your-application] [5] Tunneling with SSH: [http://www.revsys. com/writings/quicktips/ssh-tunnel.html] The Author Florian Effenberger has been a free software evangelist for many years. He is the Lead of the Marketing Project for OpenOffice.org international and a member of the OpenOffice.org Germany board. Other work involves designing and implementing enterprise and school networks, including software distribution solutions based on free software. He also writes for numerous German and English language publications, in which he mainly focuses on legal issues.

Listing 1: Configuration # Define screens section: screens ubuntu: vista: end # Alternative names section: aliases # ubuntu ‑> desktop ubuntu: desktop # vista ‑> notebook vista: notebook end # Screen arrangement section: left # vista: right of ubuntu ubuntu: right = vista # ubuntu: left of vista vista: left = ubuntu end

Admin 01

SystemTap

TO O L S

Tracing applications with OProfile and SystemTap

ÂŠ boing, Photocase.com

Data on Tap Does your application data take ages to creep off your disk or your network card, even if no noticeable activity is taking place? Tools such as OProfile and SystemTap help you find out why. By Thorsten Scherf Experienced administrators tend to use tools such as ps, vmstat, or the like when they need statistics for individual subsystems such as the network, memory, or block I/O. These tools can help identify hardware or software bottlenecks, and they are indisputably useful for a general appraisal, but if you want to delve deeper, you need something with more punch. Again, the standard toolbox offers a couple of utilities. For example, the

popular strace traces applications. In the simplest cases, the tool lists all the system calls (syscalls) with their arguments and return codes for a specific application. Setting options allows for highly selective Strace output. For example, if you need to investigate whether an application is parsing the configuration file that you painstakingly put together, you can call Strace: strace -e trace=open -o mutt.trace mutt

This command line sends all open syscalls for the Mutt application to the /tmp/mutt.trace output file. Then, you can easily grep the configuration file from the results. Profiling applications, including the popular OProfile tool [1], take this a step further by giving you details of the performance of individual applications, the kernel, or the complete system (see Figure 1). For this to happen, OProfile accesses the CPU performance counters on state-of-the-art

Listing 1: opcontrol Events opcontrol --list-events

0x30: prefetch: all inclusive

oprofile: available events for CPU type "Core 2"

0x10: prefetch: Hardware prefetch only 0x00: prefetch: exclude hardware prefetch

See Intel Architecture Developer's Manual Volume 3B, Appendix A and Intel Architecture Optimization Reference Manual (730795-001)

0x08: (M)ESI: Modified 0x04: M(E)SI: Exclusive 0x02: ME(S)I: Shared

INST_RETIRED_ANY_P: (counter: all)) number of instructions retired (min count: 6000) L2_RQSTS: (counter: all))

0x01: MES(I): Invalid LLC_MISSES: (counter: all)) L2 cache demand requests from this core that missed the L2 (min count:6000)

number of L2 cache requests (min count: 500) Unit masks (default 0x7f)

Unit masks (default 0x41)

----------

---------0x41: No unit mask

0xc0: core: all cores 0x40: core: this core

ADMIN 01

[...]

W W W. A D M I N - M AGA Z I N E .CO M

SystemTap

hardware. The counters have information on how often a specific event has occurred. In this context, an event can be RAM access or the number of interrupts. This information is very useful for identifying bottlenecks or debugging the system. To install OProfile, you need the kernel-debuginfo [2] package, which provides the symbol to machine code mappings. Note that the kerneldebuginfo version must match your kernel version. To install the package easily, use your distribution’s standard repository. On Fedora, you would do this with Yum: yum install oprofile kernel‑debuginfo

A call to RPM should confirm that the kernel and kernel-debuginfo packages have the same version number: rpm ‑q k ernel‑PAE kernel‑PAE‑debuginfo U oprofile kernel‑PAE‑2.6.32.10‑90.fc12.i686 kernel‑PAE‑debuginfo‑2.6.32.10‑90.fc12.i686 oprofile‑0.9.5‑4.fc12.i686

To profile the kernel, you need to tell OProfile where the kernel image is located with the --vmlinux option: opcontrol ‑‑setup U ‑‑vmlinux=/usr/lib/debug/lib/ U modules/`uname ‑r`/vmlinux

In normal use, you can omit the image details. The following command gives you an overview of the events that OProfile can enumerate (see Listing 1): opcontrol ‑‑list‑events

The events will differ depending on the CPU you use. The /usr/share/ oprofile directory has lists for the various architectures. An event comprises a symbolic name (L2_RQSTS), a counter (500), and an optional mask (0xc0). The counter defines the accuracy of a profile. The lower the value, the more often the event will be queried. Special properties of an event are available by query with the mask. For example, the L2_RQSTS event tells you how many requests have been made to the CPU’s L2 cache. When called with a mask of 0xc0, OProfile returns the value for all the available

w w w. a d m i n - m aga z i n e .co m

To o l s

Figure 1: OProfile architecture.

CPUs; if you set 0x40, you get the value only for the CPU actually running the OProfile process. The command in Listing 2 monitors a specific event. The --event option can occur multiple times to monitor more than one event. To make sure that the results are not distorted by historical data, the --reset option deletes them before collecting fresh data with --start. After a while, the --stop option stops monitoring the system. The data collected in this way are now available in the /var/lib/oprofile/ samples directory. For a general overview, you can access the data with opreport – either the data for the complete system or the data for a specific application (Listing 3). Depending on the event selected, this information will give you a pretty clear picture of what is happening on your system. For more details of OProfile, check out the highly informative website for the tool [1].

ample, for kernel functions (kernel. function("function")) inside of kernel modules (module("module"). function("function")) or system calls (syscall.system_call). The injected program monitors the event and collects information in the process. Thus, far more precise results can be achieved here than with OProfile, which only queries an event periodically. SystemTap has also supported querying static tracepoints in the kernel (kernel.trace("tracepoint")) and, more recently, in userspace applications. Developers build static tracepoints into the program code at Listing 2: opcontrol Events opcontrol ‑ ‑vmlinux=/usr/lib/debug/lib/modules/`uname ‑r`/vmlinux ‑‑event L2_RQSTS:500 # opcontrol ‑‑reset # opcontrol ‑‑start Using 2.6+ OProfile kernel interface. Reading module info. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started.

SystemTap The SystemTap [3] tool aims to combine the functionality of classical tracing and profile tools such as Strace and OProfile while providing a simple but powerful interface for the user. SystemTap was originally developed for monitoring the Linux kernel, although more recent versions also let you monitor userspace applications. SystemTap builds on the kprobes kernel subsystem. It lets the user insert arbitrary program code before any event in kernel space – for ex-

Profiler running.

Listing 3: opreport for Mutt opreport ‑l /usr/bin/mutt CPU: Core 2, speed 2401 MHz (estimated) Cou nted L2_RQSTS events (number of L2 cache requests) with a unit mask of 0x7f (multiple flags) count 500 samples

image name

symbol name

414

10.8377

mutt

imap_exec_msgset

162

4.2408

mutt

parse_set

161

4.2147

mutt

mutt_buffer_add

145

3.7958

mutt

mutt_extract_token

126

3.2984

mutt

ascii_strncasecmp

124

3.2461

mutt

imap_read_headers

[...]

Admin 01

SystemTap

To o l s

important locations. Because developers know their program better than anybody else, this kind of information is a big help. SystemTap programs are written in a language that is similar to Awk. A parser checks the script for syntax errors before converting it to the faster C language, which is then loaded as a kernel module (Figure 2). Using Listing 4: “Hello world” in SystemTap 01 #!/usr/bin/stap 02 probe begin {printf("Hello, world!\n");}

the module on another system that doesn’t have a compiler is not a problem, though. SystemTap lets you build modules for kernel versions besides the kernel on your own system. Then you can copy the module to the target system and run it with staprun – more on this subject later. Because all the major Linux distributions support SystemTap, you can easily install from the standard software repository. The important thing is to install the kernel-debuginfo package along with kernel-devel:

03 probe timer.sec(5) {exit();} yum install kernel‑debuginfo U

04 probe end {printf("Good‑bye, world!\n");}

kernel‑devel systemtap U

systemtap‑runtime

06 # stap helloword.stp 07 Hello, world!

The latest versions are available from the project’s Git repository:

08 <5 seconds later> 09 Good‑bye, world!

git clone U http://sources.redhat.com/git/U

Listing 5: tcpdump via SystemTap

systemtap.git systemtap

01 #!/usr/bin/stap 02 03 // A TCP dump like example 04 05 probe begin, timer.s(1) { 06 printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑ ‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑\n") 07 printf( "

Source IP

DPort

stap ‑v ‑e 'probe vfs.read U

Dest IP S

SPort

F\n")

‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑\n") 09 } 10 11 p robe tcp.receive { 13 14

%5d

%d\n", saddr, daddr, sport, dport, urg, ack, psh, rst, syn, fin)

15 }

Listing 6: Tapset tcp.stp 01 probe tcp.receive = kernel.function("tcp_v4_rcv") { 02

iphdr = __get_skb_iphdr($skb)

saddr = ip_ntop(__ip_skb_saddr(iphdr))

daddr = ip_ntop(__ip_skb_daddr(iphdr))

protocol = __ip_skb_proto(iphdr)

06 07

tcphdr = __get_skb_tcphdr($skb)

dport = __tcp_skb_dport(tcphdr)

sport = __tcp_skb_sport(tcphdr)

urg = __tcp_skb_urg(tcphdr)

ack = __tcp_skb_ack(tcphdr)

psh = __tcp_skb_psh(tcphdr)

rst = __tcp_skb_rst(tcphdr)

syn = __tcp_skb_syn(tcphdr)

fin = __tcp_skb_fin(tcphdr)

16 }

Admin 01

{printf("Reading data from U disk.\n"); exit()}'

08 printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑

12 printf(" %15s %15s

Assuming the installation is successful, you can use the following oneliner to check that SystemTap is working properly:

Tapsets are easy to integrate with your own scripts. The templates are typically located below /usr/share/ systemtap/ tapsets. Besides these synchronous events, other asynchronous events are not bound to a specific event in the kernel or program code, and typically they are used when you need to create a header or footer for your script. They are also suitable for running specific events multiple times. Listing 5 shows a simple example with two probes, each with an asynchronous and a synchronous event. The first outputs a header at onesecond intervals, the second calls the prebuilt tcp.receive tapset, which is defined in Listing 6. This example shows the extent to which the use of tapsets reduces the complexity of your own scripting. When you launch the script from Listing 1, typing stap tcpdump.stp lets you see the network packets arriving at one-second intervals with various other pieces of information. If you omit timer.s(1) in the first event, the header is only output before outputting the first network packet. The handler, also known as a body, supports instructions that will be familiar from various programming languages. For example, you can initialize variables and arrays, call functions, and query positioning parameters $ (integer) or @ (string). Of course, you wouldn’t want to do without loops (while, until, for, if/ else), which give you useful flow control options for the script.

If this accesses the kernel’s VFS subsystem, stap will send a message to standard output and terminate. The ubiquitous “Hello World” program for SystemTap is shown in Listing 4. The “Hello World” example is a great demonstration of the generic structure of a SystemTap script. A script always comprises two parts: an event and a handler – typically preceded by a probe instruction. In this example, the event is the read.vfs function and the handler is the printf command that outputs text to stdout. The handler is always executed when the specified event occurs. Events can be kernel functions, syscalls, or, as in this example, prebuilt tapsets – that is, prebuilt code blocks for specific kernel functions Figure 2: After the syntax of the SystemTap script is checked, the and system calls. script is converted to C and loaded as a kernel module.

w w w. a d m i n - m aga z i n e .co m

SystemTap

Instead of searching through a mass of data for the required information, which is the case with Strace, SystemTap lets you output the information after exceeding a specific threshold value, or when a specific event occurs. Thanks to the functional scope of the language, the choice of language constructs is more than adequate. Listing 7 shows another example that uses the vfs.read tapset. The global variable totals is an associative array in this case. It contains the process names and process IDs for all the applications that access the VFS subsystem to read data from disk. The counter is incremented each time it’s accessed. If you are interested in a specific userspace program, you’ll need to install the matching debuginfo package for the application. To make things easy, I will look at the ls tool as an example. To perform a trace here, you’ll need the coreutils-debuginfo package. Calling stap as in Listing 8 gives you an overview of the functions in a specific process. If the parameters of a specific function are also of interest, you can change the call to stap as shown in Listing 9.

build system. The target systems only require the systemtap-runtime RPM and the staprun program it contains. The following command creates a prebuilt binary kernel module for the target system: stap ‑r kernel‑PAE‑2.6.31.12‑174.2.22 U capt‑io.stp ‑m read‑io

The build system also needs the kernel-debuginfo package to match the target system version, and you must ensure that the build and target systems have the same hardware architecture. After creating a kernel module, copy it to the target system and launch it with staprun: staprun capt‑io.ko

If you want non-root users to load this kernel module, they need to be members of the stapusr group; members of the stapdev group can additionally compile their own scripts.

To o l s

comprehensive tapset library, this can be done without serious programming skills. Advanced users will enjoy the flexible, Awk-like scripting language that gives them the freedom to create highly complex tracing and profiling scripts. The SystemTap FAQ [4] and the language reference [5] are useful ports of call for more help. n

The Author Thorsten Scherf is a Senior Consultant for Red Hat EMEA. You can meet him as a speaker at conferences. He is also a keen marathon runner whenever time permits.

Listing 7: Finding I/O-Intensive Apps 01 #!/usr/bin/stap 02 03 global totals; 04 probe vfs.read 05 06 { 07

totals[execname(), pid()]++

08 }

Conclusions The SystemTap tracing and profiling tool lets regular users perform detailed analyses of kernel and userspace programs without rebooting the whole system. Thanks to the

09 10 probe end 11 { 12

printf("** Summary ** \n")

foreach ([name,pid] in totals‑)

printf( "%s (%d): %d \n", name, pid, totals[name,pid])

15 }

Listing 8: SystemTap Tracing a Userspace App

Cross-Compiling

stap ‑e 'probe process("ls").function("*").call {log (pp())}' ‑c 'ls ‑l'

If you want to run a SystemTap script on multiple systems, you will probably prefer not to have to install the compiler and the kernel debug information on all of these machines. In fact, you only need to do so on a

‑rw‑rw‑r‑‑. 1 tscherf tscherf 17347 2010‑04‑12 08:43 systemtap.txt

total 20 process("/bin/ls").function("main@/usr/src/debug/coreutils‑7.6/src/ls.c:1225").call process("/bin/ls").function("set_program_name@/usr/src/debug/coreutils‑7.6/lib/progname.c:35").call process("/bin/ls").function("human_options@/usr/src/debug/coreutils‑7.6/lib/human.c:462").call process("/bin/ls").function("clone_quoting_options@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:99").call process("/bin/ls").function("xmemdup@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:107").call process("/bin/ls").function("xmalloc@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:43").call process("/bin/ls").function("get_quoting_style@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:110").call process("/bin/ls").function("clone_quoting_options@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:99").call

Info [1] OProfile project homepage: [http://oprofile.sourceforge.net/news] [2] Kernel debuginfo information: [http://fedoraproject.org/wiki/ StackTraces#What_are_debuginfo_rpms. 2C_and_how_do_I_get_them.3F] [3] SystemTap project homepage: [http://sourceware.org/systemtap/] [4] SystemTap FAQ: [http://sourceware.org/ systemtap/wiki/SystemTapFAQ] [5] SystemTap language reference: [http:// sourceware.org/systemtap/langref/]

w w w. a d m i n - m aga z i n e .co m

process("/bin/ls").function("free_pending_ent@/usr/src/debug/coreutils‑7.6/src/ls.c:1132").call [...] process("/bin/ls").function("close_stdout@/usr/src/debug/coreutils‑7.6/lib/closeout.c:107").call process("/bin/ls").function("close_stream@/usr/src/debug/coreutils‑7.6/lib/close‑stream.c:57").call process("/bin/ls").function("close_stream@/usr/src/debug/coreutils‑7.6/lib/close‑stream.c:57").call

Listing 9: Parameter Tracing stap ‑e 'probe process("ls").function("clone_quoting_options").call {log (probefunc() . " " . $$parms) }' ‑c '/bin/ls ‑l' total 20 ‑rw‑rw‑r‑‑. 1 tscherf tscherf 18216 2010‑04‑12 09:02 systemtap.txt clone_quoting_options o=0x0 clone_quoting_options o=0x0

Admin 01

Package Your Scripts

V i rt ua l i z at i o n

Bundle your custom apps in a Debian package

It’s a Wrap Get standard scripts and custom applications into the cloud with the Debian packaging system. By Dan Frost You’ve got a cloud. It’s great. You can scale, and you’ve got redundancy. But you have about 20 scripts for a bunch of tasks (e.g., one for when an instance is booted up and another for when its IP changes) and these scripts aren’t getting any shorter, they’re getting better and longer. If you want to manage them in your favorite versioning software (which I hope is Git, but might be something else), how do you get that onto the new instances simply? Enter the not-new-at-all technology of Debian packages. They are straightforward to use across any Debianbased Linux and simple to create, and they provide an ideal way of contain-

Admin 01

ing and releasing your scripts. In this article, I’ll show how to create Debian packages and how to install them (which you probably already know). And, I’ll explain how the process will make you feel more comfortable about pushing changes live across your cloud. I talked to cloudy people about how to get code onto new instances, and I tried lots of different things, but the Debian package is such a solid, reliable format that I just had to share it.

Debian Packages Debian packages are simply archives that are very easy to install, usually in

one line. If you’ve ever worked with packages in Ubuntu or any other Debian-related Linux, you’ve probably needed to download a package from an online source and install it: apt‑get install blarblar.deb

On the inside, a Debian package is an archive of binary files, scripts, and any other resources an application needs, plus a handful of control files that the various command-line tools use to install the package. Because this standard package format is so easy to install on any Debianbased Linux, it’s a great way to get standard scripts and custom applications into the cloud. Often you need a few lines to configure a new instance or to connect the instance to the rest of your cloud, and storing all those scripts in a package keeps things very neat. The parts of a Debian package I’m most interested in are the control files, which live in a directory called DEBIAN. Control files tell Debian all about what the package contains,

w w w. a d m i n - m aga z i n e .co m

Package Your Scripts

what it’s called, the version, and so on. The second part is the code, which I’ll look at in detail later.

Creating the Package To show this process at work, I’ll create a simple package. Suppose I need a simple page that can sit on your server until an application is installed. Good practice dictates that your instances should fail nicely, so if you start 10 instances when your app gets 6 million tweets, you at least want them to deliver a nice page before they’re ready to do business. To begin, n Create a directory called myserver and the directories and files inside it: ./DEBIAN/ ./var/www/index.html

n Put the code shown in Listing 1 in ./DEBIAN/control. n Put the file shown in Listing 2 in ./var/www/index.html. If the path to the index.html file looks familiar, it’s because the file structure inside your package mirrors exactly the structure in the target instance. If you want a file in /var/some/where/ here, just create that path inside your package project. Once you’ve created this amazing page, package it up with: $ dpkg‑deb ‑‑build myserver dpkg‑deb: building package U `myserver' in `myserver.deb'

When you look in the directory, you’ll see a file called myserver.deb. Now that your project is all packaged up, you can install it. But, before you do, take a look at what’s inside: $ dpkg‑deb ‑‑contents myserver.deb

then install: $ sudo dpkg‑deb ‑i myproject.deb

After running this command, you’ll find the HTML file sitting there for when Apache starts (Listing 3): The next step is to create a skeleton set of scripts for a cloud provisioning service, such as Scalr or RightScale [1]. Indeed, the scripts I use for host-

w w w. a d m i n - m aga z i n e .co m

ing and development servers all sit inside one Debian package.

A Cloudy Package A tiny HTML file isn’t all that useful in the cloud, so I’ll look at something a bit more useful. Server configuration can be set from the Debian package simply by placing your preferred configuration file in the package: ./etc/apache2/conf.d/ our‑config.conf

As long as Apache is configured to include this file (which, in Ubuntu, it often is), it will take effect right away. Although you might want to control this with tools such as Puppet after deployment, starting the instance with a good configuration will help keep the environment sane from the outset. Cloud hosting becomes difficult when you use strange configurations – creating exceptions for some apps or generally working against the grain (e.g., using Tomcat’s configuration style and Apache’s config directories). Avoid customizing the environment too much because it will mean extra maintenance in the future and could limit how you can scale. Another common script for cloud servers will run tasks at certain points in the instance’s life cycle. For instance, a service such as Scalr will run scripts on various events, such as OnHostUp, OnHostInit, and OnIPAd‑ dressChanged. You can create some scripts for these events in your Debian package: ./usr/local/myserver/bin/on‑host‑up.sh ./usr/local/myserver/bin/on‑ip‑ U address‑changed.sh

The first script should download an HTML file or a PHP file from either S3 or your existing repository and place it in the document root:

interaction. Everything has to run automatically when new instances start up, and you really don’t want your script waiting for a human that doesn’t exist. Next, package your project into a .deb file and place it somewhere public from which you can download. This might be where you host, but it is much better to put it somewhere resilient, like S3 [2]. Next, log in to Scalr and add the following lines to a new script that will run on the event OnHostUp (Figure 1). # install the package wget ‑O myserver.deb http://mybucket. U s3.amazonaws.com/myserver.deb deb ‑i myserver.deb # run the script /usr/local/myserver/bin/on‑host‑up.sh

Save the Scalr script under the name of the event that you want to trigger it and go to the farm configuration. Now you can add your neatly organized scripts without having to edit Listing 1: Control Files Package: myserver Version: 0.0.1 Section: server Priority: optional Architecture: all Essential: no Installed‑Size: 1024 Maintainer: Dan Frost [dan@3ev.com] Description: My scripts for running stuff in the cloud

Listing 2: Message Page 01

<html>

<head>

<title>We're getting there...</title>

</head>

<body>

<h1>Give us a moment.</h1>

<p>We're just getting some more machines

plugged in ...</p> 08 09

cd /var/www/

</body> </html>

Listing 3: HTML File

wget ‑O tmp.tgz http://mybucket.s3. U amazonaws.com/website.tgz

V i rt ua l i z at i o n

01 $ cat /var/www/index.html

tar xzf tmp.tgz

02 <html>

service apache2 restart

These scripts will be like any other script you would write, with one important difference: no more human

<head> <title>We're getting there ...</title>

</head>

...

Admin 01

V i rt ua l i z at i o n

Package Your Scripts

deb, so you can install, run arbitrary tests, and repeatedly verify that your cloud scripts are stable before they hit the live environment. And you should! When everything is stable, upload to S3, which you might want to script as well: s3cmd put myserver.deb U s3://mybucket/myserver.deb

Then, all you need is a corresponding script to run on your instances. Create a new script in Scalr or RightScale that downloads and installs the latest version: wget ‑O myserver.deb U http://mybucket.s3.amazonaws.com/ U

Figure 1: Creating a new script.

myserver.deb deb ‑i myserver.deb

scripts via a web interface (Figure 2). If you have a team working on your cloud hosting, you can even start using standard code management, such as Git or SVN, to version your cloud environment’s bootup and configuration scripts. A second event script, which is called each time the IP address changes, would typically update Dynamic DNS with a one-liner (you’ll need to set up your Dynamic DNS account first): curl 'http://www.dnsmadeeasy.com/ U servlet/updateip?username= U myuser&password=mypassword&id= U 99999999&ip=123.231.123.231'

Once you’ve placed this code in the script on‑ip‑address‑changed.sh, simply package it up into your .deb file, upload it to S3 again, and start a new instance. With this approach, testing small changes takes a little longer, but because the scripts are all in a .deb,

Figure 2: Adding a script.

Admin 01

you can test them more easily outside the cloud.

The Package in Production Everything thus far might feel a bit heavy-handed. I put a lot of effort into getting a short script up onto a cloud instance. But suppose you have a running server farm, and you need to update some scripts across the farm. Several cloud services let you edit scripts via a web interface, which is fine up to a point, but beyond a few lines, you will start pining for Emacs or your favorite editor. A custom .deb package makes it easy to create and test the script on local machines or a development cloud before uploading the final version to the production environment. Installing the script on instances is simply a matter of deb ‑i myserver.

The ability to test server configuration in the cloud, for the cloud, is really important. If you’ve been running nice chunky servers for years, you wouldn’t make changes to them unless you were 100 percent sure, but with cloud computing, you can prototype your configurations and settle your nerves before putting things live. When your cloud is running, you will want every opportunity to test the scripts, so being able to install, run, and test them on any instances is valuable.

After Installation: Uninstall So far, Debian packages might just look like glorified tarballs, so why not just tar up your scripts? Well … they’re better than that. Two hooks are provided: post-install and preuninstall. Once your Debian package’s files have been copied to the filesystem, the post-install script, ./DEBIAN/postinst, is run, and when you uninstall, Debian removes your files before running ./DEBIAN/prerm. With these scripts, you can install software, start services, and call a monitoring system to tell it exactly what’s going on with the new instance. For example, open ./DEBIAN/postinst and add something like:

w w w. a d m i n - m aga z i n e .co m

Linux Magazine

ACADEMY curl U http://my‑monitor.example.com/U ?event=installed‑apache&server=U $SERVER_NAME

How you keep your monitoring systems informed depends on what you’re running, but you can add arbitrary commands here to keep yourself happy. A more typical post-install task is sym-linking your scripts into the standard path: ln ‑s /usr/local/myserver/bin/ U on‑host‑up.sh/usr/bin/ on‑host‑up.sh

Anything that gets your scripts working, such as starting any services that are provided or used by your scripts, should be done in the post-install script: service apache2 start service my‑monitor start

However, this is not where you install your web app’s code, nor is it where you grab the latest data. Stick to getting the helper scripts running in the Debian hooks and installing your site from the scripts inside your package. Remember that the key to cloud computing is scaling without friction. Your scripts must install themselves without the need for checking the OS afterward, so use the hooks to leave everything ready to go live.

Why It All Makes Sense To finish, I’ll look at a realworld example. Suppose you want to change your proxy from Apache to HAProxy, and you want your web servers to host some extra code because this makes your app more scalable. Instead of changing to HAProxy on the instance, you create the

w w w. a d m i n - m aga z i n e .co m

script that installs and configures HAProxy, but you do this on your local machine. When you’re happy, you commit this into your Debian package and install it on some cloud instances for testing. When your HAProxy scripts all work fine, simply push your Debian package, along with the new script, up to S3. Next, just terminate your HAProxy instance and wait for a new one to replace it that will run the new scripts, installing and running HAProxy instead of whatever you had before. To get extra code onto an instance, just pull it by using SVN, Git, or Wget; put it into place; and the work is done. So, if you have a huge repository of PDFs that never change or a massive archive database, your scripts can copy this down to instances so that each runs independently. Anything you can do on the command line can be scripted, and packing up your common tasks into a Debian package means your best scripts and best config will be used on all of your instances. Finally, remember that being scalable means being friction-free. The people I work with use Debian packages, because if the package installs, we’ve won half the battle: Our scripts are on the instance. It’s a standard and convenient way of deploying, and it works every time. n

Info [1] “Scaling the Cloud with Scalr” by Dan Frost, Linux Magazine, August 2010, pg. 20 [2] Amazon Simple Storage Service (S3): [http://aws.amazon.com/s3/]

Online Training with Linux Magazine Academy

Preparing for the LPIC exam - the easy way: Linux+ IA T p m o C o t Equivalent PI powered by L GET YOUR LINUX KNOWHOW CERTIFIED WITH LPIC: Professional training for the exams LPI 101 and 102

❚ hardware settings ❚ package management ❚ partitioning and file systems ❚ shell environment ❚ automate system administration ❚ network configuration ❚ security administration tasks ❚ troubleshooting

20% rrent u c r o off f srcibers sub

For more information and to order: academy.linux-magazine.com/LPIC

V i rt uA l i z At i o n

openVz

Operating system virtualization with OpenVZ

Container Service the virtualization technology market is currently concentrating on hypervisor-based systems, but hosting providers often use an alternative technology. container-based solutions such as openVz/Virtuozzo are the most efficient way to go if the guest and host systems are both linux. By thomas drilling

Hypervisor-based virtualization solutions are all the rage. Many companies use Xen, KVM, or VMware to gradually abstract their hardware landscape from its physical underpinnings. The situation is different if you look at leased servers, however. People who decide to lease a virtual server are not typically given a fully virtualized system based on Xen or ESXi, and definitely not a root server. Instead, they might be given a resource container, which is several magnitudes more efficient for Linux guest systems and also easier to set up and manage. A resource container can be implemented with the use of Linux VServer [1], OpenVZ [2], or Virtuozzo [3].

of the CPU, chipset, and peripherals. If you have state-of-the-art hardware (a CPU with a virtualization extension – VT), the performance is good. However, hypervisor-based systems do have some disadvantages. Because each guest installs its own operating system, it will perform many tasks in its own context just like the host system does, meaning that some services might run multiple times. This can affect performance because of overlapping – one example of this being cache strategies for the hard disk subsystem. Caching the emulated disks on the guest system is a waste of time because the host system already does this, and emulated hard disks are actually just files on the filesystem.

Benefits

Parallel Universes

Hypervisor-based virtualization solutions emulate a complete hardware layer for the guest system. Ideally, any operating system including applications can be installed on the guest, which will seem to have total control

Resource containers use a different principle on the basis that – from the application’s point of view – every operating system comprises a filesystem with installed software, space for data, and a number of functions for

Admin 01

accessing devices. For the application, all of this appears to be a separate universe. A container has to be designed so that the application thinks it has access to a complete operating system with a run-time environment. From the host’s point of view, containers are simply directories. Because all the guests share the same kernel, they can only be of the same type as the host operating system or its kernel. This means a Linux-based container solution like OpenVZ can only host Linux guests. From a technical point of view, resource containers extend the host system’s kernel. Adding an abstraction layer then isolates the containers from one another and provides resources, such as CPU cycles, memory, and disk capacity (Figure 1). Installing a container means creating a sub-filesystem in a directory on the host system, such as /var/lib/vz/ gast1; this is the root directory for the guest. Below /var/lib/vz/gast1 is a regular Linux filesystem hierarchy but without a kernel, just as in a normal chroot environment.

w w w. A d m i n - m AgA z i n e .co m

openVz

live migration, checkpointing and restoring OpenVZ containers can be shifted from one physical host to another during operations (Live migration). Ideally, the user will not even notice this process. However, the host environment must be configured to support live migration from a technical point of view. In other words, both virtual environments must reside on the same subnet, and data transmission rate must be high enough. Additionally, the target virtual environment (VE) must have sufficient hard disk space. If these conditions are fulfilled, the following command starts the migration: vzmigrate ‑online target IP VEID

Target IP is the network address of the VE into which you want to migrate to the VE with the ID

of VEID. Of course, the vzmigrate tool supports a plethora of different options (e.g., for migrating over secure connections). The exact syntax and other examples of applications are discussed [12]. Additionally, OpenVZ can create what it refers to as checkpoints (snapshots) of VEs: A checkpoint freezes the current state of the VE and saves it in a file. The checkpoint can be created from within the host context with the vzctl chkpnt VEID command. The checkpoint file can be used later to restore the VE on another OpenVZ host with vzctlrestore VEID.

The container abstraction layer makes sure that the guest system sees its own process namespace with separate process IDs. On top of this, the kernel extension that provides the interface is required to create, delete, shut down and assign resources to containers. Because the container data are extensible on the host file system, resource containers are easy to manage from within the host context.

ever, you can’t load any drivers or kernels from within a container. The predecessors of container virtualization in the Unix world are technologies that have been used for decades, such as chroot (Linux), jails (BSD), or Zones (Solaris). With the exception of (container) virtualization in UserMode Linux [4], only a single host kernel runs with resource containers.

OpenVZ

Efficiency Resource containers are magnitudes more efficient than hypervisor systems because each container uses only as many CPU cycles and as much memory as its active applications need. The resources the abstraction layer itself needs are negligible. The Linux installation on the guest only consumes hard disk space. How-

OpenVZ is the free variant of a commercial product called Parallels Virtuozzo. The kernel component is available under the GPL; the source code for the matching tools under the QPL. OpenVZ runs on any CPU type, including CPUs without VT extensions. It supports snapshots of active containers as well as the Live migration of containers to a different host

V i rt uA l i z At i o n

(see the box “Live Migration, Checkpointing and Restoring”). Incidentally, the host is referred to as the hardware node in OpenVZ-speak. To be able to use OpenVZ, you will need a kernel with OpenVZ patches. One problem is that the current stable release of OpenVZ is still based on kernel 2.6.18, and what is known as the super stable version is based on 2.6.9. It looks like the OpenVZ developers can’t keep pace with official kernel development. Various distributions have had an OpenVZ kernel, such as the last LTS release (v8.04) of Ubuntu, on which this article is based (Figure 2). Ubuntu 9.04 and 9.10 no longer feature OpenVZ, apart from the VZ tools; this also applies to Ubuntu 10.04. If you really need a current kernel on your host system, your only option is to download the beta release, which uses kernel 2.6.32. The option of using OpenVZ and KVM on the same host system opens up interesting possibilities for a free super virtualization solution with which administrators can experiment. If you are planning to deploy OpenVZ in a production environment, I suggest you keep to the following recommendations: You must disable SELinux because OpenVZ will not work correctly otherwise. Additionally, the host system should only be a minimal system. You will probably want to dedicate a separate partition to OpenVZ and to mount this below, /ovz, for example Besides this, you

Applications Container Context

Resource Container 1

Resource Container 3

Resource Container 2

Applications Host Context OpenVZ Abstraction Layer

Host System Kernel

Figure 1: In virtualization based on resource containers, the host and guest use

Figure 2: openSUSE with Ubuntu: System virtualization with resource containers

the same kernel; therefore, they must be of the same type. This means that a

is an interesting option if you need to host (multiple) Linux guest systems as

Linux host can only support Linux guests.

efficiently as possible on a Linux host system.

w w w. A d m i n - m AgA z i n e .co m

Admin 01

V i rt ua l i z at i o n

OpenVZ

Figure 3: I installing OpenVZ from the package sources for Ubuntu 8.04 – the

Figure 4: The OpenVZ developers provide container templates for various guest

last version of Ubuntu to officially include an OpenVZ kernel. The only package

systems; this makes installing a guest system a quick and easy experience.

needed for this was the linux‑openvz meta-package.

Templates from the community are also available.

should have at least 5GB of hard disk space, a fair amount of RAM (at least 4GB), and enough swap space.

Starting OpenVZ Installing OpenVZ is simple. Users on RPM-based operating systems such as RHEL or CentOS can simply include the Yum repository specified in the quick install manual on the project homepage. Ubuntu 8.04 users will find a linux‑openvz meta-package in the multiverse repository, which installs the required OpenVZ kernel, including the kernel modules and header files (Figure 3). At the time of writing, no OpenVZ kernel was available for Ubuntu 10.04. If you are interested in using OpenVZ with a current version of Ubuntu, you will find a prebuilt deb package in Debian’s unstable branch. To install type:

detailed information on this, refer to the sysctl section in the quick install guide, which covers providing network access to the guest systems, involving setting up packet forwarding for IPv4 [5]. Then, you need to reboot with the new kernel. If you edit sysctl after rebooting, you can reload by typing sudo sysctl ‑p. Typing sudo /etc/init.d/vz start

wakes up the virtualization machine. Next, you should make sure all the OpenVZ services are running; this is done easily (on Ubuntu) by issuing: sudo sysv‑rc‑conf ‑list vz

If the tool is missing, you can type sudo apt‑get install sysconfig

to install it. Debian and Red Hat users can run the legacy chkconfig tool. A

check of service vz status should now tell you that OpenVZ is running.

Container Templates OpenVZ users don’t need to install an operating system in the traditional sense of the word. The most convenient approach to set up OpenVZ containers is with templates (i.e., tarballs with a minimal version of the distribution you want to use in the container). Administrators can create templates themselves, although it’s not exactly trivial [6]. Downloading prebuilt templates [7] and copying them to the template folder is easier: sudo cp path_to_template U /var/lib/vz/template/cache

Besides templates provided by the OpenVZ team, the page also offers

sudo dpkg ‑i linux‑base_2.6.32‑10_all.debU linux‑image‑2.6.32‑4‑openvz‑686_U 2.6.32‑10_i386.deb

The sudo apt‑get ‑f install command will automatically retrieve any missing packages. You will also need to install the vzctl tool, which has a dependency for vzquota. Before setting up the containers and configuring the OpenVZ host environment, you need to modify a few kernel parameters that are necessary to run OpenVZ in the /etc/sysctl. conf file on the host system. For more

Admin 01

Figure 5: A couple of clear-cut commands are used for creating and starting a VE and for entering the VE.

w w w. a d m i n - m aga z i n e .co m

OpenVZ

a number of community templates (Figure 4).

Configuring the Host Environment The /etc/vz/vz.conf file lets you configure the host environment. This is where you specify the path to the container and template data on the host filesystem. If you prefer not to use the defaults of

subnet and tell them the DNS server address, which lets OpenVZ create venet devices. All of the following commands must be given in the host context. To do this, you first need to stop the VE and then set all the basic parameters. For example, you can set the hostname for the VE as follows: sudo vzctl set VEID U ‑‑hostname Hostname ‑‑save

V i rt ua l i z at i o n

The ‑‑ipadd option lets you assign a local IP address. If you need to install a large number of VEs, use VEID as the host part of the numeric address. sudo vzctl set VEID ‑‑ipadd U

IP-Address ‑‑save

The DNS server can be configured using the ‑‑nameserver option: sudo vzctl set VEID U ‑‑nameserver Nameserver-address ‑‑save

TEMPLATE=/var/lib/vz/template VE_ROOT=/var/lib/vz/root/$VEID VE_PRIVATE=/var/lib/vz/private/$VEID

you can set your own paths. VE_ROOT is the mountpoint for the root directory of the container. The private data for the container are mounted in VE_PRIVATE. VEID is a unique ID that identifies an instance of the virtual environment. All OpenVZ tools use this container ID to address the required container.

Figure 6: The virtual environment uses venet devices to communicate with the outside world.

Creating Containers The vzctl, which is only available in the host context, creates containers and handles most management tasks, too. In the following example, I used it to create a new VE based on a template for openSUSE 11.1 that I downloaded:

Figure 7: The vzlist command outputs a list of active VEs.

sudo vzctl create VEID U ‑‑ostemplate suse‑11.1‑x86_64

The template name is specified without the path and file extension. The sudo vzctl start VEID starts the VE, and sudo vzctl stop VEID stops it again (Figure 5). The commands sudo vzctl enter VEID and exit let you enter and exit the VE. Entering the VE gives you a working root shell without prompting you for a password. Unfortunately, you can’t deny root access in the host context.

Network Configuration The next step is to configure network access for the container. OpenVZ supports various network modes for this. The easiest option is to assign the VEs an IP on the local network/

w w w. a d m i n - m aga z i n e .co m

Figure 8: User Bean Counters, a set of configuration parameters, allow the administrator to limit resources for each virtual environment.

Admin 01

V i rt ua l i z at i o n

OpenVZ

After restarting the VE, you should be able to ping it from within the host context. After entering the VE, you should also be able to ping the host or another client (Figure 6). For more details on the network configuration, see the “Network Modes” box.

OpenVZ Administration The vzctl tool handles a number of additional configuration tasks. Besides starting, stopping, entering, and exiting VEs, you can use the ‑set option to set a number of operational parameters. Running vzlist in the host context displays a list of the currently active VEs, including some additional information such as the network parameter configuration (Figure 7). In the VE, you can display the process list in the usual way by typing ps. And, if the package sources are configured correctly, patches and software updates should work in the normal way using apt, yum, or yast depending on your guest system. For the next step, it is a good idea to enter the VE by typing vzctl enter VEID. Then, you can set the root password, create more users, and assign the privileges you wish to give them; otherwise, you can only use the VEs in trusted environments.

Figure 9: Virtual Ethernet devices make the VE a full-fledged member of the network with all its advantages and disadvantages.

Without additional configuration, the use of VEs is a security risk because only one host kernel exists, and each container has a superuser. Besides this, you need to be able to restrict the resources available to each VE, such as the disk and memory and the CPU cycles in the host context. OpenVZ has a set of configuration parameters for defining resource limits known as User Bean Counters (UBCs) [8]. The parameters are classified by importance in Primary, Secondary, and Auxiliary. Most of these parameters can also be set with vzctl set (Figure 8). For example, you can enter

ues lets you specify a minimum and maximum limit: sudo vzctl set 100 ‑‑diskspace 8G:10G U ‑‑quotatime 300

Incidentally, sudo vzlist ‑o lists all the set UBC parameters. Note that some UBC parameters can clash, so you will need to familiarize yourself with the details by reading the exhaustive documentation. To completely remove a container from the system, just type the sudo vzctl destroy

command.

sudo vzctl set 100 ‑‑cpus 2

Conclusions

to set the maximum number of CPUs for the VE. The maximum permitted disk space is set by the ‑‑diskspace parameter. A colon between two val-

Resource containers with OpenVZ offer a simple approach to running virtual Linux machines on a Linux host. According to the developers, the

Network Modes In many cases, a venet device is all you need in the line of network interfaces in a VE. Each venet device sets up a point-to-point connection to the host context and can be addressed using an IP address from the host context. Venet devices have a couple of drawbacks, however: They don’t have a MAC address and thus don’t support ARP or broadcasting, which makes it impossible to use DHCP to assign IP addresses. Also, network services like Samba rely on functional broadcasting. A virtual Ethernet (veth) device solves this problem (Figure 9). These devices are supported by a kernel module that uses vzctl to present a virtual network card to the VE. The vzethdev sets up two Ethernet devices: one in the host context and one in the VE. The devices can be named individually, and you can manually or automatically assign a MAC address to them. The host-side device can also reside on a bridge to give the VE a network environment

Admin 01

that is visible in the host context. Within the container, the administrator can then use Linux tools to configure the network interface with a static address or even use DHCP. The kernel module is loaded when the OpenVZ kernel boots. You can check that you have it by issuing the sudo lsmod | grep vzethdev command. To configure a veth device in the container, run sudo vzctl set 101 ‑‑netif_add eth0 ‑‑save

where eth0 is the interface name in the container context. The device name in the host context defaults to vethVEID. If needed, you can assign MAC addresses and device names explicitly. The device can be listed in the host context in the normal way with ifconfig. The number following the dot (0) refers to the device number; here, this is the first veth device in the container with the VEID of 100:

sudo ifconfig veth100.0

A bridge device is the only thing missing for host network access; to set this up host-side, give the sudo brtcl addbr vmbr0 command, then sudo brctl addif vmbr0 verth100.0 to bind it to the veth device, assuming bridge‑utils is installed. Host-side you now have the interfaces l0, eth0, venet0, and veth100.0. If the bridge device is set up correctly, brctl show gives you a listing similar to Figure 10. The additional veth device set up here, 100.1, is for test purposes only and is not important to further steps. Virtual network devices are slightly slower than venet devices. Also, security might be an issue with a veth device – this places a full-fledged Ethernet device in the container context, which the container owner could theoretically use to sniff all the traffic outside the container.

w w w. a d m i n - m aga z i n e .co m

OpenVZ

virtualization overhead with OpenVZ is only two to three percent more CPU and disk load: These numbers compare with the approximately five percent quoted by the Xen developers. The excellent values for OpenVZ are the result of the use of only one kernel. The host and guest kernels don’t need to run identical services, and caching effects for the host and guest kernels do not interfere with each other. The containers themselves provide a complete Linux environment without installing an operating system. The environment only uses the resources that the applications running in it actually need. The only disadvantage of operating system virtualization compared with paravirtualization or hardware virtualization is that, apart from the network interfaces, it is not possible to assign physical resources exclusively to a single guest. Otherwise, you can do just about anything in the containers, including installing packages and providing services. Additionally, setting up the

OpenVZ kernel requires just a couple of simple steps, and the template system gives you everything you need to set up guest Linux distributions quickly. OpenVZ has a head start of several years development compared with modern hypervisor solutions such as KVM and is thus regarded as mature. Unfortunately, the OpenVZ kernel lags behind vanilla kernel development. However, if you are thinking of deploying OpenVZ commercially, you might consider its commercial counterpart Virtuozzo. Besides support, there are a number of aspects to take into consideration when using resource containers. For example, hosting providers need to offer customers seamless administration via a web interface, with SSH and FTP, or by both methods; of course, the security concerns mentioned previously cannot be overlooked. Parallels offers seamless integration of OpenVZ with Plesk and convenient administrations tools for, say, impos-

V i rt ua l i z at i o n

ing resource limits in the form of the GUI-based Parallels Management Console [9] or Parallels Infrastructure Manager [10]. The excellent OpenVZ wiki covers many topics, such as the installation of Plesk in a VE or setting up an X11 system [11]. OpenVZ is the only system that currently offers Linux guest systems a level of performance that can compete with that of a physical system without sacrificing performance to the implementation itself. This makes OpenVZ a good choice for virtualized Linux servers of any kind. n

Info [1] Linux VServer: [http://linux‑vserver.org/ Welcome_to_Linux‑VServer.org] [2] OpenVZ: [http://wiki.openvz.org/Main_Page] [3] Virtuozzo: [http://www.parallels.com/de/ products/pvc45] [4] User-Mode Linux: [http:// user‑mode‑linux.sourceforge.net] [5] OpenVZ quick install guide: [http://wiki. openvz.org/Quick_installation] [6] Creating your own OpenVZ templates: [http://wiki.openvz.org/ Category:Templates] [7] Prebuilt OpenVZ templates: [http://wiki.openvz.org/Download/ template/precreated] [8] OpenVZ User Bean Counters: [http://wiki. openvz.org/UBC_parameters_table] [9] Parallels Management Console: [http://www.parallels.com/de/products/ virtuozzo/tools/vzmc] [10] Parallels Infrastructure Manager: [http:// www.parallels.com/de/products/pva45] [11] X11 forwarding: [http://wiki.openvz.org/X_inside_VE] [12] Live migration: [http://openvz.org/ documentation/mans/vzmigrate.8]

device (eth0) from the container context.

The Author Thomas Drilling has been a freelance journalist and editor for scientific and IT magazines for more than 10 years. With his editorial office team, he regularly writes on the subject of open source, Linux, servers, IT administration, and Mac OS X. In addition to this, Thomas Drilling is also a book author and publisher, a consultant to small and medium-sized companies, and a regular speaker on Linux, open source and IT security.

w w w. a d m i n - m aga z i n e .co m

Admin 01

Figure 10: This example includes one venet and one veth device in the host context. The latter is physically connected to the host network via a bridge device. The host-side veth bridge looks like a normal Ethernet

SUBSCRIBE NOW Save

30% or more!

The New IT New tools, new threats, new technologies... Looking for a guide to the changing world of system administration?

www.admin-magazine.com/subs

AND SAVE 30%! ADMIN Network & Security

Explore the new world of system administration

It isn’t all Windows anymore – and it isn’t all Linux. A router is more than a router. A storage device is more than a disk. And the potential intruder who is looking for a way around your security system might have some tricks that even you don’t know. Keep your network tuned and ready for the challenges with the one magazine that is all for admins. Each issue delivers technical solutions to the real-world problems you face every day. Learn the latest techniques for better: • network security • system management • troubleshooting • performance tuning

ORLD W L A E R S M E L B O PR ! SOLVED

• virtualization • cloud computing on Windows, Linux, Solaris, and popular varieties of Unix.

Special introductory offer! Order by December 31st to save 10% off the regular subscription price! Subscription prices for 1 year of ADMIN Magazine (4 issues + 4 DVDs) are $39.95 for the USA and £24.90/29.90 for UK/Europe (other regions please see our website).

www.admin-magazine.com/subs

V i rt uA l i z At i o n

Virtual machine manager

Microsoft System Center Virtual Machine Manager 2008 R2

Virtual Windows in theory, virtualizing all your old servers is a good idea, but managing them won’t necessarily become any easier. Virtual machine manager gives windows administrators an easy option. By Björn Bürstinghaus Service virtualization is no longer just an interesting topic for large corporations and data centers. In fact, virtualization of multiple server systems on a single physical machine has become an affordable option for small and medium-sized businesses, too. With virtualization and the consolidation benefits that it offers, system management also changes. The machines you are managing are

only available as virtual entities and as the number of virtual machines and virtualization hosts continues to rise, administrators need to consider centralizing their management. Microsoft System Center Virtual Machine Manager 2008 R2 (SCVMM) is a management solution for Hyper-V (R2) hosts, Virtual Server 2005 R2 hosts, and VMware ESX hosts that use the VMware VCenter. SCVMM [1] offers excellent scalability, easy management of hosts and virtual machines and many benefits for the administrator. A Workgroup Edition is available for deployment in small and medium-sized businesses: If you use a maximum of five hosts, this version is a costFigure 1: The Virtual Machine Manager startup screen shows a selection of the effective way of components to install.

Admin 01

managing an unlimited number of virtual machines.

System Requirements To install SCVMM, you need a 64bit version of Windows Server 2008 (R2), which you can run as a virtual machine in smaller environments (with a maximum of five hosts). The system on which you install SCVMM must be a member of an Active Directory domain, but you can use it to manage host systems in non-member networks. In this case, you’ll need to install the agents manually because automatic installation only works inside the domain. SCVMM relies on Microsoft SQL Server to store data. Depending on the size of your environment, you can use the free SQL Server 2005 Express Edition supplied with the bundle, which has a database size limit of 4GB, or you can use an instance of SQL Server 2005 or SQL Server 2008. It makes sense to use a separate server for the database in larger environments. You can install the SCVMM

w w w. A d m i n - m AgA z i n e .co m

Virtual Machine Manager

V i rt ua l i z at i o n

database component on a separate system; it doesnâ&#x20AC;&#x2122;t need to reside on the management server. To install the management component in SCVMM, you also need the Windows Automated Installation Kit (WAIK) 1.1, which is automatically installed when you install the management server; Windows PowerShell 1.0 or 2.0; Windows Remote Management (WinRM) 1.1 or 2.0; and the .NET framework 3.0 (SP1). The web-based SCVMM Self-Service Portal additionally requires Internet Information Server (IIS) version 7.0 or 7.5. The SCVMM Administrator Console can also be installed on client systems such as Windows Vista and Windows 7. If you also want to send command-line jobs to SCVMM via the client, you additionally need to install Windows PowerShell client-side. Figure 2: The management console gives you an overview of the virtual machines in the central panel and the

Installation

system load for the selected VM below.

After starting the installation, youâ&#x20AC;&#x2122;ll be given a choice of components to install for SCVMM (Figure 1). The management server, management console, and self-service portal are all installed separately. When you install the management server, you can place the individual modules, such as the database and the library server, on different systems. This arrangement improves performance if you have a large number of hosts and virtual machines. Before the installation starts, an automatic check is performed to make sure that the hardware and software requirements are fulfilled. If this is the case for all the components, provisioning SCVMM will take less than 20 minutes. Before installing the self-service portal, you must enable the web server role on Windows Server 2008 (IIS 7.0) or Windows Server 2008 R2 (IIS 7.5), as well as the ASP.NET, IIS-6 metacompatibility and IIS 6 WMI compatibility role services. If the role or one of the additional services is not installed, you will see an error message to this effect. Portal access privileges are configured in the management console preferences.

The agents for managing hosts via SCVMM can be installed through the management console or manually. If you use the management console for the install, an automatic check is performed to see whether the host has a hypervisor. If not, the Hyper-V role will be installed automatically on Windows Server 2008 (R2), and Virtual Server 2005 R2 will be installed on Windows Server 2003 (R2).

w w w. a d m i n - m aga z i n e .co m

Microsoft offers a free configuration analyzer for SCVMM; after the install, the analyzer checks whether all the components are installed optimally. Also, you can use the configuration analyzer to check the configuration on remote systems that you will be deploying as hosts and systems intended for P2V conversion. If any issues are detected, youâ&#x20AC;&#x2122;ll be given a detailed description and possible

Figure 3: The Job queue shows modified and outstanding jobs.

Admin 01

V i rt ua l i z at i o n

Virtual Machine Manager

Figure 4: Privileges in the self-service portal: The Web allowed to manage its own virtual machines.

solutions. To use the configuration analyzer for SCVMM, you also need the free Microsoft Baseline Security Analyzer.

The Management Console The SCVMM management console allows administrators to handle a full set of host and virtual machine administrative tasks. The management console is clear-cut and configurable in many places. The console is divided into three areas (Figure 2). The left-hand panel contains the navigation aids for the various SCVMM subsections (Hosts, Virtual Machines, Library, Jobs, and preferences). After selecting a subsection, its objects are listed in the central panel of the console. The righthand panel shows you the actions available for the selected subsection. The Hosts and Virtual Machines subsections give you a perfect overview of the status of all your systems. If you manage a large number of systems, you can use the filters in the navigation area on the left to restrict the number of systems shown. The Jobs subsection is used to check all active SCVMM jobs (Figure 3); Again, you can use the navigation aids to filter on various criteria. This helps you quickly identify and resolve errors and issues.

groups access to virtual machines or provide templates for creating new systems by way of a web-based interface. This system means you can allow developers to restart a test system themselves or allow them Admins user group is to create a new virtual machine based on templates from the library, without requiring that they access the management console. Additionally, you can assign privileges for the portal individually for various groups or users, thus allowing certain users to manage or access a virtual machine but not restart it or switch it off (Figures 4 and 5). After logging in to the self-service portal, the user sees all the virtual machine assignments. To open a connection and manage a virtual machine, an ActiveX control is installed on the client side; the control requires Internet Explorer. Using shared templates, users can create new virtual machines based on the template in the portal. The host is then assigned by the built-in intelligent placement function in SCVMM. Also note that to control virtual ma-

chines on VMware ESX hosts, SSL authentication must be enabled on the host side or the VMware ActiveX control must be installed on the client from which you will be managing the host.

Command Line-Based Controls PowerShell [2] is an extension of the well-known Windows command line; it offers a plethora of administrative commands for script-based Windows management. SCVMM includes more than 150 PowerShell commands, or Cmdlets, which you can use for command line-based management and administration without having to launch the management console (Figure 6). Thus, you can use scheduled tasks on Windows to run tasks at a predefined time on the management server â&#x20AC;&#x201C; for example, to store the status of a virtual machine.

Intelligent Placement One of SCVMMâ&#x20AC;&#x2122;s most practical features is intelligent placement of virtual machines. Because the management server monitors the load on the hosts, it automatically displays a host statistic when you add a new machine so that you can easily see which host is best suited to the task. You can customize the automatic host

The Self-Service Portal With the SCVMM self-service portal, you can grant individual users or

Admin 01

Figure 5: The self-service portal lets non-privileged users manage virtual machines.

w w w. a d m i n - m aga z i n e .co m

Virtual Machine Manager

evaluation function for intelligent placement.

Libraries and Templates The library component in SCVMM is a shared directory of virtual hard disks, ISO images, hardware, and guest operating system profiles. Templates automatically provision Windows client and server systems quickly. A template comprises a virtual hard disk and predefined hardware and operating system profiles. The hardware profile lets you specify the minimum requirements for the CPU type or the amount of RAM the virtual machine needs. When a new virtual machine with the specified CPU type is created, SCVMM automatically searches for a host with resources to match the hardware profile requirements. The operating system profile helps automate operating system provisioning. Besides selecting the operating system, you can also configure the administrator password, a license key, the computer name, and the domain membership.

P2V Conversion SCVMM also converts physical systems to virtual machines on the fly with physical-to-virtual (P2V) migration. For this, simply install a small client on the machine; the client checks the system and displays po-

tential issues before using the volume shadow copy service to create an image. On-the-fly conversion works with client systems as of Windows XP and for server systems as of Windows Server 2003. For older systems, you have only an offline conversion option. After conversion, you can shut down the physical system and boot the system as a virtual machine.

Higher Availability with Clustering Host clustering is a useful way to guarantee system availability. Instead of expensive SAN memory, the data is provided by cheaper iSCSI solutions. To create a Hyper-V cluster, you need two host systems, both of which access the same SAN or iSCSI storage. Live migration introduced in Hyper-V R2 means you can move a virtual machine between clusters without taking the virtual machine offline. The previous version only supported virtual machine migration if you used the same processor type in both clusters. Although this restriction has not been completely lifted, it only applies to the CPU vendor, thus improving support for a variety of hardware in the cluster and offering more flexibility.

Resource Monitoring SCVMM can be combined with the System Center Operations Manager

V i rt ua l i z at i o n

2007 R2 [3] for monitoring host and virtual machine availability. In this case, SCVMM not only uses its own agent to monitor the systems but also provides performance analysis and reporting for a host or virtual machine. The performance and resource optimization (PRO) function built into SCVMM can use Operations Manager 2007 R2 to collect performance data down to the application layer and thus suggest optimization strategies, which are displayed as PRO tips in the management console.

Conclusions Microsoft System Center Virtual Machine Manager 2008 R2 greatly facilitates the management and administration of homogeneous or heterogeneous virtual infrastructures under Windows. Automated provisioning of new client and server systems can be done in minutes with SCVMM. Thanks to the integration of System Center Operations Manager 2007 R2, SCVMM also directly supports performance and availability monitoring for hosts and virtual machines. Because system management always takes a great deal of your time, whether you have five or 50 host systems, it makes sense to plan a centralized solution for all aspects of virtualization, which is exactly what SCVMM offers. n

Info [1] Microsoft System Center Virtual Machine Manager 2008 R2: [http://support. microsoft.com/kb/974722] [2] PowerShell: [http://www.microsoft. com/windowsserver2003/technologies/ management/powershell/default.mspx] [3] Microsoft System Center Operations Man‑ ager 2007 R2: [http://www.microsoft.com/systemcenter/ en/us/operations‑manager.aspx]

Figure 6: Using the PowerShell to move virtual machines between hosts.

w w w. a d m i n - m aga z i n e .co m

The Author Björn Bürstinghaus is a system administrator with simyo GmbH in Düsseldorf, Germany. In his leisure time, he runs Björn’s Windows Blog, a blog on Microsoft Windows topics located at [http://blog.buerstinghaus.net].

Admin 01

Teamviewer

ÂŠ Thor Jorgen Udvang, 123RF.com

Ma n ag e m e n t

Convenient graphical remote control

Remotely Controlled Teamviewer is an impressive demonstration of how easy remote control across routers and firewalls can be. The popular software is now available for Linux. By Daniel Kottmair Some 60 million users already have the Teamviewer [1] commercial remote control solution running on Windows and Mac OS X. Because of the many requests from customers, Teamviewerâ&#x20AC;&#x2122;s manufacturer now provides a variant for Linux in version 5. Teamviewer facilitates remote access to other computers across a network. The only requirement is that the machine at the other end is also running Teamviewer. Teamviewer provides all this functionality in a standalone program; special client or server versions are not available. Teamviewer automatically generates a globally unique ID on each machine. When it is launched, Teamviewer generates a new password that the

Admin 01

computer on the opposite end of the connection can use to access the local machine. This scheme prevents anybody who has ever logged in to that machine from doing so again without the ownerâ&#x20AC;&#x2122;s authorization. You can keep the newly generated password or define one yourself.

Connections Remote access without port forwarding works across routers and firewalls thanks to one of the globally distributed Teamviewer servers on the web, which initiates a 256-bit encrypted UDP connection between the two parties. If a proxy server or a firewall with content filtering makes this con-

nection impossible, the transfer is handled directly by the Teamviewer server. The HTTP label in the window header, rather than the UDP label, identifies this kind of connection. If you are worried about using a thirdparty server, Teamviewer will sell you your own authentication server on request. Teamviewer will even let you remotely control computers that only have a modem connection. The software vendor improved compression in version 5 to reduce the amount of data crossing the wire. Video, Flash banners, and other applications that permanently change screen content are problematic, but a fast DSL connection will let even those types of applications run at an acceptable speed. Private users can run the program free of charge, and the vendor offers commercial licenses for commercial use. Teamviewer is available for Windows, Mac OS X, and Linux; any platform can remotely control any other platform. An iPhone client, available after registering for free online, lets you remotely control a computer as well. The web and iPhone clients are the only versions that can only control, rather than work in both directions. The other variants let you control

w w w. a d m i n - m aga z i n e .co m

Teamviewer

Ma n ag e m e n t

and be controlled, and you can even change directions mid-flow.

Linux Specifics Teamviewer offers downloads of deb and RPM versions 5.0.8206 packages for Linux, along with an X64 deb package and a simple tarball that you don’t even need to install. Teamviewer for Linux is still beta, and the vendor asks for feedback and bug reports. The program is based on a modified version of Wine, although the vendor has made some Linux-specific changes (e.g., to accelerate reading of X server graphics). Although it uses Wine, it is not just a copy of the Windows version. A native Linux version is not available right now, but the vendor is considering creating one depending on acceptance and popularity of the Wine-based version. The program offers a plethora of built-in remote control solutions, such as the ability to change direction, reboot, simulate a Ctrl+Alt+Del key press, and transfer files conveniently between two machines. Multiple logins on a single machine are also supported (e.g., for training purposes in which you need to demonstrate something to a group of users). The VoIP and video chat function introduced in version 5 is also useful, and the application relies on free codecs: Speex for audio and Theora for video. Video on Linux only works in the receiving direction right now. V4L-connected Linux webcams are not currently supported by Teamviewer. The Linux version has a couple of other restrictions: The whiteboard function, which lets users draw on a whiteboard at the same time, and VPN support both fail to provide the goods. The program does not transmit virtual consoles, so you need an X server. The reboot, Ctrl+Alt+Delete key sequence, and Disable Input/ Display on remote computer functions all require the remote machine to run Windows – Mac users have a similar problem. And, the same thing applies to changing the resolution and remov-

w w w. a d m i n - m aga z i n e .co m

Figure 1: The Teamviewer window at program launch.

ing the wallpaper (to avoid unnecessary data traffic).

In the Lab The Linux version works fairly well despite its beta status. One thing that always worked during testing – no matter what network the computers used or what firewalls they were hiding behind – was the connection setup. Teamviewer has thus mastered the most important discipline in remote control with flying colors. Also, no version problems emerged;

a connection between a v4 Mac client and a v5 Linux client worked without problems. The program offers three operating modes when it launches (Figure 1): Remote support, Presentation, and File transfer. Remote support lets you remotely control a system, and presentation mode lets you demonstrate an action to one or multiple users on their own machines. File transfer leaves out the administrative and graphic functions and simply sends files to, or retrieves them from, a remote machine. This mode is ac-

Figure 2: File transfers between computers are convenient and easy to keep track of.

Admin 01

Ma n ag e m e n t

Teamviewer

cessible at any time during normal remote support. From the Teamviewer startup window, use Extras | Options to change various settings, such as your own computer name, or to assign a fixed password for remote logins. Also, you can specify which privileges you want to grant a remote user accessing a system across the wire. Teamviewer will also accept a whitelist or blacklist of computers that are allowed or not allowed to access your computer.

File Transfers File transfers occur in a separate window (Figure 2) that features a twocolumn view, with your own computer on the left and the remote computer on the right. To transfer files, just select them and click the button above the column. This seems a little convoluted; drag-and-drop, or at least double-clicking, to transfer files would make more sense. The vendor aims to change this soon. Also, the Windows-style Wine symlinks and drive letters are a little irritating for Linux users. In normal remote support mode, a hideable Teamviewer function bar is displayed on the remote desktop, and you can use it to access a full set of important remote control functions.

On the bottom right is the connection monitor, which you can also hide and which tells you who is accessing the computer across the wire (Figure 3). If you so desired – again this could be useful for training purposes – the program will use screencasting to monitor activities on the remote computer. The screencast files you can save only contain the data stream transmitted across the wire by Teamviewer and are thus quite compact. Administrators can create a list of computers for single-click access to remote machines. If needed, you can change the transmission focus. The High speed option reduces the color depth to speed up the transmission. If you prefer perfect image quality at the cost of smoother operations, you can opt for this in the View menu. The Automatic setting changes the mode to reflect the connection. Unfortunately, Teamviewer changes the standard gray of some desktops (including Gnome on Ubuntu) to a horrible pink. Because I didn’t notice any speed boosts in reduced color mode, you might prefer to keep a high-quality mode, for Linux-to-Linux connections, at least. A couple of options make the remote support experience smoother. To begin with, you will want to disable desktop effects, such as those pro-

vided by Compiz. Teamviewer only transfers the window content and not the windows, so window zooms and soft fades just create unnecessary traffic. A Flash blocker for the browser is also a good idea for the same reason. Animated ads cause an unnecessarily high level of data – and the bigger the ad space, the worse the problem.

Beta Blockers The older the distribution I tried, the more difficulties I had in testing. The program performs better and is more stable on more recent versions; for example, virtually no problems occurred in connections between Ubuntu 9.04 and 9.10. I definitely advise against changing the resolution on remote Linux clients, because it typically caused Teamviewer to crash on the client. Also, you shouldn’t change the native system settings during remote access, because that can interrupt the data stream from the remote machine. Another failing: The mouse cursor doesn’t change its appearance from an arrow to a hand, for example, if it moves outside of a window or title bar. The vendor has promised to fix this before the final release.

Conclusions Teamviewer is an easy-to-use and practical piece of software. Even if you aren’t an administrator or console jockey, it gives you a really simple approach to managing machines remotely – or for accessing your computer when you’re on the road. Teamviewer works quite well on Linux, even though the beta had a few bugs. The vendor promises to have everything resolved by the time of the final release. One of Teamviewer’s main strengths is its cross-platform compatibility between Linux, Windows, Mac OS X, and even web browsers and the iPhone. Also, the connection always works. n

Figure 3: View of the remote desktop in Teamviewer.¡

Admin 01

Info [1] Teamviewer: [http://www.teamviewer.com/index.aspx]

w w w. a d m i n - m aga z i n e .co m

NOV. 7â&#x20AC;&#x201C;12

2010

San Jose

24th Large Installation System Administration Conference

california

Uncovering the Secrets of System Administration 10 0 2 e s o San J 8 go 200 ie D n a S

re Baltimo 9 200

Dallas 2007

STAPLE HERE

Program Includes: Unraveling the Mysteries oF Twitter Infrastructure, Legal issues in the Cloud, and Huge NFS at Dreamworks SPONSORED BY

KEYNOTE AD DRESS BY

Tony Cass, CERN in cooperat ion with LO PSA & SNIA

INSERT

**JOIN US TRAINING O FOR 6 DAYS OF PRACTI N TOPICS I CAL N C L U D I N * 6-day Virt G: ualization

Track by in John Arrasj structors id and Rich including ard McDoug * Advanced Ti all me Manageme nt: Team Ef * Dovecot an ficiency by d Postfix Tom Limonc by Patrick elli * 5-day Linu Be n Koetter an x Security d Ralf Hildeb and Admini randt stration Tr ack

**REGISTER BY OCTOBER 18 AND SAV www.usenix E** .org/lisa1 0/lp

**PLUS A 3 -DAY TECHNICAL PROGRAM Invited Tal ks Refereed Pa pers Workshops Vendor Exhi bition

m A n Ag e m e n t

chef

Configuration management with Chef © Alistair Cotton, 123RF.com

Chef de Config ever dream of rolling out a complete computer farm with a single mouse click? if you stick to Linux computers and you speak a little Ruby, chef can go a long way toward making that dream come true. By tim Schürmann Chef is basically a server that stores customized configuration guides for software. Clients connected to the server access the recipes and automatically configure their systems on the basis of the rulesets the recipes contain. To do so, the clients not only modify their configuration files, but – if needed – launch their package managers. If the recipes change, or new ones are added at a later date, the clients affected automatically update to reflect the changes. In an ideal environment, this just leaves it up to the administrator to manage the recipes on the server.

Bulk Shopping Before you can enjoy the benefits, the developers behind Chef expect you to put in a modicum of work. For example, recipes are made up of one or multiple standard Ruby scripts. If you need anything beyond the fairly generic recipes available on the web, you need to have a good command of the Ruby scripting language. In other words, your mileage will vary before

Admin 01

you deploy a home-grown and hometested solution. The installation is another obstacle – and a fairly complex one, too, because the Chef server depends on several other components, each of which in turn requires even more software packages. The Chef server itself is written in Ruby but relies on the RabbitMQ server and on a Java-based full-text search engine, at the same time storing its data in a CouchDB database. Finally, your choice of operating system is also important. Chef prefers Linux underpinnings, but it will also run on other Unix-flavored operating systems such as Mac OS X, Open Solaris, FreeBSD, and OpenBSD, according to the wiki [1]. The fastest approach today is offered by Debian 5, Ubuntu 8.10 or later, or CentOS 5.x. Setting up the server on any other system can be an adventure. This article mainly relates to Debian and Ubuntu for this reason. If this is the first time you have ever cooked one of Chef’s recipes, it is also a good idea to run your kitchen on a virtual machine. This prevents things boiling

over and making a mess on the server room floor.

Valuable Ingredients A full-fledged Chef installation comprises the systems you want to configure (nodes) and the server that manages and stores the recipes. Chef clients do all the hard work, picking up the recipes from the server via a REST interface and running the scripts. Each client runs on one node but can apply recipes to multiple nodes. [Figure 1] shows you how this works. For simplicity’s sake, the following examples just use the Chef server and a single client. The latter only configures the computer on which it is running. The first thing you need to have in place is Ruby version 1.8.5 through 1.9.2 (with SSL bindings). Add to this, RubyGems, which will want to build various extensions and libraries later on, thus necessitating the existence of make, gcc, g++, and the Ruby developer packages. Additionally, you need the wget tool for various downloads. The following command installs the whole enchilada on Debian and Ubuntu Linux: sudo apt‑get install ruby U ruby1.8‑dev libopenssl‑ruby1.8 U rdoc ri irb build‑essential U wget ssl‑cert

w w w. A d m i n - m AgA z i n e .co m

Chef

Server

Provides recipes

Client

you need to concentrate on the installation, particularly server-side.

Collects recipes and executes scripts

Nodes

Who’s the Chef?

Figure 1: Overview of the Chef landscape with the server, clients, and nodes.

The packages for openSUSE are called ruby, ruby-devel, wget, opensslcerts, make, gcc, and g++. The certificates from ssl-cert will be required

later. According to the how-to [1], Chef prefers RubyGems version 1.3.6 or newer, but not 1.3.7. This version contains a bug that kills the following installation mid-way. Because most distributions have an older version of RubyGems, your best bet is to head for the source code archive: cd /tmp wget http://rubyforge.org/frs/U download.php/69365/rubygems‑1.3.6.tgz tar zxf rubygems‑1.3.6.tgz cd rubygems‑1.3.6 sudo ruby setup.rb

If the last command installs the Gems executable as /usr/bin/gem1.8 (as is

Ma n ag e m e n t

the case with Debian and Ubuntu), a symbolic link will improve things: sudo ln ‑sfv /usr/bin/gem1.8 /usr/bin/gem

Now you can issue the following Gems command to install the Chef package: sudo gem install chef

When you run a Gem update, keep an eye on the JSON Gem. The version that now comes with RubyGems, 1.4.3, causes an error in Chef. If gem update installs the offending JSON package on your disk, these commands revert to the original version: sudo gem uninstall ‑aIx json sudo gem install ‑v1.4.2 json

The steps thus far provide the underpinnings for Chef operations. Now,

Chef can automate the process of installing and configuring software, so it only seems logical to let Chef install itself. The developers refer to this process as bootstrapping. Having said this, recipes that install the server in this way only exist for Debian 5, Ubuntu 8.10 or later, and CentOS 5.x. On any other distribution, you need to perform all of the steps manually as described in the [Manual Server Installation] boxout. Life is a little easier with one of the operating systems officially supported by Chef. To begin, make sure the computers involved have Fully Qualified Domain Names (FQDNs), such as chefserver.example.com. If you don’t, you will be bombarded with error messages like Attribute domain is not defined! (ArgumentError)

later on. Additionally, the repositories need to provide the runit program in

Manual Server Installation If you need to set up the Chef server manually, start by installing the RabbitMQ messaging server [2]. openSUSE users should use the openSUSE Build Service to install rabbit-mq [3]. Doing so means that YaST automatically adds repositories that you need later on. Once you have RabbitMQ in place, it’s time to start the Chef configuration: sudo rabbitmqctl add_vhost /chef

openSUSE goes for zlib-devel and libxml-devel. Now, finally, you can install the Chef server sudo gem install chef‑server chef‑server‑api chef‑server chef‑solr

and add the really practical web front end: sudo gem install chef‑server‑webui

The next task on the list concerns the CouchDB database from the CouchDB package. If needed, you can start the service manually on openSUSE by typing rccouchdb start. The Chef server also requires Sun Java SDK version 1.6.0. Some distributions keep this package in an external or special repository. On Debian, you need to enable the nonfree package source; on Ubuntu 10.04, you can add the partner repository like this:

After completing this work, create the /etc/chef/server.rb configuration file. [Listing 1] gives you a template. As a minimum, you need to replace the domain name that follows chef_server_url with the output from hostname ‑f and add a password of your choice after web_ui_admin_default_password. All the other defaults you can keep, particularly the paths, which the server automatically creates later, should the need arise. In the next step, the script shown in [Listing 2] creates a pair of SSL certificates, which you will need. The following command line creates the search index:

sudo add‑apt‑repository "deb http://archive.canonical.com/ U

sudo chef‑solr‑indexer

sudo rabbitmqctl add_user chef testing sudo rabbitmqctl set_permissions ‑p /chef chef ".*" ".*" ".*"

lucid partner" sudo apt‑get update

Now install the JDK. On Debian and Ubuntu, the JDK is hidden away in the sun-java6-jdk package, whereas openSUSE calls it java1_6_0-sun-devel. Users on openSUSE will probably want to delete the OpenJDK packages java-1_6_0-openjdk and java-1_6_0-openjdkdevel to be on the safe side. Then, you just need to install the developer packages for zlib and libxml. Debian and Ubuntu call them zlib1g-dev and libxml2-dev;

w w w. a d m i n - m aga z i n e .co m

Another command launches the Chef SOLR Server sudo chef‑solr

and the Chef server itself, sudo chef‑server ‑N ‑e production

including the graphical web interface: sudo chef‑server‑webui ‑p 4040

‑e production

Admin 01

Chef

Ma n ag e m e n t

Table 1: Directories in a Repository Directory

Content

certificates/

SSL certificates (typically created by rake ssl_cert)

config/

General configuration files for the repository

cookbooks/

Complete cookbooks

roles/

Role definitions

site-cookbooks/

Modified cookbooks; any cookbooks stored here will overwrite or modify the ones stored in cookbooks

a package named runit (don’t install this yourself!). The Chef server also requires Sun Java SDK version 1.6.0, which the distributions love to hide in a special repository. Debian users need to enable the non-free package source for this, whereas Ubuntu users can add the partner repository with the following two lines: sudo add‑apt‑repository U "deb http://archive.canonical.com/ U lucid partner"

aid to creating simple scripts – for installing the full-fledged server and clients. To do this, create a ~/solo.rb file with the following three lines on each of the systems involved: file_cache_path "/tmp/chef‑solo" cookbook_path U "/tmp/chef‑solo/cookbooks" recipe_url U "http://s3.amazonaws.com/U chef‑solo/bootstrap‑latest.tar.gz"

This tells Chef Solo where the installation recipes are located.

sudo apt‑get update

Theoretically, the Chef server will run with the OpenJDK, although the developers do not give you any guarantees.

Lonely Kitchen Helper After fulfilling all the requirements, you can create configuration files for Chef Solo on the server and the client. This Chef variant runs the recipes directly on the client without involving the server. Without the server, Chef Solo is only useful as an

Chef on Call

If the command terminates with a cryptic error message, try running it again. During testing, the installation ran without any errors. First, the command installs a Chef client, then RabbitMQ, CouchDB, the developer packages for zlib and xml, and the Chef server, including the indexer and a web GUI you can use later to manage the Chef server (WebUI). It then goes on to create matching configuration files and the required directories and adds init script entries for the server to round off the process. At the end of this procedure, the Chef server should be listening on port 4000; the web GUI is accessible on port 4040. Java and RabbitMQ use the Apache SOLR-based full-text search engine built into the Chef server. Among other things, it provides information about the existing infrastructure, which in turn can be referenced for recipes. For details of the search function, see the wiki page [4].

Workers

Now it’s time to move on to the server candidate. On this machine, create a JSON configuration file named ~/chef.json to provide information about the node. See [Listing 3] for the file content. To match your local environment, you need to modify the server name for server_fqdn. To set up the full-fledged Chef server, give the command:

Once you have the server up and running, it’s time to turn to the client. Start by creating a ~/chef.json JSON configuration file. [Listing 4] gives you the content. The server_fqdn entry here must contain the server name, not the client’s. Now you can launch Chef Solo:

sudo chef‑solo ‑c ~/solo.rb ‑j U

sudo chef‑solo ‑c ~/solo.rb U

~/chef.json

‑j ~/chef.json U

Listing 1: Template for server.rb 01 log_level

:info

19 v alidation_key

"/etc/chef/validation.pem"

02 log_location

STDOUT

20 c lient_key

"/etc/chef/client.pem"

03 ssl_verify_mode

:verify_none

21 w eb_ui_client_name

"chef‑webui"

04 chef_server_url

"http://chef.example.com:4000"

22 w eb_ui_key

"/etc/chef/webui.pem"

06 signing_ca_path

"/var/chef/ca"

07 couchdb_database

'chef'

08 09 cookbook_path

[ "/var/chef/cookbooks", "/var/chef/site‑cookbooks" ]

10 11 file_cache_path

"/var/chef/cache"

12 node_path

"/var/chef/nodes"

13 openid_store_path

"/var/chef/openid/store"

24 w eb_ui_admin_user_name "admin" 25 w eb_ui_admin_default_password "somerandompasswordhere" 26 27 s upportdir = "/srv/chef/support" 28 s olr_jetty_path File.join(supportdir, "solr", "jetty") 29 s olr_data_path

File.join(supportdir, "solr", "data")

30 s olr_home_path

File.join(supportdir, "solr", "home")

14 openid_cstore_path "/var/chef/openid/cstore"

31 s olr_heap_size

"256M"

15 search_index_path

"/var/chef/search_index"

16 role_path

"/var/chef/roles"

33 u mask 0022

18 validation_client_name "validator"

35 M ixlib::Log::Formatter.show_time = false

Admin 01

w w w. a d m i n - m aga z i n e .co m

Chef

‑r http://s3.amazonaws.com/U chef‑solo/bootstrap‑latest.tar.gz

The tool creates a couple of directories, corrects the configuration files, and adds chef-client to the init scripts. The latter ensures that the client will talk to the server on booting and execute any recipe changes that have occurred in the meantime. After this, the client has to register with the server. To allow this to happen, copy the /etc/chef/validation. pem file from the server to the / etc/ chef/directory client-side and then restart the client manually: sudo chef‑client

The client automatically creates a key, which you need to add to the /etc/ chef/ client.pem file and which will sign every transaction with the server from this point on. Then you want to delete the validation.pem file for security reasons.

Librarian Now that you have the server and the client running, the next step is to create a repository server-side for your recipes: This is simply a hierarchy of multiple, standardized (sub-)directories. Of course, you could create them all manually, but the template provided by Opscode does a quicker job; you just need to download and unpack: wget http://github.com/opscode/U chef‑repo/tarball/master tar ‑zxf opscode‑chef‑repo‑U 123454567878.tar.gz

Because this cryptic number is difficult to remember in the daily grind, you might want to rename the directory (incidentally, the number comes from the versioning system and represents the Commit ID): mv opscode‑chef‑repo‑123454567878 U chef‑repo cd chef‑repo

[Table 1] explains the directory hierarchy in chef-repo. The recipes stored here are injected into the server by a tool named knife.

w w w. a d m i n - m aga z i n e .co m

Ma n ag e m e n t

Listing 2: SSL Certificates for the Chef Server 01 server_ssl_req="/C=US/ST=Several/L=Locality/O=Example/OU=Operations/CN=chef.example.com/ emailAddress=ops@example.com" 02 openssl genrsa 2048 > /etc/chef/validation.key 03 openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/validation.key > /etc/chef/validation.crt 04 cat /etc/chef/validation.key /etc/chef/validation.crt > /etc/chef/validation.pem 05 openssl genrsa 2048 > /etc/chef/webui.key 06 openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/webui.key > / etc/chef/webui.crt 07 cat /etc/chef/webui.key /etc/chef/webui.crt > /etc/chef/webui.pem

To prepare a recipe for action, run the command knife configure ‑i

and confirm the default responses by pressing Enter – except, enter your own username when asked Your client user name?, and type . (dot) in response to the Path to a chef repository (or leave blank)? query. Knife then registers a new client on the Chef server, creates the above-mentioned certificate in / .chef/ my-knife.pem, and finally creates the / .chef/ knife.rb configuration file.

Convenience Food Multiple recipes with the same objective can be grouped in a cookbook. For example, the mysql cookbook contains all the recipes required to install and set up the free database. For an initial test, it is a good idea to look for a simple cookbook [5]. In the section that follows, I will use the cookbook for emacs from the applications group as an example. In this example, I’ll use the package manager to install the popular Emacs text editor. After downloading the Cookbook archive, unpack it in the cookbooks subdirectory, then introduce the server to the new recipes: rake upload_cookbooks

The rake command automatically calls knife with the correct parameters, and knife then uploads all the cookbooks from the corresponding directory. To upload a single cookbook to the server, do this:

The target, upload_cookbook, is defined in the Rakefile provided by the repository.

GUI Management The server now knows the emacs cookbook, but the clients don’t. To change this, launch a browser and access the web front end with http:// chefserver.example.com:4040. Chef does not offer SSL encryption here. If you prefer a more secure approach, you could use Apache as a proxy. In the form that then appears, log in by typing the admin username [Figure 2]. The matching password is stored in the web_ui_admin_default_password line of the /etc/chef/server.rb file. Listing 3: ~/c hef.json for the Server 01 { 02 "bootstrap": { 03

"chef": {

"url_type": "http",

"init_style": "runit",

"path": "/srv/chef",

"serve_path": "/srv/chef",

"server_fqdn": "chefserver.example.com",

"webui_enabled": true

}

11 }, 12 "run_list": [ "recipe[bootstrap::server]" ] 13 }

Listing 4: ~/c hef.json for the Client 01 { 02 "bootstrap": { 03

"chef": {

"url_type": "http",

"init_style": "runit",

"path": "/srv/chef",

"serve_path": "/srv/chef",

08 09

"server_fqdn": "chefserver.example.com" }

10 }, 11 "run_list": [ "recipe[bootstrap::client]" ]

rake upload_cookbook[emacs]

12 }

Admin 01

Ma n ag e m e n t

Chef

at the same time and get in each other’s way.

Role-Out To group multiple cookbooks in a role, create a new file below Roles in the repository, say, beispiel.rb, with the following content: name "beispiel" description "Example of a role" run_list("recipe[emacs]", U "recipe[zsh]", "recipe[git]")

This groups the emacs, zsh and git recipes under the beispiel role name. Then send the role to the server like this: Figure 2: The web front end login page: the default password specified on the right is incorrect.

Changing the slightly cryptic default after logging in the first time is a good idea. Now go to the Nodes menu. When you get there, click the client name, change to the Edit tab, and finally drag the recipe you want to use from Available Recipes and drop it into the Run List (the recipe will slot into the top position in the list). In the example, you would now see emacs at the top [Figure 3]. To store this assignment, press the Save Node button bottom left on the page. Client-side now, manually launch the chef-client tool:

sudo chef‑client

This command line immediately opens a server connection, picks up the recipes assigned to the client (only emacs for the time being) and executes the recipes [Figure 5]. To allow this to happen on a regular basis, you should run the client at regular intervals as a daemon: chef‑client ‑i 3600 ‑s 600

In this example, the client contacts the server every 3,600 seconds. The -s parameter lets you vary the period slightly. If you don’t set this, all of your clients might query the server

rake roles

In the web front end, you can assign roles to a node just like cookbooks using drag and drop.

Freshly Stirred Ready-made recipes and cookbooks off the Internet will only cover standard application cases. For special cases, or individual configurations, you will typically need to create your own cookbook. The following, extremely simple example, creates a text file on the client called /tmp/thoughts.txt that is based on the quick_start cookbook [6], and it adds a sentence that is

G Figure 4: The Status tab lists all the nodes with their last contact attempts and recipe assignments (following Run List). F Figure 3: Using drag and drop to assign recipes to a node. In this example, the client runs the beispiel recipe first, followed by emacs.

Admin 01

w w w. a d m i n - m aga z i n e .co m

Chef

Figure 5: The client has picked up its assigned recipe from the server and executed it. This puts a preconfigured version of Emacs on its disk.

generated dynamically in part. Start by creating a new cookbook called beispiel in chef-repo: rake new_cookbook COOKBOOK=beispiel

The command creates a beispiel folder below cookbooks, populates it with the required subdirectories, then creates an empty recipe named default.rb. Before you start filling this with content, first create a template for the file you want to create, /tmp/thoughts. txt. This will later contain the sentence Thought for the day:

and the recipe will append an ingenious thought on a daily basis. The complete template is thus: Thought for the day: <%= @thought %>

The recipe will replace the wildcard with text later on. The new template needs to be in templates/default/; you can save it as thoughts.txt.erb. Most recipes use templates like this, or, to quote the developers: “We love templates.”

Hand Mixer Now, compose a matching recipe that picks up the template and uses it to generate the /tmp/thoughts.txt file. To save work here, you can extend the existing, but empty, default.rb recipe in the recipes subdirectory. The recipe for this example looks like: template "/tmp/thoughts.txt" do source "thoughts.txt.erb" variables :thought => U node[:thought] action :create end

w w w. a d m i n - m aga z i n e .co m

This should be fairly self-explanatory for Ruby aficionados: It creates the / tmp/ thoughts.txt file from the thoughts.txt.erb template and then replaces the wildcard with the content of the thought variable. Now you just need to think about what thoughts to use here.

Spice In this example, thought will be an attribute. Attributes store node-specific settings in a cookbook for recipes to evaluate and use. A typical attribute would be, say, a command-line parameter for a program launched automatically by a recipe. The attributes are identical for each call to the recipe and, thus, no more than constants provided by the recipe author. In contrast to genuine Ruby constants, attributes can be modified via the web interface (in the window used to assign cookbooks to nodes). A cookbook groups all of the attributes in its attributes subdirectory. For the example here, you need to create a beispiel.rb file with the following content: thought "Silence is golden ..."

Now you just need to register the new cookbook with the server rake upload_cookbooks

and assign it to one or multiple nodes in the web front end. After running chef-client, the / tmp/ thoughts.txt file should appear. This recipe leaves much scope for improvement. For example, you could randomly choose the thought of the day, which Ruby programmers should handle easily. Because recipes are full-fledged Ruby scripts, you can

Ma n ag e m e n t

draw from the full scope of the language and on RubyGems. In the case of the latter, the recipe should first check to see whether the Gem exists on the client and, if not, install it. The beispiel/metadata.json file stores metadata on the new cookbook. Before you roll out the cookbook in a production environment, you might want to add some details. As the file extension suggests, the file uses the JSON format.

Conclusions Chef is a complex piece of software, and once you have it running and have finished modifying or creating your recipes, it does make the administrator’s life much easier – at least on Linux systems. Unfortunately, the learning curve is very hard going for newcomers. The online documentation for Chef is fairly chaotic and incomplete [7]. If you need to know more about writing cookbooks, it is a good idea to download prebuilt examples and investigate them. The cookbook for emacs shows you how to use action :upgrade to install a package for example. Additionally, it is hard to find help or how-tos, even on the web, if you have a problem. Your best option here is to post your questions on the mailing list [8]. n Info [1] Installation guide: [http://wiki.opscode. com/display/chef/Installation] [2] RabbitMQ: [http://www.rabbitmq.com/server.html] [3] openSUSE build service for RabbitMQ: [http://software.opensuse. org/search?q=rabbitmq‑server& baseproject=openSUSE%3A11.2] [4] Information on full-text searches in Chef: [http://wiki.opscode.com/display/chef/ Search] [5] Repository with prebuilt cookbooks: [http://cookbooks.opscode.com/] [6] Cookbook quick start: [http://wiki.opscode. com/display/chef/Cookbook+Quick+Start] [7] Chef wiki: [http://wiki.opscode.com/display/chef/] [8] Chef mailing list: [http://lists.opscode. com/sympa/lists/opensource/chef]

Admin 01

Ma n ag e m e n t

Sysinternals

System monitoring with Sysinternals

Health Check Administrators don’t need a massive arsenal of tools just to monitor a couple of systems. With Microsoft’s free Sysinternals suite, admins can handle all sorts of tasks. By Thomas Joos The Sysinternal tools are free tools from Microsoft that can help Windows administrators handle many tasks. This article introduces the Sysinternal tools that are useful for system monitoring. All of the tools described here can be downloaded free of charge from the Microsoft site [1], either as individual downloads or as part of the Sysinternals suite. One advantage of the Sysinternals utilities is that you don’t need to install them, so they can be launched conveniently from a USB stick. When launched for the first time, the programs display a license dialog; you can suppress this dialog with the /accepteula option, which can be useful in scripting. Unfortunately, this option does not work for all of the Sysinternal tools. The programs only run on a Windows system as of Windows 2000 Server. For this article, I used Windows Server 2008 R2 and Windows 7. Windows Server 2008 R2, Windows Server 2008, Windows Vista, and Windows 7 do not support access to the hidden System $ shares such as C$, or admin$ as easily as Windows XP or Windows Server 2003; the computers do not belong to a Windows

Admin 01

domain because the new operating systems block access to administrative shares by authentication of local user accounts. Some Sysinternal tools, such as PSInfo.exe, require access to the admin share and thus will not work at first. To allow access, you must enable local logins to administrative shares in the Registry of standalone computers. To do so, launch the Registry Editor by typing regedit, then navigate to HKEY_LOCAL_MACHINE\SOFT‑ WARE\Microsoft\Windows\CurrentVer‑ sion\Policies\System. Create a new Dword entry with the label LocalAc‑ countTokenFilterPolicy, set the value to 1, then restart the computer.

LDAP Microscope Insight for Active Directory, also known as AdInsight, lets you monitor the LDAP connections on a domain controller in real time with a GUI. The user interface is similar to the Sysinternal tools Regmon and Filemon. The tool investigates calls to the wldap32.dll file, which most programs, including Exchange, use for LDAP-based access to Active Directory per LDAP.

AdInsight lists all requests including those that are blocked. This gives administrators an easy option for analyzing authentication problems with Active Directory-aware programs and identifying clients and servers that set up a connection to the domain controller. AdInsight logs all requests issued to domain controllers and stores them as an HTML report or text file for troubleshooting purposes. The logfile contains the client request and responses that the client received via LDAP. AdInsight also logs access to system services (Figure 1). When a program such as Exchange accesses the domain controller, the window fills with information; then, you can right-click to display details of the individual entries, as well as filter the display via the menu. The display also includes the name of the accessing user. Unfortunately, AdInsight only lets you monitor local access; over-the-wire diagnostics via remote access are not supported. However, AdInsight’s search function does let you filter by process, error, or request response. The tool selects the response to let you perform specific monitoring.

w w w. a d m i n - m aga z i n e .co m

Sysinternals

Ma n ag e m e n t

AdInsight also supports automated deployment and offers a variety of options for this. One useful automation feature is the ability to write the log to a file without displaying the events in the GUI. AdInsight runs on Windows 2000 Server or newer and includes a help file, which can be a useful aid for tasks that require more thorough analysis.

Filesystem, Registry, Processes The Process Monitor provides a graphical user interface for monitoring and color-tagging the filesystem, registry, and process/â&#x20AC;&#x2039;thread activity in real time. The tool combines two programs standard to Sysinternals: Filemon and Regmon. With a click of the button, you can enable and disable the individual monitoring options and restrict your monitoring of registry and filesystem access and process calls to an area in which you are interested. Process Monitor is a valuable aid for monitoring stability and identifying bottlenecks; it is capable of logging all read and write access to the system and its media. Additionally, it displays comprehensive data on any process or thread that is launched or terminated. Also, you can monitor any TCP/â&#x20AC;&#x2039;IP connections that are opened, as well as UDP traffic. Note that Process Monitor does not save the content of the TCP packets or payload data; it is not specifically designed for network monitoring. If that is what you need, you might prefer a tool such as Wireshark [2]. The Process Monitor tool is also extremely useful for troubleshooting local connection and privilege problems (Figure 2). If needed, it can display additional information on active processes (e.g., their DLL files or the parameters set when the process was launched). The filter function allows you to reduce the volume of data output generated by Process Monitor. For example, it lets you hide any processes with a specific string in their names, without filtering out registry

w w w. a d m i n - m aga z i n e .co m

Figure 1: AdInsight helps you investigate LDAP access to domain controllers.

entries that contain the same string. Additionally, you can enable multiple filters at the same time and save the configuration. For more efficient diagnostics, you might want to save the logfile and then load it for analysis, applying additional filters as needed. The Tools menu gives you a selection of preconfigured views. Process Monitor can also monitor the boot process on a server because it launches at a very early stage. You can redirect all the output to a file. If Windows fails to boot, analysis of the output file gives you a fast way to identify the issue. Just like all Sysinternal tools, Process Monitor is easy

to use and does not have an extended learning curve. Process Explorer takes things a step further than Process Monitor, in that Process Explorer lists all the processes in a window and includes more detailed information on the current process, such as access to directories (Figure 3). In DLL View mode, the tool shows which libraries are used and where they originated. The process menu contains a Restart entry that first kills a process and then restarts it. Also, you can temporarily stop individual threads and highlight processes in different colors. Process Explorer nests in the taskbar, which provides an at-

Figure 2: Monitoring processes, the registry, and filesystem access with Process Monitor.

Admin 01

Ma n ag e m e n t

Sysinternals

a-glance overview of the current CPU load and disk utilization.

Logging into the Domain The LogonSessions tool lists all the active sessions on a computer at the command line (Figure 4). If you run this command without setting options, the prompt buffer size might not be sufficient to display all of the information. In this case, you can use the logonsessions | more command or modify the command-line prompt properties to increase the buffer size. Alternatively, you can redirect the output to a file by specifying > logon. txt. The ‑p option tells LogonSessions to display the active processes in the individual sessions for login users, allowing you to monitor who is logged in to a terminal server and which applications the users are running. The tool is also really useful in Active Directory environments.

Monitoring Local Logins Just like LogonSessions, PsLoggedOn is a tool for monitoring logged in users. If you type psloggedon at the command-line prompt, the tool will display all the logged in users on the local system with their login times (Figure 5). Also, you can see who is accessing a share on the server. If you launch the tool with the username as an argument, it will investigate all the computers in the network environment or domain and show you where the user is logged in. Unfortunately, this is not an entirely error-free process in Windows DOS. Although you can identify domain users logged in

Figure 3: Monitoring and controlling processes with Process Explorer.

to the local computer, you can’t actually find out where else the user is logged on in the domain. If you use psloggedon \\computername, you can also display domain users logged in remotely. To access a computer on the network, the user account with which you launch PsLoggedOn must have administrative privileges on the remote machine. Although you can check who has an active session on a computer without add-on tools, with net session, for example, you can only monitor local logins, not network logins. The tool accesses the HKEY_USERS registry key for this check (Windows creates a separate key for each logged in user by default). The ‑l option lists locally logged in users only; ‑x leaves out the login times. The PsLoggedOn tool is useful when you need to investigate unau-

thorized access to your network and see the computers on which a user is logged in. Psinfo is another tool in the collection that lets you retrieve a variety of information for the local system, such as the operating system, the CPU, the build number, and whether the computer is a member server or a domain controller. Calling psinfo /? lists more options. For example, psinfo ‑s lists the software packages installed on the machine. Although Microsoft provides programs such as system‑ info.exe and msinfo32.exe with Windows, Psinfo can also retrieve information from remote computers across the wire.

Event Lister The PsLogList utility is a commandline tool that retrieves the event

G Figure 5: Listing logged in users with PsLoggedOn. F Figure 4: Logged in users and services.

Admin 01

w w w. a d m i n - m aga z i n e .co m

Sysinternals

Table 1: Selected PsLogList Options Name

Function

File

Runs the command on all the computers listed in the file. Each computer needs a separate column in the text file.

‑a dd/mm/yy

Lists the entries after the specified date.

‑b dd/mm/yy

Lists entries before the specified date.

‑c

Deletes the event logs after displaying them in PsLogList, which is useful in the case of batch-controlled retrieval.

‑d n

Displays the entries for the past n days.

‑e id1, id3, ...

Filters entries with defined IDs.

‑f

Filters entries with specific types (e.g., ‑f w filters warnings). You can use arbitrary strings here.

‑h n

Lists entries for the past n hours.

‑i id1, id3, ...

Shows entries with IDs defined in a comma-separated list of IDs.

‑l event_log_file

Stores entries for the defined event log.

‑m n

Lists entries for the past n minutes.

‑n n

Only shows the n latest defined entries.

display from various computers and then displays and compares events. If you run the tool without any options, it will output the entries in the local system event log. The program has numerous options that give you various ways of comparing the event logs that you retrieve. Table 1 lists some of PsLogList’s command-line options. By default, the tool uses the system event log; you can select the event log by entering the first letter or by entering an abbreviation, such as sec for security.

Monitoring Shares The ShareEnum tool lets you monitor shares and their security settings by scanning either an IP range or all the PCs and servers in a domain (or all the domains in a network) for shares (Figure 6). To display all of this information reliably, you need to log in as the domain administrator, which

is the only account with privileges on all the PCs and servers in the domain. In networks without a domain, you can use the administrator account for the login; however, you need to set the password to be the same on all the computers you want to monitor. ShareEnum shows you not only the shares but also the local paths for the shares on the computer. The Refresh button tells ShareEnum to launch a scan. If you want to scan an individual computer, enter the same IP address as the start and end address of the IP range. The tool will show you all of the shares on the network in a single window and list the access privileges to boot. If you only want to see local privileges, Sysinternals gives you a choice of two tools: AccessChk and AccessEnum. AccessChk outputs an exhaustive list of a user’s rights at file, service, or registry level at the command line, giving you a quick overview of how access privileges are defined for a specific user. To see the privileges for the administrator in the C:\Windows\ System32 directory, you would type accesschk

Figure 6: ShareEnum gives you a neat list of all the shares and assigned privileges on the network.

w w w. a d m i n - m aga z i n e .co m

administra‑ tor c:\windows\ system32. For

Ma n ag e m e n t

each file, you are told whether the user has read (R), write (W), or both (RW) types of access. If you use | more to redirect the output from this command, the display will pause on each page, and you can continue by pressing any key. In a similar fashion, >file.txt redirects the output to a file. With this approach, you do not see any command-line output. The accesschk user ‑cw * command shows you the Windows services to which a group or user has write access. If you want to see a user’s access privileges for a specific registry key, the accesschk ‑kns contoso\tami hklm\software command is your best bet. The AccessChk tool is excellent for checking computers for vulnerabilities, and it also supports scripting. AccessEnum gives you a GUI that presents a full directory tree of user privileges. In other words, AccessEnum is the graphical front end for AccessChk. The download contains both files because AccessEnum relies on the AccessChk program for performing scans. In the GUI, you can select a directory and scan it for privilege assignments. The tools also show denied privileges.

Conclusions The free command-line and GUI tools in the Sysinternals suite are a featurerich addition to any Windows administrator’s toolbox. The individual programs are useful for monitoring processes, users, and network connections, and tools like AdInsight are indispensable aids if you need to troubleshoot Active Directory logins. n

Info [1] Sysinternals tools: [http://www.sysinternals.com] [2] Wireshark: [http://www.wireshark.org] The Author Thomas Joos is a freelance IT consultant who has worked in the IT industry for more than 20 years. Among his many projects, Joos writes practical guides and articles on Windows and other Microsoft topics. You can meet him online at [http://thomasjoos.spaces.live.com].

Admin 01

N u ts a n d B o lts

a Vas

PAM and Hardware

ileva

, Foto lia.co m

Flexible user authentication with PAM

Turnkey Solution PAM is a very powerful framework for handling software- and hardware-based user authentication, giving administrators a choice of implementation methods. By Thorsten Scherf Hardware innovations are daily business in user account authentication. Pluggable Authentication Modules (PAM) help transparently integrate these new devices into a system. This gives experienced administrators the option of offering a variety of different authentication methods to their users while providing scope for controlling the total user session workflow.

Old School User logins on Linux systems are traditionally handled by the /etc/

Admin 01

passwd and /etc/shadow files. When a user runs the login command to

log in to the system with a name and password, the program creates a cryptographic checksum of the password and compares the results with the checksum stored for this user in the /etc/shadow file. If the checksums match, the user is authenticated; if not, the login will fail. This approach doesn’t scale well. In larger environments, user credentials are typically stored centrally on an LDAP server, for example. In this case, the login program doesn’t retrieve the password checksum from

the /etc/shadow file but from a directory service. This task can be simplified by deploying PAM [1].

Modular Authentication Originally developed in the mid1990s by Sun Microsystems, PAM is available on most Unix-style systems today. PAM offloads the whole authentication process from the application itself to a central framework comprising an extensive collection of modules (Figure 1). Each of these modules handles a specific task; however, the application only gets to

w w w. a d m i n - m aga z i n e .co m

PAM and Hardware

N u ts a n d B o lts

Figure 1: PAM provides a centralized user management framework for the

Figure 2: A classic PAM configuration file contains modules and libraries that the

application.

administrator can use to customize PAM.

know whether or not the user logged in successfully. In other words, it is PAM’s job to find a suitable method for authenticating the user. The PAM framework defines what this method looks like, and the application remains blissfully unaware of it. PAM can use various authentication methods. Besides popular networkbased methods like LDAP, NIS, or Winbind, PAM can use more recent libraries to access a variety of hardware devices, thus supporting logins based on smartcards or the user’s digital fingerprint. One-time password systems, such as S/Key or SecurID, are also supported by PAM, and some methods even require a specific Bluetooth device to log in the user. The way PAM works is fairly simple. Each PAM-aware application (the application must be linked against the libpam library) has a separate configuration file in the /etc/pam.d/ folder. The file will typically be named after the application itself – login, for example. Within the file, modules distribute PAM tasks among themselves. Numerous libraries are available in each group, and they handle a variety of tasks within the group (Figure 2). Control flags let you manage PAM’s behavior in case of error – for example, if a user fails to provide the correct password or if the system is unable to verify a fingerprint.

Fingerprints More recent PAM libraries allow administrators to authenticate users by means of smartcards, USB tokens, or biometric features. State-of-the-art

w w w. a d m i n - m aga z i n e .co m

notebooks often include a fingerprint reader that allows the owner to use a digital fingerprint when logging into the system. The PAM ThinkFinger library [2] provides the necessary support. According to the documentation, the module will support the UPEK/ SGS Thomson Microelectronics fingerprint reader used by most recent Lenovo notebooks and many external devices. Most major Linux distributions offer prebuilt packages for the PAM libraries. You can use your distribution’s package manager to install the software from the repositories. To install the required packages on your hard disk, you would give the yum install thinkfinger

command on a Fedora system and apt‑get install thinkfinger‑tools U libpam‑thinkfinger

on Ubuntu Hardy. Gentoo admins can issue a compact command: emerge sys‑auth/thinkfinger

If you’re using openSUSE, you’ll need the libthinkfinger and pam_thinkfinger packages, the repository versions of which are not up to date. You might prefer a manual install with the typical ./configure, make, make install steps and files from the current source code archive. Debian users on Lenny will need to access the Experimental repository and then type aptitude install libthinkfinger0 U libpam‑thinkfinger thinkfinger‑tools

for the install.

Before you modify the existing PAM configuration, you might want to test the device itself. To do so, scan a fingerprint by giving the tf‑tool ‑‑acquire

command (Figure 3). Then you can use tf‑tool ‑‑verify

to verify the results. You might see a Fingerprint does *not* match message at this point; initial attempts can be fairly inaccurate because you will need to familiarize yourself with the device. If you drag your finger too quickly or too slowly across the scanner, the device could fail to identify the fingerprint correctly. In this case, it will output an error message and quit. When you achieve reliable results from fingerprint scans, you can delete the temporary file with the test scan in /tmp and create an individual file for each user on the system that will contain the user’s fingerprint. The command is tf‑tool ‑‑add‑user username

(Figure 4). Users must scan their fingerprints three times for this to work. If the fingerprint is identified correctly each time, the tool will store it in a separate file below /etc/pam_think‑ finger/. Once everything is working, you can begin the PAM configuration. Figure 2 shows a PAM configuration for the login program that lists just one authentication module: pam_unix. If you want to authenticate against the fingerprint scanner first, you need to

Admin 01

N u ts a n d B o lts

PAM and Hardware

Figure 3: tf-tool gives you an option for testing your fingerprint scanner …

call the pam_thinkfinger PAM module before pam_unix. To prevent PAM from prompting users to enter their password despite passing the fingerprint test, you need add a sufficient control flag. This tells PAM not to call any more libraries once an authentication test has succeeded and to return PAM_SUCESS to the calling program – login in this example. If the fingerprint-based login fails, pam_unix is called as a last resort and will prompt for the user’s regular password. Manually entering the PAM libraries for all of your PAM-aware applications in every single PAM configuration file would be fairly tedious. A centralized PAM configuration file gives you an alternative. On Fedora or Red Hat, this file is named /etc/

Figure 4: … and then creating a fingerprint for each user.

pam.d/system‑auth, although other Linux distributions call it /etc/pam/ common‑auth. You can enter all the libraries against which you want to authenticate your users in the file (Figure 5). The include control flag then includes the file in all your other PAM configuration files. From now on, this makes all the programs in the PAM libraries listed by the centralized configuration file available in the individual PAM files, including the pam_thinkfinger module.

USB Tokens The pam_usb library supports another hardware-based approach, in which PAM checks to see whether a specific USB device is plugged into

the machine. If so, the user is logged in; if not, access to the system is denied. The plugged in device is identified by its unique serial number and model and vendor names. Additionally, a random number is stored on the USB device and on the computer; the number changes after each successful login attempt. When a user logs in, PAM checks both the specified USB device properties and the random number. If the number stored on the USB does not match the number on the disk, the login fails. This prevents attackers from stealing the random number, placing it on their own USB device, and then modifying the properties of their own device to access the system. Because the random number on the system changes after each login,

Figure 5: On Fedora, system‑config‑authentication provides a ba‑

Figure 6: The USB device is identified by its properties. If the user tries to log in without the

sic PAM configuration tool.

device, it will not work.

Admin 01

w w w. a d m i n - m aga z i n e .co m

November 13-19, 2010 • Ernest N. Morial Convention Center • New Orleans, Louisiana Conference Dates: November 13-19, 2010 • Exhibition Dates: November 15-18, 2010

The Future of Discovery

Sponsors: IEEE Computer Society ACM SIGARCH

T h e I n t e r n a t i o n a l C o n f e re n c e fo r H i g h Pe r fo r m a n c e C o m p u t i n g , N e t wo r k i n g , S t o ra g e, a n d A n a l ys i s

N u ts a n d B o lts

PAM and Hardware

the stolen number will not match the number on the system. Gentoo and Debian Linux offer prebuilt packages of this PAM library. In both cases, you can use the package manager to install, as described for pam_thinkfinger. Users on any other Linux distribution can download the current source code archive [3] and run make; make install to compile the required files and install them on the local system. Then you need to connect any USB device – it can be a cellphone with an SD card if you like – and store its properties in the /etc/pamusb.conf file. The command for this is pamusb‑conf ‑‑add‑device USB‑device‑name

(Figure 6). The command pamusb‑conf ‑‑add‑user user

lets you add more users to the configuration and generates a matching random number. The number for each user is stored both on the USB device and on the system. Also, the tool adds each user to the XML-based /etc/pamusb.conf configuration file. You can use the file to define actions for each user; these actions will run when the USB is plugged in or unplugged. For example, the entry in Listing 1 of the configuration file Listing 1: Configuration File for pam_usb 01 <user id="tscherf"> 02 <device> 03

/dev/sdb1

04 </device> 05

< agent event="lock">gnome‑screensaver‑command

< agent event="unlock">gnome‑screensaver‑command

‑‑lock</agent> ‑‑deactivate</agent> 07 </user>

Listing 2: USB Device-Based Authentication [tscherf@tiffy ~]$ id ‑u 500 [tscherf@tiffy ~]$ su ‑ * pam_usb v0.4.2 * Authentication request for user "root" (su‑l) * Device "/dev/sdb1" is connected (good). * Performing one time pad verification... * Verification match, updating one time pads... * Access granted. [root@tiffy ~]# id ‑u 0

Admin 01

automatically blocks the screen if the USB device is unplugged: Then You need to add the pam_usb PAM library to the corresponding PAM configuration file, /etc/pam.d/system‑auth or /etc/pam.d/common‑auth. If you use the sufficient control flag, users can log in to the system by plugging in the USB device, assuming the random number for the user matches on both devices (Listing 2). To enhance security, you can replace the sufficient control flag with re‑ quired. This setting first looks for the USB device, but even if the device is identified correctly, PAM still prompts the user for a password in the next stage of the login process. Both of these tests have to complete successfully for the user to log in. All of the hardware-based login methods I have looked at thus far are easily set up, but they all have vulnerabilities, and it is easy to fake fingerprints. Also, USB sticks can be stolen, thereby putting an end to any security they offered. If you take your security seriously, you will probably want to use two-factor authentication. This method inevitably involves using chip cards with readers or USB tokens with one-time passwords and PINs.

Yubikey A small company from Sweden, Yubico [4], recently started selling Yubikeys (Figure 7), which are small USB tokens that emulate a regular USB keyboard. The key has a button on top which, when pressed, tells the token to send a one-time password (OTP) to the active application, such as a login prompt on an SSH server or the login window of a web service. The OTP is verified in real time by a Yubico authentication server. Because the software was released under an open source license, you could theoretically set up your own authentication server on your LAN. This would remove the need for an Internet connection. The way the token works is quite simple. In contrast to popular RSA tokens, Yubikey doesn’t need a battery because the OTP is not generated on

the fly; instead, one-time passwords are defined in advance. The passwords are stored on the token and in a database on the authentication server. When you press the Yubikey button, the key sends one of these OTPs to the active application, which then uses an API to access the server and verify the password. If this fails (Unknown Key) or if the password has already been used (Replayed Key), an error message is output and the login fails. If the server identifies the key as valid, it sets usage‑count to 1 and the user is authenticated. The user cannot login with this key anymore times. Because of the simple API, more and more applications are relying on authentication against the Yubico server. One example is the plugin for the popular WordPress blog, which allows users with a Yubikey to log in to the blog. A project from Google’s Summer of Code produced a PAM module that supports logging in to an SSH server [5]. Instead of typing your user password at the login prompt, you simply press the button on the Yubikey to send a 44-character, modhex-encoded password string to the SSH server. The server then verifies the string by querying the Yubico server. The first 12 characters uniquely identify the user on the Yubikey server; the remaining 32 characters represent the one-time password. You can define a central file on the SSH server to specify users permitted to log in by producing a Yubikey. To do so, first create a /etc/yubikey‑us‑ ers.txt file with a username, a colon separator, and the matching Yubikey ID (i.e., the first 12 characters of the user’s OTP) for each user. Alternatively, users can create a file (~/. yubico/authorized_yubikeys) with the same information in their home directory. You need to configure PAM to verify the OTP against the Yubico server. To do so, add a line for the Yubikey to your /etc/pam.d/sshd file (Listing 3). The configuration shown in Listing 3 runs this authentication in addition

w w w. a d m i n - m aga z i n e .co m

PAM and Hardware

to the regular, system‑auth-driven authentication method. But if you replace the required flag with suffi‑ cient, there is no need for the user to log in after the Yubikey OTP has been validated. Unfortunately, the Yubikey is not protected by an additional PIN, and the system is vulnerable if the token is stolen. An unauthorized user in possession of a token would be able to spoof a third party’s identity. The developers are working on adding PIN protection for OTPs, and an unofficial patch is already available.

X.509 Certificates and PAM Classic two-factor authentication typically relies on chip cards. The cards typically contain a certificate protected by a PIN. The PAM pam_ pkcs11 library allows users to log in to the system via an X.509 certificate. The certificate contains a private/ public key pair. Both can be stored on a suitable chip card, with the private key protected by an additional PIN to prevent identity spoofing simply by stealing a chip card. To log in, you need both the chip card and the matching PIN. If the PIN is unknown, the login fails. The details of the login process are as follows: The user inserts the chip card into the reader and enters the PIN. The system searches for the certificate with the public key and private key

on the card. If the certificate is valid, the user is mapped onto the system. The mapping process can retrieve a variety of information from the certificate, typically the Common Name or the UID stored in the certificate. To make sure the user really is who they claim to be, the system generates a random 128-bit number. A function on the chip card then encrypts the number using the private key, which is also stored on the card. The user needs to enter the right PIN to be able to access the private key. The system then uses the freely available public key to decrypt the encrypted number. If the results match the random number, the user is correctly authenticated because the two keys match. The hardware required for this setup is a chip card with a matching reader – for example, the Gemalto e-Gate or SCR USB device by SCM. You can use any Java Card 2.1.1 or Global Platform 2.0.1-compatible token: Gemalto Cyberflex tokens are widely available. Various software solutions are also available: The approach described in this article relies on the pcsc-lite and pcsc-lite-libs packages for accessing the reader.

Public Key Infrastructure It makes sense to use X.509 certificates, but only if you have a complete Public Key Infrastructure (PKI) set up.

N u ts a n d B o lts

In this example, I’ll use Dogtag [6] from the Fedora project as a PKI solution. Users with other distributions might prefer OpenSC [7]. The PAM library is the same for both variants, pam_pkcs11. Dogtag consists of various components. For this setup, you’ll also need a Certificate Authority (CA) to create the X.509 certificates. Online Certificate Status Protocol (OCSP) is used for online validation of the certificates on the chip cards. For offline validation, you just need the latest version of the Certificate Revocation List (CRL) on the client system. Of course, you also need a way of moving the user certificate from the certificate authority to the chip card. You can use the Enterprise Security Client (ESC) to open a connection to another PKI component, the Token Processing System, for this. Assuming correct authentication, the user certificate is then copied to the chip card in the enrollment process. The ESC tool then gives the user a convenient approach to managing the card. If the user needs to request a new certificate from the CA or needs a new PIN for the private key on the card, it’s no problem with ESC. If you use OpenSC to manage your Listing 3: PAM Configuration for a Yubikey auth required pam_yubico.so authfile=/etc/yubikey‑users. txt auth

include

system‑auth

account

required

pam_nologin.so

account

include

system‑auth

password

include

system‑auth

session

required

pam_selinux.so close

session

required

pam_loginuid.so

session

required

pam_selinux.so open env_params

session

optional

pam_keyinit.so force revoke

session

include

password‑auth

Listing 4: Configuration File for pam_pkcs11 01 pkcs11_module coolkey {

Figure 7: USB keyboard emulation means that the Yubikey for one-time passwords doesn’t need special drivers. The token works with the press of a button.

w w w. a d m i n - m aga z i n e .co m

module = libcoolkeypk11.so;

description = "Cool Key"

slot_num = 0;

ca_dir = /etc/pam_pkcs11/cacerts;

nss_dir = /etc/pki/nssdb;

crl_dir = /etc/pam_pkcs11/crls;

crl_policy = auto;

09 }

Admin 01

N u ts a n d B o lts

PAM and Hardware

cess should be completely transparent. The Thunderbird email program can use the card to sign and encrypt email; Firefox can use the certificate for client-side authentication against a web server. The reward for all this configuration is a large choice of deployment scenarios. The ESC Guide [10] has a more detailed description of the tool and its configuration.

Conclusions

Figure 8: The Security Client gives you an easy option for managing chip cards.

chip card, you can transfer a prebuilt PKCS#12 file [8] to the card using: pkcs15‑init U ‑‑store‑private‑key tscherf.p12 U ‑‑format pkcs12 U ‑‑auth‑id 01

The PKCS#12 file contains both the public key and private key. If you have a user certificate from a public certification authority like CACert [9], you can use your browser’s certificate management facility to export the certificate to a file and then transfer it to the chip card as described. If you don’t have a certificate, you can create a request and send it to the appropriate certificate authority. Once the authority has verified your request, it will return a certificate to you. For both approaches, you can use the /etc/pam_pkcs11/pam_pkcs11.conf file to define the driver for access to the chip card. The driver can be modified in the configuration file, as shown in Listing 4. Here, you must specify the correct paths to the local CRL and CA certificate repository. The CRL database is necessary to check that the certificate on a user’s chip card is still valid and has not been revoked by the certificate authority. You need the certificate for the certificate authority that issued the user’s certificate from the CA certificate repository. This makes

Admin 01

it possible to validate the user’s authenticity.

Certificates for Thunderbird and Co. Applications that rely on Network Security Services (NSS) for signing or encrypting email with S/MIME, such as Thunderbird, use a file in the nss_ dir as the CA database; applications based on the OpenSSL libraries use the database in the ca_dir directory. The certutil tool can import the CA certificate into the NSS database; OpenSSL-based certificates can simply be appended to the existing file. Finally, you can define the mappings between user certificates and Linux users in the pam_pkcs11 configuration file. Various mapping tools are available for this, specified as follows: use_mappers = cn, uid

Next, you still need to add the PAM pam_pkcs11 library to the correct PAM configuration file – that is, /etc/ pam.d/login or /etc/pam.d/gdm. You can edit the file manually or use the system‑config‑authentication tool referred to previously. When you insert the chip card into the reader and launch the ESC tool, you should be able to see the certificate (Figure 8). If you now attempt to log in via the console or a new GDM session, the authentication pro-

PAM is a very powerful framework for handling authentication. As you can see from the PAM libraries introduced in this article, the functional scope is not just restricted to authenticating users but also covers tasks such as authorization, password management, and session management. Administrators who take the time to familiarize themselves with configuring PAM, which isn’t always trivial, will be rewarded with a feast of feature-rich and flexible options for password- and hardware-based authentication and authorization. n

Info [1] Linux PAM: [http://www.kernel.org/pub/ linux/libs/pam/] [2] PAM ThinkFinger: [http://thinkfinger.sourceforge.net] [3] pam_usb: [http://downloads.sourceforge.net/ pamusb/pam_usb‑0.4.2.tar.gz?download] [4] Yubico website: [http://www.yubico.com/ products/yubikey/] [5] SSH server for Yubikey: [http://code. google.com/p/yubico‑pam/downloads/] [6] Dogtag PKI: [http://pki‑svn.fedora.redhat. com/wiki/PKI_Main_Page] [7] OpenSC: [http://www.opensc‑project.org] [8] PKCS specifications: [http://en.wikipedia.org/wiki/PKCS] [9] CACert certificate authority: [http://cacert.com] [10] ESC Guide: [http://directory. fedoraproject.org/wiki/ESC_Guide] The Author Thorsten Scherf is a Senior Consultant for Red Hat EMEA. You can meet him as a speaker at conferences. He is also a keen marathon runner whenever time permits.

w w w. a d m i n - m aga z i n e .co m

ON NEWSSTANDS NOW ! UBUNTU USER MAGAZINE Includes a comprehensive Discovery Guide to help new users install, ee Ubuntu r f configure, and explore s e d lu c Also in x” DVD! Ubuntu! “Lucid Lyn Find out more on www.ubuntu-user.com

n u tS A n d b o ltS

modSecurity

Apache Protector even securely configured and patched web servers can be compromised because of vulnerabilities in a web application. modSecurity is an Apache extension that acts as a web application firewall to protect the web server against attacks. by Sebastian wolfgarten Security issues on the web are no longer typically a result of poor configuration or the lack of up-to-date server software. Tomcat, Apache, and even IIS have become extremely mature over the past few years – so much so that they don’t have any noticeable vulnerabilities, although exceptions can always turn up to prove the rule. Thus, hackers have turned their attention to the web applications and scripts running on the servers. Increasingly complex user requirements are making web applications more complex, too: Ajax, interaction with external databases, back-end interfaces, and directory services are just part of the package for a modern application. And, attack vectors grow to match this development (see the “Attacks on Web Servers” box).

Firewalls for the Web In contrast to legacy packet filters, Web Application Firewalls (WAFs) don’t inspect data in the network or transport layer, but rather at the HTTP protocol level (i.e., in OSI Layer 7) [1]. They actually speak HTTP. For

Admin 01

this to happen, these firewalls analyze incoming and outgoing client requests and server responses to distinguish between benevolent and malevolent requests on the basis of rules. If necessary, they can even launch countermeasures; if configured to do so, the software will also inspect encrypted HTTPS connections.

Accessories en Masse Where classical network-based firewalls – I’m exaggerating slightly here – either permit any or no HTTP connections, WAFs target individual HTTP connections based on their content. ModSecurity is a highperformance WAF for Apache and a complex module for the Apache web server. Originally developed by Ivan Ristic, Breach Security handles its distribution and development [2]. Two variants of the software are available: the open source variant released under the GPLv2, and a commercial version with professional support, pre-configured appliances, and management consoles. ModSecurity runs on Linux, Solaris, FreeBSD, OpenBSD,

NetBSD, AIX, and Windows, with the later versions only available for Apache 2.x. This article discusses version 2.5.10; the successor 2.5.11 is merely a bugfix. The software’s functional scope is enormous but comprehensively documented [3]. It logs HTTP requests and gives administrators unrestricted access to the individual elements of a request, such as the content of a POST request. It also identifies attacks in real time based on positive or negative security models and detects anomalies based on supplied patterns for known vulnerabilities. The powerful rules discover whether credit cards are in the data stream or use GeoIP to prevent access from certain regions. ModSecurity checks not only incoming requests but also the server’s outgoing responses. The software can implement chroot environments. As a reverse proxy, it protects web applications on other web servers, such as Tomcat or IIS. Breach also provides a collection of core rules that guarantees the basic security of the web server. Comprehensive documentation, many examples, and a mailing list provide support for the user. This makes ModSecurity a good choice for protecting web servers and their applications against vulnerabilities. But before you can even consider tackling the highly

w w w. A d m i n - m AgA z i n e .co m

modSecurity

complex configuration, you first need to install the third-party module.

Packages for any Distribution If you prefer not to build the package for Apache 2 yourself, you can pick up pre-built packages for Debian, RHEL, CentOS, Fedora, FreeBSD, Gentoo, and Windows. The manual install requires the Apache mod_unique_id module, which is not automatically provided by some distributions. You can use the

n u tS A n d b o ltS

Attacks on web Servers Compared with local applications, web applications are more vulnerable because they involve so many different components – from the browser and the Internet infrastructure to the web server and the back ends beyond. Vulnerabilities can occur anywhere, but the server is always at the center of this environment. If the web application doesn’t sufficiently validate user input and instead passes it to a database running in the background, attackers could use SQL injection to inject their own commands into the command chain. Thus, the attacker would be able to read, modify, or delete data and thereby exert a major influence on the application. If an application also allows attackers to store files on the web server and execute them over the web, the intruder could set up a web shell. Because the server will execute the attacker’s files, the attacker can run operating system commands on the web server and finally escalate their privileges to interactive shell access. Although the architecture of a carefully configured Apache will not give the attacker root access, this is often unnecessary to access sensitive data. And the more services that are installed on the server, the more likely it is that the attacker will find one that is vulnerable.

LoadModule security2_module U modules/mod_security2.so

directive to integrate the module into the Apache configuration file, httpd. conf, if your distribution doesn’t do this for you. After restarting, the web server lists the module in its error_log.

Filter Rules ModSecurity has a mass of configuration options, but to understand how it works, all you need is the basic configuration. The SecRuleEngine option activates the module’s filter mechanism and allows it to process filter rules. The settings here are On, Off, and DetectionOnly, which tells the module to monitor, but not become actively involved with, the clientserver connection, even if individual rules are configured to let it do so. This setting is useful for testing the module and your own rules. To help you get started, or for debugging, you also want to enable the SecDebugLog option to define a troubleshooting log (e.g., SecDebugLog /var/log/httpd/modsec_debug.log). Additionally, you can set the SecDebugLogLevel parameter to specify verbosity, on a scale of 0 to 9, which issues comments on its own activities and the way it processes user-defined rules. Levels 4 or 5 are useful for fine tuning or troubleshooting. In production, set this parameter to 0. The SecDefaultAction parameter defines ModSecurity’s default behavior for requests that match a filter rule

w w w. A d m i n - m AgA z i n e .co m

without a corresponding action. The option also expects you to specify a processing phase for the standard action, as listed in Table 1. ModSecurity applies filter rules in five different phases of processing and responding to a client request [4]. In real life, only phase 2, in which the client requests that content (i.e., incoming data) is filtered, and phase 4, which handles the server response (i.e., outgoing data), are relevant. Additionally, you can use the option to define one or multiple actions that the software will perform when a match occurs [5]. To log incoming client requests in the auditlog in phase 2 and respond to the access attempt with a HTTP 403 (Forbidden) error message, you would do this: SecDefaultAction phase:2,log,U auditlog,deny,status:403

A massively negative default function like the default deny is highly restrictive, but it does offer maximum protection if you additionally define

filters and actions (see also the “Insider Attacks” box). The software offers a number of prebuilt alternatives here, including converting request parameters, running external scripts (e.g., to perform an antivirus scan), or forwarding malevolent requests. The latter is useful if you are trying to investigate attacks or want to forward them to a honeypot. The basic configuration (Listing 1) also logs the contents of incoming requests and the responses given in return. It enters the information in the auditlog as mentioned previously. The first directive, SecAuditEngine, enables this. The option for the second directive defines whether the software stores the auditlog entries in a single file (Serial) or writes a file for each transaction (Concurrent). Concurrency is necessary if you intend to deploy the ModSecurity Console addon product. Breach Security offers the software for managing multiple instances, provided you don’t need to monitor more than three servers. E

Table 1: modSecurity Processing Phases Number Phase 1 Preview phase

Designator

Client request

REQUEST_BODY

3 4

RESPONSE_HEADERS POST request Server response RESPONSE_BODY

Logging

REQUEST_HEADERS

LOGGING

Activities Earliest possible filtering of incoming requests before access control, authentica‑ tion, authorization, and MIME detection have taken place Apache‑side Full access to the content of a client request (normal case) Initial option for filtering server responses Full access to the content of the server re‑ sponse to an incoming client request Access to all relevant information before it is written to the Apache logfiles

Admin 01

N u ts a n d b o lts

ModSecurity

Figure 1: Once ModSecurity has been enabled, it will log suspicious activity at the detail level specified in SecAuditLogParts – in this case, an SQL injection attack.

The third instruction defines the auditlog storage location relative to the Apache installation path. Finally, the SecAuditLogParts instruction defines the information that ModSecurity logs in the auditlog (Table 2). In this case, this is the header and the content of the request, along with the ModSecurity reaction. The results are shown in Figure 1. After this preparation, you can add the most important directive to your configuration: SecRule. This defines a filter rule and optionally an action that the module will perform if it discovers a match for the rule. If you don’t define an action, the tool will run the standard command defined in the SecDefaultAction directive. Rules always follow this pattern: SecRule Variable Operator [Action]

The number of variables is huge, and they cover every single element of the client request (both for POST and for GET), as well as the most important server environment details [6]. Additionally, you can use regular expressions. For example, to investigate an HTTP request to find out whether the client requests the /etc/passwd string in a GET method, you would use this rule:

SecRule REQUEST_HEADERS: User‑Agent "nikto"

This example tells the web server to refuse requests from the Nikto security scanner [7]. If you want ModSecurity to run a specific action for a rule, you can overwrite the default action: SecRule REQUEST_HEADERS:User‑Agent "nikto" U "phase:2,pass,msg:'Nikto‑Scan logged'"

This rule tells the module to write a Nikto scan logged message to the logfile when it detects the Nikto user agent in a client request in phase 2. The rule then overwrites the drop default action, which is defined by SecDefaultAction with the pass action. This allows the client request to pass. To test ModSecurity, Listing 1 gives you an overview of the basic configuration discussed thus far.

Practice Session After restarting Apache, a request like http://www.example.com/index. html?file=/etc/passwd would trigger the sample rule in line 8. Then the action defined in line 9 would block the request. The client sees an HTTP 403 Forbidden error. At the same time, lines 3 through 7 tell ModSecurity to

SecRule REQUEST_URI "/etc/passwd"

If the request matches the rule, ModSecurity runs the default deny action. To filter by browser type, you would do this: Listing 1: Basic Configuration for ModSecurity 01 SecRuleEngine On 02 SecAuditEngine On 03 SecAuditLogType Serial 04 SecAuditLog logs/audit.log 05 SecAuditLogParts ABCFHZ

08 SecRule REQUEST_URI "/etc/passwd" 09 SecDefaultAction phase:2,log,auditlog,deny,status:403

Admin 01

The Art of Detecting Attacks To provide effective protection against a huge assortment of attacks, system administrators need to set up a robust ruleset for ModSecurity. To formulate rules that protect you against SQL injection, cross-site scripting, or local and remote file inclusion attacks, for example, you need in-depth knowledge of how attacks on web servers work. Of course, not every administrator has this knowledge or the time to re-invent the wheel. To address this, the Open Web Application Security Project (OWASP) offers a predefined ruleset for ModSecurity [8] that relies on anomaly detection to protect web servers against a number of

Table 2: SecAuditLogParts Arguments Abbreviation A B C D E F G H

06 SecDebugLog logs/debug.log 07 SecDebugLogLevel 5

log the transaction in the auditlog and send a detailed overview of the way the client request was processed to the debuglog. This means that your Apache error_log file will contain a note to the effect that a client request was effectively blocked, as shown in Listing 2. Once ModSecurity is working correctly, you can start adding rules and modifying them for the web applications you want to protect.

J K Z

Description Header for the entry (mandatory) Request header Request content; only available if content exists and ModSecurity is configured to store it Reserved Temporary response content; only available if ModSecurity is configured for this Final response header after possible manipulation by ModSecurity; Apache itself writes the Date and Server headers Reserved Auditlog trailer equivalent to C, except where the request contains form data; in this case, the software constructs a suitable request that excludes file content to simplify matches Reserved Line by line list of all matching rules in the order of their application End of entry (mandatory)

w w w. a d m i n - m aga z i n e .co m

ModSecurity

Insider Attacks A vulnerability in a web application can thus expose an Apache web server despite its being hardened and up to date. This is particularly dangerous if the server is deployed in a hosting environment where many customers share the resources of a physical web server. Protection mechanisms, such as virtualization or jails, will separate the individual instances; however, the security of the whole system depends on its weakest link. With a sniffer, for example, attackers can simply sniff password authentication conversations if they’re not encrypted. And, this gives attackers an inside vector. Scripting languages and frameworks like PHP or Ruby on Rails help web developers achieve results quickly, but they often conceal dangers that occur when security is not given sufficient attention. More complex environments, such as Java, Tomcat, and JBoss, are not necessarily the answer because they hide many aspects from the developer.

standard attacks, such as invalid client requests, SQL injection, cross-site scripting, and email or command injection. This gives you basic, fairly robust protection, which you can then modify to match your own application environment as needed. To install these core rules, download the package and unpack it in your Apache’s conf configuration directory. Then, move the rules, including the base_rules subdirectory, to the ModSecurity directory. You can look at the configuration in the modsecurity_crs_10_global_config.conf and modsecurity_crs_10_config.conf files and modify it to suit your needs. The rules are well documented. Also, you might want to enable audit and debug

logging to see exactly what the module is doing. To do so, you need to include the core rules in httpd.conf as follows: Include conf/modsecurity/*.conf Include conf/modsecurity/ base_rules/*.conf

After restarting, your Apache web server will have a solid suit of armor that responds with an HTTP 403 to any client requests classified as attacks. A word of caution: Web masters should first test the core rules extensively in a lab environment before letting them loose on a production system. Otherwise, the danger of preventing legitimate user access is possible. Also, remember that Mod-

N u ts a n d b o lts

Security will mean about a 5 percent performance overhead for your web server.

Preventing Attacks on the United Nations In some situations, you can’t fix a web application vulnerability immediately. Imagine a major online store discovering a security hole a week before Christmas and needing several days to fix the problem, meaning the shop would be offline for that time. The owner has to make a decision: Live with the risk and keep the shop, including the vulnerability, online so you can benefit from lucrative preChristmas shopping, or protect the company and its customers by taking the website down and fixing the vulnerability. ModSecurity offers a technical workaround in the form of virtual patching that allows you to define one or multiple rules that prevent the vulnerability from being exploited without actually removing it. The ModSecurity documentation refers to a case that dates back to 2007, when attackers were trying to hack the United Nations website [9]. The sub-page with talks by Secretary-General Ban Ki-moon [10] had a statID parameter that exposed an SQL injection vulnerability (Figure 2). If you discover a vulnerability of this kind, you can temporarily define a rule for ModSecurity that will prevent hackers from exploiting the vulnerability even though it still exists. Although this isn’t a good long-term solution, it does prevent a disaster until the web developers can remove the vulnerability from the source code on the page. The following solution would work for the UN bug: E Listing 2: Rule Match Sec AuditLogType Serial [Wed Nov 04 05:39:19 2009] [error] [client 192.168.209.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "/etc/passwd" at REQUEST_URI. [file "/usr/local/

Figure 2: The website of UN Secretary-General Ban Ki-moon was affected by an input parameter variable validation vulnerability. A ModSecurity virtual patching rule protected the website temporarily until the UN system administrator fixed the problem.

w w w. a d m i n - m aga z i n e .co m

httpd‑2.2.14/conf/httpd.conf"] [line "420"] [hostname "www.example.com"] [uri "/index.html"] [unique_id "SvFZ138AAQEAAAc4AgQAAAAA"]

Admin 01

N u ts a n d b o lts

ModSecurity

<Location /apps/news/infocus/ U sgspeeches/statments_full.asp> SecRule &ARGS "!@eq 1" SecRule ARGS_NAMES "!^statid$" SecRule ARGS:statID "!^\d{1,3}$" </Location>

Three lines embedded in an Apache location container state that valid user requests for the statements_ full.asp file are only allowed to have one argument (first rule) called statid (second rule) with numbers of one to three digits (third rule) as their parameters. Any requests that do not follow this pattern are cleaned up by the default action, as defined in SecDefaultAction. This would effectively prevent an attacker exploiting the SQL injection vulnerability.

No Inside Information ModSecurity also filters outgoing data, especially server responses to incoming requests. The PHP programming language throws error messages such as this: Fatal error: Connecting to MySQL server 'dbserv.example.com' failed

Although you can disable PHP error messages in responses, Google still lists a bunch of websites where PHP error messages reveal many juicy details of applications. This information is very useful to an attacker, because it can help them understand the internal workings and structure of a web application and attack it in a more targeted way. To tell ModSecurity to catch PHP error messages and prevent them from being sent to users, you can define a rule like this: SecRule RESPONSE_BODY "Fatal error:"

RESPONSE_BODY refers to the content

of the server response to the client, Listing 3: GeoIP Access 01 LoadModule geoip_module modules/mod_geoip.so 02 LoadModule security2_module modules/mod_security2.so 03 GeoIPEnable On 04 GeoIPDBFile /usr/tmp/GeoLiteCity.dat 05 SecRuleEngine On 06 SecGeoLookupDb /usr/tmp/GeoLiteCity.dat 07 Sec Rule REMOTE_ADDR "@geoLookup" "chain,drop,msg:'Connection attempt from .CN!'" 08 SecRule GEO:COUNTRY_CODE "@streq CN" "t:none"

Admin 01

and although it is not particularly elegant, it does indicate what potential you have. With a carefully crafted regular expression, you can use the same technique to prevent credit card numbers from being revealed by, for example, a compromised application in the aftermath of a successful SQL injection attack.

The Chinese Wall in Reverse Another advanced scenario for ModSecurity involves cooperating with the GeoIP provider, Maxmind. GeoIP locates users geographically on the basis of their IP address, which means you can restrict access to a website to a specific region, such as Pennsylvania – if you have a site in Pennsylvanian Dutch that nobody else would understand – or block a country entirely. To do this, you would install the mod_geoip2 module on Apache 2, along with the GeoIP software and GeoLiteCity.dat geographical database [11]. Imagine a mechanical engineering company in Germany’s Swabian region that is afraid of industrial espionage from the Far East; in this case, they could use the configuration in Listing 3 to prevent access from China – if the people in China didn’t spoof their origins. The last two lines form a filter rule chain. Line 6 locates the geographical region for the requesting IP address, then line 7 dumps the request and a message into the logfile if the request comes from China. This might not be politically correct, but it is technically effective.

Full Insurance Coverage ModSecurity has an enormous feature scope, and it can take some time to understand it completely. But if you go to the trouble to plum the depths of the module, it will pay dividends with comprehensive methods that give you additional protection against attacks on web applications. Thankfully, the prebuilt rulesets make it easier to get started. And, the vendor behind the project offers com-

mercial products and services such as training. Armed with ModSecurity, administrators can sit up tall in their saddles, even if attackers are trying to make their horses bolt. n

Info [1] OWASP Best Practices: Use of Web Appli‑ cation Firewalls: [http://www.owasp.org/ index.php/Category:OWASP_Best_Prac‑ tices:_Use_of_Web_Application_Firewalls] [2] ModSecurity: [http://modsecurity.org] [3] ModSecurity Reference Manual: [http://modsecurity.org/documentation/ modsecurity‑apache/2.5.10/ html‑multipage] [4] ModSecurity Processing Phases: [http:// www.modsecurity.org/documentation/ modsecurity‑apache/2.5.0/ html‑multipage/processing‑phases.html] [5] ModSecurity Actions: [http://www. modsecurity.org/documentation/ modsecurity‑apache/1.9.3/ html‑multipage/05‑actions.html] [6] ModSecurity Variables: [http://www. modsecurity.org/documentation/ modsecurity‑apache/2.1.0/ html‑multipage/05‑variables.html] [7] Nikto: [http://cirt.net/nikto2] [8] OWASP ModSecurity Core Rule Set Pro‑ ject: [http://www.owasp.org/index.php/ Category:OWASP_ModSecurity_Core_Rule_ Set_Project] [9] ModSecurity blog, “Virtual Patching Dur‑ ing Incident Response: United Nations Defacement”: [http://blog.modsecurity. org/2007/08/27/] [10] Talks by the UN General Secretary: [http://www.un.org/apps/news/infocus/ sgspeeches/] [11] “Apache ModSecurity with GeoIP blocking country specific traffic: ModSecurity + GeoIP” by Suvabrata Mukherjee: [http:// linuxhelp123.wordpress.com/2008/12/11/ apache] The Author Sebastian Wolfgarten works as an IT Security Expert with the European Central Bank as an advisor, manager, and auditor of internal proj‑ ects designed to improve the security of the IT infrastructure. Before this, he spent several years working for Ernst & Young AG in Germany, and as an Advisor for Information Security in Ireland. He has also worked as an IT security expert with T-Mobile Germany.

w w w. a d m i n - m aga z i n e .co m

REAL SOLUTIONS FOR REAL NETWORKS Each issue delivers technical solutions to the real-world problems you face every day. Learn the latest techniques for better: • network security • system management • troubleshooting • performance tuning • virtualization • cloud computing

FREE DVD Inside: Knoppix 6.3 + Backtrack

on Windows, Linux, Solaris, and popular varieties of Unix.

m/su o c . e n i z aga

ORDER

in-m m d a . w T ww A E N I L N O

N U TS A N D B O LTS

Daemon Monitoring

Monitoring daemons with shell tools

Watching the Daemons Administrators often write custom monitoring programs to make sure their daemons are providing the intended functionality. But simple shell tools are just as well suited to this task, and not just for systems that are low on resources. By Harald Zisler Unix daemons typically go about their work discreetly in the background. The process table, which is output with the ps command, only shows you that these willing helpers have been launched, although in the worst case they could just be hanging around as zombies. Whether or not a daemon is actually working is not something that the process table will tell you. In other words, you need more granular diagnostics. The underlying idea is to write a “sensor” script for each service that performs a tangible check of its individual functionality.

Because almost every program outputs standardized exit codes when it terminates, you can use Unix conventions. 0 stands for error-free processing, whereas 1 indicates some problems were encountered. This value is stored in the $? shell variable, which a shell script evaluates immediately after launching the sensor. Various programs are suitable for automated, “unmanned” access to the service provided by a given daemon; all of them will run in the shell without a GUI. These programs often provide an option (typically -q) that suppresses output, and this is fine

for accessing the exit code. Error logs are obtainable by redirecting the error output to a file or, if available, by setting the corresponding program option. The only thing left to do is to find the matching client program test the functionality of each service.

Web Servers To check a web server, you could use wget. The shell script command line for this would be: wget --spider -q ip-address

The --spider option tells wget to check that the page exists but not to load it. Defining the IP address instead of the hostname avoids a false positive if DNS-based name resolution fails for some reason.

Listing 1: Database Monitoring 01 #! /bin/sh

03 while true

04 do

06 zeit=$(date +%d.%m.%y\ %H:%M\ )

34 if [ $? -eq 0 ]; then echo "$time: Database online! +++++++++" >> dba.log

08 psql -U monitor -d monitor -c "select * from watch;"

24 25

10 if [ $? -eq 2 ];

echo "$time: Database: serious error! ***************" >> dba.log

then

echo "$time:

echo "$time: Database is not accessible!

while true

****************" >> dba.log

/usr/local/etc/rc.d/002pgsql.sh start

psql -U monitor -d monitor

sleep 15

psql -U monitor -d monitor

Unable to restart!

****************" >> dba.log

ADMIN 01

-c "select * from watch;"

time=$(date +%d.%m.%y\ %H:%M\ )

echo "$time: Database online!

break

+++++++++" >> dba.log

sleep 15

done

44 45

46 47 fi 48 sleep 15 49

32 33

else

-c "select * from watch;"

then

if [ $? -eq 0 ];

50 done

W W W. A D M I N - M AGA Z I N E .CO M

Daemon Monitoring

Figure 1: After starting, the script outputs the log at the console: availability, error, restart, database running.

Almost all known databases include a client program for the shell – for example, mysql for MySQL or psql for PostgreSQL. Alternatively, you can use ODBC to access the database in your scripted monitoring, such as the isql tool provided by the Unix ODBC project. For ease of access, you might need to set up a (non-privileged) user, a database, and a table for the test query on the database server. If you choose the ODBC option, you also need a .odbc.ini file with the right access credentials. The psql shell client for the Postgres database also poses the problem of non-standard exit codes. 1 stands for an error in the query, although the connect attempt has been successful; 2 indicates a connection error. A connection test with psql would look like this: psql ‑U User ‑d Database ‑c U "select * from test_table;"

For ODBC access, you would need to pipe the SQL query to the client: echo "select * from test_table;" | U isql ODBC_data_source user

For the cups printer daemon, lpq gives you a simple method of checking whether the daemon is alive. If you need to check access to individual printers, you additionally need to provide the print queue name and then grep the exit code. To make sure the exit code complies with this behavior, Grep checks the output that you receive if the printer is active: lpq ‑Pprinter | grep ‑q U "printer is ready"

To match the output from lpq, you need to modify the search string for grep.

w w w. a d m i n - m aga z i n e .co m

The ping command checks network connections. The exit error codes differ, depending on your operating system. The FreeBSD ping uses 2, the Linux

ping uses 1. The number of test packets is restricted by the ‑c packets option; this improves the script run time and avoids unnecessary network traffic. If you use the IP address as the target, you avoid the risk of false positives from buggy name resolution. ping ‑c1 ip_address

Sensor scripts can obviously be extended to cover many other system parameters, such as disk space usage (df), logged in users (who), and much, much more. If an error or threshold value infringement occurs, the script can use this information to generate a message and notify the system administrator. The message text should include the hostname, date, and time. Messages can be stored in a file to which the administrator has permanent access. To allow this to happen, you simply have to display the logfile in a terminal and use tail ‑f, but other forms of communication are also possible – texting, for example. If the shell script has the correct privileges, it can become involved and restart a daemon, remove block files, or even reboot the whole system. Because you should avoid running this kind of script as root, you can instead set up special users and groups to own the script and the process (which is the case with many daemons).

Database Restart The sample script in Listing 1 monitors an active database instance and notifies the administrator if the database happens to fail and then is successfully restarted (Figure 1). If it can’t start the daemon, it waits

N u ts a n d B o lts

for the administrator to step in and handle the situation.

Printer Restart The second sample script relates to the printing service. The one shown here is taken from a production example, in which the cupsd server has an unknown problem with a network printer. The printer was disabled time and time again, causing no end of frustration to users and unnecessary work for the system admins. The shell script shown in Listing 2 doesn’t output messages; instead, it simply restarts the service. Either run these scripts manually (for a temporary fix or quick check) or as RC scripts.

Conclusions Administrators don’t need a complex monitoring framework that covers every aspect of the environment and has a multi-week learning curve. With some scripting know-how, you can easily create your own shell scripts to monitor server daemon processes and restart them autonomously if so desired. The use of shell scripts to monitor daemons and other system functions is by no means restricted to small embedded systems. With scripts tailored to match your requirements, you can establish your own troubleshooting arsenal. n

The Author Harald Zisler has worked with Unix-flavored operating systems since the early 1990s.

Listing 2: CUPS Monitoring 01 #! /bin/sh 02 03 while true 04 do 05 06 lpq ‑Plp | grep ‑q "lp is ready" 07 08 if [ $? ‑gt 0 ] 09

then

cupsenable lp

11 fi 12 13 sleep 15 14 15 done

Admin 01

VPNs with SSTP

N u ts a n d B o lts

Ma n,

z Ka om F.c

3R 12

State-of-the-art virtual private networks

Private Affair Because Microsoft’s legacy VPN protocol, PPTP, has a couple of vulnerabilities, SSTP, which routes data via an SSL connection, was introduced as the new VPN protocol with Vista, Windows Server 2008, and Windows 7. By Thomas Drilling

Virtual private networks (VPNs) have established themselves as a standard solution for convenient remote access to enterprise networks. However, they can cause some issues in combination with standard tunneling protocols like PPTP if, for example, NAT routers are involved or you need to work around the local firewall. Typically, it is not in the administrator’s best interest to modify the firewall, NAT, or proxy configuration to suit requirements for remote

Admin 01

access. The Secure Socket Tunneling Protocol (SSTP), which was introduced with Microsoft Windows Server 2008, provides a solution by setting up a VPN tunnel that encapsulates PPP or L2TP traffic on a Secure Sockets Layer (SSL) channel (Figure 1). For administrators, this means that SSTP is a new VPN tunnel type in the Windows Server 2008 routing and RAS server role. It encapsulates PPP (point-to-point protocol) packets in HTTPS, thus supporting the VPN

connection through a firewall, a NAT device, or a proxy. Like all SSL VPNs, SSTP uses TCP port 443 (HTTPS) for data transfer. Compared with other commercial or proprietary solutions (e.g., IPsec, L2TP, or PPTP), the advantage is that port 443 is open in almost any router or server configuration, and SSTP packets can thus pass through without any additional configuration overhead. The “Handshake” box explains what Windows does with all of these protocols. Strong encryption in SSL 3.0 ensures maximum security and performance. SSTP VPNs are thus a class of SSL VPNs, like Cisco’s WebVPN or the Vigor Router by Draytek, that basically work in the same way as IPsec,

w w w. a d m i n - m aga z i n e .co m

VPns with SSTP

n u TS A n d B o LTS

Handshake Microsoft’s SSTP is basically another proprietary implementation of an SSL VPN. SSTP relies on standards such as SSL and TLS for encryption and authentication, but Microsoft has modified the tried-and-trusted SSL handshake by introducing proprietary extensions that bind the proprietary PPP protocol. If you look at the basic handshake, SSTP at first keeps to the standard SSL handshake procedure: 1. The client opens a connection to TCP port 443 on the server. 2. The client sends an “SSL Session Setup Message” to indicate that it wants to set up an SSL connection to the server. 3. The server sends an SSL certificate to the client. 4. The client validates the server certificate, identifies the correct encryption method for the SSL session, and creates a session key, which it encrypts using the public key from the server certificate.

L2TP, or PPTP, but use SSL to handle the data transfer. Because SSTP encapsulates complete IP packets, the connections act just like a PPTP or IPsec tunnel on the client side. According to the Microsoft definition, SSTP is a protocol mainly intended for dialup connections in the application layer that guarantees the confidentiality, authenticity, and integrity of the data to be transferred. A public key infrastructure (PKI) is used for authentication purposes. Microsoft introduced SSTP with Windows Server 2008 and Vista SP1. Today, Windows Server 2008 R2 and Windows 7 also support SSTP [1]. But what does SSTP offer the administrator, and how do you set up a VPN server with SSTP?

Sample Scenario In this example, I am using Windows Server 2008 R2 to provide an SSTPbased VPN server behind a NAT device. The server is configured as the domain controller and needs two network cards for the VPN setup. At

5. The client sends the encrypted SSL session key to the server. 6. The server decrypts the client’s SSL session key using its own private key and encrypts the communications with the session key. Up to this point, the procedure is no different from standard SSL communication. However, Microsoft then implements additional handshake steps that build on what has happened thus far. 7. The client sends an HTTP-over-SSL request message to the server and negotiates an SSTP tunnel with the server. 8. After this, the client negotiates a PPP connection with the SSTP server, which includes authenticating the user’s login credentials with a PPP authentication method and configuring the settings for the data traffic. 9. The client starts to transfer data via the PPP connection.

least one card (preferably both) needs a static IP address. The client will be MS Windows 7. The server and clients are both members of an Active Directory (AD) domain. Additionally, the server and clients should have all the current updates in place, such as the current Service Pack 2 for Windows Server 2008 R2. After installing a Windows server, you need to install the Active Directory Domain Services to make the server a domain controller. The easiest way to do this is to run dcpromo at the command line and then follow the wizard. The server also needs to provide DNS, DHCP, and certificate services, which you can achieve by configuring the matching server roles in the role wizard. VPN functionality is also provided via a server role Network policy and access services.

PKI Because SSTP uses HTTPS (port 443) to handle all the data traffic, there is

no alternative to configuring a public key infrastructure. At a minimum, this means installing at least one certificate on the SSTP server and a root certificate authority certificate on all SSTP VPN clients. You might have to modify the packet filter rules, too, even though SSTP doesn’t actually need any additional NAT configuration because port 443 is typically open. The “Port Customizer” box explains how you can use a port other than 443 for SSTP. Next, you’ll need to configure an Active Directory-integrated root certification authority on the domain controller. In combination with a group policy, this causes clients that are domain members to request certificates automatically when they open a connection. Certificates are then issued Port customizer SSTP normally uses TCP port 443, which is open in most router and NAT configurations. Security-conscious Windows administrators might prefer to modify the standard port used by SSTP. To do so, you need to edit the following registry key HKEY_LOCAL_MACHINE\SYSTEM\ U CurrentControlSet\Services\SstpSvc\ U

SSTP VPN connection (TCP port 443)

Parameters

1. Open TCP connection 2. Set up SSL connection and validate certificate 3. HTTPS request 4. Initiate SSL tunnel 5. Data communication via PPP SSTP gateway server

SSTP VPN client

Figure 1: The SSTP handshake is not much different from a standard SSL handshake. In contrast to IPsec, SSTP sends PPP packets (not IP packets) through the tunnel.

w w w. A d m i n - m AgA z i n e .co m

in the Registry Editor. Look for the ListenerPort parameter. Changing the view to Decimal (right-click) lets you specify a different port. Then you need to restart the Routing and RAS service. If you change the ListenerPort, you need to reconfigure your NAT device to match and forward all incoming traffic addressed to port 443 to the newly configured port on the SSTP-based VPN server.

Admin 01

N u ts a n d B o lts

VPNs with SSTP

Setting Up a Certification Authority To set up your own (private) certification authority in Active Directory, you need to launch the MMC console on your Windows server and add the certificate snap-in. Then, in the Certificate Snap‑In dialog box, select Computer account and Local computer for Select computer. n To begin, right-click on Own certificate in the MMC, followed by All tasks | Request new certificate. n You will see a selection of templates: Web Server will do the trick here. Now you need to provide a name, such as fw.example.local. For this certificate, the name of the certificate requestor must match the hostname of the device with which the VPN client will open the connection. This step is essential for a successful SSL handshake. n If the request worked as intended, the certificate will install automatically and should be visible in the MMC console: Certificates snap-in below Certificates (local computer) | Own certificates. The CA certificate must also be visible in Certificates (local computer) | Trusted root certificate authorities; you can check this if you experience difficulty with the certificate request.

by the domain controller and placed in the client’s local certificate store along with the certification authority certificate. If you are unable or prefer not to purchase a commercial certificate, you can use a private certificate issued by Active Directory’s built-in CA. The “Setting up a Certification Authority” box explains how to do this. You need to configure the firewall to allow network traffic to pass through to the certificate authority in order for the certificate request to work. To get

this to happen, the firewall must also be authorized to request certificates (Figure 2). You can set this up under the Web Server Properties Security tab; Read and Enroll privileges are required at minimum. In production, it often makes more sense to purchase a commercial certificate and install a dedicated firewall server with Microsoft’s new Forefront Threat Management Gateway (TMG) 2010 [2]. The gateway offers a wizard-based configuration in the TMG manage-

ment console. Note that you cannot install the Forefront 2010 server component on a domain controller. Additionally, you will want to publish a certificate revocation list in a production scenario.

Revocation A Certificate Revocation List (CRL) details any certificates revoked before their expiry date. Revocation lists are always available at CLR Distribution Points (CDPs). A CDP can be set in the Extensions tab of the CA Properties dialog box (Figure 3). Additionally, the certificate revocation list must be accessible to all clients at all times via the Internet, which will mean configuring the packet filter on the local firewall. In Forefront TMG 2010, you can use the website publishing wizard for this.

VPN Server Setting up the VPN server itself is easily done. Once you have added the network policy and access services role by clicking Add roles in

Figure 2: If you are unable to request a certificate, you can check the privileges

Figure 3: A CLR distribution point controls the availability of the certificate

in the certificate template for web servers to see whether Enroll is allowed.

revocation list, which all clients need to be able to access at any time.

Admin 01

w w w. a d m i n - m aga z i n e .co m

VPNs with SSTP

N u ts a n d B o lts

Figure 4: You need to set up the RAS services to run a VPN server.

the Customize this server section of the Server Manager, the Routing and Remote Access tool is available below Management. The Action | Configure and enable routing and RAS button takes you to the Routing and Remote Access Server Setup Wizard. At the second Configuration step (Figure 4), you’ll want to enable Virtual private network (VPN) access and NAT, then click Next and select the required network interface. At the following step, Address assignment defines how the VPN server assigns IP addresses to remote clients. If you have the DHCP service running, Automatic is the quickest and cleanest option. Then you can define an IP address pool for the DHCP server to use in the next step, Address range assignment. In the final step, you can choose whether or not to use a Radius server to authenticate clients on a large-scale network; this is disabled by default. The wizard will then instruct you to set up the DHCP relay agent for Windows to support the forwarding of

IPv4 or IPv6

TCP

SSTP

DHCP messages to RAS clients. To do this, you need to enable the Relay DHCP Figure 5: Configuring Windows to forward DHCP messages to the RAS clients. packets option in the DHCP Relay Properties dialog box ets. In IPsec, ESP builds directly on (Figure 5). IP. Microsoft is quite obviously seeking to set itself apart by encapsulating in PPP. Conclusions Apart from the fairly complex Windows server configuration, which Microsoft’s SSTP encapsulation strucmainly involves setting up the certure is like a Russian doll. Just as tificate services, and possibly packet with PPTP, Microsoft uses the PPP filters for transporting the certificate protocol with SSTP, which leads to a fairly complex encapsulation structure requests, SSTP offers a secure, wellperforming tunnel technology for the (see Figure 6), in which an IP header future. contains a TCP header, which in turn n contains an SSTP header, which then contains a PPP header, which finally contains the IP packets themselves. Info Although the method seems to be [1] Microsoft support for SSTP: [http:// slightly more efficient than Encapsusupport.microsoft.com/kb/947032/] lating Security Payload (ESP), with [2] TMG 2010: [http://www.microsoft.com/ an overhead of 8 bytes in the PPP downloads/details.aspx?familyid=e05a header, compared with 20 bytes in ecbc‑d0eb‑4e0f‑a5db‑8f236995bccd& IPsec over HTTPS, there is actually no displaylang=en] real need to encapsulate in PPP pack-

PPP

IPv4 or IPv6 packet

Encapsulated SSL session Figure 6: The SSTP encapsulation structure is like a Russian doll. Microsoft has gone to considerable trouble to make something proprietary from what are basically open protocols. From a technical point of view, there seems to be no real reason to use PPP.

w w w. a d m i n - m aga z i n e .co m

Admin 01

S E RV I C E

Contact Info / Authors

WRITE FOR US Admin: Network and Security is looking for good, practical articles on system administration topics. We love to hear from IT professionals who have discovered innovative tools or techniques for solving real-world problems. Tell us about your favorite: • interoperability solutions • practical tools for cloud environments • security problems and how you solved them • ingenious custom scripts

• unheralded open source utilities • Windows networking techniques that aren’t explained (or aren’t explained well) in the standard documentation. We need concrete, fully developed solutions: installation steps, configuration files, examples – we are looking for a complete discussion, not just a “hot tip” that leaves the details to the reader. If you have an idea for an article, send a 1-2 paragraph proposal describing your topic to: edit@admin-magazine.com.

AUTHORS Falko Benthin

ADMIN 01

Björn Bürstinghaus

25, 60

Thomas Drilling

52, 94

Florian Effenberger

Dan Frost

Thomas Joos

Daniel Kottmair

Caspar Clemens Mierau

James Mohr

Thorsten Scherf

8, 44, 78

Tim Schürmann

Udo Seidel

Kurt Seifried

Sebastian Wolfgarten

Harald Zisler

Editor in Chief Joe Casad, jcasad@admin-magazine.com Managing Editor Rita L Sooby, rsooby@admin-magazine.com Contributing Editors Oliver Frommel, Uli Bantle, Andreas Bohle, Jens-Christoph Brendel, Hans-Georg Eßer, Markus Feilner, Marcel Hilzinger, Mathias Huber, Anika Kehrer, Kristian Kißling, Jan Kleinert, Daniel Kottmair, Thomas Leichtenstern, Jörg Luther, Nils Magnus Localization & Translation Ian Travis Proofreading & Polishing Amber Ankerholz Layout Klaus Rehfeld, Judith Erb Cover Illustration based on graphics by James Thew, 123RF Advertising www.admin-magazine.com/Advertise United Kingdom and Ireland Penny Wilby, pwilby@admin-magazine.com Phone: +44 1787 211 100 North America Amy Phalen, aphalen@admin-magazine.com Phone: +1 785 856 3434 All other countries Hubert Wiest, anzeigen@admin-magazine.com Phone: +49 89 9934 1123 Corporate Management (Vorstand) Hermann Plank, hplank@linuxnewmedia.com Brian Osborn, bosborn@linuxnewmedia.com Management North America Brian Osborn, bosborn@linuxnewmedia.com Associate Publisher Rikki Kite, rkite@linuxnewmedia.com Product Management Hans-Jörg Ehren, hjehren@linuxnewmedia.com Customer Service / Subscription For USA and Canada: Email: subs@admin-magazine.com Phone: 1-866-247-2802 (Toll Free from the US and Canada) Fax: 1-785-274-4305 For all other countries: Email: subs@admin-magazine.com Phone: +49 89 9934 1167 Fax: +49 89 9934 1199 Admin Magazine • c/o Linux New Media • Putzbrunner Str 71 • 81739 Munich • Germany www.admin-magazine.com While every care has been taken in the content of the magazine, the publishers cannot be held responsible for the accuracy of the information contained within it or any consequences arising from the use of it. The use of the DVD provided with the magazine or any material provided on it is at your own risk. Copyright and Trademarks © 2010 Linux New Media Ltd. No material may be reproduced in any form whatsoever in whole or in part without the written permission of the publishers. It is assumed that all correspondence sent, for example, letters, emails, faxes, photographs, articles, drawings, are supplied for publication or license to third parties on a non-exclusive worldwide basis by Linux New Media unless otherwise stated in writing. Printed in Germany Distributed by COMAG Specialist, Tavistock Road, West Drayton, Middlesex, UB7 7QE, United Kingdom Admin Magazine

ISSN 2045-0702

Admin Magazine is published by Linux New Media USA, LLC, 719 Massachusetts, Lawrence, KS 66044, USA, and Linux New Media Ltd, Manchester, England. Company registered in England. Linux is a trademark of Linus Torvalds.

W W W. A D M I N - M AGA Z I N E .CO M

DEDICATED £ 59 SERVERS FROM

PER MONT H

BRAND NEW DELL RANGE WITH WINDOWS OR LINUX

FREE CONTROL PANEL FREE SAME-DAY SETUP 100 MBPS UNMETERED CONNECTION NO MINIMUM CONTRACT DX-250

£59

per month

D3-240

£89

per month

D3-340

£129 per month

D3-520

£189 per month

D3-620

£299 per month

PROCESSOR

Intel® Xeon

CORES

Quad 2.4Ghz

Quad 2.66Ghz

Quad 2.26Ghz

2 x Quad 2.26Ghz

MEMORY

1GB DDR2 ECC

4GB DDR3 ECC

8GB DDR3 ECC

16GB DDR3 ECC

HARD DISKS

1 x 160GB SATA

2 x 250GB SATA

2 x 500GB SATA

2 x 1000GB SATA

4 x 1000GB SATA

RAID

None

Hardware RAID

100 Mbps

CONNECTION 100 Mbps

OR BUILD YOUR OWN FULLY CUSTOMISED DELL DEDICATED SERVER AT: linux.redstation.com

DELL POWEREDGE SERVERS

100 MBPS UNMETERED

SECURE PRIVATE NETWORK

WINDOWS OR LINUX

NO MINIMUM CONTRACT

24/7 TELEPHONE SUPPORT

FREE SAME-DAY SETUP

REMOTE SERVER CONTROL

PRIVATE UK DATA CENTRE

CALL FREE ON:

0800 622 6655 OR VISIT: linux.redstation.com