Semi-Reliable Periodic Commands

Impromptu

cron is great. Using cron, you can schedule commands to be run at regular intervals or at specific times. One of the major drawbacks of cron is that it doesn't generally keep state regarding job's status. If a job is scheduled to run at midnight, but the system is powered down at midnight, the command will never be run.

There are a few solutions out there to take care of this shortcomings, but many are designed for system-wide use only. What if you don't want to intermingle your personal jobs with the system-wide ones? Here is a fairly simple script that works as a wrapper, providing a little insurance to help make sure jobs get run.

#!/bin/bash

export P_ENV=~/.profile
export P_TRACK=~/.p_track

if [ -e "$P_ENV" ]; then
 . $P_ENV
fi

case "$1" in
 h) 
  export P_NOW=$(date +%Y-%m-%d-%H)
  ;;
 d)
  export P_NOW=$(date +%Y-%m-%d)
  ;;
 w)
  export P_NOW=$(date +%Y-%U)
  ;;
 m)
  export P_NOW=$(date +%Y-%m)
  ;;
 *)
  echo ERROR: Term not specified. Must be one of h, d, w, m .
  exit 1
esac

if [ -z "$2" ]; then
 echo ERROR: Command to run not specified.
 exit 2
fi

export P_TAG=$(echo $2 | sed -e 's/[^A-Za-z0-9]/-/g')
export P_FILE=$P_TRACK/$P_TAG-$P_NOW

if [ -e "$P_FILE" ]; then
 exit 0
else
 rm -f $P_TRACK/$P_TAG*
 echo Executing $2 at $(date)
 $2 $3 $4 $5 $6 $7 $8 $9
 echo Done
 
 if [ "$?" -eq "0" ]; then
  touch $P_FILE
 fi
fi

The script has 4 operating modes. It can help ensure command are run hourly, daily, weekly, or monthly. It uses empty files in a configurable directory (~/.p_track by default) to keep tabs on the last time the command was run. Entries in the tracking directory are only created if the command run returns exit status 0 (no error.)

To use this wrapper, place it and a period (h for hourly, d for daily, and so on) in front of commands in your crontab like so:

*/20 * * * * ~/bin/periodic h ~/bin/command param1 param2

The above will check to see if the command needs to be run every 20 minutes, but will only execute it every hour. The advantage of wrapping individual commands instead using periodic command directories (check out the run-parts command) is less administrative overhead.

Note: The "real" periodic command is generally used to run system-wide periodic commands, often stored in /etc/periodic. UNIX-y systems tend to use other mechanisms and directories to do something similar.

Update 1: Changed the script from a group of if/elif to case (thanks for pointing that out Dave) and added a quick import of .profile or another file to set up environment.

Flash Objects and the Z-Index

... On my Mac using Safari the drop down menus appear. When I move my cursor down on the drop down the rest of that drop down disappears In IE the drop down menus go only as far as the the top of the Flash file. I am not sure if it does the same in Mozilla. Anyway to circument this? Perhaps I could just do a pop-up page? - Michael

Yes! This is easy. First, you'll need to add a param tag to your object tag. Set the name to wmode and the value to transparent.

<param name="wmode" value="transparent" />

Then, force your Flash object to be below everything else by changing its z-index to 0. For the unfamiliar, think back to Geometry class in high school. You had X, Y, and Z axis. In the user interface world, the Z axis values specify how objects are stacked, and typically don't deal with perspective as you would see in a 3D game.

For some reason, applying z-index directly to a Flash object doesn't work properly. Wrap your Flash object in a div, and apply the z-index to it.

<div style="z-index: 0">...</div>

That's all there is too it!

Update 1: Unfortunately, the above code only works for Internet Explorer! The object tag is for IE only. Most embeded Flash content is wrapped with two tags in order to support both IE and other browsers. These tags are the object and embed tags. In order for this fix to apply to all browsers, modify the embed tag as well, adding the wmode property with a value of transparent.

Bad Programming

Rant

The self-checkout kiosk at the local supermarket blocked, waiting for employee input instead of giving me errors. Why? Bad programming.

Planet keeps brining up my old posts if I modify them (typically to correct errors). Why? Yup. Bad programming.

Bad programming doesn't necessarily mean bugs and buffer overflows. It also means bad predictions about program flow. If a narrow sequence of events is all code can accommodate without some easy and automatic method to attempt recovery and/or to bring things back on track, its badly programmed.

Programmers aren't mind readers, but they need to consider the "what ifs". When a program behaves in a manner that is just plain stupid, its probably badly designed!

Silhouette Clone: Part 3

Impromptu

Previously, we created a Subversion repository and automatically populated it with data from a directory which was shared over the network. This allowed users to work on the share while automated processes on the server stored versioned copies of the directory for later access. Now we will provide point-in-time recovery options for files in our repository without making users install and use a Subversion client.

Subversion repositories can be accessed over the network via WebDAV by means of special Apache HTTPD modules. The problem with this approach is that old versions aren't accessible using this method. Even after employing simple workarounds to present users with a prior versions, users would still have two places to look for data: the actual network share and a the URL of the WebDAV interface to the repository.

We can get around this by using WebDAV to serve up the Subversion repository and then re-sharing the WebDAV resource using the same protocols we shared the original network share with. Confused? Follow along.

The first thing we need to do is implement point-in-time tags in our Subversion repository. Subversion can copy files from one location in the repository to another without duplicating the file data. This means we can implement tags by simply copying files! For minimal impact, we will perform this copy entirely within the repository.

Commonly, Subversion repositories have a trunk/ or head/ directory where most of the work happens. We will also add a point-in-time/ directory to store our point-in-time copies. An easy way to create these directories is to use the svn mkdir command on our repository.

$ svn mkdir /path/to/repository/head -m "Created head/"
$ svn mkdir /path/to/repository/point-in-time -m "Created /point-in-time/"

Note the use of the -m option to specify a commit message. We are performing operations directly on the repository and are therefore creating new revisions of it. Subversion demands you leave a message (even an empty one) when creating a new revision.

Now that we are using a head/ directory, all of our day-to-day work should be performed there. Users do not need to be aware of the head/ directory at this point so we will simply pretend that head/ is the root of our repository.

$ svn checkout /path/to/repository/head /path/to/working/copy

Any changes to the working copy will be committed to the head/ directory in our repository transparently.

Now that we have a repository and working copy set up to utilize a head/ directory, we can continue as we did in parts 1 and 2 to enable automatic commits. From this point, implementing point-in-time copies is quick and easy. A simple shell script to copy head/ to a subdirectory of point-in-time/ at regular intervals will get the job done.

#!/bin/bash
svn copy file:///path/to/repository/head "file:///path/to/repository/snapshots/`date +%F\ %T`" -m "Point-in-time marker added"

Note that while we can put the above command in the same file as our previous commands to commit changes to the Subversion repository, we don't have to. Separate scripts will let us use separate schedules for our actual backups and point-in-time tags. This allows us to back up data frequently without presenting a user attempting to recover a file him or herself with an overwhelming number of file iterations to wade through.

Once our backup commands and point-in-time tags commands are running at regular intervals, we will have a repository layout that is fairly self explanatory.

$ svn list /path/to/repository
head/
point-in-time/

$svn list path/to/repository/point-in-time
2007-05-08 17:00:00/
2007-05-08 17:15:00/
2007-05-08 17:30:00/

Note that by modifying the date format string in our point-in-time tag command we can change how subdirectories of our point-in-time/ directory are named. 24-hour time is used instead of 12-hour time to make sure directories are always sorted in chronological order.

Now that we have our data efficiently stored and laid out, we have to provide access to it. Most file sharing packages do not understand Subversion repositories, so we will have to build a bridge. We can start by using Apache HTTPD to provide WebDAV access to the repository. After installing and enabling the mod_dav_svn module, a few lines in an Apache configuration file will do.

<Location /repository>
DAV svn
SVNPath /path/to/repository
</Location>

Ideally, you will want to lock-down this location using a combination of users, passwords, and IP addresses. See the Apache documentation for more information.

Once Apache is reloaded and serving the repository, install davfs2. When installed properly, you can mount the Subversion repository via WebDAV as you would any other file system. A simple setup is to create a directory for the network share, and place your data in a subdirectory. This will let you mount the Subversion repository along side your data.

$ mkdir /path/to/network/share
$ mkdir /path/to/network/share/data
$ mkdir /path/to/network/share/backups

$ mount -t davfs -o ro,noaskauth http://localhost/repository/point-in-time/ /path/to/network/share/backups

Note that you do not want to allow the mounted backup directory to be modified.

Mirroring our Subversion repository layout in our local file sytem seems silly, but we used this setup for a reason. Providing users direct access to the live Subversion repository for use as the live network share would generate up to 3 new revisions in our repository each time a file is saved! This is due to the rename-write-delete method commonly used to avoid data loss when saving files. Multiple revisions aren't too bad, but the way in which these revisions come about prevents Subversion from storing data efficiently.

Using Subversion as a (mostly) transparent replacement for Microsoft Window Server's Shadow Copy for Shared Folders can be quite cumbersome, but its a viable alternative. Current developments in Linux file systems will likely render the need for this workaround obsolete in the near future. Until then, take it one step at a time and enjoy automatic, versioned backups of your network shares.

Silhouette Clone: Part 2

Impromptu

Read Part 1

Now that we have already set up a Subversion repository as well as automatic file additions and commits, we need to account for deletions and provide access to versioned data.

When a file is deleted from a Subversion working copy without the proper procedures (which is pretty much what we are going to do) Subversion is shocked not to find the file when checking repository status.

$ svn status /path/to/working/copy
! deleted-file

All we need to do is let Subversion know we want to delete the file using the svn delete command. As in the previous installment, a little finagling from a shell script gets the job done.

#!/bin/bash
svn status /path/to/working/copy | grep ^\? | cut -c 8- | xargs svn add
svn status /path/to/working/copy | grep ^\! | cut -c 8- | xargs svn delete
svn commit -m "Automatic snapshot" /path/to/working/copy

You may note that now we have two calls to the svn status command. Below the above is rewritten to cache the output of this command in order to improve performance, but I wanted to show this format just once to compare to the previous version of the script.

#!/bin/bash
svn status /path/to/working/copy > /tmp/svn-status.txt
grep ^\? < /tmp/svn-status.txt | cut -c 8- | xargs svn add
grep ^\! < /tmp/svn-status.txt | cut -c 8- | xargs svn delete
svn commit -m "Automatic snapshot" /path/to/working/copy

Unfortunately, without handling file moves and renames through Subversion, they cannot efficiently. A moved or renamed file looks like a missing and new file pair to Subversion.

$ svn status /path/to/working/copy
! old-file-path
? new-file-path

Even though we can't handle this as efficiently as Subversion can, we can still handle it. No additional work is required to handle file renames or moves.

Next time, we will provide access to the versioned file system over the network without the need for clients to use the svn command line tool.