Semi-Reliable Periodic Commands

Impromptu

cron is great. Using cron, you can schedule commands to be run at regular intervals or at specific times. One of the major drawbacks of cron is that it doesn't generally keep state regarding job's status. If a job is scheduled to run at midnight, but the system is powered down at midnight, the command will never be run.

There are a few solutions out there to take care of this shortcomings, but many are designed for system-wide use only. What if you don't want to intermingle your personal jobs with the system-wide ones? Here is a fairly simple script that works as a wrapper, providing a little insurance to help make sure jobs get run.

#!/bin/bash

export P_ENV=~/.profile
export P_TRACK=~/.p_track

if [ -e "$P_ENV" ]; then
 . $P_ENV
fi

case "$1" in
 h) 
  export P_NOW=$(date +%Y-%m-%d-%H)
  ;;
 d)
  export P_NOW=$(date +%Y-%m-%d)
  ;;
 w)
  export P_NOW=$(date +%Y-%U)
  ;;
 m)
  export P_NOW=$(date +%Y-%m)
  ;;
 *)
  echo ERROR: Term not specified. Must be one of h, d, w, m .
  exit 1
esac

if [ -z "$2" ]; then
 echo ERROR: Command to run not specified.
 exit 2
fi

export P_TAG=$(echo $2 | sed -e 's/[^A-Za-z0-9]/-/g')
export P_FILE=$P_TRACK/$P_TAG-$P_NOW

if [ -e "$P_FILE" ]; then
 exit 0
else
 rm -f $P_TRACK/$P_TAG*
 echo Executing $2 at $(date)
 $2 $3 $4 $5 $6 $7 $8 $9
 echo Done
 
 if [ "$?" -eq "0" ]; then
  touch $P_FILE
 fi
fi

The script has 4 operating modes. It can help ensure command are run hourly, daily, weekly, or monthly. It uses empty files in a configurable directory (~/.p_track by default) to keep tabs on the last time the command was run. Entries in the tracking directory are only created if the command run returns exit status 0 (no error.)

To use this wrapper, place it and a period (h for hourly, d for daily, and so on) in front of commands in your crontab like so:

*/20 * * * * ~/bin/periodic h ~/bin/command param1 param2

The above will check to see if the command needs to be run every 20 minutes, but will only execute it every hour. The advantage of wrapping individual commands instead using periodic command directories (check out the run-parts command) is less administrative overhead.

Note: The "real" periodic command is generally used to run system-wide periodic commands, often stored in /etc/periodic. UNIX-y systems tend to use other mechanisms and directories to do something similar.

Update 1: Changed the script from a group of if/elif to case (thanks for pointing that out Dave) and added a quick import of .profile or another file to set up environment.

Silhouette Clone: Part 2

Impromptu

Read Part 1

Now that we have already set up a Subversion repository as well as automatic file additions and commits, we need to account for deletions and provide access to versioned data.

When a file is deleted from a Subversion working copy without the proper procedures (which is pretty much what we are going to do) Subversion is shocked not to find the file when checking repository status.

$ svn status /path/to/working/copy
! deleted-file

All we need to do is let Subversion know we want to delete the file using the svn delete command. As in the previous installment, a little finagling from a shell script gets the job done.

#!/bin/bash
svn status /path/to/working/copy | grep ^\? | cut -c 8- | xargs svn add
svn status /path/to/working/copy | grep ^\! | cut -c 8- | xargs svn delete
svn commit -m "Automatic snapshot" /path/to/working/copy

You may note that now we have two calls to the svn status command. Below the above is rewritten to cache the output of this command in order to improve performance, but I wanted to show this format just once to compare to the previous version of the script.

#!/bin/bash
svn status /path/to/working/copy > /tmp/svn-status.txt
grep ^\? < /tmp/svn-status.txt | cut -c 8- | xargs svn add
grep ^\! < /tmp/svn-status.txt | cut -c 8- | xargs svn delete
svn commit -m "Automatic snapshot" /path/to/working/copy

Unfortunately, without handling file moves and renames through Subversion, they cannot efficiently. A moved or renamed file looks like a missing and new file pair to Subversion.

$ svn status /path/to/working/copy
! old-file-path
? new-file-path

Even though we can't handle this as efficiently as Subversion can, we can still handle it. No additional work is required to handle file renames or moves.

Next time, we will provide access to the versioned file system over the network without the need for clients to use the svn command line tool.

Simple Command Line Text Processing

Impromptu

I almost didn't write this, because I decided to check the latest goings on at GnuJersey.org, and happened to see Dave posted something about sed. I didn't want to be a copy cat. Then I remembered that nothing is original anymore. Plus, you will find no mention of sed (other than this and the previous one) or anything else in Dave's post here. so here we go.

The other day I needed to sort a few dozen lines of text. For some reason, the behemoth text editor I was using didn't have a sort function. (What gives?) A commercial competitor I recently switch from had this functionality, but I didn't want to reinstall it just for one quick sort. Instead, I turn to the old standby of the UNIX user: piping for text processing!

In order to demonstrate some key functionality, I am going to operate on the contents of a simple text file. The file contains the top 10 most common passwords and respective frequencies, according to this article. For reference, here are the contents of the file:

letmein 1.76%
thomas 0.99%
arsenal 1.11%
monkey 1.33%
charlie 1.39%
qwerty 1.41%
qwerty 1.41%
123456 1.63%
letmein 1.76%
liverpool 1.82%
password 3.780%
arsenal 1.11%
123 3.784%

In order to understand what is going on, you need to be familiar with 3 input redirectors used in the UNIX world. They are as follows.

• < reads data from a file into the command on the left


• > writes data from a command into a file on the right


• | forwards data from one command to the next

Input redirection is actually far more complicated and capable than that, but that is another post.

The first key command is sort. sort does what you may expert, it sorts lines in a text file.

$ sort < demo.txt
123 3.784%
123456 1.63%
arsenal 1.11%
arsenal 1.11%
charlie 1.39%
letmein 1.76%
letmein 1.76%
liverpool 1.82%
monkey 1.33%
password 3.780%
qwerty 1.41%
qwerty 1.41%
thomas 0.99%

In order to clean things up, sort is often combined with uniq. uniq preserves unique lines in a text file, but it has a serious limitation: it only works when duplicate lines are adjacent. Therefore, if you want to use uniq, use sort first.

$ sort < demo.txt | uniq
123 3.784%
123456 1.63%
arsenal 1.11%
charlie 1.39%
letmein 1.76%
liverpool 1.82%
monkey 1.33%
password 3.780%
qwerty 1.41%
thomas 0.99%

Another handy one is cut. cut splits lines into fields (think spreadsheet) and allows you to cut out the fields you want to keep. cut can work with any simple field delimiter given by the -d option. A comma separated list of fields to display is given by the -f option.

$ sort < demo.txt | uniq | cut -d ' ' -f 1
123
123456
arsenal
charlie
letmein
liverpool
monkey
password
qwerty
thomas

If you want to get back to your command line roots, or just like the speed and simplicity of throwing a few small, highly specialized commands at a your problems, keep exploring. There are several more fun text processing commands available on most UNIX-like operating systems. More complicated solutions like awk and sed open up even more quick solutions to common text processing problems.

Restricting Interactive Access

How can I give someone one SSH access to my server without compromising security? - Plenty of People

[How can I] allow shell access but NOT allow [arbitrary] remote commands to be run? - TyphoonJoe @ Ubuntu Forums (Impromptu)

First, every point of access to your system is a potential weak spot. Learn to accept it.

The long and short of this is if you want to allow someone interactive access to a machine, but want to control what programs they can run, you need to do one of two things. Either force a customized shell upon the user that will implement some sort of access control, or use regular, old-fashioned file system permissions to lock out certain programs.

Wrapping a shell is best left to experts. There are some interrupt issues to handle as well as some special case scenarios to worry about. Plus, its usually overkill!

File system permissions are quick, easy, and effective. There are just six simple steps to take:

1. Identify a restricted program or group of programs (such as ifconfig or gcc and ld.)


2. Create a group for the program or program group.


3. Add users permitted to use the restricted programs to the corresponding group.


4. Change the group owner on the program(s) to the specially create group.


5. Change file system permissions on the program so that only the owner or group can read or execute the program.


6. Pray you didn't botch steps 1 through 5.

Keep in mind that man other programs, scripts, and services may need to use what you have restricted. When locking out commands with file system permissions make sure you are as thorough as possible.