Ask Cold Drink

ACD Go Down the Hole

2007-10-15T17:00:00.000-05:00

As you probably have not noticed but will now recognize since I'm pointing it out, I haven't really posted here since June or so. Why? I have been far to preoccupied with other things to post anything, and far too bitter to be bothered when I'm not. Considering that, as well as the fact that people don't ask me things I'd share for free anymore, and GnuJersey.org's imminent shutdown, I see no reason to pretend like I'm ever going to post again.

I have other interesting things in mind for the future, but we'll see what happens.

*Poof*, suckas!

I Realy Don't Care

2007-08-12T14:56:00.000-05:00

Rant

I really don’t care.

I don’t care that your computer is infested with spyware, causing pictures to open up by themselves. I don’t care how much money you spent just to have a giant, 2.8GHz paperweight. I don’t care that your network is home to names, home addresses, and social security numbers of 10,000 individuals, and protected by security which my cat could break through. One more time: I don’t care.

At the end of the day, there is one thing I do care about. You annoying wanna-be power users. I hate you. I despise you. I hope you get crippling arthritis in your fingers and can never type again.

Why on earth do you think that being able to change passwords and set a few profile restrictions on your domain controller makes you qualified to be your company’s domain administrator? Half the professional techs I know aren’t qualified to be administrators for a Windows domain, and you think a few help documents you found on the Internet are better than their years of experience and training?

“No, I didn’t graduate from medical school, but I found some tutorials on the web that will get me through your appendectomy.”

Sure, you go ahead and believe that, but when you make the 11 o’clock news because the 10,000 records of names, addresses, and social security numbers you maintain wound up on the Internet, don’t call me. I’m not going to touch it. I warned you about passwords that can be cracked faster than you can enter them.

As you are not qualified to do the job, you certainly aren’t qualified to dish out advice about it. Thanks you you, I have to waste my time telling others like you and novice users alike how they aren’t as impenetrable as they think just because they installed a desktop firewall or are behind a NAT router. Since you are the long-time trusted source for information (that guy in accounting who is really good with computers, or your nephew who plays computer games a lot) there is no way I convince victims of your ignorance to trust my advice.

False authority syndrome sucks. Look it up.

On the rare occasion I do feel sorry enough for you to get involved if your tragic technical situations, shut the hell up and let me do my job. I don’t need you second guessing my every decision, and demanding I explain everything I do. When the bill gets run up because you won’t leave me alone, or because your network looks like it was designed by retarded lemming, just shut up and pay the bill.

I could have redeployed all of your repaired workstations in a fraction of the time, but considering that I had to manually set up every printer on your network by hand instead of pointing to a share on the server, well, your lucky it didn’t take even longer. Sure, I’ll recover old, tainted profiles off of your dying machines and manually move them to a new workstation, but its going to cost you. Your employees were too dumb and hardheaded to save documents to the server share? Sorry, they are gone. I told you about folder redirection, and the ability to, for the most part, keep users from saving documents to their local machines, but you didn’t listen.

But, in the end, I don’t care. You’ve dug your own grave, and its yours to lie in. If you pay me enough, I’ll pretend to care, at least for a little while. If you’ve got the money, I’ll be over here playing with my Mac and Linux boxes. No, don’t ask me to explain those to you.

Managing E-mail Lists

2007-06-11T20:17:00.000-05:00

[I need to set up an e-mail distribution list that doesn't reveal list members.] Now I'm having a problem with my list being too large. When I send out a message most of the recipients don't get it. I get a system administrator message saying my e mail was "truncated" or didn't even go through... - Virginia

There are four common ways mass e-mails are distributed.

Mailing list managers

Group E-mail Alias

Unassisted in a regular e-mail client

Desktop mass-mailing software

Mailing List Managers

Mailing list managers are the best choice for fluid memberships, or member lists over two dozen or so people. Mailing list managers typically reside on or near an e-mail server, taking incoming messages and redistributing them to everyone on the mailing list. Other features available in more popular packages are automatic bounce detection (to remove invalid e-mail addresses from the list automatically), subscription management, and moderation.

Mailing list managers allow you to take a hand-off approach to mass e-mail distribution. All you have to do is send a single e-mail to the manager, and it will figure out the rest. Users who want to be removed from the list can often do so without any interaction from another human. Better still, many mailing list managers will maintain an archive of list activity. Past newsletters and announcements can be preserved with no additional effort on the part of the sender.

GNU Mailman is a very popular choice in mailing list managers. It provides all of the features one typically needs and more. The biggest drawback of Mailman is the invasiveness of its install. It is not practical or even possible to install Mailman on most low-end web hosting accounts. If this isn't a problem, Mailman is an easy choice.

Another option is phplist. phplist requires no more than your average PHP-based web application to run. Most web hosting accounts, even cheap ones, have what is required to support phplist. Unlike Mailman, phplist is not a group mailing list manager. Its intended for one way communication only. For people only interested in sending newsletters and announcements, phplist can get the job done without the complication of Mailman.

Group E-mail Aliases

Group e-mail aliases allow a single e-mail to a single address to be sent to multiple recipients. This is where the similarity with mailing list managers ends. Group aliases must be managed manually by an e-mail server administrator or user-accessible control panel. Aliases don't work well for large numbers of users, and typically have no access control available. Anyone can send any e-mail to an alias group, and each member e-mail address will receive a copy.

As far as user-generated e-mails are concerned, there is really no need to use a group alias. An unassisted desktop e-mail client can get the job done just as well.

Desktop E-mail Clients

Doing things the hard (easy?) way by dumping large numbers of e-mail addresses into a standard desktop e-mail client is likely the most popular way of dealing with mass mailing list. This solution is cheap, easy, and effective... until you break a few dozen recipients, that is. Large amounts of e-mail being forced through your mail server without the intelligent batching and throttling of a good list manager can be a disaster. If SMTP server recipient limits don't trip things up, an angry mail administrator with a choking server or spam detection and prevention mechanisms might.

If a desktop e-mail client is to be used unassisted for mass mailings, be sure to respect recipients privacy. Listing several recipients in the To or CC fields is only acceptable in small group settings, when each recipient is already aware of the e-mail addresses and participation of the others. The easiest way around this is to address mass mails to yourself, and BCC all other recipients.

Desktop Mass Mailing Software

Desktop mass mailing software is often the tool of choice for spammers. This shouldn't deter you should you have legitimate needs. That said, if you are running a legitimate e-mail list, you'll probably be better off with one of the two previous solutions. Mass mailing software will yield better results than an e-mail client alone, but is still a stop-gap measure with most of the disadvantages that come with using anything but a server-based mailing list manager.

All things considered, just get a decent mailing list manager. Everyone stands a very good chance of getting either Mailman or phplist up and running. There is little excuse for not using a decent mailing list manager. If you have to run with something else until you've got a real mailing list manager set up, so be it. But make sure your temporary solution doesn't become a permanent one.

Open Source Public Relations

2007-06-01T13:17:00.000-05:00

Marketing says “Buy me.” PR says, “Its OK, go ahead.”

To be blunt, open source public relations sucks. Linux public relations sucks. (While the open source and Linux communities are very close, there are members of one who are not members of the other. Despite this, I will henceforth refer to both groups simply as the open source community.) If the open source community expects any significant adoption of their ideals and products, decent public relations is a necessity. This is due to the nature of the community and the market.

With the exception of Apache, user bases of various open source software products are far below that of commercial rivals. The results are a smaller talent pool, fewer businesses or business-like points of contact for help, and most importantly, greater effect of a single individual on the image of the user and support community as a whole. As members of the community, we must recognize this and attempt to keep our perception positive, and remain active in public relations to that end.

Would you send your child into someone's household storm?

We tend to be a very passionate bunch. While people rage on and fight wars over issues of religion and politics, we do the same over software licenses and patents, feature sets, and implementation details. Its human nature to get defensive, and often offensive, when it comes to beliefs. Being passionate about something is a good thing, but perhaps we need to keep our passions behind closed doors. When outsiders look in, without context and understanding, how are they likely to perceive conflict? Will they see colleagues battling things out for the greater good of the product and community? Will they see unprofessional people who are still emotionally children crying over bruised egos? Regardless of what the truth is, its the perception that matters

Consider for a moment a household in turmoil. Most of us know at least one. It could be parents who are constantly fighting, or perhaps its a single parent with questionable friends or a poor choice for a love interest. Maybe its just the family that is falling apart because nothing seems to be going right. Regardless, would you send your child into such a household to stay even a night? Nobody is perfect, but when you see significant, obvious problems which could affect your child, could you really send them into the fire? The situation may not seem relevant, but the reality is its more significant than most would believe.

Technology is becoming a greater and more significant part of our lives every day. It could be your business and the hopes, dreams, and mortgage it carries that is effected by your technology choices. Maybe its the memories of a lost loved one immortalized in digital photos and video clips. It could also be your finances, your recreation time, your educational tool, or your connections to distant friends. Software, operating systems, media formats – they can all be key parts of our lives. Who are you going to trust that to?

Who's legs to stand on?

Most people don't have the passion about technology that we do, so choices are made in a much more practical way. Would you trust your digital life to a successful business with strong leaders, excellent market position, and the ability to command and define the landscape? Would you trust it to a bunch of people in the game as a hobby who's allegiance is fleeting, and needs defined by ideals instead of practicality? To us, the decision is clear. The problem is, its also clear to the people we seek to bring on board.

We have to understand the needs of others, and what is important to them. We can't even begin to speak to them on the right level until we know where they are coming from. We can't possibly devalue the importance of going with the crowed when we don't even know why other's have made that choice in the first place.

Do you care?

You can't help someone who doesn't want to be helped, but there is more to the situation than saving users from themselves. Today, the open source community doesn't have the power it needs to drive the world in the direction it wants. We dig and we claw, we make some headway, but in the end we are still being kept down. If we are to reform or eliminate software patents, promote and standardize open formats, eliminate DRM, or accomplish any of our other far-reaching goals, we need all of the help we can get. Let's present the best of ourselves to others, and seek to understand their wants and needs. Maybe if we work on our PR, we can rally more troops to the cause.

Why Use RCS for Backups?

2007-05-31T12:43:00.000-05:00

Indrect response to Subversion for backup critics

A little while ago I wrote a series on using Subversion along with some shell scripts and other fun things to provide versioned backups of files. Subversion and other Revision Control Systems have some key features that make it very well suited for this task. After all, that's at the core of what RCS do.

RCS aren't always the best choice for this type of thing. Dedicated, versioned backup systems are much better at this core task. They are more specialized, have less cruft, and less chance of breaking. All that said, sometimes doing things the unnecessarily hard way has huge benefits. What are they?

Tagging

A common feature RCS offers that one wont find in backup offerings is tagging. Tagging allows you to mark a copy of your data, or subset there-of, in some meaningful way. This is a bit different than just marking a copy of your backup from some specific point in time. Tagging allows you to be selective about what gets marked, and marked items can span different time periods.

As an example, you can mark a selection of family photos in your backups as “Pictures for Grandma”. Exporting images with that tag will allow you to quickly put together a set, and store that set with very little overhead. When your aunt sees grandma's new photo-filled DVD and requests a copy, its a simple matter to re-export that set and send it off. Photo album software can do this, but what if you want to include video clips as well? How about the poetry your kid wrote in school? Tagging the files lets you mix and match data of any type.

Branching

Branching allows you to maintain multiple, parallel data sets. This feature will let you back up originals of files and work on them separately from your originals while maintaining backups of both. This is less useful in a backup scenario than tagging because you can always roll back files to specific versions with a regular backup setup. The real advantage comes into play when you want to have easy access to both the original and modified data sets at the same time.

One of the more obvious applications is photo editing. Photo sets may be thumbed through, with a limited number of shots making it through to the next stage, be it another elimination, basic manipulation, cataloging, or other actions. Utilizing branching, backups can be made of both the raw, original set as well as intermediate steps without wasting storage space on-disk and in backups. If you have 5 rounds of eliminations or selections, and want to preserve each round, you don't have to store your final selections 5 times.

Multiple Working Copies

One of the biggest benefits of using a RCS for backups is the ability to have multiple working copies. You can mirror your data on multiple systems, editing what you want, where you want. Changes at all locations are merged back into your backup data, and can be propagated back out to all working copies. Simply, you can have the same data anywhere, with changes at all locations backed up an an efficient manner.

In the very common case of having a work and a home computer, or having one desktop machine and one laptop, the convenience of multiple working copies becomes apparent immediately. So long as you keep up with minor housekeeping, the work you do on one computer is available on another. Should one machine be unavailable for any reason, you can pick up where you left off elsewhere.

Portable Backup Data

Many snapshot-preserving backup methods use file system-specific tricks or features, making it difficult if not impossible to move backup data to a new system or volume without loosing key information. RCS like Subversion can be quite happily picked up and moved to different locations, regardless of disk format or operating system. The ability to store this data easily on portable, rewrite able media like a USB drive is a very convenient feature as well, since you can have all of your versions readily accessible wherever you go.

Hopefully I have shed some light on the craze regarding using revision control systems for backing up data. In most situations, its more trouble than its worth. But for those of us with special needs, nothing else will do.

Semi-Reliable Periodic Commands

2007-05-18T14:57:00.000-05:00

Impromptu

cron is great. Using cron, you can schedule commands to be run at regular intervals or at specific times. One of the major drawbacks of cron is that it doesn't generally keep state regarding job's status. If a job is scheduled to run at midnight, but the system is powered down at midnight, the command will never be run.

There are a few solutions out there to take care of this shortcomings, but many are designed for system-wide use only. What if you don't want to intermingle your personal jobs with the system-wide ones? Here is a fairly simple script that works as a wrapper, providing a little insurance to help make sure jobs get run.

#!/bin/bash

export P_ENV=~/.profile
export P_TRACK=~/.p_track

if [ -e "$P_ENV" ]; then
 . $P_ENV
fi

case "$1" in
 h) 
  export P_NOW=$(date +%Y-%m-%d-%H)
  ;;
 d)
  export P_NOW=$(date +%Y-%m-%d)
  ;;
 w)
  export P_NOW=$(date +%Y-%U)
  ;;
 m)
  export P_NOW=$(date +%Y-%m)
  ;;
 *)
  echo ERROR: Term not specified. Must be one of h, d, w, m .
  exit 1
esac

if [ -z "$2" ]; then
 echo ERROR: Command to run not specified.
 exit 2
fi

export P_TAG=$(echo $2 | sed -e 's/[^A-Za-z0-9]/-/g')
export P_FILE=$P_TRACK/$P_TAG-$P_NOW

if [ -e "$P_FILE" ]; then
 exit 0
else
 rm -f $P_TRACK/$P_TAG*
 echo Executing $2 at $(date)
 $2 $3 $4 $5 $6 $7 $8 $9
 echo Done
 
 if [ "$?" -eq "0" ]; then
  touch $P_FILE
 fi
fi

The script has 4 operating modes. It can help ensure command are run hourly, daily, weekly, or monthly. It uses empty files in a configurable directory (~/.p_track by default) to keep tabs on the last time the command was run. Entries in the tracking directory are only created if the command run returns exit status 0 (no error.)

To use this wrapper, place it and a period (h for hourly, d for daily, and so on) in front of commands in your crontab like so:

*/20 * * * * ~/bin/periodic h ~/bin/command param1 param2

The above will check to see if the command needs to be run every 20 minutes, but will only execute it every hour. The advantage of wrapping individual commands instead using periodic command directories (check out the run-parts command) is less administrative overhead.

Note: The "real" periodic command is generally used to run system-wide periodic commands, often stored in /etc/periodic. UNIX-y systems tend to use other mechanisms and directories to do something similar.

Update 1: Changed the script from a group of if/elif to case (thanks for pointing that out Dave) and added a quick import of .profile or another file to set up environment.

Flash Objects and the Z-Index

2007-05-16T12:53:00.000-05:00

... On my Mac using Safari the drop down menus appear. When I move my cursor down on the drop down the rest of that drop down disappears In IE the drop down menus go only as far as the the top of the Flash file. I am not sure if it does the same in Mozilla. Anyway to circument this? Perhaps I could just do a pop-up page? - Michael

Yes! This is easy. First, you'll need to add a param tag to your object tag. Set the name to wmode and the value to transparent.

<param name="wmode" value="transparent" />

Then, force your Flash object to be below everything else by changing its z-index to 0. For the unfamiliar, think back to Geometry class in high school. You had X, Y, and Z axis. In the user interface world, the Z axis values specify how objects are stacked, and typically don't deal with perspective as you would see in a 3D game.

For some reason, applying z-index directly to a Flash object doesn't work properly. Wrap your Flash object in a div, and apply the z-index to it.

<div style="z-index: 0">...</div>

That's all there is too it!

Update 1: Unfortunately, the above code only works for Internet Explorer! The object tag is for IE only. Most embeded Flash content is wrapped with two tags in order to support both IE and other browsers. These tags are the object and embed tags. In order for this fix to apply to all browsers, modify the embed tag as well, adding the wmode property with a value of transparent.

Bad Programming

2007-05-13T15:20:00.000-05:00

Rant

The self-checkout kiosk at the local supermarket blocked, waiting for employee input instead of giving me errors. Why? Bad programming.

Planet keeps brining up my old posts if I modify them (typically to correct errors). Why? Yup. Bad programming.

Bad programming doesn't necessarily mean bugs and buffer overflows. It also means bad predictions about program flow. If a narrow sequence of events is all code can accommodate without some easy and automatic method to attempt recovery and/or to bring things back on track, its badly programmed.

Programmers aren't mind readers, but they need to consider the "what ifs". When a program behaves in a manner that is just plain stupid, its probably badly designed!

Silhouette Clone: Part 3

2007-05-08T17:27:00.000-05:00

Impromptu

Previously, we created a Subversion repository and automatically populated it with data from a directory which was shared over the network. This allowed users to work on the share while automated processes on the server stored versioned copies of the directory for later access. Now we will provide point-in-time recovery options for files in our repository without making users install and use a Subversion client.

Subversion repositories can be accessed over the network via WebDAV by means of special Apache HTTPD modules. The problem with this approach is that old versions aren't accessible using this method. Even after employing simple workarounds to present users with a prior versions, users would still have two places to look for data: the actual network share and a the URL of the WebDAV interface to the repository.

We can get around this by using WebDAV to serve up the Subversion repository and then re-sharing the WebDAV resource using the same protocols we shared the original network share with. Confused? Follow along.

The first thing we need to do is implement point-in-time tags in our Subversion repository. Subversion can copy files from one location in the repository to another without duplicating the file data. This means we can implement tags by simply copying files! For minimal impact, we will perform this copy entirely within the repository.

Commonly, Subversion repositories have a trunk/ or head/ directory where most of the work happens. We will also add a point-in-time/ directory to store our point-in-time copies. An easy way to create these directories is to use the svn mkdir command on our repository.

$ svn mkdir /path/to/repository/head -m "Created head/" $ svn mkdir /path/to/repository/point-in-time -m "Created /point-in-time/"

Note the use of the -m option to specify a commit message. We are performing operations directly on the repository and are therefore creating new revisions of it. Subversion demands you leave a message (even an empty one) when creating a new revision.

Now that we are using a head/ directory, all of our day-to-day work should be performed there. Users do not need to be aware of the head/ directory at this point so we will simply pretend that head/ is the root of our repository.

$ svn checkout /path/to/repository/head /path/to/working/copy

Any changes to the working copy will be committed to the head/ directory in our repository transparently.

Now that we have a repository and working copy set up to utilize a head/ directory, we can continue as we did in parts 1 and 2 to enable automatic commits. From this point, implementing point-in-time copies is quick and easy. A simple shell script to copy head/ to a subdirectory of point-in-time/ at regular intervals will get the job done.

#!/bin/bash svn copy file:///path/to/repository/head "file:///path/to/repository/snapshots/`date +%F\ %T`" -m "Point-in-time marker added"

Note that while we can put the above command in the same file as our previous commands to commit changes to the Subversion repository, we don't have to. Separate scripts will let us use separate schedules for our actual backups and point-in-time tags. This allows us to back up data frequently without presenting a user attempting to recover a file him or herself with an overwhelming number of file iterations to wade through.

Once our backup commands and point-in-time tags commands are running at regular intervals, we will have a repository layout that is fairly self explanatory.

$ svn list /path/to/repository head/ point-in-time/ $svn list path/to/repository/point-in-time 2007-05-08 17:00:00/ 2007-05-08 17:15:00/ 2007-05-08 17:30:00/

Note that by modifying the date format string in our point-in-time tag command we can change how subdirectories of our point-in-time/ directory are named. 24-hour time is used instead of 12-hour time to make sure directories are always sorted in chronological order.

Now that we have our data efficiently stored and laid out, we have to provide access to it. Most file sharing packages do not understand Subversion repositories, so we will have to build a bridge. We can start by using Apache HTTPD to provide WebDAV access to the repository. After installing and enabling the mod_dav_svn module, a few lines in an Apache configuration file will do.

<Location /repository> DAV svn SVNPath /path/to/repository </Location>

Ideally, you will want to lock-down this location using a combination of users, passwords, and IP addresses. See the Apache documentation for more information.

Once Apache is reloaded and serving the repository, install davfs2. When installed properly, you can mount the Subversion repository via WebDAV as you would any other file system. A simple setup is to create a directory for the network share, and place your data in a subdirectory. This will let you mount the Subversion repository along side your data.

$ mkdir /path/to/network/share $ mkdir /path/to/network/share/data $ mkdir /path/to/network/share/backups $ mount -t davfs -o ro,noaskauth http://localhost/repository/point-in-time/ /path/to/network/share/backups

Note that you do not want to allow the mounted backup directory to be modified.

Mirroring our Subversion repository layout in our local file sytem seems silly, but we used this setup for a reason. Providing users direct access to the live Subversion repository for use as the live network share would generate up to 3 new revisions in our repository each time a file is saved! This is due to the rename-write-delete method commonly used to avoid data loss when saving files. Multiple revisions aren't too bad, but the way in which these revisions come about prevents Subversion from storing data efficiently.

Using Subversion as a (mostly) transparent replacement for Microsoft Window Server's Shadow Copy for Shared Folders can be quite cumbersome, but its a viable alternative. Current developments in Linux file systems will likely render the need for this workaround obsolete in the near future. Until then, take it one step at a time and enjoy automatic, versioned backups of your network shares.

Silhouette Clone: Part 2

2007-05-07T14:38:00.000-05:00

Impromptu

Read Part 1

Now that we have already set up a Subversion repository as well as automatic file additions and commits, we need to account for deletions and provide access to versioned data.

When a file is deleted from a Subversion working copy without the proper procedures (which is pretty much what we are going to do) Subversion is shocked not to find the file when checking repository status.

$ svn status /path/to/working/copy ! deleted-file

All we need to do is let Subversion know we want to delete the file using the svn delete command. As in the previous installment, a little finagling from a shell script gets the job done.

#!/bin/bash svn status /path/to/working/copy | grep ^\? | cut -c 8- | xargs svn add svn status /path/to/working/copy | grep ^\! | cut -c 8- | xargs svn delete svn commit -m "Automatic snapshot" /path/to/working/copy

You may note that now we have two calls to the svn status command. Below the above is rewritten to cache the output of this command in order to improve performance, but I wanted to show this format just once to compare to the previous version of the script.

#!/bin/bash svn status /path/to/working/copy > /tmp/svn-status.txt grep ^\? < /tmp/svn-status.txt | cut -c 8- | xargs svn add grep ^\! < /tmp/svn-status.txt | cut -c 8- | xargs svn delete svn commit -m "Automatic snapshot" /path/to/working/copy

Unfortunately, without handling file moves and renames through Subversion, they cannot efficiently. A moved or renamed file looks like a missing and new file pair to Subversion.

$ svn status /path/to/working/copy ! old-file-path ? new-file-path

Even though we can't handle this as efficiently as Subversion can, we can still handle it. No additional work is required to handle file renames or moves.

Next time, we will provide access to the versioned file system over the network without the need for clients to use the svn command line tool.

Silhouette Clone: Part 1

2007-05-05T18:28:00.000-05:00

Impromptu

A feature of a server operating system I will not name enables administrators to create snapshots of network shares automagically. This is above and beyond storing data on a server to facilitate regular backups. Users who clobber each other's work can view the share as it was previously and recover missing or modified documents themselves. If this sounds familiar, it should. Its simply automated version control.

Subversion is a great tool for storing versioned copies of filesystems. Combined with Apache HTTPD and a few modules, access to versioned, writable Subversion repositories over the network is possible, but not ideal for many situations. In order to prevent data loss, many applications do a 3-step shuffle when saving files. Even this would be fine, except for the fact that this doesn't always happen efficiently over WebDAV, the protocol used to share Subversion repositories in this manner.

A decent compromise is to share data over the network as one normally would, and automatically commit changes to a Subversion repository at regular intervals.

Setting up a Subversion repository is quite simple.

svnadmin create /path/to/repository

After that, simply check out your (empty) repository.

svn checkout /path/to/repository /path/to/working/copy

Then, simply share the working copy via the method of your choice. You may want to restrict access to the .svn directory to prevent accidental destruction.

A simple shell script can be run from cron at regular intervals to add new documents to the repository. Modified documents will be committed automatically during the commit command.

#!/bin/bash svn status /path/to/working/copy | grep \? | cut -c 8- | xargs svn add svn commit -m "Automatic snapshot" /path/to/working/copy

Once your cron job starts firing, you will have point-in-time snapshots of your share. In future posts, we'll deal with deletes and simple remote access to the versioned data.

Simple Command Line Text Processing

2007-04-21T15:03:00.000-05:00

Impromptu

I almost didn't write this, because I decided to check the latest goings on at GnuJersey.org, and happened to see Dave posted something about sed. I didn't want to be a copy cat. Then I remembered that nothing is original anymore. Plus, you will find no mention of sed (other than this and the previous one) or anything else in Dave's post here. so here we go.

The other day I needed to sort a few dozen lines of text. For some reason, the behemoth text editor I was using didn't have a sort function. (What gives?) A commercial competitor I recently switch from had this functionality, but I didn't want to reinstall it just for one quick sort. Instead, I turn to the old standby of the UNIX user: piping for text processing!

In order to demonstrate some key functionality, I am going to operate on the contents of a simple text file. The file contains the top 10 most common passwords and respective frequencies, according to this article. For reference, here are the contents of the file:

letmein 1.76% thomas 0.99% arsenal 1.11% monkey 1.33% charlie 1.39% qwerty 1.41% qwerty 1.41% 123456 1.63% letmein 1.76% liverpool 1.82% password 3.780% arsenal 1.11% 123 3.784%

In order to understand what is going on, you need to be familiar with 3 input redirectors used in the UNIX world. They are as follows.

< reads data from a file into the command on the left

> writes data from a command into a file on the right

| forwards data from one command to the next

Input redirection is actually far more complicated and capable than that, but that is another post.

The first key command is sort. sort does what you may expert, it sorts lines in a text file.

$ sort < demo.txt 123 3.784% 123456 1.63% arsenal 1.11% arsenal 1.11% charlie 1.39% letmein 1.76% letmein 1.76% liverpool 1.82% monkey 1.33% password 3.780% qwerty 1.41% qwerty 1.41% thomas 0.99%

In order to clean things up, sort is often combined with uniq. uniq preserves unique lines in a text file, but it has a serious limitation: it only works when duplicate lines are adjacent. Therefore, if you want to use uniq, use sort first.

$ sort < demo.txt | uniq 123 3.784% 123456 1.63% arsenal 1.11% charlie 1.39% letmein 1.76% liverpool 1.82% monkey 1.33% password 3.780% qwerty 1.41% thomas 0.99%

Another handy one is cut. cut splits lines into fields (think spreadsheet) and allows you to cut out the fields you want to keep. cut can work with any simple field delimiter given by the -d option. A comma separated list of fields to display is given by the -f option.

$ sort < demo.txt | uniq | cut -d ' ' -f 1 123 123456 arsenal charlie letmein liverpool monkey password qwerty thomas

If you want to get back to your command line roots, or just like the speed and simplicity of throwing a few small, highly specialized commands at a your problems, keep exploring. There are several more fun text processing commands available on most UNIX-like operating systems. More complicated solutions like awk and sed open up even more quick solutions to common text processing problems.

Wireless Stupidity

2007-04-20T19:18:00.000-05:00

Impromptu

People still just don't get it. Most of the wireless networks pictured are protected by WPA Pre-Shared Key authentication, and the rest WEP. (WEP sucks, but its better than nothing. Or is it...) Obviously, one wasn't protected at all.

(Possibly) worse, a wireless network I am in no way responsible for, but am closely tied to, was serving up tasty, free Internet access to patrons. When asked if the network was encrypted, I said no, and briefly mentioned that we want it to be quick and easy for patrons to hop on. No fuss, no muss.

The response was an awkward, almost accusatory look. Apparently there is some problem with unencrypted networks. They are unsafe and give you STDs like the crazy, promiscuous girl even the desperate guys won't go near. The reality is, even an encrypted network isn't necessarily “safe”. If you want safe and private computerized communication, you want VPNs and SSL/TLS. After all, that's what they were invented for. The right tool for the right job makes all the difference.

Ugh

2007-04-20T01:39:00.000-05:00

Impromptu

Every human life is different. From the moment consciousness sparks while we are in the womb, every thing we see and experience shapes our perception of the world. We reflect as we grow and the sum of our experiences shape our perception and personal philosophy.

The problem is that this divides us. We segregate sometimes because we want to. Other times, its because we are forced to. Worst of all, those who should be friends push each other away by making others feel unwanted, unappreciated, or alienated.

Intelligent people can disagree and even argue while maintaining allegiance, but third parties only see the battle. When outsiders see this in a community already known to be fragmented and filled with infighting, the perception is several times worse.

Who would want to invest in such a community?

Background Bookmarklets

2007-04-15T15:29:00.000-05:00

To summarize for the sake of brevity: [How can I load a page with a browser bookmarklet without actually viewing the page?] – Dave

First, to provide a little background, Dave wanted to save bookmarks to a server on the Internet (like del.icio.us) but at a location under his control. More importantly, leaving the current page to do so (like del.icio.us) was unnecessary and unwanted.

So, how do we do that? If you said AJAX like I did at first, your wrong! The XMLHTTPRequest object Ajax uses won't operate across domains for security reasons. So, what is the solution then? We need to use JavaScript to ensure a one-click affair, but the champion of JavaScript and Web 2.0 has left us out in the cold. We must use the old stand by of JavaScript and DHTML, the Image object.

The Image object provides some very useful behavior for us.

Unless you are crossing the HTTPS boundary, there aren't really any security mechanisms around by default to get in our way.

We can create an manipulate Image objects without every displaying them, or anything else, to the user.

Image objects are fairly cheap to create an use, unlike trying to fire up ActiveX, Java, or Flash.

Image objects will load any arbitrary URL we give them immediately and in the background, providing us with flexible, asynchronous behavior.

So, here is the code.

var img = new Image(); img.src = "http://localhost/bookmark/";

But wait! Where is the URL we want to bookmark? Easy, its in the referrer header sent to the target web server. If you don't want to trust referrers, all you need to do is escape and use the location.href property.

var img = new Image(); img.src = "http://localhost/bookmark/" + escape(location.href);

Now, there are a few things you can do with this. Start the entire thing off with javascript: and you've got a suitable bookmarklet. Wrap that in a tags and you have a click-able link you can embed in pages.

If you are interested, here is the mailing list thread this was discussed in.

6 Steps for Moving Websites

2007-04-10T14:38:00.000-05:00

How do I move my web site to another host? - DFRI, and others

If you have a 100% static web site, moving from one host to another is fairly easy. Of course, how often do we get to do things the easy way? Assuming a dynamic web site with some database-housed content (say, a content management system or forum software) there is very particular sequence of events that needs to happen to minimize pain. Some of the steps below only apply to web sites with database-backed content, and can be skipped for sites without it.

Step 1: Set up the new hosting account.

This step is obvious to anyone in the web hosting universe, but is not apartment to others. Moving a web site takes time. If you try and shut down your old service without having a new spot ready, you can wind up in web server limbo. Give your new host 24 hours to get your account set up, and DNS records propagated to their outward-facing DNS servers. Some hosts only update their DNS records once every 12 to 24 hours!

Step 2: Lock your existing web site.

If you aren't the only person updating your web site data, you'll need to lock it temporarily. If something changes after you have copied your data, but before you have finished, you'll have two separate versions of your data! If you don't want to get a stack of e-mails that say “Where did my post go?”, don't skip this step.

Step 3: Copy your data.

All of your files and database data will need to copied to your new web host. This should be fairly easy, so long as you don't forget the incidentals. Be mindful of file permissions, as some files may be created by the web server, and a loss of proper permissions could break your web site. The biggest culprits here are upload directories, uploaded files, and cache files. Consult your original deployment instructions or setup manuals if you are unsure about file permissions. Also, keep a close eye on database host names, database names, user names, and passwords. If any can't be copied exactly, you'll need to note the changes for later.

Step 4: Connect your old web site to your new database.

If you have any dynamic content, you will need to point the code on your old host to the database on your new host. Keep in mind that this is against the terms of service for many web hosts, and not even possible with many. If your new web host won't let you do this, find another one and go back to step 1. If your old web host won't let you do this, well, its a good thing you are moving. A few quick edits to configuration files should be all that is necessary. When you are done, unlock your existing web site, but leave any file uploads disabled. It's OK, all of those edits to the database are happening on your new web host.

Step 5: Connect your new web site to... your new web site.

You've already copied files, now you need to reconfigure your code. Make the same edits to files you did in step 4. Some options may be a little different (localhost vs sql.hostdomain.tld, for example.) Wow, this step was short.

Step 6: Updating DNS information.

The final step is to update your DNS information to point to your new web host. In most cases, this means changing your domain's name servers to the values provided by your new web host. Those with third party DNS hosting (or DNS hosting from the domain registrar) can update the DNS records for the domain. Your new host can provide you with all of the necessary information.

That's it! Your done! It make take up to 72 hours for the switch to be complete, but since you carefully followed the above instructions, it doesn't matter! It will all come together. After you are satisfied that no one is being pointed to the old web host, cancel your old account.

What? You want to move e-mail accounts too... Well, that is a different story...

Firewalls Suck

2007-04-05T15:28:00.000-05:00

Impromptu

Firewalls suck. I hate them. Sure, a firewall can do you a lot of good in the hands of someone who knows what they are doing, but commonly, firewalls are the little Dutch boy's finger of the network security world. The problem is, no one else comes by to help.

Security charlatans sell firewalls like snake oil. They claim firewalls are mythical pieces of equipment or software that can protect the user from every computer security issue out there. So what happens? Your novice or otherwise uneducated user (or even worse, users who think they are well versed, but really aren't) runs a firewall and thinks they are safe. They don't patch their systems promptly because they believe the firewall will protect them. They don't password protect sensitive services, or encrypt sensitive information because they think the firewall will protect them. They generally do stupid things because there is no need to be concerned; the firewall will protect them.

Just think about immortality for a moment. If you could never be hurt or killed, would you wear your seatbelt?

All that said, firewalls are actually quite useful as temporary solutions to problems, or insurance against possible issues that may come up in the future. If a vulnerability is discovered in a service your computer runs, you can quickly disable access to this service at the network level if, for some reason, you cannot disable the service outright.

In the end, a properly configured, patched system does not need a firewall. If a firewall is your main defense, you've got other problems.

Perils of Rootkit Detection

2007-03-30T13:35:00.000-05:00

i admit i haven't followed security as closely as i did when i just ran windows, but are there now rootkits that can hide themselves from computer b when it's on computer a? - ice60

Yes, but they do not explicitly. In fact, any rootkit hiding itself from other processes is also hiding itself from other systems on the network as a free side-effect.

If computer b is looking for a rootkit on computer a, it will ask for information about the system. Details about running processes, files, logs, and so forth may be queried. The problem is all of these requests must pass though the infected system, and therefor cannot be trusted. Involving a daemon in your quest to sniff out a rootkit only provides an additional opportunity for the rootkit to hide itself.

Let's look at one of my infamous metaphors as an example. We can equate a rootkit-infected system with a brainwashed individual.

If you were brainwashed, you wouldn't necessarily know you were brainwashed. Any decent brainwasher would hide the brainwashing from you, perhaps with false memories.

If I asked you if you were brainwashed, you wouldn't be able to tell me you are brainwashed. Quite simply, you wouldn't know.

If I asked you indirect questions relating to your brainwashing, I *might* be able to infer you were brainwashed. There are certain inconsistencies that may come up in your responses. Perhaps you claim you were eating an anchovy pizza, but I know you hate anchovies.

Since any response you give could be effected by the brainwashing, I still wouldn't be able to tell with any certainty that you had been brainwashed.

The only 100% reliable way to detect a rootkit, assuming you have a signature for it, is to bring the infected system off line, and look for it without relying on any part of the infected system. This could mean running the system from a separate, bootable disk, or transplanting the infected systems hard drive into another clean system.

For further reading, please see this article in Computerworld.

Apache HTTPD Virtual Hosts and SSL

2007-03-27T10:17:00.000-05:00

how do i do configure Apache2 to answer for all subdomains requested. as i want to be ble to create sub domains on my server using my main domain name. - patty552 @ UbuntuForums

Various statements of misconception in this thread at UbuntuForums.

Apache HTTPD Virtual Hosts make the web go round. They allow a single server to host many web sites with different addresses. In the web hosting world, this allows for efficient, cheap web hosting. For your average person running Apache HTTPD, it means efficient, cheap home networking and web development testing environments.

Virtual hosts are easy to set up, just check the documentation at http://httpd.apache.org/docs/ . That said, there are two main ways to configure virtual hosting, which you have to keep in mind when starting out. One method involves matching the request host name, IP address, port, or any combination of them to a separate block of HTTPD configuration statements. The other method specifies a directory pattern to use for the document root and cgi-bin based on parts of the host name.

For the former method of configuring virtual hosts, all one needs to do is add a wildcard ServerAlias directive to the VirtualHost block for your domain name.

<VirtualHost 1.2.3.4>
 ServerName domain.tld
 ServerAlias *.domain.tld
 DocumentRoot /var/www/
</VirtualHost>

For the latter, all one needs to do is match against only the domain name, or include subdomains in the pattern, making sure to create the appropriate directory structure.

VirtualDocumentRoot /var/www/%-2/
VirtualDocumentRoot /var/www/%-2/%-3/

Web hosts tend to use the larger VirtualHost method. Smaller shops, or generic mass hosts (departmental or employee hosting within an organization,for example) will find the latter very helpful, particularly when serving out of user's home directories.

All this is great, but what about SSL? You could start up a separate instance of Apache HTTPD to serve over an SSL connection, but you probably don't want to do that. There are some advantages, which but that is beyond the scope of this piece. The easiest way is to use a VirtualHost block to match against connections on port 443, the default HTTPS port. Contrary to popular belief, you do not need a separate IP address to do this.

<VirtualHost *:443>
 SSLEngine On
 SSLCertificateFile /etc/httpd/ssl.pem
 DocumentRoot /var/www/
</VirtualHost>

This works just fine if you are only serving one site over HTTPS. The problem comes in when you have multiple domain names being served from the same server which need SSL. Since the SSL certificate needs to be used before the web browser sends a request to the server, the server has no way of picking a domain-specific SSL certificate to use. Name-based matching just won't work for SSL. This is why proprietors of shared web hosting services demand that you purchase a dedicated IP address if you want to use SSL. IP addresses are known before SSL certificates are used, so by matching based on IP address, we can use domain-specific SSL certificates.

<VirtualHost 1.2.3.4:443>
 SSLEngine On
 SSLCertificateFile /etc/httpd/dom1-ssl.pem
 DocumentRoot /var/www/dom1/
</VirtualHost>

<VirtualHost 1.2.3.5:443>
 SSLEngine On
 SSLCertificateFile /etc/httpd/dom2-ssl.pem
 DocumentRoot /var/www/dom2/
</VirtualHost>

So, to recap, you do not need a separate IP address to use HTTPS. You do need separate IP addresses to use HTTPS on servers with multiple domains using SSL.

New Jersey Ubuntu LoCo Team

2007-03-25T13:39:00.000-05:00

The New Jersey Ubuntu LoCo team's first South Jersey meeting is pictured above. Even though New Jersey is the 4th smallest state in the United States of America, for some reason we need to have to seperate meeting locations. Sad, no? The group is headed by Joe Terranova, so you can bitch at him for the sillyness.

Password Strength

2007-03-17T16:15:00.000-05:00

Impromptu

I came across a thread at UbuntuForums where someone asked how safe the password “s8fd7fg67fdg6” is. All these years, and we still have password education issues.

Excusing the fact that the guy just mashed on the keyboard producing something unlike what he would actually use when password-picking time came, this is a horrible password. It would take a while to crack, but its hard to remember and could easily be a lot stronger. 600+ times stronger, actually.

So, putting easy to remember aside, I will make a few assertions. If you want proof, do the math.

The strength of a password is determined by the maximum number of attempts a brute force attack would require to find the password under ideal conditions. Ideal conditions in this case are knowing the maximum length of the password and the character set it was derived from. There is more to intelligent brute forcing (no, not a misnomer) than that, but that is beyond the scope of this piece.

The maximum number of attempts required, as described above, can be calculated by adding one size of the character set s, and raising the result to the length of the password l. That would be (s+1)^l, of course. The +1 is in there because I said “maximum” above. We have to account for the possibility that a password is shorter than the maximum by including an empty character in the character set.

Working under the assumption that we know the maximum length of the password as well as its character set is quite reasonable, considering that the average password is 8 characters long*, the vast majority of passwords do not contain non-alphanumeric characters*, and minimum and maximum password lengths are very often imposed on users.

*These numbers came from a group of passwords acquired from MySpace users that fell for a phishing attack. You'd think that this represents the bottom rung of users, and you'd be right, but previous password auditing results have given similar numbers. Your average computer and Internet user isn't really that savvy. Juicy details at http://www.schneier.com/blog/archives/2006/12/realworld_passw.html.

If you start with a password up to 6 characters long, composed of numbers and lowercase letters, there are 2,565,726,409 possible combinations. Increasing the length by just one character yields 94,931,877,133 possible combinations. If you include capital letters in the character set instead of increasing the password length, you get 62,523,502,209 possibilities. Obviously, its better to increase your password length. But wait!

If you start with a 7 character password made up of lower case letters and numbers, and increase the length to 8 characters, you have 3,512,479,453,921 possibilities. If you change the character set instead, there are 3,938,980,639,167 possibilities. Woah!

So, what does this tell us? To over-simplify, if your password is going to be more than 7 characters long, character set matters more than password length.

Restricting Interactive Access

2007-03-15T15:40:00.000-05:00

How can I give someone one SSH access to my server without compromising security? - Plenty of People

[How can I] allow shell access but NOT allow [arbitrary] remote commands to be run? - TyphoonJoe @ Ubuntu Forums (Impromptu)

First, every point of access to your system is a potential weak spot. Learn to accept it.

The long and short of this is if you want to allow someone interactive access to a machine, but want to control what programs they can run, you need to do one of two things. Either force a customized shell upon the user that will implement some sort of access control, or use regular, old-fashioned file system permissions to lock out certain programs.

Wrapping a shell is best left to experts. There are some interrupt issues to handle as well as some special case scenarios to worry about. Plus, its usually overkill!

File system permissions are quick, easy, and effective. There are just six simple steps to take:

Identify a restricted program or group of programs (such as ifconfig or gcc and ld.)

Create a group for the program or program group.

Add users permitted to use the restricted programs to the corresponding group.

Change the group owner on the program(s) to the specially create group.

Change file system permissions on the program so that only the owner or group can read or execute the program.

Pray you didn't botch steps 1 through 5.

Keep in mind that man other programs, scripts, and services may need to use what you have restricted. When locking out commands with file system permissions make sure you are as thorough as possible.

Mirroring Web Sites

2007-03-11T21:44:00.000-05:00

Impromptu

Mirroring a web site, or subset thereof, is actually very quick and easy. All you need is the handy utility wget. There are actually several good reasons for wanting to do this.

Snap-shotting a compromised web site for off-line analysis

Snap-shotting a healthy web site for off-line analysis

Setting up a decoy or dummy website

Legitimate mirroring to help someone else out with bandwidth

wget has quite a few options. Read the man page if you care. If not, here is a sample command to snatch a web site domain.tld.

wget --mirror --wait=2 --random-wait --force-directories --recursive --convert-links --page-requisites –domains=domain.tld http://domain.tld/

Most of the options above are the long form so that you can understand what they do without me having to explain each one. One pair worth noting is -wait=2 -random-wait.

Plenty of web hosts and administrators run statistical analysis on their logs. You don't want to set off any alarms, even if your intentions are pure. If some overzealous administrator sees some idiot beating the hell out of their website, they may decided to teach the wannabe DoSing bastard a lesson and phone the feds. The two above options are an attempt at keeping your full footprint in the logs from being noticed.

--wait=2 sets a pause between page fetches of 2 seconds, and -random-wait skews this by 0 to 200% per request. Logs will still show a lot of hits within a short time frame, but hopefully you will avoid some flagging thresholds. These two options will also help you dodge DoS filters as well.

Stealthy Sniffing

2007-03-09T14:44:00.000-05:00

Will get caught if I try and sniff packets at my local coffee shop/library/school/office? - About 50 people

YES! At least, if you ask that question I'm sure you will. If you are sniffing packets, probing ports, or just plain doing something nefarious, no amount of black clothing will help you when someone sees “Johnnie Smith's Library” show up in their iTunes Window. This is probably the least of your worries if you are conspicuously huddled in the back corner pointing a cantenna at everyone with a laptop.

Before we get into this, its important to note why you should care. The legality of packet sniffing is arguable. That said, its fairly easy to bring someone down for sniffing. People having been charged with computer trespass for simply browsing the web over someone else's wireless connection. Even if you don't get convicted, do you have the money to fight a legal battle, especially after the FBI confiscates your laptop as evidence? I didn't think so. If you are doing anything even remotely shady, keep your footprint in the logs as small as possible. Yes, many low-end consumer routers do keep logs too!

From the network's point of view, its the little things that give you away. You will hardly ever see promiscuous mode detection deployed, but when some administrator sees someone that shouldn't be there screaming packets over the wire, its a pretty big hint. Most operating systems will periodically poll servers for updates. Since these are typically long transactions, you are going to be fairly easy to spot on the wire if this happens. You may think to reconfigure or not use programs like iTunes (but all 1337 hackers listen to techno MP3s while sniffing!) but you would be quite surprised at the amount of software that phones home for updates when you aren't looking.

Worse still are noisy operating environment components. Some common culprits include DHCP clients, various zero configuration protocols, network browsers, and ARP requests triggered by the above. Turn off or reconfigure as many of these services as possible if you are going sniffing. Depending on what operating system you are running, you can set things up so that you can switch to a now “location”, and have many of these services turned off at once.

When you think you have configured everything correctly, flip on your packet sniffer and watch your own traffic. See what is leaking. No packet should be beyond scrutiny. At least one popular file sharing client uses DNS requests to check for version updates. Its efficient use of the architecture of the Internet, but it makes leak checking just a little bit more difficult.

When are are set to go out in the world and do your deeds, remember the ninja. While ninja did wear all black when jumping from rooftop to rooftop at night, somehow I doubt that is what you will be doing. Good ninja hid in plain sight, indistinguishable from everyone else. If you are near a university, ratty jeans and t-shirts are acceptable. If you are in a ritzy neighborhood, where your khakis and a sweater. If you look like a computer geek and there is no LUG meeting in sight, you will be suspect.

The Skinny

2007-03-05T14:29:00.000-05:00

I am very frequently asked questions about computers, electronics, information security, and other fun topics. I'm tired of answering the same questions over and over again. This archive of questions and answers will handle the repeats for me. Update: There are plenty questions I have anwsered and long since forgotten why. There will be no shortge of impromptu posts as a result.

If you have a question you would like to ask, e-mail me at frostycolddrink@gmail.com

As a side note, posts taged with scenario are discussions of scenarios and techniques from a high-level perspective. Posts taged with how to are more technical and include practical instruction you can use.