Managing E-mail Lists

[I need to set up an e-mail distribution list that doesn't reveal list members.] Now I'm having a problem with my list being too large. When I send out a message most of the recipients don't get it. I get a system administrator message saying my e mail was "truncated" or didn't even go through... - Virginia

There are four common ways mass e-mails are distributed.

1. Mailing list managers


2. Group E-mail Alias


3. Unassisted in a regular e-mail client


4. Desktop mass-mailing software

Mailing List Managers

Mailing list managers are the best choice for fluid memberships, or member lists over two dozen or so people. Mailing list managers typically reside on or near an e-mail server, taking incoming messages and redistributing them to everyone on the mailing list. Other features available in more popular packages are automatic bounce detection (to remove invalid e-mail addresses from the list automatically), subscription management, and moderation.

Mailing list managers allow you to take a hand-off approach to mass e-mail distribution. All you have to do is send a single e-mail to the manager, and it will figure out the rest. Users who want to be removed from the list can often do so without any interaction from another human. Better still, many mailing list managers will maintain an archive of list activity. Past newsletters and announcements can be preserved with no additional effort on the part of the sender.

GNU Mailman is a very popular choice in mailing list managers. It provides all of the features one typically needs and more. The biggest drawback of Mailman is the invasiveness of its install. It is not practical or even possible to install Mailman on most low-end web hosting accounts. If this isn't a problem, Mailman is an easy choice.

Another option is phplist. phplist requires no more than your average PHP-based web application to run. Most web hosting accounts, even cheap ones, have what is required to support phplist. Unlike Mailman, phplist is not a group mailing list manager. Its intended for one way communication only. For people only interested in sending newsletters and announcements, phplist can get the job done without the complication of Mailman.

Group E-mail Aliases

Group e-mail aliases allow a single e-mail to a single address to be sent to multiple recipients. This is where the similarity with mailing list managers ends. Group aliases must be managed manually by an e-mail server administrator or user-accessible control panel. Aliases don't work well for large numbers of users, and typically have no access control available. Anyone can send any e-mail to an alias group, and each member e-mail address will receive a copy.

As far as user-generated e-mails are concerned, there is really no need to use a group alias. An unassisted desktop e-mail client can get the job done just as well.

Desktop E-mail Clients

Doing things the hard (easy?) way by dumping large numbers of e-mail addresses into a standard desktop e-mail client is likely the most popular way of dealing with mass mailing list. This solution is cheap, easy, and effective... until you break a few dozen recipients, that is. Large amounts of e-mail being forced through your mail server without the intelligent batching and throttling of a good list manager can be a disaster. If SMTP server recipient limits don't trip things up, an angry mail administrator with a choking server or spam detection and prevention mechanisms might.

If a desktop e-mail client is to be used unassisted for mass mailings, be sure to respect recipients privacy. Listing several recipients in the To or CC fields is only acceptable in small group settings, when each recipient is already aware of the e-mail addresses and participation of the others. The easiest way around this is to address mass mails to yourself, and BCC all other recipients.

Desktop Mass Mailing Software

Desktop mass mailing software is often the tool of choice for spammers. This shouldn't deter you should you have legitimate needs. That said, if you are running a legitimate e-mail list, you'll probably be better off with one of the two previous solutions. Mass mailing software will yield better results than an e-mail client alone, but is still a stop-gap measure with most of the disadvantages that come with using anything but a server-based mailing list manager.

All things considered, just get a decent mailing list manager. Everyone stands a very good chance of getting either Mailman or phplist up and running. There is little excuse for not using a decent mailing list manager. If you have to run with something else until you've got a real mailing list manager set up, so be it. But make sure your temporary solution doesn't become a permanent one.

Open Source Public Relations

Marketing says “Buy me.” PR says, “Its OK, go ahead.”

To be blunt, open source public relations sucks. Linux public relations sucks. (While the open source and Linux communities are very close, there are members of one who are not members of the other. Despite this, I will henceforth refer to both groups simply as the open source community.) If the open source community expects any significant adoption of their ideals and products, decent public relations is a necessity. This is due to the nature of the community and the market.

With the exception of Apache, user bases of various open source software products are far below that of commercial rivals. The results are a smaller talent pool, fewer businesses or business-like points of contact for help, and most importantly, greater effect of a single individual on the image of the user and support community as a whole. As members of the community, we must recognize this and attempt to keep our perception positive, and remain active in public relations to that end.

Would you send your child into someone's household storm?

We tend to be a very passionate bunch. While people rage on and fight wars over issues of religion and politics, we do the same over software licenses and patents, feature sets, and implementation details. Its human nature to get defensive, and often offensive, when it comes to beliefs. Being passionate about something is a good thing, but perhaps we need to keep our passions behind closed doors. When outsiders look in, without context and understanding, how are they likely to perceive conflict? Will they see colleagues battling things out for the greater good of the product and community? Will they see unprofessional people who are still emotionally children crying over bruised egos? Regardless of what the truth is, its the perception that matters

Consider for a moment a household in turmoil. Most of us know at least one. It could be parents who are constantly fighting, or perhaps its a single parent with questionable friends or a poor choice for a love interest. Maybe its just the family that is falling apart because nothing seems to be going right. Regardless, would you send your child into such a household to stay even a night? Nobody is perfect, but when you see significant, obvious problems which could affect your child, could you really send them into the fire? The situation may not seem relevant, but the reality is its more significant than most would believe.

Technology is becoming a greater and more significant part of our lives every day. It could be your business and the hopes, dreams, and mortgage it carries that is effected by your technology choices. Maybe its the memories of a lost loved one immortalized in digital photos and video clips. It could also be your finances, your recreation time, your educational tool, or your connections to distant friends. Software, operating systems, media formats – they can all be key parts of our lives. Who are you going to trust that to?

Who's legs to stand on?

Most people don't have the passion about technology that we do, so choices are made in a much more practical way. Would you trust your digital life to a successful business with strong leaders, excellent market position, and the ability to command and define the landscape? Would you trust it to a bunch of people in the game as a hobby who's allegiance is fleeting, and needs defined by ideals instead of practicality? To us, the decision is clear. The problem is, its also clear to the people we seek to bring on board.

We have to understand the needs of others, and what is important to them. We can't even begin to speak to them on the right level until we know where they are coming from. We can't possibly devalue the importance of going with the crowed when we don't even know why other's have made that choice in the first place.

Do you care?

You can't help someone who doesn't want to be helped, but there is more to the situation than saving users from themselves. Today, the open source community doesn't have the power it needs to drive the world in the direction it wants. We dig and we claw, we make some headway, but in the end we are still being kept down. If we are to reform or eliminate software patents, promote and standardize open formats, eliminate DRM, or accomplish any of our other far-reaching goals, we need all of the help we can get. Let's present the best of ourselves to others, and seek to understand their wants and needs. Maybe if we work on our PR, we can rally more troops to the cause.

Why Use RCS for Backups?

Indrect response to Subversion for backup critics

A little while ago I wrote a series on using Subversion along with some shell scripts and other fun things to provide versioned backups of files. Subversion and other Revision Control Systems have some key features that make it very well suited for this task. After all, that's at the core of what RCS do.

RCS aren't always the best choice for this type of thing. Dedicated, versioned backup systems are much better at this core task. They are more specialized, have less cruft, and less chance of breaking. All that said, sometimes doing things the unnecessarily hard way has huge benefits. What are they?

Tagging

A common feature RCS offers that one wont find in backup offerings is tagging. Tagging allows you to mark a copy of your data, or subset there-of, in some meaningful way. This is a bit different than just marking a copy of your backup from some specific point in time. Tagging allows you to be selective about what gets marked, and marked items can span different time periods.

As an example, you can mark a selection of family photos in your backups as “Pictures for Grandma”. Exporting images with that tag will allow you to quickly put together a set, and store that set with very little overhead. When your aunt sees grandma's new photo-filled DVD and requests a copy, its a simple matter to re-export that set and send it off. Photo album software can do this, but what if you want to include video clips as well? How about the poetry your kid wrote in school? Tagging the files lets you mix and match data of any type.

Branching

Branching allows you to maintain multiple, parallel data sets. This feature will let you back up originals of files and work on them separately from your originals while maintaining backups of both. This is less useful in a backup scenario than tagging because you can always roll back files to specific versions with a regular backup setup. The real advantage comes into play when you want to have easy access to both the original and modified data sets at the same time.

One of the more obvious applications is photo editing. Photo sets may be thumbed through, with a limited number of shots making it through to the next stage, be it another elimination, basic manipulation, cataloging, or other actions. Utilizing branching, backups can be made of both the raw, original set as well as intermediate steps without wasting storage space on-disk and in backups. If you have 5 rounds of eliminations or selections, and want to preserve each round, you don't have to store your final selections 5 times.

Multiple Working Copies

One of the biggest benefits of using a RCS for backups is the ability to have multiple working copies. You can mirror your data on multiple systems, editing what you want, where you want. Changes at all locations are merged back into your backup data, and can be propagated back out to all working copies. Simply, you can have the same data anywhere, with changes at all locations backed up an an efficient manner.

In the very common case of having a work and a home computer, or having one desktop machine and one laptop, the convenience of multiple working copies becomes apparent immediately. So long as you keep up with minor housekeeping, the work you do on one computer is available on another. Should one machine be unavailable for any reason, you can pick up where you left off elsewhere.

Portable Backup Data

Many snapshot-preserving backup methods use file system-specific tricks or features, making it difficult if not impossible to move backup data to a new system or volume without loosing key information. RCS like Subversion can be quite happily picked up and moved to different locations, regardless of disk format or operating system. The ability to store this data easily on portable, rewrite able media like a USB drive is a very convenient feature as well, since you can have all of your versions readily accessible wherever you go.

Hopefully I have shed some light on the craze regarding using revision control systems for backing up data. In most situations, its more trouble than its worth. But for those of us with special needs, nothing else will do.

Wireless Stupidity

Impromptu

People still just don't get it. Most of the wireless networks pictured are protected by WPA Pre-Shared Key authentication, and the rest WEP. (WEP sucks, but its better than nothing. Or is it...) Obviously, one wasn't protected at all.

(Possibly) worse, a wireless network I am in no way responsible for, but am closely tied to, was serving up tasty, free Internet access to patrons. When asked if the network was encrypted, I said no, and briefly mentioned that we want it to be quick and easy for patrons to hop on. No fuss, no muss.

The response was an awkward, almost accusatory look. Apparently there is some problem with unencrypted networks. They are unsafe and give you STDs like the crazy, promiscuous girl even the desperate guys won't go near. The reality is, even an encrypted network isn't necessarily “safe”. If you want safe and private computerized communication, you want VPNs and SSL/TLS. After all, that's what they were invented for. The right tool for the right job makes all the difference.

Firewalls Suck

Impromptu

Firewalls suck. I hate them. Sure, a firewall can do you a lot of good in the hands of someone who knows what they are doing, but commonly, firewalls are the little Dutch boy's finger of the network security world. The problem is, no one else comes by to help.

Security charlatans sell firewalls like snake oil. They claim firewalls are mythical pieces of equipment or software that can protect the user from every computer security issue out there. So what happens? Your novice or otherwise uneducated user (or even worse, users who think they are well versed, but really aren't) runs a firewall and thinks they are safe. They don't patch their systems promptly because they believe the firewall will protect them. They don't password protect sensitive services, or encrypt sensitive information because they think the firewall will protect them. They generally do stupid things because there is no need to be concerned; the firewall will protect them.

Just think about immortality for a moment. If you could never be hurt or killed, would you wear your seatbelt?

All that said, firewalls are actually quite useful as temporary solutions to problems, or insurance against possible issues that may come up in the future. If a vulnerability is discovered in a service your computer runs, you can quickly disable access to this service at the network level if, for some reason, you cannot disable the service outright.

In the end, a properly configured, patched system does not need a firewall. If a firewall is your main defense, you've got other problems.

Perils of Rootkit Detection

i admit i haven't followed security as closely as i did when i just ran windows, but are there now rootkits that can hide themselves from computer b when it's on computer a? - ice60

Yes, but they do not explicitly. In fact, any rootkit hiding itself from other processes is also hiding itself from other systems on the network as a free side-effect.

If computer b is looking for a rootkit on computer a, it will ask for information about the system. Details about running processes, files, logs, and so forth may be queried. The problem is all of these requests must pass though the infected system, and therefor cannot be trusted. Involving a daemon in your quest to sniff out a rootkit only provides an additional opportunity for the rootkit to hide itself.

Let's look at one of my infamous metaphors as an example. We can equate a rootkit-infected system with a brainwashed individual.

• If you were brainwashed, you wouldn't necessarily know you were brainwashed. Any decent brainwasher would hide the brainwashing from you, perhaps with false memories.


• If I asked you if you were brainwashed, you wouldn't be able to tell me you are brainwashed. Quite simply, you wouldn't know.


• If I asked you indirect questions relating to your brainwashing, I *might* be able to infer you were brainwashed. There are certain inconsistencies that may come up in your responses. Perhaps you claim you were eating an anchovy pizza, but I know you hate anchovies.


• Since any response you give could be effected by the brainwashing, I still wouldn't be able to tell with any certainty that you had been brainwashed.

The only 100% reliable way to detect a rootkit, assuming you have a signature for it, is to bring the infected system off line, and look for it without relying on any part of the infected system. This could mean running the system from a separate, bootable disk, or transplanting the infected systems hard drive into another clean system.

For further reading, please see this article in Computerworld.

Password Strength

Impromptu

I came across a thread at UbuntuForums where someone asked how safe the password “s8fd7fg67fdg6” is. All these years, and we still have password education issues.

Excusing the fact that the guy just mashed on the keyboard producing something unlike what he would actually use when password-picking time came, this is a horrible password. It would take a while to crack, but its hard to remember and could easily be a lot stronger. 600+ times stronger, actually.

So, putting easy to remember aside, I will make a few assertions. If you want proof, do the math.

• The strength of a password is determined by the maximum number of attempts a brute force attack would require to find the password under ideal conditions. Ideal conditions in this case are knowing the maximum length of the password and the character set it was derived from. There is more to intelligent brute forcing (no, not a misnomer) than that, but that is beyond the scope of this piece.


• The maximum number of attempts required, as described above, can be calculated by adding one size of the character set s, and raising the result to the length of the password l. That would be (s+1)l, of course. The +1 is in there because I said “maximum” above. We have to account for the possibility that a password is shorter than the maximum by including an empty character in the character set.


• Working under the assumption that we know the maximum length of the password as well as its character set is quite reasonable, considering that the average password is 8 characters long*, the vast majority of passwords do not contain non-alphanumeric characters*, and minimum and maximum password lengths are very often imposed on users.

*These numbers came from a group of passwords acquired from MySpace users that fell for a phishing attack. You'd think that this represents the bottom rung of users, and you'd be right, but previous password auditing results have given similar numbers. Your average computer and Internet user isn't really that savvy. Juicy details at http://www.schneier.com/blog/archives/2006/12/realworld_passw.html.

If you start with a password up to 6 characters long, composed of numbers and lowercase letters, there are 2,565,726,409 possible combinations. Increasing the length by just one character yields 94,931,877,133 possible combinations. If you include capital letters in the character set instead of increasing the password length, you get 62,523,502,209 possibilities. Obviously, its better to increase your password length. But wait!

If you start with a 7 character password made up of lower case letters and numbers, and increase the length to 8 characters, you have 3,512,479,453,921 possibilities. If you change the character set instead, there are 3,938,980,639,167 possibilities. Woah!

So, what does this tell us? To over-simplify, if your password is going to be more than 7 characters long, character set matters more than password length.

Stealthy Sniffing

Will get caught if I try and sniff packets at my local coffee shop/library/school/office? - About 50 people

YES! At least, if you ask that question I'm sure you will. If you are sniffing packets, probing ports, or just plain doing something nefarious, no amount of black clothing will help you when someone sees “Johnnie Smith's Library” show up in their iTunes Window. This is probably the least of your worries if you are conspicuously huddled in the back corner pointing a cantenna at everyone with a laptop.

Before we get into this, its important to note why you should care. The legality of packet sniffing is arguable. That said, its fairly easy to bring someone down for sniffing. People having been charged with computer trespass for simply browsing the web over someone else's wireless connection. Even if you don't get convicted, do you have the money to fight a legal battle, especially after the FBI confiscates your laptop as evidence? I didn't think so. If you are doing anything even remotely shady, keep your footprint in the logs as small as possible. Yes, many low-end consumer routers do keep logs too!

From the network's point of view, its the little things that give you away. You will hardly ever see promiscuous mode detection deployed, but when some administrator sees someone that shouldn't be there screaming packets over the wire, its a pretty big hint. Most operating systems will periodically poll servers for updates. Since these are typically long transactions, you are going to be fairly easy to spot on the wire if this happens. You may think to reconfigure or not use programs like iTunes (but all 1337 hackers listen to techno MP3s while sniffing!) but you would be quite surprised at the amount of software that phones home for updates when you aren't looking.

Worse still are noisy operating environment components. Some common culprits include DHCP clients, various zero configuration protocols, network browsers, and ARP requests triggered by the above. Turn off or reconfigure as many of these services as possible if you are going sniffing. Depending on what operating system you are running, you can set things up so that you can switch to a now “location”, and have many of these services turned off at once.

When you think you have configured everything correctly, flip on your packet sniffer and watch your own traffic. See what is leaking. No packet should be beyond scrutiny. At least one popular file sharing client uses DNS requests to check for version updates. Its efficient use of the architecture of the Internet, but it makes leak checking just a little bit more difficult.

When are are set to go out in the world and do your deeds, remember the ninja. While ninja did wear all black when jumping from rooftop to rooftop at night, somehow I doubt that is what you will be doing. Good ninja hid in plain sight, indistinguishable from everyone else. If you are near a university, ratty jeans and t-shirts are acceptable. If you are in a ritzy neighborhood, where your khakis and a sweater. If you look like a computer geek and there is no LUG meeting in sight, you will be suspect.

The Skinny

I am very frequently asked questions about computers, electronics, information security, and other fun topics. I'm tired of answering the same questions over and over again. This archive of questions and answers will handle the repeats for me. Update: There are plenty questions I have anwsered and long since forgotten why. There will be no shortge of impromptu posts as a result.

If you have a question you would like to ask, e-mail me at frostycolddrink@gmail.com

As a side note, posts taged with scenario are discussions of scenarios and techniques from a high-level perspective. Posts taged with how to are more technical and include practical instruction you can use.