Shell Scripting Tips

Process Automation

We like Bash.
We like Cron.

The combination of Bash and Cron makes our department run more smoothly by giving my a huge degree of flexibility for automation. For instance:

• I've got a cron job that checks for FreeBSD updates on each of our FreeBSD servers. When an update is available, I get an email.
• We're paid on a bi-weekly basis. I have a weekly cron job that calls a script to remind me to turn in my time card on every 2nd run.
• We back up our VMWare ESXi machines each weekend. Cron calls a backup script that enumerates our VMs, then backs them up to our NFS server.
• We use a Hanoi Rotation for our backup tapes. My predecessor wrote a Heroku app to calculate the weekly tape rotation (whichtape.heroku.com). I have a cron job that gets the tape label from the web page and emails it to me with a reminder to change out the tape.

This is a very potent combination. We've run into a few gotchas--mostly foibles in Bash--that produce script behavior which is quite odd if you don't know what you're looking for.

Two things that will make you shell scripting life easier:

1. "Internal Field Separator:" When using "for i in" loops to parse lists, Bash uses a field separator value stored in the IFS variable. By default, this consists of space, tab, and newline. Frequently (almost always), I'm using a loop to parse strings containing spaces, and want the field separator to consist solely of a newline character. The behavior can be achieved by setting the IFS variable to the newline character at the beginning of your script:

IFS=$'\n'



2. while [ $TRIP -ne SOMESTRING ] ; do ; done  is great for delaying execution of a script until $TRIP matches a defined value. It's even better when your strings are validated:

while [ "$TRIP" -ne "SOMESTRING" ] ; do ; done

Happy Automating!

Poor Network Performance?

Check your patch cables.

We're a low-budget shop (see blog title, above), so we reuse patch cables. We've been reusing patch cables for about twelve years, and some have gotten a bit worn.

I installed a new server (reprovisioned an old server in a new location) yesterday, and was having difficulty initializing a Remote Desktop Connection to it. My initial thought was that the problem was some sort of bizzare hardware/software thing,* but it turned out to be much simpler: The patch cable in the telephone closet was faulty. It was adequate to get a DHCP lease and maintain a network presence, but just couldn't keep an RDP connection active. Replacing the patch cable to the switch fixed it, and its network connection now works as well as you'd expect it to.

If there's an odd performance issue on your network, the problem might be in a lower layer than you think.

* This server configuration is just weird.** It's got two Xeon 64-bit processors, but no VT extensions. It's got twin 15K SCSI drives, but the optical drive is CD-ROM. IT's got 2GB of RAM installed, but only the first 256MB are reported, etc. And it's got loud fans.
** I inherited it.

GMail as an MTA on Debian

Our company outsourced our email system to Google Apps when it became available a few years ago, so we don't maintain an MTA (mail transfer agent) inside our network. We do like some of our servers to have the ability to send email alerts, though, so we configure them on a case-by-case basis to send email via one of our (Google Apps) hosted accounts.

O/S: Debian Squeeze

Default MTA: exim4

For Debian machines, we configure exim4 to route email through Gmail's smtp server. This way, there is minimal change from the default system configuration, and we can forward any local mail to outside accounts via the traditional .forward files.

I found directions for this configuration on the Debian Wiki, here. They were pretty straightforward, and Debian Squeeze didn't need all the steps given.

Configure exim4

# dpkg-reconfigure exim4-config

• Choose mail sent by SMARTHOST; received via SMTP or fetchmail;
• Accept given hostname, if desired;
• Listen on loopback addresses (127.0.0.1, ::1);
• Don't accept mail for any other destinations (leave blank)
• Don't relay mail for other machines (leave blank)
• Machine handling outgoing mail for this host (smarthost): smtp.gmail.com::587
• Don't hide local mail name in outgoing mail
• Don't keep number of DNS-queries minimal
• Yes, split configuration into smaller files

Enter account credentials

# vi /etc/exim4/passwd.client

Enter the following line:

*.google.com:yourAccountName@domain.com:yourReallySecurePassword

Register the configuration update

# update-exim4.conf

Validate your account credentials

If you haven't logged in to the Google Apps email account interactively, you'll probably need to do so before the account credentials will be accepted. I was watching the log file (# tail -f /var/log/exim4/mainlog) and saw the following:

2011-08-24 15:37:21 1QwKCP-0001yy-2O plain authenticator failed H=gmail-smtp-msa.l.google.com [74.125.159.109] 535-5.7.1 Username and Password not accepted. Learn more at

535 5.7.1 http://mail.google.com/support/bin/answer.py?answer=14257 e4sm1192393anb.15

Which leads to information on "Capcha Unlocking" which is solved by logging in the the account interactively.

Set up forwards

I configure our servers to forward all mail sent to local root on to a department status mailbox (which is then monitored by the tech on duty). For Linux/BSD systems, this is dead simple:

# echo "itstatus@mycompany.com" > /root/.forward

Conclusion

And that's it. If everything works, any system status messages sent to the local root will be forwarded on to your outside address, where it can be retrieved, forwarded, ignored, etc.

With a functional MTA configured, you can also integrate email notifications into shell scripts and cron jobs:

# Send SMS notification when system boots

@reboot date | mail -s "`hostname` just booted" yourphonenumber@txt.att.net

For a simple notification system on a budget, this is hard to beat.