Perils of Remote Access

I thought since I've shared quite a bit of technical information in my past few posts that it was time for an admin's parable. I wrote a little about the perils of remote access servers earlier, but I didn't really go into detail about how 'normal' activities should be rethought in the environment of sole access being remote. Remote administration can be a wisening experience, unless you have some out-of-band administration tool (such as a DRAC card). With may hosts, you may only have the option to do a hard reboot or a reimaging (often at a fee).

In one case, CurlyFries was looking to add a static IP to the server to move critical services off the primary IP and domain. Most services, such as SSH, can listen on any address and can be restricted to certain addresses. It's handily secure to only have ports 80, 443, and 5222 open on a heavy-trafficked IP address.

It's fairly straightforward to add a static IP to a system. I would never dream of being timid about making such a change. At least not nearly as timid as when making a critical Apache or MySQL configuration change. Nonetheless, I managed to screw up with a typo in the /etc/network/interfaces file and got a serious error when restarting the networking. Thankfully, I hadn't lost terminal access, so I went back to look for the problem.

When I thought that I found it and fixed the problem, I restarted networking again. No joy. I was well aware of the implications of having a bad network interfaces file: no network interface access. So, I restored the backup configuration and restarted the service. At that point, my terminal died and I was then unable to establish any SSH connections to the server. The web server was down. Confounded, I cursed Debian for being so ridiculous and hoped that the configuration had taken effect. Within the hour, we had our TorrentFries rebooted, and to my relief the original configuration was working.

Had the networking stopped working when I tried to load the invalid configuration instead of when I loaded the original working configuration, we would have been toast. Not only would we have been forced to do a complete reinstall, but we would have lost quite a bit of data in the process due to not moving the rather infrequent non-automated backups off site regularly. *cough*

We barely escaped a disaster that would have spelled the second time in my capacity as an admin to need an OS reinstall on the remote server. The only time we've had to reinstall was when a certain curly type of administrator dropped the MySQL privileged users and my consequent mucking with things resulted in an inoperable system. The lesson to take away is threefold:

1. Backup offsite before making any changes that could effect the operation of the site.
2. Be mindful of what senarios could leave you with no server access. These generally include but are not limited to network configuration, SSH configuration, firewall configuration, and boot loader configurations.
3. Don't close an open terminal session unless you're sure you can get back in. After I make some changes, I open a second SSH session to ensure access before exiting out of my working session.

I hope you found that at least mildly entertaining enough for you to remember when it's your server you'll be losing access to. There's nothing quite as disappointing as the realization of failures easily avoided.

Backups

Okay, I can't stress this enough: an off-site backup is essential to the operation of any website, especially a tracker. Database backups are the most important, but backing up your files from time to time is necessary as well.

For example, if your thumb-fingered database maintenance *cough* results in a few thousand damaged or deleted rows (done that at least twice) or in the accidental deletion of your privileged user from the database (done that, too), you need a backup to fall back on. It's nice and convenient to have a local backup for that, but you also need to upload that backup to another server (or your home computer) in case the server runs into technical problems or you're suddenly taken down.

I was pretty lazy with backups while running on a shared web host, so I don't have a lot of wisdom to offer there. My host informed me that he ran nightly backups of all databases, so there was no need for me to maintain my own. (My PHP-based backup script had become pretty heavy for his server.) I think this is pretty normal behavior for shared hosts, although I don't know that all of them would give you access to those backups on demand. The point for them is protection from pissed-off customers in case they muck something up on their end or the server crashes or some such. So, when in doubt, back up.

The crux of the matter is that I'm lazy, and you probably are too. If it were left up to me, there'd be thorough backups for a few days, followed by a span of weeks or even months without backup. If you're lucky, that'll be all you need. How lucky do you feel, punk?

Therefore, you need to automate two things: the backup itself, and the process of uploading the backup to an external server. I'm not going to walk you through the setup process since that sort of info is pretty easy to find online, but I'll run over the general concept.

For the backup process, I set up cron jobs with two shell scripts: daily.sh and weekly.sh. When they're set to run is pretty self-evident. daily backs up the database (mysqldump -uroot -prootdatabasepassword --all-databases > backup.sql) and the uploaded metainfo (.torrent) files, while weekly backs up the entire web directory, all the PHP sources and so forth. weekly doesn't have to touch the database, since daily will also be running on that day. Each script then runs tar -czf backup.tar.gz files, with whatever files you intend to compress.

That's all it takes to ensure that you're safeguarded against data corruption. Naturally, there are other pitfalls that you can encounter. Hardware failure, hacking, inept administration, and of course the ever-present DMCA message—all can threaten your data and your backups. For that reason, it's a good idea to maintain off-site backups as well. You can download them manually, of course, but that's not going to be any easier to remember to do on a regular basis than to make the backups in the first place. Instead, I have my backup script upload to an external server via scp. I use that command instead of ftp because it's more secure and easier to automate.

However, scp does require that you have a UNIX box at the end. (Well, there are Windows ssh clients, so maybe there are scp servers as well. Stranger things have happened.) Since I run Mac OS X on my home computer, it's simple to automate the upload to there, or to any other server that I operate. If your shared host gives you rudimentary shell access, it's even okay to use that.

I'm sorry for being so lazy with the tutorial-writing in this one, but this blog entry is already getting rather lengthy and hopefully you can follow me anyway. I just have a few more points to make before I close.

Where one backup is good, archives of backups is better. If you set your script up to name its output files with the creation date, they won't overwrite one another and you'll have a nice collection in case you need to roll back a few days. I have plenty of hard drive space, so I still have backups from 2007. Who knows when you might need them?

Putting all this together, a simple shell script might look something like this (lines are double-spaced to avoid confusion in the case of line breaks):

#!/bin/sh

date=`date -I`

mysqldump --all-databases | gzip > /var/backup/backup-$date.sql.gz

scp /var/backup/backup-$date.sql.gz user@host:/dir/backup-$date.sql.gz

Okay, that's all for now, folks. I meant to throw in some horror stories about maintaining/not maintaining backups, but we're out of time. Maybe I'll get to that later on. Rest assured, such stories do exist, and don't forget: always maintain an off-site backup!

Retrospective 1: Conception

Since starting TorrentFries, I've had a number of people ask me how to start a successful tracker. The first question I ask is always "What sort of material do you want on the site?" Only one person gave me a response other than "everything". Incidentally, I'm now involved in some small way in the development of that other person's tracker. The two aren't directly related (I'm not a sword for hire), but they're not entirely unrelated either.

Listen, I'll let you in on a secret. There are tons of "everything" trackers out there. Come up with something unique to pull people in.

Personally, I believe in unconventional activism. In this case, deep-fried goods are too expensive, so I thought a BitTorrent tracker might get people talking about the problem. Since nobody else had one going, I set out to create one of my own.

The first step in the process was to find a host and a tracker. In this situation, personal connections are nice. I was lucky enough to know a guy that ran a shared web host, so I was able to get things running for $5/month, plus the cost of registering a domain name ($12/year). However, any shared host worth their salt should figure out that someone is running a tracker pretty quickly, so don't count on slipping that under the radar. Instead, you might consider a VPS, which allows you to rent part of a server, rather than having to pay for the whole thing. You should have plenty of control over your server, and get more leeway in terms of traffic. A basic VPS will cost you around $40/month.

However, be aware that there's no way you can buy security for any price you can afford at this point. The best you can hope for is a web host that will look the other way until they hear from a copyright holder. Most dedicated and VPS hosts I've encountered will do this; in fact, some hosts have told me this directly. What you can do is maintain daily off-site backups and have a backup host lined up in advance in case of the worst. We'll talk about contingency planning a bit later on.

Choosing the software involves a fair bit of research. I was pretty lazy on that point, and have been paying for it ever since. The only two trackers I checked out were TBdev and TorrentTrader Classic. TBdev is used by a lot of big-name private trackers such as Torrent-Damage, TorrentLeech, the late OiNK, LearnBits, BitMe, BlackCats, BrokenStones and so forth. However, the administration functions are almost non-existent, and the amount of customization available in TorrentTrader appealed to me. I didn't want to spend much time digging around in the code to get everything to work properly.

I've got a fair bit of experience with graphic and web design, so I took it as given that I'd build a new theme from the ground up. The product was in retrospect not that impressive, but it was certainly usable and unquestionably better than what I'd started with.

Finally, before going live, I asked a friend to take a look at the site. I expected him to spend perhaps 5 minutes and respond with a "looks good", but he ended up giving me a two-page essay including everything down to copy-editing in the FAQ. It was great help, and I was able to refine a lot of features that way.

Tomorrow: details on how to pull members in.