Guest Post: Forum Integration With SMFTracker

A friend of mine is working on a BitTorrent tracker mod for the Simple Machines Forum system. He was kind enough to provide us with a peek at his development process. He is basing his mod's announce on the basic framework I developed and published on Monday.

I've had to edit his post a bit, since he strayed into matters regarding our true identity pretty frequently, but the gist of the tracker mod development is there. Hey, what's life without a little mystery?

This is ChiliDog, co-collaborator of our ever-elusive stand-alone tracker script.  I was asked to make a guest post for the TorrentFries blog by CurlyFries.  Unfortunately, being the uninteresting person that I am, I’ve been racking my brain, trying to figure out what to write about.  I eventually came to the idea of writing about my current project: a SMF mod, with complete tracker integration.

Our main goal with the collaborative project was to successfully bridge any forum system’s [be it SMF, vBulletin, phpBB, etc.] database, however, that turned out to be a far-fetched dream [for now], due to its resource intensive code [blame CurlyFries ¬_¬].  But to tell you the truth, I couldn’t care less for any forum system other than SMF.  It’s powerful, feature-rich, and masterfully safe [not to mention completely free].  

I’ve been keeping an eye out for any tracker mods for SMF for the past 5 years or so, but not one has ever appeared.  There have been a few makeshift attempts at bridging a tracker system and SMF’s user table, but nothing worth using.  So after waiting so long, I finally set out to make one.

But, ‘Why a mod?  Why not just a bridge?’ you ask?  We’ll, I’ll tell you… It’s always been my philosophy to take advantage of what’s there, rather than to create something that does the same thing.  A bridge will still require all the things that SMF already does, in your tracker’s source.  CurlyFries is apparently the opposite of me.  He likes to create, rather than use what’s available [but more importantly, give people more options for the forum system].  That in itself is fine, and I’ll follow along for the main project, but I’ll be doing the SMFT mod in my spare time.

Other the announce.php, everything takes full advantage of SMF’s functions, including its permission system, themes, security, and even the language and modifications/upgrade system.  With three possible modes, open [ratio tracking is disabled], semi-private [non-members can still connect, just no ratio tracking and they might be restricted in site permissions], and private tracking [only members can connect], you will have a choice of what type of tracker you want to create.

There’s really nothing ground breaking about my project, but it should be the first SMF bittorrent tracker available; hence my project’s name: SMFTracker® [genius naming sense, eh?].  A lot of the front end source code, is the code CurlyFries and I created for our project, albeit most of it [if not all], was rewritten to work with SMF.

Our original announce [which I really liked] was ‘accidentally’ deleted by CurlyFries, but the [barebones] announce posted here is just as efficient [although, I prefer the switch system code he posted first].  I’m not ashamed to say I used that as a base to work with [credit was given where credit was due].

Although, I’m almost complete in coding SMFT [the remainder is just fluff and whatnot], it probably won’t be released for distribution this year.  I intend to beta test it for a while on my site and work out the kinks.  Once testing is completed, I’ll probably ask permission to add it to the SMF modification repository.  I say ‘ask permission’ because I’m not sure of SMF’s stance on a tracker mod.  They’re a legitimate corporation, and might decline my request, due to bittorrent being synonymous with copyright infringement [who’d have thought?].

On a side note, neither I, nor CurlyFries have abandoned our own project, and we do intend to [eventually] complete it.  However, SMFT is just something to keep me occupied until CurlyFries has extra free time on his hands for a time intensive project.

I definitely agree that SMF's mod system is ridiculously versatile, but with it comes unnecessary bloat and overhead. I personally prefer to build from scratch for maximum efficiency, and to me it makes more sense to build a forum around a tracker rather than a tracker around a forum.

I've subtly hinted that he may want to think about load testing the announces of various popular PHP/MySQL trackers, perhaps including his own and the one I wrote. It'd be interesting to see how they stack up.

The Tracker Demystified - Part 3: Reality

Okay, it's time to get back to the coding. On Tuesday, I developed a script that would play by the rules assuming everybody else did too. It was awesome: 29 lines, with a maximum of two database queries. And I wouldn't give it five minutes in the real world.

Why? Well, first, no input data was validated leaving the script open to SQL injection. Hopefully I don't need to say much more about the problems presented there, aside from that it would be a very bad thing were we not to address this issue. If you haven't heard of injection, go take a few minutes and read up on the implications. Since infohashes and peer ids are binary, any value containing 0x27 (apostrophe) would invalidate the SQL even without any malicious intent.

Secondly, even if we assume that the user is always benevolent, the script fails to anticipate that they may not be as true to the protocol as we are. The most glaring issue is that if the stopped message is never received, that particular row will remain in the database forever. This happens in the real world all the time; some poorly-designed clients will even neglect to send stopped when closed.

This time around, I'm just going to link the finished code and then go over some of the highlights. It'll be faster than just detailing every change in minute detail.

In my previous code, there wasn't even an error function. I've corrected that now, using the failure reason response provided by the protocol spec. I like my code to fail gracefully, since it cuts down on errors for users to come running to you with, but occasionally that's just not possible. Using failure reason with the message "Invalid port number", the user will see something like this in their client:

Obviously, you don't have a lot of room to work with, but you can get some pertinent information across. In this case, it is used to inform the user that their client is fucked. (Side note: this is an artificially-generated error. Transmission would never do anything bad.)

I've added a new field to the database, a timestamp field called last_connect. It has ON UPDATE CURRENT_TIMESTAMP set, so whenever we make a change to a row, the timestamp will be updated to match. This will be handy later on.

A bunch of validation has been added to the top of the script, which should be pretty self-explanatory. The BitTorrent spec describes the optional ip value as being "generally used for the origin if [the client] is on the same machine as the tracker", so the script makes an exception for 127.0.0.1, ignoring the value otherwise and simply going on the server's record.

I've shuffled the core about a bit so that they make more sense to the computer and less to a human. Since the maximum announce interval has been set to 30 minutes, if a row hasn't been updated in an hour, the script will assume it to be trash and delete it. In order to update the timestamp, a dummy query (SET uploader = $uploader) is run. Well, it's not entirely useless – this way, if the completed event is never received, the script won't even notice. In fact, it doesn't even check for completed anymore. Both it and empty are now treated in exactly the same way. 

If you rewind a bit to my previous entry on the subject, you'll see that empty was never dealt with. That's because it doesn't involve any change in state, which the tracker would need to record. Well, now the state is simply an assertion of its continued existence, which the tracker will need to keep track of.

It's not necessary to clear old entries with every connection; in fact, it is quite wasteful to do so. However, since this is a one-script application, we can't rely on a cron job or other external function to do our dirty work for us.

This is where I will leave you. There are a ton of features I could add, including ratio tracking, IP bans, client identification, protocol extensions (some trackers include swarm info on announce), defending against browser attacks, and so forth. However, I promised to deliver a script that functioned as a basic barebones open tracker in the real world, and that's exactly what I did. I hope you'll be able to use it as a framework to work from in writing your own code.

On the subject of working as a framework, you should now see why I left the script half-finished last time. It's a lot easier to understand the early version than the completed version, and hopefully helped a bit to clarify the basic structure we were working with. I'm sorry for any bugs or security holes that may have crept in here. This is a last-minute job as always, and it's entirely possible that I'm being sloppy.

That's all for now, folks.

Building a Foundation of Knowledge

Since CurlyFries was tied up yesterday, he's unable to continue his Tracker Demystified series today. Instead, I bring you some learning advice and an ambitious reading list from a true bookworm, or book-collector — whichever you prefer. As an admin, the most important thing you can have is a friend that knows what they're talking about when it comes to Linux. Second best is a book to tell you how to do it and give you more detail. Since I'm not a Linux guru by trade, having plenty of people and books around got me out of quite a few tight spots. I wouldn't have normally been able to wiggle out them on my own just Googling. Sometimes even the smallest pieces of advice are worthwhile.

For example, we were trying to do some optimization on the server because TorrentFries' forums were slowing down. When I asked a friend what he knew about optimization, he asked me "How does htop look?" Quite frankly, I had never heard of it, but it's proven to be one of the most useful tools, not to mention the most used. (htop is a more advanced visual extension to top. You'll never use top again.)

Anyways, I've latched onto over half a dozen such people to draw from their Linux expertise now and again, and also so I don't have to go back to the same person time and time again. Networking is more than just TCP/IP. ;)

I'll start you off with my recommendations from the books I've read. I'm not going to claim that they're the best, just that they've provided me with some good background.

Unix Administration

How Linux Works
UNIX, Third Ed.
The Book of Postfix (email - just a casual read)

Development

PHP 6 and MySQL 5 for Dynamic Web Sites: Visual QuickPro Guide
PHP Solutions: Dynamic Web Design Made Easy
SQL Power!: The Comprehensive Guide

Web Design

Web Designer's Reference
Web Standards Solutions: The Markup and Style Handbook
CSS Mastery: Advanced Web Standards Solutions
Pro CSS Techniques

I've read quite a few others, but for the sake of brevity, I'll leave them off. Here are a couple other books that I might also suggest reading even though I haven't read them yet, or only have read excerpts out of.

Apache Cookbook: Solutions and Examples for Apache Administrators
High Performance MySQL: Optimization, Backups, Replication, and More
Apache: The Definitive Guide
Running Linux
Hardening Linux

There you have it. That's well over a year's worth of reading by my standard, but it's a solid foundation of knowledge. Hopefully tomorrow will bear more tracker information.

The Tracker Demystified – Part 2: Structure

Now it's time to get started on the coding itself. For debugging purposes, we're going to need a lot more than the single-line tracker status field in normal BitTorrent clients. Happily, since the connectivity is simply via HTTP GET, you could just type the announce URL into your browser's address bar to test the effect of various values. In fact, this is the way I did my testing last time I took it upon myself to write a tracker.

This time, I'm going a little more high class. I've made a simple form with fields for each of the variables (info_hash, peer_id and so forth) that is submitted to announce.php using GET. It's much easier to play around with different values this way. Since forms are pretty simple and not the subject of discussion here, I won't go into any more depth here.

First, even before connecting to the database, I'll load the passed variables and perform a minimum of validation. Like yesterday, I'll be referring to the protocol spec to determine the expected values and ranges of particular variables. I don't like throwing a ton of errors at my user, so I tend to design for graceful failure. For example, if event is anything other than started, completed, stopped, or empty, it will simply default to empty.

Speaking of event, it is going to form the basis of our code structure. The heart of the script consists of a series of conditionals based on the four possible variables. Actually, since empty merely denotes that the client is requesting more peers, we only need the three that require updates on the tracker's end: started, completed, and stopped. These three events map cleanly to three MySQL statements: INSERT, UPDATE, and DELETE, respectively. When a torrent is started, the peer needs a new row in the database, which should be updated when the download has finished and removed when the connection is closed.

Now, stopped only takes input – there's no point in making output when nobody is going to use it. For everything else, we'll use BitTorrent's bencode protocol (detailed in the spec) to output the proper response. Again, I encourage you to go back and peruse the spec. Even if you don't muck around in the backend much, it's nice to be able to pop the hood of any metainfo (.torrent) file you may come across and get a general idea of what's going on in there. The bencoded data may look intimidating, but it's perfectly readable if you take a few minutes to understand how it works.

So, back to the announce. The proper response I mentioned above is a dictionary consisting of two keys/value pairs: interval to indicate the number of seconds until the tracker suggests that you scrape again, and the peers to provide a list of the peer id, ip, and port of each provided peer. There's one bit in the official specification that I feel the need to quote for all the private tracker elitists out there:

Note that downloaders may rerequest on nonscheduled times if an event happens or they need more peers.

Announcing more often than this is perfectly acceptable practice. I do hate to hold it up for ridicule, but BitMe has ironically banned the Mainline (BitTorrent) client originally developed by Bram Cohen himself, developer of the protocol, because it "does not honor the protocol requirements of private trackers". I'm not going to knock on BitMe too much because they're more open than many private trackers in that regard, and they're the only one that I know of that lists banned clients and justifications for banning. However, allow me to take this opportunity to say... what the fuck?

That's a little off-topic. Anyway, now we just have to pull the data from the database. In practice, clients can and do indicate the number of peers they're looking for with the numwant variable, but that's not part of the spec, so I'm ignoring it for now. Instead, I've specified 25 as an arbitrary maximum, and used ORDER BY RAND() in my MySQL query to ensure that all peers get a statistically equal showing no matter where they sit in the database.

Now that we've got the pertinent data, it has to be output, so we'll bencode it according to the structure laid out in the protocol, and shove it out the door. I might talk about writing full functions to read and write bencoded data at some point, but it's a lot more work. The implementation in the announce is simple because we only have to account for the variation between different volumes of data, which is nevertheless being output in a rigid format. Super easy.

Now, while I may not have done a particularly satisfying of describing the last bit of the announce, it is in fact done. The finished script is only 29 lines, including whitespace. I had it heavily commented, but many of the comments repeated what I'm saying here, and I want to emphasize how simple an announce can really be, so I've left it blank and hopefully the minimalism will speak for itself.

At the moment, I'm running three peers: Transmission, Azureus, and my own web browser masquerading as a client. Transmission has successfully seeded the file to Azureus, and both are trying to connect to my web browser to share with it too. Obviously, since it isn't actually a torrent client at all, that's not going to happen.

Now, I see the seasoned coders cringing already, so I'll repeat this once again: this is a barebones demo script. It is designed to provide a framework for you to work off of in forming a more complex announce. The idea is to illustrate the way the protocol works, and the way the tracker interfaces with the protocol. As it stands now, the script doesn't even sanitize input, so it is vulnerable to SQL injection as well as just about every other damn thing.

Tomorrow, I hope to elaborate a bit on these issues, refining this code to create a perfectly usable announce. Ratio tracking is still outside the scope of the project, but we can certainly tighten things up and maybe lay the necessary groundwork for you to go on and add ratios and such frivolity. I say "I hope" because I'm burdened with a lot of work tomorrow and may not get time for a lot of coding. I'll make sure you get an article, and hopefully part 3 of this series, but no promises.

And yes, you're welcome to take the code I wrote above and use it without attribution for whatever purposes you want. However, if you're going to do that, I really suggest that you fix the holes first, or wait until I have a chance to do it for you.

The Tracker Demystified – Part 1: Building the Database

You may have noticed a post on this subject a while back that was unintentionally released before its time. Well, if you have already read that one, re-read this anyway. It's finished, for one thing.

Anyway, this week is going to be devoted to building a simple tracker from scratch. This tracker will do no more than accept and share IP addresses, with no front-end for uploading and downloading metainfo (.torrent) files. However, if you've worked with PHP much, you can probably already work out how to do file uploads and downloads and pretty or at least usable interfaces. This is the big obstacle, and also the bit that has the potential to be interesting for the non-coder, if I can manage to write clearly enough to keep them engaged and slightly comprehending.

Hopefully, over the course of the week, I'll dispel any notions of BitTorrent as a complex or incomprehensible protocol. Once you get your head around it, it's actually quite easy to understand and use. The peer-to-peer bit is a little less straightforward, but happily, we don't have to deal with that. We're writing a tracker, not a client.

My resource in all of this is going to be the official BitTorrent spec, which I know almost by heart. I'll be referring to the spec from time to time, so read it over and my posts might start to make some kind of sense. My MySQL abilities are a little more touch-and-go, so optimization may not be as fantastic as it could be and I'd welcome any constructive criticism on that front.

On that note, we'll be starting today with outlining the basic structure of the announce through the creation of a database table for the peers. This is the only table we'll need, which is handy. Basically, we need to store the pertinent bits of the data that's received by the announce, and enough to provide a coherent response. There are eight variables passed by the client to the tracker: info_hash, peer_id, ip, port, uploaded, downloaded, left, and event. Official or unofficial extensions may add extra values, but we're writing a barebones tracker, so we can safely ignore them. The spec does a perfectly good job of clearly outlining the purpose of each of these variables, so I won't repeat it here.

Now, all of the provided variables are pretty important for various things, but again, this isn't a full-featured tracker, so we'll ignore some of them. We'll save info_hash so we can connect peers on the same torrent to one another, peer_id so we can distinguish one peer from another, and ip and port so we can share the user's address with others on request. As well, while we have no need to track ratios here, we do need to know who is seeding and who is leeching so that we don't waste time sharing seeds' IPs with other seeds. We'll call this variable uploader, and it'll simply be a bit assigned based on the test of left == 0.

At this point, our database looks like this:

If you're familiar with phpMyAdmin, you'll see that I've set a primary index on id, which is standard practice. I've also set an index on info_hash, since we'll be running a lot of queries for it.

Now, at this point, before the more knowledgeable members start tearing holes in my post, I want to point out that I'm working with an ideal model here. No information is lost, all clients follow the protocol to the letter (particularly in always cleanly closing connections), and there are no clients that spoof information for personal gain. Obviously, none of these are true, but since I'm aiming to explain an implementation of the BitTorrent protocol, I'm going to work with these assumptions for the time being, just like how friction is often ignored in introductory physics courses.

Leaving a Host

There comes a time when you need to cover your tracks in a hurry. Since you often can't stick a boot CD in a dedicated host to do a DOD hard drive wipe like you should, you can settle for a few other options. It's nice to feel like you haven't left any incriminating footprints behind.

I find that the best tool for the job is secure remove. The command srm will allow you to do a 38-pass deletion wipe that's good enough for most purposes. Note that it's not impossible to recover the data deleted in this method, just beyond the reach of any small to medium sized organization.

Install secure remove:

$ sudo apt-get install secure-delete

Now, you could use this command to bash script a quick exit, but without forking off your processes, a quicker way would be to run each part of your secure removal separately. This works very well if you've got multiple cores or processors. You can use process management tools such as screen or background processes to run multiple srm commands at one time.

The three basic parts of your server you should consider deleting would be the MySQL databases, Apache www folder, and your logs. If you've installed any other software, be sure to check those out as well.

MySQL
sudo srm -rv /var/lib/mysql/

Web root
sudo srm -rv /var/www/

Logs
sudo srm -rv /var/log/*

You'll have to use your judgment on the rest of the system on an individual basis. Don't forget to delete any backups you've made. The last thing you can do is clear out your private files before you exit.

sudo srm -r ~/.*


That's all. SRM uses a lot of processor and takes some time on larger directories and files, so be sure to start your deletes in a relatively parallel fashion if you've got to high-tail it out.

Expect to see the series on "The Tracker Demystified" next week. Hopefully, if I write loud enough, that might happen. ;)

CurlyFries adds: Only stuff that you delete with srm will be securely removed. For this reason, do not drop your MySQL databases prior to hitting /var/lib/mysql with your srm stick. If you do so, MySQL will (insecurely) delete your database files, and running srm afterwards will do you no good.

Oh yeah, and hush up. I'm getting to it.

Perils of Remote Access

I thought since I've shared quite a bit of technical information in my past few posts that it was time for an admin's parable. I wrote a little about the perils of remote access servers earlier, but I didn't really go into detail about how 'normal' activities should be rethought in the environment of sole access being remote. Remote administration can be a wisening experience, unless you have some out-of-band administration tool (such as a DRAC card). With may hosts, you may only have the option to do a hard reboot or a reimaging (often at a fee).

In one case, CurlyFries was looking to add a static IP to the server to move critical services off the primary IP and domain. Most services, such as SSH, can listen on any address and can be restricted to certain addresses. It's handily secure to only have ports 80, 443, and 5222 open on a heavy-trafficked IP address.

It's fairly straightforward to add a static IP to a system. I would never dream of being timid about making such a change. At least not nearly as timid as when making a critical Apache or MySQL configuration change. Nonetheless, I managed to screw up with a typo in the /etc/network/interfaces file and got a serious error when restarting the networking. Thankfully, I hadn't lost terminal access, so I went back to look for the problem.

When I thought that I found it and fixed the problem, I restarted networking again. No joy. I was well aware of the implications of having a bad network interfaces file: no network interface access. So, I restored the backup configuration and restarted the service. At that point, my terminal died and I was then unable to establish any SSH connections to the server. The web server was down. Confounded, I cursed Debian for being so ridiculous and hoped that the configuration had taken effect. Within the hour, we had our TorrentFries rebooted, and to my relief the original configuration was working.

Had the networking stopped working when I tried to load the invalid configuration instead of when I loaded the original working configuration, we would have been toast. Not only would we have been forced to do a complete reinstall, but we would have lost quite a bit of data in the process due to not moving the rather infrequent non-automated backups off site regularly. *cough*

We barely escaped a disaster that would have spelled the second time in my capacity as an admin to need an OS reinstall on the remote server. The only time we've had to reinstall was when a certain curly type of administrator dropped the MySQL privileged users and my consequent mucking with things resulted in an inoperable system. The lesson to take away is threefold:

1. Backup offsite before making any changes that could effect the operation of the site.
2. Be mindful of what senarios could leave you with no server access. These generally include but are not limited to network configuration, SSH configuration, firewall configuration, and boot loader configurations.
3. Don't close an open terminal session unless you're sure you can get back in. After I make some changes, I open a second SSH session to ensure access before exiting out of my working session.

I hope you found that at least mildly entertaining enough for you to remember when it's your server you'll be losing access to. There's nothing quite as disappointing as the realization of failures easily avoided.

Protocol

Okay, this is going to be a quick post because OnionRings and I both have a ton of work to do and Ketchup seems to be maintaining our collective laziness quota.

I'm planning to offer a detailed examination of what exactly goes into the technical end of a BitTorrent tracker by offering explanations as I write my own, but that's well beyond my time allotment for the evening. Instead, I'll write a simple summary of the protocol.

The peer-to-peer aspect is not important for our purposes. Unless you're planning on writing a client, which is well outside the scope of this site, you don't even need to think about it. I know I don't. I really haven't taken the time to study it in much detail, aside from understanding what's going on in general terms.

Happily, the peer-to-tracker protocol is very simple to understand. The tracker is inherently passive, never initiating connections itself. Instead, clients connect to the tracker via HTTP. They provide the infohash (unique identifier) for the torrent they're trying to download, their current port, and a bunch of other information indicating the state of the client. The tracker responds with a simple list of IP addresses that are also downloading the same file. It's up to the client to initiate connections with the provided peers. Since the protocol is peer-to-peer, any client can initiate the connection, assuming that the other computer is properly configured to accept incoming connections.

That's all a tracker has to do: log the IP addresses that connect, and respond with a list of suggestions for possible peers. The process is fast, light on resources, and very easy to code. You can develop a tracker just like any website, using just about any language you want: PHP, ASP, Ruby, whatever. The biggest trackers run on C or other such languages, since it's fast and lightweight beyond the dreams of PHP. However, for your purposes, web languages are all you need.

That said, I'll pick up on the subject later on when I actually find the time to get going on development. Like I say, things have been nuts of late.

Back to the Basics

I realize that our focus has of late moved from tracker-specific discussion to more-general stuff that applies to any site or server and can be found in a bazillion places online. Since this blog is unique because it does chronicle the operation of a BitTorrent tracker, I'm going to get back to the meat of the thing for a bit. OnionRings plans to continue with his Linux series, but is sadly indisposed this evening.

In some senses I will cover some of the same material as our second-ever blog entry, but with a different focus. While that entry gave a quick overview of why and how to set up a tracker, I'm going to look a little more closely at the technical and logistical aspects of bringing a tracker online.

First off, what do you personally need to bring to the table? (You're probably the only one at the metaphorical table at this point, but the bringing should commence now.) Beyond anything else, you need to understand the BitTorrent protocol, and understand your tracker of choice. We'll talk a bit about choosing trackers shortly. If you plan on running a PHP/MySQL tracker, you'd better have at least a working knowledge of PHP and MySQL. From the get-go, you'll need to be able to peer into the inner workings of your site and see what makes it tick. You don't just need an understanding of the protocol, but of your own tracker. Trust me, admin panels are nice, but some things just require you to get down and dirty. However complete your chosen script may look at first glance, there will quickly come a time when you or your users bemoan the lack or poor design of some feature or another, and you will be forced to go in and remedy the situation. As the tracker grows and moves to its first dedicated server, it will also become important to know your way around Linux to some extent.

Secondly, you need a hook to pull people in. The most obvious aspect is to provide something that others can't. A unique idea is neat, but it's entirely possible to carve out a niche in a "market" that is well-developed but not saturated. For example, if you have terabytes of obscure movies that you have spent years collecting, starting a movie tracker to share these can be an excellent starting point. Of course, you will need to keep all the uploaded torrents active, even though you will be getting very few peers at first. For this purpose, you may want to consider renting a seedbox. The added cost may be hard for a brand new tracker admin to swallow, but your ability to saturate the connections of your first members will do a lot to encourage them to stick around and maybe even contribute some stuff of their own. As well, a unique design may sound frivolous, but it will lend your site credibility and a sense of longevity. I've covered some methods of early promotion in the post I mentioned above, so I won't rehash them here.

I hate to be defeatist, but if you can't meet these requirements, you should think twice about starting a tracker. Of course, the lovely thing about being a human is the capacity to learn, but you should get going on that learning well before you even consider getting into tracker territory. While I'm trying to demystify the role of the tracker administrator, it's certainly a job that not everyone has the skill and disposition to fill with any great success.

Moving on to more technical requirements, you will also need to choose a tracker. That's a given. The three most popular trackers under active development are TBdev, TorrentTrader, and the new Project Gazelle, although there are a ton of other options out there, so you shouldn't feel constrained to choosing one of these three. Shop around a bit. Detailed comparison of these and other trackers is well outside the scope of this entry, and perhaps even of this blog, but this should point you in the right direction. At the end of the day, it's up to you to select the tracker that best suits your particular tastes and needs. As a side note: for the love of God, don't use TorrentTrader Classic.

So, why not just boot Azureus or µTorrent or one of these handy ubiquitous little torrent clients that happen to include the ability to operate as standalone trackers? Hopefully this isn't a question you were asking yourself, but I'll answer it anyway. First, this isn't the purpose they were designed for. Just because they can run a tracker doesn't mean they should. They're not optimized for the purpose, they don't include features that are necessary for the smooth operation of a tracker (they're inherently open to anyone that wants to use them), and they include a ton of features and bloat that will just bog down your server. What's more, part of the sort of tracker we're discussing here is the index, or frontend. This is the bit where torrents are uploaded, downloaded, where users interact with one another, all that sort of stuff. The tracker itself is ridiculously simple in operation, as we will see in my eventual piece-by-piece breakdown of what it takes to build one from scratch. The important business is the complex frontend, and BitTorrent clients' wannabe trackers just can't provide this.

So, once you've selected the script that best fits your needs, it's time to choose a host. I've had some requests to list a number of torrent-friendly hosts, but that's not something I'm going to do for a number of reasons. The most important reason here is that I don't want to endorse a host that turns out to be unsafe, or to lull readers into a false sense of security. You may have noticed that even (especially) the biggest trackers tend to move around a lot. The world of web hosts is intrinsically volatile, so to proclaim that one host is perfectly safe, even if presently true, is to ignore the fact that this may not continue to be the case in the future. LeaseWeb was once a safe haven for trackers, but after losing a lengthy court battle on behalf of Demonoid and other trackers, all the will to fight has gone out of them. Nowadays, if a copyright holder says "boo", they'll turn around and shut you down.

So simply accept that no place is perfectly safe, and operate under the assumption that you could be taken down tomorrow – you could be. A good strategy to find places that are safer is to run whois lookups on established trackers' IP addresses. Presumably, if they are able to operate on a particular host, that host must be somewhat resistant to legal threats. The only problem is that many larger trackers own their own servers which they operate via colocation in the host's datacenter, while you are likely unable to afford more than a rented dedicated server (an important distinction to make). Not all hosts supply both options, and not all do so for reasonable rates. In addition to colocated and dedicated servers, I've talked about shared and virtual hosting in that previous article, so I'm not going to rehash that here.

This should be all you need to get you off the ground. Again, I'm not going into exacting detail on some of the technical points because I'm assuming that you will be able to handle the fiddly bits of your own accord. If you can't, like I mentioned, you probably shouldn't be running a tracker.

If there are any points on the subject (or any other, really) that you would like us to cover, leave a comment on this post and we'll do our best to touch on them in future posts.

Site Administration Tools

Monday I wrote a technical piece about installing LAMP so today I'll talk a little about the type of administrative tools we use on the site. I've always been a big fan of using the tools UNIX gave us. That means strict command line pleasure. It means things are clean on the system and no extra resources are used or security holes opened. But not everybody agrees. Even I find it nice to use a GUI once and awhile so as not to get bogged down with the mechanics of the command line.

MySQL seems to be the toughest to administer with the tools UNIX gurus gave us. Writing SQL statements isn't bad if you're familiar with SQL, but you'll likely feel better with PHPMyAdmin or MySQL Administrator (MySQL Query Browser). The difference is that PHPMyAdmin is web-based and MySQL Administrator is a remote client. To me, the difference is that there's no garbage on the server when you use a remote client to connect to to your server like there is with installing PMA.

Consider the security implications as well. When installing PMA, you open up an avenue for attack via the web. When using MySQL Admin allowing remote connections, you're doing even worse. Therefore, the safest and most efficient option is to tunnel your MySQL Admin connection over SSH to get a localhost connection using a remote tool. I think this method applies for other remote tools as well. Ketchup likes using MySQL-Front. Using a remote tool allows room for preference.

In other areas, it can be handy to have a GUI as well. If you have many servers running on your box (such as mail, Jabber, and Apache/MySQL) it can be handy to increase your proficiency and ability to edit obscure configuration files with greater ease by installing a management tool. I've used Webmin before, and I have mixed feelings. It gives very good control over the multitude of configurations on a Linux box, but I always see it as another security vulnerability.

That's all for now, unfortunately. Next time I post, we'll discuss more about LAMP configurations.

Linux Part 2: Installing LAMP

I'm going to resume from Linux Part 1: Installing Configuring Sudoers and OpenSSH with part 2: Installing LAMP. In the world of Ubuntu, virtually no one installs LAMP from a command line on Ubuntu Server. Ubuntu provides the option to automatically install LAMP during OS setup, so most of the time, it's done that way. Purchasing a dedicated server often doesn't give you that option, however. Yes, many tutorials cover installing LAMP, and they are just as good, but note that they're installing on Ubuntu Desktop, sometimes assuming you have a GUI. Thankfully, installing LAMP on Ubuntu from the command-line isn't hard at all.

Let's start by running the best command ever, apt-get. This will get all the packages we need.

sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 mysql-server libapache2-mod-auth-mysql

Alternatively, you can try

sudo tasksel install lamp-server

Apt will download the packages and install. Then, you'll see this as MySQL is being configured:



You'll have to confirm your password, and then you're done. Restart Apache.

sudo /etc/init.d/apache2 restart

Let's make sure everything went okay. Browse to your IP address. You should see an "It Works!" page. This means Apache is running. Let's check PHP. Run

sudo nano /var/www/test.php

Write out this single line to the file:

<?php phpinfo(); ?>

Browse to your addresss/test.php and you should see lots of information about the current state of PHP.

Securing MySQL

That installation was so easy, we'll do a little security cleanup too.

Log into MySQL:

mysql -uroot -p

Your prompt will change to mysql>

Create a user you'll be using to administrate the databases. By default, MySQL is secure in the fact that it will only allow root to login locally. That's excellent until you realize that MySQL can be a pain to administer locally. I don't mind too much, but some people like to install PHPMyAdmin. This can open you up to potential remote access attacks on the root user's password. Therefore, if you're going to install PMA, we'll get rid of the root user to make it more difficult to enumerate users. Let's start by creating a MySQL account, Zeus.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'localhost'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;

You should consider also adding the privileges to this user @ 127.0.0.1
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'127.0.0.1'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;

At this point, you should create a user that will be the one accessing the database. Using your "God" user to access MySQL from your web application is bad karma and invites security risks. Google 'create mysql user' for the syntax on creating a user with less privileges.

Feel free to be creative with your names...
Flush the privileges and exit.

mysql> FLUSH PRIVILEGES;
mysql> quit

DO NOT ATTEMPT TO FORGET THIS PASSWORD! It's a pain to restore a MySQL root password, and even more of a pain when the root user doesn't exist, (some say impossible) so take it from me-- remember your privileged user's password for cripes' sake.

Login and delete the root and anonymous user once zeus was created.

mysql -uzeus -p

mysql> DELETE FROM mysql.user WHERE User = ' ';
mysql> DELETE FROM mysql.user WHERE User = 'root';
mysql> FLUSH PRIVILEGES;

Check out your work:

mysql> SELECT User, Host FROM mysql.user;

Don't drop the debian-sys-maint account.

Apache and MySQL Control

Here's how you can restart Apache and MySQL without rebooting your server if you have changed a configuration file.

sudo /etc/init.d/apache2 restart
sudo /etc/init.d/mysql restart

Replace 'restart' with 'stop' or 'start' if needed.

Thanks for reading again, next we'll show you a bit about the different administrative tools available for Linux.

Backups

Okay, I can't stress this enough: an off-site backup is essential to the operation of any website, especially a tracker. Database backups are the most important, but backing up your files from time to time is necessary as well.

For example, if your thumb-fingered database maintenance *cough* results in a few thousand damaged or deleted rows (done that at least twice) or in the accidental deletion of your privileged user from the database (done that, too), you need a backup to fall back on. It's nice and convenient to have a local backup for that, but you also need to upload that backup to another server (or your home computer) in case the server runs into technical problems or you're suddenly taken down.

I was pretty lazy with backups while running on a shared web host, so I don't have a lot of wisdom to offer there. My host informed me that he ran nightly backups of all databases, so there was no need for me to maintain my own. (My PHP-based backup script had become pretty heavy for his server.) I think this is pretty normal behavior for shared hosts, although I don't know that all of them would give you access to those backups on demand. The point for them is protection from pissed-off customers in case they muck something up on their end or the server crashes or some such. So, when in doubt, back up.

The crux of the matter is that I'm lazy, and you probably are too. If it were left up to me, there'd be thorough backups for a few days, followed by a span of weeks or even months without backup. If you're lucky, that'll be all you need. How lucky do you feel, punk?

Therefore, you need to automate two things: the backup itself, and the process of uploading the backup to an external server. I'm not going to walk you through the setup process since that sort of info is pretty easy to find online, but I'll run over the general concept.

For the backup process, I set up cron jobs with two shell scripts: daily.sh and weekly.sh. When they're set to run is pretty self-evident. daily backs up the database (mysqldump -uroot -prootdatabasepassword --all-databases > backup.sql) and the uploaded metainfo (.torrent) files, while weekly backs up the entire web directory, all the PHP sources and so forth. weekly doesn't have to touch the database, since daily will also be running on that day. Each script then runs tar -czf backup.tar.gz files, with whatever files you intend to compress.

That's all it takes to ensure that you're safeguarded against data corruption. Naturally, there are other pitfalls that you can encounter. Hardware failure, hacking, inept administration, and of course the ever-present DMCA message—all can threaten your data and your backups. For that reason, it's a good idea to maintain off-site backups as well. You can download them manually, of course, but that's not going to be any easier to remember to do on a regular basis than to make the backups in the first place. Instead, I have my backup script upload to an external server via scp. I use that command instead of ftp because it's more secure and easier to automate.

However, scp does require that you have a UNIX box at the end. (Well, there are Windows ssh clients, so maybe there are scp servers as well. Stranger things have happened.) Since I run Mac OS X on my home computer, it's simple to automate the upload to there, or to any other server that I operate. If your shared host gives you rudimentary shell access, it's even okay to use that.

I'm sorry for being so lazy with the tutorial-writing in this one, but this blog entry is already getting rather lengthy and hopefully you can follow me anyway. I just have a few more points to make before I close.

Where one backup is good, archives of backups is better. If you set your script up to name its output files with the creation date, they won't overwrite one another and you'll have a nice collection in case you need to roll back a few days. I have plenty of hard drive space, so I still have backups from 2007. Who knows when you might need them?

Putting all this together, a simple shell script might look something like this (lines are double-spaced to avoid confusion in the case of line breaks):

#!/bin/sh

date=`date -I`

mysqldump --all-databases | gzip > /var/backup/backup-$date.sql.gz

scp /var/backup/backup-$date.sql.gz user@host:/dir/backup-$date.sql.gz

Okay, that's all for now, folks. I meant to throw in some horror stories about maintaining/not maintaining backups, but we're out of time. Maybe I'll get to that later on. Rest assured, such stories do exist, and don't forget: always maintain an off-site backup!