The Tracker Demystified – Part 1: Building the Database

You may have noticed a post on this subject a while back that was unintentionally released before its time. Well, if you have already read that one, re-read this anyway. It's finished, for one thing.

Anyway, this week is going to be devoted to building a simple tracker from scratch. This tracker will do no more than accept and share IP addresses, with no front-end for uploading and downloading metainfo (.torrent) files. However, if you've worked with PHP much, you can probably already work out how to do file uploads and downloads and pretty or at least usable interfaces. This is the big obstacle, and also the bit that has the potential to be interesting for the non-coder, if I can manage to write clearly enough to keep them engaged and slightly comprehending.

Hopefully, over the course of the week, I'll dispel any notions of BitTorrent as a complex or incomprehensible protocol. Once you get your head around it, it's actually quite easy to understand and use. The peer-to-peer bit is a little less straightforward, but happily, we don't have to deal with that. We're writing a tracker, not a client.

My resource in all of this is going to be the official BitTorrent spec, which I know almost by heart. I'll be referring to the spec from time to time, so read it over and my posts might start to make some kind of sense. My MySQL abilities are a little more touch-and-go, so optimization may not be as fantastic as it could be and I'd welcome any constructive criticism on that front.

On that note, we'll be starting today with outlining the basic structure of the announce through the creation of a database table for the peers. This is the only table we'll need, which is handy. Basically, we need to store the pertinent bits of the data that's received by the announce, and enough to provide a coherent response. There are eight variables passed by the client to the tracker: info_hash, peer_id, ip, port, uploaded, downloaded, left, and event. Official or unofficial extensions may add extra values, but we're writing a barebones tracker, so we can safely ignore them. The spec does a perfectly good job of clearly outlining the purpose of each of these variables, so I won't repeat it here.

Now, all of the provided variables are pretty important for various things, but again, this isn't a full-featured tracker, so we'll ignore some of them. We'll save info_hash so we can connect peers on the same torrent to one another, peer_id so we can distinguish one peer from another, and ip and port so we can share the user's address with others on request. As well, while we have no need to track ratios here, we do need to know who is seeding and who is leeching so that we don't waste time sharing seeds' IPs with other seeds. We'll call this variable uploader, and it'll simply be a bit assigned based on the test of left == 0.

At this point, our database looks like this:

If you're familiar with phpMyAdmin, you'll see that I've set a primary index on id, which is standard practice. I've also set an index on info_hash, since we'll be running a lot of queries for it.

Now, at this point, before the more knowledgeable members start tearing holes in my post, I want to point out that I'm working with an ideal model here. No information is lost, all clients follow the protocol to the letter (particularly in always cleanly closing connections), and there are no clients that spoof information for personal gain. Obviously, none of these are true, but since I'm aiming to explain an implementation of the BitTorrent protocol, I'm going to work with these assumptions for the time being, just like how friction is often ignored in introductory physics courses.

The Rules: Writing a Constitution

Yesterday I walked into a shop and was greeted by a sign thanking me for not urinating in the produce. Wait, no I wasn't. You know why? Because there are certain codes of behavior that are implicitly expected of all members of a society. Yet somehow sites feel the need to inform me that I could be banned if I start stirring up hell in the forum. Great, thanks for that.

Quite simply, there are some things that just don't need saying. Sure, you're going to get trolls in your forum, but making a rule telling them to go away is going to do nothing for you. Rules are for the people that genuinely care about their standing with the site. The people that are just around to make trouble aren't going to be reading your rules, and certainly won't make a point of abiding by them. You can't legislate common sense.

So what are your rules for? They're for people like me. I'm a member of a bunch of different trackers, and whenever I sign up, I always take a peek at the rules before I get down to business. I know what a BitTorrent tracker is, I know the usual etiquette, but I just want to find out if there's anything special about how your site does business. Do I have to post in rhyme? Have a cute, fuzzy avatar? Download only certain torrents? I don't want to dig through a bunch of bullshit about obeying "the moderators [sic] expressed wishes! [sic]" in order to find out what really matters.

Conversely, a lot of sites that have automatic ratio bans (which I've ranted about in previous posts) won't detail these bans in their rules. The hardest and fastest (and most variable) rule of all is somehow overlooked. The staff don't have to touch it as the banbot does all the work, but it's the rule with the most impact on new users, good and bad alike. Whether or not I agree with the concept is a moot point, at least tell me.

You're writing a constitution here, not a criminal code. If you've never read your country's constitution, go do it. It's probably available online. In general, they're pretty succinct and accessible for all their importance, and you'd be surprised what rights are being violated. Your criminal code, by contrast, is probably a few thousand pages long. The constitution is basically a vague statement of purpose, while a criminal code gets into the nitty gritties. Since your moderators are not police officers bound by the letter of the law, you don't need a criminal code. You simply need to give guidelines by which your site should operate.

A lot of sites have 15-20 rules, mostly the stuff that came preset with TBdev or whatever they're using. In my mind, you should be able to get everything you need in less than 10, preferably 5 general rules and 5 upload rules, in addition to the behavior that one normally expects of a generic good member elsewhere on the internet. The fewer rules you have, the better.

That's not to say you should have people responding to your moderation actions complaining that they didn't know something was against the rules. If it's the sort of rule a person can't be normally expected to know, then by all means throw it in there. Otherwise, forget about it. You're entrusting your moderators with plenty of discretionary powers, so you don't need to spell everything out.

I'd finish up with some examples from TorrentFries, but quite simply, it's stuff that applies only to that site and to none other. That's kinda the point.

Peer Management

Okay, I'm not in fantastic condition at the moment, but I want to keep things going. I'll just grab on to a recent suggestion – discussing ratios in a community context – and run with it.

Basically, any simple, closed system will be balanced, no matter the context: total uploads will equal total downloads. No matter how your tracker records data, it will always be tracking as much uploaded data as it does downloaded data, neglecting hash fails.

Therefore, when the ratio tracking system is altered, you will do nothing but alter the community dynamic. There will always be supposed gormless bottom-feeders at the bottom of your scale, and theoretical paragons of virtue at the top. Sorta. In my experience, chronic leechers are mostly those that simply haven't been initiated into proper torrent etiquette. I know in my early days of torrenting, I was quite unnerved by the prospect of others being able to connect to and download from me. To be sure, I didn't go out of my way to do this, until it became a point of personal pride to maintain a ratio above 1.0. It is only my involvement in the community that drove me to do this.

Now, think of an open site like The Pirate Bay. The tracker is simply a collection of its parts, each behaving according to their own conscience. The leechers leech according to their desire for gratification (or lesser experience, or lesser ability), while the seeders do so because of pride or generosity or simply because the swarm is so slow they end up uploading far more data than the download size by the time they complete. And you know what? It works. It may not work well, but it works.

Now, ratio-obsessed private trackers choose to whack off that bottom demographic, the people that are disinclined to share. That's all well and good, but you simply move the community's scale. Instead of having the true leechers at the bottom end of the scale, now you have the metaphorical middle class down there. In my experience, such systems tend to drift ever upwards as users are banned for poor ratios – in other words, as the bottom end of the spectrum is chopped off again and again. Obviously, these systems rely on new registrations to add a continual stream of new members to slow or halt that drift.

This is compensated for by free leech days, golden torrents, what have you. However, the only people that will participate in this activity are those that are genuinely concerned about ratios. They're not downloading torrents because they want to, but because it will improve their status in the community. To me, this is contrary to the purpose of file sharing. You share what you've got because you want to share it, and people accept what you're sharing because they want to have it.

Basically, these events are designed to fix a system that doesn't work properly all the time. If you need to schedule special events to compensate for your model, you must be doing something wrong. I would prefer to have a system that works in the same way and with the same effectiveness every day, rather than just once in a while.

So where's the solution? I don't have an answer. I don't think anyone does. I certainly do use a couple of private sites, but I also use the big boys when they have what I need or when I don't feel like destroying my ratio trying to seed back. "If it ain't broke, don't fix it," comes to mind, but I can't deny that swarms on private trackers do seem to perform better, so clearly something has been fixed or at least improved upon. Personally, I would prefer to see more personal treatment towards leechers. I believe that most leechers simply haven't been properly introduced to the social protocol (heh) associated with torrenting. If someone is actually intentionally and knowingly violating accepted behavior, they will take pains to hide it by using a cheating client of some sort. Therefore, wouldn't it be logical to take some means to inform the uninformed? I prefer not to participate in a community that is ultimately built on fear. It smells bad.

This is just meant as some food for thought, and hopefully will dispel some feelings of superiority on either side. Again, I'm sorry for my unfocused meandering, for my inability to make a solid point, and for any typographical errors I may have made. I'll probably read this over sometime when I'm in better intellectual shape and say, "I can't believe I said that," but there's nothing to be done. At least you got a post.

I had intended to write my tracker tutorial this week, but ah well. I'll do it sometime in the future. Stay tuned for that.