Guest Post: Forum Integration With SMFTracker

A friend of mine is working on a BitTorrent tracker mod for the Simple Machines Forum system. He was kind enough to provide us with a peek at his development process. He is basing his mod's announce on the basic framework I developed and published on Monday.

I've had to edit his post a bit, since he strayed into matters regarding our true identity pretty frequently, but the gist of the tracker mod development is there. Hey, what's life without a little mystery?

This is ChiliDog, co-collaborator of our ever-elusive stand-alone tracker script.  I was asked to make a guest post for the TorrentFries blog by CurlyFries.  Unfortunately, being the uninteresting person that I am, I’ve been racking my brain, trying to figure out what to write about.  I eventually came to the idea of writing about my current project: a SMF mod, with complete tracker integration.

Our main goal with the collaborative project was to successfully bridge any forum system’s [be it SMF, vBulletin, phpBB, etc.] database, however, that turned out to be a far-fetched dream [for now], due to its resource intensive code [blame CurlyFries ¬_¬].  But to tell you the truth, I couldn’t care less for any forum system other than SMF.  It’s powerful, feature-rich, and masterfully safe [not to mention completely free].  

I’ve been keeping an eye out for any tracker mods for SMF for the past 5 years or so, but not one has ever appeared.  There have been a few makeshift attempts at bridging a tracker system and SMF’s user table, but nothing worth using.  So after waiting so long, I finally set out to make one.

But, ‘Why a mod?  Why not just a bridge?’ you ask?  We’ll, I’ll tell you… It’s always been my philosophy to take advantage of what’s there, rather than to create something that does the same thing.  A bridge will still require all the things that SMF already does, in your tracker’s source.  CurlyFries is apparently the opposite of me.  He likes to create, rather than use what’s available [but more importantly, give people more options for the forum system].  That in itself is fine, and I’ll follow along for the main project, but I’ll be doing the SMFT mod in my spare time.

Other the announce.php, everything takes full advantage of SMF’s functions, including its permission system, themes, security, and even the language and modifications/upgrade system.  With three possible modes, open [ratio tracking is disabled], semi-private [non-members can still connect, just no ratio tracking and they might be restricted in site permissions], and private tracking [only members can connect], you will have a choice of what type of tracker you want to create.

There’s really nothing ground breaking about my project, but it should be the first SMF bittorrent tracker available; hence my project’s name: SMFTracker® [genius naming sense, eh?].  A lot of the front end source code, is the code CurlyFries and I created for our project, albeit most of it [if not all], was rewritten to work with SMF.

Our original announce [which I really liked] was ‘accidentally’ deleted by CurlyFries, but the [barebones] announce posted here is just as efficient [although, I prefer the switch system code he posted first].  I’m not ashamed to say I used that as a base to work with [credit was given where credit was due].

Although, I’m almost complete in coding SMFT [the remainder is just fluff and whatnot], it probably won’t be released for distribution this year.  I intend to beta test it for a while on my site and work out the kinks.  Once testing is completed, I’ll probably ask permission to add it to the SMF modification repository.  I say ‘ask permission’ because I’m not sure of SMF’s stance on a tracker mod.  They’re a legitimate corporation, and might decline my request, due to bittorrent being synonymous with copyright infringement [who’d have thought?].

On a side note, neither I, nor CurlyFries have abandoned our own project, and we do intend to [eventually] complete it.  However, SMFT is just something to keep me occupied until CurlyFries has extra free time on his hands for a time intensive project.

I definitely agree that SMF's mod system is ridiculously versatile, but with it comes unnecessary bloat and overhead. I personally prefer to build from scratch for maximum efficiency, and to me it makes more sense to build a forum around a tracker rather than a tracker around a forum.

I've subtly hinted that he may want to think about load testing the announces of various popular PHP/MySQL trackers, perhaps including his own and the one I wrote. It'd be interesting to see how they stack up.

The Tracker Demystified - Part 3: Reality

Okay, it's time to get back to the coding. On Tuesday, I developed a script that would play by the rules assuming everybody else did too. It was awesome: 29 lines, with a maximum of two database queries. And I wouldn't give it five minutes in the real world.

Why? Well, first, no input data was validated leaving the script open to SQL injection. Hopefully I don't need to say much more about the problems presented there, aside from that it would be a very bad thing were we not to address this issue. If you haven't heard of injection, go take a few minutes and read up on the implications. Since infohashes and peer ids are binary, any value containing 0x27 (apostrophe) would invalidate the SQL even without any malicious intent.

Secondly, even if we assume that the user is always benevolent, the script fails to anticipate that they may not be as true to the protocol as we are. The most glaring issue is that if the stopped message is never received, that particular row will remain in the database forever. This happens in the real world all the time; some poorly-designed clients will even neglect to send stopped when closed.

This time around, I'm just going to link the finished code and then go over some of the highlights. It'll be faster than just detailing every change in minute detail.

In my previous code, there wasn't even an error function. I've corrected that now, using the failure reason response provided by the protocol spec. I like my code to fail gracefully, since it cuts down on errors for users to come running to you with, but occasionally that's just not possible. Using failure reason with the message "Invalid port number", the user will see something like this in their client:

Obviously, you don't have a lot of room to work with, but you can get some pertinent information across. In this case, it is used to inform the user that their client is fucked. (Side note: this is an artificially-generated error. Transmission would never do anything bad.)

I've added a new field to the database, a timestamp field called last_connect. It has ON UPDATE CURRENT_TIMESTAMP set, so whenever we make a change to a row, the timestamp will be updated to match. This will be handy later on.

A bunch of validation has been added to the top of the script, which should be pretty self-explanatory. The BitTorrent spec describes the optional ip value as being "generally used for the origin if [the client] is on the same machine as the tracker", so the script makes an exception for 127.0.0.1, ignoring the value otherwise and simply going on the server's record.

I've shuffled the core about a bit so that they make more sense to the computer and less to a human. Since the maximum announce interval has been set to 30 minutes, if a row hasn't been updated in an hour, the script will assume it to be trash and delete it. In order to update the timestamp, a dummy query (SET uploader = $uploader) is run. Well, it's not entirely useless – this way, if the completed event is never received, the script won't even notice. In fact, it doesn't even check for completed anymore. Both it and empty are now treated in exactly the same way. 

If you rewind a bit to my previous entry on the subject, you'll see that empty was never dealt with. That's because it doesn't involve any change in state, which the tracker would need to record. Well, now the state is simply an assertion of its continued existence, which the tracker will need to keep track of.

It's not necessary to clear old entries with every connection; in fact, it is quite wasteful to do so. However, since this is a one-script application, we can't rely on a cron job or other external function to do our dirty work for us.

This is where I will leave you. There are a ton of features I could add, including ratio tracking, IP bans, client identification, protocol extensions (some trackers include swarm info on announce), defending against browser attacks, and so forth. However, I promised to deliver a script that functioned as a basic barebones open tracker in the real world, and that's exactly what I did. I hope you'll be able to use it as a framework to work from in writing your own code.

On the subject of working as a framework, you should now see why I left the script half-finished last time. It's a lot easier to understand the early version than the completed version, and hopefully helped a bit to clarify the basic structure we were working with. I'm sorry for any bugs or security holes that may have crept in here. This is a last-minute job as always, and it's entirely possible that I'm being sloppy.

That's all for now, folks.

The Tracker Demystified – Part 2: Structure

Now it's time to get started on the coding itself. For debugging purposes, we're going to need a lot more than the single-line tracker status field in normal BitTorrent clients. Happily, since the connectivity is simply via HTTP GET, you could just type the announce URL into your browser's address bar to test the effect of various values. In fact, this is the way I did my testing last time I took it upon myself to write a tracker.

This time, I'm going a little more high class. I've made a simple form with fields for each of the variables (info_hash, peer_id and so forth) that is submitted to announce.php using GET. It's much easier to play around with different values this way. Since forms are pretty simple and not the subject of discussion here, I won't go into any more depth here.

First, even before connecting to the database, I'll load the passed variables and perform a minimum of validation. Like yesterday, I'll be referring to the protocol spec to determine the expected values and ranges of particular variables. I don't like throwing a ton of errors at my user, so I tend to design for graceful failure. For example, if event is anything other than started, completed, stopped, or empty, it will simply default to empty.

Speaking of event, it is going to form the basis of our code structure. The heart of the script consists of a series of conditionals based on the four possible variables. Actually, since empty merely denotes that the client is requesting more peers, we only need the three that require updates on the tracker's end: started, completed, and stopped. These three events map cleanly to three MySQL statements: INSERT, UPDATE, and DELETE, respectively. When a torrent is started, the peer needs a new row in the database, which should be updated when the download has finished and removed when the connection is closed.

Now, stopped only takes input – there's no point in making output when nobody is going to use it. For everything else, we'll use BitTorrent's bencode protocol (detailed in the spec) to output the proper response. Again, I encourage you to go back and peruse the spec. Even if you don't muck around in the backend much, it's nice to be able to pop the hood of any metainfo (.torrent) file you may come across and get a general idea of what's going on in there. The bencoded data may look intimidating, but it's perfectly readable if you take a few minutes to understand how it works.

So, back to the announce. The proper response I mentioned above is a dictionary consisting of two keys/value pairs: interval to indicate the number of seconds until the tracker suggests that you scrape again, and the peers to provide a list of the peer id, ip, and port of each provided peer. There's one bit in the official specification that I feel the need to quote for all the private tracker elitists out there:

Note that downloaders may rerequest on nonscheduled times if an event happens or they need more peers.

Announcing more often than this is perfectly acceptable practice. I do hate to hold it up for ridicule, but BitMe has ironically banned the Mainline (BitTorrent) client originally developed by Bram Cohen himself, developer of the protocol, because it "does not honor the protocol requirements of private trackers". I'm not going to knock on BitMe too much because they're more open than many private trackers in that regard, and they're the only one that I know of that lists banned clients and justifications for banning. However, allow me to take this opportunity to say... what the fuck?

That's a little off-topic. Anyway, now we just have to pull the data from the database. In practice, clients can and do indicate the number of peers they're looking for with the numwant variable, but that's not part of the spec, so I'm ignoring it for now. Instead, I've specified 25 as an arbitrary maximum, and used ORDER BY RAND() in my MySQL query to ensure that all peers get a statistically equal showing no matter where they sit in the database.

Now that we've got the pertinent data, it has to be output, so we'll bencode it according to the structure laid out in the protocol, and shove it out the door. I might talk about writing full functions to read and write bencoded data at some point, but it's a lot more work. The implementation in the announce is simple because we only have to account for the variation between different volumes of data, which is nevertheless being output in a rigid format. Super easy.

Now, while I may not have done a particularly satisfying of describing the last bit of the announce, it is in fact done. The finished script is only 29 lines, including whitespace. I had it heavily commented, but many of the comments repeated what I'm saying here, and I want to emphasize how simple an announce can really be, so I've left it blank and hopefully the minimalism will speak for itself.

At the moment, I'm running three peers: Transmission, Azureus, and my own web browser masquerading as a client. Transmission has successfully seeded the file to Azureus, and both are trying to connect to my web browser to share with it too. Obviously, since it isn't actually a torrent client at all, that's not going to happen.

Now, I see the seasoned coders cringing already, so I'll repeat this once again: this is a barebones demo script. It is designed to provide a framework for you to work off of in forming a more complex announce. The idea is to illustrate the way the protocol works, and the way the tracker interfaces with the protocol. As it stands now, the script doesn't even sanitize input, so it is vulnerable to SQL injection as well as just about every other damn thing.

Tomorrow, I hope to elaborate a bit on these issues, refining this code to create a perfectly usable announce. Ratio tracking is still outside the scope of the project, but we can certainly tighten things up and maybe lay the necessary groundwork for you to go on and add ratios and such frivolity. I say "I hope" because I'm burdened with a lot of work tomorrow and may not get time for a lot of coding. I'll make sure you get an article, and hopefully part 3 of this series, but no promises.

And yes, you're welcome to take the code I wrote above and use it without attribution for whatever purposes you want. However, if you're going to do that, I really suggest that you fix the holes first, or wait until I have a chance to do it for you.

The Tracker Demystified – Part 1: Building the Database

You may have noticed a post on this subject a while back that was unintentionally released before its time. Well, if you have already read that one, re-read this anyway. It's finished, for one thing.

Anyway, this week is going to be devoted to building a simple tracker from scratch. This tracker will do no more than accept and share IP addresses, with no front-end for uploading and downloading metainfo (.torrent) files. However, if you've worked with PHP much, you can probably already work out how to do file uploads and downloads and pretty or at least usable interfaces. This is the big obstacle, and also the bit that has the potential to be interesting for the non-coder, if I can manage to write clearly enough to keep them engaged and slightly comprehending.

Hopefully, over the course of the week, I'll dispel any notions of BitTorrent as a complex or incomprehensible protocol. Once you get your head around it, it's actually quite easy to understand and use. The peer-to-peer bit is a little less straightforward, but happily, we don't have to deal with that. We're writing a tracker, not a client.

My resource in all of this is going to be the official BitTorrent spec, which I know almost by heart. I'll be referring to the spec from time to time, so read it over and my posts might start to make some kind of sense. My MySQL abilities are a little more touch-and-go, so optimization may not be as fantastic as it could be and I'd welcome any constructive criticism on that front.

On that note, we'll be starting today with outlining the basic structure of the announce through the creation of a database table for the peers. This is the only table we'll need, which is handy. Basically, we need to store the pertinent bits of the data that's received by the announce, and enough to provide a coherent response. There are eight variables passed by the client to the tracker: info_hash, peer_id, ip, port, uploaded, downloaded, left, and event. Official or unofficial extensions may add extra values, but we're writing a barebones tracker, so we can safely ignore them. The spec does a perfectly good job of clearly outlining the purpose of each of these variables, so I won't repeat it here.

Now, all of the provided variables are pretty important for various things, but again, this isn't a full-featured tracker, so we'll ignore some of them. We'll save info_hash so we can connect peers on the same torrent to one another, peer_id so we can distinguish one peer from another, and ip and port so we can share the user's address with others on request. As well, while we have no need to track ratios here, we do need to know who is seeding and who is leeching so that we don't waste time sharing seeds' IPs with other seeds. We'll call this variable uploader, and it'll simply be a bit assigned based on the test of left == 0.

At this point, our database looks like this:

If you're familiar with phpMyAdmin, you'll see that I've set a primary index on id, which is standard practice. I've also set an index on info_hash, since we'll be running a lot of queries for it.

Now, at this point, before the more knowledgeable members start tearing holes in my post, I want to point out that I'm working with an ideal model here. No information is lost, all clients follow the protocol to the letter (particularly in always cleanly closing connections), and there are no clients that spoof information for personal gain. Obviously, none of these are true, but since I'm aiming to explain an implementation of the BitTorrent protocol, I'm going to work with these assumptions for the time being, just like how friction is often ignored in introductory physics courses.

Protocol

Okay, this is going to be a quick post because OnionRings and I both have a ton of work to do and Ketchup seems to be maintaining our collective laziness quota.

I'm planning to offer a detailed examination of what exactly goes into the technical end of a BitTorrent tracker by offering explanations as I write my own, but that's well beyond my time allotment for the evening. Instead, I'll write a simple summary of the protocol.

The peer-to-peer aspect is not important for our purposes. Unless you're planning on writing a client, which is well outside the scope of this site, you don't even need to think about it. I know I don't. I really haven't taken the time to study it in much detail, aside from understanding what's going on in general terms.

Happily, the peer-to-tracker protocol is very simple to understand. The tracker is inherently passive, never initiating connections itself. Instead, clients connect to the tracker via HTTP. They provide the infohash (unique identifier) for the torrent they're trying to download, their current port, and a bunch of other information indicating the state of the client. The tracker responds with a simple list of IP addresses that are also downloading the same file. It's up to the client to initiate connections with the provided peers. Since the protocol is peer-to-peer, any client can initiate the connection, assuming that the other computer is properly configured to accept incoming connections.

That's all a tracker has to do: log the IP addresses that connect, and respond with a list of suggestions for possible peers. The process is fast, light on resources, and very easy to code. You can develop a tracker just like any website, using just about any language you want: PHP, ASP, Ruby, whatever. The biggest trackers run on C or other such languages, since it's fast and lightweight beyond the dreams of PHP. However, for your purposes, web languages are all you need.

That said, I'll pick up on the subject later on when I actually find the time to get going on development. Like I say, things have been nuts of late.

Back to the Basics

I realize that our focus has of late moved from tracker-specific discussion to more-general stuff that applies to any site or server and can be found in a bazillion places online. Since this blog is unique because it does chronicle the operation of a BitTorrent tracker, I'm going to get back to the meat of the thing for a bit. OnionRings plans to continue with his Linux series, but is sadly indisposed this evening.

In some senses I will cover some of the same material as our second-ever blog entry, but with a different focus. While that entry gave a quick overview of why and how to set up a tracker, I'm going to look a little more closely at the technical and logistical aspects of bringing a tracker online.

First off, what do you personally need to bring to the table? (You're probably the only one at the metaphorical table at this point, but the bringing should commence now.) Beyond anything else, you need to understand the BitTorrent protocol, and understand your tracker of choice. We'll talk a bit about choosing trackers shortly. If you plan on running a PHP/MySQL tracker, you'd better have at least a working knowledge of PHP and MySQL. From the get-go, you'll need to be able to peer into the inner workings of your site and see what makes it tick. You don't just need an understanding of the protocol, but of your own tracker. Trust me, admin panels are nice, but some things just require you to get down and dirty. However complete your chosen script may look at first glance, there will quickly come a time when you or your users bemoan the lack or poor design of some feature or another, and you will be forced to go in and remedy the situation. As the tracker grows and moves to its first dedicated server, it will also become important to know your way around Linux to some extent.

Secondly, you need a hook to pull people in. The most obvious aspect is to provide something that others can't. A unique idea is neat, but it's entirely possible to carve out a niche in a "market" that is well-developed but not saturated. For example, if you have terabytes of obscure movies that you have spent years collecting, starting a movie tracker to share these can be an excellent starting point. Of course, you will need to keep all the uploaded torrents active, even though you will be getting very few peers at first. For this purpose, you may want to consider renting a seedbox. The added cost may be hard for a brand new tracker admin to swallow, but your ability to saturate the connections of your first members will do a lot to encourage them to stick around and maybe even contribute some stuff of their own. As well, a unique design may sound frivolous, but it will lend your site credibility and a sense of longevity. I've covered some methods of early promotion in the post I mentioned above, so I won't rehash them here.

I hate to be defeatist, but if you can't meet these requirements, you should think twice about starting a tracker. Of course, the lovely thing about being a human is the capacity to learn, but you should get going on that learning well before you even consider getting into tracker territory. While I'm trying to demystify the role of the tracker administrator, it's certainly a job that not everyone has the skill and disposition to fill with any great success.

Moving on to more technical requirements, you will also need to choose a tracker. That's a given. The three most popular trackers under active development are TBdev, TorrentTrader, and the new Project Gazelle, although there are a ton of other options out there, so you shouldn't feel constrained to choosing one of these three. Shop around a bit. Detailed comparison of these and other trackers is well outside the scope of this entry, and perhaps even of this blog, but this should point you in the right direction. At the end of the day, it's up to you to select the tracker that best suits your particular tastes and needs. As a side note: for the love of God, don't use TorrentTrader Classic.

So, why not just boot Azureus or µTorrent or one of these handy ubiquitous little torrent clients that happen to include the ability to operate as standalone trackers? Hopefully this isn't a question you were asking yourself, but I'll answer it anyway. First, this isn't the purpose they were designed for. Just because they can run a tracker doesn't mean they should. They're not optimized for the purpose, they don't include features that are necessary for the smooth operation of a tracker (they're inherently open to anyone that wants to use them), and they include a ton of features and bloat that will just bog down your server. What's more, part of the sort of tracker we're discussing here is the index, or frontend. This is the bit where torrents are uploaded, downloaded, where users interact with one another, all that sort of stuff. The tracker itself is ridiculously simple in operation, as we will see in my eventual piece-by-piece breakdown of what it takes to build one from scratch. The important business is the complex frontend, and BitTorrent clients' wannabe trackers just can't provide this.

So, once you've selected the script that best fits your needs, it's time to choose a host. I've had some requests to list a number of torrent-friendly hosts, but that's not something I'm going to do for a number of reasons. The most important reason here is that I don't want to endorse a host that turns out to be unsafe, or to lull readers into a false sense of security. You may have noticed that even (especially) the biggest trackers tend to move around a lot. The world of web hosts is intrinsically volatile, so to proclaim that one host is perfectly safe, even if presently true, is to ignore the fact that this may not continue to be the case in the future. LeaseWeb was once a safe haven for trackers, but after losing a lengthy court battle on behalf of Demonoid and other trackers, all the will to fight has gone out of them. Nowadays, if a copyright holder says "boo", they'll turn around and shut you down.

So simply accept that no place is perfectly safe, and operate under the assumption that you could be taken down tomorrow – you could be. A good strategy to find places that are safer is to run whois lookups on established trackers' IP addresses. Presumably, if they are able to operate on a particular host, that host must be somewhat resistant to legal threats. The only problem is that many larger trackers own their own servers which they operate via colocation in the host's datacenter, while you are likely unable to afford more than a rented dedicated server (an important distinction to make). Not all hosts supply both options, and not all do so for reasonable rates. In addition to colocated and dedicated servers, I've talked about shared and virtual hosting in that previous article, so I'm not going to rehash that here.

This should be all you need to get you off the ground. Again, I'm not going into exacting detail on some of the technical points because I'm assuming that you will be able to handle the fiddly bits of your own accord. If you can't, like I mentioned, you probably shouldn't be running a tracker.

If there are any points on the subject (or any other, really) that you would like us to cover, leave a comment on this post and we'll do our best to touch on them in future posts.

Retrospective 1: Conception

Since starting TorrentFries, I've had a number of people ask me how to start a successful tracker. The first question I ask is always "What sort of material do you want on the site?" Only one person gave me a response other than "everything". Incidentally, I'm now involved in some small way in the development of that other person's tracker. The two aren't directly related (I'm not a sword for hire), but they're not entirely unrelated either.

Listen, I'll let you in on a secret. There are tons of "everything" trackers out there. Come up with something unique to pull people in.

Personally, I believe in unconventional activism. In this case, deep-fried goods are too expensive, so I thought a BitTorrent tracker might get people talking about the problem. Since nobody else had one going, I set out to create one of my own.

The first step in the process was to find a host and a tracker. In this situation, personal connections are nice. I was lucky enough to know a guy that ran a shared web host, so I was able to get things running for $5/month, plus the cost of registering a domain name ($12/year). However, any shared host worth their salt should figure out that someone is running a tracker pretty quickly, so don't count on slipping that under the radar. Instead, you might consider a VPS, which allows you to rent part of a server, rather than having to pay for the whole thing. You should have plenty of control over your server, and get more leeway in terms of traffic. A basic VPS will cost you around $40/month.

However, be aware that there's no way you can buy security for any price you can afford at this point. The best you can hope for is a web host that will look the other way until they hear from a copyright holder. Most dedicated and VPS hosts I've encountered will do this; in fact, some hosts have told me this directly. What you can do is maintain daily off-site backups and have a backup host lined up in advance in case of the worst. We'll talk about contingency planning a bit later on.

Choosing the software involves a fair bit of research. I was pretty lazy on that point, and have been paying for it ever since. The only two trackers I checked out were TBdev and TorrentTrader Classic. TBdev is used by a lot of big-name private trackers such as Torrent-Damage, TorrentLeech, the late OiNK, LearnBits, BitMe, BlackCats, BrokenStones and so forth. However, the administration functions are almost non-existent, and the amount of customization available in TorrentTrader appealed to me. I didn't want to spend much time digging around in the code to get everything to work properly.

I've got a fair bit of experience with graphic and web design, so I took it as given that I'd build a new theme from the ground up. The product was in retrospect not that impressive, but it was certainly usable and unquestionably better than what I'd started with.

Finally, before going live, I asked a friend to take a look at the site. I expected him to spend perhaps 5 minutes and respond with a "looks good", but he ended up giving me a two-page essay including everything down to copy-editing in the FAQ. It was great help, and I was able to refine a lot of features that way.

Tomorrow: details on how to pull members in.