TorrentFries

Introducing the New Forum

2008-11-10T12:22:00.000-08:00

As previously mentioned, we've been working on creating a forum to replace the blog and hopefully provide a similar source for useful information and discussion. Well, the TorrentFries forum is now online. We encourage you to head over there, sign up, and check it out. This blog will remain online for reference purposes but will no longer be updated. Any further entries we feel like adding will be made over at the forum.

I hope you have enjoyed the blog, and I hope you will find the forum useful and perhaps entertaining. I'd like to extend extra-special thanks to Chris for registering the domain for us before some filthy squatter could get to it. torrentfries.com and www.torrentfries.com will continue to redirect to the blog to prevent broken links, and forum.torrentfries.com will take you to the forum.

A Public Service Announcement

2008-11-04T06:45:00.000-08:00

No serious post today. I just wanted to remind any Americans in the crowd to get out and vote. Vote Obama, vote McCain, vote motherfucking Communist, just vote.

Trust me. It'll be good for you.

That'll be all. This has been a public service announcement paid for by nobody, unfortunately.

Irregular Posting Schedule

2008-11-03T02:44:00.001-08:00

Since the beginning of September, we've been posting religiously, 5 days a week. Our steady stream of subjects of discussion having slowed to a trickle, we're moving to a less regular schedule. We will continue posting at least once a week for the time being. I'm not one to leave you hanging, so when the blog reaches its end, you'll know it.

We will be launching a TorrentFries community, so stay tuned for announcements on that. In the meantime, if you have any questions that you'd like to have addressed, please do leave a comment and we'll do our best to answer. I've got a few to respond to at the moment, and will be doing so in the coming days.

Guest Post: Forum Integration With SMFTracker

2008-10-31T00:50:00.000-07:00

A friend of mine is working on a BitTorrent tracker mod for the Simple Machines Forum system. He was kind enough to provide us with a peek at his development process. He is basing his mod's announce on the basic framework I developed and published on Monday.

I've had to edit his post a bit, since he strayed into matters regarding our true identity pretty frequently, but the gist of the tracker mod development is there. Hey, what's life without a little mystery?

This is ChiliDog, co-collaborator of our ever-elusive stand-alone tracker script. I was asked to make a guest post for the TorrentFries blog by CurlyFries. Unfortunately, being the uninteresting person that I am, I’ve been racking my brain, trying to figure out what to write about. I eventually came to the idea of writing about my current project: a SMF mod, with complete tracker integration.

Our main goal with the collaborative project was to successfully bridge any forum system’s [be it SMF, vBulletin, phpBB, etc.] database, however, that turned out to be a far-fetched dream [for now], due to its resource intensive code [blame CurlyFries ¬_¬]. But to tell you the truth, I couldn’t care less for any forum system other than SMF. It’s powerful, feature-rich, and masterfully safe [not to mention completely free].

I’ve been keeping an eye out for any tracker mods for SMF for the past 5 years or so, but not one has ever appeared. There have been a few makeshift attempts at bridging a tracker system and SMF’s user table, but nothing worth using. So after waiting so long, I finally set out to make one.

But, ‘Why a mod? Why not just a bridge?’ you ask? We’ll, I’ll tell you… It’s always been my philosophy to take advantage of what’s there, rather than to create something that does the same thing. A bridge will still require all the things that SMF already does, in your tracker’s source. CurlyFries is apparently the opposite of me. He likes to create, rather than use what’s available [but more importantly, give people more options for the forum system]. That in itself is fine, and I’ll follow along for the main project, but I’ll be doing the SMFT mod in my spare time.

Other the announce.php, everything takes full advantage of SMF’s functions, including its permission system, themes, security, and even the language and modifications/upgrade system. With three possible modes, open [ratio tracking is disabled], semi-private [non-members can still connect, just no ratio tracking and they might be restricted in site permissions], and private tracking [only members can connect], you will have a choice of what type of tracker you want to create.

There’s really nothing ground breaking about my project, but it should be the first SMF bittorrent tracker available; hence my project’s name: SMFTracker® [genius naming sense, eh?]. A lot of the front end source code, is the code CurlyFries and I created for our project, albeit most of it [if not all], was rewritten to work with SMF.

Our original announce [which I really liked] was ‘accidentally’ deleted by CurlyFries, but the [barebones] announce posted here is just as efficient [although, I prefer the switch system code he posted first]. I’m not ashamed to say I used that as a base to work with [credit was given where credit was due].

Although, I’m almost complete in coding SMFT [the remainder is just fluff and whatnot], it probably won’t be released for distribution this year. I intend to beta test it for a while on my site and work out the kinks. Once testing is completed, I’ll probably ask permission to add it to the SMF modification repository. I say ‘ask permission’ because I’m not sure of SMF’s stance on a tracker mod. They’re a legitimate corporation, and might decline my request, due to bittorrent being synonymous with copyright infringement [who’d have thought?].

On a side note, neither I, nor CurlyFries have abandoned our own project, and we do intend to [eventually] complete it. However, SMFT is just something to keep me occupied until CurlyFries has extra free time on his hands for a time intensive project.

I definitely agree that SMF's mod system is ridiculously versatile, but with it comes unnecessary bloat and overhead. I personally prefer to build from scratch for maximum efficiency, and to me it makes more sense to build a forum around a tracker rather than a tracker around a forum.

I've subtly hinted that he may want to think about load testing the announces of various popular PHP/MySQL trackers, perhaps including his own and the one I wrote. It'd be interesting to see how they stack up.

Building a Community

2008-10-30T04:26:00.000-07:00

The heart of a good BitTorrent site is a good community. This is a hard subject to address, which is one reason I've hitherto avoided it. However, it's also a very important one, so I'll do my best.

Easier than describing how to make a good community is perhaps how not to do so. I know of a group of people that are trying to create a community out of whole cloth, which strikes me as ridiculously naïve. That said, they're more knowledgeable than I am, and if they succeed, the more power to them.

However, in my experience, good communities are things that are not made but simply come about of their own accord. The best you can do as administrator is to focus on creating an environment that fosters a strong community. I've already outlined factors that can poison a community: strong emphasis on ratio, paranoid pre-emptive action against potential undesirables, heavy-handed and arbitrary moderation, and so forth.

People like to be free, to be able to speak their minds without fear of repercussions. They will naturally gravitate to a place where they feel that they can do this. Make no mistake, the world doesn't need another 4chan. In fact, the world doesn't need the 4chan it already has. As always, it is necessary to find a balance between the extremes of staff control and utter chaos – to moderate in moderation, as it were. Trying to entice people become active in the community is doomed to failure. Rewards for high post counts are ridiculous; people are too hung up over post counts anyway, and it's harder to break in to a community where post count equals status equals worth as a person. It may not seem that way to you, but it will to the newcomers.

This is where things get difficult. It's up to you to find your own balance. All I can do is lay out the questions and leave you to answer.

If you've got any more points you'd like us to cover before shutting this bad boy down, now's the time. Don't be shy!

Donations

2008-10-29T05:21:00.000-07:00

In light of yesterday's post, I'm going to spend the next few days addressing questions as they come up. Today's question is regarding encouraging donations.

In my experience, people are generally pretty decent sorts and will do what they can to support their favorite site, and I'm assuming that's you. I'm not into pushy salesmanship, and I don't think it's necessary. It is annoying and compromises the tone of your site, but more importantly, it crosses a rather important legal line between facilitating the sharing of copyrighted material and outright selling it. Where you want to stand on that question is entirely your own choice, but I like to stay on the light side.

However, I'm not opposed to the idea of buying ratio, as long as your site doesn't make it necessary for users to do so. Ratio represents a contribution to the site, and there are all kinds of ways to contribute. Some sites also provide ratio bonuses for long-term seeders, and I think there is very much the same thing going on here. If you've contributed, I have no problem with recognizing you for it.

I have a slightly less sunny image of donor/ratio perks, though. Any features that you provide to VIPs are features that you are denying to other users, which in turn prevents your site from being as usable as possible to as many people as possible - kinda the point in my view. Providing recognition is fantastic, but adding concrete features is a little more questionable. VIP-only forum boards will fracture your community, since the VIPs will tend to hang out with their kin and other users will never really be pulled into the fold.

In terms of recognition, I suggest simply giving users more control over their own profiles and the way they are presented to others. I know some sites allow avatars only to power users. However, since I only ever notice peoples' avatars rather than their usernames, I want to encourage as many people to have avatars as possible. You could allow people to customize the way their username is displayed, add more info to their profiles, maybe add custom stylesheets. None of these things have any practical use, but they provide a degree of social status as a reward, and that is enough. Of course, the ubiquitous donor star is a must.

I suggest placing a clear donate button on every page, but nowhere obtrusive. In keeping with my previous point regarding selling copyrighted material, I would keep it away from torrent description pages. Make it clear to your users that donations are always appreciated but never expected. A simple button in the header or footer ought to be plenty. I like Demonoid's approach of using an animated GIF that flashes briefly and subtly every couple of seconds. It draws attention without being too distracting.

Demonoid also has a neat donation system. I'm not sure I should be detailing it here, but you should be able to get some idea by making a donation yourself, or going through the process up to the point where you're asked to key in your PayPal account. Basically, donated money is received not through Demonoid but through a third party site, so PayPal can have no complaint. Very clever.

The Future of TorrentFries' Blog

2008-10-28T00:09:00.000-07:00

You may have noticed that we've been running a little short on subjects for discussion of late. This was intended to be an account of the day-to-day operation of a tracker, but with the site's untimely demise, that fell by the wayside and we ended up relying on a series of exposés instead. That's all well and good, and I really hope you've found them useful, interesting and/or insightful up to this point. However, there is only so much that one can discuss without the constant influx of new material resulting from running a live tracker.

Therefore, I'm interested to get your take on where we should go from here. We have been discussing creating a forum for those interested in the subjects we've discussed here, so that remains an option. We may also move to a less intense publication schedule, although that's only prolonging the inevitable. One way or another, the blog's days are numbered, so if you have any questions that we've left unaddressed, by all means, leave us a comment and we'll see what we can do for you.

The Tracker Demystified - Part 3: Reality

2008-10-27T01:05:00.000-07:00

Okay, it's time to get back to the coding. On Tuesday, I developed a script that would play by the rules assuming everybody else did too. It was awesome: 29 lines, with a maximum of two database queries. And I wouldn't give it five minutes in the real world.

Why? Well, first, no input data was validated leaving the script open to SQL injection. Hopefully I don't need to say much more about the problems presented there, aside from that it would be a very bad thing were we not to address this issue. If you haven't heard of injection, go take a few minutes and read up on the implications. Since infohashes and peer ids are binary, any value containing 0x27 (apostrophe) would invalidate the SQL even without any malicious intent.

Secondly, even if we assume that the user is always benevolent, the script fails to anticipate that they may not be as true to the protocol as we are. The most glaring issue is that if the stopped message is never received, that particular row will remain in the database forever. This happens in the real world all the time; some poorly-designed clients will even neglect to send stopped when closed.

This time around, I'm just going to link the finished code and then go over some of the highlights. It'll be faster than just detailing every change in minute detail.

In my previous code, there wasn't even an error function. I've corrected that now, using the failure reason response provided by the protocol spec. I like my code to fail gracefully, since it cuts down on errors for users to come running to you with, but occasionally that's just not possible. Using failure reason with the message "Invalid port number", the user will see something like this in their client:

Obviously, you don't have a lot of room to work with, but you can get some pertinent information across. In this case, it is used to inform the user that their client is fucked. (Side note: this is an artificially-generated error. Transmission would never do anything bad.)

I've added a new field to the database, a timestamp field called last_connect. It has ON UPDATE CURRENT_TIMESTAMP set, so whenever we make a change to a row, the timestamp will be updated to match. This will be handy later on.

A bunch of validation has been added to the top of the script, which should be pretty self-explanatory. The BitTorrent spec describes the optional ip value as being "generally used for the origin if [the client] is on the same machine as the tracker", so the script makes an exception for 127.0.0.1, ignoring the value otherwise and simply going on the server's record.

I've shuffled the core about a bit so that they make more sense to the computer and less to a human. Since the maximum announce interval has been set to 30 minutes, if a row hasn't been updated in an hour, the script will assume it to be trash and delete it. In order to update the timestamp, a dummy query (SET uploader = $uploader) is run. Well, it's not entirely useless – this way, if the completed event is never received, the script won't even notice. In fact, it doesn't even check for completed anymore. Both it and empty are now treated in exactly the same way.

If you rewind a bit to my previous entry on the subject, you'll see that empty was never dealt with. That's because it doesn't involve any change in state, which the tracker would need to record. Well, now the state is simply an assertion of its continued existence, which the tracker will need to keep track of.

It's not necessary to clear old entries with every connection; in fact, it is quite wasteful to do so. However, since this is a one-script application, we can't rely on a cron job or other external function to do our dirty work for us.

This is where I will leave you. There are a ton of features I could add, including ratio tracking, IP bans, client identification, protocol extensions (some trackers include swarm info on announce), defending against browser attacks, and so forth. However, I promised to deliver a script that functioned as a basic barebones open tracker in the real world, and that's exactly what I did. I hope you'll be able to use it as a framework to work from in writing your own code.

On the subject of working as a framework, you should now see why I left the script half-finished last time. It's a lot easier to understand the early version than the completed version, and hopefully helped a bit to clarify the basic structure we were working with. I'm sorry for any bugs or security holes that may have crept in here. This is a last-minute job as always, and it's entirely possible that I'm being sloppy.

That's all for now, folks.

Big-Tracker Finances

2008-10-24T03:14:00.000-07:00

As I mentioned yesterday, I am seriously strapped for time this evening and won't be able to continue my announce-writing series as I had intended. I'll do some coding over the weekend and hopefully get things going again on Monday.

Early in the life of the blog, I discussed the financial logistics of running a small tracker. Well, TorrentFries grew out of that phase. As you grow, so does the complexity of your bookkeeping.

Obviously, the cost of hosting increases as you move to faster and faster servers. However, things also become a lot less predictable. I've previously advised that contingency is a good thing, and nowhere is this more true than in the case of finances. While you're small beans, you will probably be paying the same cost every month for a long time. Hosting is predictable, although we were impacted by the dropping US dollar, which caused our hosting costs (charged in euro) to increase without any benefit to us. Of course, we were growing and becoming richer as time went on, so that was more annoying than difficult.

Eventually, it is likely that donations will outstrip expenses. Don't let it go to your head and don't squander the money. A big site is always in danger of being taken offline or even coming under threat from the copyright police. Having a slush fund handy is invaluable, since you won't even be able to beg for money with the site down. At one point, we had as much as $800 in reserve, plenty to get us a new server if need be. In fact, we were saving up to build our own box to colocate somewhere friendly. That didn't end up happening, but it certainly could have with a few more months. At the time, we were making about $30/mo in ad revenues (Project Wonderful is not the most profitable provider around) and over $500/mo in donations, although donors are fickle and hard to rely on – another good argument for a slush fund.

Now, running into trouble with PayPal (which I assume you are using for your finances) is another concern. I suggest spreading your wealth across multiple accounts, a few hundred dollars in each. If one goes down, you still have access to enough money to tide you through until things get sorted out. Don't use Tor while accessing PayPal. If your geographical IP jumps around too much, PayPal will automatically flag your account as potentially compromised and will freeze your funds until you confirm the account with a credit card. Prepaid cards are handy there, but a massive pain in the ass nonetheless.

How you choose to use your donated money is entirely your own choice, but I have no ethical problem with paying for personal projects that have direct benefit to the site. For example, I bought Coda with donated money, and paid for a course in UNIX administration that significantly improved my ability to do my job. I went from absolutely thumb-fingered to only moderately thumb-fingered, and I think the cost was worth it.

Since there is a massive time investment involved in the operation of a torrent site, I wouldn't begrudge you making a profit, either. The excess donations I received would have amounted to something in the ballpark of $1/hr, and I don't think that's unreasonable. However, profiting from the site opens a whole new can of worms. Profiting from copyright violation puts you in different ethical and legal territory. The implication of sale is what landed the EliteTorrents admins in deep shit, and will get you kicked out of Spain, too. As always, it's a personal decision as to where you want your tracker to go, and I decided to take the high road.

Botbusting

2008-10-23T05:52:00.000-07:00

Okay, this week has heated up a great deal for me, so it doesn't look like I'll be able to continue my coding series until Monday. In the meantime, I want to talk a little about combating bots of different sorts. OnionRings knows more about the technical and security stuff than I do, so he may have some stuff to add or correct. Maybe he'll do some stuff about tightening up the backend, but my expertise is on the frontend/touchy feely stuff, so I'll just focus on that.

A CAPTCHA is an image that is (ideally) impossible for a computer to read but easy for a human. It's used to verify that a user is not a bot, rendering brute-force attacks unfeasible. However, even if it is easy for a human, it's also very annoying. For that reason, you need to use them with moderation.

The biggest technical threat is brute-force attacks on account logins. If someone gets the password of the admin account... well, I don't have to tell you the kind of hurt that can result. Combating this can be as simple as implementing brief (15-minute) bans after a certain number of failed login attempts. I don't think it's feasible to use CAPTCHA in this case because it's something that everyone will be presented with, especially if they don't have their browsers set to keep them logged in. I dislike entering more than a couple of them a day, and any unnecessary ones annoy me. However, if you implement temporary IP bans and change your password regularly, it should afford you a reasonable amount of protection. Other options here may be to require CAPTCHA on all attempts after the first one (decided by IP, not cookie), or to create an account that is used only for administration and give lower permissions to your regular account.

You can use CAPTCHA on registration, but I don't like to spend too long filling out registration forms, either. Besides, a human can register, then let a bot loose on the site. It's still not a great solution, although if you have fairly generic registration forms (or use an unmodified script), that may be a concern.

Instead, I suggest requiring CAPTCHA on a user's first few posts/comments/messages. After the first few, no more annoyances. One idea I just came up with and have yet to implement is simply to require CAPTCHA for all users with a 0/0 ratio. If you're using your account, you're likely not a bot.

No matter how careful you are, some will inevitably slip through. As I've mentioned before, proactively promoting moderators means that what does get through can be dealt with reasonably quickly. This is especially nice in the case of porn spam with images attached, which is never nice to be presented with unexpectedly.

If you have any other suggestions, I'd love to hear them.

Building a Foundation of Knowledge

2008-10-22T01:00:00.000-07:00

Since CurlyFries was tied up yesterday, he's unable to continue his Tracker Demystified series today. Instead, I bring you some learning advice and an ambitious reading list from a true bookworm, or book-collector — whichever you prefer. As an admin, the most important thing you can have is a friend that knows what they're talking about when it comes to Linux. Second best is a book to tell you how to do it and give you more detail. Since I'm not a Linux guru by trade, having plenty of people and books around got me out of quite a few tight spots. I wouldn't have normally been able to wiggle out them on my own just Googling. Sometimes even the smallest pieces of advice are worthwhile.

For example, we were trying to do some optimization on the server because TorrentFries' forums were slowing down. When I asked a friend what he knew about optimization, he asked me "How does htop look?" Quite frankly, I had never heard of it, but it's proven to be one of the most useful tools, not to mention the most used. (htop is a more advanced visual extension to top. You'll never use top again.)

Anyways, I've latched onto over half a dozen such people to draw from their Linux expertise now and again, and also so I don't have to go back to the same person time and time again. Networking is more than just TCP/IP. ;)

I'll start you off with my recommendations from the books I've read. I'm not going to claim that they're the best, just that they've provided me with some good background.

Unix Administration

How Linux Works
UNIX, Third Ed.
The Book of Postfix (email - just a casual read)

Development

PHP 6 and MySQL 5 for Dynamic Web Sites: Visual QuickPro Guide
PHP Solutions: Dynamic Web Design Made Easy
SQL Power!: The Comprehensive Guide

Web Design

Web Designer's Reference
Web Standards Solutions: The Markup and Style Handbook
CSS Mastery: Advanced Web Standards Solutions
Pro CSS Techniques

I've read quite a few others, but for the sake of brevity, I'll leave them off. Here are a couple other books that I might also suggest reading even though I haven't read them yet, or only have read excerpts out of.

Apache Cookbook: Solutions and Examples for Apache Administrators
High Performance MySQL: Optimization, Backups, Replication, and More
Apache: The Definitive Guide
Running Linux
Hardening Linux

There you have it. That's well over a year's worth of reading by my standard, but it's a solid foundation of knowledge. Hopefully tomorrow will bear more tracker information.

The Tracker Demystified – Part 2: Structure

2008-10-21T01:01:00.000-07:00

Now it's time to get started on the coding itself. For debugging purposes, we're going to need a lot more than the single-line tracker status field in normal BitTorrent clients. Happily, since the connectivity is simply via HTTP GET, you could just type the announce URL into your browser's address bar to test the effect of various values. In fact, this is the way I did my testing last time I took it upon myself to write a tracker.

This time, I'm going a little more high class. I've made a simple form with fields for each of the variables (info_hash, peer_id and so forth) that is submitted to announce.php using GET. It's much easier to play around with different values this way. Since forms are pretty simple and not the subject of discussion here, I won't go into any more depth here.

First, even before connecting to the database, I'll load the passed variables and perform a minimum of validation. Like yesterday, I'll be referring to the protocol spec to determine the expected values and ranges of particular variables. I don't like throwing a ton of errors at my user, so I tend to design for graceful failure. For example, if event is anything other than started, completed, stopped, or empty, it will simply default to empty.

Speaking of event, it is going to form the basis of our code structure. The heart of the script consists of a series of conditionals based on the four possible variables. Actually, since empty merely denotes that the client is requesting more peers, we only need the three that require updates on the tracker's end: started, completed, and stopped. These three events map cleanly to three MySQL statements: INSERT, UPDATE, and DELETE, respectively. When a torrent is started, the peer needs a new row in the database, which should be updated when the download has finished and removed when the connection is closed.

Now, stopped only takes input – there's no point in making output when nobody is going to use it. For everything else, we'll use BitTorrent's bencode protocol (detailed in the spec) to output the proper response. Again, I encourage you to go back and peruse the spec. Even if you don't muck around in the backend much, it's nice to be able to pop the hood of any metainfo (.torrent) file you may come across and get a general idea of what's going on in there. The bencoded data may look intimidating, but it's perfectly readable if you take a few minutes to understand how it works.

So, back to the announce. The proper response I mentioned above is a dictionary consisting of two keys/value pairs: interval to indicate the number of seconds until the tracker suggests that you scrape again, and the peers to provide a list of the peer id, ip, and port of each provided peer. There's one bit in the official specification that I feel the need to quote for all the private tracker elitists out there:

Note that downloaders may rerequest on nonscheduled times if an event happens or they need more peers.

Announcing more often than this is perfectly acceptable practice. I do hate to hold it up for ridicule, but BitMe has ironically banned the Mainline (BitTorrent) client originally developed by Bram Cohen himself, developer of the protocol, because it "does not honor the protocol requirements of private trackers". I'm not going to knock on BitMe too much because they're more open than many private trackers in that regard, and they're the only one that I know of that lists banned clients and justifications for banning. However, allow me to take this opportunity to say... what the fuck?

That's a little off-topic. Anyway, now we just have to pull the data from the database. In practice, clients can and do indicate the number of peers they're looking for with the numwant variable, but that's not part of the spec, so I'm ignoring it for now. Instead, I've specified 25 as an arbitrary maximum, and used ORDER BY RAND() in my MySQL query to ensure that all peers get a statistically equal showing no matter where they sit in the database.

Now that we've got the pertinent data, it has to be output, so we'll bencode it according to the structure laid out in the protocol, and shove it out the door. I might talk about writing full functions to read and write bencoded data at some point, but it's a lot more work. The implementation in the announce is simple because we only have to account for the variation between different volumes of data, which is nevertheless being output in a rigid format. Super easy.

Now, while I may not have done a particularly satisfying of describing the last bit of the announce, it is in fact done. The finished script is only 29 lines, including whitespace. I had it heavily commented, but many of the comments repeated what I'm saying here, and I want to emphasize how simple an announce can really be, so I've left it blank and hopefully the minimalism will speak for itself.

At the moment, I'm running three peers: Transmission, Azureus, and my own web browser masquerading as a client. Transmission has successfully seeded the file to Azureus, and both are trying to connect to my web browser to share with it too. Obviously, since it isn't actually a torrent client at all, that's not going to happen.

Now, I see the seasoned coders cringing already, so I'll repeat this once again: this is a barebones demo script. It is designed to provide a framework for you to work off of in forming a more complex announce. The idea is to illustrate the way the protocol works, and the way the tracker interfaces with the protocol. As it stands now, the script doesn't even sanitize input, so it is vulnerable to SQL injection as well as just about every other damn thing.

Tomorrow, I hope to elaborate a bit on these issues, refining this code to create a perfectly usable announce. Ratio tracking is still outside the scope of the project, but we can certainly tighten things up and maybe lay the necessary groundwork for you to go on and add ratios and such frivolity. I say "I hope" because I'm burdened with a lot of work tomorrow and may not get time for a lot of coding. I'll make sure you get an article, and hopefully part 3 of this series, but no promises.

And yes, you're welcome to take the code I wrote above and use it without attribution for whatever purposes you want. However, if you're going to do that, I really suggest that you fix the holes first, or wait until I have a chance to do it for you.

The Tracker Demystified – Part 1: Building the Database

2008-10-20T01:02:00.001-07:00

You may have noticed a post on this subject a while back that was unintentionally released before its time. Well, if you have already read that one, re-read this anyway. It's finished, for one thing.

Anyway, this week is going to be devoted to building a simple tracker from scratch. This tracker will do no more than accept and share IP addresses, with no front-end for uploading and downloading metainfo (.torrent) files. However, if you've worked with PHP much, you can probably already work out how to do file uploads and downloads and pretty or at least usable interfaces. This is the big obstacle, and also the bit that has the potential to be interesting for the non-coder, if I can manage to write clearly enough to keep them engaged and slightly comprehending.

Hopefully, over the course of the week, I'll dispel any notions of BitTorrent as a complex or incomprehensible protocol. Once you get your head around it, it's actually quite easy to understand and use. The peer-to-peer bit is a little less straightforward, but happily, we don't have to deal with that. We're writing a tracker, not a client.

My resource in all of this is going to be the official BitTorrent spec, which I know almost by heart. I'll be referring to the spec from time to time, so read it over and my posts might start to make some kind of sense. My MySQL abilities are a little more touch-and-go, so optimization may not be as fantastic as it could be and I'd welcome any constructive criticism on that front.

On that note, we'll be starting today with outlining the basic structure of the announce through the creation of a database table for the peers. This is the only table we'll need, which is handy. Basically, we need to store the pertinent bits of the data that's received by the announce, and enough to provide a coherent response. There are eight variables passed by the client to the tracker: info_hash, peer_id, ip, port, uploaded, downloaded, left, and event. Official or unofficial extensions may add extra values, but we're writing a barebones tracker, so we can safely ignore them. The spec does a perfectly good job of clearly outlining the purpose of each of these variables, so I won't repeat it here.

Now, all of the provided variables are pretty important for various things, but again, this isn't a full-featured tracker, so we'll ignore some of them. We'll save info_hash so we can connect peers on the same torrent to one another, peer_id so we can distinguish one peer from another, and ip and port so we can share the user's address with others on request. As well, while we have no need to track ratios here, we do need to know who is seeding and who is leeching so that we don't waste time sharing seeds' IPs with other seeds. We'll call this variable uploader, and it'll simply be a bit assigned based on the test of left == 0.

At this point, our database looks like this:

If you're familiar with phpMyAdmin, you'll see that I've set a primary index on id, which is standard practice. I've also set an index on info_hash, since we'll be running a lot of queries for it.

Now, at this point, before the more knowledgeable members start tearing holes in my post, I want to point out that I'm working with an ideal model here. No information is lost, all clients follow the protocol to the letter (particularly in always cleanly closing connections), and there are no clients that spoof information for personal gain. Obviously, none of these are true, but since I'm aiming to explain an implementation of the BitTorrent protocol, I'm going to work with these assumptions for the time being, just like how friction is often ignored in introductory physics courses.

TorrentFries, inc.

2008-10-17T00:56:00.000-07:00

In the past, I've made a point of steering clear of the legal nuances of tracker-running. The law varies dramatically from jurisdiction to jurisdiction, and more importantly, I know jack shit about legal matters. However, I just thought I'd throw you a tidbit gleaned from one of my lawyer friends: the beauty of a capitalist society.

Basically, in the US, Canada, and the UK, corporations can only be sued for the assets of that corporation. This applies even if that corporation is just you and the assets are zilch. So long as you make it clear that you are always acting on behalf of the corporation, you're out of any legal troubles scott free.

Obviously, the practical implications are a bit more complicated, but that's my understanding, anyway. I invited him to write a guest entry for us, so if he takes me up on that, you'll get to hear a bit more. In the meantime, please take my "legal advice" with all the salt in the Dead Sea.

A Moderator's View

2008-10-16T02:00:00.000-07:00

Unfortunately, I fell asleep before I could write what should have been yesterday's entry. I hope you didn't miss us too much. I'd like to focus a bit on some experiences I've had as a moderator of TorrentFries. I wasn't always an admin or even moderator at TorrentFries. One day I logged in to find a full host of "shiny new buttons," as we like to call them at TorrentFries, and what a true surprise that was. It's a gratifying feeling to be an uploader and to give to the community, but as a moderator and admin, that responsibility takes on a deeper meaning of giving to the community. I'm glad to have been able to give in that unique way.

As a new moderator, some of my initial duties included moderating the forum, responding to torrent reports and weeding out torrents that didn't follow the rules. As a moderator team, it's obviously pretty important that everyone gets along so the work gets done without too much hassle. If things get slack and neglected, it's apparent right away.

For instance, at one time we were experiencing problems that required users to use the staff contact form to fix, but nobody was addressing the problems. When 25 or 30 messages piled up, it became difficult to manage. I eventually got enough time to clear out the problems, but it took a full 2 hours to take care of. The point is this: our team is good, but without clear direction or responsibility, things tend to not get done if there's a lot of work to get done. The admin has to either pick up this slack or divvy it up, but sometimes that choice has to be made.

Another problem we saw was spam in the forums. Thankfully, there were mods checking the forums, but our registration system didn't include CAPTCHA or any other deterrent to bots, so we began seeing a heavy influx of pornographic material posts. Without fixing the root of the problem, it was difficult to keep the issue under control, and with CurlyFries busy he couldn't rewrite a part of the login. No one else could do it, so we just kept getting bombarded with spam.

Save small issues like these two mentioned, the site as a whole could easily run itself without CurlyFries logging in at all. Nevertheless, there needs to be some direction and fiddling done by the sysop no matter how well the mods work.

Additionally, there always seems to be something that mods should be doing instead of bantering on the forums. It's difficult to moderate torrent comments with any efficiency, and even ensuring that the uploaded content follows the rules seems take a backseat at times. So, in short, as an admin, aim to get a staff that can run the site, but expect that they won't always do everything on their own accord.

How Not to Admin a Tracker

2008-10-14T01:12:00.000-07:00

I'm assuming most of the audience that is actually moving in the tracker direction is running or planning to run a small operation, for the time being at least. That means spamming all over the place for your site, and that means your previous identity (with all its slip-ups) is irrevocably connected with your site.

Chances are, when you first got involved in the P2P community, you were scared shitless. If you weren't, you should've been. If you were paranoid, you should have made a new account unrelated to any other you might have online. If you didn't, go do that now. Otherwise you're opening yourself up to a whole world of hurt. After a while, you start to relax, to become blasé about the whole copyright issue, and you start to get sloppy again.

Now take this unconcerned you and put it in charge of a BitTorrent tracker. You're probably a little concerned, but that goes away fast. It's just a small operation, after all. It can't hurt.

The problem is that small operations have a way of becoming big operations, and your behavior has a way of sticking around. Even if you've been careful to delete posts as you grow, the curse/blessing that is archive.org will remember your transgressions. All of a sudden, you're at the head of a booming site, and oh shit, what have you done? You go to cover your tracks, but it's too late, of course.

Chances are that if you're headed down this road, you're not far along. You can still change now. Don't make these silly mistakes. If you're going to be involved in your own community, which I recommend, create a new username for it. Don't let hubris enter into the equation. It's tempting to join other torrent sites and go "look at my awesomeness!". Don't. Your admin username should be used only in the context of your own site, and in your dealings with other sites. It's tainted in a different way, and it's dangerous to mix things together.

As well, it's helpful to have an alter ego. Create somebody that is plausibly of the same character as you, but comes from a dramatically different background. Trust me, it's a good idea. You're role-playing, and you should never drop that identity, even with your closest online friends.

Of course, this is the product of much painful experience. Do as I say, not as I do. Accessible as I want the prospect to be, it can be dangerous work, and you have to take it seriously, no matter the size of your site.

The Rules: Writing a Constitution

2008-10-13T03:02:00.000-07:00

Yesterday I walked into a shop and was greeted by a sign thanking me for not urinating in the produce. Wait, no I wasn't. You know why? Because there are certain codes of behavior that are implicitly expected of all members of a society. Yet somehow sites feel the need to inform me that I could be banned if I start stirring up hell in the forum. Great, thanks for that.

Quite simply, there are some things that just don't need saying. Sure, you're going to get trolls in your forum, but making a rule telling them to go away is going to do nothing for you. Rules are for the people that genuinely care about their standing with the site. The people that are just around to make trouble aren't going to be reading your rules, and certainly won't make a point of abiding by them. You can't legislate common sense.

So what are your rules for? They're for people like me. I'm a member of a bunch of different trackers, and whenever I sign up, I always take a peek at the rules before I get down to business. I know what a BitTorrent tracker is, I know the usual etiquette, but I just want to find out if there's anything special about how your site does business. Do I have to post in rhyme? Have a cute, fuzzy avatar? Download only certain torrents? I don't want to dig through a bunch of bullshit about obeying "the moderators [sic] expressed wishes! [sic]" in order to find out what really matters.

Conversely, a lot of sites that have automatic ratio bans (which I've ranted about in previous posts) won't detail these bans in their rules. The hardest and fastest (and most variable) rule of all is somehow overlooked. The staff don't have to touch it as the banbot does all the work, but it's the rule with the most impact on new users, good and bad alike. Whether or not I agree with the concept is a moot point, at least tell me.

You're writing a constitution here, not a criminal code. If you've never read your country's constitution, go do it. It's probably available online. In general, they're pretty succinct and accessible for all their importance, and you'd be surprised what rights are being violated. Your criminal code, by contrast, is probably a few thousand pages long. The constitution is basically a vague statement of purpose, while a criminal code gets into the nitty gritties. Since your moderators are not police officers bound by the letter of the law, you don't need a criminal code. You simply need to give guidelines by which your site should operate.

A lot of sites have 15-20 rules, mostly the stuff that came preset with TBdev or whatever they're using. In my mind, you should be able to get everything you need in less than 10, preferably 5 general rules and 5 upload rules, in addition to the behavior that one normally expects of a generic good member elsewhere on the internet. The fewer rules you have, the better.

That's not to say you should have people responding to your moderation actions complaining that they didn't know something was against the rules. If it's the sort of rule a person can't be normally expected to know, then by all means throw it in there. Otherwise, forget about it. You're entrusting your moderators with plenty of discretionary powers, so you don't need to spell everything out.

I'd finish up with some examples from TorrentFries, but quite simply, it's stuff that applies only to that site and to none other. That's kinda the point.

Leaving a Host

2008-10-10T00:58:00.000-07:00

There comes a time when you need to cover your tracks in a hurry. Since you often can't stick a boot CD in a dedicated host to do a DOD hard drive wipe like you should, you can settle for a few other options. It's nice to feel like you haven't left any incriminating footprints behind.

I find that the best tool for the job is secure remove. The command srm will allow you to do a 38-pass deletion wipe that's good enough for most purposes. Note that it's not impossible to recover the data deleted in this method, just beyond the reach of any small to medium sized organization.

Install secure remove:

$ sudo apt-get install secure-delete

Now, you could use this command to bash script a quick exit, but without forking off your processes, a quicker way would be to run each part of your secure removal separately. This works very well if you've got multiple cores or processors. You can use process management tools such as screen or background processes to run multiple srm commands at one time.

The three basic parts of your server you should consider deleting would be the MySQL databases, Apache www folder, and your logs. If you've installed any other software, be sure to check those out as well.

MySQL

sudo srm -rv /var/lib/mysql/

Web root

sudo srm -rv /var/www/

Logs

sudo srm -rv /var/log/*

You'll have to use your judgment on the rest of the system on an individual basis. Don't forget to delete any backups you've made. The last thing you can do is clear out your private files before you exit.

sudo srm -r ~/.*

That's all. SRM uses a lot of processor and takes some time on larger directories and files, so be sure to start your deletes in a relatively parallel fashion if you've got to high-tail it out.

Expect to see the series on "The Tracker Demystified" next week. Hopefully, if I write loud enough, that might happen. ;)

CurlyFries adds: Only stuff that you delete with srm will be securely removed. For this reason, do not drop your MySQL databases prior to hitting /var/lib/mysql with your srm stick. If you do so, MySQL will (insecurely) delete your database files, and running srm afterwards will do you no good.

Oh yeah, and hush up. I'm getting to it.

404 Not Found

2008-10-09T00:38:00.000-07:00

The requested URL / was not found on this server.

Apache/2.2.3 (Debian) PHP/5.2.0-8+etch111 Server at ... Port 80

Rest in peace.

January 4, 2007 – October 8, 2008

See you all on Friday.

Perils of Remote Access

2008-10-08T00:11:00.000-07:00

I thought since I've shared quite a bit of technical information in my past few posts that it was time for an admin's parable. I wrote a little about the perils of remote access servers earlier, but I didn't really go into detail about how 'normal' activities should be rethought in the environment of sole access being remote. Remote administration can be a wisening experience, unless you have some out-of-band administration tool (such as a DRAC card). With may hosts, you may only have the option to do a hard reboot or a reimaging (often at a fee).

In one case, CurlyFries was looking to add a static IP to the server to move critical services off the primary IP and domain. Most services, such as SSH, can listen on any address and can be restricted to certain addresses. It's handily secure to only have ports 80, 443, and 5222 open on a heavy-trafficked IP address.

It's fairly straightforward to add a static IP to a system. I would never dream of being timid about making such a change. At least not nearly as timid as when making a critical Apache or MySQL configuration change. Nonetheless, I managed to screw up with a typo in the /etc/network/interfaces file and got a serious error when restarting the networking. Thankfully, I hadn't lost terminal access, so I went back to look for the problem.

When I thought that I found it and fixed the problem, I restarted networking again. No joy. I was well aware of the implications of having a bad network interfaces file: no network interface access. So, I restored the backup configuration and restarted the service. At that point, my terminal died and I was then unable to establish any SSH connections to the server. The web server was down. Confounded, I cursed Debian for being so ridiculous and hoped that the configuration had taken effect. Within the hour, we had our TorrentFries rebooted, and to my relief the original configuration was working.

Had the networking stopped working when I tried to load the invalid configuration instead of when I loaded the original working configuration, we would have been toast. Not only would we have been forced to do a complete reinstall, but we would have lost quite a bit of data in the process due to not moving the rather infrequent non-automated backups off site regularly. *cough*

We barely escaped a disaster that would have spelled the second time in my capacity as an admin to need an OS reinstall on the remote server. The only time we've had to reinstall was when a certain curly type of administrator dropped the MySQL privileged users and my consequent mucking with things resulted in an inoperable system. The lesson to take away is threefold:

Backup offsite before making any changes that could effect the operation of the site.
Be mindful of what senarios could leave you with no server access. These generally include but are not limited to network configuration, SSH configuration, firewall configuration, and boot loader configurations.
Don't close an open terminal session unless you're sure you can get back in. After I make some changes, I open a second SSH session to ensure access before exiting out of my working session.

I hope you found that at least mildly entertaining enough for you to remember when it's your server you'll be losing access to. There's nothing quite as disappointing as the realization of failures easily avoided.

Peer Management

2008-10-07T00:38:00.000-07:00

Okay, I'm not in fantastic condition at the moment, but I want to keep things going. I'll just grab on to a recent suggestion – discussing ratios in a community context – and run with it.

Basically, any simple, closed system will be balanced, no matter the context: total uploads will equal total downloads. No matter how your tracker records data, it will always be tracking as much uploaded data as it does downloaded data, neglecting hash fails.

Therefore, when the ratio tracking system is altered, you will do nothing but alter the community dynamic. There will always be supposed gormless bottom-feeders at the bottom of your scale, and theoretical paragons of virtue at the top. Sorta. In my experience, chronic leechers are mostly those that simply haven't been initiated into proper torrent etiquette. I know in my early days of torrenting, I was quite unnerved by the prospect of others being able to connect to and download from me. To be sure, I didn't go out of my way to do this, until it became a point of personal pride to maintain a ratio above 1.0. It is only my involvement in the community that drove me to do this.

Now, think of an open site like The Pirate Bay. The tracker is simply a collection of its parts, each behaving according to their own conscience. The leechers leech according to their desire for gratification (or lesser experience, or lesser ability), while the seeders do so because of pride or generosity or simply because the swarm is so slow they end up uploading far more data than the download size by the time they complete. And you know what? It works. It may not work well, but it works.

Now, ratio-obsessed private trackers choose to whack off that bottom demographic, the people that are disinclined to share. That's all well and good, but you simply move the community's scale. Instead of having the true leechers at the bottom end of the scale, now you have the metaphorical middle class down there. In my experience, such systems tend to drift ever upwards as users are banned for poor ratios – in other words, as the bottom end of the spectrum is chopped off again and again. Obviously, these systems rely on new registrations to add a continual stream of new members to slow or halt that drift.

This is compensated for by free leech days, golden torrents, what have you. However, the only people that will participate in this activity are those that are genuinely concerned about ratios. They're not downloading torrents because they want to, but because it will improve their status in the community. To me, this is contrary to the purpose of file sharing. You share what you've got because you want to share it, and people accept what you're sharing because they want to have it.

Basically, these events are designed to fix a system that doesn't work properly all the time. If you need to schedule special events to compensate for your model, you must be doing something wrong. I would prefer to have a system that works in the same way and with the same effectiveness every day, rather than just once in a while.

So where's the solution? I don't have an answer. I don't think anyone does. I certainly do use a couple of private sites, but I also use the big boys when they have what I need or when I don't feel like destroying my ratio trying to seed back. "If it ain't broke, don't fix it," comes to mind, but I can't deny that swarms on private trackers do seem to perform better, so clearly something has been fixed or at least improved upon. Personally, I would prefer to see more personal treatment towards leechers. I believe that most leechers simply haven't been properly introduced to the social protocol (heh) associated with torrenting. If someone is actually intentionally and knowingly violating accepted behavior, they will take pains to hide it by using a cheating client of some sort. Therefore, wouldn't it be logical to take some means to inform the uninformed? I prefer not to participate in a community that is ultimately built on fear. It smells bad.

This is just meant as some food for thought, and hopefully will dispel some feelings of superiority on either side. Again, I'm sorry for my unfocused meandering, for my inability to make a solid point, and for any typographical errors I may have made. I'll probably read this over sometime when I'm in better intellectual shape and say, "I can't believe I said that," but there's nothing to be done. At least you got a post.

I had intended to write my tracker tutorial this week, but ah well. I'll do it sometime in the future. Stay tuned for that.

We Interrupt Your Regular Programming...

2008-10-06T14:45:00.000-07:00

Sorry for the service interruption. My internet was offline yesterday, so I couldn't publish a post, and it seems that my organization was working against me as Blogger automatically published a draft I had scheduled for today. We'll return to our regularly-scheduled programming tomorrow.

BitTorrent Communities

2008-10-03T01:15:00.000-07:00

I didn't want to bore you with too much Linux, so today I'll write about the elements of a BitTorrent community from the eyes of a member of a handful and an admin of one. I would encourage anyone that still reads our posts to comment with any questions you might have, and we will try to address everything we can.

When analyzing BitTorrent communities from an admin perspective, I often consider the following goals.

Maximize uploaded material quantity
Maximize upload quality
Maximize user involvement

This three-legged stool seems to have a visible balance. In the case of large tracker sites, often there is more quantity paired with less quality and user involvement. Smaller private sites sometimes have higher levels of quality and user involvement, but sometimes less quantity. These three legs on my stool help each other. For instance, more user involvement helps upload quality and quantity. Good upload qualities and quantities attract user involvement. When creating a community, it's critical to have a clear idea of your goals for these three items and let that define the 'feel' of your site.

Maintaining higher standards on torrent descriptions coupled with user comments improves at least perceived quality. When users have a way to gauge the details of the torrent data without downloading, they'll be more comfortable engaging. Have an explicit example of how torrent descriptions should look on the user upload page. Instead of simply titling your comments section "Comments" on the torrent page, consider something like "Comment on this torrent's quality".

When users aren't downloading or uploading material, what are they doing? Many sites consume users' excess time through interactive forums and shoutboxes. A lively forum can be the mainstay of a community. Conversely, a dead or hostile forum can turn away would-be outstanding members away. Be friendly and don't let users or moderators become short with users even if they are n00bs, the whole community will align on the staff and prominent posters' viewable culture.

User interface is key. Make it easy for users to drill down the material they're looking for using categories and subgroups. Search is the most important thing for a torrent site, so make sure it is prominently positioned, works well, and is predictable and consistent. Otherwise, negative results are a reflection on the material tracked. Also, I can't stress the importance of a good site design. Mediocrity won't get you very far when attempting to gain users and material early on.

As mentioned before, ratio tracking helps to mentally encourage users to give back to the community. Distinguish uploaders and otherwise exceptional members in the community or figure out a way to give them a warm fuzzy feeling. Make this visible wherever users interact.

The overlying theme is culture. Define a culture you find appealing and align your entire site around it. Culture exudes itself in every aspect of your site from the site rules to each forum post, so take some time to think about what you want to display. A little psychology and management will go a long way.

Protocol

2008-10-02T01:10:00.000-07:00

Okay, this is going to be a quick post because OnionRings and I both have a ton of work to do and Ketchup seems to be maintaining our collective laziness quota.

I'm planning to offer a detailed examination of what exactly goes into the technical end of a BitTorrent tracker by offering explanations as I write my own, but that's well beyond my time allotment for the evening. Instead, I'll write a simple summary of the protocol.

The peer-to-peer aspect is not important for our purposes. Unless you're planning on writing a client, which is well outside the scope of this site, you don't even need to think about it. I know I don't. I really haven't taken the time to study it in much detail, aside from understanding what's going on in general terms.

Happily, the peer-to-tracker protocol is very simple to understand. The tracker is inherently passive, never initiating connections itself. Instead, clients connect to the tracker via HTTP. They provide the infohash (unique identifier) for the torrent they're trying to download, their current port, and a bunch of other information indicating the state of the client. The tracker responds with a simple list of IP addresses that are also downloading the same file. It's up to the client to initiate connections with the provided peers. Since the protocol is peer-to-peer, any client can initiate the connection, assuming that the other computer is properly configured to accept incoming connections.

That's all a tracker has to do: log the IP addresses that connect, and respond with a list of suggestions for possible peers. The process is fast, light on resources, and very easy to code. You can develop a tracker just like any website, using just about any language you want: PHP, ASP, Ruby, whatever. The biggest trackers run on C or other such languages, since it's fast and lightweight beyond the dreams of PHP. However, for your purposes, web languages are all you need.

That said, I'll pick up on the subject later on when I actually find the time to get going on development. Like I say, things have been nuts of late.

Back to the Basics

2008-10-01T02:01:00.000-07:00

I realize that our focus has of late moved from tracker-specific discussion to more-general stuff that applies to any site or server and can be found in a bazillion places online. Since this blog is unique because it does chronicle the operation of a BitTorrent tracker, I'm going to get back to the meat of the thing for a bit. OnionRings plans to continue with his Linux series, but is sadly indisposed this evening.

In some senses I will cover some of the same material as our second-ever blog entry, but with a different focus. While that entry gave a quick overview of why and how to set up a tracker, I'm going to look a little more closely at the technical and logistical aspects of bringing a tracker online.

First off, what do you personally need to bring to the table? (You're probably the only one at the metaphorical table at this point, but the bringing should commence now.) Beyond anything else, you need to understand the BitTorrent protocol, and understand your tracker of choice. We'll talk a bit about choosing trackers shortly. If you plan on running a PHP/MySQL tracker, you'd better have at least a working knowledge of PHP and MySQL. From the get-go, you'll need to be able to peer into the inner workings of your site and see what makes it tick. You don't just need an understanding of the protocol, but of your own tracker. Trust me, admin panels are nice, but some things just require you to get down and dirty. However complete your chosen script may look at first glance, there will quickly come a time when you or your users bemoan the lack or poor design of some feature or another, and you will be forced to go in and remedy the situation. As the tracker grows and moves to its first dedicated server, it will also become important to know your way around Linux to some extent.

Secondly, you need a hook to pull people in. The most obvious aspect is to provide something that others can't. A unique idea is neat, but it's entirely possible to carve out a niche in a "market" that is well-developed but not saturated. For example, if you have terabytes of obscure movies that you have spent years collecting, starting a movie tracker to share these can be an excellent starting point. Of course, you will need to keep all the uploaded torrents active, even though you will be getting very few peers at first. For this purpose, you may want to consider renting a seedbox. The added cost may be hard for a brand new tracker admin to swallow, but your ability to saturate the connections of your first members will do a lot to encourage them to stick around and maybe even contribute some stuff of their own. As well, a unique design may sound frivolous, but it will lend your site credibility and a sense of longevity. I've covered some methods of early promotion in the post I mentioned above, so I won't rehash them here.

I hate to be defeatist, but if you can't meet these requirements, you should think twice about starting a tracker. Of course, the lovely thing about being a human is the capacity to learn, but you should get going on that learning well before you even consider getting into tracker territory. While I'm trying to demystify the role of the tracker administrator, it's certainly a job that not everyone has the skill and disposition to fill with any great success.

Moving on to more technical requirements, you will also need to choose a tracker. That's a given. The three most popular trackers under active development are TBdev, TorrentTrader, and the new Project Gazelle, although there are a ton of other options out there, so you shouldn't feel constrained to choosing one of these three. Shop around a bit. Detailed comparison of these and other trackers is well outside the scope of this entry, and perhaps even of this blog, but this should point you in the right direction. At the end of the day, it's up to you to select the tracker that best suits your particular tastes and needs. As a side note: for the love of God, don't use TorrentTrader Classic.

So, why not just boot Azureus or µTorrent or one of these handy ubiquitous little torrent clients that happen to include the ability to operate as standalone trackers? Hopefully this isn't a question you were asking yourself, but I'll answer it anyway. First, this isn't the purpose they were designed for. Just because they can run a tracker doesn't mean they should. They're not optimized for the purpose, they don't include features that are necessary for the smooth operation of a tracker (they're inherently open to anyone that wants to use them), and they include a ton of features and bloat that will just bog down your server. What's more, part of the sort of tracker we're discussing here is the index, or frontend. This is the bit where torrents are uploaded, downloaded, where users interact with one another, all that sort of stuff. The tracker itself is ridiculously simple in operation, as we will see in my eventual piece-by-piece breakdown of what it takes to build one from scratch. The important business is the complex frontend, and BitTorrent clients' wannabe trackers just can't provide this.

So, once you've selected the script that best fits your needs, it's time to choose a host. I've had some requests to list a number of torrent-friendly hosts, but that's not something I'm going to do for a number of reasons. The most important reason here is that I don't want to endorse a host that turns out to be unsafe, or to lull readers into a false sense of security. You may have noticed that even (especially) the biggest trackers tend to move around a lot. The world of web hosts is intrinsically volatile, so to proclaim that one host is perfectly safe, even if presently true, is to ignore the fact that this may not continue to be the case in the future. LeaseWeb was once a safe haven for trackers, but after losing a lengthy court battle on behalf of Demonoid and other trackers, all the will to fight has gone out of them. Nowadays, if a copyright holder says "boo", they'll turn around and shut you down.

So simply accept that no place is perfectly safe, and operate under the assumption that you could be taken down tomorrow – you could be. A good strategy to find places that are safer is to run whois lookups on established trackers' IP addresses. Presumably, if they are able to operate on a particular host, that host must be somewhat resistant to legal threats. The only problem is that many larger trackers own their own servers which they operate via colocation in the host's datacenter, while you are likely unable to afford more than a rented dedicated server (an important distinction to make). Not all hosts supply both options, and not all do so for reasonable rates. In addition to colocated and dedicated servers, I've talked about shared and virtual hosting in that previous article, so I'm not going to rehash that here.

This should be all you need to get you off the ground. Again, I'm not going into exacting detail on some of the technical points because I'm assuming that you will be able to handle the fiddly bits of your own accord. If you can't, like I mentioned, you probably shouldn't be running a tracker.

If there are any points on the subject (or any other, really) that you would like us to cover, leave a comment on this post and we'll do our best to touch on them in future posts.

Site Administration Tools

2008-09-30T13:55:00.000-07:00

Monday I wrote a technical piece about installing LAMP so today I'll talk a little about the type of administrative tools we use on the site. I've always been a big fan of using the tools UNIX gave us. That means strict command line pleasure. It means things are clean on the system and no extra resources are used or security holes opened. But not everybody agrees. Even I find it nice to use a GUI once and awhile so as not to get bogged down with the mechanics of the command line.

MySQL seems to be the toughest to administer with the tools UNIX gurus gave us. Writing SQL statements isn't bad if you're familiar with SQL, but you'll likely feel better with PHPMyAdmin or MySQL Administrator (MySQL Query Browser). The difference is that PHPMyAdmin is web-based and MySQL Administrator is a remote client. To me, the difference is that there's no garbage on the server when you use a remote client to connect to to your server like there is with installing PMA.

Consider the security implications as well. When installing PMA, you open up an avenue for attack via the web. When using MySQL Admin allowing remote connections, you're doing even worse. Therefore, the safest and most efficient option is to tunnel your MySQL Admin connection over SSH to get a localhost connection using a remote tool. I think this method applies for other remote tools as well. Ketchup likes using MySQL-Front. Using a remote tool allows room for preference.

In other areas, it can be handy to have a GUI as well. If you have many servers running on your box (such as mail, Jabber, and Apache/MySQL) it can be handy to increase your proficiency and ability to edit obscure configuration files with greater ease by installing a management tool. I've used Webmin before, and I have mixed feelings. It gives very good control over the multitude of configurations on a Linux box, but I always see it as another security vulnerability.

That's all for now, unfortunately. Next time I post, we'll discuss more about LAMP configurations.

Linux Part 2: Installing LAMP

2008-09-29T00:30:00.000-07:00

I'm going to resume from Linux Part 1: Installing Configuring Sudoers and OpenSSH with part 2: Installing LAMP. In the world of Ubuntu, virtually no one installs LAMP from a command line on Ubuntu Server. Ubuntu provides the option to automatically install LAMP during OS setup, so most of the time, it's done that way. Purchasing a dedicated server often doesn't give you that option, however. Yes, many tutorials cover installing LAMP, and they are just as good, but note that they're installing on Ubuntu Desktop, sometimes assuming you have a GUI. Thankfully, installing LAMP on Ubuntu from the command-line isn't hard at all.

Let's start by running the best command ever, apt-get. This will get all the packages we need.

sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 mysql-server libapache2-mod-auth-mysql

Alternatively, you can try

sudo tasksel install lamp-server

Apt will download the packages and install. Then, you'll see this as MySQL is being configured:

You'll have to confirm your password, and then you're done. Restart Apache.

sudo /etc/init.d/apache2 restart

Let's make sure everything went okay. Browse to your IP address. You should see an "It Works!" page. This means Apache is running. Let's check PHP. Run

sudo nano /var/www/test.php

Write out this single line to the file:

<?php phpinfo(); ?>

Browse to your addresss/test.php and you should see lots of information about the current state of PHP.

Securing MySQL

That installation was so easy, we'll do a little security cleanup too.

Log into MySQL:

mysql -uroot -p

Your prompt will change to mysql>

Create a user you'll be using to administrate the databases. By default, MySQL is secure in the fact that it will only allow root to login locally. That's excellent until you realize that MySQL can be a pain to administer locally. I don't mind too much, but some people like to install PHPMyAdmin. This can open you up to potential remote access attacks on the root user's password. Therefore, if you're going to install PMA, we'll get rid of the root user to make it more difficult to enumerate users. Let's start by creating a MySQL account, Zeus.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'localhost'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;

You should consider also adding the privileges to this user @ 127.0.0.1

mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'127.0.0.1'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;

At this point, you should create a user that will be the one accessing the database. Using your "God" user to access MySQL from your web application is bad karma and invites security risks. Google 'create mysql user' for the syntax on creating a user with less privileges.

Feel free to be creative with your names...
Flush the privileges and exit.

mysql> FLUSH PRIVILEGES;
mysql> quit

DO NOT ATTEMPT TO FORGET THIS PASSWORD! It's a pain to restore a MySQL root password, and even more of a pain when the root user doesn't exist, (some say impossible) so take it from me-- remember your privileged user's password for cripes' sake.

Login and delete the root and anonymous user once zeus was created.

mysql -uzeus -p

mysql> DELETE FROM mysql.user WHERE User = ' ';
mysql> DELETE FROM mysql.user WHERE User = 'root';
mysql> FLUSH PRIVILEGES;

Check out your work:

mysql> SELECT User, Host FROM mysql.user;

Don't drop the debian-sys-maint account.

Apache and MySQL Control

Here's how you can restart Apache and MySQL without rebooting your server if you have changed a configuration file.

sudo /etc/init.d/apache2 restart
sudo /etc/init.d/mysql restart

Replace 'restart' with 'stop' or 'start' if needed.

Thanks for reading again, next we'll show you a bit about the different administrative tools available for Linux.

Terminal Case

2008-09-26T00:28:00.000-07:00

I was going to continue OnionRings' series on Linux configuration, but I don't have access to a clean Ubuntu install at the moment (or any other kind, actually), so I'll leave that topic for now. Instead, I'll rant a little about the future of the MAFIAA, and the RIAA in particular.

The popular opinion among pirates is that downloading music hurts no one. I've heard a variety of justifications for this, but the most popular opinion is that the free publicity (which presumably leads to album sales) counteracts lost sales. I'm no expert, but you're sure as fucking hell less of an expert than I am. What I will say is that, if this is true, it's true only for small bands and independent labels. The big boys can afford to buy as much publicity as they want. So they're right, it is hurting their sales. It's not hurting their sales by the number of copies downloaded times the full sticker price of the album, but it's hurting them nonetheless.

Don't mistake this for a bad thing. The big labels would have you believe that without them, there would be no music. Listen, bud. There was music before you came along, and there will be music long after you are laid to rest. What we will see the death of is McMusicians, singers and bands that everyone has heard of and that the most gullible of us consider to be the epitome of good music. You know who I'm talking about.

What will be left without the big labels? Musicians that aren't in the business of making millions of dollars. Record labels that are employed by the musicians, not the other way around. A world where the music industry is easier to break into because people are more open-minded about music they haven't heard on the radio at least 15 times. A world without the fat, corrupt husk that pretends to serve the needs of musicians and music-lovers alike.

The best bit of all: you can live in that world today. Listen to internet radio and find new bands. Go anywhere with live music, be it a concert, bar, club, coffee shop, or street corner. Buy independent records from independent record stores. Learn to play an instrument yourself! The labels aren't afraid of downloaders. They're afraid we'll realize that we don't need them. Downloading music won't hurt them as much as the realization that you never need to buy anything from them, ever again.

Naturally, all of the sentiments above can be directly applied to the film industry as well. Notice how movies are all shit these days? Try independent cinema. Trust me, there are still movies being made that don't suck.

Okay, that's all for now. I'll see you on Monday, or better yet, OnionRings will. I'll see if I can make him do an article next time.

A brief update: apologies to anyone that was expecting this to be a pun and for me to cover command-line stuff. I think OnionRings is indeed planning to get his series going again for Monday, and I'll try to get a little further ahead on my posting as well.

Linux Part 1: Installing Configuring Sudoers and OpenSSH

2008-09-24T21:40:00.001-07:00

Okay, I'm going to start you off with a little advice for beginning Linux users looking to get rolling with a torrent site such as ours. Note that startup sites will most likely look for shared hosting whereas the following advice is aimed at admin's moving from a shared host to a virtual private server (VPS) or a dedicated server (dedi).

In these posts, I'll be customizing the advice for a PHP/MySQL tracker such as ours at TorrentFries. I'll also be running under the assumption that you're running Ubuntu Server Linux, specifically Hardy Heron 8.04. Additionally, these are pretty common instructions you can find at a variety of blogs across the net.

Let's just get some things straight before we get technical. If you're new-ish to Linux, I'd recommend Ubuntu. If you disagree, you don't need to be reading this anyway. Ubuntu isn't the most rock-solid distro available, it's just the easiest to administer (in my realm of experience) and it's stable enough that I've never crashed it on accident. It can also be found with most hosting companies. If not, Debian is the closest to Ubuntu and feels nearly the same.

If you're concerned, check out these survey-based distribution choosers:

Logging In

You'll find that your hosting company will give you nothing more than an IP address (or a couple if that's the case) and a root login password. This means that you'll be accessing a server via Command Line Interface (CLI) over Secure SHell (SSH).

On Windows, I recommend using PuTTY for terminal access.
On Macs, you have Terminal!

Starting Configuration

Once you're logged in, create an admin group.

groupadd admin

If the admin group already exists, that's fine. Now create the first user with -g being the primary group (often just the username), -G being secondary group (admins), -s is the location to the shell, bash, -p is the password (you can omit this flag if you want), -d being the home directory, and -m creates the directory if it doesn't exist, and finally the username itself.

useradd -Gadmin -s/bin/bash -pPassword -d/home/UserName -m UserName

The password you just set can be tricky, so to be sure...

passwd UserName

You can add any other users in this fashion.
Let's give your new user sudo access first. Sudo lets you run commands as root while not logged in as root. Once, you'll have to use the UNIX text editor 'vim', but in the future, you can use a much easier text editor called 'nano'.

Edit: Ubuntu Intrepid Ibex (8.10) now uses nano for visudo.

visudo

Here's a quick tutorial on vi(m). Only use visudo when editing the sudoers file. Using any other method may cause corruption in the file, so make it a habit to use visudo.
Make sure your sudoers file looks like this in the group section:

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

Now that you have sudo access, try connecting with a new session. You should now see something like

UserName@host$

instead of

root@host#

Let's use our new user from now on.

Configuring OpenSSH

Now, lets lock down SSH. First, copy the config file so we can restore the original settings at any time.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original

Now, let's edit the sshd_config file with nano text editor.

sudo nano /etc/ssh/sshd_config

Look for:
LogLevel INFO

Change to:
LogLevel VERBOSE

This will allow us to get more information about SSH activity. It's handy to see who's logged in and who's tried to log in. Run grep -ir ssh /var/log/* | less to get that good data you're collecting.

Look for:
PermitRootLogin yes

Change to:
PermitRootLogin no

This will prevent hackers from brute forcing the root password, since root login is disabled. They'll still try to enumerate your users, but this is a critical step.

Look for:
LoginGraceTime 120

Change to:
LoginGraceTime 20

Changing the login time will slow down any would-be automated attacks.

If you'd like, you can change the port you connect to on SSH if you feel better being more obscure. Just change the 22 to what you wish:

Port 22

Save and exit the file and restart OpenSSH:

sudo /etc/init.d/ssh restart

You won't notice any changes until you reconnect. When you do, remember that you won't be able to log in as root. If you need root, you will have to use sudo or su. Also, if you changed your port, remember to change it in PuTTY when you reconnect.

For more info on OpenSSH, check this out.

Up next: installing and configuring LAMP.

Guest Post: To Track Ratios, Or Not To Track Ratios

2008-09-24T01:04:00.000-07:00

I'm a little short on time tonight, so I'm going to share a brief essay, courtesy of a friend of mine. He's quite knowledgeable about the BitTorrent protocol, and fairly critical of the status quo.

Talk about things that make you go hmmm...when TorrentFries was a private tracker, only allowing members to upload and download, they collectively reported uploading 100x more than they reported downloading.

The fact is it's impossible to track ratios accurately--the BitTorrent protocol wasn't designed with statistics in mind and doesn't support them reliably. Ratio updates are not infrequently lost and it's easy to fake ratio, undetectably, using any client.

Given its unreliability it is counter-productive to use ratio as a basis for rewards/punishments. You encourage cheating and end up rewarding the dishonest and punishing (by way of not rewarding) the virtuous.

You can try to identify cheats, but someone with a sky high ratio may have a seedbox and an inclination to support your community. At least until you mistakenly accuse them. And, of course, the smart cheats don't do so outrageously--so they never stand out in any way.

If you don't track ratios? The Pirate Bay continues to grow in spite of not doing so. Certainly their seed:peer ratio is lower than that of any ratio tracker, but their torrents survive. Download speed isn't everything--if people can wait weeks or months for a DVDRip to come out they can survive a day or two of downloading.

In spite of all that, I recommend you track ratios. They remind people that sharing is what torrenting is all about, and they are mostly accurate. Just don't make a big deal of them and, when people do complain, admit their failings and point out that they shouldn't be taken too seriously. Don't accede to request to "fix" people's ratios though--you open yourself up to a lot of work with no real benefit.

While I'm on the subject, I'd like to debunk a common myth--that it's impossible for everyone to have a ratio > 1. The theory goes that since everything which is uploaded has to be downloaded, the average ratio must equal 1. So, for everyone who has a ratio > 1 there must be someone who has a ratio < 1. So, people with low ratios are victims of bastards who seed too much, who should be pitied rather than punished.

Nonsense. The theory only works if you assume that nobody ever leaves--totally unrealistic, particularly if your policy is to ban people with ratios < 1. Even if nobody ever left, so long as momentum is maintained (ie. so long as people upload within a reasonable period of downloading, thus restoring their ratios) it would continue to fly.

Green Salad
Not an administrator. Not always wrong.

One thing I'd take exception to is his claim that the global ratio does not always have to be 1.0. It does when you factor in users that have left or been banned. If something is uploaded on a private tracker, another user must be downloading it. In my mind, it's also a little incongruous with his arguments against rewards or penalties based on ratio, since his argument relies on users being banned for having poor ratios. Since the global ratio must inevitably be 1.0, someone will inevitably end up banned. Allowing people to register with the expectation that some will receive summary bans seems pretty ridiculous.

That's all for now, folks.

Staff Privacy

2008-09-23T01:22:00.000-07:00

Running a tracker can gather quite a bit of heat under your persona, especially if you live in a country that dislikes sharing the type of material on your site. It's good to insulate yourself as an admin from these type of things as much as possible, so be diligent and conscientious. Think ahead for the long term when exposing yourself. There was a good writeup on this sort of thing at TorrentFreak, so I'm going to expand from there.

Encryption is king, but nothing quite takes the crown like OTR messaging. For chatting between staff, running OTR offers encryption, authentication, deniability, and perfect forward secrecy. As a supplement and/or alternative, it's very convenient to run your own private XMPP chat server such as OpenFire or Jabberd. XMPP/Jabber allows greater control over configuration and supports SSL encryption.

Paranoia is a positive feeling as a admin, and nothing satiates that feeling quite like running your network connections over Tor, the encrypted onion routing software. This works well when you want to obscure your location or ISP. Practically, it's a difficult thing to put up with considering you'll likely see a significant drop in network speeds over Tor. Nevertheless, it's possible to run two browsers or their mobile package in order to not disturb your normal surfing. Using a VPN solution such as OpenVPN is another handy option to encrypt traffic or create a tunnel.

As a brief aside, never torrent over Tor. Despite the tempting alliteration in the names, they do not go well together. Tor is, by and large, extremely slow, and running BitTorrent through it is a gross abuse of the bandwidth donated by willing volunteers that believe in the free flow of information. Tor is slow for all of us, and if you're torrenting on it, that speed is your fault.

Anyway, if you're like me and don't trust a free email host with your information, set up an email server on your box with webmail. The most easily configured mail setup I've come across is Postfix with RoundCube Webmail, but be aware that the learning curve for the inexperienced is daunting. I still suffer headaches and driving pains to my geeky ego. :P At any rate, this allows you to send and receive site-related email without worrying about your origin or alias. Note that it wouldn't be wise to send email via your server's SMTP service as this could expose your IP in the mail headers.

If you've been good about obfuscating yourself, you won't need to delete server logs that might point to an IP you used, but it's reassuring to not log those things just in case. Scrutinize both Apache and SSH server logs as well as any other services you connect to.

I hope I've provided you with some options to protect your privacy. The best summary would be to quote the TorrentFreak article, "Identity is everything , If you never tell anyone, no-one will ever know."

Choosing a Server

2008-09-22T00:03:00.000-07:00

BitTorrent trackers are strange animals. Servers are generally pretty well-rounded in terms of system resources (hard drive is proportionate to memory/processor/bandwidth), but trackers don't use each resource equally.

Trackers use very little bandwidth and disk space, but are comparatively heavy on processor and memory. The announce/response and scrape/response cycles contain very little data and are only infrequently initiated with any given peer. By bandwidth alone, it would be theoretically possible to track tens of thousands of simultaneous peers on a residential internet line. Obviously, a consumer ISP would be unable up to handle that volume of connections, but that's a pretty compelling illustration of the bandwidth situation. Similarly, since even a complete BitTorrent site (index + tracker) only needs to serve small metainfo (.torrent) files, the site and database combined for a reasonably-sized tracker should be no more than a few gigabytes. However, the sheer amount of traffic involved in running a tracker makes for a fair bit of work for the memory and processor.

To estimate the traffic on any tracker, remember that each peer will connect to the announce roughly once every 30 minutes, or however long your tracker's announce interval is. The protocol specifies that clients should connect more often when sending event=started/completed/stopped, so you'll have to round up a bit to compensate. Basically, just divide the number/estimated number of peers by the announce interval in seconds (30 minutes * 60 = 1800 seconds). This will give you a rough idea of the number of hits per second on your announce. On TorrentFries, the announce accounts for roughly 50% of the site traffic, 30% for scrape, and 20% for all other activity. Simply double the result to get a rough idea of the total number of hits on the web server.

hits per second = (peers / announce interval) * 2

This doesn't tell you a whole lot in terms of servers, though. However, since site growth is generally pretty gradual, there's no need to worry about what you'll eventually be running on. Just start basic and upgrade as the need strikes.

Our first server had an AMD Athlon 64 3500+ processor with 512 MiB of memory, a 1500 GiB bandwidth cap on a 100 Mbps line, and a 160 GB SATA drive. It was pretty rudimentary, but worked fine to about 8,000 peers (10 hits/sec). Right now we're tracking 25,000 peers on a dual Xeon machine with 4 GiB of memory and two hard drives in a RAID 1 array, and we're looking at about 10% utilization. It's got a bit of room to grow.

If you're curious to see what makes bigger sites tick, take a look at The Pirate Bay's setup.

Tough Calls and Conflicts of Interest

2008-09-19T12:11:00.000-07:00

Okay, I figure I may as well finish the week off with a subject that's something of a continuation from yesterday's discussion of black-and-white dealings with members. As "nowhere" said on Monday, this is the sort of thing all staff should already know, although in my experience this is not always the case. Anyway, sometimes things aren't so clear, even for the more levelheaded staffers out there.

If an argument occurs between two established members, it can be a difficult call to make. The best choice with heated discussions is to lock or delete the thread in question. If the thread is locked, include a simple post saying something like "this discussion is getting out of hand". Don't take sides. Usually, this is all it will take. It's the topic's fault, not the members'. If necessary, send each involved party a cautionary message. Don't get into a "who started it" game, or allow the members to pull you into one. Depending on the circumstances, you could caution one member against baiting others and another against rising to the bait. Warnings are rarely necessary where otherwise-upstanding members are concerned, since they have a stake in the community and should understand that they should behave accordingly. However, repeat offenders should be officially sanctioned no matter their status.

That's still something of a no-brainer. Here's the bigger point, and the one where I slipped up as I'll tell you in a minute: don't get involved. Even if the subject of discussion is concerning the site, do your best to remain impartial. If it's a topic of some concern to you, step back and ask another staff member to deal with it. Similarly, don't use the locking method I described above to foreclose debate. Unless a subject is clearly inciting a polarized and heated debate (as in the example I gave on Monday of the "What do you think about homosexuals?" thread on Facebook), it should be allowed to run its course unless it gets out of hand. Just because you may not like the way the discussion is going doesn't give you any right to bring your extra powers to bear.

Sometimes, you just want to leap into the fray. Hey, a good argument can be fun, so long as it remains civil. However, if you're going to join up, leave your badge at the door, and make it clear to the members that you are doing so. If necessary, message a fellow staff member asking them to keep an eye on the thread and moderate any or all participants if necessary, even you. If you bring your powers into a discussion, your relationship with members will change forever. It can't be undone.

This is the point where I bring in my own painful example, where I became involved in a discussion and a user ended up leaving the site for good as a result of my behavior. During a political debate, I brought up my rather polarized views against Israel's foreign policy. The other user responded with equally-polarized view in support, and we fell into discussing the country's enemies. Eventually, I felt that the other had become quite offensive, both to the Islamic religion and Arabs themselves. I issued a summary ban. I realized almost immediately that this was an overreaction, so I scaled it back to a warning. Here's the worst part: I then asked one of my fellow moderators to send a message explaining the reasons behind the warning that I had issued. I didn't want it to look like I had become emotionally involved in the discussion, and didn't believe that I had. In retrospect, that was certainly the case. The other member reacted in an understandable fashion towards myself and my middleman: he sent us both indignant messages, deleted all his torrents, then his own account. I had, in the end, effectively banned him from the site.

The biggest lesson to take away from this example is that there's a right and a wrong way to involve other staff members. Asking another to do the work I didn't feel comfortable doing myself should have sent up red flags that what I was doing was not a good idea. Part of the problem is that I don't think he had quite internalized The Moderator's Bible that I posted on Monday, and didn't feel comfortable enough to say no to me. That's another reason that mods need to be relaxed and comfortable in their roles, since they may need to call you out when you behave improperly yourself. Hey, we're all human. Well, we are under a layer of internet, anyway.

Troll Control

2008-09-18T00:28:00.000-07:00

Anyone who has spent any length of time in an online community knows about trolls: insecure teenagers looking for recognition by eliciting a reaction from community members, be it positive ("LMFAO") or negative ("fuck off"). The easiest way to draw a reaction is to incite the contempt of others, so that's exactly what they aim to do. Having this sort of person around is naturally detrimental to the tone and integrity of the community. But what can you do to get rid of them?

The obvious response is to ban them outright. Well, this one has a few problems. First, as negative reactions go, summary bans are pretty close to the top of the list and will incite the troll to offend again, both on your site and on others. Proxies do exist, after all, and trolls know this on account of having been banned from other sites before. As well, this sort of reaction is entirely arbitrary and may make other community members nervous to say things to which you may take offense. In short, they become afraid of you. Some administrators thrive on this, but really, how does that make you any different from the trolls? Just because you hold the banstick doesn't (shouldn't) mean that your own behavior in the community is beyond reproach.

Of course, the best response to a troll is to cut off their food supply. To post responses, as well as conduct overt moderation, will feed their insecurity. Instead, caution your community not to respond. If necessary, delete a few posts - other posts - so it appears that no response has been made. Again, it is a bad idea to moderate posts that are annoying but not rule-breaking. However, you may be able to find more creative alternatives. I once edited the shoutbox code on TorrentFries so that the shouts of a chronic troll would appear only to himself and me. He left of his own accord soon after. (I'm pretty sure the troll was male. To the best of my knowledge, I've never encountered a female troll.)

On larger sites, this sort of time-intensive personal response is not practical. Instead, liberally delete affected posts, or entire threads where they have become infested with the troll/response cycle. However, do not hand out warnings or even directly acknowledge the deletion and underlying reasons until rules are directly broken. "No trolling" isn't a good rule, since it is very arbitrary in definition and to acknowledge this status makes the troll feel successful as this is their goal.

However, true rule-breaking will likely happen within a few days, and you should respond swiftly at this point. By waiting until rules are broken before responding, you make it clear to the troll that they will not be dealt with in any special manner. Since they want to feel special, this will leave them unfulfilled and allow you to respond without making the situation worse.

It is possible that the troll will attempt to contact you or another staff member directly. Respond to their messages politely, formally, and cooly. Answer any questions directly and as succinctly as possible, or provide a concrete reason why responding to a particular question is not an option (privileged information and so forth). Do not make any other comments, particularly regarding the character or behavior of the troll.

Never ban a troll without sufficient cause, although you can certainly make the rules a little less tolerant where the undesirables are concerned. You cannot allow them to feel like a part of the community or they will get the idea that the staff and the staff alone hate them, leading them to return via proxy "for the sake of the community". To the contrary, if you do your job right, they will leave of their own accord after a few days and a ban will not be necessary. I have never had a troll that lasted more than 4 days.

If you stray from this path, particularly by arbitrarily banning the troll or rising to their bait, your site may end up with a poltergeist to call its own, and they will become extremely difficult to shift. One way or another, best of luck in dealing with these little shits.

Write Your Own Disclaimer (in your spare time)

2008-09-17T01:05:00.000-07:00

By popular demand, I give you: do-it-yourself disclaimers.

Okay, first I'd like to offer a disclaimer about this post. I'm not a lawyer, and I haven't consulted with one in writing mine. I'm planning to have a lawyer member look over/rewrite our own disclaimer. Well, a member that is ostensibly a lawyer—see my previous comments on judging online character.

Anyway, there are a number of important points to cover. I know Demonoid has 34 pages of text in their disclaimer, which apparently contains some amusing points for those with the attention span to read through it. I guess they figure the length is a deterrent to people actually reading it, but lawyers are renown for their long attention spans. I think it's more important to make your point in a concise manner that your friends can understand as well as your enemies.

Basically, you need to address the following points:

No "uploaded"/copyrighted material is stored on this server. It's not. Only metainfo files (.torrent) are stored on the server, and they only contain SHA1 hashes which describe the material. All material is located on individual users' computers, so you're perfectly safe in making this argument.

Given that part of the disclaimer, for the love of god don't use your server as a seedbox. That'll just increase your culpability in this sort of thing and will make the bit about copyrighted material a total lie. What's more, running it as a seedbox can have all manner of detrimental effects on your server's performance, not just in terms of bandwidth but also drive speed. I know you probably have a 10 or 100 Mbps line with very low utilization, but resist the temptation.

The site cannot be held accountable for the behavior of its users. It is merely offers an indexing service. It's true. You can make the analogy to Google potentially indexing child pornography sites, but that's a little extreme. The point is that you merely provide a service.

It is the responsibility of the users to ensure that they hold the copyright or attain proper permission prior to uploading material. You may want to hide this statement a little, since some members will inevitably take it to heart. I've definitely had some concerned posts to that effect, to which I've given some rather tongue-in-cheek responses. Obviously, the idea is to turn one big fish into a whole lot of little fishies, reducing your own culpability without greatly affecting the masses.

Finally, you will comply with any and all formal copyright notices sent from the proper jurisdiction (where your host is located, since they shouldn't know where you are personally located). Obviously, you're not hosted in the US... are you? The idea is to make a hell of a lot of work for anyone that would attack you, encouraging them to go after other sites. You also need to provide a clear email address for copyright complaints. Better that they go whining to you than to your web host, which will always have contact information prominently displayed and will (with some exceptions) likely take you offline if a copyright owner says "boo".

It's doubtful that this will do a whole lot of good, but you never know. Once I get some better legal advice, I'll make another post correcting any glaring mistakes...

Site Contingency and Redundancy

2008-09-16T00:15:00.000-07:00

Today's post will cover contingency and redundancy, of which we've learned the hard way over time running TorrentFries. Backups are crucial to a holistic approach on contingency and redundancy, but there’s more. By the way, I’m OnionRings, the “fellow admin and general Linux and security geek.”

If you’re just beginning a site, hardware redundancy isn’t possible since you will likely have a shared or dedicated server host, but there are a few other areas of redundancy to think about. Attack issues of redundancy and weaknesses before they occur and in the order of likelihood. In the case of TorrentFries, the most likely thing to happen is for a host to get a copyright infringement notice.

Selecting a host that’s least likely to take fright in a copyright notice is the best choice to reduce risk of downtime, but that can be difficult to know right away. We’ve often asked our hosts what their policy on copyright material was before moving to them. Usually they are happy to take your money up until they receive a notice.

In the past, a host has given us 12 hours notice to remove material before the server would be shut off. That might sound like a lot of time, but when they issue them at 5 am in your time zone, it could be a few hours before you even notice the warning. At that point, you need a new host quickly. Therefore, select a host or two that would be sufficient while you have the time to research it and before you get a takedown notice. It can take days for a hosting company to get you login credentials much less the time it takes to research your best option. Plan ahead.

TorrentFries was once put out of commission after receiving an infringement notice while our host and the domain registrar were the same company. In that case, we were locked from our account and couldn’t even redirect our domain to the new server we purchased. So if uptime means anything to you, register your domain with a different company than you host with.

Short of having an expensive hot spare server wasting CPU cycles somewhere, there are other ways to mitigate risks of takedown. Parceling out the server’s work to subdomains that reside on different hosts is one way to achieve better contingency. In this case, put the tracker announce and scrapes on a separate server than the main site so the tracker doesn’t go down if the main site is taken down. Leveraging this technique can be a handy method to keep staff communications up during downtime where forums, email, or Jabber are critical. If set up with some thought, your database latency won’t take a huge hit and purchasing a couple cheaper servers might be only slightly more expensive than buying one powerful server.

Finally, there’s quite a bit of rework involved in installing and configuring software each time you move to a new server if you’re on a VPS or dedicated server. Even though I’m a Linux geek, I have to look up some things that I don’t use everyday. It’s nice to have a list of the tutorials you’ll be using or a checklist of the things that need to be done starting from disabling root SSH login and ending with the final tweaks in Apache or MySQL configurations. If you’re savvy enough, write up a bash script to handle all those dull tasks.

Hopefully your site will learn from some of our mistakes and you’ll be able to keep your uptime positive.

The Moderator's Bible

2008-09-15T00:20:00.000-07:00

I'm going to rewind a bit today and expound a little on a subject I touched on last Thursday, namely, the task of trying to get my moderators to loosen up and maybe understand their role a little better. This entry is perhaps something of a cop-out, but it's going to consist entirely of a message that is permanently stickied in the staff lounge. Hopefully it'll be of use to somebody.

Disclaimer: Take what I say at face value here - I'm being serious and saying everything straight up and no sarcasm involved. This is where I need that serious smiley.

IMO, this is a general thread that should be stickied across all community sites, like those "How to Upload a Torrent" threads often are. There's nothing specifically directed at any of you or all of you as a group, but everything that should be taken to heart for all mods everywhere - as the subject line indicates, a "Moderator's Bible".

The following is an example of what happens from time to time in a forum. You've all read the exchange and know who is involved, but I've genericized the names anyway since it really doesn't matter who is involved - we're all culpable at some point, staff or no. This is more to point out that in a community the term "moderator" doesn't mean "a person who wields shiny buttons". It means "a person who moderates". Think of a moderator in a debate. They have the ability to cut off somebody's microphone, but the job description is to keep the debate moderate and on track (well, in this site's case abandon all hope on that last point).

Here, things are admittedly a little different. In a debate setting, everyone in the debate is seriously devoted to the debate, even if they get sidetracked or passionate or whatever and need to be brought to order. On most sites, there are retards and spammers and all sorts of undesirables, so the heavy hand of the law needs to be used more often. But never forget that everything you do needs to stem from the basic need to keep the entire site as a whole on track and civil, to moderate the site as a whole, not just its parts.

So, without further ado, here's the excerpt:
Bystander:
Ah, another one of my flock, I'm studying engineering ;D
Combatant 1:
Which Engineering discipline? If its Civil then YOU are one of my flock, I like my coffee with milk and 2 sugars please ::)
Combatant 2:
Hey, hey, take it easy on the nice lady... You don't want us pressing any buttons, do you? :P
If it wasn't for that sarcastic smiley at the end, you would have made my day. ;)
The stage is set... nothing would have come of it since Combatants 1 & 2 are both mature, upstanding members of the community.
Concerned Person:
and YOU, Combatant 2, would appear to be correct that Combatant 1 was sheparding Bystander.
When a third party enters the game and takes sides, the sarcastic, joking air evaporates. Next, Combatant 2 attempts to explain his position, setting the stage for potential further conflicts:
Combatant 2:
Thus my comment about being nice to the lady. Sorry Combatant 1, but I won't have that, be it joking or not.
At this point, the mod steps in with a innocuous attempt to calm things down with a straightforward comment:
Moderator:
:-\ Is it just me or do I sense a sniff whiff of drama forming here?
Combatant 1 tries to explain his position. It's pretty effective, although this wouldn't always be the case.
Combatant 1:
I was implying that if she is a Civil Engineer, then I have a few years on her and if she came to work at my company she would be the general dogs body, degree or not she would still get stitched up with making the tea/coffee or doing the bacon butty run. :)
Moderator completely diffuses the situation with a straight-up remark, followed by a comment to hijack the thread in a whole new exciting direction. :D
Moderator:
Yeah, I figured as much. Usually just a matter of __ took __ the wrong way, so __ says __, and things just go downhill from there.

Now that we've got all that sexual tension out in the open ;) we can move on.
Sorry again for my choice of examples here, but this sort of thing is so rare around here that I kind of had to take what I could get. It's not a good example nor a neutral one, but it'll have to do. Around [popular site that I used to moderate], these things happen every day, partly because of the number of posts, but also because the community isn't as tight-knit and most people don't know each other that well. In fact, what happens over there is a thousand times worse, and with dozens of members taking four or five sides, some of them just saying that another side is idiotic for no reason at all. You can understand how quickly something like this could get out of hand if one user can't tell when another is being honest, sarcastic, ironic, mocking or even deceitful.

In this case it would've resolved itself without intervention, but I intervened (moderated) just to be sure. What made me think of writing this was a discussion thread I saw on Facebook asking for users' views on homosexuals. This is an obvious but somehow more insidious form of trolling, in my opinion. The thread will get hundreds of posts in an hour, but they will become increasingly heated, and in most communities the general opinion will result in the press of numbers attacking those of the minority opinion, and the minority posting nonstop making the same points over and over in an attempt to save face. Unless you are willing to watch every post, I would suggest you lock it immediately with a neutral comment like "this is an important discussion that should not be conducted here for the integrity of the community". This isn't the lazy way out; in fact, it's the safest way.

This isn't something that I'm telling you because I think you're performing badly as moderators. If anything, I want to make your more confident and aware of what your role is. You delete the porn spam, yes, and split the off-topic threads if you have a lot of time on your hands, but the rest of what you do has to stem from your own sense of keeping things level, in good faith and without any sort of bad feelings. The rules are guidelines both for members and for yourselves, but never feel you cannot overstep the limits of the rules or permit things not specifically allowed there as governed by your own judgement. There is no constitution here, and unlike in the real world, the intent of the law reigns supreme over the letter of the law. Above all, it is your role in keeping the site running smoothly and keeping its members in good harmony that you must adhere to.

So, if you've read this whole essay, kudos to you, and I hope it helps you out in some small manner. I, clearly, have a long attention span at the moment. And that's really all I have to say on the matter.

First person to reply to this thread with "I'm sorry" gets shot.

Backups

2008-09-12T00:43:00.000-07:00

Okay, I can't stress this enough: an off-site backup is essential to the operation of any website, especially a tracker. Database backups are the most important, but backing up your files from time to time is necessary as well.

For example, if your thumb-fingered database maintenance *cough* results in a few thousand damaged or deleted rows (done that at least twice) or in the accidental deletion of your privileged user from the database (done that, too), you need a backup to fall back on. It's nice and convenient to have a local backup for that, but you also need to upload that backup to another server (or your home computer) in case the server runs into technical problems or you're suddenly taken down.

I was pretty lazy with backups while running on a shared web host, so I don't have a lot of wisdom to offer there. My host informed me that he ran nightly backups of all databases, so there was no need for me to maintain my own. (My PHP-based backup script had become pretty heavy for his server.) I think this is pretty normal behavior for shared hosts, although I don't know that all of them would give you access to those backups on demand. The point for them is protection from pissed-off customers in case they muck something up on their end or the server crashes or some such. So, when in doubt, back up.

The crux of the matter is that I'm lazy, and you probably are too. If it were left up to me, there'd be thorough backups for a few days, followed by a span of weeks or even months without backup. If you're lucky, that'll be all you need. How lucky do you feel, punk?

Therefore, you need to automate two things: the backup itself, and the process of uploading the backup to an external server. I'm not going to walk you through the setup process since that sort of info is pretty easy to find online, but I'll run over the general concept.

For the backup process, I set up cron jobs with two shell scripts: daily.sh and weekly.sh. When they're set to run is pretty self-evident. daily backs up the database (mysqldump -uroot -prootdatabasepassword --all-databases > backup.sql) and the uploaded metainfo (.torrent) files, while weekly backs up the entire web directory, all the PHP sources and so forth. weekly doesn't have to touch the database, since daily will also be running on that day. Each script then runs tar -czf backup.tar.gz files, with whatever files you intend to compress.

That's all it takes to ensure that you're safeguarded against data corruption. Naturally, there are other pitfalls that you can encounter. Hardware failure, hacking, inept administration, and of course the ever-present DMCA message—all can threaten your data and your backups. For that reason, it's a good idea to maintain off-site backups as well. You can download them manually, of course, but that's not going to be any easier to remember to do on a regular basis than to make the backups in the first place. Instead, I have my backup script upload to an external server via scp. I use that command instead of ftp because it's more secure and easier to automate.

However, scp does require that you have a UNIX box at the end. (Well, there are Windows ssh clients, so maybe there are scp servers as well. Stranger things have happened.) Since I run Mac OS X on my home computer, it's simple to automate the upload to there, or to any other server that I operate. If your shared host gives you rudimentary shell access, it's even okay to use that.

I'm sorry for being so lazy with the tutorial-writing in this one, but this blog entry is already getting rather lengthy and hopefully you can follow me anyway. I just have a few more points to make before I close.

Where one backup is good, archives of backups is better. If you set your script up to name its output files with the creation date, they won't overwrite one another and you'll have a nice collection in case you need to roll back a few days. I have plenty of hard drive space, so I still have backups from 2007. Who knows when you might need them?

Putting all this together, a simple shell script might look something like this (lines are double-spaced to avoid confusion in the case of line breaks):

#!/bin/sh

date=`date -I`

mysqldump --all-databases | gzip > /var/backup/backup-$date.sql.gz

scp /var/backup/backup-$date.sql.gz user@host:/dir/backup-$date.sql.gz

Okay, that's all for now, folks. I meant to throw in some horror stories about maintaining/not maintaining backups, but we're out of time. Maybe I'll get to that later on. Rest assured, such stories do exist, and don't forget: always maintain an off-site backup!

Recruiting Staff

2008-09-11T00:35:00.000-07:00

Okay, I don't generally intend to follow the order of my subject list in yesterday's post, but since this is the first subject that came to mind, I'll hit it first and go from there. Maybe I'll end up following that order anyway, but not by any grand plan.

I was a moderator on a major site for a while, until I left due in no small part to a personality conflict with the administrator. What I took away from the experience (aside from a bunch of friends among fellow mods) was a shopping list of what not to do as an admin. Specifically, I decided to be open with my mods, democratizing the relationship as much as is practical. I'm not all-knowing, and I certainly don't feel comfortable with my word being taken as gospel. It took a while to convince them to relax, but I think it was a good move and it seems to have worked well.

As mentioned in the title, the point I want to make is specifically related to how to tell when members are ready to be promoted. Your mother probably told you something about strangers on the internet, and it's true, albeit with a lot less paranoia in the equation. You never do know who someone you're talking to online really is, but honestly, I don't care. If someone is reliable online, I don't care if they're a pedophile or serial killer in real life. I'm going to be dealing with their online persona exclusively, so the fact that I can rely on them online is all that's important from a staff perspective.

When I say recruiting your friends is a good idea, I want you to understand exactly what I'm talking about. These people need to be first and foremost the community's friends, but you're going to be working with them on a daily basis. The staff needs to be a strong team, and when there are conflicts, there's a problem. Besides, if you're doing your job right, you have nothing but friends in the community, and not the sort that come with being a figure of authority.

What I don't mean is that you should have your friends from real life or elsewhere on the internet join up and be promoted almost immediately. In this case, it doesn't matter how long you have known them or how much you trust them. The community in general has to trust them as much as you do, and they have to be committed to the community. That's not to say your personal trust isn't a great asset, but it can't be the primary factor. I did make that mistake once, allowing an online friend to join up and be promoted after some token site activity. As of this posting, he hasn't been seen on TorrentFries in over a year. He didn't do anything reprehensible, just disappeared one day. Just because someone is trustworthy doesn't mean they're committed to the site.

The other point I want to make is regarding timing. Moderating a site can be a big job, requires some serious team effort, and has its lulls and periods of frantic activity. I see no sense in hiring "enough" mods while leaving others on your waiting list in case you need them. People have lives, and it's nice for your mods to know they're free to take off whenever they need to without crippling the site in their absence. Hell, I could do that if necessary, since my mods had all the power needed to ensure that the site kept going on a day-to-day basis. These days, the only thing none of my fellow staff members have access to is the money, but that's a story for another time.

A side benefit of this duplication is that any of the little day-to-day issues that need dealing with such as trolling, spamming for porn sites, general rule violations and so forth can be dealt with very quickly. Even with "enough" staff members, not everyone is going to be online all the time. The more people that have the power to deal with this sort of thing, the more likely it'll be that someone will be able to deal with this sort of offense immediately.

Okay, one last thing. This shouldn't even need saying, but: if someone asks to be promoted, for fuck's sake don't do it. If it becomes apparent that they want or expect the position, a big black mark goes down beside their name. This sort of power-hungry person is going to be unreliable at best. The best staffers are the ones that will be baffled to sign on and find that they now have a ton of extra buttons on the site.

That's enough for today. Tune in tomorrow for... uh, something else. I'll work it out later.

Retrospective 7: Et Cetera

2008-09-10T01:02:00.000-07:00

While we moved to a dedicated server well over a year ago, the time after the move was (until recently) characterized by a large number of gradual changes rather than the quick bursts of activity that I've discussed so far. The process of administration definitely became more complex, but in doing so, the issues we were dealing with became longer term in nature. The urgent stuff had been dealt with, but now the business of governing lay before us. It's very French Revolution, really, except without the guillotine and general in-fighting.

All of that is a very different ballgame. In fact, this gradual change is going to force me to abandon this "retrospective" structure altogether, favoring instead a more topical approach, dealing with each subject individually. I'm still going to be speaking historically for the most part, but story time is over. Now we're going to be talking about the real deal, what it takes to keep something like the site running on a day-to-day basis. Once that subject has been exhausted, we'll probably end up at something more closely resembling your traditional blog, but my fellow administrators will finally be introduced and will therefore be able to post, so you won't have to read my deranged ramblings exclusively from then on.

For now, a brief overview of what we're going to be talking about:

Recruiting staff: My approach is pretty easygoing. Generally, if someone is clearly ready to be on the staff, they are promoted. There's no need to wait until everyone else is swamped before looking for help.
Dealing with troublesome users: "Ignore them until they go away" does actually work. Sometimes.
Conflict of interest: A case study in what not to do, hindsight lesson courtesy of yours truly.
Backups: It's easy to neglect the importance of having up-to-date off-site backups. However, when disaster hits, you will either turn to your members and gloat at your awesomeness or shamefacedly admit that you just lost a month's worth of torrents and forum posts.
The tracker announce: It's not as scary and impenetrable as it may look. I'll take you through the basic workings as I write one from scratch.

Hopefully that'll pique your interest enough to make you stick around for a bit longer. More to come, folks! I'll try to keep up this 5-day release schedule, at least until I run out of interesting things to talk about.

Retrospective 6: Early Linux Experiences

2008-09-09T00:17:00.000-07:00

As I mentioned on Friday, I was completely hopeless with Linux when TorrentFries moved to its first dedicated server. Happily, I quickly discovered the most magical feature of Ubuntu: apt-get. With it, it's possible to download and install applications with a simple command, such as apt-get install htop. (By the way, htop is something well worth checking out. For those that don't know, top is a command that allows you to monitor the processes and load on a server. htop takes that a step further and allows more advanced process manipulation and general feedback.)

What I did know was what needed doing, specifically, setting up a LAMP (Linux/Apache/MySQL/PHP) server and somehow getting everything to work properly. For working this out, the Ubuntu documentation (both official and unofficial) was invaluable. I've found the unofficial docs to be more comprehensive since they do a better job of covering command-line and server material, while the official stuff seems more focused on the desktop GUI. Obviously, since the server version doesn't have a GUI, that's not especially useful.

Thanks to the magic of Ubuntu, a LAMP server needs only one command to install. Obviously, there's a bit of fiddly configuration that follows, but it's still a walk in the park compared to the downloading and compiling some OSes would have you doing. The masochists that prefer that sort of thing are welcome to it, and it's certainly true that the customization available during compile makes for a faster and more effective install... but I can still have my server online hours before you, and that's a matter of days if we're using beginner timeframes here.

The bottom line is that Ubuntu is a good choice for an introduction to Linux server administration. It's be messy at first, but Google is a great place to find tutorials that can walk you through just about anything. Once OnionRings came onto the scene, he was able to help me out with a lot of the more complicated stuff, although it took even him days to figure out how to set up a proper mail server. Trust me, Google Apps is the way to go for email.

Since Linux still isn't my strong point, I'm not going to go into any more detail here, but some of OnionRings' eventual posts should be a little more helpful in that regard.

Retrospective 5: Money

2008-09-08T00:43:00.000-07:00

Continuing from where we left off on Friday's post: as a result of the move to a dedicated server, our monthly costs went from $5 (or $0, depending how you do the math) to roughly $80. That's a huge jump for a small site. I made a news post pleading for donations. EasySpeedy required 3 months' worth of hosting payments in advance, plus a €60 setup fee, when paying with PayPal. As a result, we had to raise €240 up front, or $360 US. The community came through. In two days we'd raised $250 and had enough with our previous donations to make the move.

However, going to the member base with hand outstretched is very poor form, and something I like to avoid at all costs. By the way, when you see a host do a donation drive, don't forget that there are a lot of generous people out there and the host may be making more money than you think. I heard secondhand of a site where things slowed down and the administrator announced they needed a new server and started a donation drive. A few weeks later, the site sped up again, "we're on our new server", and the admin walked away with a cool $10,000. Don't let that prevent you from being charitable, especially to a small site—please don't—but that's something to bear in mind.

Anyhow, we obviously needed to find a more sustainable way to supplement our income to ensure that we could make our next hosting payment. I hadn't gotten around to looking into advertising up to that point because it often requires credit cards, mailing addresses and other undesirables. The provider that we did end up with was Project Wonderful. They're wonderfully subversive to normal advertising models, which appealed to my rebelliousness. They certainly don't pay as well as other providers might, but it's all right. Instead of scams, spyware and pyramid schemes, we got ads for web comics and independent online clothing stores. Again, a little subversive.

I also implemented a few perks for donors. There's not a lot of incentive to offer without removing features for regular members, so I added the usual formula of donor star, your choice of username color (staff names in italic), and x days of status per dollar donated. It seems to have worked all right, and we did manage to meet our next quarterly payment with a bit to spare.

I have a feeling there was something else I was planning to mention here, but since I'm actually writing this days in advance as a result of my unnecessarily long post attempt on Thursday, I imagine I'll have time for it to come to me. If not, maybe you'll be lucky and I'll remember later on and tell you then.

Retrospective 4: Dedicated Hosting

2008-09-05T13:59:00.000-07:00

In April, our host began complaining about the load of the burgeoning site, so we started shopping around for dedicated servers. Ketchup ended up finding a server for €60/month ($80 US at the time) with EasySpeedy, a Danish web host. I asked them in advance how they felt about running a BitTorrent tracker, and they responded with the attitude I mentioned in Tuesday's post: "if we don't hear about it, it's fine". And it was. We were hosted there for over a year before we outgrew that server and had to look elsewhere, and no copyright holder contacted them in that time.

There are a couple of notable things about this move. First, it was the first time I'd used Linux to any great extent, and certainly my first experience with server administration. I had enough experience with UNIX on Mac OS X to know how to navigate directory structures and edit files, but that's about it. I selected Ubuntu to install on the server because I'd heard it was the friendliest, and that certainly did help me a lot. I'm still amazed that I was able to keep the site running with such limited experience, basically working off what tutorials I could find with Google. In fact, I'm still learning, so later posts (particularly once OnionRings comes onto the scene) ought to have some useful info on that front.

Naturally, the second point is the financial. Sorry for the short posts lately, but I had to break what was originally a ridiculously long post down a bit. So, on Monday, we'll talk money.

Retrospective 3: Shared Hosting

2008-09-04T08:55:00.000-07:00

Up to now, I've been talking mostly about setup. Now it's time to delve a little more into the day-to-day operation of the early months of the site's life.

As one might expect, things were pretty slow for several months after our January launch. With the low activity on the site, there wasn't a lot to do aside from tinker with features as suggestions filtered in. The more I customized, the more I realized how woefully unequal TorrentTrader was to the task. Unfortunately, I was committed and to change at that point would mean more work still. Instead, I quickly developed a policy of rewriting entire pages as a whole rather than tweaking what was already there.

One of the suggestions was that I add a favicon to the site. The existing logo didn't fit the small size very well, so I set out to make a new one. The product ended up looking pretty good, so I redid the entire site design based on the icon. What I wouldn't give to have that kind of time at my disposal nowadays...

Aside from the constant refinement of features, there's not a lot involved in running a small tracker, no more than what's involved in tweaking any other pet website. Since someone else may be taking care of your server for you, you may not even need to worry about that. The whole experience feels a little like running a lemonade stand on a deserted street, sitting in the sun while the ice in your mother's lemonade pitcher melts. It's just that exciting.

Despite the last few paragraphs of doom and gloom, we did grow slowly thanks to some of the methods I detailed yesterday. In March, a member asked how to donate to the site. Bemused, I opened a PayPal account. The $5 donation felt like a fortune. Since the site was sharing a hosting account with my personal stuff, we were essentially being hosted for free at the time.

Don't worry, all that changes soon enough. More on that tomorrow.

Retrospective 2: Promotion

2008-09-03T12:38:00.000-07:00

Promoting any new site is a major issue. Word-of-mouth on the internet is not to be underestimated, but it won't do much for a site with no unique features—see yesterday's entry for a bit of discussion of that one. Even on an unusual site, word-of-mouth won't get things going very fast from the beginning, and you end up in the vicious cycle of having no material, making it difficult to pull in new members to add new material. Some people have ridiculously large collections of material gathered from various sources, such as audiobooks or movies. I know of people whose porn collections are measured in terabytes, but I prefer not to know them personally. Regardless, if you can upload your own collection piece by piece, you may be able to make your tracker appear much larger than it actually is, and use your unique material to pull people in.

Unfortunately, there's not a lot you can do by way of promotion in the early days aside from shameless spamming. A point related to my earlier mention of the usefulness of connections: I am active enough on some high-profile communities that I could get away with throwing links into my signature and posting a few spam threads. Don't try this as a new member, and certainly don't try to join communities exclusively for the purpose of posting spam messages. It will only generate ill will towards you and your site. If you don't have a place to do this, it may be a good idea to hold off on your plans until you do. It's important.

Dump sites are great promotional methods. Sites such as MiniNova and Demonoid allow you to upload torrents tracked elsewhere, so configure your new tracker to accept unregistered IP addresses (temporarily if you intend to go private) and upload your torrents to a bunch of dump sites like that. In the torrents' descriptions, include a comment such as "find more great torrents like this at www.example.com". You can even throw a text file inside the torrent to the same effect. You'd be amazed by how well it works.

I'll leave you today with a couple of graphs showing the rate of growth of the site. As you can see, it pretty quickly surpasses the size of an average private tracker, so the processes involved may start to diverge a bit as we progress and as I start to deal with a larger and larger tracker.

Retrospective 1: Conception

2008-09-02T11:56:00.000-07:00

Since starting TorrentFries, I've had a number of people ask me how to start a successful tracker. The first question I ask is always "What sort of material do you want on the site?" Only one person gave me a response other than "everything". Incidentally, I'm now involved in some small way in the development of that other person's tracker. The two aren't directly related (I'm not a sword for hire), but they're not entirely unrelated either.

Listen, I'll let you in on a secret. There are tons of "everything" trackers out there. Come up with something unique to pull people in.

Personally, I believe in unconventional activism. In this case, deep-fried goods are too expensive, so I thought a BitTorrent tracker might get people talking about the problem. Since nobody else had one going, I set out to create one of my own.

The first step in the process was to find a host and a tracker. In this situation, personal connections are nice. I was lucky enough to know a guy that ran a shared web host, so I was able to get things running for $5/month, plus the cost of registering a domain name ($12/year). However, any shared host worth their salt should figure out that someone is running a tracker pretty quickly, so don't count on slipping that under the radar. Instead, you might consider a VPS, which allows you to rent part of a server, rather than having to pay for the whole thing. You should have plenty of control over your server, and get more leeway in terms of traffic. A basic VPS will cost you around $40/month.

However, be aware that there's no way you can buy security for any price you can afford at this point. The best you can hope for is a web host that will look the other way until they hear from a copyright holder. Most dedicated and VPS hosts I've encountered will do this; in fact, some hosts have told me this directly. What you can do is maintain daily off-site backups and have a backup host lined up in advance in case of the worst. We'll talk about contingency planning a bit later on.

Choosing the software involves a fair bit of research. I was pretty lazy on that point, and have been paying for it ever since. The only two trackers I checked out were TBdev and TorrentTrader Classic. TBdev is used by a lot of big-name private trackers such as Torrent-Damage, TorrentLeech, the late OiNK, LearnBits, BitMe, BlackCats, BrokenStones and so forth. However, the administration functions are almost non-existent, and the amount of customization available in TorrentTrader appealed to me. I didn't want to spend much time digging around in the code to get everything to work properly.

I've got a fair bit of experience with graphic and web design, so I took it as given that I'd build a new theme from the ground up. The product was in retrospect not that impressive, but it was certainly usable and unquestionably better than what I'd started with.

Finally, before going live, I asked a friend to take a look at the site. I expected him to spend perhaps 5 minutes and respond with a "looks good", but he ended up giving me a two-page essay including everything down to copy-editing in the FAQ. It was great help, and I was able to refine a lot of features that way.

Tomorrow: details on how to pull members in.

Background

2008-09-01T14:47:00.000-07:00

Okay, this is my first real blog attempt, but I thought it might be a good idea to talk a bit about the life of a BitTorrent tracker admin (or SysOp, for those that subscribe to the TBdev lingo). The challenges of running a tracker are pretty unique to the subject, and I haven't seen any really in-depth discussions of what this entails. The closest thing I've seen is a TorrentFreak article which only really focuses on the personal implications. Hopefully, this blog will give readers a good idea of what the process entails, to warn away the unprepared and prepare those that want to go ahead with starting their own sites.

I've been running a tracker for almost two years now. I won't say what tracker for security reasons (the same reason I'm using a different alias here), but the astute may figure it out anyway. If you do, please be quiet about it. I'll be talking frankly about security measures and stuff that is normally kept under wraps, so I don't want most people to make the direct connection to the site. Instead, I'll be referring to the site as TorrentFries.

TorrentFries is middlish in size, no Demonoid, but certainly enough to keep me occupied. Signups are permanently open, and accounts never expire. At the moment, the tracker is open as well, so you don't need an account to download stuff. The tracker occupies a pretty unique niche in the torrent world, and is the only one of its kind that I know of. I would love to see more of its kin appear, but so far there's nothing that I've heard of.

This brings me back to the original point of the blog. We need more trackers out there, and it's up to you to make them. So, learn from my mistakes and the things I've accidentally done right and you'll do just fine.

Over the next few weeks, I'll make a series of retrospective posts covering the process we've gone through up to this point. Once that's done with, we can get going on the here-and-now. So hop on and enjoy the ride. :)