I hope you have enjoyed the blog, and I hope you will find the forum useful and perhaps entertaining. I'd like to extend extra-special thanks to Chris for registering the domain for us before some filthy squatter could get to it. torrentfries.com and www.torrentfries.com will continue to redirect to the blog to prevent broken links, and forum.torrentfries.com will take you to the forum.
Monday, November 10, 2008
Introducing the New Forum
As previously mentioned, we've been working on creating a forum to replace the blog and hopefully provide a similar source for useful information and discussion. Well, the TorrentFries forum is now online. We encourage you to head over there, sign up, and check it out. This blog will remain online for reference purposes but will no longer be updated. Any further entries we feel like adding will be made over at the forum.
Tuesday, November 4, 2008
A Public Service Announcement
No serious post today. I just wanted to remind any Americans in the crowd to get out and vote. Vote Obama, vote McCain, vote motherfucking Communist, just vote.
Trust me. It'll be good for you.
That'll be all. This has been a public service announcement paid for by nobody, unfortunately.
Monday, November 3, 2008
Irregular Posting Schedule
Since the beginning of September, we've been posting religiously, 5 days a week. Our steady stream of subjects of discussion having slowed to a trickle, we're moving to a less regular schedule. We will continue posting at least once a week for the time being. I'm not one to leave you hanging, so when the blog reaches its end, you'll know it.
We will be launching a TorrentFries community, so stay tuned for announcements on that. In the meantime, if you have any questions that you'd like to have addressed, please do leave a comment and we'll do our best to answer. I've got a few to respond to at the moment, and will be doing so in the coming days.
Friday, October 31, 2008
Guest Post: Forum Integration With SMFTracker
A friend of mine is working on a BitTorrent tracker mod for the Simple Machines Forum system. He was kind enough to provide us with a peek at his development process. He is basing his mod's announce on the basic framework I developed and published on Monday.
I've had to edit his post a bit, since he strayed into matters regarding our true identity pretty frequently, but the gist of the tracker mod development is there. Hey, what's life without a little mystery?
This is ChiliDog, co-collaborator of our ever-elusive stand-alone tracker script. I was asked to make a guest post for the TorrentFries blog by CurlyFries. Unfortunately, being the uninteresting person that I am, I’ve been racking my brain, trying to figure out what to write about. I eventually came to the idea of writing about my current project: a SMF mod, with complete tracker integration.
Our main goal with the collaborative project was to successfully bridge any forum system’s [be it SMF, vBulletin, phpBB, etc.] database, however, that turned out to be a far-fetched dream [for now], due to its resource intensive code [blame CurlyFries ¬_¬]. But to tell you the truth, I couldn’t care less for any forum system other than SMF. It’s powerful, feature-rich, and masterfully safe [not to mention completely free].
I’ve been keeping an eye out for any tracker mods for SMF for the past 5 years or so, but not one has ever appeared. There have been a few makeshift attempts at bridging a tracker system and SMF’s user table, but nothing worth using. So after waiting so long, I finally set out to make one.
But, ‘Why a mod? Why not just a bridge?’ you ask? We’ll, I’ll tell you… It’s always been my philosophy to take advantage of what’s there, rather than to create something that does the same thing. A bridge will still require all the things that SMF already does, in your tracker’s source. CurlyFries is apparently the opposite of me. He likes to create, rather than use what’s available [but more importantly, give people more options for the forum system]. That in itself is fine, and I’ll follow along for the main project, but I’ll be doing the SMFT mod in my spare time.
Other the announce.php, everything takes full advantage of SMF’s functions, including its permission system, themes, security, and even the language and modifications/upgrade system. With three possible modes, open [ratio tracking is disabled], semi-private [non-members can still connect, just no ratio tracking and they might be restricted in site permissions], and private tracking [only members can connect], you will have a choice of what type of tracker you want to create.
There’s really nothing ground breaking about my project, but it should be the first SMF bittorrent tracker available; hence my project’s name: SMFTracker® [genius naming sense, eh?]. A lot of the front end source code, is the code CurlyFries and I created for our project, albeit most of it [if not all], was rewritten to work with SMF.
Our original announce [which I really liked] was ‘accidentally’ deleted by CurlyFries, but the [barebones] announce posted here is just as efficient [although, I prefer the switch system code he posted first]. I’m not ashamed to say I used that as a base to work with [credit was given where credit was due].
Although, I’m almost complete in coding SMFT [the remainder is just fluff and whatnot], it probably won’t be released for distribution this year. I intend to beta test it for a while on my site and work out the kinks. Once testing is completed, I’ll probably ask permission to add it to the SMF modification repository. I say ‘ask permission’ because I’m not sure of SMF’s stance on a tracker mod. They’re a legitimate corporation, and might decline my request, due to bittorrent being synonymous with copyright infringement [who’d have thought?].
On a side note, neither I, nor CurlyFries have abandoned our own project, and we do intend to [eventually] complete it. However, SMFT is just something to keep me occupied until CurlyFries has extra free time on his hands for a time intensive project.
I definitely agree that SMF's mod system is ridiculously versatile, but with it comes unnecessary bloat and overhead. I personally prefer to build from scratch for maximum efficiency, and to me it makes more sense to build a forum around a tracker rather than a tracker around a forum.
I've subtly hinted that he may want to think about load testing the announces of various popular PHP/MySQL trackers, perhaps including his own and the one I wrote. It'd be interesting to see how they stack up.
Labels:
development,
forum,
MySQL,
PHP,
Simple Machines Forum,
software
Thursday, October 30, 2008
Building a Community
The heart of a good BitTorrent site is a good community. This is a hard subject to address, which is one reason I've hitherto avoided it. However, it's also a very important one, so I'll do my best.
Easier than describing how to make a good community is perhaps how not to do so. I know of a group of people that are trying to create a community out of whole cloth, which strikes me as ridiculously naïve. That said, they're more knowledgeable than I am, and if they succeed, the more power to them.
However, in my experience, good communities are things that are not made but simply come about of their own accord. The best you can do as administrator is to focus on creating an environment that fosters a strong community. I've already outlined factors that can poison a community: strong emphasis on ratio, paranoid pre-emptive action against potential undesirables, heavy-handed and arbitrary moderation, and so forth.
People like to be free, to be able to speak their minds without fear of repercussions. They will naturally gravitate to a place where they feel that they can do this. Make no mistake, the world doesn't need another 4chan. In fact, the world doesn't need the 4chan it already has. As always, it is necessary to find a balance between the extremes of staff control and utter chaos – to moderate in moderation, as it were. Trying to entice people become active in the community is doomed to failure. Rewards for high post counts are ridiculous; people are too hung up over post counts anyway, and it's harder to break in to a community where post count equals status equals worth as a person. It may not seem that way to you, but it will to the newcomers.
This is where things get difficult. It's up to you to find your own balance. All I can do is lay out the questions and leave you to answer.
If you've got any more points you'd like us to cover before shutting this bad boy down, now's the time. Don't be shy!
Wednesday, October 29, 2008
Donations
In light of yesterday's post, I'm going to spend the next few days addressing questions as they come up. Today's question is regarding encouraging donations.
In my experience, people are generally pretty decent sorts and will do what they can to support their favorite site, and I'm assuming that's you. I'm not into pushy salesmanship, and I don't think it's necessary. It is annoying and compromises the tone of your site, but more importantly, it crosses a rather important legal line between facilitating the sharing of copyrighted material and outright selling it. Where you want to stand on that question is entirely your own choice, but I like to stay on the light side.
However, I'm not opposed to the idea of buying ratio, as long as your site doesn't make it necessary for users to do so. Ratio represents a contribution to the site, and there are all kinds of ways to contribute. Some sites also provide ratio bonuses for long-term seeders, and I think there is very much the same thing going on here. If you've contributed, I have no problem with recognizing you for it.
I have a slightly less sunny image of donor/ratio perks, though. Any features that you provide to VIPs are features that you are denying to other users, which in turn prevents your site from being as usable as possible to as many people as possible - kinda the point in my view. Providing recognition is fantastic, but adding concrete features is a little more questionable. VIP-only forum boards will fracture your community, since the VIPs will tend to hang out with their kin and other users will never really be pulled into the fold.
In terms of recognition, I suggest simply giving users more control over their own profiles and the way they are presented to others. I know some sites allow avatars only to power users. However, since I only ever notice peoples' avatars rather than their usernames, I want to encourage as many people to have avatars as possible. You could allow people to customize the way their username is displayed, add more info to their profiles, maybe add custom stylesheets. None of these things have any practical use, but they provide a degree of social status as a reward, and that is enough. Of course, the ubiquitous donor star is a must.
I suggest placing a clear donate button on every page, but nowhere obtrusive. In keeping with my previous point regarding selling copyrighted material, I would keep it away from torrent description pages. Make it clear to your users that donations are always appreciated but never expected. A simple button in the header or footer ought to be plenty. I like Demonoid's approach of using an animated GIF that flashes briefly and subtly every couple of seconds. It draws attention without being too distracting.
Demonoid also has a neat donation system. I'm not sure I should be detailing it here, but you should be able to get some idea by making a donation yourself, or going through the process up to the point where you're asked to key in your PayPal account. Basically, donated money is received not through Demonoid but through a third party site, so PayPal can have no complaint. Very clever.
In my experience, people are generally pretty decent sorts and will do what they can to support their favorite site, and I'm assuming that's you. I'm not into pushy salesmanship, and I don't think it's necessary. It is annoying and compromises the tone of your site, but more importantly, it crosses a rather important legal line between facilitating the sharing of copyrighted material and outright selling it. Where you want to stand on that question is entirely your own choice, but I like to stay on the light side.
However, I'm not opposed to the idea of buying ratio, as long as your site doesn't make it necessary for users to do so. Ratio represents a contribution to the site, and there are all kinds of ways to contribute. Some sites also provide ratio bonuses for long-term seeders, and I think there is very much the same thing going on here. If you've contributed, I have no problem with recognizing you for it.
I have a slightly less sunny image of donor/ratio perks, though. Any features that you provide to VIPs are features that you are denying to other users, which in turn prevents your site from being as usable as possible to as many people as possible - kinda the point in my view. Providing recognition is fantastic, but adding concrete features is a little more questionable. VIP-only forum boards will fracture your community, since the VIPs will tend to hang out with their kin and other users will never really be pulled into the fold.
In terms of recognition, I suggest simply giving users more control over their own profiles and the way they are presented to others. I know some sites allow avatars only to power users. However, since I only ever notice peoples' avatars rather than their usernames, I want to encourage as many people to have avatars as possible. You could allow people to customize the way their username is displayed, add more info to their profiles, maybe add custom stylesheets. None of these things have any practical use, but they provide a degree of social status as a reward, and that is enough. Of course, the ubiquitous donor star is a must.
I suggest placing a clear donate button on every page, but nowhere obtrusive. In keeping with my previous point regarding selling copyrighted material, I would keep it away from torrent description pages. Make it clear to your users that donations are always appreciated but never expected. A simple button in the header or footer ought to be plenty. I like Demonoid's approach of using an animated GIF that flashes briefly and subtly every couple of seconds. It draws attention without being too distracting.
Demonoid also has a neat donation system. I'm not sure I should be detailing it here, but you should be able to get some idea by making a donation yourself, or going through the process up to the point where you're asked to key in your PayPal account. Basically, donated money is received not through Demonoid but through a third party site, so PayPal can have no complaint. Very clever.
Tuesday, October 28, 2008
The Future of TorrentFries' Blog
You may have noticed that we've been running a little short on subjects for discussion of late. This was intended to be an account of the day-to-day operation of a tracker, but with the site's untimely demise, that fell by the wayside and we ended up relying on a series of exposés instead. That's all well and good, and I really hope you've found them useful, interesting and/or insightful up to this point. However, there is only so much that one can discuss without the constant influx of new material resulting from running a live tracker.
Therefore, I'm interested to get your take on where we should go from here. We have been discussing creating a forum for those interested in the subjects we've discussed here, so that remains an option. We may also move to a less intense publication schedule, although that's only prolonging the inevitable. One way or another, the blog's days are numbered, so if you have any questions that we've left unaddressed, by all means, leave us a comment and we'll see what we can do for you.
Monday, October 27, 2008
The Tracker Demystified - Part 3: Reality
Okay, it's time to get back to the coding. On Tuesday, I developed a script that would play by the rules assuming everybody else did too. It was awesome: 29 lines, with a maximum of two database queries. And I wouldn't give it five minutes in the real world.
Why? Well, first, no input data was validated leaving the script open to SQL injection. Hopefully I don't need to say much more about the problems presented there, aside from that it would be a very bad thing were we not to address this issue. If you haven't heard of injection, go take a few minutes and read up on the implications. Since infohashes and peer ids are binary, any value containing 0x27 (apostrophe) would invalidate the SQL even without any malicious intent.
Secondly, even if we assume that the user is always benevolent, the script fails to anticipate that they may not be as true to the protocol as we are. The most glaring issue is that if the stopped message is never received, that particular row will remain in the database forever. This happens in the real world all the time; some poorly-designed clients will even neglect to send stopped when closed.
This time around, I'm just going to link the finished code and then go over some of the highlights. It'll be faster than just detailing every change in minute detail.
In my previous code, there wasn't even an error function. I've corrected that now, using the failure reason response provided by the protocol spec. I like my code to fail gracefully, since it cuts down on errors for users to come running to you with, but occasionally that's just not possible. Using failure reason with the message "Invalid port number", the user will see something like this in their client:
Obviously, you don't have a lot of room to work with, but you can get some pertinent information across. In this case, it is used to inform the user that their client is fucked. (Side note: this is an artificially-generated error. Transmission would never do anything bad.)
I've added a new field to the database, a timestamp field called last_connect. It has ON UPDATE CURRENT_TIMESTAMP set, so whenever we make a change to a row, the timestamp will be updated to match. This will be handy later on.
A bunch of validation has been added to the top of the script, which should be pretty self-explanatory. The BitTorrent spec describes the optional ip value as being "generally used for the origin if [the client] is on the same machine as the tracker", so the script makes an exception for 127.0.0.1, ignoring the value otherwise and simply going on the server's record.
I've shuffled the core about a bit so that they make more sense to the computer and less to a human. Since the maximum announce interval has been set to 30 minutes, if a row hasn't been updated in an hour, the script will assume it to be trash and delete it. In order to update the timestamp, a dummy query (SET uploader = $uploader) is run. Well, it's not entirely useless – this way, if the completed event is never received, the script won't even notice. In fact, it doesn't even check for completed anymore. Both it and empty are now treated in exactly the same way.
If you rewind a bit to my previous entry on the subject, you'll see that empty was never dealt with. That's because it doesn't involve any change in state, which the tracker would need to record. Well, now the state is simply an assertion of its continued existence, which the tracker will need to keep track of.
It's not necessary to clear old entries with every connection; in fact, it is quite wasteful to do so. However, since this is a one-script application, we can't rely on a cron job or other external function to do our dirty work for us.
This is where I will leave you. There are a ton of features I could add, including ratio tracking, IP bans, client identification, protocol extensions (some trackers include swarm info on announce), defending against browser attacks, and so forth. However, I promised to deliver a script that functioned as a basic barebones open tracker in the real world, and that's exactly what I did. I hope you'll be able to use it as a framework to work from in writing your own code.
On the subject of working as a framework, you should now see why I left the script half-finished last time. It's a lot easier to understand the early version than the completed version, and hopefully helped a bit to clarify the basic structure we were working with. I'm sorry for any bugs or security holes that may have crept in here. This is a last-minute job as always, and it's entirely possible that I'm being sloppy.
That's all for now, folks.
Friday, October 24, 2008
Big-Tracker Finances
As I mentioned yesterday, I am seriously strapped for time this evening and won't be able to continue my announce-writing series as I had intended. I'll do some coding over the weekend and hopefully get things going again on Monday.
Early in the life of the blog, I discussed the financial logistics of running a small tracker. Well, TorrentFries grew out of that phase. As you grow, so does the complexity of your bookkeeping.
Obviously, the cost of hosting increases as you move to faster and faster servers. However, things also become a lot less predictable. I've previously advised that contingency is a good thing, and nowhere is this more true than in the case of finances. While you're small beans, you will probably be paying the same cost every month for a long time. Hosting is predictable, although we were impacted by the dropping US dollar, which caused our hosting costs (charged in euro) to increase without any benefit to us. Of course, we were growing and becoming richer as time went on, so that was more annoying than difficult.
Eventually, it is likely that donations will outstrip expenses. Don't let it go to your head and don't squander the money. A big site is always in danger of being taken offline or even coming under threat from the copyright police. Having a slush fund handy is invaluable, since you won't even be able to beg for money with the site down. At one point, we had as much as $800 in reserve, plenty to get us a new server if need be. In fact, we were saving up to build our own box to colocate somewhere friendly. That didn't end up happening, but it certainly could have with a few more months. At the time, we were making about $30/mo in ad revenues (Project Wonderful is not the most profitable provider around) and over $500/mo in donations, although donors are fickle and hard to rely on – another good argument for a slush fund.
Now, running into trouble with PayPal (which I assume you are using for your finances) is another concern. I suggest spreading your wealth across multiple accounts, a few hundred dollars in each. If one goes down, you still have access to enough money to tide you through until things get sorted out. Don't use Tor while accessing PayPal. If your geographical IP jumps around too much, PayPal will automatically flag your account as potentially compromised and will freeze your funds until you confirm the account with a credit card. Prepaid cards are handy there, but a massive pain in the ass nonetheless.
How you choose to use your donated money is entirely your own choice, but I have no ethical problem with paying for personal projects that have direct benefit to the site. For example, I bought Coda with donated money, and paid for a course in UNIX administration that significantly improved my ability to do my job. I went from absolutely thumb-fingered to only moderately thumb-fingered, and I think the cost was worth it.
Since there is a massive time investment involved in the operation of a torrent site, I wouldn't begrudge you making a profit, either. The excess donations I received would have amounted to something in the ballpark of $1/hr, and I don't think that's unreasonable. However, profiting from the site opens a whole new can of worms. Profiting from copyright violation puts you in different ethical and legal territory. The implication of sale is what landed the EliteTorrents admins in deep shit, and will get you kicked out of Spain, too. As always, it's a personal decision as to where you want your tracker to go, and I decided to take the high road.
Labels:
advertising,
community,
contingency,
donations,
finance,
hosting,
legal
Thursday, October 23, 2008
Botbusting
Okay, this week has heated up a great deal for me, so it doesn't look like I'll be able to continue my coding series until Monday. In the meantime, I want to talk a little about combating bots of different sorts. OnionRings knows more about the technical and security stuff than I do, so he may have some stuff to add or correct. Maybe he'll do some stuff about tightening up the backend, but my expertise is on the frontend/touchy feely stuff, so I'll just focus on that.
A CAPTCHA is an image that is (ideally) impossible for a computer to read but easy for a human. It's used to verify that a user is not a bot, rendering brute-force attacks unfeasible. However, even if it is easy for a human, it's also very annoying. For that reason, you need to use them with moderation.
The biggest technical threat is brute-force attacks on account logins. If someone gets the password of the admin account... well, I don't have to tell you the kind of hurt that can result. Combating this can be as simple as implementing brief (15-minute) bans after a certain number of failed login attempts. I don't think it's feasible to use CAPTCHA in this case because it's something that everyone will be presented with, especially if they don't have their browsers set to keep them logged in. I dislike entering more than a couple of them a day, and any unnecessary ones annoy me. However, if you implement temporary IP bans and change your password regularly, it should afford you a reasonable amount of protection. Other options here may be to require CAPTCHA on all attempts after the first one (decided by IP, not cookie), or to create an account that is used only for administration and give lower permissions to your regular account.
You can use CAPTCHA on registration, but I don't like to spend too long filling out registration forms, either. Besides, a human can register, then let a bot loose on the site. It's still not a great solution, although if you have fairly generic registration forms (or use an unmodified script), that may be a concern.
Instead, I suggest requiring CAPTCHA on a user's first few posts/comments/messages. After the first few, no more annoyances. One idea I just came up with and have yet to implement is simply to require CAPTCHA for all users with a 0/0 ratio. If you're using your account, you're likely not a bot.
No matter how careful you are, some will inevitably slip through. As I've mentioned before, proactively promoting moderators means that what does get through can be dealt with reasonably quickly. This is especially nice in the case of porn spam with images attached, which is never nice to be presented with unexpectedly.
If you have any other suggestions, I'd love to hear them.
Labels:
administration,
bots,
community,
security,
staff
Wednesday, October 22, 2008
Building a Foundation of Knowledge
Since CurlyFries was tied up yesterday, he's unable to continue his Tracker Demystified series today. Instead, I bring you some learning advice and an ambitious reading list from a true bookworm, or book-collector — whichever you prefer. As an admin, the most important thing you can have is a friend that knows what they're talking about when it comes to Linux. Second best is a book to tell you how to do it and give you more detail. Since I'm not a Linux guru by trade, having plenty of people and books around got me out of quite a few tight spots. I wouldn't have normally been able to wiggle out them on my own just Googling. Sometimes even the smallest pieces of advice are worthwhile.
For example, we were trying to do some optimization on the server because TorrentFries' forums were slowing down. When I asked a friend what he knew about optimization, he asked me "How does htop look?" Quite frankly, I had never heard of it, but it's proven to be one of the most useful tools, not to mention the most used. (htop is a more advanced visual extension to top. You'll never use top again.)
Anyways, I've latched onto over half a dozen such people to draw from their Linux expertise now and again, and also so I don't have to go back to the same person time and time again. Networking is more than just TCP/IP. ;)
I'll start you off with my recommendations from the books I've read. I'm not going to claim that they're the best, just that they've provided me with some good background.
Unix Administration
How Linux Works
UNIX, Third Ed.
The Book of Postfix (email - just a casual read)
Development
PHP 6 and MySQL 5 for Dynamic Web Sites: Visual QuickPro Guide
PHP Solutions: Dynamic Web Design Made Easy
SQL Power!: The Comprehensive Guide
Web Design
Web Designer's Reference
Web Standards Solutions: The Markup and Style Handbook
CSS Mastery: Advanced Web Standards Solutions
Pro CSS Techniques
I've read quite a few others, but for the sake of brevity, I'll leave them off. Here are a couple other books that I might also suggest reading even though I haven't read them yet, or only have read excerpts out of.
Apache Cookbook: Solutions and Examples for Apache Administrators
High Performance MySQL: Optimization, Backups, Replication, and More
Apache: The Definitive Guide
Running Linux
Hardening Linux
There you have it. That's well over a year's worth of reading by my standard, but it's a solid foundation of knowledge. Hopefully tomorrow will bear more tracker information.
For example, we were trying to do some optimization on the server because TorrentFries' forums were slowing down. When I asked a friend what he knew about optimization, he asked me "How does htop look?" Quite frankly, I had never heard of it, but it's proven to be one of the most useful tools, not to mention the most used. (htop is a more advanced visual extension to top. You'll never use top again.)
Anyways, I've latched onto over half a dozen such people to draw from their Linux expertise now and again, and also so I don't have to go back to the same person time and time again. Networking is more than just TCP/IP. ;)
I'll start you off with my recommendations from the books I've read. I'm not going to claim that they're the best, just that they've provided me with some good background.
Unix Administration
How Linux Works
UNIX, Third Ed.
The Book of Postfix (email - just a casual read)
Development
PHP 6 and MySQL 5 for Dynamic Web Sites: Visual QuickPro Guide
PHP Solutions: Dynamic Web Design Made Easy
SQL Power!: The Comprehensive Guide
Web Design
Web Designer's Reference
Web Standards Solutions: The Markup and Style Handbook
CSS Mastery: Advanced Web Standards Solutions
Pro CSS Techniques
I've read quite a few others, but for the sake of brevity, I'll leave them off. Here are a couple other books that I might also suggest reading even though I haven't read them yet, or only have read excerpts out of.
Apache Cookbook: Solutions and Examples for Apache Administrators
High Performance MySQL: Optimization, Backups, Replication, and More
Apache: The Definitive Guide
Running Linux
Hardening Linux
There you have it. That's well over a year's worth of reading by my standard, but it's a solid foundation of knowledge. Hopefully tomorrow will bear more tracker information.
Labels:
Apache,
books,
development,
learning,
Linux,
MySQL,
reading,
reference,
Unix,
web design
Tuesday, October 21, 2008
The Tracker Demystified – Part 2: Structure
Now it's time to get started on the coding itself. For debugging purposes, we're going to need a lot more than the single-line tracker status field in normal BitTorrent clients. Happily, since the connectivity is simply via HTTP GET, you could just type the announce URL into your browser's address bar to test the effect of various values. In fact, this is the way I did my testing last time I took it upon myself to write a tracker.
This time, I'm going a little more high class. I've made a simple form with fields for each of the variables (info_hash, peer_id and so forth) that is submitted to announce.php using GET. It's much easier to play around with different values this way. Since forms are pretty simple and not the subject of discussion here, I won't go into any more depth here.
First, even before connecting to the database, I'll load the passed variables and perform a minimum of validation. Like yesterday, I'll be referring to the protocol spec to determine the expected values and ranges of particular variables. I don't like throwing a ton of errors at my user, so I tend to design for graceful failure. For example, if event is anything other than started, completed, stopped, or empty, it will simply default to empty.
Speaking of event, it is going to form the basis of our code structure. The heart of the script consists of a series of conditionals based on the four possible variables. Actually, since empty merely denotes that the client is requesting more peers, we only need the three that require updates on the tracker's end: started, completed, and stopped. These three events map cleanly to three MySQL statements: INSERT, UPDATE, and DELETE, respectively. When a torrent is started, the peer needs a new row in the database, which should be updated when the download has finished and removed when the connection is closed.
Now, stopped only takes input – there's no point in making output when nobody is going to use it. For everything else, we'll use BitTorrent's bencode protocol (detailed in the spec) to output the proper response. Again, I encourage you to go back and peruse the spec. Even if you don't muck around in the backend much, it's nice to be able to pop the hood of any metainfo (.torrent) file you may come across and get a general idea of what's going on in there. The bencoded data may look intimidating, but it's perfectly readable if you take a few minutes to understand how it works.
So, back to the announce. The proper response I mentioned above is a dictionary consisting of two keys/value pairs: interval to indicate the number of seconds until the tracker suggests that you scrape again, and the peers to provide a list of the peer id, ip, and port of each provided peer. There's one bit in the official specification that I feel the need to quote for all the private tracker elitists out there:
Note that downloaders may rerequest on nonscheduled times if an event happens or they need more peers.
Announcing more often than this is perfectly acceptable practice. I do hate to hold it up for ridicule, but BitMe has ironically banned the Mainline (BitTorrent) client originally developed by Bram Cohen himself, developer of the protocol, because it "does not honor the protocol requirements of private trackers". I'm not going to knock on BitMe too much because they're more open than many private trackers in that regard, and they're the only one that I know of that lists banned clients and justifications for banning. However, allow me to take this opportunity to say... what the fuck?
That's a little off-topic. Anyway, now we just have to pull the data from the database. In practice, clients can and do indicate the number of peers they're looking for with the numwant variable, but that's not part of the spec, so I'm ignoring it for now. Instead, I've specified 25 as an arbitrary maximum, and used ORDER BY RAND() in my MySQL query to ensure that all peers get a statistically equal showing no matter where they sit in the database.
Now that we've got the pertinent data, it has to be output, so we'll bencode it according to the structure laid out in the protocol, and shove it out the door. I might talk about writing full functions to read and write bencoded data at some point, but it's a lot more work. The implementation in the announce is simple because we only have to account for the variation between different volumes of data, which is nevertheless being output in a rigid format. Super easy.
Now, while I may not have done a particularly satisfying of describing the last bit of the announce, it is in fact done. The finished script is only 29 lines, including whitespace. I had it heavily commented, but many of the comments repeated what I'm saying here, and I want to emphasize how simple an announce can really be, so I've left it blank and hopefully the minimalism will speak for itself.
At the moment, I'm running three peers: Transmission, Azureus, and my own web browser masquerading as a client. Transmission has successfully seeded the file to Azureus, and both are trying to connect to my web browser to share with it too. Obviously, since it isn't actually a torrent client at all, that's not going to happen.
Now, I see the seasoned coders cringing already, so I'll repeat this once again: this is a barebones demo script. It is designed to provide a framework for you to work off of in forming a more complex announce. The idea is to illustrate the way the protocol works, and the way the tracker interfaces with the protocol. As it stands now, the script doesn't even sanitize input, so it is vulnerable to SQL injection as well as just about every other damn thing.
Tomorrow, I hope to elaborate a bit on these issues, refining this code to create a perfectly usable announce. Ratio tracking is still outside the scope of the project, but we can certainly tighten things up and maybe lay the necessary groundwork for you to go on and add ratios and such frivolity. I say "I hope" because I'm burdened with a lot of work tomorrow and may not get time for a lot of coding. I'll make sure you get an article, and hopefully part 3 of this series, but no promises.
And yes, you're welcome to take the code I wrote above and use it without attribution for whatever purposes you want. However, if you're going to do that, I really suggest that you fix the holes first, or wait until I have a chance to do it for you.
Monday, October 20, 2008
The Tracker Demystified – Part 1: Building the Database
You may have noticed a post on this subject a while back that was unintentionally released before its time. Well, if you have already read that one, re-read this anyway. It's finished, for one thing.
Anyway, this week is going to be devoted to building a simple tracker from scratch. This tracker will do no more than accept and share IP addresses, with no front-end for uploading and downloading metainfo (.torrent) files. However, if you've worked with PHP much, you can probably already work out how to do file uploads and downloads and pretty or at least usable interfaces. This is the big obstacle, and also the bit that has the potential to be interesting for the non-coder, if I can manage to write clearly enough to keep them engaged and slightly comprehending.
Hopefully, over the course of the week, I'll dispel any notions of BitTorrent as a complex or incomprehensible protocol. Once you get your head around it, it's actually quite easy to understand and use. The peer-to-peer bit is a little less straightforward, but happily, we don't have to deal with that. We're writing a tracker, not a client.
My resource in all of this is going to be the official BitTorrent spec, which I know almost by heart. I'll be referring to the spec from time to time, so read it over and my posts might start to make some kind of sense. My MySQL abilities are a little more touch-and-go, so optimization may not be as fantastic as it could be and I'd welcome any constructive criticism on that front.
On that note, we'll be starting today with outlining the basic structure of the announce through the creation of a database table for the peers. This is the only table we'll need, which is handy. Basically, we need to store the pertinent bits of the data that's received by the announce, and enough to provide a coherent response. There are eight variables passed by the client to the tracker: info_hash, peer_id, ip, port, uploaded, downloaded, left, and event. Official or unofficial extensions may add extra values, but we're writing a barebones tracker, so we can safely ignore them. The spec does a perfectly good job of clearly outlining the purpose of each of these variables, so I won't repeat it here.
Now, all of the provided variables are pretty important for various things, but again, this isn't a full-featured tracker, so we'll ignore some of them. We'll save info_hash so we can connect peers on the same torrent to one another, peer_id so we can distinguish one peer from another, and ip and port so we can share the user's address with others on request. As well, while we have no need to track ratios here, we do need to know who is seeding and who is leeching so that we don't waste time sharing seeds' IPs with other seeds. We'll call this variable uploader, and it'll simply be a bit assigned based on the test of left == 0.
At this point, our database looks like this:
If you're familiar with phpMyAdmin, you'll see that I've set a primary index on id, which is standard practice. I've also set an index on info_hash, since we'll be running a lot of queries for it.
Now, at this point, before the more knowledgeable members start tearing holes in my post, I want to point out that I'm working with an ideal model here. No information is lost, all clients follow the protocol to the letter (particularly in always cleanly closing connections), and there are no clients that spoof information for personal gain. Obviously, none of these are true, but since I'm aiming to explain an implementation of the BitTorrent protocol, I'm going to work with these assumptions for the time being, just like how friction is often ignored in introductory physics courses.
Labels:
announce,
backend,
BitTorrent protocol,
MySQL,
PHP,
ratio tracking,
software,
technical
Friday, October 17, 2008
TorrentFries, inc.
In the past, I've made a point of steering clear of the legal nuances of tracker-running. The law varies dramatically from jurisdiction to jurisdiction, and more importantly, I know jack shit about legal matters. However, I just thought I'd throw you a tidbit gleaned from one of my lawyer friends: the beauty of a capitalist society.
Basically, in the US, Canada, and the UK, corporations can only be sued for the assets of that corporation. This applies even if that corporation is just you and the assets are zilch. So long as you make it clear that you are always acting on behalf of the corporation, you're out of any legal troubles scott free.
Obviously, the practical implications are a bit more complicated, but that's my understanding, anyway. I invited him to write a guest entry for us, so if he takes me up on that, you'll get to hear a bit more. In the meantime, please take my "legal advice" with all the salt in the Dead Sea.
Basically, in the US, Canada, and the UK, corporations can only be sued for the assets of that corporation. This applies even if that corporation is just you and the assets are zilch. So long as you make it clear that you are always acting on behalf of the corporation, you're out of any legal troubles scott free.
Obviously, the practical implications are a bit more complicated, but that's my understanding, anyway. I invited him to write a guest entry for us, so if he takes me up on that, you'll get to hear a bit more. In the meantime, please take my "legal advice" with all the salt in the Dead Sea.
Thursday, October 16, 2008
A Moderator's View
Unfortunately, I fell asleep before I could write what should have been yesterday's entry. I hope you didn't miss us too much. I'd like to focus a bit on some experiences I've had as a moderator of TorrentFries. I wasn't always an admin or even moderator at TorrentFries. One day I logged in to find a full host of "shiny new buttons," as we like to call them at TorrentFries, and what a true surprise that was. It's a gratifying feeling to be an uploader and to give to the community, but as a moderator and admin, that responsibility takes on a deeper meaning of giving to the community. I'm glad to have been able to give in that unique way.
As a new moderator, some of my initial duties included moderating the forum, responding to torrent reports and weeding out torrents that didn't follow the rules. As a moderator team, it's obviously pretty important that everyone gets along so the work gets done without too much hassle. If things get slack and neglected, it's apparent right away.
For instance, at one time we were experiencing problems that required users to use the staff contact form to fix, but nobody was addressing the problems. When 25 or 30 messages piled up, it became difficult to manage. I eventually got enough time to clear out the problems, but it took a full 2 hours to take care of. The point is this: our team is good, but without clear direction or responsibility, things tend to not get done if there's a lot of work to get done. The admin has to either pick up this slack or divvy it up, but sometimes that choice has to be made.
Another problem we saw was spam in the forums. Thankfully, there were mods checking the forums, but our registration system didn't include CAPTCHA or any other deterrent to bots, so we began seeing a heavy influx of pornographic material posts. Without fixing the root of the problem, it was difficult to keep the issue under control, and with CurlyFries busy he couldn't rewrite a part of the login. No one else could do it, so we just kept getting bombarded with spam.
Save small issues like these two mentioned, the site as a whole could easily run itself without CurlyFries logging in at all. Nevertheless, there needs to be some direction and fiddling done by the sysop no matter how well the mods work.
Additionally, there always seems to be something that mods should be doing instead of bantering on the forums. It's difficult to moderate torrent comments with any efficiency, and even ensuring that the uploaded content follows the rules seems take a backseat at times. So, in short, as an admin, aim to get a staff that can run the site, but expect that they won't always do everything on their own accord.
As a new moderator, some of my initial duties included moderating the forum, responding to torrent reports and weeding out torrents that didn't follow the rules. As a moderator team, it's obviously pretty important that everyone gets along so the work gets done without too much hassle. If things get slack and neglected, it's apparent right away.
For instance, at one time we were experiencing problems that required users to use the staff contact form to fix, but nobody was addressing the problems. When 25 or 30 messages piled up, it became difficult to manage. I eventually got enough time to clear out the problems, but it took a full 2 hours to take care of. The point is this: our team is good, but without clear direction or responsibility, things tend to not get done if there's a lot of work to get done. The admin has to either pick up this slack or divvy it up, but sometimes that choice has to be made.
Another problem we saw was spam in the forums. Thankfully, there were mods checking the forums, but our registration system didn't include CAPTCHA or any other deterrent to bots, so we began seeing a heavy influx of pornographic material posts. Without fixing the root of the problem, it was difficult to keep the issue under control, and with CurlyFries busy he couldn't rewrite a part of the login. No one else could do it, so we just kept getting bombarded with spam.
Save small issues like these two mentioned, the site as a whole could easily run itself without CurlyFries logging in at all. Nevertheless, there needs to be some direction and fiddling done by the sysop no matter how well the mods work.
Additionally, there always seems to be something that mods should be doing instead of bantering on the forums. It's difficult to moderate torrent comments with any efficiency, and even ensuring that the uploaded content follows the rules seems take a backseat at times. So, in short, as an admin, aim to get a staff that can run the site, but expect that they won't always do everything on their own accord.
Tuesday, October 14, 2008
How Not to Admin a Tracker
I'm assuming most of the audience that is actually moving in the tracker direction is running or planning to run a small operation, for the time being at least. That means spamming all over the place for your site, and that means your previous identity (with all its slip-ups) is irrevocably connected with your site.
Chances are, when you first got involved in the P2P community, you were scared shitless. If you weren't, you should've been. If you were paranoid, you should have made a new account unrelated to any other you might have online. If you didn't, go do that now. Otherwise you're opening yourself up to a whole world of hurt. After a while, you start to relax, to become blasé about the whole copyright issue, and you start to get sloppy again.
Now take this unconcerned you and put it in charge of a BitTorrent tracker. You're probably a little concerned, but that goes away fast. It's just a small operation, after all. It can't hurt.
The problem is that small operations have a way of becoming big operations, and your behavior has a way of sticking around. Even if you've been careful to delete posts as you grow, the curse/blessing that is archive.org will remember your transgressions. All of a sudden, you're at the head of a booming site, and oh shit, what have you done? You go to cover your tracks, but it's too late, of course.
Chances are that if you're headed down this road, you're not far along. You can still change now. Don't make these silly mistakes. If you're going to be involved in your own community, which I recommend, create a new username for it. Don't let hubris enter into the equation. It's tempting to join other torrent sites and go "look at my awesomeness!". Don't. Your admin username should be used only in the context of your own site, and in your dealings with other sites. It's tainted in a different way, and it's dangerous to mix things together.
As well, it's helpful to have an alter ego. Create somebody that is plausibly of the same character as you, but comes from a dramatically different background. Trust me, it's a good idea. You're role-playing, and you should never drop that identity, even with your closest online friends.
Of course, this is the product of much painful experience. Do as I say, not as I do. Accessible as I want the prospect to be, it can be dangerous work, and you have to take it seriously, no matter the size of your site.
Monday, October 13, 2008
The Rules: Writing a Constitution
Yesterday I walked into a shop and was greeted by a sign thanking me for not urinating in the produce. Wait, no I wasn't. You know why? Because there are certain codes of behavior that are implicitly expected of all members of a society. Yet somehow sites feel the need to inform me that I could be banned if I start stirring up hell in the forum. Great, thanks for that.
Quite simply, there are some things that just don't need saying. Sure, you're going to get trolls in your forum, but making a rule telling them to go away is going to do nothing for you. Rules are for the people that genuinely care about their standing with the site. The people that are just around to make trouble aren't going to be reading your rules, and certainly won't make a point of abiding by them. You can't legislate common sense.
So what are your rules for? They're for people like me. I'm a member of a bunch of different trackers, and whenever I sign up, I always take a peek at the rules before I get down to business. I know what a BitTorrent tracker is, I know the usual etiquette, but I just want to find out if there's anything special about how your site does business. Do I have to post in rhyme? Have a cute, fuzzy avatar? Download only certain torrents? I don't want to dig through a bunch of bullshit about obeying "the moderators [sic] expressed wishes! [sic]" in order to find out what really matters.
Conversely, a lot of sites that have automatic ratio bans (which I've ranted about in previous posts) won't detail these bans in their rules. The hardest and fastest (and most variable) rule of all is somehow overlooked. The staff don't have to touch it as the banbot does all the work, but it's the rule with the most impact on new users, good and bad alike. Whether or not I agree with the concept is a moot point, at least tell me.
You're writing a constitution here, not a criminal code. If you've never read your country's constitution, go do it. It's probably available online. In general, they're pretty succinct and accessible for all their importance, and you'd be surprised what rights are being violated. Your criminal code, by contrast, is probably a few thousand pages long. The constitution is basically a vague statement of purpose, while a criminal code gets into the nitty gritties. Since your moderators are not police officers bound by the letter of the law, you don't need a criminal code. You simply need to give guidelines by which your site should operate.
A lot of sites have 15-20 rules, mostly the stuff that came preset with TBdev or whatever they're using. In my mind, you should be able to get everything you need in less than 10, preferably 5 general rules and 5 upload rules, in addition to the behavior that one normally expects of a generic good member elsewhere on the internet. The fewer rules you have, the better.
That's not to say you should have people responding to your moderation actions complaining that they didn't know something was against the rules. If it's the sort of rule a person can't be normally expected to know, then by all means throw it in there. Otherwise, forget about it. You're entrusting your moderators with plenty of discretionary powers, so you don't need to spell everything out.
I'd finish up with some examples from TorrentFries, but quite simply, it's stuff that applies only to that site and to none other. That's kinda the point.
Labels:
administration,
community,
conflict,
ratio tracking,
rules,
staff,
users
Friday, October 10, 2008
Leaving a Host
There comes a time when you need to cover your tracks in a hurry. Since you often can't stick a boot CD in a dedicated host to do a DOD hard drive wipe like you should, you can settle for a few other options. It's nice to feel like you haven't left any incriminating footprints behind.
I find that the best tool for the job is secure remove. The command srm will allow you to do a 38-pass deletion wipe that's good enough for most purposes. Note that it's not impossible to recover the data deleted in this method, just beyond the reach of any small to medium sized organization.
Install secure remove:
Now, you could use this command to bash script a quick exit, but without forking off your processes, a quicker way would be to run each part of your secure removal separately. This works very well if you've got multiple cores or processors. You can use process management tools such as screen or background processes to run multiple srm commands at one time.
The three basic parts of your server you should consider deleting would be the MySQL databases, Apache www folder, and your logs. If you've installed any other software, be sure to check those out as well.
MySQL
Web root
Logs
You'll have to use your judgment on the rest of the system on an individual basis. Don't forget to delete any backups you've made. The last thing you can do is clear out your private files before you exit.
That's all. SRM uses a lot of processor and takes some time on larger directories and files, so be sure to start your deletes in a relatively parallel fashion if you've got to high-tail it out.
Expect to see the series on "The Tracker Demystified" next week. Hopefully, if I write loud enough, that might happen. ;)
I find that the best tool for the job is secure remove. The command srm will allow you to do a 38-pass deletion wipe that's good enough for most purposes. Note that it's not impossible to recover the data deleted in this method, just beyond the reach of any small to medium sized organization.
Install secure remove:
$ sudo apt-get install secure-delete
Now, you could use this command to bash script a quick exit, but without forking off your processes, a quicker way would be to run each part of your secure removal separately. This works very well if you've got multiple cores or processors. You can use process management tools such as screen or background processes to run multiple srm commands at one time.
The three basic parts of your server you should consider deleting would be the MySQL databases, Apache www folder, and your logs. If you've installed any other software, be sure to check those out as well.
MySQL
sudo srm -rv /var/lib/mysql/
Web root
sudo srm -rv /var/www/
Logs
sudo srm -rv /var/log/*
You'll have to use your judgment on the rest of the system on an individual basis. Don't forget to delete any backups you've made. The last thing you can do is clear out your private files before you exit.
sudo srm -r ~/.*
That's all. SRM uses a lot of processor and takes some time on larger directories and files, so be sure to start your deletes in a relatively parallel fashion if you've got to high-tail it out.
Expect to see the series on "The Tracker Demystified" next week. Hopefully, if I write loud enough, that might happen. ;)
CurlyFries adds: Only stuff that you delete with srm will be securely removed. For this reason, do not drop your MySQL databases prior to hitting /var/lib/mysql with your srm stick. If you do so, MySQL will (insecurely) delete your database files, and running srm afterwards will do you no good.
Oh yeah, and hush up. I'm getting to it.
Thursday, October 9, 2008
404 Not Found
The requested URL / was not found on this server.
Apache/2.2.3 (Debian) PHP/5.2.0-8+etch111 Server at ... Port 80
Rest in peace.
January 4, 2007 – October 8, 2008
See you all on Friday.
Wednesday, October 8, 2008
Perils of Remote Access
I thought since I've shared quite a bit of technical information in my past few posts that it was time for an admin's parable. I wrote a little about the perils of remote access servers earlier, but I didn't really go into detail about how 'normal' activities should be rethought in the environment of sole access being remote. Remote administration can be a wisening experience, unless you have some out-of-band administration tool (such as a DRAC card). With may hosts, you may only have the option to do a hard reboot or a reimaging (often at a fee).
In one case, CurlyFries was looking to add a static IP to the server to move critical services off the primary IP and domain. Most services, such as SSH, can listen on any address and can be restricted to certain addresses. It's handily secure to only have ports 80, 443, and 5222 open on a heavy-trafficked IP address.
It's fairly straightforward to add a static IP to a system. I would never dream of being timid about making such a change. At least not nearly as timid as when making a critical Apache or MySQL configuration change. Nonetheless, I managed to screw up with a typo in the /etc/network/interfaces file and got a serious error when restarting the networking. Thankfully, I hadn't lost terminal access, so I went back to look for the problem.
When I thought that I found it and fixed the problem, I restarted networking again. No joy. I was well aware of the implications of having a bad network interfaces file: no network interface access. So, I restored the backup configuration and restarted the service. At that point, my terminal died and I was then unable to establish any SSH connections to the server. The web server was down. Confounded, I cursed Debian for being so ridiculous and hoped that the configuration had taken effect. Within the hour, we had our TorrentFries rebooted, and to my relief the original configuration was working.
Had the networking stopped working when I tried to load the invalid configuration instead of when I loaded the original working configuration, we would have been toast. Not only would we have been forced to do a complete reinstall, but we would have lost quite a bit of data in the process due to not moving the rather infrequent non-automated backups off site regularly. *cough*
We barely escaped a disaster that would have spelled the second time in my capacity as an admin to need an OS reinstall on the remote server. The only time we've had to reinstall was when a certain curly type of administrator dropped the MySQL privileged users and my consequent mucking with things resulted in an inoperable system. The lesson to take away is threefold:
In one case, CurlyFries was looking to add a static IP to the server to move critical services off the primary IP and domain. Most services, such as SSH, can listen on any address and can be restricted to certain addresses. It's handily secure to only have ports 80, 443, and 5222 open on a heavy-trafficked IP address.
It's fairly straightforward to add a static IP to a system. I would never dream of being timid about making such a change. At least not nearly as timid as when making a critical Apache or MySQL configuration change. Nonetheless, I managed to screw up with a typo in the /etc/network/interfaces file and got a serious error when restarting the networking. Thankfully, I hadn't lost terminal access, so I went back to look for the problem.
When I thought that I found it and fixed the problem, I restarted networking again. No joy. I was well aware of the implications of having a bad network interfaces file: no network interface access. So, I restored the backup configuration and restarted the service. At that point, my terminal died and I was then unable to establish any SSH connections to the server. The web server was down. Confounded, I cursed Debian for being so ridiculous and hoped that the configuration had taken effect. Within the hour, we had our TorrentFries rebooted, and to my relief the original configuration was working.
Had the networking stopped working when I tried to load the invalid configuration instead of when I loaded the original working configuration, we would have been toast. Not only would we have been forced to do a complete reinstall, but we would have lost quite a bit of data in the process due to not moving the rather infrequent non-automated backups off site regularly. *cough*
We barely escaped a disaster that would have spelled the second time in my capacity as an admin to need an OS reinstall on the remote server. The only time we've had to reinstall was when a certain curly type of administrator dropped the MySQL privileged users and my consequent mucking with things resulted in an inoperable system. The lesson to take away is threefold:
- Backup offsite before making any changes that could effect the operation of the site.
- Be mindful of what senarios could leave you with no server access. These generally include but are not limited to network configuration, SSH configuration, firewall configuration, and boot loader configurations.
- Don't close an open terminal session unless you're sure you can get back in. After I make some changes, I open a second SSH session to ensure access before exiting out of my working session.
Labels:
administration,
backup,
MySQL,
network,
remote access,
SSH
Tuesday, October 7, 2008
Peer Management
Okay, I'm not in fantastic condition at the moment, but I want to keep things going. I'll just grab on to a recent suggestion – discussing ratios in a community context – and run with it.
Basically, any simple, closed system will be balanced, no matter the context: total uploads will equal total downloads. No matter how your tracker records data, it will always be tracking as much uploaded data as it does downloaded data, neglecting hash fails.
Therefore, when the ratio tracking system is altered, you will do nothing but alter the community dynamic. There will always be supposed gormless bottom-feeders at the bottom of your scale, and theoretical paragons of virtue at the top. Sorta. In my experience, chronic leechers are mostly those that simply haven't been initiated into proper torrent etiquette. I know in my early days of torrenting, I was quite unnerved by the prospect of others being able to connect to and download from me. To be sure, I didn't go out of my way to do this, until it became a point of personal pride to maintain a ratio above 1.0. It is only my involvement in the community that drove me to do this.
Now, think of an open site like The Pirate Bay. The tracker is simply a collection of its parts, each behaving according to their own conscience. The leechers leech according to their desire for gratification (or lesser experience, or lesser ability), while the seeders do so because of pride or generosity or simply because the swarm is so slow they end up uploading far more data than the download size by the time they complete. And you know what? It works. It may not work well, but it works.
Now, ratio-obsessed private trackers choose to whack off that bottom demographic, the people that are disinclined to share. That's all well and good, but you simply move the community's scale. Instead of having the true leechers at the bottom end of the scale, now you have the metaphorical middle class down there. In my experience, such systems tend to drift ever upwards as users are banned for poor ratios – in other words, as the bottom end of the spectrum is chopped off again and again. Obviously, these systems rely on new registrations to add a continual stream of new members to slow or halt that drift.
This is compensated for by free leech days, golden torrents, what have you. However, the only people that will participate in this activity are those that are genuinely concerned about ratios. They're not downloading torrents because they want to, but because it will improve their status in the community. To me, this is contrary to the purpose of file sharing. You share what you've got because you want to share it, and people accept what you're sharing because they want to have it.
Basically, these events are designed to fix a system that doesn't work properly all the time. If you need to schedule special events to compensate for your model, you must be doing something wrong. I would prefer to have a system that works in the same way and with the same effectiveness every day, rather than just once in a while.
So where's the solution? I don't have an answer. I don't think anyone does. I certainly do use a couple of private sites, but I also use the big boys when they have what I need or when I don't feel like destroying my ratio trying to seed back. "If it ain't broke, don't fix it," comes to mind, but I can't deny that swarms on private trackers do seem to perform better, so clearly something has been fixed or at least improved upon. Personally, I would prefer to see more personal treatment towards leechers. I believe that most leechers simply haven't been properly introduced to the social protocol (heh) associated with torrenting. If someone is actually intentionally and knowingly violating accepted behavior, they will take pains to hide it by using a cheating client of some sort. Therefore, wouldn't it be logical to take some means to inform the uninformed? I prefer not to participate in a community that is ultimately built on fear. It smells bad.
This is just meant as some food for thought, and hopefully will dispel some feelings of superiority on either side. Again, I'm sorry for my unfocused meandering, for my inability to make a solid point, and for any typographical errors I may have made. I'll probably read this over sometime when I'm in better intellectual shape and say, "I can't believe I said that," but there's nothing to be done. At least you got a post.
I had intended to write my tracker tutorial this week, but ah well. I'll do it sometime in the future. Stay tuned for that.
Labels:
community,
conflict,
culture,
dump sites,
ratio tracking
Monday, October 6, 2008
We Interrupt Your Regular Programming...
Sorry for the service interruption. My internet was offline yesterday, so I couldn't publish a post, and it seems that my organization was working against me as Blogger automatically published a draft I had scheduled for today. We'll return to our regularly-scheduled programming tomorrow.
Friday, October 3, 2008
BitTorrent Communities
I didn't want to bore you with too much Linux, so today I'll write about the elements of a BitTorrent community from the eyes of a member of a handful and an admin of one. I would encourage anyone that still reads our posts to comment with any questions you might have, and we will try to address everything we can.
When analyzing BitTorrent communities from an admin perspective, I often consider the following goals.
Maintaining higher standards on torrent descriptions coupled with user comments improves at least perceived quality. When users have a way to gauge the details of the torrent data without downloading, they'll be more comfortable engaging. Have an explicit example of how torrent descriptions should look on the user upload page. Instead of simply titling your comments section "Comments" on the torrent page, consider something like "Comment on this torrent's quality".
When users aren't downloading or uploading material, what are they doing? Many sites consume users' excess time through interactive forums and shoutboxes. A lively forum can be the mainstay of a community. Conversely, a dead or hostile forum can turn away would-be outstanding members away. Be friendly and don't let users or moderators become short with users even if they are n00bs, the whole community will align on the staff and prominent posters' viewable culture.
User interface is key. Make it easy for users to drill down the material they're looking for using categories and subgroups. Search is the most important thing for a torrent site, so make sure it is prominently positioned, works well, and is predictable and consistent. Otherwise, negative results are a reflection on the material tracked. Also, I can't stress the importance of a good site design. Mediocrity won't get you very far when attempting to gain users and material early on.
As mentioned before, ratio tracking helps to mentally encourage users to give back to the community. Distinguish uploaders and otherwise exceptional members in the community or figure out a way to give them a warm fuzzy feeling. Make this visible wherever users interact.
The overlying theme is culture. Define a culture you find appealing and align your entire site around it. Culture exudes itself in every aspect of your site from the site rules to each forum post, so take some time to think about what you want to display. A little psychology and management will go a long way.
When analyzing BitTorrent communities from an admin perspective, I often consider the following goals.
- Maximize uploaded material quantity
- Maximize upload quality
- Maximize user involvement
Maintaining higher standards on torrent descriptions coupled with user comments improves at least perceived quality. When users have a way to gauge the details of the torrent data without downloading, they'll be more comfortable engaging. Have an explicit example of how torrent descriptions should look on the user upload page. Instead of simply titling your comments section "Comments" on the torrent page, consider something like "Comment on this torrent's quality".
When users aren't downloading or uploading material, what are they doing? Many sites consume users' excess time through interactive forums and shoutboxes. A lively forum can be the mainstay of a community. Conversely, a dead or hostile forum can turn away would-be outstanding members away. Be friendly and don't let users or moderators become short with users even if they are n00bs, the whole community will align on the staff and prominent posters' viewable culture.
User interface is key. Make it easy for users to drill down the material they're looking for using categories and subgroups. Search is the most important thing for a torrent site, so make sure it is prominently positioned, works well, and is predictable and consistent. Otherwise, negative results are a reflection on the material tracked. Also, I can't stress the importance of a good site design. Mediocrity won't get you very far when attempting to gain users and material early on.
As mentioned before, ratio tracking helps to mentally encourage users to give back to the community. Distinguish uploaders and otherwise exceptional members in the community or figure out a way to give them a warm fuzzy feeling. Make this visible wherever users interact.
The overlying theme is culture. Define a culture you find appealing and align your entire site around it. Culture exudes itself in every aspect of your site from the site rules to each forum post, so take some time to think about what you want to display. A little psychology and management will go a long way.
Thursday, October 2, 2008
Protocol
Okay, this is going to be a quick post because OnionRings and I both have a ton of work to do and Ketchup seems to be maintaining our collective laziness quota.
I'm planning to offer a detailed examination of what exactly goes into the technical end of a BitTorrent tracker by offering explanations as I write my own, but that's well beyond my time allotment for the evening. Instead, I'll write a simple summary of the protocol.
The peer-to-peer aspect is not important for our purposes. Unless you're planning on writing a client, which is well outside the scope of this site, you don't even need to think about it. I know I don't. I really haven't taken the time to study it in much detail, aside from understanding what's going on in general terms.
Happily, the peer-to-tracker protocol is very simple to understand. The tracker is inherently passive, never initiating connections itself. Instead, clients connect to the tracker via HTTP. They provide the infohash (unique identifier) for the torrent they're trying to download, their current port, and a bunch of other information indicating the state of the client. The tracker responds with a simple list of IP addresses that are also downloading the same file. It's up to the client to initiate connections with the provided peers. Since the protocol is peer-to-peer, any client can initiate the connection, assuming that the other computer is properly configured to accept incoming connections.
That's all a tracker has to do: log the IP addresses that connect, and respond with a list of suggestions for possible peers. The process is fast, light on resources, and very easy to code. You can develop a tracker just like any website, using just about any language you want: PHP, ASP, Ruby, whatever. The biggest trackers run on C or other such languages, since it's fast and lightweight beyond the dreams of PHP. However, for your purposes, web languages are all you need.
That said, I'll pick up on the subject later on when I actually find the time to get going on development. Like I say, things have been nuts of late.
Wednesday, October 1, 2008
Back to the Basics
I realize that our focus has of late moved from tracker-specific discussion to more-general stuff that applies to any site or server and can be found in a bazillion places online. Since this blog is unique because it does chronicle the operation of a BitTorrent tracker, I'm going to get back to the meat of the thing for a bit. OnionRings plans to continue with his Linux series, but is sadly indisposed this evening.
In some senses I will cover some of the same material as our second-ever blog entry, but with a different focus. While that entry gave a quick overview of why and how to set up a tracker, I'm going to look a little more closely at the technical and logistical aspects of bringing a tracker online.
First off, what do you personally need to bring to the table? (You're probably the only one at the metaphorical table at this point, but the bringing should commence now.) Beyond anything else, you need to understand the BitTorrent protocol, and understand your tracker of choice. We'll talk a bit about choosing trackers shortly. If you plan on running a PHP/MySQL tracker, you'd better have at least a working knowledge of PHP and MySQL. From the get-go, you'll need to be able to peer into the inner workings of your site and see what makes it tick. You don't just need an understanding of the protocol, but of your own tracker. Trust me, admin panels are nice, but some things just require you to get down and dirty. However complete your chosen script may look at first glance, there will quickly come a time when you or your users bemoan the lack or poor design of some feature or another, and you will be forced to go in and remedy the situation. As the tracker grows and moves to its first dedicated server, it will also become important to know your way around Linux to some extent.
Secondly, you need a hook to pull people in. The most obvious aspect is to provide something that others can't. A unique idea is neat, but it's entirely possible to carve out a niche in a "market" that is well-developed but not saturated. For example, if you have terabytes of obscure movies that you have spent years collecting, starting a movie tracker to share these can be an excellent starting point. Of course, you will need to keep all the uploaded torrents active, even though you will be getting very few peers at first. For this purpose, you may want to consider renting a seedbox. The added cost may be hard for a brand new tracker admin to swallow, but your ability to saturate the connections of your first members will do a lot to encourage them to stick around and maybe even contribute some stuff of their own. As well, a unique design may sound frivolous, but it will lend your site credibility and a sense of longevity. I've covered some methods of early promotion in the post I mentioned above, so I won't rehash them here.
I hate to be defeatist, but if you can't meet these requirements, you should think twice about starting a tracker. Of course, the lovely thing about being a human is the capacity to learn, but you should get going on that learning well before you even consider getting into tracker territory. While I'm trying to demystify the role of the tracker administrator, it's certainly a job that not everyone has the skill and disposition to fill with any great success.
Moving on to more technical requirements, you will also need to choose a tracker. That's a given. The three most popular trackers under active development are TBdev, TorrentTrader, and the new Project Gazelle, although there are a ton of other options out there, so you shouldn't feel constrained to choosing one of these three. Shop around a bit. Detailed comparison of these and other trackers is well outside the scope of this entry, and perhaps even of this blog, but this should point you in the right direction. At the end of the day, it's up to you to select the tracker that best suits your particular tastes and needs. As a side note: for the love of God, don't use TorrentTrader Classic.
So, why not just boot Azureus or µTorrent or one of these handy ubiquitous little torrent clients that happen to include the ability to operate as standalone trackers? Hopefully this isn't a question you were asking yourself, but I'll answer it anyway. First, this isn't the purpose they were designed for. Just because they can run a tracker doesn't mean they should. They're not optimized for the purpose, they don't include features that are necessary for the smooth operation of a tracker (they're inherently open to anyone that wants to use them), and they include a ton of features and bloat that will just bog down your server. What's more, part of the sort of tracker we're discussing here is the index, or frontend. This is the bit where torrents are uploaded, downloaded, where users interact with one another, all that sort of stuff. The tracker itself is ridiculously simple in operation, as we will see in my eventual piece-by-piece breakdown of what it takes to build one from scratch. The important business is the complex frontend, and BitTorrent clients' wannabe trackers just can't provide this.
So, once you've selected the script that best fits your needs, it's time to choose a host. I've had some requests to list a number of torrent-friendly hosts, but that's not something I'm going to do for a number of reasons. The most important reason here is that I don't want to endorse a host that turns out to be unsafe, or to lull readers into a false sense of security. You may have noticed that even (especially) the biggest trackers tend to move around a lot. The world of web hosts is intrinsically volatile, so to proclaim that one host is perfectly safe, even if presently true, is to ignore the fact that this may not continue to be the case in the future. LeaseWeb was once a safe haven for trackers, but after losing a lengthy court battle on behalf of Demonoid and other trackers, all the will to fight has gone out of them. Nowadays, if a copyright holder says "boo", they'll turn around and shut you down.
So simply accept that no place is perfectly safe, and operate under the assumption that you could be taken down tomorrow – you could be. A good strategy to find places that are safer is to run whois lookups on established trackers' IP addresses. Presumably, if they are able to operate on a particular host, that host must be somewhat resistant to legal threats. The only problem is that many larger trackers own their own servers which they operate via colocation in the host's datacenter, while you are likely unable to afford more than a rented dedicated server (an important distinction to make). Not all hosts supply both options, and not all do so for reasonable rates. In addition to colocated and dedicated servers, I've talked about shared and virtual hosting in that previous article, so I'm not going to rehash that here.
This should be all you need to get you off the ground. Again, I'm not going into exacting detail on some of the technical points because I'm assuming that you will be able to handle the fiddly bits of your own accord. If you can't, like I mentioned, you probably shouldn't be running a tracker.
If there are any points on the subject (or any other, really) that you would like us to cover, leave a comment on this post and we'll do our best to touch on them in future posts.
Labels:
administration,
announce,
BitTorrent protocol,
configuration,
contingency,
copyright,
hosting,
MySQL,
PHP,
planning,
software,
technical
Tuesday, September 30, 2008
Site Administration Tools
Monday I wrote a technical piece about installing LAMP so today I'll talk a little about the type of administrative tools we use on the site. I've always been a big fan of using the tools UNIX gave us. That means strict command line pleasure. It means things are clean on the system and no extra resources are used or security holes opened. But not everybody agrees. Even I find it nice to use a GUI once and awhile so as not to get bogged down with the mechanics of the command line.
MySQL seems to be the toughest to administer with the tools UNIX gurus gave us. Writing SQL statements isn't bad if you're familiar with SQL, but you'll likely feel better with PHPMyAdmin or MySQL Administrator (MySQL Query Browser). The difference is that PHPMyAdmin is web-based and MySQL Administrator is a remote client. To me, the difference is that there's no garbage on the server when you use a remote client to connect to to your server like there is with installing PMA.
Consider the security implications as well. When installing PMA, you open up an avenue for attack via the web. When using MySQL Admin allowing remote connections, you're doing even worse. Therefore, the safest and most efficient option is to tunnel your MySQL Admin connection over SSH to get a localhost connection using a remote tool. I think this method applies for other remote tools as well. Ketchup likes using MySQL-Front. Using a remote tool allows room for preference.
In other areas, it can be handy to have a GUI as well. If you have many servers running on your box (such as mail, Jabber, and Apache/MySQL) it can be handy to increase your proficiency and ability to edit obscure configuration files with greater ease by installing a management tool. I've used Webmin before, and I have mixed feelings. It gives very good control over the multitude of configurations on a Linux box, but I always see it as another security vulnerability.
That's all for now, unfortunately. Next time I post, we'll discuss more about LAMP configurations.
MySQL seems to be the toughest to administer with the tools UNIX gurus gave us. Writing SQL statements isn't bad if you're familiar with SQL, but you'll likely feel better with PHPMyAdmin or MySQL Administrator (MySQL Query Browser). The difference is that PHPMyAdmin is web-based and MySQL Administrator is a remote client. To me, the difference is that there's no garbage on the server when you use a remote client to connect to to your server like there is with installing PMA.
Consider the security implications as well. When installing PMA, you open up an avenue for attack via the web. When using MySQL Admin allowing remote connections, you're doing even worse. Therefore, the safest and most efficient option is to tunnel your MySQL Admin connection over SSH to get a localhost connection using a remote tool. I think this method applies for other remote tools as well. Ketchup likes using MySQL-Front. Using a remote tool allows room for preference.
In other areas, it can be handy to have a GUI as well. If you have many servers running on your box (such as mail, Jabber, and Apache/MySQL) it can be handy to increase your proficiency and ability to edit obscure configuration files with greater ease by installing a management tool. I've used Webmin before, and I have mixed feelings. It gives very good control over the multitude of configurations on a Linux box, but I always see it as another security vulnerability.
That's all for now, unfortunately. Next time I post, we'll discuss more about LAMP configurations.
Monday, September 29, 2008
Linux Part 2: Installing LAMP
I'm going to resume from Linux Part 1: Installing Configuring Sudoers and OpenSSH with part 2: Installing LAMP. In the world of Ubuntu, virtually no one installs LAMP from a command line on Ubuntu Server. Ubuntu provides the option to automatically install LAMP during OS setup, so most of the time, it's done that way. Purchasing a dedicated server often doesn't give you that option, however. Yes, many tutorials cover installing LAMP, and they are just as good, but note that they're installing on Ubuntu Desktop, sometimes assuming you have a GUI. Thankfully, installing LAMP on Ubuntu from the command-line isn't hard at all.
Let's start by running the best command ever, apt-get. This will get all the packages we need.
Alternatively, you can try
Apt will download the packages and install. Then, you'll see this as MySQL is being configured:
You'll have to confirm your password, and then you're done. Restart Apache.
Let's make sure everything went okay. Browse to your IP address. You should see an "It Works!" page. This means Apache is running. Let's check PHP. Run
Write out this single line to the file:
Browse to your addresss/test.php and you should see lots of information about the current state of PHP.
Securing MySQL
That installation was so easy, we'll do a little security cleanup too.
Log into MySQL:
Your prompt will change to mysql>
Create a user you'll be using to administrate the databases. By default, MySQL is secure in the fact that it will only allow root to login locally. That's excellent until you realize that MySQL can be a pain to administer locally. I don't mind too much, but some people like to install PHPMyAdmin. This can open you up to potential remote access attacks on the root user's password. Therefore, if you're going to install PMA, we'll get rid of the root user to make it more difficult to enumerate users. Let's start by creating a MySQL account, Zeus.
You should consider also adding the privileges to this user @ 127.0.0.1
At this point, you should create a user that will be the one accessing the database. Using your "God" user to access MySQL from your web application is bad karma and invites security risks. Google 'create mysql user' for the syntax on creating a user with less privileges.
Feel free to be creative with your names...
Flush the privileges and exit.
DO NOT ATTEMPT TO FORGET THIS PASSWORD! It's a pain to restore a MySQL root password, and even more of a pain when the root user doesn't exist, (some say impossible) so take it from me-- remember your privileged user's password for cripes' sake.
Login and delete the root and anonymous user once zeus was created.
Check out your work:
Don't drop the debian-sys-maint account.
Apache and MySQL Control
Here's how you can restart Apache and MySQL without rebooting your server if you have changed a configuration file.
Replace 'restart' with 'stop' or 'start' if needed.
Thanks for reading again, next we'll show you a bit about the different administrative tools available for Linux.
Let's start by running the best command ever, apt-get. This will get all the packages we need.
sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 mysql-server libapache2-mod-auth-mysql
Alternatively, you can try
sudo tasksel install lamp-server
Apt will download the packages and install. Then, you'll see this as MySQL is being configured:
You'll have to confirm your password, and then you're done. Restart Apache.
sudo /etc/init.d/apache2 restart
Let's make sure everything went okay. Browse to your IP address. You should see an "It Works!" page. This means Apache is running. Let's check PHP. Run
sudo nano /var/www/test.php
Write out this single line to the file:
<?php phpinfo(); ?>
Browse to your addresss/test.php and you should see lots of information about the current state of PHP.
Securing MySQL
That installation was so easy, we'll do a little security cleanup too.
Log into MySQL:
mysql -uroot -p
Your prompt will change to mysql>
Create a user you'll be using to administrate the databases. By default, MySQL is secure in the fact that it will only allow root to login locally. That's excellent until you realize that MySQL can be a pain to administer locally. I don't mind too much, but some people like to install PHPMyAdmin. This can open you up to potential remote access attacks on the root user's password. Therefore, if you're going to install PMA, we'll get rid of the root user to make it more difficult to enumerate users. Let's start by creating a MySQL account, Zeus.
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'localhost'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;
You should consider also adding the privileges to this user @ 127.0.0.1
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zeus'@'127.0.0.1'
-> IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;
At this point, you should create a user that will be the one accessing the database. Using your "God" user to access MySQL from your web application is bad karma and invites security risks. Google 'create mysql user' for the syntax on creating a user with less privileges.
Feel free to be creative with your names...
Flush the privileges and exit.
mysql> FLUSH PRIVILEGES;
mysql> quit
DO NOT ATTEMPT TO FORGET THIS PASSWORD! It's a pain to restore a MySQL root password, and even more of a pain when the root user doesn't exist, (some say impossible) so take it from me-- remember your privileged user's password for cripes' sake.
Login and delete the root and anonymous user once zeus was created.
mysql -uzeus -p
mysql> DELETE FROM mysql.user WHERE User = ' ';
mysql> DELETE FROM mysql.user WHERE User = 'root';
mysql> FLUSH PRIVILEGES;
Check out your work:
mysql> SELECT User, Host FROM mysql.user;
Don't drop the debian-sys-maint account.
Apache and MySQL Control
Here's how you can restart Apache and MySQL without rebooting your server if you have changed a configuration file.
sudo /etc/init.d/apache2 restart
sudo /etc/init.d/mysql restart
Replace 'restart' with 'stop' or 'start' if needed.
Thanks for reading again, next we'll show you a bit about the different administrative tools available for Linux.
Labels:
Apache,
configuration,
installation,
LAMP,
Linux,
MySQL,
Ubuntu
Friday, September 26, 2008
Terminal Case
I was going to continue OnionRings' series on Linux configuration, but I don't have access to a clean Ubuntu install at the moment (or any other kind, actually), so I'll leave that topic for now. Instead, I'll rant a little about the future of the MAFIAA, and the RIAA in particular.
The popular opinion among pirates is that downloading music hurts no one. I've heard a variety of justifications for this, but the most popular opinion is that the free publicity (which presumably leads to album sales) counteracts lost sales. I'm no expert, but you're sure as fucking hell less of an expert than I am. What I will say is that, if this is true, it's true only for small bands and independent labels. The big boys can afford to buy as much publicity as they want. So they're right, it is hurting their sales. It's not hurting their sales by the number of copies downloaded times the full sticker price of the album, but it's hurting them nonetheless.
Don't mistake this for a bad thing. The big labels would have you believe that without them, there would be no music. Listen, bud. There was music before you came along, and there will be music long after you are laid to rest. What we will see the death of is McMusicians, singers and bands that everyone has heard of and that the most gullible of us consider to be the epitome of good music. You know who I'm talking about.
What will be left without the big labels? Musicians that aren't in the business of making millions of dollars. Record labels that are employed by the musicians, not the other way around. A world where the music industry is easier to break into because people are more open-minded about music they haven't heard on the radio at least 15 times. A world without the fat, corrupt husk that pretends to serve the needs of musicians and music-lovers alike.
The best bit of all: you can live in that world today. Listen to internet radio and find new bands. Go anywhere with live music, be it a concert, bar, club, coffee shop, or street corner. Buy independent records from independent record stores. Learn to play an instrument yourself! The labels aren't afraid of downloaders. They're afraid we'll realize that we don't need them. Downloading music won't hurt them as much as the realization that you never need to buy anything from them, ever again.
Naturally, all of the sentiments above can be directly applied to the film industry as well. Notice how movies are all shit these days? Try independent cinema. Trust me, there are still movies being made that don't suck.
Okay, that's all for now. I'll see you on Monday, or better yet, OnionRings will. I'll see if I can make him do an article next time.
A brief update: apologies to anyone that was expecting this to be a pun and for me to cover command-line stuff. I think OnionRings is indeed planning to get his series going again for Monday, and I'll try to get a little further ahead on my posting as well.
Wednesday, September 24, 2008
Linux Part 1: Installing Configuring Sudoers and OpenSSH
Okay, I'm going to start you off with a little advice for beginning Linux users looking to get rolling with a torrent site such as ours. Note that startup sites will most likely look for shared hosting whereas the following advice is aimed at admin's moving from a shared host to a virtual private server (VPS) or a dedicated server (dedi).
In these posts, I'll be customizing the advice for a PHP/MySQL tracker such as ours at TorrentFries. I'll also be running under the assumption that you're running Ubuntu Server Linux, specifically Hardy Heron 8.04. Additionally, these are pretty common instructions you can find at a variety of blogs across the net.
Let's just get some things straight before we get technical. If you're new-ish to Linux, I'd recommend Ubuntu. If you disagree, you don't need to be reading this anyway. Ubuntu isn't the most rock-solid distro available, it's just the easiest to administer (in my realm of experience) and it's stable enough that I've never crashed it on accident. It can also be found with most hosting companies. If not, Debian is the closest to Ubuntu and feels nearly the same.
If you're concerned, check out these survey-based distribution choosers:
Logging In
You'll find that your hosting company will give you nothing more than an IP address (or a couple if that's the case) and a root login password. This means that you'll be accessing a server via Command Line Interface (CLI) over Secure SHell (SSH).
On Windows, I recommend using PuTTY for terminal access.
On Macs, you have Terminal!
Starting Configuration
Once you're logged in, create an admin group.
If the admin group already exists, that's fine. Now create the first user with -g being the primary group (often just the username), -G being secondary group (admins), -s is the location to the shell, bash, -p is the password (you can omit this flag if you want), -d being the home directory, and -m creates the directory if it doesn't exist, and finally the username itself.
The password you just set can be tricky, so to be sure...
You can add any other users in this fashion.
Let's give your new user sudo access first. Sudo lets you run commands as root while not logged in as root. Once, you'll have to use the UNIX text editor 'vim', but in the future, you can use a much easier text editor called 'nano'.
Edit: Ubuntu Intrepid Ibex (8.10) now uses nano for visudo.
Here's a quick tutorial on vi(m). Only use visudo when editing the sudoers file. Using any other method may cause corruption in the file, so make it a habit to use visudo.
Make sure your sudoers file looks like this in the group section:
Now that you have sudo access, try connecting with a new session. You should now see something like
instead of
Let's use our new user from now on.
Configuring OpenSSH
Now, lets lock down SSH. First, copy the config file so we can restore the original settings at any time.
Now, let's edit the sshd_config file with nano text editor.
Look for:
LogLevel INFO
Change to:
LogLevel VERBOSE
This will allow us to get more information about SSH activity. It's handy to see who's logged in and who's tried to log in. Run grep -ir ssh /var/log/* | less to get that good data you're collecting.
Look for:
PermitRootLogin yes
Change to:
PermitRootLogin no
This will prevent hackers from brute forcing the root password, since root login is disabled. They'll still try to enumerate your users, but this is a critical step.
Look for:
LoginGraceTime 120
Change to:
LoginGraceTime 20
Changing the login time will slow down any would-be automated attacks.
If you'd like, you can change the port you connect to on SSH if you feel better being more obscure. Just change the 22 to what you wish:
Save and exit the file and restart OpenSSH:
You won't notice any changes until you reconnect. When you do, remember that you won't be able to log in as root. If you need root, you will have to use sudo or su. Also, if you changed your port, remember to change it in PuTTY when you reconnect.
For more info on OpenSSH, check this out.
Up next: installing and configuring LAMP.
In these posts, I'll be customizing the advice for a PHP/MySQL tracker such as ours at TorrentFries. I'll also be running under the assumption that you're running Ubuntu Server Linux, specifically Hardy Heron 8.04. Additionally, these are pretty common instructions you can find at a variety of blogs across the net.
Let's just get some things straight before we get technical. If you're new-ish to Linux, I'd recommend Ubuntu. If you disagree, you don't need to be reading this anyway. Ubuntu isn't the most rock-solid distro available, it's just the easiest to administer (in my realm of experience) and it's stable enough that I've never crashed it on accident. It can also be found with most hosting companies. If not, Debian is the closest to Ubuntu and feels nearly the same.
If you're concerned, check out these survey-based distribution choosers:
Logging In
You'll find that your hosting company will give you nothing more than an IP address (or a couple if that's the case) and a root login password. This means that you'll be accessing a server via Command Line Interface (CLI) over Secure SHell (SSH).
On Windows, I recommend using PuTTY for terminal access.
On Macs, you have Terminal!
Starting Configuration
Once you're logged in, create an admin group.
groupadd admin
If the admin group already exists, that's fine. Now create the first user with -g being the primary group (often just the username), -G being secondary group (admins), -s is the location to the shell, bash, -p is the password (you can omit this flag if you want), -d being the home directory, and -m creates the directory if it doesn't exist, and finally the username itself.
useradd -Gadmin -s/bin/bash -pPassword -d/home/UserName -m UserName
The password you just set can be tricky, so to be sure...
passwd UserName
You can add any other users in this fashion.
Let's give your new user sudo access first. Sudo lets you run commands as root while not logged in as root. Once, you'll have to use the UNIX text editor 'vim', but in the future, you can use a much easier text editor called 'nano'.
Edit: Ubuntu Intrepid Ibex (8.10) now uses nano for visudo.
visudo
Here's a quick tutorial on vi(m). Only use visudo when editing the sudoers file. Using any other method may cause corruption in the file, so make it a habit to use visudo.
Make sure your sudoers file looks like this in the group section:
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
Now that you have sudo access, try connecting with a new session. You should now see something like
UserName@host$
instead of
root@host#
Let's use our new user from now on.
Configuring OpenSSH
Now, lets lock down SSH. First, copy the config file so we can restore the original settings at any time.
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original
Now, let's edit the sshd_config file with nano text editor.
sudo nano /etc/ssh/sshd_config
Look for:
LogLevel INFO
Change to:
LogLevel VERBOSE
This will allow us to get more information about SSH activity. It's handy to see who's logged in and who's tried to log in. Run grep -ir ssh /var/log/* | less to get that good data you're collecting.
Look for:
PermitRootLogin yes
Change to:
PermitRootLogin no
This will prevent hackers from brute forcing the root password, since root login is disabled. They'll still try to enumerate your users, but this is a critical step.
Look for:
LoginGraceTime 120
Change to:
LoginGraceTime 20
Changing the login time will slow down any would-be automated attacks.
If you'd like, you can change the port you connect to on SSH if you feel better being more obscure. Just change the 22 to what you wish:
Port 22
Save and exit the file and restart OpenSSH:
sudo /etc/init.d/ssh restart
You won't notice any changes until you reconnect. When you do, remember that you won't be able to log in as root. If you need root, you will have to use sudo or su. Also, if you changed your port, remember to change it in PuTTY when you reconnect.
For more info on OpenSSH, check this out.
Up next: installing and configuring LAMP.
Subscribe to:
Posts (Atom)