BadCheese.com

This is my response to this blog entry: http://gibsonandlily.com/blogs/48

This is an open letter from us, that is, the system administrators, to you, management, users, and everybody that has to deal with us and our sometimes very bizarre behavior.

I'm 41 years old and I've been a linux/unix system administrator for about 11 years. Before that I was a unix programmer for about 5 years. One day when I was only a few years out of my C.S. degree, I shared an office with our solaris sysadmin. We became good friends and got along great, except for when I asked him to install something on a system for me, he'd give me this look that said to me, "Dude, we're great friends, why do you want to spoil that and have me do some work for you?". I never really understood that look. I figured that it was his job to support me and to do what I ask him to do. If I need something done to do my job, he should do it without any attitude and just let me know when it's done. We continued to be great friends throughout the company's existence and still contact him from time to time, but I could never get over or understand that look ... until now.

Many people treat me and my fellow system administrators as an angry god when they want us to do some work for them. They offer a peace offering and apologize 5-10 times for interrupting us and then explain the problem hoping that we won't get angry. Or perhaps they see the same look that I used to get from my friend when I asked him to do work.

Being a system administrator for 11 years taught me a few things. I now understand the sysadmin attitude. It's not that we're assholes. It's not that we are lazy and don't like to work. It's not that we hate or think less of a certain type of employee. It's very simple.

The cardinal rule: Don't make us do any unnecessary work

That's it. Keep that in mind and you'll never feel weird when approaching your local sysadmin and asking him/her to do something for you.

We used to serve up the images for http://pronto.com from a couple of linux machines running reiserfs 3.6 with 6x 750GB SATA harddrives in them. We began to run out of room and decided to go to a SAN. We got a good deal on a SUN 7410 SAN through our corporate deals (22TB, 14TB usable) and it had all kinds of great stuff in it so we said what the heck.

Now, the problem. Our storage platform holds 366M 5-8 kilobyte files. This is a LOT of little files. At the top directory, we have a 256 directories (00-FF) and under each of those, we have another 256 directories, so we have 65,536 directories total. This means that we have about 5000-6000 images in each directory.

This is where we start to notice differences. Reiserfs does an "ls" in a directory with 6000 files in it in about 3-5 seconds. The SUN SAN does it in about 1-2 minutes. Serious problems here.

Our only option is to add a third level of hashed directories. This will give us about 16.7M directories and will solve our performance problem, but to all of you people out there that think that ZFS is hot shit, I say to you, "It depends on your situation".

I'm a linux guy. I've been using linux primarily since before the 1.0 kernel days back in 1993. I've never owned or used a macintosh for more than just helping someone get their mail or VPN up and running.

On friday, I installed Leopard (not Snow Leopard) on a MacBook Pro laptop for work which is about a year old and I decided that I was going to take it home and get to know what all of the fuss is about.

Friday night, first impressions: Nice hardware. Beautiful screen. Clean fonts. Nice, quiet keyboard. I liked the initial 'take a picture for your profile' which uses the screen as a flash - nice touch. I found more wireless access points from my home living room than I've ever found using any laptop that I've ever used before. Nice wireless/antenna. I changed the battery to the most conservative option and I got > 3 hours of battery life from it the first night. Nice. The magnetic power connector is also a nice touch. I had known about that before, but using it several times was nice. Pretty, well-engineered, productive and useful right out of the box. Good job Jobs! :)

Saturday night, I mainly tried out the software that came with the OS. I poked around with things and got to know how good safari was and ichat, mail, etc ... Some initial impressions: iTunes is *way* faster on a mac than on windows. Javascript on Safari is very zippy! Finding things on a mac is not easy for a Linux or Windows user. Finding installed applications could be made easier in my opinion. Time capsule is a great feature. Rock band is fun. I think that I would like to buy parallels or something so I could dual-boot with it though. I liked that the VOIP client just worked out of the box and was very easy to set up and configure.

I began to miss my > 1 mouse buttons. I tried holding the mouse button down, I tried double-clicking, I tried everything and had to figure out the mac-way of doing things without a second or third mouse button. I'm sure that I could hook up an external USB mouse, but I wanted to do things the 'mac' way, so I worked without it. Also, I'm used to tapping the touchpad for a left-click. I dont know how many times I tried that and waited for something to happen ... :)

Saturday night, I tried to download the xcode and iphone SDK (2 GB). It was a normal HTTP download from the Apple site but the transfer failed twice (not sure if it was the wireless connection, or the web server or what, so I don't really blame the mac or safari). I tried it with wget on linux and it downloaded great the first time, so I copied it to the mac over scp (which was standard on the mac) and I was good-to-go. I tried a torrent client - tomato torrent was the first hit on google for "torrent mac", so I downloaded it, and it pretty much sucks and never made one connection nor downloaded anything at all.

Sunday night, I installed the Xcode stuff and tried out some of my code that I am writing on linux. CVS worked great, but I couldn't compile something I was writing that needed PCRE (perl-compatible regular expressions) and I wasn't quite sure how to install it, so I went on to a different set of source code that I am working on. It's a multi-threaded application that uses pthreads. It compiled fine, but when ran, anything over 10 threads, it would hang. Using 10 threads, it ran fine. My perl scripts ran fine without having to install any CPAN modules at all.

Now, before all of you mac people start responding about how stupid I am or how easy it is to solve my problems. Let me explain. I didn't have days to figure things out. I just wanted a good taste of a mac and overall, I was very impressed. Some things very good. Some things not quite as good, some things a little weird and different. I'm sure that if I spent more than 60-ish hours (only about 5 actual hours) with a Mac and OS X, I'd probably figure things out and everything would be great. I would love to do some iphone development and would love to really learn to make this nice laptop really sing, but I have to return this machine to its rightful owner tomorrow morning and I'll be returning to my Ubuntu 9.04 box on my quad-core 4GB PC box. :)

I like linux and I like the mac, but being a little linux-biased, I think that more linux people should taste the macintosh experience a little bit and bring some of the mac flavor over to the linux side. Personally, I think that Windows is a lost cause and Mac and Linux are the way of the future. If anyone at Apple is reading this and wants to send me a free MacBook Pro, I'd love to use it as my primary machine every day, but if I were to spend my own hard earned money on a laptop today, I'd spend $500-$800 on an HP or Acer. I just can't justify $2k on a laptop no matter how nice it is. I don't think that I'd really go the hacintosh route mainly because it makes me feel dirty. :)

I never really figured out if there was a package manager or something else like apt or yum on linux. Is there?

Crawling the whole internet using a home internet connection and a few PCs.  Can it be done in a reasonable amount of time and without buying a million harddrives and computers?  How big is this problem?  Let's assume that we want to do the simplest type of analysis – just pulling all of the pages on the internet, grabbing the urls and then deleting the file and crawling more.  We only need temporary storage of the html, so we only have to store the urls.  Recently, Google celebrated the indexing of it trillionth web page (2007, I think).  So, let's say that there are 1T urls that we need to store.  Are those unique pages, or 1T pages, including dupes?  For each url, we need to store its return code (200, 404, …).  Normally, we'd store the date/time last fetched, Last-modified, mime-type, etc, but we're just going to store the url.  If we store it in a flat file, and we store urls in a per-domain file, we can cut off the domain part of the urls (aka http://mydomain.com/) and just store the path and filename in the files.  Making a wild guess at the average url length is across the entire internet, let's say it's somewhere in-between 40-100 characters.  We'll choose 80 chars as a stab in the dark.  1T urls * 80 chars = 80TB worth of urls.  If we store all of the URLs in a key/value database that supports compression (tokyo cabinet) and use the highest compression possible, we might be able to get the storage of all of the URLs down to below 10TB.  So, that's possible with a single PC for a couple of grand.  We now have a machine that's capable of storing every possible url on the internet (or close to it).

Downloading it all?  Can I actually download the entire internet on a home internet connection?  Assuming the same 1 trillion web pages, and the average web page size has increased to about 25k ( see: http://www.optimizationweek.com/reviews/average-web-page/ ).  Doing the math, that's 25Petabytes over a 6Mbit connection.  Wolfram Alpha says that that if I could max out my home internet connection, it would take 1057 years.  Turning on compression would help, but I doubt that many servers actually use compression, so let's say 1000 years, so that's not really do-able until we get Gigabit fiber to the door (only 6.5 years) or something much faster.

So, I guess that crawling the whole internet is not really possible at this point for me.  I can find a niche however and crawl a subset of the internet.  Perhaps I'll do that.  :)