We used to serve up the images for http://pronto.com from a couple of linux machines running reiserfs 3.6 with 6x 750GB SATA harddrives in them. We began to run out of room and decided to go to a SAN. We got a good deal on a SUN 7410 SAN through our corporate deals (22TB, 14TB usable) and it had all kinds of great stuff in it so we said what the heck.

Now, the problem. Our storage platform holds 366M 5-8 kilobyte files. This is a LOT of little files. At the top directory, we have a 256 directories (00-FF) and under each of those, we have another 256 directories, so we have 65,536 directories total. This means that we have about 5000-6000 images in each directory.

This is where we start to notice differences. Reiserfs does an "ls" in a directory with 6000 files in it in about 3-5 seconds. The SUN SAN does it in about 1-2 minutes. Serious problems here.

Our only option is to add a third level of hashed directories. This will give us about 16.7M directories and will solve our performance problem, but to all of you people out there that think that ZFS is hot shit, I say to you, "It depends on your situation".