The unintended consequences of large-scale storage

Jeff Atwood over at Coding Horror has posted about something called Rainbow Tables. Now, I don't want to turn this blog into a discussion of encryption so let me boil it down for you.

Windows passwords are stored in an encrypted format known as "hashes". When you enter your password, Windows encrypts it for you and compares it to the stored hashed version. If it matches, you're let in. If it doesn't you're not. There's no way to decrypt the hashed version of your password in any reasonable amount of time, if at all, and is therefore considered a secure method of storage.

The problem now is that you can get a database of pre-hashed content. Known as rainbow tables, these are basically a table with just two columns, first column, a word (or other combination of letters) and in the second, the matching hash. Now, if you have a hash, you can look it up in the table and see what the original password is. In other words, it's not decrypting the hash, its hashing all possible passwords in advance.

This is such a simple hack. So, why is it coming to light now. Well, the problem is large-scale portable storage. In the past, tables such as these were considered to big to bring to the computer you're trying to hack. But these days, a 1GB flash drive would allow you to carry a rainbow table that covered all conceivable passwords between one an 14 characters in length, containing just English letters. Here's Jeff's chart showing example storage requirements:

If you're suddenly not worried about a Rainbow Table measuring 64BG I've got a 500GB portable USB hard drive I'd like to show you.

Here's the bottom line, in Jeff's example, the password "Fgpyyih804423" (one that's probably a hell of a lot stronger than any password you use) was broken in just 160 seconds using a rainbow table.