> so you ended up seeing a lot more interesting smaller sites?

A lot of smaller sites just aren't indexed any more, it's a common complaint you read here...

"Smaller site, we don't use adwords, we're getting crawled 500x/day which is 40% of our bandwidth, still not indexed after 6 months"