There's a lot of heuristics. Big sites get more crawl time. Some crawlers will back off if pages are slow. There's usually some sort of 'interestingness' calculation, so repetitive content won't get crawled as much.