I'm fascinated by this idea of a deep web only accessible by the cognoscenti. Presumably if a link slips out then the deep web becomes a lot shallower?
Well actually the relatively hard part is hosting a crawler of decent size, and then if you crawl in violation of robots.txt its pretty straight forward to use iptables to ban you, of course you then spend money on hiring a botnet to mask your traffic footprint, except that on that same darknet there might be people who are friends of the owner of that botnet.
Does this not rather beg the question - is there a deep web, and how big (or small) is it. I can easily understand the desire for a coherent group of people putting up vpns etc to keep their world seperate from others - but that implies you join in based on some other criteria, which sounds not very deep web but pretty secret-VPN-we-are-not-telling-you-about-unless-we-cross-your-AS-Number-when-something-is-obvious
It just has that feel of "secret society" to it, which tended just to reflect the informal power structures of the wider world anyway.
No, the "deep web" is real in the sense that there are billions of network addresses that contain content or services which are not accessible through the 'standard' discovery services (Google). In many ways things like Usenet are still part of it as there are netnews groups, and they get used, but there isn't a lot of indexing going on. Further there are at least two 'separated' NNTP type networks that are invitation only.
So it is a "collection" of secret societies, each with their own quirks. As a collection is constitutes a 'web' and perhaps the only commonality is the desire to not be part of the "public" web.
