We're actually getting rid of export credits because it's caused confusion over the years. We now just have query credits to download data/ do searches, and scan credits for users that want to request on-demand scans. We announced this change in the most recent Shodan Update newsletter. You can already use our new website (https://beta.shodan.io) to download data using your query credits.
Export credits were the first way I tried to monetize Shodan and it became a legacy system that lots of companies used so I was hesitant to get rid of it until something better was in place.
I'll also add that the API was purposely not designed for downloading lots of search results. The API is designed for security operations center (SOC) use cases. Companies that need large-scale, bulk access to our data would need to check out our enterprise platform (https://enterprise.shodan.io).
This is what I've assumed, but it's in a pretty uncomfortable place right now as e.g. the documentation often refers to export credits with a broken link.
The API is somewhat unsuitable for exporting large volumes because it seems remarkably unstable as to ordering, it suggests that you can do paginated requests but the second page tends to have 30% overlap with the first page.
I 100% understand the product motive to move large exports to an "Enterprise" feature, but it's rather disappointing because as a small-scale independent operation I don't expect to be able to afford it, and that would go for a lot of productive people in security research. But then, that's capitalism.
I decided that a broken link is better than having people spend money on something that will be deprecated. We're obviously working on cleaning up those broken links but it's an easy thing to explain if anybody emails support@shodan.io
The ordering is based on timestamp and it can happen that new results were indexed in between your 1st request and 2nd request which creates an overlapping result. A 30% overlap is unusual and sounds like it's for a query with many results.
Finally, most researchers don't actually need to download data. They could just use our free API and facet queries to get the information without downloading the actual data. This entire website is powered by a free API key that uses facets:
I think a lot of researchers go into the default mode of "I want to have the data" but using facets is way easier, faster and doesn't cost any money at all. And you can navigate the available facets using our new beta website (another area we're trying to make things a bit clearer). For example:
Note that we provide free upgrades to universities/ students/ professors as well as routinely work together with researchers so we're not trying to push them into the enterprise product. We also let universities monitor up to ~120k IPs for free using Shodan Monitor (https://monitor.shodan.io). But the use cases for researchers are few and we figure that if you need lots of data then you can send us an email.
Export credits were the first way I tried to monetize Shodan and it became a legacy system that lots of companies used so I was hesitant to get rid of it until something better was in place.
I'll also add that the API was purposely not designed for downloading lots of search results. The API is designed for security operations center (SOC) use cases. Companies that need large-scale, bulk access to our data would need to check out our enterprise platform (https://enterprise.shodan.io).