Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You're right, that's why I'm questioning the reason Anubis implemented it this way. Lots of big AI companies are at least honest about their crawlers and have proper user agents (which Anubis outright blocks). So "unethical" companies who change the user-agent to something normal will have an advantage with the way Anubis is currently set up by default.

I'm aware that end users can modify the rules, but in reality most will just use the defaults.



Shitty heuristics buy time to gather data and make better heuristics.


Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will have an advantage, but I don't give swindlers a pass just because they rob me to my face. I'm okay with defaults that punish all bots.


You can have a bot allow list. I think it's also being planned as a subscription service (not sure about this part).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: