Our business operates in a market that has a lot of wannabe hacker types: bored teenagers in the gaming community with the minimum knowledge required to order a DDOS. As such, we're very very careful about never leaking our IPs anywhere. This has been tricky at times but it's well worth it. Cloudflare has made our lives a lot easier as it handles the bulk of the requests. Still, a lot of what it takes to be DDOS resilient is just being able to scale to high concurrency very quickly: ability to cache everything at every level, read-only mode, CDN for everything including HTML when possible, API gateways and self imposed rate limiting to third party services. With that said, there's not much you can do if your IP leaks. Some often overlooked areas for IP leaks: user generated content (your server makes a request to a URL specified by the user), or even sending an email (even many services provide the origin sender IP - such as SendGrid - as a matter of policy, so we don't use those services.)
Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)
That is indeed an interesting observation. I am not a network engineer so take all of the below with a handful of salt :)
Can't you drop all incoming traffic on "protected" IPs other than that coming through known-trusted networks like CDN's one? Your networking equipment still needs to be able to handle that load as well, which is not a given.
I guess DDoS attacker could fake the IP packet headers to appear as if they are coming from Cloudflare (in your case), though even then there'd be some obscurity in figuring what subnet you are allowing through.
I guess CDN providers could provide a secret-key-seeded time-based (like TOTP) source network, and your firewall would let packets in by that.
The benefit of this is that it is cheaper than doing full packet encryption, and probably much easier to implement in hardware/software.
>Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)
Search first. I would try something like,
detect irregular OR abnormal behavior internet OR requests OR databases
The plan I have is to use an isolation forest with dimensions like requests per minute, unique pages per session, etc. and just present a recaptcha or inaccurate data when they're outside some threshold.
I like the approach. It'll still return a 200, but just slow them down. In a botnet, that's a pretty interesting approach instead of returning a captcha so they'll keep using the IP.
This is very well written and practical. It's exactly how you think about security when you own a company. I'm guessing the author is/was holding a large amount of stock in Heroku/Salesforce.
Summary:
1. Don't publicly expose endpoints that are either slow or require a ton of resources.
2. If you can't get rid of a slow endpoint; put authentication in front of it so you have a lever to pull in the event of an attack.
3. Throttle / rate limit everything with high barriers so not to impede normal traffic patterns.
4. Don't make it easy for someone to DoS you; reduce or eliminate well-know attack vectors and vulnerabilities.
5. Scan your app for regex and zip bombs.
6. The bad guys will sniff-out your N+1 queries, so fix them.
7. If necessary: pay for DDoS mitigation from a cloud provider.
8. Set timeouts for the maximum time a client can take to send a request.
For Example, the attacker will send a GET request using many small packets with a large time gap between them. They do this by first sending the “G” in its own packet, after a while they will send the “E” and after some more time the “T” and so on. While this is a legitimate behavior according to the HTTP and TCP protocols, this behavior is consuming a lot of resources from the server – it has to keep the connection open, waiting for the full request to arrive. Since the connection pool is limited, it is very easy to reach pool saturation, with very little traffic.
In order to defend against slow-client (Slowloris) attacks, reasonable timeouts are necessary but not sufficient if your server has a limited pool of request handlers (or if it uses a separate resource-intensive process/thread per connection). You need a HTTP server (or proxy such as nginx) that buffers incoming requests using async IO syscalls (e.g., `epoll` or at least `select`) until a complete request is ready to be handled by the application.
Even with async IO buffering requests it's still possible for a slow-client DoS to exhaust other system resources (such as file descriptors), but with properly-tuned limits the required attack will need to be many orders of magnitude greater to succeed.
"There are ways to mitigate or reduce the impact of such an attack. In general, these involve increasing the maximum number of clients the server will allow, limiting the number of connections a single IP address is allowed to make, imposing restrictions on the minimum transfer speed a connection is allowed to have, and restricting the length of time a client is allowed to stay connected."
Yup all great points for mitigation strategies. Sometimes admins think DDoS = Traffic overload and forget to consider that slow connections can be just as effective as flooding if the configuration allows.
Some of the best DDOS attacks work by sending data as character by character as slow as possible before the connection times out so as to mimic extremely slow connections and consume handler threads. Also HTTP1.1 downgrade attacks can force the usage of more connections especially if keep-alive is false.
Still to do: developing our own online ML based monitor which can detect abnormal user behaviour before it becomes a problem (ie. aggressive HTML scrapers, burp suite scans, etc.)