Document various DDoS attacks #141
Comments
|
This took quite some time on my 4 GB access log file, but: I only went to the first bunch that went to 10,000. Would be interesting to do a little IP research and see what all these IPs are doing. The first one looks to be a German DigitalOcean IP address. |
|
I used the 'report abuse' feature on DigitalOcean to report |
|
The pattern here suggests DDoS as a service. Such services leverage compromised hosts from their botnet networks to DDoS victims and each infected host is limited to around 1-5Mbps to avoid detection by their owners/users. So that's 1-5Mbps per host times a few million infected hosts. So "report abuse" for in these cases, will probably not solve anything. DDoS as a service is growing exponentially every year. |
|
I had a similar issue with my wife's wordpress some weeks ago. I first |
True, but that one IP was orders of magnitude beyond the rest. Hopefully it will at least prompt DigitalOcean to clean up that server. |
|
A few more quick stats since things are calming down (at least now that Cloudflare's handling the brunt of it:
And looking at my logs from the time before Cloudflare started working... I'm seeing |
|
Can you post your log for the attack? I don't think we need all of it, but I would like to see if there is some sort of fail2ban and/or rate limiting rules that can be done. I mostly just need to see the IP address, time, and request method. |
|
I would also look at Cactus Comments. It is a decentralized comment system based off of Matrix. That will allow you to have comments that don't break through the cache That should prevent some DDOS attacks on your site in the future as well. I don't know how easy it is to integrate into Drupal, and I haven't found a plugin for it, but you know more about how Drupal works then I do. |
|
@minecraftchest1 - Eventually if I find a good comment system that integrates with spam prevention services that I can integrate with a static site and don't feel like it will turn out to be another Discord, I'd love to go that route and convert from Drupal to Hugo or some other static generator. |
|
Well someone is at it again. Approximately 5 minutes ago I got an outage alert, and saw the exact same patterns. Hundreds of thousands of requests like: At least 150 unique IP addresses and here's a list of the top ones, sampling just a second or two of accesses: |
|
Cloudflare IPs are popping into the list, since I switched it on again. Wish there were a simple plugin or something for Nginx to just pop on Cloudflare mode and only allow access from those IPs (without me having to pull down the list from Cloudflare itself). |
|
For now, since I'm not on the site and I'm okay with basically disabling comments overnight, I've re-enabled my Cloudflare firewall rule to block all POST requests. The attack is not sophisticated... I might just add a rule in Nginx to block POST requests to anything that's not a valid comment, and see if that holds up (since I don't have the ability to get that deep on Cloudflare). |
|
Seems like the exact same attack that plays out over on Nixcraft: https://www.cyberciti.biz/faq/nginx-block-post-requests-urls-for-spammer-ip-address-cidr/ Here's a fancy bandwidth graph on my server:
|
|
This might be of some help. |
I don't understand your comment about another Discord. Since you are using GitHub, you can also look at https://utteranc.es/ as well. Just some options that may help lower attack surfaces. |
|
@minecraftchest1 - I meant Disqus :D — basically, anything that I don't self-host will eventually have an incentive to jam ads and tracking into their product (even if they're a paid service, most likely). |
That makes more sense. I created #143 to discuss this more as it is off topic for this thread. |
|
Added a note that Edit: Someone else also noted this in newsboat, so it's not an isolated incident:
|
|
Opened new issue for the duplicate RSS feed items: #145 |
|
Another attack after I posted the video How I survived a DDoS attack. Nginx request log:
Set Cloudflare to 'under attack' mode for now since I have other things to do this morning :P Obviously most of the requests are coming through CF—I need to modify my command (will do later) to grab the real IP address. |
|
This time the script is hitting /user/login, which I set to bypass (will have to switch that off, lol): |
|
Decided to just block the |
|
One of the top IPs seems to belong to an elementary school... https://ipinfo.io/165.139.238.20
Probably a hacked teacher's computer. Hundreds of thousands of requests blocked now:
|
|
Requests are back down to a more normal rate on the backend:
|
|
Up to 1.3 million now.
Edit: 1.8 million after another minute. lol Edit 2: 2.2 million after another minute |
|
(And to give an update on why I didn't fix that sooner—I was out getting lab work done for Crohn's medicine. I have plenty of other stuff in my life that are higher priority than my website 😉) |
|
I have diabetes myself, so I fully understand |
|
@PH4NTOMiki - I might just add a couple more rules in soon and block off anything besides comment forms that isn't a public-facing page. |
I hate when I have to wait for some stuff just to be able to get insulin and insulin pump supplies for myself, so no worries. |
|
All right, here it is sorted by true IPs since I've had a little time without needing to head off to an appointment: That's just IPs that sent more than 10,000 requests in the past couple hours. |
|
Up to 7,000,000 blocked today on Cloudflare (and a few million more that went through):
|
|
Just a few more graphs from while I was getting my labs drawn at the hospital:
|
|
lol, back at it again: |
|
I guess your only option is to switch to some static site generator (I suggest Hexo) when you have time |
|
I can help with the migration if you want |
|
https://www.drupal.org/project/static_generator appears to be able to generate a static version of a drupal site that could be deployed to cloudflare/github/gitlab pages, or served via ipfs using an ipfs gateway.
|
|
If I stuck on Drupal, I'd probably use Tome (https://www.drupal.org/project/tome). But for other projects I either use Jekyll or Hugo :) |
|
So the attacker finally became wise enough to start posting comments, and in a stroke of actual intelligence, decided to use the edit domain. I left that route open to see whether the attacker would eventually make it through or not, and it looks like he or she did. So this morning I:
It seems like some requests are still squeaking through DO's firewall rule, though—for example: Maybe it's just queued requests. |
|
Will have to clean up these comments later:
Comment reply blocking rule was working for a while:
But now DigitalOcean seems to be tiring of handling this idiot's barrage:
I may move to another server, and drop all DNS routes to the server itself. |
geerlingguy
changed the title
|
At least they recommend a service you use.
|
|
I ended up basically dropping the IP associated with the backend services and am on a new IP at this point, with full proxy coverage on Cloudflare so the attacker can't get around it. I also set Cloudflare's security level 'high' for now (I'll back that off once things stabilize), but in some browsers I'm getting a 'Too many redirects' warning. Not exactly sure what's going on there, but I just rebooted the web server. I'm wondering if by changing IP addresses, Nginx may have somehow gotten confused about what IP it was serving from, and tried redirecting through the old IP? Not sure, we'll see. |
|
That might've been it. Before the reboot, requests were coming through like: And after the reboot: Probably could've just restarted Nginx itself. |
|
Also heard from someone running some security honeypots that there were a few other amplification attacks that were attempted. Maybe Putin has nothing better to do since his army is kinda stalled out, and he's assigned someone to try taking down my website since I mentioned something about supporting Ukraine 🤣. |
|
You aren't the only one under attack. I have been seeing some failed logins into my VPS. |
|
Yeah, I see them frequently too. But I use WP and change url so that
mitigates most of entries.
…On Thu, 17 Mar 2022 at 20:43, Minecraftchest1 ***@***.***> wrote:
You aren't the only one under attack. I have been seeing some failed
logins into my VPS.
—
Reply to this email directly, view it on GitHub
<#141 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AIN22MPBFEA3ZINOSTVZGMTVAODNXANCNFSM5N6KB6RA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@PH4NTOMiki Most of what I have been seeing is ssh login attempts. For some reason, Debian 11 allows root login via ssh keys. I turned that off just now. I need to see what else I need to monitor with fail2ban. |
|
@minecraftchest1 - Almost every minute, I see dozens of script kiddie attacks against any kind of URL under the sun that could be left open—at this point it looks like Cloudflare is eating up most of those, but it's basically a constant barrage against any device that has ports 80 or 443 open. And alternately, if you have any port that's running SSH (especially if you leave 22 open), you'll get periods with hundreds or thousands of login attempts no matter how secure your server is. I use fail2ban to rate limit that... but it's just a fact of life. It'd be like if you had a house in a neighborhood, and every passerby went around and tried every door handle and window latch. That's like running any service accessible over the public Internet. |
|
@geerlingguy For now, I'll be watching postfix, dovecot, apache2, nginx, and matrix synapse to make sure everything goes okay. To bad journalctl doesn't have an option to filter out services. I see a lot of spam from docker and systemd that makes it harder to find important stuff. I've got my web servers behind cloudflare, dovecot and postfix I am still working on getting configured correctly, I need to check my WordPress log, and see if Synapse is giving any intersting log outputs. sshd is being monitored with fail2ban. What would recomend for monitoring? I am thinking about setting up Grafana and Prometheus, but I wonder what other free options there are? |
|
I need to figure out how this is possible |
|
Reading this is depressing. Modern Internet is a cesspool, and the vandals realistically don’t face any consequences. I hope you won’t give up sharing your useful content because of those twerps. Anyway, I’ve gotten enamored by Crowdsec.net (think of it as a collaborative fail2ban for pretty much any service). It would be interesting what percentage Crowdsec can take off the top. |
|
At it again:
|
|
any updates? |
|
Not really, just every now and then there's another attempt. I guess someone just has it on their schedule to DDoS the site now and then. Hasn't affected anything for over a year. |
So, shortly after my post Hosting this website on a farm - or anywhere went live, the site was pummeled by an average of 5,000,000 POST requests per hour (in addition to a ton of other traffic, I don't even have the full metrics because I turned some of that off once the server load was hitting 25-30).
There was no way the poor Turing Pi 2 cluster could handle that load—and not even my 2GB DigitalOcean VPS could.
So I had to turn to Cloudflare. I went 13 years without needing a hosted CDN/cache layer in front of my site, and went through dozens of HN front page sessions without an issue.
But when someone points a DDoS cannon at the site (I could confirm at least 1,000 unique IPs were sending dozens of requests per second), requiring Cloudflare is inevitable.
A few things I need to clean up after all this insanity:
curl https://www.jeffgeerling.com/blog.xml, so I'm not sure what's happening for these users. (see RSS feed showing duplicates in some feed readers #145)The text was updated successfully, but these errors were encountered: