A while ago I published a post showing how to start a script on boot using systemd. In the example I used a simple bash script to make a curl request against my website. I believe someone tried the exact same example, activated it and forgot all about it.
So now I have a steady stream of curl requests coming in to my server. Even though these requests are harmless and pose no threat, I am going to show you how you can block these requests if you use Cloudflare like I do.
I am hoping that at some point of time the person who is running the curl realizes that his/her curl is failing and stops it. If not, then my Cloudflare rule will continue to block it.
The method demonstrated here can be used to block any user agent that you see in your log file. You could block a bad user agent or a robot that is causing issues for your site.
Note: I am not promoting Cloudflare, neither is this a sponsored post from Cloudflare. I just happen to use Cloudflare with my WordPress site.
Login to your Cloudflare account and navigate to the ‘Firewall’ page and then select the ‘Tools’ option as shown in the image below.
Scroll down to ‘User Agent Block’ and then click on ‘Create Blocking Rule’ as shown below.
In the next screen as show below, enter a ‘Name Description’ for your rule, set ‘Action’ to Block and the exact user agent that you wish to block. After this is done click on ‘Save and Deploy’. This will activate your Cloudflare User Agent Blocking Rule.
There are few other options available in addition to block, do take a look at them as you might want a different kind of action for the user agent.
To check if this is really working, you can go to your web server log and tail the access log and see if the curl requests are coming or not. You can also verify this on the Cloudflare site.
To confirm this on the Cloudflare site, navigate to ‘Overview’ under the ‘Firewall’ page as shown below and set a filter if needed. Click on the block event to see its details.
The above shows you the ip address, the user agent, the request and the action taken, which in this case is set to ‘Block’.
My plan allows me to add 10 User Agent Blocking rules and keep them active. Please do check your account and plan before you activate any rules.
In addition to this I have some other firewall rules that I use to protect my WordPress site. To see how you can protect your login and admin see this post.
This post shows that it is fairly easy for you to block a User Agent from accessing your website. Having said that, please be careful what you block. If you use a common user agent you might end up blocking more traffic then you intended.
I’m searching for a list of bad user agents to block.
I already blocked ApacheBench/2.3 which was responsible for a couple of worryingly powerful single ip address DOS attacks.
Anyone have a list of bad user agents to share?
I would imagine the list would be very big. Also a bad bot can identify itself as a valid user agent. You would be better off using a different method to block, say based on number of requests in a short period of time.