Block AI Scrapers with Anubis
This tutorial will demonstrate how to set up the Anubis web firewall to block automated bots from accessing your website. Watch the video version here.
Prerequisites: You will need a server running Debian 13 (visit linode.nots.co, digitalocean.nots.co, or vultr.nots.co if you don't have one). You will also need a domain name to register an SSL/TLS certificate with Let's Encrypt (visit hover.nots.co if you don't have one).
Example parameters: This tutorial will use example1.nerdonthestreet.com for the Apache example and example2.nerdonthestreet.com for the NGINX example.
Go to Part 1 for Apache or Part 2 for NGINX.
Part 1: Using Anubis with Apache
If you already have a working Apache web server, you can skip to Step 4.
Step 1: Make sure your server is entirely up-to-date, then install Apache and Certbot:
Step 2: Enter the default Apache configuration file and set the ServerName directive:
Uncomment ServerName and set it to your domain name (e.g. example1.nerdonthestreet.com). Save the file, then reload the Apache configuration:
Step 3: Run Certbot to enable TLS:
Step 4: Download and install your preferred version from the Anubis release list on GitHub.
Step 5: Create global and Apache-specific Anubis configuration files by copying them from the defaults, then enable Anubis using that configuration:
cp /etc/anubis/default.env /etc/anubis/apache.env
cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/apache.botPolicies.yaml
systemctl enable --now anubis@apache.service
Step 6: Create a copy of your Apache configuration file:
cp /etc/apache2/sites-available/000-default-le-ssl.conf /etc/apache2/sites-available/000-default-le-ssl-anubis.conf
Step 7: Reconfigure Apache to send traffic through Anubis:
nano /etc/apache2/sites-enabled/000-default-le-ssl-anubis.conf
Add the following lines in the <VirtualHost *:443> section:
# Proxy to Anubis
RequestHeader set "X-Real-Ip" expr=%{REMOTE_ADDR}
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set "X-Http-Version" "%{SERVER_PROTOCOL}s"
ProxyPreserveHost On
ProxyRequests Off
ProxyVia Off
ProxyPass / http://[::1]:8923/
ProxyPassReverse / http://[::1]:8923/
Add the following new section at the bottom of the file:
<VirtualHost *:3000>
ServerAdmin email@example.com
ServerName example1.nerdonthestreet.com
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/error-proxy.log
CustomLog /var/log/apache2/access-proxy.log combined
RemoteIPHeader X-Real-IP
RemoteIPTrustedProxy 127.0.0.1/32
</VirtualHost>
Save and close the Apache configuration file.
Step 8: Enable the necessary Apache modules:
a2enmod headers proxy_http remoteip
Step 9: Define and enable a new listener for the port being used for the actual website:
echo "Listen 127.0.0.1:3000" > /etc/apache2/conf-available/anubis-listener.conf
a2enconf anubis-listener
Step 10: Disable the old Apache configuration file and enable the new one:
a2dissite 000-default-le-ssl.conf
a2ensite 000-default-le-ssl-anubis.conf
Step 11: Restart Apache to use the new configuration and modules:
systemctl restart apache2
To customize the Anubis graphics, continue to Part 3.
Part 2: Using Anubis with NGINX
If you already have a working NGINX web server, you can skip to Step 4.
Step 1: Make sure your server is entirely up-to-date, then install NGINX and Certbot:
Step 2: Enter the default NGINX configuration file and set the server_name directive:
Find server_name and set it to your domain name (e.g. example2.nerdonthestreet.com). Save the file, then reload the NGINX configuration:
Step 3: Run Certbot to enable TLS:
Step 4: Download and install your preferred version from the Anubis release list on GitHub.
Step 5: Create global and NGINX-specific Anubis configuration files by copying them from the defaults, then enable Anubis using that configuration:
cp /etc/anubis/default.env /etc/anubis/nginx.env
cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/nginx.botPolicies.yaml
systemctl enable --now anubis@nginx.service
Step 6: Create an NGINX configuration file containing the proxy configuration:
nano /etc/nginx/anubis-proxy.inc
Add the following lines:
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Http-Version $server_protocol;
proxy_pass http://anubis;
}
Step 7: Create a copy of your main NGINX configuration file:
cp /etc/nginx/sites-available/default /etc/nginx/sites-available/default-anubis
Step 8: Reconfigure NGINX to send traffic through Anubis:
nano /etc/nginx/sites-available/default-anubis
Replace the default "location /" block with the Anubis proxy configuration:
include "anubis-proxy.inc";
Add the following new server block at the bottom of the file:
server {
listen 3000;
server_name example2.nerdonthestreet.com;
port_in_redirect off;
root "/var/www/html";
index index.html index.htm index.nginx-debian.html;
}
Save and close the NGINX configuration file.
Step 9: Configure an NGINX upstream for Anubis:
/etc/nginx/conf.d/upstream-anubis.conf
Enter the following configuration:
upstream anubis {
server 127.0.0.1:8923;
# Optional: fall back to serving the websites directly.
#server 127.0.0.1:3000 backup;
}
Save and close the NGINX configuration file.
There are no pages beneath this page
Discussion (0 posts)