Crowdsec and Dealing with domains behind Cloudflare

How we accidentally banned all traffic from Cloudflare

(and how we fixed it)

We recently received a support request from a hosting client that a particular site was not working properly. Upon further investigation...

Turns out there is an official postoverflow for Crowdsec to handle this very situation, which basically keeps an updated list of Cloudflare IPs on a permanent allowlist so they will never be banned. Had we installed this initially, we would have avoided this whole ordeal. However, we did not, and installing this does not do anything about IPs that have already been banned, so we had to come up with a way to manually unban all Cloudflare IPs that have been wrongfully banned.

However, this is not exactly a straightforward process. Cloudflare does keep a current list of CIDR ranges that are used for their services, which is handy. But unfortunately, Crowdsec does not have a built-in way to unban a CIDR range, which means we were going to need to expand the CIDR ranges into the hundreds of thousands of individual IP addresses they represent,and unban each one explicitly. Yikes! 😬

We run a Crowdsec cluster with many nodes, and have amassed a very large database of banned IPs, which makes even unbanning a single IP make the CPU sweat a little - it takes several seconds to complete even one unban requeston our modest hardware. This was clearly not going to scale to hundreds of thousands of IPs! So we had to get creative.

Although Crowdsec uses Postgres to store bans and the metadata that goes along with them, it also uses ipset to manage the raw list of banned IPs in memory, presumably as a more direct way of interfacing with the OS firewall, which can also talk to ipset.

Running ipset list will dump a complete list of all indivudual IPs currently banned by Crowdsec to stdout, one per line.

So first, I redirected this to a text file:

ipset list > bad-ips.txt

Even though this file is quite large, it's extremely fast to check IPs against using grep.

So now all we needed to do was:

• Expand the CIDR ranges into individual IP addresses using prips
• Check each individual IP against this textfile
• If a match is found, call cscli decisions delete on the IP to properly unban it

I came up with this script to do exactly that. It took a few hours to run, but in the end it found thousands of Cloudflare IPs that were banned by mistake, and unbanned them. It worked perfectly😎

unban-cloudflare-ips.sh

#!/bin/bash
TMPFILE="/tmp/bad-ips.txt"

curl https://www.cloudflare.com/ips-v4/ > cloudflare-ips.txt
CIDR_LIST="cloudflare-ips.txt"

# Check the file exists
if [[ ! -f "$CIDR_LIST" ]]; then
    echo "CIDR list file not found: $CIDR_LIST"
    exit 1
fi

# Dump all IPs currently blocked by Crowdsec into a text file
ipset list > $TMPFILE

# Loop through each CIDR
while IFS= read -r CIDR; do
    [[ -z "$CIDR" ]] && continue
    echo "Processing CIDR: $CIDR"

    # Expand CIDR to IPs using prips
    for IPADDRESS in $(prips "$CIDR"); do
        # Check if this IP is in CrowdSec decisions
        if grep  -F "$IPADDRESS" $TMPFILE; then
        IPADDRESS="$IPADDRESS" unban-ip.sh
        fi
    done

done < "$CIDR_LIST"

Note that in order for this to work, you need to have prips installed on your system. On Debian based systems, you can install it with a simple sudo apt-get install prips. However it does not appear to come packaged for RHEL-based distros. If this is you, you'll need to compile it from source, which you can get from the official GitLab page or their website.