Web developer. Lead developer of PieFed

  • 6 Posts
  • 190 Comments
Joined 3 years ago
cake
Cake day: January 4th, 2024

help-circle




















  • I used fail2ban as an anti-scraper strategy for a few months. It helped with the really basic scrapers that only use one or a few IP addresses.

    But then they adapted and started using hundreds or thousands of different IP addresses, each one only making a couple of requests. fail2ban became counter-productive as it slowed my server down - the log files it was monitoring became massive under high load and it needs to scan the log files often to check if an IP has been seen before.

    I’m not saying don’t try it, I’m saying it’s only effective in some situations.