Apparently the AI crawler bots that are currently busy vacuuming up as much data from the web as possible have become fancy enough to run even complex Single-Page Applications like SOTLAS, including loading maps etc. And identifying with a proper User-Agent string seems to be a thing of the past too.
As a result, map hosting costs (via MapTiler) have exploded. In an effort to put an end to this, I have had to implement an invisible “CAPTCHA” using Cloudflare Turnstile that the browser must complete before being able to load the map. We’ll see if that is enough to keep the bots out.
Users who are logged in with their SOTA account are exempt from the CAPTCHA verification, so if you see a message saying that anti-bot verification failed, simply log in and the map should display as usual.
It is unfortunate that you are forced to do this but all I can say is THANK YOU for the wonderful tool that you provide to the SOTA community. It is truly excellent, and when on my computer at home I always have SOTLAS open on my browser!
I know a lot of us would find it very much more inconvenient without this site. Is there any way we can contribute towards the costs? I know some people are not in a position to help financially, I’m not suggesting anyone is obligated to pay of course.
SOTLAS is one of the most useful SOTA sites on the web. I’ve noticed recently that sometimes it can take a while to load - could this be down to crawlers placing additional demand on the systems I wonder? Anyway, it’s a wonderful resource and very much appreciated.
Thanks! SOTLAS is hosted on a global CDN with almost unlimited capacity, and the maps are on a (separate) CDN as well. So it’s unlikely that anything is slowed down due to the number of requests.
Delays during initial map loading when not logged in could be caused by the new CAPTCHA validation. And anything involving SOTA database requests (filtering the map for activated summits by callsign, loading activator detail pages, activation logs etc.) is subject to response times from the database.
Manuel, thanks for keeping all the bots at bay! I’m sure it’s a never-ending challenge. I was just reading about some of the new Cloudflare opt-in options that will allow you to filter AI-bot traffic, or optionally opt in to allow specific crawlers the option to pay for content. I haven’t leveraged this, but it sounds like this is trending toward rewarding AI bots that have more ethical practices and denying access for those that don’t. All while offering monetization for content. I did see that Cloudflare is offering a 2-hour informational/educational session for this new feature. I’m not sure if any of that is viable for the SOTLAS implementation, but I thought I would mention it in case there are some helpful takeaways.
As a regular user who leverages the SOTLAS site for activation planning and ongoing exploration, I would like to say thanks for the behind-the-scenes work that goes into keeping this site running optimally for the community.
Thanks for the pointer, Robert! SOTLAS is hosted on Cloudflare, but indirectly through DigitalOcean’s app platform, so I don’t have access to the bot blocking settings. But that doesn’t matter, as the invisible CAPTCHA solution to keep the bots from loading the map and causing costs is working very well
Luckily it appears that the anti-bot measures have worked.
