+- MFGG Forums (https://forums.mfgg.net)
+-- Forum: MFGG (https://forums.mfgg.net/forumdisplay.php?fid=4)
+--- Forum: Suggestions (https://forums.mfgg.net/forumdisplay.php?fid=11)
+--- Thread: OpenAI is crawling the MFGG forums (/showthread.php?tid=3290)
OpenAI is crawling the MFGG forums - Q-Nova - 02-23-2025
I noticed that according to the "Users online" box, OpenAI is crawling the MFGG forums. I know a while ago ChatGPT got blocked from this place, so perhaps we should do the same to OpenAI? I'm sure a lot of people here wouldn't want AI to know the answer to question #17 of Super Trivia Bros...
RE: OpenAI is crawling the MFGG forums - AGuyCalledKlaz - 02-23-2025
You've got to be freakin' kidding me. Even here? This is starting to feel shameless on the part of... Whoever is even doing this.
RE: OpenAI is crawling the MFGG forums - VinnyVideo - 02-24-2025
Yeah, these LLM bots can be pretty thirsty for bandwidth. I don't think OpenAI's crawler is adding much value for us, so let's see if a robots.txt update will be enough to get it to pack its bags.
RE: OpenAI is crawling the MFGG forums - Q-Nova - 03-05-2025
I just noticed that ChatGPT (along with OpenAI) is also crawling the forums. I thought that bot got blocked off? The main site has been getting slow sometimes, and I suspect these bots are the cause of it.
RE: OpenAI is crawling the MFGG forums - DJ Coco - 03-05-2025
On the bright (or not so bright) side, Grok already knows all about me.
RE: OpenAI is crawling the MFGG forums - VinnyVideo - 03-08-2025
Actually, there's now at least three different bots related to ChatGPT. Several other companies have their own bots for training or operating LLMs. Many of these LLM bots are extremely resource-intensive, and stopping/throttling them is a constant game of Whack-a-Bot.
RE: OpenAI is crawling the MFGG forums - Vert - 03-10-2025
Maybe someone ought to just put in some bot protection. Something simple like detecting if the limit rate is above what's humanly possible?
RE: OpenAI is crawling the MFGG forums - VinnyVideo - 03-10-2025
@Vert I've tried to use robots.txt to throttle specific problematic user agents, though not all bots respect robots.txt. Maybe I could beg R II to spring for Cloudflare...
RE: OpenAI is crawling the MFGG forums - Q-Nova - 04-17-2025
I highly agree with @Vert! I've been meaning to say this for a while, but the site has been about unusable for the last few days. I feel lucky that I was able to make this post, considering how slow it's been taking for the pages to load. Though, I'm not exactly sure about Cloudfare. I've heard they track people and can have issues with browsers that aren't Chrome. On the other hand, you can set up a firewall and that's really expensive from what I hear. I guess whatever you use, you'll have to decide on it soon rather than later, otherwise we might lose a lot of users.