It looks like the main culprit is a webcrawler called Claudebot. Here is a description of what it does: "ClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude." Many website operators are complaining that ClaudeBot is using up all its resources by flooding the website with too many requests, several every second. There are methods for denying access to certain users, but they have limited effect on ClaudeBot. I will keep trying until I am successful in blocking this dangerous bot.
|