“I’m not a robot.” This assertion to the familiar Captcha question is increasingly not a simple click as bot traffic is reported to have surpassed human web surfers. But parsing out the intentions of bots can be tricky.
A new report from edge cloud platform Fastly traces the new landscape of bots online based on research conducted in Q2 of this year.
- Most bots (87%) are considered malicious; they pose threats like account takeover attacks and ad fraud.
- A small but growing slice, however, now scrapes data for AI training, crawls sites for AI search indexing, and pulls information in real-time for user queries to AI chatbots.
This rise in bot traffic can strain web infrastructure and create new challenges for businesses concerned with protecting or promoting web content and marketing materials.
“The challenge for website owners often is in letting the good bots in while keeping the bad ones out,” the authors wrote.
Making fetch happen: The new class of AI bots fall into two categories, according to Fastly: crawlers and fetchers. Crawlers, which constitute around 80% of this segment of AI bots, are primarily concerned with amassing training data and indexing. Fetchers—only 20% of AI bot traffic—fetch content from the web in response to user queries.
While these fetchers currently make up a smaller portion of overall bot traffic, their proportion could grow as more users turn to chatbots for search and a growing number of AI agents rely on this type of information retrieval, according to Arun Kumar, a senior security researcher at Fastly.
“Imagining a world where everyone tries to access content, discover content, through ChatGPT, or similar applications…as that grows, we would definitely see way more fetchers, and I would expect that proportion between fetchers and crawlers might probably flip,” Kumar told Tech Brew.
Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
Whose bots? More often than not, it is indeed ChatGPT that is doing the fetching—it and another OpenAI bot accounted for 98% of fetcher bot traffic analyzed by the authors, while AI search engine Perplexity made up about 1.5%.
Meanwhile, Meta made up the bulk of crawler traffic at around 52%, followed by Google (23%) and OpenAI (20%). Kumar told us Meta’s inordinate share of crawling activity could be a fluke of the three-month time period in which the analysis took place; it’s possible that Meta was simply doing more crawling during that time.
Commerce, media and entertainment, and tech websites see the highest levels of AI scraping for training, according to the report, while education and media are the two categories of website where the fetchers outnumber the crawlers.
Traffic jam: The increased volume in traffic from fetcher bots can overwhelm websites with the amount of activity they direct their way, “even without malicious intent.”
- Fastly reported one instance in which a fetcher bot made 39,000 requests per minute to a single website, which may cause effects similar to a distributed denial of service (DDoS) attack.
- AI crawler bots can reach 1,000 requests per minute.
In order for this relationship between website owners and AI companies to be sustainable in the long run, Kumar said, AI companies must create bots that don’t stress infrastructure unduly and work out revenue deals for training.
“[Companies should] have some kind of a rate limit depending on what the sites could take on,” Kumar said. “And also have these revenue models which content owners can use to regain some of the lost revenue. But then there are also so many open questions about pricing and all of that, which we are keeping an eye on.”