Disclosure
“What follows is what one month of AI bots scraping data actually looks like on a small independent site.”
01 · The Traffic Divergence Problem
Something has been quietly breaking across independent publishing for the past eighteen months. Traffic from Google Search is down. Time-on-site is holding. Direct visits are stable. But referral traffic — the kind that comes from someone clicking a link in a search result — has been eroding in ways that don’t show up cleanly in Google Analytics.
The instinct is to blame algorithm updates. Sometimes that’s right. But there’s a second explanation that most site owners haven’t been able to measure until now: AI systems are reading your content, extracting value from it, and returning answers to users who never visit your site at all. No click. No referrer. No record.
This is what happens when you actually start measuring that.
02 · The Signal That Started It: Bing 4,415 Impressions, 137 Clicks
TechReviewsUK has been building Bing Search Console data over recent months. The headline number is a 3% click-through rate — 4,415 impressions generating 137 clicks. That figure is low by any benchmark. Bing users are seeing TechReviewsUK content surface in results, reading the preview, and not clicking through.
One explanation is that Bing’s AI-assisted results are summarising the content before the user ever reaches the link. The answer is surfaced; the visit is not. This is not a ranking problem. The impressions are there. It is a value-extraction problem: the content is being used to answer the query without crediting the source.
That question — who is accessing the content, and what happens to it afterward — is what prompted the monitoring setup described below.
03 · The Bot Data: 1,468 Requests, Four Identified Crawlers
Unsourced monitors incoming bot traffic by matching user-agent strings and verified IP ranges against each AI company’s published crawler lists, with reverse DNS confirmation. Over the monitored period, TechReviewsUK logged 1,468 total AI bot requests across four identified crawlers.
That is not background noise. That is systematic, repeated access from machines that exist specifically to ingest content for AI training or real-time retrieval.
04 · Who Was Doing It?
The breakdown by crawler reveals a significant concentration. OpenAI‘s infrastructure accounts for the overwhelming majority of traffic, with two separate agents operating independently.
- ChatGPT-User — 1,160 requests, 629 unique IPs. Real-time retrieval agent — fires when a ChatGPT user asks a question that triggers a web fetch. High volume, high frequency. Each request represents a live user query that returned an answer using TechReviewsUK content, without a click to the site and without a citation in the response.
- GPTBot — 305 requests. OpenAI’s training crawler. Systematic page-level access for model training data.
- ClaudeBot — 14 requests. Anthropic’s crawler. Present but at low volume in this period.
- PerplexityBot — 4 requests. Perplexity AI’s indexing crawler. Low volume, consistent presence.
Following identification of these bot signatures in the Unsourced dashboard, the site owner implemented WAF blocks for the relevant crawlers. Unsourced surfaces the data; enforcement decisions remain with the publisher.
05 · The Competitor Intelligence Layer: Who AI Recommends Instead
Bot detection answers one question: who is accessing the content. The citation monitoring layer answers a different and arguably more commercially important question: when AI systems respond to queries that TechReviewsUK’s content should answer, which domains are they recommending instead?
The answer is direct. Eight competitor domains are consistently surfaced by AI models in response to queries seeded from TechReviewsUK’s own content. TechReviewsUK itself is not among them.
- TechRadar — 6 citations
- Tom’s Guide — 5 citations
- NotebookCheck — 4 citations
- Android Central — 4 citations
- Eufy — 3 citations
- DJI — 3 citations
- HowToGeek — 3 citations
- RTings — 2 citations
The content was trained on. The citation went elsewhere. This is the gap that standard analytics cannot see.
This is the most actionable data in the dashboard. TechReviewsUK is a growing independent site — it is not yet in the citation layer of these models. But knowing precisely which domains are occupying that space, and on which topics, turns a vague competitive disadvantage into a specific, addressable problem. It tells you where to focus content, where to build authority, and which domains are benefiting from training data that originated here.
06 · The Evidence: Reports Generated
For each monitoring period, Unsourced generates two report types — an Evidence Report (structured evidence for licensing or legal discussion) and an AI Visibility Audit (competitive strategy and content gap analysis). Both are exportable as PDFs.
“Screenshots captured across multiple sessions — live data updates continuously so figures may vary between images.”
07 · The Watcher: Origin Signal
The final layer in the monitoring setup is Origin Signal — a hidden proprietary marker embedded invisibly in published content. It is not readable by human visitors. It is addressable by crawlers. If any AI system reproduces a passage of TechReviewsUK content — in a training output, a model response, or a generated article — the Signal is designed to surface in that output and trigger a logged detection event.
It has not fired yet. That is not a failure state. It is a baseline. The Signal is now embedded and the watcher is running. Every week that passes without a detection is recorded data. Every future detection becomes a timestamped, verifiable event — not an inference, not a probability estimate.
The value of Origin Signal is not in the moment it fires. It is in the chain of custody it creates: crawl event, content ingestion, model reproduction, detection — each stage documented. If and when TechReviewsUK content appears in an AI output without attribution, there will be a record that holds up to scrutiny.
08 · What This Means for Publishers
The numbers above are specific to TechReviewsUK. The pattern they describe is not. Every independent publisher operating in a competitive vertical is dealing with some version of this: AI systems trained on original content, citing aggregators and large platforms in their responses, while smaller original sources remain invisible in the output layer.
The problem is not that AI is reading independent content. It will do that regardless. The problem is that until recently, publishers had no way to see it happening, no way to measure the gap between ingestion and citation, and no structured evidence if they ever wanted to act on it.
The data from TechReviewsUK does not require alarm. It requires visibility. 1,468 requests from known AI infrastructure is not a crisis — it is a measurement. Zero citations is not a verdict — it is a starting point. Eight competitor domains being recommended instead is not a catastrophe — it is a brief.
What you do not measure, you cannot change. That is true in SEO, in paid media, and now — demonstrably — in AI.
Stop Guessing. Start Measuring.
Unsourced monitors AI bot scraping, tracks citation gaps, and generates structured Evidence Reports and AI Visibility Audit reports for independent publishers. 14-day free trial, free WordPress Plugin available for WordPress hosted sites. No card required.
Case study by René R — Editor, TechReviewsUK & Founder, Unsourced.
Leave a Reply