LLMs and web data: how AI collects, filters and uses information.

We often hear that “LLMs find everything on the Internet.” This statement, both fascinating and somewhat magical, comes up frequently when discussing generative artificial intelligence. But how does it really work? Let’s try to see more clearly.

Two major phases: training & search.

Model Training

First crucial step: LLMs (Large Language Models) are trained on massive amounts of text from the web, books, articles, forums, etc. This giant database, regularly updated, has two roles:

• Integrate content: knowledge, facts, references.
  ps: statistical calculation on word usage
• Understand language: syntax, nuances, writing styles.

It’s this dual capability that allows LLMs to be both precise and adaptable to varied questions.

Direct Web Search

Second step, increasingly crucial: real-time search. Here, the LLM queries the web to complete its response with fresh, relevant information for the question asked. This is what allows, for example, obtaining in a few seconds a synthesis of the latest reviews on a product or recent news.

Where do the data come from? The war of bots and indexers

How is data retrieved to train LLMs? Major AI companies have developed their own indexing bots — AI2Bot, ClaudeBot, GPTBot, Bytespider, etc. Their mission: crawl the web, collect text, index, then filter.

Post-collection filtering: a mandatory step

After collection, data goes through a technical pipeline in several steps:

Deduplication and cleaning

Algorithms detect duplicates; HTML code and metadata are removed; encoding is normalized and classifiers identify the language.

Quality scoring

Automatic criteria evaluate readability (sentence length, vocabulary, structure), spam or generated text detection, and content coherence.

”Unlearning”

New potential step: there are publications about making models forget, parameter modifications, but none of the major players has yet spoken on the subject.

This process is framed by discussions and agreements on data usage rights, between publishers, platforms, and authorities, according to rules specific to each country.

Who feeds these web searches?

But the other revolution is the ability to query the web directly to answer a question. For the user, it’s unprecedented comfort: no need to compare 15 product sheets or 8 consumer reviews, just one prompt, and the LLM synthesizes everything for you in a few seconds.
For brands, the stakes are huge: being in the top 4 of proposed answers is the new Grail.

Each AI player relies on its own engine:

• ChatGPT Search (OpenAI): Uses a system developed by OpenAI, relying on Bing’s index, but the processing layer (filtering, synthesis, citations) remains OpenAI’s own.
• Claude (Anthropic): Now relies on Brave Search as the primary source for its real-time web queries, confirmed by several technical analyses and communications.
• Perplexity.ai: Has its own proprietary index for the majority of its searches, but continues to aggregate signals from other engines, particularly Bing.
• Google Gemini: Uses Google Search’s index directly (logical!).

Each player optimizes its queries, filters and ranks results, which changes the nature of responses depending on the engine.

Data usage policy: Meta and others

Contrary to what we sometimes imagine, Meta (and other major players) do not automatically use all public web data to train their models. Their policy depends on the law, content type, and consent obtained.

A strict legal framework

• GDPR in Europe: requires explicit consent for any personal data, even public.
• Copyright: protected content is only usable with a license agreement (negotiations ongoing with publishers).
• Robots.txt and opt-out: sites can refuse indexing, and this choice is increasingly respected.
• Geographic zoning: for example, Meta exploits US data but excludes certain European content for its models in Europe.

Generally excluded data

• Sensitive medical data and automatically detected private information
• Content subject to restrictive licenses or protected by a paywall
• Private forums and spaces requiring authentication

In the end, collection is never total: most countries impose strict rules that protect certain content. The challenge remains finding the balance between technological innovation and respect for rights.

Don’t forget the limitations and risks of LLMs

While LLMs impress with their ability to synthesize web information, they are not without flaws. Two major problems deserve to be highlighted.

Hallucinations: when AI invents

“Hallucinations” refer to those moments when an LLM generates factually incorrect information, but presented with confidence. This phenomenon occurs particularly when the model “fills in the gaps” of missing information or draws hasty conclusions from insufficient clues.

Biases: distorting mirror of the web

LLMs inherit and amplify biases present in their training and web search sources. Overrepresentation of dominant content, reproduction of existing stereotypes: AI reflects the web’s imbalances rather than an objective view of the world.

Other risks to monitor

• Fake news propagation: Poorly filtered real-time searches can relay or amplify false information, with a viral effect.
• Environmental impacts: Mass web scraping and model training consume a lot of energy, with a non-negligible ecological impact.

These limitations remind us of the importance of keeping a critical eye on AI responses, however sophisticated they may be.

In summary:

LLMs do indeed exploit enormous web resources, but all this relies on a much more complex mechanics than simple “magical absorption” of the Internet. Collection, filtering, rule compliance, and web search strategies form a moving ecosystem, with a lot of human behind the machine.

And finally, a word on SEO:
One might think, faced with the rise of AI, that natural referencing (SEO) is doomed to disappear. Not at all! On the contrary, in a world where LLMs draw from the best web results to inform and build their responses, SEO remains more strategic than ever, you also need to be selected as a reliable source by AIs. This implies:

• Structured and well-documented content.
• Regular updates to stay relevant.
• Optimization for conversational queries (questions/answers).

In short, artificial intelligence is transforming the rules of the game… but the game continues!

Resources

Final list of sources for your bibliography

1. Kyle Wiggers, TechCrunch — Anthropic appears to be using Brave to power web search for its Claude chatbot (TechCrunch)

2. Simon Willison’s Weblog — Anthropic Trust Center: Brave Search added as a subprocessor (March 19, 2025) (Simon Willison’s Weblog)

3. Article « Claude Web Search Explained » (Profound) – statistical proof of Brave/Claude correspondence (~86.7%) (tryprofound.com)

4. Financial Times / she‑velenko interview – Perplexity has its own index, not just Bing (Financial Times)

5. PYMNTS / BBC legal claim – Conflicts over unauthorized content reuse by Perplexity (PYMNTS.com)

6. Wikipedia / Brave Search – Independence of Brave index since August 2023 (Wikipedia)