The uneven rise of AI-generated web content

arxiv.org/pdf/2610.36631

Just learned that Patrick Stewart has passed away. Such a sad loss. Here he is in one of the best things ever put on television, the Star Trek episode “The Measure of a Man.” I show it to my students in Philosophy and Science Fiction every year.

In 1950, five people at MIT passed notes through slots. Today the math from that experiment helps decide what AI gets trained on.
stephenburns.ai/blog/harmonic-…

Common Crawl Foundation will be at IETF 127 in SF, week of November 15. If you are around, we'd love to chat!

October 2026 Crawl Archive Now Available

We are pleased to announce the release of the October 2026 crawl archive, consisting of 2.16 billion pages (or 371.19 TiB of uncompressed content)

Findings of the WMT 2026 Shared Task of the Open Language Data Initiative

Laurie Burchell Common Crawl Foundation

www2.statmt.org/wmt26/pdf/2026…

We present the results of the WMT 2026 shared task of the Open Language Data Initiative (OLDI). Participants were invited to contribute to the massively multilingual open datasets supported by OLDI, FLORES+ and OLDI Seed, or to create new resources in line with OLDI’s ethos. We accepted ten submissions: five extending FLORES+, and five extending other open and massively-parallel datasets (BOUQuET, Seed, SMOL, and WMT24+). These contributions advance the coverage and quality of multilingual datasets, especially for underserved language varieties. All contributions are released under permissive open-source licenses.

@CommonCrawl My bet: low-resource languages would lose at the filter.

Quality classifiers trained mostly on English throw out good Icelandic pages as noise. Smarter crawl budgets won't help if the filter can't read the language.

The Common Crawl Foundation at the TPDL Conference

The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.

arxiv.org/abs/2610.13321

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages.

We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data.

Common Crawl - the ultimate time capsule for knowledge | Amazon Web Services

youtube.com/watch?v=rGT-al…

we will have two papers @NeurIPSConf this year from the croissant community 🥐

1. "croissant baker" led by rafi al attrach and @tompollard
2. "croissant miner" led by berke arda and @akhtarmubashara

i took a quick glance at the rearview mirror

since early 2023, our idea for s layer has grown into an expansive ecosystem, incl

- data aggregators spanning hundreds of thousands of datasets incl @huggingface @kaggle @dataverseorg @CommonCrawl and others

- tooling spanning auto-generation and -verification of croissant files

- the formal spec maintenance under @MLCommons

- and a growing list of community extensions spanning verticals such geo, responsible AI, health and others

onwards towards the composable data dream(:

There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?

For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.

Here’s the work between downloading those and training a model 🧵

Information Stewardship Forum 2026: Creating Community and Purpose Around US Government Information

A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 rve and provide access to United States government information.