Introducing warc2zip
We are pleased to introduce warc2zip, a tool that allows anyone to turn WARC archives into simpler, zipped files that can be inspected with ordinary tools.
We are pleased to introduce warc2zip, a tool that allows anyone to turn WARC archives into simpler, zipped files that can be inspected with ordinary tools.
3
0
10
Common Crawl Foundation will be at IETF 127 in SF, week of November 15. If you are around, we'd love to chat!
1
4
11
October 2026 Crawl Archive Now Available
We are pleased to announce the release of the October 2026 crawl archive, consisting of 2.16 billion pages (or 371.19 TiB of uncompressed content)
We are pleased to announce the release of the October 2026 crawl archive, consisting of 2.16 billion pages (or 371.19 TiB of uncompressed content)
1
1
19
The Common Crawl Foundation at the TPDL Conference
The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.
The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.
1
1
2
Common Crawl - the ultimate time capsule for knowledge | Amazon Web Services
youtube.com/watch?v=rGT-al…
youtube.com/watch?v=rGT-al…
1
3
5
Information Stewardship Forum 2026: Creating Community and Purpose Around US Government Information
A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 rve and provide access to United States government information.
A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 rve and provide access to United States government information.
1
0
7
September 2026 Crawl Archive Now Available
We are pleased to announce the release of the September 202 6 crawl archive, consisting of 2.17 billion pages (or 361.4 TiB of uncompressed content.)
📷
We are pleased to announce the release of the September 202 6 crawl archive, consisting of 2.17 billion pages (or 361.4 TiB of uncompressed content.)
📷
1
1
12
Getting Started with Common Crawl Data on Hugging Face
We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build.
We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build.
1
5
14
CommonsDB, ISCC, Spaghetti, and Meatballs
Notes from the CommonsDB final conference in Alicante: a registry of 6.5 million open works keyed on ISCC, and the EUIPO's push for federated copyright infrastructure.
Notes from the CommonsDB final conference in Alicante: a registry of 6.5 million open works keyed on ISCC, and the EUIPO's push for federated copyright infrastructure.
1
1
4
Bodhium Labs: From web pages to a question corpus
FineQuestions draws on 110 snapshots of FineWeb, the filtered English-language corpus derived from Common Crawl, spanning 2013–2025. Extraction yielded roughly 13.4 billion raw questions filling 22 TB. Normalization and se to approximately 1.5 billion unique questions.
FineQuestions draws on 110 snapshots of FineWeb, the filtered English-language corpus derived from Common Crawl, spanning 2013–2025. Extraction yielded roughly 13.4 billion raw questions filling 22 TB. Normalization and se to approximately 1.5 billion unique questions.
2
0
3
Web Graph Embeddings: An Experimental Dataset Release
We are publishing an experimental dataset of graph embeddings built from the Common Crawl Web Graph: a 128-dimensional vector for each of 52.9 million web hosts, learned from hyperlinks alone, along with two interactive Hugging Face Spaces.
We are publishing an experimental dataset of graph embeddings built from the Common Crawl Web Graph: a 128-dimensional vector for each of 52.9 million web hosts, learned from hyperlinks alone, along with two interactive Hugging Face Spaces.
3
2
14
Common Crawl Rate Limit
The CDX index server is heavily rate limited. What triggers a block, how long it lasts, and the interface that has no limit.
The CDX index server is heavily rate limited. What triggers a block, how long it lasts, and the interface that has no limit.
2
2
6
A Content Analysis of llms.txt Files from the July 2026 Crawl Archive
We analyzed the contents of 584,107 llms.txt files from the July 2026 crawl. Two thirds are produced by a plugin, half follow the structure the specification defines, and a few files even contain prompt injections.
We analyzed the contents of 584,107 llms.txt files from the July 2026 crawl. Two thirds are produced by a plugin, half follow the structure the specification defines, and a few files even contain prompt injections.
1
1
5
Thom Vaughan of Common Crawl, speaking from the Paris Open Source AI Summit recently.
opensourceaisummit.eu
opensourceaisummit.eu
0
2
7
August 2026 Crawl Archive Now Available
We are pleased to announce that the crawl archive for August 2026 is now available, containing 2.14 billion web pages or 360 TiB of uncompressed content.
We are pleased to announce that the crawl archive for August 2026 is now available, containing 2.14 billion web pages or 360 TiB of uncompressed content.
3
1
15
Announcing the First Stable Release of CC-Downloader
Over a year ago we released cc-downloader, an experimental tool to politely download Common Crawl data. Today we're releasing its first stable version, with a Rust library and Python bindings.
Over a year ago we released cc-downloader, an experimental tool to politely download Common Crawl data. Today we're releasing its first stable version, with a Rust library and Python bindings.
1
3
12
Notes from HTTP Workshop Basel and IETF 126 Vienna
Two weeks in Basel and Vienna, at the HTTP Workshop and IETF 126. Protocol adoption measured across the whole web, and an attempt to define what "machine readable" actually means.
Two weeks in Basel and Vienna, at the HTTP Workshop and IETF 126. Protocol adoption measured across the whole web, and an attempt to define what "machine readable" actually means.
1
1
6
July 2026 Crawl Archive Now Available
The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce some improvements and changes.
The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce some improvements and changes.
2
1
5
Common Crawl Joins Project Tapestry
Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and culturally informed AI development.
📷
Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and culturally informed AI development.
📷
3
2
6
Measuring Crawled Coverage of a Website in Common Crawl
How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think.
How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think.
2
0
6
Is one vantage point enough? IPv6 across the top million web hosts
We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 327% work from everywhere, but one vantage point turns out to be enough for the headline rate.
We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 327% work from everywhere, but one vantage point turns out to be enough for the headline rate.
2
0
5
Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls
Amazing research by {micha,hande,luca} @ commoncrawl
Amazing research by {micha,hande,luca} @ commoncrawl
1
1
8
Turning 30,000 Arabic Domains Into a Better Crawl
How we filtered, geolocated and categorised a donation of Arabic seed domains
How we filtered, geolocated and categorised a donation of Arabic seed domains
1
2
4
Common Crawl Foundation at LREC 2026
The Common Crawl team attended the 16th International Conference on Language Resources and Evaluation in Palma, Mallorca, co-organizing a tutorial, presenting recent published work, and strengthening links with the research community.
The Common Crawl team attended the 16th International Conference on Language Resources and Evaluation in Palma, Mallorca, co-organizing a tutorial, presenting recent published work, and strengthening links with the research community.
1
0
7
Common Crawl Foundation at IIPC-WAC 2026
Common Crawl was well represented with contributions at the 2026 IIPC Web Archiving Conference and General Assembly.
Common Crawl was well represented with contributions at the 2026 IIPC Web Archiving Conference and General Assembly.
1
0
8
CommonLID Update: New Tools, Growing Impact
CommonLID, a community-built language ID benchmark, has a new website and interactive leaderboard. Its paper was accepted to ACL 2026, with a poster session on 7 July. Source code, a PyPI package, and the dataset are now available.
CommonLID, a community-built language ID benchmark, has a new website and interactive leaderboard. Its paper was accepted to ACL 2026, with a poster session on 7 July. Source code, a PyPI package, and the dataset are now available.
1
0
1
Introducing the AI Visibility Audit
A free guide for SEOs and GEOs on how to check whether AI systems can actually reach a site, and how to stay visible in the crawl that trains them.
A free guide for SEOs and GEOs on how to check whether AI systems can actually reach a site, and how to stay visible in the crawl that trains them.
2
0
7
Under-represented languages deserve better tools! On June 4th, The Common Crawl Foundation and Mozilla Data Collective will host a webinar to test language identification for the languages you care about.
1
0
4
March 2026 Crawl Archive Now Available
We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content. We also observed a dramatic increase in fetches over IPv6, explained by the enabling of Happy Eyeballs in the OkHttp library.
We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content. We also observed a dramatic increase in fetches over IPv6, explained by the enabling of Happy Eyeballs in the OkHttp library.
1
2
13
IPv6 Adoption Across the Top 100K Web Hosts
We probed the 100,000 most-linked web hosts for IPv6 support using the Common Crawl Web Graph. Only 36.9% are fully reachable over IPv6, with adoption ranging from 71% among the top 100 to 32% in the long tail.
We probed the 100,000 most-linked web hosts for IPv6 support using the Common Crawl Web Graph. Only 36.9% are fully reachable over IPv6, with adoption ranging from 71% among the top 100 to 32% in the long tail.
1
1
4
Introducing the New Examples & Resources Browser
We've replaced our old Examples and Use Cases pages with a single searchable, filterable browser. 119 resources from 115 contributors, all in one place. Search, filter by type or language, sort, and share links. We welcome community submissions.
We've replaced our old Examples and Use Cases pages with a single searchable, filterable browser. 119 resources from 115 contributors, all in one place. Search, filter by type or language, sort, and share links. We welcome community submissions.
4
7
15
AI Plumbers at FOSDEM’26
Common Crawl was invited to the AI Plumbers unconference held at FOSDEM this year. The contrast between the 100 people at the unconference, compared to the 10,000 people at the main event, couldn't be bigger.
Common Crawl was invited to the AI Plumbers unconference held at FOSDEM this year. The contrast between the 100 people at the unconference, compared to the 10,000 people at the main event, couldn't be bigger.
1
0
1
IBM researchers created a high-quality, 10-trillion-token dataset called GneissWeb, which makes up Granite’s core linguistic capabilities. Common Crawl has open-sourced Gneiss Web's annotations providing direct access to the data.
1
0
4
CC-Citations: A Visualization of Research Papers Referencing Common Crawl
We are proud to release an interactive visualization of thousands of research papers using or citing Common Crawl data.
We are proud to release an interactive visualization of thousands of research papers using or citing Common Crawl data.
1
0
5
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
We are excited to announce the release of CommonLID, a language identification benchmark for the web, covering 109 languages. CommonLID was developed in collaboration with multiple ns and language community groups.
We are excited to announce the release of CommonLID, a language identification benchmark for the web, covering 109 languages. CommonLID was developed in collaboration with multiple ns and language community groups.
2
5
17
Web Archives for Social Sciences Datathon, Bristol
Recently, a two-day Bristol datathon used Common Crawl web archives to analyse UK industries and policy, strengthening social science research through hands-on, team-based work.
Recently, a two-day Bristol datathon used Common Crawl web archives to analyse UK industries and policy, strengthening social science research through hands-on, team-based work.
3
1
5
