Scrapling has 100,136 GitHub stars. Scrapy has 65,286.
Scrapy got 4.7x the downloads last month.
Stars are attention. Downloads are deployments.
Scrapy got 4.7x the downloads last month.
Stars are attention. Downloads are deployments.
1
0
0
15
17 of the 42 tools were created since 2024. They hold 58% of the stars.
AI-ready crawlers, browser agents, and MCP servers are where the new attention went.
AI-ready crawlers, browser agents, and MCP servers are where the new attention went.
1
0
0
5
We downloaded the IP lists that 8 major AI bot operators publish. 14 lists, about 94,000 IPv4 addresses.
ChatGPT-User's covers 69,280. GPTBot's: 2,704.
A user agent is a claim. The IP list is how you check it.
ChatGPT-User's covers 69,280. GPTBot's: 2,704.
A user agent is a claim. The IP list is how you check it.
1
1
3
150
23 AI bots from 9 operators, and how to verify each: joinmassive.com/blog/ai-crawle…
1
0
0
62
Wikimedia told a federal court on Oct 29: training an LLM is likely fair use. Copying pages into chatbot answers is not.
News sites' robots.txt files draw the line the other way.
News sites' robots.txt files draw the line the other way.
2
0
2
117
The brief, in the consolidated OpenAI copyright case in New York, backs neither side, and the court hasn't ruled on it.
Its fix for retrieval: follow the source's license, like Wikipedia's CC BY-SA credit. Not legal advice.
Its fix for retrieval: follow the source's license, like Wikipedia's CC BY-SA credit. Not legal advice.
1
0
0
35
The brief: commons.wikimedia.org/wiki/File:26.1… What 49 news sites block, bot by bot: joinmassive.com/blog/should-i-…
0
0
0
31
Google just had its fastest query growth in five years. The new searches earn it less.
AI Mode queries are: 3x longer and 1 in 5 voice, photo or document mostly brand new
Tracking 3-word keywords? You're measuring the old Google.
AI Mode queries are: 3x longer and 1 in 5 voice, photo or document mostly brand new
Tracking 3-word keywords? You're measuring the old Google.
1
0
3
107
Massive's network didn't start as a pool of IP addresses. It started as an app-monetization SDK.
1
2
3
183
Today about 1.3M devices are active on it daily, across 195+ countries.
That's why we count devices, not IPs. Every one joined through that opt-in.
That's why we count devices, not IPs. Every one joined through that opt-in.
1
0
0
23
How the opt-in works, step by step: joinmassive.com/blog/how-massi…
0
0
0
16
Blocking GPTBot doesn't keep you out of ChatGPT search. That's OAI-SearchBot.
Blocking Google-Extended doesn't keep you out of AI Overviews. That's Googlebot.
Each AI company runs separate bots for training, search, and user fetches.
Blocking Google-Extended doesn't keep you out of AI Overviews. That's Googlebot.
Each AI company runs separate bots for training, search, and user fetches.
2
0
1
140
On 49 news sites (Nov 1):
35 block PerplexityBot, which Perplexity says isn't used for model training
30 block Google-Extended but not Googlebot
12 block GPTBot but not OAI-SearchBot
35 block PerplexityBot, which Perplexity says isn't used for model training
30 block Google-Extended but not Googlebot
12 block GPTBot but not OAI-SearchBot
1
0
0
33
Every bot's job, from each company's docs, and what 49 news sites block: joinmassive.com/blog/should-i-…
0
0
0
23
One robots.txt group will soon tell Google, OpenAI, Microsoft and Anthropic not to train on your pages:
User-Agent: __usage_ai-training
Disallow: /
It's a draft, and none of the four honor it yet.
User-Agent: __usage_ai-training
Disallow: /
It's a draft, and none of the four honor it yet.
1
0
4
150
We checked 49 news sites on Nov 1: 0 use the token yet.
10 of 33 homepages we could load rely on NoArchive or NoCache, which Microsoft stops reading for training or grounding after March 1, 2027.
10 of 33 homepages we could load rely on NoArchive or NoCache, which Microsoft stops reading for training or grounding after March 1, 2027.
1
0
0
17
How the opt-out works, each company's commitment, and our 49-site check: joinmassive.com/blog/robots-tx…
0
0
0
16