Pinned Tweet
Today, we launched Pangram 4.1.
We better calibrated Pangram's confidence, and we now show this tooltip indicating an AI range for documents we are less certain about.
We better calibrated Pangram's confidence, and we now show this tooltip indicating an AI range for documents we are less certain about.
55
50
983
54,739
I’m gonna sound like a schizo when I explain to my normie friends why I don’t have a TV
turns out it classifies what audio it hears, before reporting back to samsung.
sensor preferences, dog-barking and baby-crying event counters, and camera streaming duration are built into telemetry paths.
23
11
440
8,834
I went to a restaurant today that made me order by QR code. Scanning the code took me to an app that immediately opened a live activity on my phone. Closing the tab involved a default 22% tip and declining an offer for Polymarket credits
25
4
234
4,690
To clarify some common misconceptions, Pangram is not a measure of Good Writing.
A 100% human score on Pangram does not mean your writing is Good.
A 100% AI score on Pangram does not mean your writing is Bad.
Ultimately, Pangram is a fancy authorship attribution algorithm with near-infinite writing samples from a few dozen language models. The goal is to understand the degree of AI use and differentiate between human, AI-assisted, and AI-generated texts.
Pangram typically generalizes to unseen language models when any of the following conditions are met:
1) The LLM is part of the same family as a model in the training set (i.e. Opus 4.5 to Opus 4.6).
2) The LLM is trained on synthetic data from a model in the training set (i.e. Llama SFT'ed on 4o outputs).
3) The LLM is similar to a model in the training set (i.e. trained on mostly common crawl).
Pangram will not generalize to your language model when any of the following conditions are met:
1) Your LLM is a base model.
2) Your LLM is a narrow fine-tune (i.e. trained on only Hemingway or only /r/wallstreetbets).
3) Your LLM is producing degraded outputs (i.e. it's incoherent, full of grammatical mistakes, Binglish, or o3 CoT).
My experience is that choosing to optimize an LLM against Pangram comes with serious trade-offs to output quality. However, I fear that Pangram's increased mindshare will lead to us being used as a static benchmark to hill climb rather than its intended use as a tool to infer the provenance of text.
I have no ill will to anyone training writing models. We genuinely need more people working on this problem. But I am building for the future where we can detect all frontier LLMs, even those that produce clear, concise, "good" writing.
A 100% human score on Pangram does not mean your writing is Good.
A 100% AI score on Pangram does not mean your writing is Bad.
Ultimately, Pangram is a fancy authorship attribution algorithm with near-infinite writing samples from a few dozen language models. The goal is to understand the degree of AI use and differentiate between human, AI-assisted, and AI-generated texts.
Pangram typically generalizes to unseen language models when any of the following conditions are met:
1) The LLM is part of the same family as a model in the training set (i.e. Opus 4.5 to Opus 4.6).
2) The LLM is trained on synthetic data from a model in the training set (i.e. Llama SFT'ed on 4o outputs).
3) The LLM is similar to a model in the training set (i.e. trained on mostly common crawl).
Pangram will not generalize to your language model when any of the following conditions are met:
1) Your LLM is a base model.
2) Your LLM is a narrow fine-tune (i.e. trained on only Hemingway or only /r/wallstreetbets).
3) Your LLM is producing degraded outputs (i.e. it's incoherent, full of grammatical mistakes, Binglish, or o3 CoT).
My experience is that choosing to optimize an LLM against Pangram comes with serious trade-offs to output quality. However, I fear that Pangram's increased mindshare will lead to us being used as a static benchmark to hill climb rather than its intended use as a tool to infer the provenance of text.
I have no ill will to anyone training writing models. We genuinely need more people working on this problem. But I am building for the future where we can detect all frontier LLMs, even those that produce clear, concise, "good" writing.
45
42
836
41,748
Why doesn't someone just run large-scale plagiarism detection to find more fraudulent papers? I know a little bit about this space so I'll give it a stab.
There are two primary entities to know about: Turnitin and Crossref.
Turnitin today is the world's biggest provider of plagiarism detection software. Founded in 1998, they started in higher education and worked primarily on academic integrity.
Meanwhile, in the early internet, people wanted better way to link to references or objects without the links dying so they invented the DOI (I am glossing over a lot here). Scholarly publishers wanted a way to link references across journals, and through this, Crossref was born. Crossref quickly became the definitive DOI registry for scholarly publishing.
After Turnitin had been operating for years in academic integrity, they wanted to get into research integrity. Turnitin had the tech, Crossref had the database. Turnitin made a deal with them, winning exclusive access in exchange for discounts to publishers who share their data. Their tool was called iThenticate, and it is essentially the one and only plagiarism detection tool for research integrity, because of this exclusive deal for Crossref data. iThenticate is based on an N-gram matching algorithm that produces a similarity score, and similarity scores above a certain threshold are evidence of potential plagiarism.
Today, Turnitin sells iThenticate bidirectionally. They sell to the journals, to help them confirm no plagiarism before they accept and publish a paper. This is a revenue source, but because of the way Turnitin grants discounts to every Crossref member who shares their data, basically every journal buys iThenticate. This is because the universities are the bigger fish: they have more money and there are many more of them. They want to confirm that the articles they submit don't get desk rejected or accused of plagiarism. This would be a great reputational risk to the university, and essentially the service that Turnitin provides to both sides is a large reduction in reputational risk.
So, why is there so much plagiarism that doesn't get caught?
The important thing is that basically every major institution has iThenticate, and they can run it on manuscripts before submitting. This means, if somebody actually plagiarized, and they go to submit, they would see the similarity score is higher than it should be, and go back and rewrite parts of the text until the similarity score is acceptable. In the end, it's less about preventing plagiarism and more about flagging text that would be a smoking gun.
And what is stopping someone from running large scale plagiarism detection?
Turnitin's iThenticate algorithm is largely static. If it was good enough last year, it would simply be confusing to change it for this year. It would be embarrassing to all parties involved if an already-published article was found to have plagiarism that iThenticate didn't catch last year.
But even if iThenticate did update their algorithm, their business is to protect reputation, not cause a public reckoning. A large-scale re-scan would be reputationally destructive to many of their customers. No one would be happy.
And no one else has the data to build a similar plagiarism detector at scale – the paywalled data required would be massively cost-prohibitive for any new entrant coming into the scene.
There are two primary entities to know about: Turnitin and Crossref.
Turnitin today is the world's biggest provider of plagiarism detection software. Founded in 1998, they started in higher education and worked primarily on academic integrity.
Meanwhile, in the early internet, people wanted better way to link to references or objects without the links dying so they invented the DOI (I am glossing over a lot here). Scholarly publishers wanted a way to link references across journals, and through this, Crossref was born. Crossref quickly became the definitive DOI registry for scholarly publishing.
After Turnitin had been operating for years in academic integrity, they wanted to get into research integrity. Turnitin had the tech, Crossref had the database. Turnitin made a deal with them, winning exclusive access in exchange for discounts to publishers who share their data. Their tool was called iThenticate, and it is essentially the one and only plagiarism detection tool for research integrity, because of this exclusive deal for Crossref data. iThenticate is based on an N-gram matching algorithm that produces a similarity score, and similarity scores above a certain threshold are evidence of potential plagiarism.
Today, Turnitin sells iThenticate bidirectionally. They sell to the journals, to help them confirm no plagiarism before they accept and publish a paper. This is a revenue source, but because of the way Turnitin grants discounts to every Crossref member who shares their data, basically every journal buys iThenticate. This is because the universities are the bigger fish: they have more money and there are many more of them. They want to confirm that the articles they submit don't get desk rejected or accused of plagiarism. This would be a great reputational risk to the university, and essentially the service that Turnitin provides to both sides is a large reduction in reputational risk.
So, why is there so much plagiarism that doesn't get caught?
The important thing is that basically every major institution has iThenticate, and they can run it on manuscripts before submitting. This means, if somebody actually plagiarized, and they go to submit, they would see the similarity score is higher than it should be, and go back and rewrite parts of the text until the similarity score is acceptable. In the end, it's less about preventing plagiarism and more about flagging text that would be a smoking gun.
And what is stopping someone from running large scale plagiarism detection?
Turnitin's iThenticate algorithm is largely static. If it was good enough last year, it would simply be confusing to change it for this year. It would be embarrassing to all parties involved if an already-published article was found to have plagiarism that iThenticate didn't catch last year.
But even if iThenticate did update their algorithm, their business is to protect reputation, not cause a public reckoning. A large-scale re-scan would be reputationally destructive to many of their customers. No one would be happy.
And no one else has the data to build a similar plagiarism detector at scale – the paywalled data required would be massively cost-prohibitive for any new entrant coming into the scene.
@literalbanana hmm what's confusing to me is that plagiarizing detection software already exists for student papers? so is just running all of jstor through that just a cost issue? an API issue? 'no one cares' issue?
fuzzy matching shouldn't be a problem idt, or "80 words of 100 in same order"
23
34
465
65,987
I fear that any free API or hosted content will become untenably expensive to serve in the age of AI agents
if you're using claude to edit videos, give it 17M+ films and 14M audio recordings for $0
the internet archive has a free api. no key.
10,000+ prelinger films (public domain stock footage)
187,000 78rpm records
300,000+ live concert recordings
give it archive.org/developers/ need archival b-roll or period music, search the prelinger and 78rpm collections on archive.org and use them in the edit”
some great clips that are amazing for slightly more artsy ads for tech
35
16
770
26,182
Pangram is hiring a Senior Brand Designer! We're looking for a talented designer to connect Pangram’s visual identity directly to our mission of becoming the voice of authenticity across the Internet.
Please reach out if you're interested!
jobs.ashbyhq.com/pangramlabs/7f…
Please reach out if you're interested!
jobs.ashbyhq.com/pangramlabs/7f…
7
7
86
4,072
i am going insane, what is going on here
@QualiaQuanta I feel like I am going crazy, I searched Scope of New Mechanism and looked at all of the page one results. Is there something different I am not seeing? Can you just send a link?
42
1
617
58,419
to give maximum benefit of the doubt, this may be true and the author is just very confused about it
x.com/birchlse/statu…
x.com/birchlse/statu…
@max_spero_ When an abstractless paper is uploaded to PhilPapers, either people or bots (I don't know which) will sometimes add an AI-generated abstract. In this case, I think this has confused the author of the paper itself, who correctly remembers the paper as being human written.
1
0
22
828
holy shit this is crazy technology
@pleometric @theojaffee What does this post actually mean? 📼
12
6
326
23,111