Something the team and I have been quietly building for the past months. Stoked to release the first benchmark on how well AI models actually perform at small molecule preclinical discovery. More to come soon on biologics, oligos, gene & cell therapies across disease areas.
We introduce TherapeuticsBench Preclinical Pharmacology (TxBench-PP), a verifiable benchmark for small-molecule preclinical pharmacology and the first focused slice of a broader benchmarking effort across drug-discovery stages and therapeutic modalities.
TxBench-PP tests whether rate conclusions from realistic assay artifacts rather than memorized facts from the literature. The benchmark contains 100 evaluations indexed by program stage, assay type, and task structure, spanning mechanism-of-action (MoA) and pharmacodynamic (PD) reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy.
The strongest model-harness configuration was Claude Opus 4.8 + Pi at 59.3%, followed by GPT-5.5 + Pi at 55.3%.
While experiments are rate-limited by natural processes, human decisions and organizational consensus often make up significant components of program timelines in drug discovery. Agents promise to accelerate discovery, development, and translation by compressing these interpretation and decision-making loops.
However, the practical use of agentic systems in industrial workflows requires standardized and trusted methods of evaluating performance. This is especially challenging in drug discovery because the ecosystem is a sprawling landscape of assay categories, development stages, therapeutic modalities, and decision types.
Benchmarks must therefore measure realistic tasks while providing focused treatment of the many local scientific judgments that make up the biotech ecosystem.
We evaluated 16 model-harness configurations, comprising 11 models across three agent harnesses, on 100 preclinical pharmacology tasks. Each configuration was run three independent times per task, yielding 4,800 agent trajectories.
Performance varied by program stage: accuracy ranged from 27% in screening and hit prioritization to 55% in drug response. Difficult program stages involved decisions across QC, statistics, and chemical or biological judgment of molecular candidates.
Trajectory analysis reveals gaps in scientific judgement. Failures included incorrect perception of assay outputs, reliance on literature priors over supplied evidence, and assay-specific reasoning mistakes.
Manuscript, results and subset of evals/trajectories available below:
0
3
16
2,500
Hannah Han Le retweeted
We introduce TherapeuticsBench Preclinical Pharmacology (TxBench-PP), a verifiable benchmark for small-molecule preclinical pharmacology and the first focused slice of a broader benchmarking effort across drug-discovery stages and therapeutic modalities.
TxBench-PP tests whether rate conclusions from realistic assay artifacts rather than memorized facts from the literature. The benchmark contains 100 evaluations indexed by program stage, assay type, and task structure, spanning mechanism-of-action (MoA) and pharmacodynamic (PD) reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy.
The strongest model-harness configuration was Claude Opus 4.8 + Pi at 59.3%, followed by GPT-5.5 + Pi at 55.3%.
While experiments are rate-limited by natural processes, human decisions and organizational consensus often make up significant components of program timelines in drug discovery. Agents promise to accelerate discovery, development, and translation by compressing these interpretation and decision-making loops.
However, the practical use of agentic systems in industrial workflows requires standardized and trusted methods of evaluating performance. This is especially challenging in drug discovery because the ecosystem is a sprawling landscape of assay categories, development stages, therapeutic modalities, and decision types.
Benchmarks must therefore measure realistic tasks while providing focused treatment of the many local scientific judgments that make up the biotech ecosystem.
We evaluated 16 model-harness configurations, comprising 11 models across three agent harnesses, on 100 preclinical pharmacology tasks. Each configuration was run three independent times per task, yielding 4,800 agent trajectories.
Performance varied by program stage: accuracy ranged from 27% in screening and hit prioritization to 55% in drug response. Difficult program stages involved decisions across QC, statistics, and chemical or biological judgment of molecular candidates.
Trajectory analysis reveals gaps in scientific judgement. Failures included incorrect perception of assay outputs, reliance on literature priors over supplied evidence, and assay-specific reasoning mistakes.
Manuscript, results and subset of evals/trajectories available below:
TxBench-PP tests whether rate conclusions from realistic assay artifacts rather than memorized facts from the literature. The benchmark contains 100 evaluations indexed by program stage, assay type, and task structure, spanning mechanism-of-action (MoA) and pharmacodynamic (PD) reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy.
The strongest model-harness configuration was Claude Opus 4.8 + Pi at 59.3%, followed by GPT-5.5 + Pi at 55.3%.
While experiments are rate-limited by natural processes, human decisions and organizational consensus often make up significant components of program timelines in drug discovery. Agents promise to accelerate discovery, development, and translation by compressing these interpretation and decision-making loops.
However, the practical use of agentic systems in industrial workflows requires standardized and trusted methods of evaluating performance. This is especially challenging in drug discovery because the ecosystem is a sprawling landscape of assay categories, development stages, therapeutic modalities, and decision types.
Benchmarks must therefore measure realistic tasks while providing focused treatment of the many local scientific judgments that make up the biotech ecosystem.
We evaluated 16 model-harness configurations, comprising 11 models across three agent harnesses, on 100 preclinical pharmacology tasks. Each configuration was run three independent times per task, yielding 4,800 agent trajectories.
Performance varied by program stage: accuracy ranged from 27% in screening and hit prioritization to 55% in drug response. Difficult program stages involved decisions across QC, statistics, and chemical or biological judgment of molecular candidates.
Trajectory analysis reveals gaps in scientific judgement. Failures included incorrect perception of assay outputs, reliance on literature priors over supplied evidence, and assay-specific reasoning mistakes.
Manuscript, results and subset of evals/trajectories available below:
12
28
159
54,422
Hannah Han Le retweeted
China produces over 98% of the world's Lithium Iron Phosphate (LFP) cathode material, effectively holding a global monopoly over arguably the most important and industrially critical battery chemistry today. What people don’t realize is that this was an invention made in the U.S. focused on IP and legalities, China was handed a decade-long head start to build the entire industrial stack and focused on scaling.
I think this is one of the most interesting case-studies detailing the consequence of focusing too much on the discovery and losing sight of the real goal, manufacturability.
I think this is one of the most interesting case-studies detailing the consequence of focusing too much on the discovery and losing sight of the real goal, manufacturability.
1
4
19
2,347
Surprising how frontier models are still pretty weak at biology! It's been fun working with Kenny, Zhen, @Harihara_subrah to build this benchmark.
Below is a further breakdown of a model’s analysis journey, where it fails, plus insights that nearly 2X performance in our tests:
Below is a further breakdown of a model’s analysis journey, where it fails, plus insights that nearly 2X performance in our tests:
2026 will be the year of agents in biology. But we need better benchmarks.
We worked with scientists to turn real world analysis into verifiable problems. SpatialBench stratifies frontier models, shows harnesses matter, and reveals distinct failure modes between model families:
2
6
17
6,924
5/ Spatial analysis (i.e. how neighboring cells interact)
Models lack fluency with spatial-specific tools (Squidpy, LIANA, CellPhoneDB). One model wrote a segmentation algorithm from scratch (impressively but unnecessarily).
Another failure mode is lack spatial reasoning: integrating coordinates, expression, and cell type context simultaneously.
When asked to count follicles, models picked arbitrary radius defaults without checking cell composition. A biologist knows each follicle should contain roughly one oocyte.
Models lack fluency with spatial-specific tools (Squidpy, LIANA, CellPhoneDB). One model wrote a segmentation algorithm from scratch (impressively but unnecessarily).
Another failure mode is lack spatial reasoning: integrating coordinates, expression, and cell type context simultaneously.
When asked to count follicles, models picked arbitrary radius defaults without checking cell composition. A biologist knows each follicle should contain roughly one oocyte.
1
0
2
156
A few implications for building better system
> Tacit biology knowledge is not codified anywhere. An underrated and a unique reason why making models good at biology is harder than coding. There is no StackOverflow equivalent for biology.
Learning bio is like artisanal apprenticeship where much knowledge is quietly trapped inside the heads of thousands of scientists, earned through years of intuition building in the lab.
Opportunity here to build a database that captures heuristics at each of the above steps for every kit and experiment type.
> Better self-evaluation mechanisms: Models seem to lack judgment of what "good" means. Encoding biologists' intuition in areas like cell typing (e.g., canonical markers separating predicted cell types well, reasonable percentage of cell type proportions) is important.
> Gold-standard tools with correct abstractions for agents to manage compute infrastructure: Most agent "skills" are just prompts plus API docs. But real biological analysis requires orchestrating tens to hundreds of cloud machines (protein structure prediction, for instance, takes minutes to an hour per protein on a GPU, and you often need hundreds of predictions) Agents must reliably manage this scale.
More on these here: blog.latch.bio/p/agents-for-s…
> Tacit biology knowledge is not codified anywhere. An underrated and a unique reason why making models good at biology is harder than coding. There is no StackOverflow equivalent for biology.
Learning bio is like artisanal apprenticeship where much knowledge is quietly trapped inside the heads of thousands of scientists, earned through years of intuition building in the lab.
Opportunity here to build a database that captures heuristics at each of the above steps for every kit and experiment type.
> Better self-evaluation mechanisms: Models seem to lack judgment of what "good" means. Encoding biologists' intuition in areas like cell typing (e.g., canonical markers separating predicted cell types well, reasonable percentage of cell type proportions) is important.
> Gold-standard tools with correct abstractions for agents to manage compute infrastructure: Most agent "skills" are just prompts plus API docs. But real biological analysis requires orchestrating tens to hundreds of cloud machines (protein structure prediction, for instance, takes minutes to an hour per protein on a GPU, and you often need hundreds of predictions) Agents must reliably manage this scale.
More on these here: blog.latch.bio/p/agents-for-s…
0
0
3
130
Hannah Han Le retweeted
Launching a public agent sandbox for spatial biology. Five demo flows tailored to specific kits/machines
Try it now: agent.bio
This is a shippable intermediary towards reliable and widely deployed agentic systems used to make expensive scientific decisions.
Try it now: agent.bio
This is a shippable intermediary towards reliable and widely deployed agentic systems used to make expensive scientific decisions.
3
17
80
21,339
Example of product shipments recently:
> Universal H5 viewer for spatial biology (AnnData, PMTiles, DuckDB to render millions of cells, transcripts and tiled high content images in the browser)
> latch-curate: Python agentic framework to clean the Internet's molecular data
> Forch: Recratch to install Latch in pharma clouds <20 minutes
... and more
> Universal H5 viewer for spatial biology (AnnData, PMTiles, DuckDB to render millions of cells, transcripts and tiled high content images in the browser)
> latch-curate: Python agentic framework to clean the Internet's molecular data
> Forch: Recratch to install Latch in pharma clouds <20 minutes
... and more
0
0
0
195
(7/11)
Is cost per Gb all that matters?
While cost per gigabase has become the dominant metric for comparing sequencing platforms, this narrow focus obscures the substantial computational infrastructure investments required to transform raw sequencing data into useful insights.
Different companies adopt different hardware + software strategies:
- Illumina with DRAGEN for FPGA-accelerated genomics, but high license costs
- Oxford Nanopore and PacBio Revio put A100 GPUs right inside the device
- Element streams data to customer cloud and automatically start Bases2Fastq
- Ultima has built-in HPC but also plug into ecosystem of pipelines on AWS Omics.
Is cost per Gb all that matters?
While cost per gigabase has become the dominant metric for comparing sequencing platforms, this narrow focus obscures the substantial computational infrastructure investments required to transform raw sequencing data into useful insights.
Different companies adopt different hardware + software strategies:
- Illumina with DRAGEN for FPGA-accelerated genomics, but high license costs
- Oxford Nanopore and PacBio Revio put A100 GPUs right inside the device
- Element streams data to customer cloud and automatically start Bases2Fastq
- Ultima has built-in HPC but also plug into ecosystem of pipelines on AWS Omics.
1
0
3
435
(10/11)
Are sequencing machines curiously evolving into something more?
- Illumina announced spatial technology program in 2025.
- Singular introduced Direct-seq and G4X to image transcripts + proteins in FFPE tissue.
- Element’s AVITI24 for “5D multiomics" (DNA, RNA, proteins, , and cell morphology)
- PacBio’s SPRQ chemistry for calling DNA sequence, methylation, and chromatin accessibility.
Originally built as DNA readers, these machines are becoming far more powerful than their original architects ever imagined.
Are sequencing machines curiously evolving into something more?
- Illumina announced spatial technology program in 2025.
- Singular introduced Direct-seq and G4X to image transcripts + proteins in FFPE tissue.
- Element’s AVITI24 for “5D multiomics" (DNA, RNA, proteins, , and cell morphology)
- PacBio’s SPRQ chemistry for calling DNA sequence, methylation, and chromatin accessibility.
Originally built as DNA readers, these machines are becoming far more powerful than their original architects ever imagined.
1
0
3
439
(11/11)
Resources to further explore:
- Database: latch.bio/omics-provider…
- Blog post: blog.latch.bio/p/a-primer-on-…
Resources to further explore:
- Database: latch.bio/omics-provider…
- Blog post: blog.latch.bio/p/a-primer-on-…
1
0
7
282
It’s wild to think that Sanger sequencing, which famously decoded the first 5,000-base genome of bacteriophage φX174, was invented almost half a century ago.
The space has exploded with 10 companies and 37 instruments. If you factor in the different flow cell types and be used with each machine, you end up with an overwhelming 68 options!
I’ve been digging into the market, technical aspects of their products, and wrote a piece comparing existing techs.
9.7K words, 50 minute read time. Plus a website to compare their technical specs.
The space has exploded with 10 companies and 37 instruments. If you factor in the different flow cell types and be used with each machine, you end up with an overwhelming 68 options!
I’ve been digging into the market, technical aspects of their products, and wrote a piece comparing existing techs.
9.7K words, 50 minute read time. Plus a website to compare their technical specs.
5
17
72
11,312
(6/11)
Scientists need to navigate the cost per genome vs instrument price paradox.
The most expensive machines often deliver the lowest cost per genome, while affordable instruments can carry surprisingly high per-sample costs. Looking at cost per genome alone can be misleading.
Scientists need to navigate the cost per genome vs instrument price paradox.
The most expensive machines often deliver the lowest cost per genome, while affordable instruments can carry surprisingly high per-sample costs. Looking at cost per genome alone can be misleading.
1
1
1
348
(5/11)
Let’s dive into a comparison by attributes.
The U.S. seems to be trailing behind Chinese manufacturers in terms of throughput and speed.
Topping the chart is Complete Genomics (owned by Chinese‑based MGI), which dominates when it comes to throughput per day. Their ields 72 TB data for one 80-hour run, translating to ~21.6 Tb/day.
GeneMind, founded in Shenzhen, developed the SURFSeq Q, which outputs 14 TB in 36 hours, or over 9.3 Tb/day.
Then there’s Illumina's flagship NovaSeq X Plus at 8 Tb/ day.
Let’s dive into a comparison by attributes.
The U.S. seems to be trailing behind Chinese manufacturers in terms of throughput and speed.
Topping the chart is Complete Genomics (owned by Chinese‑based MGI), which dominates when it comes to throughput per day. Their ields 72 TB data for one 80-hour run, translating to ~21.6 Tb/day.
GeneMind, founded in Shenzhen, developed the SURFSeq Q, which outputs 14 TB in 36 hours, or over 9.3 Tb/day.
Then there’s Illumina's flagship NovaSeq X Plus at 8 Tb/ day.
1
0
1
302
(4/11)
There are many flavors of chemistries with various accuracy scores. We walk through the latest ones:
- Oxford Nanopore Q20+ Kit14 (2024) - Q30+ duplex:
Duplex sequencing reads both DNA strands through the same nanopore using hairpin adapters, jumping from Q15-Q18 simplex while keeping ultra-long reads.
- PacBio HiFi Chemistry (2019) - Q30-Q40: DNA polymerase reads in circles. SMRTbell templates loop fragments so the enzyme passes over the same sequence 10-20 times, averaging observations into Q30-Q40 consensus sequences at 10-25kb lengths.
- PacBio SPRQ Chemistry (2024) - Q30-Q40: Multi-omics chemistry: transposase inserts 6mA adapters into open chromatin, then HiFi detects sequence plus polymerase kinetics for DNA, chromatin accessibility, and methylation in one run.
- Element Cloudbreak UltraQ (2024) - Q50+: Rolling circle amplification creates dense clusters of identical DNA copies. UltraQ averages signals across thousands of copies per cluster, achieving Q50+ accuracy, one error in 100,000 bases.
- Illumina XLEAP-SBS (2023) - Q40+: Chemistry X overhauls sequencing-by-synthesis with thermostable polymerase and optimized terminators. Result are 2× faster cycles, 3× better accuracy, ≥85% of bases reaching Q40.
- Complete Genomics CoolMPS (2025) - Q30: Uses unlabeled nucleotides during incorporation, then fluorescent antibodies detect which base was added. Indirect detection reduces chemical scarring for cleaner 2×300bp reads.
- Ultima ppmSeq (2025) - Q30-Q40: Spins silicon wafers with billions of nanowells, flowing one nucleotide type at a time. Sequences both DNA strands independently and cross-validates to correct errors.
- Singular Direct-Seq (2025) - Q30-Q40: Sequencing-by-synthesis directly on tissue sections without RNA extraction. Reveals gene expression, proteins, and variants while preserving spatial context in one run.
There are many flavors of chemistries with various accuracy scores. We walk through the latest ones:
- Oxford Nanopore Q20+ Kit14 (2024) - Q30+ duplex:
Duplex sequencing reads both DNA strands through the same nanopore using hairpin adapters, jumping from Q15-Q18 simplex while keeping ultra-long reads.
- PacBio HiFi Chemistry (2019) - Q30-Q40: DNA polymerase reads in circles. SMRTbell templates loop fragments so the enzyme passes over the same sequence 10-20 times, averaging observations into Q30-Q40 consensus sequences at 10-25kb lengths.
- PacBio SPRQ Chemistry (2024) - Q30-Q40: Multi-omics chemistry: transposase inserts 6mA adapters into open chromatin, then HiFi detects sequence plus polymerase kinetics for DNA, chromatin accessibility, and methylation in one run.
- Element Cloudbreak UltraQ (2024) - Q50+: Rolling circle amplification creates dense clusters of identical DNA copies. UltraQ averages signals across thousands of copies per cluster, achieving Q50+ accuracy, one error in 100,000 bases.
- Illumina XLEAP-SBS (2023) - Q40+: Chemistry X overhauls sequencing-by-synthesis with thermostable polymerase and optimized terminators. Result are 2× faster cycles, 3× better accuracy, ≥85% of bases reaching Q40.
- Complete Genomics CoolMPS (2025) - Q30: Uses unlabeled nucleotides during incorporation, then fluorescent antibodies detect which base was added. Indirect detection reduces chemical scarring for cleaner 2×300bp reads.
- Ultima ppmSeq (2025) - Q30-Q40: Spins silicon wafers with billions of nanowells, flowing one nucleotide type at a time. Sequences both DNA strands independently and cross-validates to correct errors.
- Singular Direct-Seq (2025) - Q30-Q40: Sequencing-by-synthesis directly on tissue sections without RNA extraction. Reveals gene expression, proteins, and variants while preserving spatial context in one run.
1
0
2
397