Nigam Shah retweeted
1/🧵How do we know if AI is actually ready for healthcare? We built a benchmark, MedHELM, that tests LMs on real clinical tasks instead of just medical exams. #AIinHealthcare
Blog, GitHub, and link to leaderboard in thread!
Blog, GitHub, and link to leaderboard in thread!
9
72
335
60,721
Superb tweet chain by @jasonafries reg the work on medalign.stanford.edu and why it matters. Check out the series 👇
LLMs have made impressive progress on medical benchmarks like MedQA since 2021. Both open and closed medical LLMs have focused a lot on MedQA performance, but this has a number of problems when we think about deploying LLMs in hospitals. /1
1
5
32
5,309
Nigam Shah retweeted
Excited to be in Vancouver 🇨🇦 for #AAAI24 with @_scott_fleming_ to present:
"MedAlign: A Clinician-Generated Benchmark Dataset for Instruction Following with Electronic Medical Records"
If you are interested in healthcare LLMs and preference alignment, checkout our talk and poster today (2/22)
🎤 Oral (presented by @_scott_fleming_): Thursday 3:45PM Room 217
🖼️ Poster Session: Thursday 7-9PM West Exhibit Hall
Website: medalign.stanford.edu
Paper: arxiv.org/abs/2308.14089
"MedAlign: A Clinician-Generated Benchmark Dataset for Instruction Following with Electronic Medical Records"
If you are interested in healthcare LLMs and preference alignment, checkout our talk and poster today (2/22)
🎤 Oral (presented by @_scott_fleming_): Thursday 3:45PM Room 217
🖼️ Poster Session: Thursday 7-9PM West Exhibit Hall
Website: medalign.stanford.edu
Paper: arxiv.org/abs/2308.14089
0
8
52
7,723
It was fun to summarize lessons learned from research in partnership with @StanfordHAI, @StanfordCRFM, @StanfordMed for our clinical colleagues. We have to verify the claimed value propositions (hai.stanford.edu/news/how-found…) because they don't always pan out (hai.stanford.edu/news/how-well-…)!
Special Communication discusses whether large language models (LLMs) are being trained with the right kind of self-supervision and if the purported value propositions of using LLMs in medicine are being verified. ja.ma/3KxDT0S
0
15
79
24,139
Thank you @katieelink for sharing this work. In the spirit of the HELM project, crfm.stanford.edu/helm/latest/ @StanfordHAI it would be amazing to get more focus on the need of "evaluations that matter" rather than "evaluations we can do"!
Here's the link to read more:
hai.stanford.edu/news/shaky-fou…
Thanks so much to the authors including @drnigam @birjupatel @StanfordHAI for this extensive and thoughtful analysis!
And thank you @BrigitteTousi for sharing this post with me 🤗
0
0
10
2,812
You have probably heard of #chatGPT, which is a class of models called "large language models" or Foundation Models .. what else could such models be useful for? Check out below ..
In healthcare, the transition from development to deployment of AI is rife with challenge. Can foundation models help? Scholars outline key opportunities: stanford.io/3GLezSt
1
2
12
3,937
Nigam Shah retweeted
Come present your work at SAIL 2023 - Rio Grande, Puerto Rico from May 9th to 12th.
Call for abstracts and travel support applications will be open until January 20th.
sail.health/call_for_paper…
Call for abstracts and travel support applications will be open until January 20th.
sail.health/call_for_paper…
0
3
5
Nigam Shah retweeted
#StanfordAIHealth is almost here! Join us Dec 6-7 to explore critical & emerging issues related to AI's impact across the spectrum of healthcare. Stellar program with talks, panels, firesides & live Q&A. 9 CME credits. Register: aihealth2022.stanford.edu @StanfordHAI @StanfordCME
0
14
22
Nigam Shah retweeted
“As a community, I think we’re hung up on the performance of the model and not asking the question, is the model useful? We need to think outside the model.” @drnigam, Chief Data Scientist for @StanfordHealth. stanford.io/3NXiKNk
0
2
15
Very thoughtful analysis @StanfordDeptMed by @agatafshsh, @stephenpfohl, @birjupatel on the interaction between imposing fairness constraints and practice guideline adherence. Good example of our holistic view at @StanfordMed as suggested by @StanfordHAI
Excited to share our paper on evaluating fairness of algorithms in the context of medical guidelines, written with @stephenpfohl @birjupatel and @drnigam published in @BMJ_HCI! Paper: informatics.bmj.com/content/29/1/e…. A thread:
1
4
25
Nigam Shah retweeted
Join us May 24 for AIMI Symposium 2022 as we convene in person & online to discuss key advances and future vision for AI in medicine: scientific understanding of disease, clinical applications, delivery science, open data & ethics. Register: stanford.io/3v4z86z #AIMI22
1
16
27
Superb signout by @stephenpfohl! His is one of 179 papers accepted out of 711, completing his trilogy scholar.google.com/citations?view…, scholar.google.com/citations?view…, scholar.google.com/citations?view… on #algorithmic #fairness @StanfordMed and @StanfordHAI
The last paper from my PhD, "Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare", has been accepted at FAccT! With @YizheXu6, @agatafsh, @nikosIgnatiadis, @julian_genkins, @drnigam. Preprint: arxiv.org/abs/2202.01906.
0
0
7
What are you doing on Nov 18th?
Hopefully attending SMILE, a symposium dedicated to ML implementation & evaluation!
Featuring:
@drnigam
@ctlin1
Anna Goldenberg, PhD
@shalmali_joshi_
@MarkSendak
@AnthonyChangMD
@rkamaleswaran
and more!
Register here: eventbrite.com/e/symposium-fo…
0
1
7
Nigam Shah retweeted
Join us THIS AFTERNOON for an amazing #AMIA2021 panel!
Want to learn more about how provider org's, academics, government & industry came together to respond to #covid and what comes next? Join me +
@atulbutte @CALonghurst @dramyabernethy &
@IBMWatsonHealth's Gretchen Jackson @
@AMIAinformatics #amia2021 - LB08, Tues 11/2 @ 3:30p!
0
7
20
Nigam Shah retweeted
A new platform at Stanford AIMI offers datasets at no cost, with the aim of spurring crowd-sourced AI applications in healthcare. stanford.io/3mKhm4p
0
24
53
Results in @nejmcatalyst article, catalyst.nejm.org/doi/full/10.10….
"Medicine has been dreaming about doing this for 50 years," says #StanDOM's @drnigam of a tool recently developed by @StanfordMed researchers that pulls from thousands of #EHRs to guide physicians on diagnosis & treatment. stan.md/3i3bx0i
0
7
18
Nigam Shah retweeted
Assessing the Future of #PublicHealth Data Exchange, #Interoperability ehrintelligence.com/news/assessing…
“We need to rethink public health data [as a] fundamental construct,” @donrucker said.
This involves shifting a mandated reporting mindset to a data reuse framework, he explained.
“We need to rethink public health data [as a] fundamental construct,” @donrucker said.
This involves shifting a mandated reporting mindset to a data reuse framework, he explained.
1
2
2
Thank you @NLM_NIH for supporting the #DataScience that led to the results in @nejmcatalyst. Nice example of how #informatics to drive innovations in #DigitalHealth delivery @StanfordHealth via a diverse team of @clssfr @DoctorGombar @KMorseMD @emcahan @HeartBobH + others!
To advance data-powered health, @StanfordMed researchers used aggregate patient data from electronic health records to answer physician questions & support their decisions. This NLM-funded study shows how past records can inform future patient care. catalyst.nejm.org/doi/full/10.10… #EHR
0
10
29
Nice summary @JonathanLu11's work by @StanfordHAI “Flying in the Dark”: Hospital AI Tools Aren’t Well Documented hai.stanford.edu/news/flying-da…
0
0
15
Nigam Shah retweeted
Do ML models with better AUC performance necessarily have higher utility?🤔
Our new paper assesses utility of ML models for allocating interventions to avoid unplanned readmissions.
Led by @mlko53 & @emma_ychen @ashwinagrawal15 w/ @drnigam et al.
1/n
sciencedirect.com/science/articl…
Our new paper assesses utility of ML models for allocating interventions to avoid unplanned readmissions.
Led by @mlko53 & @emma_ychen @ashwinagrawal15 w/ @drnigam et al.
1/n
sciencedirect.com/science/articl…
1
5
32