Steven Xia retweeted
Starting a new blog with my colleagues at CMU about the future of software engineering and SE education in an AI world, called "The Last Software Engineer?" My first article on the possible end code review and why this might not be so bad: thelastsoftwareengineer.substack.com/p/the-end-of-cโฆ
0
17
106
6,759
Steven Xia retweeted
๐๐ ๐๐ง๐ญ๐ข๐ ๐๐จ๐๐ข๐ง๐ ๐ญ๐จ๐จ๐ฅ๐ฌ ๐ก๐๐ฅ๐ฉ ๐ฌ๐ก๐ข๐ฉ ๐๐จ๐๐ ๐๐๐ฌ๐ญ๐๐ซ, ๐๐ฎ๐ญ ๐๐ญ ๐ฐ๐ก๐๐ญ ๐๐จ๐ฌ๐ญ?
๐๐ก๐๐ญ ๐ฐ๐ ๐๐ข๐:
๐ฏAcross 182 open source projects, we tracked >30K commits from agentic coding tools to measure the true cost of adoption.
๐งต (1/6)
๐๐ก๐๐ญ ๐ฐ๐ ๐๐ข๐:
๐ฏAcross 182 open source projects, we tracked >30K commits from agentic coding tools to measure the true cost of adoption.
๐งต (1/6)
1
3
5
746
Steven Xia retweeted
Introducing ๐ฆ๐๐ฎ๐ฟ๐บ๐ฅ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต: agent skills that orchestrate a population of independent research agents for autonomous discovery.
SwarmResearch achieves over 2ร the throughput of vanilla autoresearch when designing speculative decoding implementations. While refining locally optimal approaches, SwarmResearch continues to explore diverse approaches after 11 hours.
Repo: github.com/SwarmResearch/โฆ
SwarmResearch achieves over 2ร the throughput of vanilla autoresearch when designing speculative decoding implementations. While refining locally optimal approaches, SwarmResearch continues to explore diverse approaches after 11 hours.
Repo: github.com/SwarmResearch/โฆ
1
5
16
1,490
Steven Xia retweeted
Weโre happy to release ๐๐๐-๐๐๐ง๐๐ก ๐๐ซ๐จ: a benchmark for measuring the bug-hunting capabilities of AI agents in critical software systems such as Chromium V8, Firefox SpiderMonkey, and more.
Explore the details here: sec-bench.github.io
Explore the details here: sec-bench.github.io
0
3
4
514
Steven Xia retweeted
๐คฏ๐คฏ๐คฏ Gemini 3 Pro + Live-SWE-agent hits 77.4% on SWE-bench Verified, beating ALL existing models, including Claude 4.5!!
๐ค Live-SWE-agent is the first live software agent that autonomously self-evolves on the fly โ and it even outperforms the manually engineered scaffold o team (76.2%)
๐ค Live-SWE-agent is the first live software agent that autonomously self-evolves on the fly โ and it even outperforms the manually engineered scaffold o team (76.2%)
31
66
470
115,303
Steven Xia retweeted
๐ Introducing Live-SWE-agent:
๐ค An autonomous coding agent that self-evolves on the fly while solving real-world issues.
โจ Simplistic design: no offline training, no heavy workflows.
๐ Surprising gains:
๐ 75.4% on SWE-bench Verified
๐ 45.8% on SWE-Bench Pro (new SOTA!)
๐ค An autonomous coding agent that self-evolves on the fly while solving real-world issues.
โจ Simplistic design: no offline training, no heavy workflows.
๐ Surprising gains:
๐ 75.4% on SWE-bench Verified
๐ 45.8% on SWE-Bench Pro (new SOTA!)
3
12
43
4,914
Steven Xia retweeted
We're releasing a new iteration of SWE-bench, in collaboration with the original authors, to more reliably evaluate AI models on their ability to solve real-world software issues. openai.com/index/introducโฆ
355
392
2,646
830,761
Steven Xia retweeted
I'm on the way to USENIX Security'24. We will present one paper about privacy-preserving app authentication. I'm also happy to discuss TEE-based AI security and other AI-related security topics. If you're interested, please get in touch with me! #usenix #USESEC2024
0
1
3
1,025
Steven Xia retweeted
Introducing OpenAutoCoder-Agentless๐บ:
A simple agentless solution solves 27.3% GitHub issues on SWE-bench Lite with ~$0.34 each, outperforming all open-source AI SW agents!
It's fully open-source, try it out:
๐งโ๐ปgithub.com/OpenAutoCoder/โฆ
๐huggingface.co/papers/2407.01โฆ
A simple agentless solution solves 27.3% GitHub issues on SWE-bench Lite with ~$0.34 each, outperforming all open-source AI SW agents!
It's fully open-source, try it out:
๐งโ๐ปgithub.com/OpenAutoCoder/โฆ
๐huggingface.co/papers/2407.01โฆ
18
128
621
97,305
Steven Xia retweeted
Magicoder: Source Code Is All You Need
paper page: huggingface.co/papers/2312.02โฆ
introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B odels are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate high-quality instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs by empowering them with a wealth of open-source references for the production of more diverse, realistic, and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks, including Python text-to-code generation, multilingual coding, and data-science program completion. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1). Overall, OSS-Instruct opens a new direction for low-bias and high-quality instruction tuning using abundant open-source references.
paper page: huggingface.co/papers/2312.02โฆ
introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B odels are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate high-quality instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs by empowering them with a wealth of open-source references for the production of more diverse, realistic, and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks, including Python text-to-code generation, multilingual coding, and data-science program completion. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1). Overall, OSS-Instruct opens a new direction for low-bias and high-quality instruction tuning using abundant open-source references.
6
174
805
127,560
Steven Xia retweeted
In the past 6-mon release of HumanEval+ we have been improving its toolchain usability and dataset quality from v0.1.0 to v0.1.7 releases.
๐ฅ Now we release MBPP+, a new benchmark in EvalPlus v0.2.0: tinyurl.com/4pw82wb8
๐งต
๐ฅ Now we release MBPP+, a new benchmark in EvalPlus v0.2.0: tinyurl.com/4pw82wb8
๐งต
2
7
50
7,756
Steven Xia retweeted
Introducing the EvalPlus leaderboard! evalplus.github.io/leaderboard.htโฆ
๐ฅ28 models have been evaluated on coding HumanEval & HumanEval+
๐ฅ7B CodeLlama outperforms ~16B models e.g. StarCoder&CodeGen
๐ฅPhind-CodeLlama-34B-v2 and WizardCoder-Python-34B-V1 as open models both beat ChatGPT
๐งต
๐ฅ28 models have been evaluated on coding HumanEval & HumanEval+
๐ฅ7B CodeLlama outperforms ~16B models e.g. StarCoder&CodeGen
๐ฅPhind-CodeLlama-34B-v2 and WizardCoder-Python-34B-V1 as open models both beat ChatGPT
๐งต
4
17
140
40,439
Steven Xia retweeted
OK now we have @steven_xia_ on his work with @YuxiangWei9 and @LingmingZhang, all of @plfmse. Talking about program repair using large pre-trained language models
2
6
14
3,507
Steven Xia retweeted
We welcome everyone to try out ๐๐๐ฎ๐ฆ๐๐ง๐๐ฏ๐๐ฅ+! A dataset to reflect the "real" correctness of LLM-generated code.
Using๐๐๐ฎ๐ฆ๐๐ง๐๐ฏ๐๐ฅ+ is the same as HumanEval. You can easily pip install it and evaluate in our prepared sandbox (optional). github.com/evalplus/evalpโฆ
Using๐๐๐ฎ๐ฆ๐๐ง๐๐ฏ๐๐ฅ+ is the same as HumanEval. You can easily pip install it and evaluate in our prepared sandbox (optional). github.com/evalplus/evalpโฆ
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
extensive evaluation across 14 popular LLMs (including GPT-4 and ChatGPT) demonstrates that HUMANEVAL+ is able to catch significant amounts of previously undetected by LLMs, reducing the pass@k by 15.1% on average! For example, the pass@k of widely studied open-source models like CODEGEN-16B can drop by over 18.0%, while the performance of state-of-the-art commercial models like ChatGPT and GPT-4 can also drop by at least 13.0%, largely affect the result analysis for almost all recent work on LLM-based code generation
abs: arxiv.org/abs/2305.01210
github: github.com/evalplus/evalpโฆ
2
17
103
14,985
Steven Xia retweeted
๐จ Evaluating LLM-generated code on datasets with just "3 test-cases" is NOT enough! ๐จ
We built โจHumanEval+โจ: improving HumanEval with up to thousands of new tests to fully evaluate functional correctness of LLM generated code!
@JiaweiLiu_ @YuyaoStarling @LingmingZhang
We built โจHumanEval+โจ: improving HumanEval with up to thousands of new tests to fully evaluate functional correctness of LLM generated code!
@JiaweiLiu_ @YuyaoStarling @LingmingZhang
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
extensive evaluation across 14 popular LLMs (including GPT-4 and ChatGPT) demonstrates that HUMANEVAL+ is able to catch significant amounts of previously undetected by LLMs, reducing the pass@k by 15.1% on average! For example, the pass@k of widely studied open-source models like CODEGEN-16B can drop by over 18.0%, while the performance of state-of-the-art commercial models like ChatGPT and GPT-4 can also drop by at least 13.0%, largely affect the result analysis for almost all recent work on LLM-based code generation
abs: arxiv.org/abs/2305.01210
github: github.com/evalplus/evalpโฆ
1
12
39
11,192
๐จ Evaluating LLM-generated code on datasets with just "3 test-cases" is NOT enough! ๐จ
We built โจHumanEval+โจ: improving HumanEval with up to thousands of new tests to fully evaluate functional correctness of LLM generated code!
@JiaweiLiu_ @YuyaoStarling @LingmingZhang
We built โจHumanEval+โจ: improving HumanEval with up to thousands of new tests to fully evaluate functional correctness of LLM generated code!
@JiaweiLiu_ @YuyaoStarling @LingmingZhang
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
extensive evaluation across 14 popular LLMs (including GPT-4 and ChatGPT) demonstrates that HUMANEVAL+ is able to catch significant amounts of previously undetected by LLMs, reducing the pass@k by 15.1% on average! For example, the pass@k of widely studied open-source models like CODEGEN-16B can drop by over 18.0%, while the performance of state-of-the-art commercial models like ChatGPT and GPT-4 can also drop by at least 13.0%, largely affect the result analysis for almost all recent work on LLM-based code generation
abs: arxiv.org/abs/2305.01210
github: github.com/evalplus/evalpโฆ
1
12
39
11,192
Compared to the original HumanEval, our results on our new dataset show that on average, the pass@K performance of current popular state-of-the-art LLMs drops by ๐15% including both ChatGPT and GPT-4.
1
1
3
992
We have released our dataset as well as our code at: github.com/evalplus/evalpโฆ
Furthermore, we also included all studied LLM-generated code at: github.com/evalplus/evalpโฆ
โจ x 14 models (10 model types)
โจ x 5 temperature settings
โจ x 200 code samples
Furthermore, we also included all studied LLM-generated code at: github.com/evalplus/evalpโฆ
โจ x 14 models (10 model types)
โจ x 5 temperature settings
โจ x 200 code samples
0
0
4
380