Beyond human votes, we release two new benchmarks:

- BigCodeReward: tests reward models on 4.7K human preference votes. Execution feedback improves judgment accuracy.

- AutoCodeArena: automated evaluation of 20+ LLMs. GPT-5 leads, followed by Claude-Opus-4 and Claude-Sonnet-4.

In 5 months, BigCodeArena collected 14K conversations & 4.7K preference votes across 10 frontier LLMs.

Findings:
1. o3-mini & o1-mini consistently top Elo rankings
2. Claude-3.5-Sonnet excels in matched-language settings
3. Previous open models still lag

Why does this matter?

Benchmarks like HumanEval only scratch the surface. Reading code “by eye” is error-prone. True quality emerges when you actually run it: web apps render, games play, edge cases break.

BigCodeArena makes execution feedback the default.

Introducing BigCodeArena, a human-in-the-loop platform for evaluating code through execution.

Unlike current open evaluation platforms that collect human preferences on text, it enables interaction with runnable code to assess functionality and quality across any language.

Releasing BigCodeBench-Hard: a subset of more challenging and user-facing tasks.

BigCodeBench-Hard provides more accurate model performance evaluations and we also investigate some recent model updates.

Read more: huggingface.co/blog/terryyz/b…
Leaderboard: huggingface.co/spaces/bigcode…

Introducing 🌸BigCodeBench: Benchmarking Large Language Models on Solving Practical and Challenging Programming Tasks!

BigCodeBench goes beyond simple evals like HumanEval and MBPP and tests LLMs on more realistic and challenging coding tasks.

Releasing StarCoder2 Instruct! 🚀

Achieves 72% HumanEval score using only self-generated content without any GPT-3.5/4 data. This work demonstrates that self-instruct works already well at the 15B scale without data from proprietary models!

Read more: huggingface.co/blog/sc2-instr…

This is the result of hard work by the BigCode community and supported by @ServiceNowRSRCH, @huggingface and @nvidia to train the 3B, 7B and 15B models!

The Stack v2 was built with @SoftwareHeritage and the full processed training data is coming soon!

huggingface.co/datasets/bigco…

StarCoder2 performs well on a wide range of coding and math tasks. The 15B model is best in its class, while the 3B model is at the performance of StarCoder1-15B.

This makes StarCoder2 models more efficient and performant!

Read the full report: hf.co/bigcode/report

Introducing: StarCoder2 and The Stack v2 ⭐️

StarCoder2 is trained with a 16k token context and repo-level information for 4T+ tokens. All built on The Stack v2 - the largest code dataset with 900B+ tokens.

All code, data and models are fully open!

hf.co/bigcode/starco…

We just added it along with CodeGeeX-2

@nomic_ai team already added support for StarCoderBase-3B in their GPT4ALL local models.

Download the model at: gpt4all.io/models/starcod…
& follow the docs: docs.gpt4all.io/gpt4all_python…

Stay tuned for the 7B model integration!

StarCoder Models generate more tokens per second for a single batch, but things getter more fun when we increase the batch size thanks to MQA 🚀

The sharing of values/keys among attention heads significantly reduces the memory bandwidth requirements.

📊 We are also releasing a leaderboard showcasing the performance of base multilingual code models. We additionally compare throughputs so you don't have to trade performance for efficiency:

huggingface.co/spaces/bigcode…

💫 Each model demonstrates the strongest performance for its size across various programming languages. 7B StarCoderBase reaches 28.37% pass@1 on HumanEval.

7B: huggingface.co/bigcode/starco…
3B: huggingface.co/bigcode/starco…
1B: huggingface.co/bigcode/starco…

🌌 News from the StarCoder cosmos!

We trained smaller versions of StarCoder: 1B, 3B and 7B models.

1T tokens, 80+ programming languages with 8k context window, MQA & FIM.

You can watch the full online session on how StarCoder was built and where BigCode might head next here:

youtube.com/watch?v=sQFWE_…

It can build HTML websites and much more...

Give it a try 🚀

We instruction-tuned StarCoder+ on the OpenAssistant Guanaco dataset to get StarChat-beta: a strong chat assistant

Model: huggingface.co/HuggingFaceH4/…
Demo: huggingface.co/spaces/Hugging…

The result: ⭐ StarCoder+ a powerful English Language Model with strong coding abilities.

It outperforms all LLaMa models and PaLM-540B on HumanEval and stands out in the LLM leaderboard for < 30B models with a 45.1 MMLU score!

huggingface.co/spaces/Hugging…

📣 Introducing ⭐ StarCoder+ & StarChat Beta!

We trained StarCoder on the Falcon model's English web dataset and Instruction-tuned it. Both models rank high in the LLM leaderboard, with strong natural language performance and coding capabilities.

huggingface.co/HuggingFaceH4/…

👀 A glimpse of our latest mystery model's performance. Not just acing the coding tasks, but also mastering natural language! Intrigued yet?

Join us at our StarCoder webinar this Thursday to find out: servicenow.zoom.us/j/99103734103

🛡️ We provide Docker containers for improved security and reproducible execution. They are especially helpful for certain languages like Bash to avoid executing commands on your system.

+ Execution works out of the box for all supported programming languages.

🚀 We leverage data parallelism using accelerate library to run the generations across many GPUs. Generating full HumanEval solutions for StarCoder 15B with 50 generations per problem takes less than 15 minutes on 2 A100s 80GB.

You also want to be able to separate generation from execution, the first part usually requires GPUs, while the second only needs CPUs.

Similarly to lm-evaluation-harness, we build a framework combining several benchmarks:

- HumanEval, MBPP & APPS
- MultiPL-E (HumanEval translated to 18 programming languages)
- DS-1000
- PaL for GSM8K
...

Introducing the BigCode Evaluation Harness for Code LLMs:

github.com/bigcode-projec…

Inspired by the lm-evaluation-harness from @AiEleuther, it ensures ease-of-use, reproducibility and efficiency. Let’s explore its key features 🧵:

For example the folks at @refact_ai are working on a shiny VSCode extension that can now make use of StarCoder to autocomplete or refactor code as well as writing code from an instruction!

refact.ai/blog/2023/self…

StarCoder was also trained on JupyterNotebooks and with Jupyter plugin from @JiaLi52524397 it can make use of previous code and markdown cells as well as outputs to predict the next cell.

You can install it here or search on chrome store: github.com/bigcode-projec…

We present the most extensive evaluation of code LLMs to date in the full tech report with 68 (!) authors.

You can also read up on all the details from data preprocessing and governance to training at scale!

drive.google.com/file/d/1cN-b9G…

Today we release two open-access models!

StarCoderBase: trained on 1T tokens in 80+ programming languages huggingface.co/bigcode/starco…

StarCoder: additionally trained on 35B Python tokens that can be prompted to reach 40.8% pass@1 huggingface.co/bigcode/starco…

In addition to chatting with StarCoder, it can also help you code in the new VSCode plugin. By pressing CTRL+ESC you can also check if the current code was in the pretraining dataset!

marketplace.visualstudio.com/items?itemName…

Introducing: 💫StarCoder

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. It can be prompted to reach 40% pass@1 on HumanEval and act as a Tech Assistant.

Try it here: shorturl.at/cYZ06r

Release thread🧵

Day 18: Weather is clear and the loss is still going down ...

We started training something big and the daily training updates have degenerated to weather reports 🌦:

We also worked on the governance side and simplified the opt-out process:

Check "Am I In The Stack" (hf.co/spaces/bigcode…) to see if your data is in the current version of The Stack and click the link at the bottom to create an opt-out request directly.

It is also interesting to analyze repositories in terms of GitHub stars, number of forks or issues (e.g. to build custom filters). The metadata dataset contains that information and can be joined with the code dataset.

huggingface.co/datasets/bigco…

Exciting new release: The Stack v1.2 with many new features!
- The Stack Issues
- The Stack Metadata
- The Stack Commits (coming soon!)

Along with a new and simplified opt-out request and an even better near-deduplicated version of the code dataset.
hf.co/bigcode

🧵👇

Finally, we summarized our findings in a technical report with a wonderful group of collaborators:

Paper: hf.co/datasets/bigco…

So what's next? Scaling to larger models and training more languages early next year! 🚀

We release all models and intermediate checkpoints on the Hugging Face Hub and load the via the revision:

huggingface.co/bigcode/santac…

The compute for these experiments was sponsored by @ServiceNowRSRCH's research cluster.

With these new insights we trained a final model called SantaCoder. We applied both the extra near-deduplication and code-to-comment ratio filters and trained for 600K steps (236B tokens). The result is an efficient (MQA) and flexible (FIM) multilingual model:

The thorough evaluation was made possible by the evaluation working group. They evaluated on the MultiPL-E benchmark (multilingual HumanEval and MBPP) and CodeXGLUE and extended the code evaluation harness!

github.com/bigcode-projec…

In addition to the standard near-deduplication and heuristics pipeline, we ran 4 filtering experiments: GitHub stars, tokenizer fertility, comment-to-code ratio and more near-deduplication.

Filtering for GitHub stars hurts performance while comments and near-dedup help!

Fill-in-the-Middle is a clever approach where a sequence is reordered such that prefix|middle|suffix becomes prefix|suffix|middle. With that you can use normal left-to-right generation to fill the middle part.

While some claim FIM is for free we found it's rather FIM for cheap:

In Multi Head Attention every head has a set of queries, keys, and values. In MQA, the queries are unique while keys and values are shared. This saves memory and speeds up inference for large batches. We found it had only a minor impact on performance:

Before training any models we looked into removing sensitive information from code such as email addresses, secret keys and IP addresses. For that purpose we annotated 400 samples and then built and continuously refined RegEx rules to remove the information before training.

SantaCoder is trained on Python, Java, and JavaScript and outperforms other large multilingual models such as InCoder (6.7B) or CodeGen-multi (2.7B) considerably!

A lot of pieces from a lot of collaborators came together to get to that result:

Announcing a holiday gift: 🎅SantaCoder - a 1.1B multilingual LM for code that outperforms much larger open-source models on both left-to-right generation and infilling!

Demo: hf.co/spaces/bigcode…
Paper: hf.co/datasets/bigco…
Attribution: hf.co/spaces/bigcode…

A🧵:

It was a blast to meet some of the BigCode community last night in person. Thank you to Juan’s Flying Burrito and team for helping to make the event a success.

Today we are releasing The Stack v1.1! 🚀

We added more data, included more programming languages, and extended the list of permissive licenses used.

huggingface.co/datasets/bigco…

Also the first batch of opt-out requests was removed from the dataset.

Between now and Christmas🎄 we are running a series on experiments to figure out what the best pre-processing is for code datasets such as The Stack. We'll share the W&B dashboards of these 🎅-models so if you are interested you can follow along!

Roughly 235B tokens (2048 seq x 300k steps x 384 bs):

Real fun facts: The dataset contains 1.5M files that include "hello world". It is also 150x larger than English Wikipedia and if printed (double sided) it would almost reach 11x the height of the Mount Everest.

For easy experimentation we set up a small subset with 10k files per programming language here: hf.co/datasets/bigco…

Usage: There are several ways to use the dataset. You can download it with git:

git lfs install
git clone huggingface.co/datasets/bigco…

Or you can use the 🤗Datasets library to load the dataset. With the streaming this requires no disk space at all!

Governance: For over 80% of all repos a license couldn't be detected and were thus excluded. The rest was filtered for permissive licenses.

Devs might not be comfortable that their code is used to train such models. We are working on an opt-out process:
bigcode-project.org/docs/about/the…

Evaluation: to see if the dataset is useful to train generative models for code we trained a 350M parameter GPT-2 model on several datasets: training without license filtering performs best. Interestingly, removing near duplicates improves the performance on all datasets!

The dataset includes ~30 programming languages covering common languages such as Java, C/C++ and Python as well as lower resource languages (2GB of Dockerfiles 🐳). If you'd like to see a new language added, feel free to add it in this issue:

github.com/orgs/bigcode-p…

Dataset collection: With gharchive.org over 220M repos were identified and 137M successfully cloned with over 50B files and 90TB of data. Filtered by extension and permissive licenses this yields 3TB of data. We also make a near-deduplicated version (1.5TB) available.

Introducing 📑 The Stack - a 3TB dataset of permissively licensed code in 30 programming languages.

hf.co/datasets/bigco…

You want your code excluded from the model training? There is an opt-out form and data governance plan:

bigcode-project.org/docs/about/the…

Let's take a tour🧵

print("Hello world! 🎉")

Excited to announce the BigCode project led by @ServiceNowRSRCH and @huggingface! In the spirit of BigScience we aim to develop large language models for code in an open and responsible way.

Join here: bigcode-project.org/docs/about/joi…

A thread with our goals🧵