Models currently lack the skills to achieve true software engineering automation. Emulated is a data company aiming to close the gaps in the software development loop.

We're releasing the Software Development Automation Benchmark, a benchmark focused on model abilities beyond code generation to cover the full software development lifecycle.

Read more here: emulated.so/sdab

Announcing Autoresearch Bench

A benchmark for coding agents autonomously tackling research problems

While existing coding benchmarks are saturating quickly, the difference between models in autoresearch loops is stark

Nearly all benchmarks focus on codegen-only tasks.

Emulated's environments contain running services serving real traffic.

We not only mock integrations with Slack, Linear, Sentry, Prometheus, and CI/CD, but we also emulate full cloud services like AWS, GCP, and Azure.

These let us build incredibly rich tasks for distributed databases, training clusters, and network infrastructure!