ORCA-bench: How Ready Are Language Model Agents for Oncall?
- We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting.
- ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and
Unverified
- We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting.
- ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and
Sources: Arxiv