BigCode Evaluation Harness — бенчмаркинг моделей кодогенерации

★ 7.7 · testing

bigcode-evaluation-harness is a Claude Code skill that benchmarks code generation models across 15+ tasks — including HumanEval (164 problems), HumanEval+, MBPP, MBPP+, and MultiPL-E spanning 18 programming languages — using pass@k metrics. Built on the BigCode Evaluation Harness from the BigCode Project, it serves as the industry standard behind HuggingFace leaderboards for comparing coding LLMs. The skill supports quantized (4-bit) models, instruction-tuned models with configurable instruction tokens, Docker-based safe code execution for multi-language tasks, and JSON output with pass@1, pass@10, and pass@100 scores. It is the go-to choice for ML engineers who need to rigorously evaluate code model quality, test multilingual generation, or prepare a model submission for public leaderboards.