Оценка LLM на академических бенчмарках

★ 7.7 · testing

lm-evaluation-harness is a Claude Code skill that evaluates language models across 60+ academic benchmarks, including MMLU (57 subjects), HumanEval (164 Python problems), GSM8K, TruthfulQA, and HellaSwag. It supports HuggingFace models, vLLM, and APIs — including quantized (4-bit/8-bit) checkpoints with automatic batch size detection. Results are saved as JSON with per-task accuracy and standard error, making it straightforward to generate comparison tables or plot learning curves across training steps. An industry standard adopted by EleutherAI, HuggingFace, and major labs, the skill fits researchers and engineers who need reproducible benchmark numbers for model releases or training runs. Dependencies: lm-eval, transformers, vllm.

• #llm-evaluation
• #benchmarking
• #mmlu
• #humaneval
• #model-comparison