Тестирование и сравнение LLM через omniroute CLI

★ 7.9 · testing

cli-eval is a Claude Code skill that lets you create and run LLM evaluation suites, watch live benchmark progress, view scorecards, and compare model performance directly from the terminal using the omniroute CLI. It covers the full eval lifecycle: defining suites with rubrics (`exact-match`, `contains`, `llm-judge`, or `regex`) via a JSONL samples file, running them against specific models with `omniroute eval suites run <suiteId> --model <id> --watch` for a live TUI dashboard, and inspecting per-sample pass/fail results through `eval scorecard`. Comparing multiple models — Claude, GPT-4o, Gemini — is done by looping the same suite across targets and diffing the returned scores. CI integration is built around a score-threshold check that exits with a non-zero code when quality drops below an acceptable level, making it straightforward to gate deployments on eval results. The skill is aimed at teams who need automated regression testing for language models and continuous quality monitoring across releases.