NeMo Curator — подготовка данных для обучения LLM
★ 7.7 · data
nemo-curator is a Claude Code skill that streamlines GPU-accelerated data curation with NVIDIA's NeMo Curator toolkit for building high-quality LLM training datasets. It covers the full pipeline: quality filtering with 30+ heuristics, exact and fuzzy deduplication via MinHash+LSH (16× faster than CPU on datasets like 8TB RedPajama v2), embedding-based semantic deduplication, PII redaction, and NSFW detection. All major modalities are supported — text, images, video, and audio — with near-linear multi-GPU scaling powered by RAPIDS and Dask. The skill is ideal for teams processing Common Crawl web scrapes, deduplicating large corpora, or assembling multimodal datasets ahead of model training.
- #data-curation
- #gpu-acceleration
- #deduplication
- #text-processing
- #llm-training-data
- #rapids
- #nvidia