Ray Data — распределённая обработка данных для ML

★ 7.7 · data

ray-data is a Claude Code skill that enables scalable distributed data processing for ML and AI workloads with streaming execution across CPU and GPU resources. It reads Parquet, CSV, JSON, images, audio, and video from cloud storage or local sources, with dependencies on ray[data], pyarrow, and pandas. The skill integrates with PyTorch, TensorFlow, HuggingFace, and Ray Train, supporting batch inference pipelines, multi-step preprocessing, and distributed ETL that scales from a laptop to hundreds of cluster nodes. Core operations include map_batches, filter, groupby, repartition, GPU-accelerated transforms, and streaming iteration over datasets larger than available memory. Ideal for ML engineers and data engineers working with datasets over 100 GB who need cluster-level parallelism without rewriting existing Python logic.