large-scale-data-processing

Here are 3 public repositories matching this topic...

Language: All

Filter by language

NVIDIA-NeMo / Curator

Scalable data pre processing and curation toolkit for LLMs

python data data-processing data-preparation deduplication data-quality data-curation data-prep fine-tuning fast-data-processing data-processing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes semantic-deduplication llm-data-quality

Updated Mar 10, 2026
Python

data-prep-kit / data-prep-kit

Star 906

Open source project for data preparation for GenAI applications

python data spark malware code-quality data-preprocessing ray data-preparation deduplication data-prep finetuning data-preprocessing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes

Updated Mar 2, 2026
HTML

EN : Water Pipe Leakage Risk Prediction System v0-3 | Industrial-grade GPR system achieving R2=0.9894 accuracy with 1,907x parallel processing efficiency | Complete scalability proven from N=44 to N=6000 datasets. JP : Shui Dao Guan Lou Shui risukuYu Ce GPRshisutemu v0-3 | Chan Ye reberuYu Ce Jing Du R2=0.9894, 1,907Bei Bing Lie Chu Li Xiao Lu Hua woShi Xian shitagausuGuo Cheng Hui Gui niyoruGao Jing Du Lou Shui risukuFen Xi shisutemu | N=44-N=6000Wan Quan suke-rabiriteiShi Zheng Ji mi