Distributed Multimodal Dataset Curation & Deduplication at Scale
Client identifiers anonymized under strict Non-Disclosure Agreements (NDAs). Performance metrics reflect architectural benchmark simulations.
About the Project
This infrastructure was built for a legal analytics and enterprise consulting firm preparing to pre-train a proprietary domain-specific language model. The raw data lake contained 45 million historical documents spanning SEC filings, contract archives, legal opinions, balance sheets, and research briefs.
The engineering mandate was to transform petabytes of noisy, semi-structured files into a pristine, deduplicated token stream suitable for pre-training, stripping boilerplate, corrupted OCR artifacts, and duplicated syndicated text in under 72 hours of cluster compute.
Challenges We Faced
1. 38% Redundancy from Syndicated Boilerplate & Disclaimers
Raw enterprise document archives are bloated with duplicated text: identical legal disclaimers, repeated headers, email thread quoting, and syndicated whitepapers. Training on repetitive tokens wastes GPU compute and leads to severe model memorization.
2. Corrupted OCR Scans & Unicode Artifacts
Decades of scanned microfiches and legacy PDFs contained garbled Unicode sequences, broken ligatures, and fragmented hyphenations that distorted token distributions and degraded subword tokenizer vocabularies.
3. Petabyte-Scale Compute Bottlenecks
Running single-node Python cleaners across 45 million documents was projected to take 14 months of runtime. The pipeline required a fault-tolerant, horizontally scalable compute cluster capable of streaming data directly from object storage without disk I/O bottlenecks.
Solution Architecture
Acadify engineered a distributed data processing pipeline on Apache Spark and Ray, delivered through our product engineering practice.
We implemented a two-stage deduplication strategy combining 13-gram MinHash with Locality Sensitive Hashing (LSH) for document-level deduplication, followed by suffix array string matching to identify and strip recurring boilerplate paragraphs across documents.
- Distributed Ray Processing Cluster: Deployed Ray workers across 32 AWS EC2 spot instances, streaming data in memory using zero-copy Apache Arrow buffers to eliminate intermediate disk writes.
- Two-Tier MinHash LSH Deduplication: Processed document fingerprints through MinHash with Jaccard similarity thresholds of 0.85, eliminating 17 million near-duplicate records while preserving rare domain-specific documents.
- Heuristic & Perplexity Filtering: Implemented a multi-tier filter combining fastText language identification, KenLM perplexity scoring against authoritative domain text, and symbol-to-word ratio checks to eliminate OCR garbage.
- Lineage Tracking & Parquet Partitioning: Wrote cleaned token streams to partitioned Apache Parquet files on S3 with cryptographic manifest hashes, ensuring 100% reproducible training datasets.
Why Acadify's Engineering Approach Fit the Project
Model capabilities are strictly bounded by training data hygiene. Acadify brings deep distributed systems and data engineering experience, orchestrating Ray and Spark clusters to process petabytes of unstructured text efficiently without runaway cloud compute costs.
System Architecture & Tech Stack
Related Acadify Solution Services
This case study demonstrates capabilities from Acadify Solution's Data Annotation Services and AI Development practices. For teams building training data pipelines, explore our data annotation capabilities.