The Role
We’re hiring a Pre-Training Data & Acquisition Engineer to build the data systems powering Prometheus’s foundation models for the physical world. You’ll work closely with research and infrastructure teams to acquire, process, and deliver large-scale training datasets across engineering, scientific, and multimodal domains. This role spans distributed crawling, source integration, data processing, and production operation, with end-to-end ownership from raw content to training-ready datasets.
What You’ll Do
Identify and integrate valuable data sources across engineering and scientific domains.
Build distributed crawlers, API integrations, and bulk ingestion systems with effective scheduling, rate limiting, retries, and incremental updates.
Develop pipelines for parsing, extraction, normalization, deduplication, quality filtering, and tokenization across heterogeneous formats.
Optimize throughput and cost across networking, compute, storage, and databases as acquisition and processing workloads scale.
Build monitoring and tooling to track source coverage, ingestion failures, processing throughput, and usable data yield.
Work closely with pre-training researchers to translate data requirements into reliable pipelines and deliver datasets ready for large-scale training.
Own dataset reproducibility, versioning, provenance, and recovery from acquisition through delivery.
What We’re Looking For
Experience building and operating large-scale distributed systems, web crawlers, or data processing pipelines.
Strong programming ability in Python and Rust, Go, C++, or a comparable systems language.
A practical understanding of web infrastructure, including HTTP, DNS, concurrency, caching, and common failure modes.
Hands-on experience with databases, object storage, and distributed batch or streaming processing.
Experience designing fault-tolerant systems that handle partial failures, resume interrupted work, and prevent unintended duplication or data loss.
Ability to profile and debug performance across CPU, memory, disk, and network usage.
Strong technical judgment when integrating unfamiliar sources, APIs, and file formats.
Bias toward fast iteration and end-to-end ownership, from initial implementation through reliable production operation.
Experience with search indexing, document extraction, or foundation-model data pipelines is a plus.
Why Join Us
Work with world-class researchers on frontier AI systems for the physical world.
Build the acquisition and processing systems that supply engineering, scientific, and multimodal data to large-scale model training.
Competitive compensation and flexible work arrangements.
High-impact, mission-driven environment.
