Building an 8-Stage AI Pipeline for Contact Discovery
A technical deep-dive into how Scrapll's pipeline turns a natural language query into scored, verified leads in under 3 minutes.
When we set out to build Scrapll, the core technical challenge was clear: take a natural language description like "SaaS companies in SF that raised Series A" and turn it into a list of verified contacts with emails, enrichment data, and relevance scores — all in under 3 minutes.
Here's how our 8-stage pipeline works.
Stage 1: Query Parser
The query parser extracts structured intent from natural language. It identifies target industry, company size, geography, job titles, and behavioral signals. This stage converts fuzzy human language into actionable search parameters.
Stage 2: Company Discovery
Using the parsed query, we search Google, DuckDuckGo, and a curated database of real companies across SaaS, Tech, Finance, Healthcare, AI, and E-commerce. A multi-strategy approach (search engine regex, library search, and curated fallback) ensures reliable results even when search engines rate-limit.
Stage 3: Contact Finder
For each discovered company, we find decision-makers via LinkedIn search with strict name validation — capitalized names, 2+ words, no digits. This ensures we're finding real people, not artifacts.
Stage 4: Email Verifier
Every email address is verified in real-time: MX record checks, SMTP deliverability testing, and disposable domain filtering. This is what gets us to 98% accuracy.
Stage 5: Enrichment
We enrich each contact with company intel: industry, size, revenue estimates, location, technology stack, and recent news. Failed domains are cached per pipeline run to avoid redundant requests.
Stage 6: Scorer
Each lead receives a 0-100 score based on query relevance, data completeness, company fit, and engagement signals. Hot leads (70+) float to the top.
Stage 7: Summarizer
The final stage generates a concise, human-readable summary for each lead: why they matched, what makes them a good fit, and suggested outreach angles.
The orchestrator
All seven stages are coordinated by an orchestrator that manages parallel execution, error handling, and graceful degradation. If one source is slow, the pipeline doesn't block — it returns partial results and continues enriching in the background.
The entire pipeline runs in about 3 minutes for a typical query, delivering 10-50 verified leads depending on your plan.