Key takeaways
- Prefer an API wherever one exists; scraping is a deliberate decision, not a default.
- One canonical shape with an adapter per source, and the raw payload kept alongside.
- Use a model only on prose — parsing a structured field with one is slower and worse.
- Fetch incrementally with a stored cursor, and back off properly when throttled.
- The expensive failure is silent, so alert on volume and field fill rates rather than errors.
Prefer an API to scraping, always
Where a source offers an API or a feed, use it. It is faster, it is stable, it usually comes with terms you can actually comply with, and it fails loudly rather than returning a page of markup that no longer contains what you expected.
Scraping is what you do when nothing else exists, and it should be a deliberate decision with the terms of service read rather than a default. Many job boards explicitly prohibit automated collection, and that matters more once your project has users.
Register properly and behave well on the sources you do use. A documented key, a stated user agent and respect for the published limits are what turn an integration into a relationship, and the alternative is a block that arrives without warning on the morning it matters.
The schemas will not agree
Every source models a job differently. Salary might be a range, a single number, a string, or absent. Location might be a city, a region, a code, or the word "remote" in a field meant for a city. Seniority might be structured, implied by the title, or nowhere.
So the first real component is a normaliser: one internal representation, and an adapter per source that maps into it. Keep the raw payload alongside the normalised record, because your normaliser is wrong in ways you have not discovered yet and reprocessing history is the only way to fix it.
Resist defaulting a missing field to something convenient. A salary recorded as zero, or a location defaulted to remote, silently distorts every filter downstream — and unknown is a value the rest of the system can handle correctly if you let it exist.
- One canonical job shape, adapters per source
- Raw payload stored beside the normalised record
- Explicit "unknown" rather than defaulting a missing field
- Source and fetch timestamp on every record
| Field | Variation | What it breaks |
|---|---|---|
| Salary | Range, point, text, absent, per hour | Filtering and sorting |
| Location | City, region, code, "remote" | Eligibility gating |
| Posted date | Posted, updated, re-syndicated | Freshness ranking |
| Employer | Legal name, brand, agency | Deduplication |
| Description | HTML, markdown, truncated | Requirement extraction |
| Job id | Unstable across fetches | Change detection |
Where a model helps, and where it does not
Do not use a model for what the API already gives you structured. Parsing a JSON field with an LLM is slow, costly and less reliable than reading the field.
Use it for the parts that arrive as prose: pulling required skills out of a description, inferring seniority when only a title exists, deciding whether "hybrid — 2 days" means the same as another source’s "flexible". That is genuine language work and it is where the token spend is justified.
Run that work once per posting rather than once per arrival. The same role comes through four sources, and keying the expensive extraction to a content hash means the second, third and fourth encounters cost a lookup.
Rate limits and incremental fetching
Refetching everything on every run burns quota and gets you throttled. Most APIs support fetching since a timestamp or a cursor — use it, and store the cursor so a restart resumes rather than starting over.
Back off properly when throttled. A retry loop with no delay against a rate-limited endpoint will get your key suspended, and the suspension will last longer than the outage would have.
Poll each source at the rate it actually changes. A large aggregator and a small company’s careers feed differ by an order of magnitude in how often anything new appears, and one interval for both is either wasteful or too slow.
Isolate sources from each other so one cannot starve the rest. A per-source budget and a circuit breaker mean a feed that suddenly returns ten times its usual volume consumes its own allocation rather than everyone else’s.
Detecting the silent failure
The failure that costs weeks is not an error. It is a source that starts returning an empty list, or a field that quietly becomes null, while your pipeline reports success.
Alert on shape, not just on errors. If a source that normally returns two hundred postings returns four, that is an incident even though nothing threw. The same applies per field: a sudden rise in missing salaries usually means the API changed, not that employers stopped paying.
Watch the normaliser’s own rejection rate as a leading indicator. A mapping that suddenly cannot interpret a fifth of the records is telling you the schema moved, and it says so before the missing jobs reach anyone’s feed.
- Alert when a source’s volume deviates from its own baseline
- Track the fill rate of each important field over time
- Fail the run when the normaliser rejects an unusual share of records
- Keep a dashboard showing last successful fetch per source
Frequently asked questions
Should I use job APIs or scrape job boards?
Use APIs or feeds wherever they exist — faster, stable, and with terms you can comply with. Scraping should be a deliberate decision after reading the terms, not a default.
Why keep the raw payload if I normalise it?
Because your normaliser is wrong in ways you have not found yet, and postings get taken down. The raw record is the only thing that lets you reprocess history after fixing a mapping.
Where should I use an LLM in an API-based pipeline?
Only on prose: extracting skills, inferring seniority from a title, reconciling work-model wording — and once per posting, keyed by content hash rather than per arrival.
How do I catch a source that silently stops returning jobs?
Alert on volume deviating from that source’s own baseline, track field fill rates, and watch the normaliser rejection rate as a leading indicator.
Should every source be polled at the same interval?
No. Poll at the rate each source actually changes; one interval for a large aggregator and a small careers feed is either wasteful or too slow.
Why give each source its own budget?
So a feed returning ten times its usual volume consumes its own allocation rather than starving every other source in the same run.
Check this against your own resume
Scan your CV against a real job description, or build a parse-safe one from scratch. Your first scan costs nothing.