Searching many sources at once

Not every source is available to you. Official APIs and published feeds come with terms; many sites prohibit automated collection outright, and some jurisdictions treat breach of those terms as more than a contractual matter.

Decide this per source before building. Retrofitting a legal review onto a running aggregator usually means removing sources, and by then users depend on the coverage they provided.

Behave well on the sources you do use. A published crawl delay, a rate limit and an honest user agent cost nothing and are the difference between a long-term arrangement and a block that arrives without warning on the morning it matters most.

• Official APIs and partner feeds — preferred, with stated limits
• Published syndication feeds — usually explicit about reuse
• Employer career pages — check terms, respect crawl directives
• Anything prohibited by terms — not a technical question

Every source describes a job differently: salary as a range, a single figure, hourly or annual, in different currencies, sometimes as text. Location may be a city, a region, a code or "remote (US)". Employment type has no shared vocabulary.

Define your own schema and map into it at the boundary, keeping the raw payload alongside. Source-shaped data leaking into the rest of the system means every consumer handles five formats, and a new source touches everything.

Let fields be genuinely absent rather than defaulted. A missing salary recorded as zero, or a missing location recorded as "remote", produces filtering that silently excludes or includes the wrong roles — and unknown is a value the rest of the system can handle correctly if you let it exist.

The same posting appears across aggregators with different links, slightly different titles and different descriptions. URL-based deduplication does nothing, and a candidate seeing one role five times loses confidence in the whole feed.

Match on employer, normalised title, location and a similarity threshold on the description. Then choose a canonical record — the employer’s own page where available, since it is the most likely to be current and the one that accepts applications.

Agencies are the case that defeats naive matching. The same role appears under three recruitment firms and the employer, with the employer’s name removed from the agency versions, and collapsing those requires matching on the description rather than on the employer field that is deliberately blank.

Aggregated feeds carry closed roles, sometimes for weeks. A candidate who is sent to two dead links stops trusting the source, and that judgement extends to everything else it shows them.

Track when each posting was last confirmed present at its source, re-check the ones you surface most, and show the date. Where you cannot verify, say so rather than presenting stale data as current.

Concentrate the re-checking where it pays. Verifying the fifty postings you are about to recommend today is cheap and catches almost all of the damage; verifying an entire index nightly is expensive and mostly re-confirms roles nobody will see.

Sources go down, rate-limit and change their formats, and they will not do it in a coordinated way. An aggregator that fails entirely when one source misbehaves is offline far more often than any of its inputs.

Run sources independently with their own timeouts and error handling, return what succeeded, and say which source was unavailable. Partial results with an honest note are far better than an error page.

Watch for the quiet failure too. A source that returns a valid empty response because its format changed looks like a slow week rather than an outage, so alert on a source’s volume dropping sharply rather than only on errors.

• Independent timeouts and error handling per source
• Return what succeeded, and name what did not
• Alert on a volume drop, not only on an exception
• A circuit breaker, so a broken source is skipped rather than retried
• Last-good results served with their age shown

Polling every source every few minutes is expensive and mostly returns what you already have. Poll frequency should follow how fast each source actually changes, which varies by an order of magnitude between a large aggregator and a small company’s careers page.

Incremental fetching, where a source supports it, is the difference between a viable aggregator and one whose bill scales with the size of the index rather than with the number of new postings. Ask for what changed since your last cursor before falling back to fetching everything.

Spend the expensive processing once. Embedding, requirement extraction and classification should run on first sight of a posting, keyed by its identity, so the same role arriving through a fourth source costs a lookup rather than another full pass.