iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
If a job-search API returns the same vacancy from several boards, start by comparing normalized apply URLs and looking for a shared employer or applicant-tracking-system (ATS) requisition ID. Titles and source-specific job IDs can differ even when listings point to the same opening. Keep each board’s record intact, group copies with a traceable requisition-level identity, and use fuzzy matching only when stronger identifiers are missing.
Why the same vacancy appears under different titles
A job title is descriptive text, not a dependable unique key. One board might shorten or rewrite a title while another preserves the employer’s wording. In a reported example, a search for “python developer” with a remote filter returned 30 results, including four copies of one Berlin fintech requisition across three aggregators and the employer’s board. Exact-title matching and a company-plus-lowercased-title check did not catch every copy; an apply URL containing a canonical requisition ID provided the useful clue. These are figures from that article’s example, not a measure of how often job feeds contain duplicates. DEV Community’s account is the source for the scenario.
Research on duplicate job-posting detection also describes the underlying challenge: separate postings for the same job can vary in text. A 2024 preprint found value in combining character-level similarity, text embeddings, and curated skill lists rather than relying on any single technique. It does not establish a universal matching threshold or a performance figure suitable for every feed. Read the paper on arXiv.
Free tools Windows power users keep installed
One-click scans. No signup required.
What the apply URL can tell you
An apply link may lead to an employer’s ATS and include a requisition identifier that remains stable across listings copied to other boards. That can be stronger evidence of a shared vacancy than matching titles or descriptions. It is not a universal guarantee: URLs vary by ATS, and a link can lead to a generic application page rather than a specific requisition.
#1 Best Overall
- Works on Windows 11, 10, & 8
- Build a Professional Resume Fast with the step-by-step guide to help you create a professional resume that showcases your unique experience and skills
- ResumeMaker & Resume Maker are registered trademarks & box images and screenshots are copyrights of Individual Software Inc.
- Modern Resume Styles - Choose from 60 styles and customize any style with choice of header, colors, graphics and a photograph plus Powerful Ways to Search for Jobs
- Video Resumes & Expert Advice - View Sample Video Resumes and video resume scripts you can customize plus Email & Share Your Resume on LinkedIn, Facebook & Twitter
Before comparing links, preserve the original value and derive a separate normalized form. Follow redirects only when permitted and practical, remove known tracking parameters, and normalize the host and path. Do not strip query parameters blindly: one may contain the requisition ID you need. Parse identifiers from known ATS patterns, validate that an extracted value refers to a particular posting, and version your URL parsers so later changes can be audited.
- Raw apply URL: the exact value received, retained for provenance and troubleshooting.
- Normalized apply URL: a comparison value after carefully handling redirects, host, path, and known tracking parameters.
- Extracted requisition ID: a parsed identifier, stored separately with its ATS or employer context and parser version.
Keep vacancy identity separate from source records
A requisition-level identity answers “Which opening is this?” A source-posting ID answers “Which record did this board or API return?” Those are different questions. Do not overwrite a board’s ID with an ATS identifier or collapse all fields into a single guessed key.
For each result, retain the provider job ID, source or board, employer, title, location, posting URL, apply URL, and retrieval time. Store extracted identifiers and normalized URLs alongside those source fields. When you decide that several listings belong to one vacancy, represent that as a reversible association or cluster with its supporting evidence. That lets a user inspect the grouped records and lets your system undo a mistaken match without losing the originals.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsUse a matching hierarchy, not one brittle rule
Prefer the strongest available evidence, and stop short of an automatic merge when the evidence conflicts. In particular, similar wording is not enough to prove that two locations or requisitions are the same opening.
- Shared explicit requisition ID: match when the ID is extracted from a validated, job-specific ATS or employer URL. Include the ATS or employer context if IDs are not globally unique.
- Employer and ATS evidence: compare employer identity, ATS host, and requisition ID together when that combination is the reliable scope for the identifier.
- URL and employer evidence: use normalized apply or posting URLs with employer information when no validated requisition ID is available.
- Scored content similarity: as a fallback, combine title, location, and description signals. Character similarity, embeddings, and curated skill lists are complementary approaches described in the 2024 preprint, not a ready-made universal rule.
Keep uncertain matches reviewable. A shared employer and near-identical description may still represent distinct openings, while a changed title may refer to the same requisition. Record which signals drove each association rather than presenting a fuzzy score as proof.
Choose a deduplication method by its risks
| Method | Identity strength | Cross-board coverage | Main risk or limitation |
|---|---|---|---|
| Validated requisition ID from an ATS or employer link | Strong when the ID is job-specific and scoped to the correct ATS or employer | Can connect mirrors that expose the same identifier | Some links hide the ID, use generic routes, or contain IDs whose scope is unclear |
| Normalized URL plus employer evidence | Useful when URLs resolve to a job-specific destination | Can connect listings that share a destination despite different board IDs | Redirects, tracking parameters, and meaningful query parameters complicate normalization |
| Title, location, and description similarity | Suggestive rather than definitive | Can help when source links do not expose shared identifiers | May miss rewritten titles or falsely merge similar roles, locations, or separate openings |
| Provider exclusion or “seen before” filters | Useful for repeated calls to that provider | Limited to the provider’s own records and filter semantics | Does not necessarily combine different source records for one requisition |
The available sources do not provide measured scores for false-merge rates, cross-board coverage, URL stability, or operating cost across these methods. Choose thresholds against your own feed and keep the evidence visible.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Do not confuse retries with cross-board deduplication
Idempotency protects a write operation from being applied repeatedly after a retry; it does not identify two different source records as one vacancy in a search result set. SEEK’s developer guidance, which is marked draft, recommends reusing the same idempotencyId for retries of the same posting and not reusing it for a different job. That is a mutation-safety rule, not a cross-board matching algorithm. See SEEK Developer documentation.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchLikewise, some search providers offer filters for avoiding results already returned on earlier calls. TheirStack documents discovered_at_gte and job_id_not for this purpose. These are provider-specific options for recurring queries; they do not necessarily collapse separate records that refer to one requisition. See TheirStack’s guide.
Best Value
Prefer a trustworthy apply destination
When listings have been grouped as the same requisition, an active employer-hosted ATS link is a sensible apply destination if it clearly represents that opening. The reported example recommends trusting the employer’s ATS link over an aggregator mirror, but that is a recommendation from that account, not a guarantee that every employer link is current or correct. Check that the destination still resolves to the intended role before presenting it as the preferred link.
Keep search indexing guidance in its lane
Google Search Central advises using canonical URLs for copies of a job posting and placing JobPosting structured data on the page that describes a single job. That guidance concerns search indexing and presentation; it is not a universal database key for merging API results. Read Google’s JobPosting documentation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
