How do you tell that two listings from different sources are the same thing?
We pull job posts from a few dozen boards and company career pages, and each one is keyed on its source plus that source's own id. That works inside one source and does nothing across them: the same opening can arrive from the company's own page, an aggregator and a recruiter repost, each with a slightly different title and location line. Matching on company plus title looks obvious until you meet one company with two genuinely separate Backend Developer openings in two cities, next to one opening titled three different ways. We still haven't found a rule we trust, so for now we let duplicates leak through rather than risk merging real jobs away. If you run anything that aggregates listings, products or events from many places, which signal ended up being the one you trust for deciding two records are the same?
Replies
Much smaller scale than yours, but one thing from importing birthdays from a phone's calendar into my app transfers: the id of each imported record is a hash of what identifies it in the source, so syncing again updates the same record instead of adding a copy. And a record the user removed is kept with a dismissed flag rather than deleted, otherwise the next sync brings it straight back. For cross source jobs I would keep your choice of letting duplicates leak: a duplicate costs a second of scrolling, a wrong merge silently deletes a real opening. Maybe show them grouped as possibly the same job and let the user collapse them.