Anton Kylikov

We pull job posts from a few dozen boards and company career pages, and each one is keyed on its source plus that source's own id. That works inside one source and does nothing across them: the same opening can arrive from the company's own page, an aggregator and a recruiter repost, each with a slightly different title and location line. Matching on company plus title looks obvious until you meet one company with two genuinely separate Backend Developer openings in two cities, next to one opening titled three different ways. We still haven't found a rule we trust, so for now we let duplicates leak through rather than risk merging real jobs away. If you run anything that aggregates listings, products or events from many places, which signal ended up being the one you trust for deciding two records are the same?

Our interface runs in twelve languages, and the short strings turned out to be the easy part. The hard part is long pages: an article gets translated, then someone fixes a figure or rewrites a paragraph in the original, and every other version quietly keeps the old text. Right now we only keep pages without a translation out of search, which stops thin duplicates but does nothing for a version that was correct last month and is wrong today. I keep thinking about storing a hash of each source paragraph next to its translation and flagging any mismatch, but I have not built it, and I am not even sure a paragraph is the right unit. If you run a site in more than two languages, how do you find out that a translation is out of date, and who ends up fixing it?