fix(dedup): distinct non-Latin companies merged into one, deleting a row

[darkpandawarrior]

The bug

dedup-tracker.mjs kept its own local normalizeCompany() using the pre-#2429 ASCII fold:

.replace(/[^a-z0-9 ]/g, '')

Any company name written entirely in a non-Latin script folds to '' under that. Two different employers therefore share a grouping key, pass roleMatch() when their titles agree — common, since "Engineer" / "Backend Engineer" recur everywhere — and the lower-scored row is deleted.

| 1 | アクメ株式会社       | Engineer | 4.2/5 |
| 2 | グロベックス合同会社 | Engineer | 3.0/5 |

before: 🗑️  Remove #2 (グロベックス合同会社 — Engineer, 3.0/5) → kept #1 (4.2/5)
after:  📊 0 duplicates removed

The run prints its normal success output. Nothing warns. Only the .bak preserves the deleted application.

This is the third instance of the same class here

That's the part I think matters more than the fix:

• f589ce83 (merge-tracker: every non-Latin company name normalizes to "", so different companies silently overwrite each other #2429) made tracker-utils.mjs's normalizeCompany Unicode-aware. merge-tracker.mjs and set-status.mjs picked it up automatically because they import the shared helper. dedup-tracker.mjs had forked its own copy, so it didn't.
• fe4561b (fix(dedup): use Unicode-aware Via key for blind-employer grouping #2397) then fixed this same file's blind-via key — while the ordinary company path twelve lines below kept the old fold.
• The comment block right above that code already describes this exact defect, for the Via key.

So the fix is to delete the local copy and import the shared one, rather than patch the regex — that's what stops a fourth instance.

Why this is safe

The value is only ever an in-memory grouping key (groups Map, dedup-tracker.mjs:328). It is never persisted, never written to a row, and never compared against a key produced by another script. Adopting the shared normalization changes no stored data.

I checked the Latin path explicitly rather than assuming: Acme (Inc.) and Acme Inc still merge, and Globex / Initech still stay apart.

Tests

A sibling of the existing #2393 blind-via block, placed next to it, on the ordinary company path:

• two distinct non-Latin employers stay apart (fails before this change)
• punctuation variants of one Latin employer still merge
• two distinct Latin employers still stay apart

The last two are the regression guards for the normalization swap.

node test-all.mjs → 0 failed.

Filed directly per CONTRIBUTING.md (bug fix, no issue needed).

Summary by CodeRabbit

• Bug Fixes

  • Improved duplicate detection for company names using Unicode-aware normalization.
  • Distinct non-Latin company names, including Japanese names, are now kept separate.
  • Equivalent Latin company names with punctuation or spacing differences continue to be combined while retaining the higher-scoring entry.
  • Distinct Latin companies are no longer incorrectly grouped together.

• Tests

  • Added regression coverage for multilingual names and company-name variations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a47a6864-28ac-40e6-bdc1-93d419b2ecd9

📥 Commits

Reviewing files that changed from the base of the PR and between ccb7b0d and 63c554f.

📒 Files selected for processing (2)

• dedup-tracker.mjs
• test-all.mjs

---

📝 Walkthrough

Walkthrough

dedup-tracker.mjs now uses shared Unicode-aware company normalization. Its grouping documentation reflects this behavior. test-all.mjs adds end-to-end coverage for non-Latin company separation and Latin company variant deduplication.

Changes

Unicode company deduplication

Layer / File(s)	Summary
Shared normalization and grouping contract dedup-tracker.mjs	The tracker imports normalizeCompany from tracker-utils.mjs and documents Unicode-aware normalization for company and Via grouping.
Deduplication regression coverage test-all.mjs	The end-to-end test verifies separate non-Latin and distinct Latin companies, and merges equivalent Latin variants while retaining the higher-scored row.

Estimated code review effort: 3 (Moderate) | ~15–30 minutes

Possibly related PRs

• santifer/career-ops#2397: Both changes update dedup-tracker.mjs and regression tests for non-Latin company and Via names.
• santifer/career-ops#2445: Both changes use shared Unicode-aware normalization for company and Via grouping.
• santifer/career-ops#2502: Both changes prevent incorrect grouping of distinct non-Latin companies.

Suggested labels: p1

Suggested reviewers: santifer, scott-emberson, luochen211

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main fix: preventing distinct non-Latin companies from merging and deleting a row.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[santifer]

Merged, @darkpandawarrior, and this is the right shape for it. #2429 made tracker-utils's normalizeCompany Unicode-aware and merge-tracker/set-status inherited it; dedup-tracker kept a private copy, so it stayed on the old [^a-z0-9] fold. Deleting the copy rather than patching it is what stops the two from drifting apart a third time.

Reproduced here before merging, with both controls: the local function on main folds アクメ株式会社 and グロベックス合同会社 to the same empty key (so one of two genuinely separate applications got deleted), while the shared one keeps them apart and still merges Acme (Inc.) with Acme Inc and still keeps Acme and Globex separate. Main 3070/0 after the merge.

Worth naming: the blind-via channel path right above it was fixed for this exact class in #2393, and the ordinary path underneath it stayed broken. A fix that lands next to its sibling and doesn't get applied to it is a pattern I want to catch earlier 🚀