First/Middle name normalization needs to be improved

[sergeyf]

Current normalization is done in two different way:

From text.py

def normalize_text(text: Optional[str], special_case_apostrophes: bool = False) -> str:
    """
    Normalize text.

    Parameters
    ----------
    text: string
        the text to normalize
    special_case_apostrophie: bool
        whether to replace apostrophes with empty strings rather than spaces

    Returns
    -------
    string: the normalized text
    """
    if text is None or len(text) == 0:
        return ""

    norm_text = unidecode(text).lower()

    if special_case_apostrophes:
        norm_text = norm_text.replace("'", "")

    norm_text = RE_NORMALIZE_WHOLE_NAME.sub(" ", norm_text)
    norm_text = re.sub(r"\s+", " ", norm_text).strip()

    return norm_text

1. For name tuples only everything is replaced with a space, including dots, dashes etc.
2. For all other purposes including features, apostrophes are replaced with nothing and all else is replaced with spaces.

Then first and middle are concatted. Then a split is applied. The first element of the split is first, and the rest becomes middle.

Suboptimal because we want Jo Ann, JoAnn, Jo-Ann to be mapped to the same thing. This is not currently happening with the above rules, it is possible and will even match Joann if lower-casing is applied universally, which is part of the text normalization.

This should precede #38 and these better normalizations can be applied to the name tuples before counting as well.

[sergeyf]

J Johnson block, name counts for those first names that have apostraphe or dash.

Jo-Ann          92
Jamlik-Omari    52
Jean-Martial    22
Jo-Anne         15
Jane            11
Jacque-Lynne    10
J-A              4
Jo‐Ann           3
Jo‐Ann M.        3
Jenna-Lyn        2
Jung-chen        2
Jo‐ann           2
Jenni-Louise     2
J.               2
Jean-Ellen       2
Jay-Anne         2
Janet            2
Joann-Lee        2
JO‐ANN M.        1
Jan-Michael      1
John-Irwin       1
Joseph           1
J.-O.            1
Jung-Chen        1
J.-X             1
Jason            1
John-David       1
Ju-Cinta         1
JEAN-MARTIAL     1
Jodi-Ann         1
Jaz-Munn         1
JAMIE M.‐P.      1
Joanne           1
J.W.K.           1
J'Na             1
Jacquelynn       1
John-Michael     1
Juli-Anna        1
Jeri-Lynn        1
J.-A.            1
Jo‐Anne          1
J‐A              1
Jean‐Martial     1

[sergeyf]

Same for Y Wang, first 50 rows. There are 5696 rows total...

Yu-Zhong      821
Yu-Ping       580
Yao-Yu        549
Yu-Ming       502
Yeong-Her     483
Yue-Sheng     482
Ya-Nan        467
Yuan-Yuan     453
Yi-Ping       451
Yan-Hong      439
Yu-Lin        427
You-Nian      408
Yu-Feng       350
Yu-Chun       342
Yi-Ming       338
Yi-Ting       336
Ying-Ying     329
Yong-Jun      316
Yu-Ting       309
Ya-Hui        306
Ying-Ming     302
Ying-Ping     297
Yi-Fei        293
Yong-Qiang    273
Ying-Hong     272
Yan-Jiang     268
Yi-Xiang      261
Yuan-Fang     260
Yu-Jie        259
Yong-Sheng    257
Yu-Chi        256
Ya-Ping       256
Yu-Xiang      253
Yi-Fan        248
Yu-Cheng      246
Yi-Min        244
Yi-Wen        240
Yu-Long       239
Yu-Chen       236
Yu-Ling       235
Yan-Ling      233
Yu-Fang       228
Yu-Sheng      227
Yan-Yan       226
Yi-Xin        225
Ya-nan        222
Yu-Fei        222
Yan-Feng      221
Yu-Min        220
Yu-Mei        220

Note that the thing after the dash isn't always capitalized.

There are also ALL CAPS in places.

[sergeyf]

J Johnson, middle name counts

M-F                             5
F.                              3
M                               3
M.‐F.                           3
Rogers-Clarke                   2
M.-F.                           2
K.                              2
M.                              2
F A                             2
C.                              2
Elizabeth-Sidnie                1
F.A.                            1
M‐F.                            1
Curry O'Connell Marilyn J.      1
Pascual-Leone and Janice        1
Chou                            1
Degraft-                        1
M‐F                             1
J. Dahling and Hazel-Anne M.    1
F                               1
Dolyniuk                        1
De-Graft                        1

[sergeyf]

Same for Y Wang:

Frank           195
J               138
Eric             92
J.               80
Lin              35
C.               26
Tina             22
-M.              14
– Li             13
A.               13
Jim              12
Troy             11
C                11
Lily             10
W.                9
John              9
-H.               8
W                 8
-F.               7
-J.               7
-S.               6
A                 6
Lawrence          6
H.                5
Dr.-Ing.          5
-C.               5
L.                5
-Z.               5
.                 5
-L.               4
-Y.               4
-P.               3
-X.               3
E.                3
Eureka            2
-H                2
David             2
Wang              2
(Joyce)           2
Chen              2
'Tai'             2
Shiuan            2
-Shung            2
(Jim)             2
Laureen           2
Joyce             2
X.                2
Alan              2
Wang‡Yan-Guo      2
Annie             2

[sergeyf]

From David: "Should there be a consideration for multi-initial names like "H. G. Wells"?"

[sergeyf]

This looks very promising https://github.com/datamade/probablepeople/tree/main

Can be tried here https://parserator.datamade.us/probablepeople/

There is a way to make sure it never returns "company" in Python.

[sergeyf]

Survey of how the two types of normalizations are used:

(1) get_name_counts.py - special_case_apostrophes=False
(2) make_s2and_name_tuples.py - special_case_apostrophes=False
(3) get_orcid_name_prefix_counts.py - special_case_apostrophes=True
(4) data.py - special_case_apostrophes=False for everything except
- special_case_apostrophes=True for author_info_first_normalized_without_apostrophe and author_info_middle_normalized_without_apostrophe

What are author_info_first_normalized_without_apostrophe and author_info_middle_normalized_without_apostrophe used for?

• In get_orcid_name_prefix_counts.py recomputed in the similar normalize_names function
• In model.py for Clusterer.predict and predict_incremental as the canonical names
• In featurizer.py to extract features
• In subblocking.py in make_subblocks

Where is author_info_first_normalized used?

• For pre-processing keys in name counts, looking things up in name_counts
• For pre-processing name tuples.

---

There is no one single normalization that fixes to the Jo-Ann issue. A probablepeople parser

for i in (parse("Jo Ann Taylor-O'neal", 'person'), parse("Jo-Ann Taylor-O'neal", 'person'), parse("JoAnn Taylor-O'neal", 'person')):
    print(i)

Gives you

[('Jo', 'GivenName'), ('Ann', 'GivenName'), ("Taylor-O'neal", 'Surname')]
[('Jo-Ann', 'GivenName'), ("Taylor-O'neal", 'Surname')]
[('JoAnn', 'GivenName'), ("Taylor-O'neal", 'Surname')]

So why was there a special case for the apostrophes? I really don't remember... It's almost certainly related to the logic I used to figure out what is first vs middle:

Name-normalization workflow

1. normalize_text()

   • Empty/None → "".
   • ASCII-fold and lowercase with unidecode.
   • Optionally drop apostrophes (if special_case_apostrophes=True).
   • Replace any punctuation/diacritics matched by RE_NORMALIZE_WHOLE_NAME with a space.
   • Collapse multiple spaces, trim ends.

2. First + middle processing

   • Build two versions of the “first + middle” string:
     – normal (first_normalized + " " + middle_normalized)
     – apostrophe-stripped (first_normalized_without_apostrophe + " " + middle_normalized).
   • Split each into tokens.
   • If the first token is a known prefix in NAME_PREFIXES (e.g., de, van, d’), drop it.
   • After this adjustment:
     – First name = first token of each list.
     – Middle name(s) = remaining tokens of the apostrophe-stripped list (joined by spaces).

3. Last and suffix

   • author_info_last_normalized = normalize_text(author_info_last).
   • author_info_suffix_normalized = normalize_text(author_info_suffix or "").

4. Stored results in the signature

   • author_info_first_normalized – first token with punctuation retained.
   • author_info_first_normalized_without_apostrophe – same, but apostrophes already removed.
   • author_info_middle_normalized_without_apostrophe – middle component(s) without apostrophes.
   • Plus the normalized last name and suffix above.

Since we are splitting on space, I probably found all kinds of scenarios where there were apostrophe-having first or middle names, and wanted them to not be split up.

IF, we move away from this approach and just take what is given to us by probablepeople, then maybe a single form of normalization is totally fine. In that form: should apostrophes be replaced by a space or by nothing?

[sergeyf]

Gemini: "The existing normalization scheme in the S2AND pipeline is designed to handle name variation correctly and will not place a large constraint between any pair of Jo-Ann, JoAnn, and Jo Ann.

Here is a breakdown of why it works:

Normalization (normalize_text): This function replaces non-alphanumeric characters (like hyphens) with spaces.

Jo-Ann becomes "jo ann".
Jo Ann becomes "jo ann".
JoAnn becomes "joann".
Name Parsing (preprocess_signatures): This step splits the normalized string to identify the first name.

For "jo ann", the first name is parsed as "jo".
For "joann", the first name is parsed as "joann".
Constraint Logic (get_constraint): This is the crucial part. The function does not require an exact match. It avoids applying a constraint if one name is a prefix of the other.

Let's look at the comparisons:

Jo-Ann vs. Jo Ann: Both are parsed to have the first name "jo". Since they are identical, no constraint is applied.
Jo-Ann vs. JoAnn: The first names are "jo" and "joann". The code checks if "joann".startswith("jo"), which is True. Because it's a valid prefix, no constraint is applied.
Jo Ann vs. JoAnn: This is the same comparison as above, and it also passes the prefix check.
Therefore, the current system correctly identifies these as compatible names. The combination of replacing hyphens with spaces and using a prefix-based check for constraints makes the logic robust to these common name variations."

[sergeyf]

Important example is Chinese names:

• Yu Zhong
• YuZhong
• Yuzhong
• Yu-Zhong
• Y. Z.

Are all the same FIRST NAME. The current scheme allows any pair of these to be plausible for each because the scheme gets us:

• yu
• yuzhong
• yu
• y

As the normalized first name. So the prefix rule (a.startswith(b) or b.startswith(a)) comes in as True for all pairs.

Replacing everything non-alphanumeric with '' scheme would fail that and we'd get a do not merge constraint.

So we are stuck here. The featurization logic has to recover from this issue where Zhong is sometimes a middle name and sometimes part of the first name.

I think that means this ticket should be closed and we should instead really focus on looking at Chinese names, which I have started to do.

[sergeyf]

Generally speaking, things with dashes in them are a single name like "John-David" or "J-D"/"J.-D.". Our "startswith". Things with spaces are sometimes just the first name name (e.g. "Jo Ann"), always one name for Chinese names (like Yu Zhong Li) (and other East/SouthEast Asian?), and mostly are first name and then middle name for non-Asian names.

To deal with this, we can update the a.startswith(b) or b.startswith(a) rule to work with multiple token first names.

Assume we can eventually normalize multiple token first names to have only spaces.

Then we can generalize a.startswith(b) to check every token up to the minimum number of tokens.

E.g.

• "john david" vs "j" comes out as true
• "john david" vs "j d" comes out as true
• "john" vs "j d" comes out as true
• "john david" vs "j f" comes out as false
• "jon david" vs "john david" comes out as false

# ≈180 ns on CPython 3.11 for two-token names, says o3
def same_prefix_tokens(a: str, b: str) -> bool:
    """
    Symmetric multi-token “startswith”.
    True ⇔ for every aligned pair of tokens (up to the shorter list),
           one token is a prefix of the other (case-insensitive).
    """
    ta, tb = a.lower().split(), b.lower().split()
    for x, y in zip(ta, tb):
        if not (x.startswith(y) or y.startswith(x)):
            return False
    return True

[sergeyf]

Findings:

• Removing apostrophes from the first name is harmless. And it might be harmless EVERYWHERE???
• Doug-Steve in the first name is really reliably a hyphenated name.
• A space in the first name is really tricky. If the name is Chinese, we can assume compound first name. If not, It's probably safer to assume that the "first name" field has a middle name in it too, even if it's wrong sometimes:
• "T. C." -> No idea.
• "Tomi S." -> S. is probably a middle name
• "Kevin Chen-Chuan" -> Kevin is probably the Westernized name, so we'd get this wrong.
• "Miguel Ángel" -> No idea.
• "B. V. K. S." -> No idea.
• "Halil Önder" -> No idea.

So, we should JUST look at dashes for creating multi-token first names. by replacing

            first_normalized = normalize_text(signature.author_info_first or "")
            first_normalized_without_apostrophe = normalize_text(
                signature.author_info_first or "", special_case_apostrophes=True
            )

            middle_normalized = normalize_text(signature.author_info_middle or "")
            first_middle_normalized_split = (first_normalized + " " + middle_normalized).split(" ")
            if first_middle_normalized_split[0] in NAME_PREFIXES:
                first_middle_normalized_split = first_middle_normalized_split[1:]
            first_middle_normalized_split_without_apostrophe = (
                first_normalized_without_apostrophe + " " + middle_normalized
            ).split(" ")
            if first_middle_normalized_split_without_apostrophe[0] in NAME_PREFIXES:
                first_middle_normalized_split_without_apostrophe = first_middle_normalized_split_without_apostrophe[1:]

with (needs testing)

def split_into_first_middle(first_raw: str, middle_raw: str, NAME_PREFIXES: set[str]):
    """
    Normalize and split fields with the rules:
      - Apostrophes are removed globally (never create spaces).
      - Hyphen *in raw first* => keep all first tokens as first (no spill).
      - Otherwise, first token is first; remaining first tokens spill into middle.
      - Known prefixes at the *start of first* are dropped.
    Returns:
      author_info_first,
      author_info_last
    """

    # Detect dash in RAW first (normalize will erase '-')
    has_dash_in_first = "-" in (first_raw or "")

    # Single, apostrophe-less normalization everywhere
    first_norm  = normalize_text(first_raw)   # now always deletes "'"
    middle_norm = normalize_text(middle_raw)

    f_parts = first_norm.split()
    m_parts = middle_norm.split()

    # Drop leading prefix (only one) if present
    if f_parts and f_parts[0] in NAME_PREFIXES:
        f_parts = f_parts[1:]

    if not f_parts:
        first_out  = ""
        middle_out = " ".join(m_parts)
    else:
        if has_dash_in_first:
            # e.g., "Anna-Marie" -> first="anna marie"; no spill
            first_out  = " ".join(f_parts)
            middle_out = " ".join(m_parts)
        else:
            # e.g., "John David" -> first="john"; spill "david" to middle
            first_out  = f_parts[0]
            middle_out = " ".join(f_parts[1:] + m_parts)

    # With apostrophes always removed, these two are identical by design
    author_info_first_normalized = first_out
    author_info_first_normalized_without_apostrophe = first_out
    author_info_middle_normalized_without_apostrophe = middle_out

    return first_out, middle_out

Also, we can probably cover more apostrophes: a few “apostrophe-like” marks won’t be removed by your current code:

• Grave/backtick: ` (U+0060) → stays as ` after unidecode, so you don’t delete it (e.g., O\Brien`).
• Spacing acute: ´ (U+00B4) often becomes ', but not always consistently across sources.
• Rare glottal/okina & cousins: ʻ (U+02BB), ʼ (U+02BC), saltillo ꞌ (U+A78C), primes ′ ‵ (U+2032/U+2035) — unidecode typically maps these to ', but normalizing them explicitly is safer.

Keep your pipeline the same, just delete the “apostrophe-likes” after unidecode:

APOSTROPHE_DELETE_AFTER_UNIDECODE = str.maketrans({
    "'": "",   # ASCII apostrophe
    "`": "",   # backtick/grave accent (O`Brien variants)
})

def normalize_text(text: Optional[str], special_case_apostrophes: bool = False) -> str:
    if not text:
        return ""
    # First make sure all apostrophe-like code points become ASCII "'"
    # (unidecode already does most of this, but pre-unifying helps edge inputs)
    text = _unify_apostrophes_pre_unidecode(text)

    norm_text = unidecode(text).lower()

    if special_case_apostrophes:
        norm_text = norm_text.translate(APOSTROPHE_DELETE_AFTER_UNIDECODE)

    norm_text = RE_NORMALIZE_WHOLE_NAME.sub(" ", norm_text)
    norm_text = re.sub(r"\s+", " ", norm_text).strip()
    return norm_text

# Optional: tighten inputs before unidecode (covers rare sources reliably)
_APOS_PRE_MAP = {ord(c): "'" for c in "’‘ʼʻ′‵´ꞌ﹍＇"}
def _unify_apostrophes_pre_unidecode(s: str) -> str:
    return s.translate(_APOS_PRE_MAP)

• The pre-unidecode step coerces curly quotes, okina, modifier-apostrophe, primes, spacing acute, saltillo, and fullwidth quote to plain '.
• The post-unidecode translate then removes both ' and the backtick ` when special_case_apostrophes=True.

---

TODO:

• Remove special_case_apostrophes flag and just ALWAYS replace "'" with nothing.
• Harden against more apostrophes.
• For Chinese names, wire in Sinonym and make it so dashed first names stay together, then confirm no regression using the prod model [DONE]
• Get the name counts again (with new pre-normalization) (get_name_counts.py)
• Get the name tuples again (with new pre-normalization) (have to write a new script to make a s2and_name_tuples_filtered.txt from s2and_unnormalized_filtered_name_tuples.txt)
• Same for get_orcid_name_prefix_counts.py
• For the 3 above, see Name tuples updates #38
• Wire in split_into_first_middle and so we no longer have a distinction between author_info_first_normalized and author_info_first_normalized_without_apostrophe anywhere so we can actually rename these variables everywhere.
• Update everywhere to use same_prefix_tokens [DONE]
• Test to make sure no regressions. And no massive speed reductions.
• [OPTINAL] Integrate SPECTER2
• Retrain the prod model

Question:

• Why did I originally want to have two streams w.r.t. apostrophe?

[sergeyf]

IMPORTANT: when retraining the model, the training data has to match what eventually inference data looks like.

IF future inference data will have already gone through sinonym, then the training data must also go through sinonym in the same way. As such, we will have to make sinonym a dependency no matter what happens.

[sergeyf]

From codex, after working on integrating sinonym and hyphenated first names:

Normalization Unification Migration Plan

Scope

• Unify name normalization for first/middle/last across data preparation, modeling, subblocking, and auxiliary datasets (name counts, name tuples, ORCID prefix counts).

Current State (post-Sinonym hyphen pass)

• Canonical first/middle preserve hyphenated Chinese given names:
  • Implemented via s2and.text.split_first_middle_hyphen_aware.
• Backward-compat shims for artifacts built with legacy normalization (to be removed):
  • Name counts (first): when raw first had a hyphen, join spaces in canonical first for count keys (e.g., "qi xin" → "qixin").
  • Name counts (last): join spaces in canonical last for compound/hyphenated surnames (e.g., "ou yang" → "ouyang").
    • Helpers: _canonicalize_last_for_counts(...).
  • Constraints: last-name disallow uses a space-insensitive comparison ("ou yang" == "ouyang").
    • Helper: _lasts_equivalent_for_constraint(...).
  • Subblocking: temporary ORCID prefix map probe uses the first token from canonical first when multi-token.
  • Sinonym overwrite gating (optional, off by default unless invoked):
    • Compute allowlist per normalized name using multi-author evidence priority:
      • If multi-author evidence exists: overwrite when flips x satisfy x >= min_ratio * y (not-flips).
      • Else (single-author only): overwrite when any flip evidence exists (a > 0).
  • Blocks on Sinonym overwrite (inference-only): recompute author_info_block as first_initial + compact_surname where
    compact_surname removes spaces/hyphens (e.g., q ouyang) to keep compound surnames atomic for blocking.

Target State

• Single, unified normalization for names (apostrophes always stripped; hyphen/space variants treated consistently; Sinonym for Chinese names keeping given-name tokens together; surname handling consistent with given names).
• Remove the distinction between author_info_first_normalized and author_info_first_normalized_without_apostrophe; use a single canonical field consumed everywhere (features, constraints, counts, tuples, subblocking).
• No special-case shims for counts or constraints; artifacts regenerated to match canonicalization.

Steps

1. Finalize normalization policy

   • Apostrophes: always remove (no replacement with spaces in canonical fields).
   • Hyphen/compound names: treat hyphen and space variants equivalently in canonicalization. For Chinese given names, keep tokens together per Sinonym; for surnames, adopt consistent joining/preservation policy (see 4).
   • Language-agnostic defaults: ensure non-Chinese names remain unaffected other than consistent punctuation handling.

2. Implement unified canonicalization

   • Provide a single canonicalization path for first/middle/last (extends split_first_middle_hyphen_aware to surnames or replaces with a unified routine).
   • Replace all reads/writes of author_info_*_normalized* with the unified canonical fields.

3. Regenerate external artifacts

   • Name counts: regenerate with unified canonicalization (first, last, first_last, last_first_initial).
   • Name tuples: regenerate with canonical forms aligned to the unified logic.
   • ORCID prefix counts: regenerate first_k_letter_counts_from_orcid.json using canonical first names (no first-token fallback).

4. Blocks and downstream features

   • Define block computation on canonical names (first-initial + surname). Decide whether canonical surname should be joined (preferred) or space-separated; update compute_block and Sinonym overwrite code accordingly.
   • Remove inference-only surname compaction once blocks universally use canonicalization.

5. Remove compatibility shims

   • Drop _canonicalize_last_for_counts, _lasts_equivalent_for_constraint, and related TODOs.
   • Remove the subblocking ORCID first-token probe.

6. Validation

   • Run pairwise and clustering evaluations on representative datasets; compare against baseline.
   • Inspect subblocks and merges; verify Chinese/Western hyphenated cases perform as expected.
   • Rebuild caches as needed (featurizer cache keyed by version may need a bump).

Rollback/Compat Notes

• Keep a feature flag or version switch temporarily to load and operate on legacy artifacts during transition.

References in code (as of this migration doc)

• Given-name canonicalization: s2and.text.split_first_middle_hyphen_aware.
• Surname compat for counts: _canonicalize_last_for_counts in s2and/data.py.
• Last-name constraint shim: _lasts_equivalent_for_constraint in s2and/data.py.
• ORCID prefix fallback in subblocking: comment and lookup in s2and/subblocking.py near the counts probe.
• Sinonym overwrite gating and application: compute_sinonym_overwrite_allowlist, apply_sinonym_overwrites in s2and/data.py.

Tests added for regression

• tests/test_surname_hyphen_aware.py
  • Validates surname count canonicalization, last-name constraint equivalence, and block compaction behavior under Sinonym overwrites.