-
Notifications
You must be signed in to change notification settings - Fork 112
Expand file tree
/
Copy pathexpected_changes.toml
More file actions
345 lines (330 loc) · 20.1 KB
/
Copy pathexpected_changes.toml
File metadata and controls
345 lines (330 loc) · 20.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
# Every rule needs `issue`; optional `name_regex` and `fields` narrow
# it. An unexplained diff is a release blocker until classified (spec
# S5). Rules are seeded from docs/superpowers/plans/notes-m12-diffs.md
# and the `classification="fix(...)"` rows in tests/v2/cases.py; keep
# each entry's `name_regex`/`fields` as tight as the diff allows.
# File order is not load-bearing: compare.py sorts `name_regex` rules
# ahead of `fields`-only ones before matching.
[[change]]
issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots"
# '毛 泽东', '김민준': script_orders flips first/last for a name written
# wholly in Han or Hangul, and the Korean surnames that now ship as
# default vocabulary additionally split an unspaced hangul token into
# last + first. '高橋 みなみ', 'マイケル・ジャクソン': #272 extends the
# same order table to kana-bearing Japanese names and makes the
# katakana middle dot a token separator, which moves pieces between the
# same three fields. Seeded from the fix(#271) and fix(#272) rows in
# tests/v2/cases.py.
#
# One rule, not two, because it is one diff shape: both issues move
# name pieces between first/middle/last on native-script CJK input, and
# splitting the class by issue would need a rule per script with no
# gain in tightness (the fields list is already the narrow half).
#
# Scoped by script, which is exactly the scope of the behavior --
# nothing here can touch a name with no CJK character in it. Han's
# astral block (U+20000-U+323AF) is deliberately left out of the class:
# no name in either corpus reaches it, and a rule should be no wider
# than the diffs it has to explain. Extend it -- and the sync pin in
# tests/v2/test_regex_sync.py, which fails if the two disagree -- if
# one ever does. The kana spans are BMP-only for a different reason:
# supplementary-plane kana is hentaigana and other archaic forms no
# modern name uses, so _SCRIPT_RANGES does not list it either.
#
# U+FF65 is the one span here that _SCRIPT_RANGES does NOT have, and
# it is deliberate. The halfwidth middle dot separates tokens like
# its fullwidth twin, so 'マイケル・ジャクソン' splits where 1.4 left one
# token, but halfwidth kana is excluded from CLASSIFICATION on
# purpose (a separate normalization problem). The rest of the
# halfwidth block (U+FF66-U+FF9F) is left out for the usual tightness
# reason, and it was measured rather than assumed: a dotless
# halfwidth name such as 'マイケル ジャクソン' is byte-identical on both
# sides, so covering the block would pre-excuse a future regression
# on a shape that is parity today. U+00B7, the context-sensitive
# 间隔号 (#298), deliberately gets NO span even though it too changes
# parses: its flank guard divides only between classified-script
# characters, so every name it can change already matches this class
# through a flanking character -- and a B7 span's only actual effect
# would be letting this rule claim first/middle/last diffs on
# punt-volat Latin names ('Gal·la Marcet'), pre-excusing a regression
# on exactly the input class the guard exists to protect.
#
# corpus_cjk.jsonl (#295) exists so this rule fires in every real
# run: build_cjk_corpus.py harvests every CJK-bearing text in
# tests/v2/cases.py -- the fix(#271)/fix(#272)/fix(shime-mark) rows
# AND the parity rows, whose value here is pinning NON-diffs (マイケル
# must not change against 1.4) -- closing the blind spot the other
# two corpora have by construction: v1's banks had no reason to test
# CJK, and build_issues_corpus.py requires an internal space, which
# unspaced names never have.
name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]"
fields = ["first", "middle", "last"]
[[change]]
issue = "fix(#274) maiden markers consumed"
name_regex = "(?i)\\b(n[ée]e|born|geb\\.?|roz\\.?)\\b"
fields = ["maiden", "middle", "last"]
[[change]]
issue = "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip"
# Its own rule rather than a widening of fix(#274) above: that rule's
# fields stop at maiden/middle/last because a Latin marker moves only
# those, while a Han one also flips `first` -- the name left after the
# marker is consumed is wholly Han, so it reads family-first (#271).
# Adding `first` to #274's list would make that rule broader than its
# prose and let it absorb diffs that have nothing to do with markers
# (#328). The regex is the marker itself, so this rule can claim
# nothing else.
name_regex = "旧姓"
fields = ["first", "middle", "last", "maiden"]
[[change]]
issue = "fix(comma-family) lone post-comma piece routes to suffix/title, not first"
# 'Smith, Dr.' / 'Andrews, M.D.': v1 put the lone strict-suffix-or-title
# post-comma piece in `first`; 2.0 routes it to `suffix`/`title` instead
# (family/`last` is unchanged either way -- "pre-comma is definitionally
# family").
#
# #312 (the glued-honorific peel site move) sent two of its five
# DIFFING corpus rows here instead of the CJK-comma rule below, and it
# is worth recording why, since this rule's prose has nothing to do
# with CJK. Diffing, not new: four of the five are rows #312 added,
# while '田中さん, PhD' was already in the corpus and #312 changed only
# what it parses to. #312 added a fifth row that does NOT appear here,
# '田中さん, 太郎' -- it agrees with 1.4.0, so it never reaches
# classify() at all; it is the pinned guard for the case a naive fix
# breaks (the peel site there is 太郎, so nothing peels).
# '김, 민준씨' and '田中, 太郎さん' diff as {first, suffix}: the glued
# honorific peels off the post-comma given name into `suffix`
# (민준씨->민준/씨, 太郎さん->太郎/さん), and that field pair happens to
# be a subset of this rule's [first, title, suffix]. No title moved and
# the peeled token is CJK, not a Latin credential -- this rule doesn't
# know or care, because compare.py's classify() matches purely on
# regex-then-field-superset, first rule to satisfy both wins, and this
# rule sits earlier in file order (name_regex tier, written-order tie-
# break) than fix(cjk-comma-compound) below. The other two comma rows,
# 'Dr 김민준씨, V.' and '田中さん, PhD', diff with `last` in the mix
# (v1 read the whole pre-comma run as `first`; 2.0 splits family into
# `last`), which is outside this rule's fields, so they correctly fall
# through to fix(cjk-comma-compound). The fifth row, '威廉·莎士比亚さん',
# has no comma at all -- its dot is U+00B7 (间隔号), not a comma -- so
# it skips every comma-keyed rule and the honorific-suffix rule too (no
# space precedes its glued さん), landing on the fields-only
# fix(suffix-routing) tier by its plain {first, last, suffix} shape,
# same as any glued Latin honorific. Verified by the 2026-08-01 run,
# not assumed: traced through compare.py's classify(), which tries
# name_regex rules before fields-only ones and, within a tier, file
# order, returning the first rule whose regex (if any) matches the
# string and whose fields are a superset of the diff. No rule here
# inspects the honorific token itself -- the field shape alone decides.
# The coverage pass that followed added one more row here, '김, 민준씨
# (Jimmy)': same {first, suffix} diff as '김, 민준씨' -- the extracted
# nickname matches on both sides and so is not part of the diff -- and
# it lands on this rule for the same reason. The limit row recorded
# next to it, '田中さん, V.', appears under no rule at all: it is
# parity with 1.4.0, so it never reaches classify().
name_regex = ","
fields = ["first", "title", "suffix"]
[[change]]
issue = "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix"
# 'Johnson PhD' / 'Mr. Johnson PhD': v1 routed a lone trailing suffix
# to family/first (no comma present); 2.0 keeps recognized suffixes in
# `suffix`.
# #312: also claims comma-less glued-honorific names like
# '威廉·莎士比亚さん' by the same {first, last, suffix} shape as any
# glued Latin honorific -- see the note on fix(comma-family) above for
# why its four comma-bearing siblings split between that rule and
# fix(cjk-comma-compound) instead.
fields = ["first", "last", "suffix"]
[[change]]
issue = "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole"
# Only fires when a custom suffix delimiter is configured (Policy /
# Constants.suffix_delimiters); the corpus runs with default policy, so
# this rule is expected to match nothing here. Kept for documentation
# parity with tests/v2/cases.py's 'suffix_delimiter_no_space_core' row
# (anti-#100, migration plan deviation 5).
name_regex = "/"
fields = ["suffix"]
[[change]]
issue = "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname"
# suffix_acronyms_ambiguous gained 'ma'/'do' so bare 'Jack Ma' keeps its
# surname (v1 parity restored); side effect: parenthesized/quoted "MA"
# or "DO" no longer escape to suffix (v1 did, since v1 treated them as
# unambiguous there) -- they now fall through to nickname parsing.
# Not expected to fire against this corpus (no such strings survived
# into the v1 banks); kept for documentation completeness.
name_regex = "(?i)[(\"'](m\\.?a\\.?|d\\.?o\\.?)[)\"']"
fields = ["suffix", "nickname"]
[[change]]
issue = "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)"
# 'محمد بن سلمان': #269 adds the native-script Arabic
# patronymic particle بن ("bin"/"son of") to PREFIXES/
# NON_FIRST_NAME_PREFIXES. v1 had no such entry, so it left بن a plain
# middle-name token ('سلمان' alone as last); 2.0 now chains it onto the
# family the same way 'von'/'bin' (Latin) do, giving family 'بن سلمان'.
# This is new-recognition on non-Latin input -- the exact behavior
# #269 exists to add -- not a Latin-corpus regression, so it is
# classified rather than reverted.
# Word-bounded: a bare "بن" would also match the substring inside e.g.
# لبنان ("Lebanon") and silently absorb unrelated middle/last diffs.
name_regex = "\\bبن\\b"
fields = ["middle", "last"]
[[change]]
issue = "feat(#273) typographic nickname delimiters recognized by default"
# 'John “Jack” Kennedy', 'Hans „Hansi“ Müller', 'Jean «Petit» Dupont':
# v1 knew only straight quotes and parentheses, so a smart-quoted or
# guillemet-wrapped nickname leaked into `middle` as literal text.
# Matches only when such a delimiter is actually present -- the ASCII
# pairs stay unclassified, since a diff there would be a regression.
name_regex = "[“”„«»「」『』()]"
fields = ["middle", "nickname"]
[[change]]
issue = "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip"
# '山田「タロ」太郎', '山田 太郎 (マイケル・ジャクソン)': one name, two
# intended changes at once -- the corner-bracket/nakaguro handling
# (2.0's typographic delimiters; 2.1's dot separator rendering the
# nickname join with a space) changes `nickname`, while the same
# name's wholly-CJK remainder takes the 2.1 family-first flip in
# first/last. Neither single-change rule may claim the union (the
# delimiter rule's fields exclude first/last, the CJK rule's exclude
# nickname -- each on purpose, so a lone regression in the other's
# fields stays loud). Both lookaheads are required: the delimiters
# alone would match 'John 「Jack」 Kennedy' -- the brackets sit in CJK
# Symbols and Punctuation, OUTSIDE every classified span, so a Latin
# name can carry them -- and would then absorb a bare first/last
# regression on it. Requiring a classified codepoint too confines the
# rule to names the order flip can actually reach. The second
# lookahead's class is the same hand copy of _SCRIPT_RANGES the rule
# above carries, pinned by the same sync test. One absorption is
# inherent and accepted: a nickname-ONLY diff on a nakaguro name
# ('マイケル・ジャクソン') classifies here, because subset-matching
# cannot REQUIRE a field to have changed. Positional note: this rule
# overlaps the CJK rule above and the delimiter rule below; within
# the name_regex tier file order decides which label a diff reports
# under, so it sits after the tighter single-change rules on purpose.
# The issue slug deliberately avoids the literal #271/#272 strings --
# test_regex_sync's differential pin selects the canonical CJK rule
# by those substrings and asserts it is unique.
name_regex = "(?s)(?=.*[「」『』・・])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])"
fields = ["first", "last", "nickname"]
[[change]]
issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip"
# '威廉·莎士比亚, PhD': one name, two intended changes at once -- the
# post-comma lone-piece routing (fix(comma-family): v1 read PhD as
# title; 2.x routes it to suffix) and the CJK source-order/family-
# first movement in first/middle/last on the pre-comma name. Neither
# single-change rule may claim the union: comma-family's fields
# exclude last, the CJK rule's exclude title/suffix -- each on
# purpose, so a lone regression in the other's fields stays loud.
# Both lookaheads are required, by the same reasoning as the
# delimiter compound above: a comma alone matches every Latin
# 'Smith, Jr.' in the corpus, and the classified-codepoint lookahead
# confines the rule to names the CJK behaviors can actually reach.
# The class is the same hand copy of _SCRIPT_RANGES the rules above
# carry, pinned by the same sync test. The slug avoids the literal
# #271/#272 substrings the canonical-rule pin selects by.
#
# #312 sends only the two glued-honorific comma rows whose diff
# includes `last` here ('Dr 김민준씨, V.', '田中さん, PhD'); the two
# whose diff is only {first, suffix} ('김, 민준씨', '田中, 太郎さん')
# match fix(comma-family) first instead, purely on file order -- see
# the note there for why.
name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])"
fields = ["first", "middle", "last", "title", "suffix"]
[[change]]
issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip"
# '王小明 先生', '김민준 씨', '田中 太郎 様': #307 ships the spaced CJK
# honorifics as suffix vocabulary, so a trailing 先生/씨/様 moves to
# `suffix` where 1.4 read it as a name part -- and the remaining
# wholly-CJK name simultaneously takes the family-first flip, so a
# three-token name moves `middle` too, past the fields-only
# suffix-routing rule. Keyed on the honorific alternation itself
# (the maiden-marker rule's pattern): the trigger tokens are wholly
# CJK characters, so the rule matches only a name whose TRAILING
# token is a listed honorific -- a mostly-Latin name with one
# ('Wang Xiaoming 先生') is inside its shadow, accepted because the
# recognized honorific is the diff's cause there too, and the
# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries --
# pinned by tests/v2/test_regex_sync.py, which derives the expected
# set from the config by script membership. Anchored to a WHOLE
# trailing token ((?:^| )...$), judged on the NAME STRING: without
# it, any name string merely ENDING in 양/군 -- the glued given name
# 김지양, the spaced 김 지양 -- would match and could have a real
# regression absorbed as intentional (the case table pins the
# parser-side twin: suffix matching is whole-token, never endswith).
# GLUED diffs (田中さん, 김민준씨, and the 김씨 that segmentation
# already reached) never match this rule -- the anchor is judged on
# the name STRING, and a glued form has no space before its
# honorific. They fall to the fields-only suffix-routing rule by
# their {first, last, suffix} shape (v1 held the whole token in
# first, so last moves too), which is field-honest even though that
# rule's prose describes the two-token Latin case. Verified by the
# 2026-07-30 run, not assumed: #308's peel put a dozen glued names
# in the corpus (Latin+glued like Andersonさん included) and every
# one classified there -- none carried a middle-field diff.
name_regex = "(?:^| )(?:씨|박사|박사님|선생님|교수님|군|양|님|先生|女士|小姐|博士|教授|様|氏|殿|さん|さま|くん|ちゃん)$"
fields = ["first", "middle", "last", "suffix"]
[[change]]
issue = "feat(#269) non-Latin titles/conjunctions recognized"
# 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic
# conjunction И joining two given names). v1 shipped Latin-only
# vocabulary, so these were plain name tokens and everything shifted
# one position. New recognition on non-Latin input -- the point of
# #269 -- not a Latin-corpus regression.
#
# Cyrillic only, and only the three fields the shift can touch. The
# first draft allowed five fields across four scripts, which put every
# Arabic name in the corpus -- including ones that are currently
# PARITY -- into its shadow, and pre-excused Greek and Hebrew, against
# which nothing has ever been compared. A leading title or an absorbed
# conjunction moves a word from `first` to `title`, or from `middle`
# into `first`; it cannot reach `last` or `suffix`. Widen this only
# with a name that demonstrates the need, and prefer a new rule per
# script over loosening this one -- the \bبن\b rule below is the
# model.
name_regex = "[\\u0400-\\u04FF]"
fields = ["title", "first", "middle"]
[[change]]
issue = "fix(leading-credential) a split 'Ph. D.' before the name stays one unit"
# 'Ph. D. John Smith'. v1 healed the adjacent 'Ph.'/'D.' pair only when
# it TRAILED; leading, it split them -- title 'Ph.', given 'D.', and
# the real given name pushed to middle. 2.0 keeps the credential whole
# and routes it to `suffix`, which is the post-nominal field, so a
# leading credential lands there rather than in `title`. Slightly odd
# positionally, and still strictly better than manufacturing a given
# name out of 'D.'.
#
# Anchored to the START on purpose. Trailing 'Ph. D.' is PARITY and
# stays unclassified below; widening this regex would mask a
# regression in the shape that is the whole reason fix_phd exists.
name_regex = "^Ph\\. ?D\\."
fields = ["title", "first", "middle", "suffix"]
# Deliberately NOT a [[change]] rule: TRAILING 'Ph. D.' split-token
# healing ('John Ph. D.', 'John Smith, Ph. D.') is PARITY, not a 2.0
# behavior change -- v1's fix_phd healing of the adjacent 'Ph.'/'D.'
# pair into one suffix unit is replicated exactly by 2.0's 'joined'
# vocab tag (tests/v2/cases.py classification="parity" on both
# 'phd_split' and 'suffix_comma_split_phd'; verified empirically, zero
# diff against the 1.4 worker). Adding a suppression rule for it would
# risk masking a real regression in this exact shape, so it is
# intentionally left unclassified: if it ever starts diffing, the
# harness must fail. (The leading case above is a separate shape, which
# is why it needed its own anchored rule rather than a widened one.)
#
# That last sentence does NOT hold today, and the scope of what is
# parity has to be read narrowly because of it. Parity is the two
# inputs named above, where the pair LEADS its run. A trailing pair
# after another suffix diffs on ORDER -- 'John Smith, Jr. Ph. D.' is
# 1.4.0 suffix 'Ph. D., Jr.' against 2.0's 'Jr. Ph. D.', because
# fix_phd extracted the credential pre-parse and re-appended it while
# 2.0 renders the tail as written (tests/v2/cases.py
# 'suffix_comma_split_phd_after_another_suffix',
# classification="fix(credential-pair-order)"). Measured on a probe
# corpus: the run comes out unexplained 0, absorbed by
# fix(comma-family) below, whose name_regex is a bare comma and whose
# `fields` list contains `suffix`. So the shape is guarded by the case
# table and not by this file. Left that way deliberately rather than
# fixed in passing: classify() takes the FIRST matching rule and the
# sort has only two tiers (name_regex before fields-only, stable
# within a tier), so a narrower rule for this shape could only win by
# being written earlier in the file -- making file order load-bearing
# again, which is the thing the sort exists to prevent. Giving
# classify() a real specificity order is the fix, and it is a change
# to the harness rather than to a rule.