v1.5.0 - LLM functionalities (optimised tokenisation, anonymisation)

What's new in v1.5

Everything from v1.4 still works exactly as before. v1.5 adds two things that belong together: it can hand the review to a language model, and it can do that without handing over the people in it.

The export a model can actually use

records = parser.to_llm_dataset()

{
  "comment_id": 4,
  "author": "Dave Architect",
  "context_before": "Section 5 describes data handling obligations under applicable law.",
  "referenced_text": "Records are retained for seven years.",
  "context_after": "Deletion requests are processed within 30 days.",
  "comment": "Why seven years? GDPR needs a documented lawful basis for that.",
  "thread": [...],
  "category": "Compliance",
  "priority": "high",
  "intent": "question",
  "action_required": true
}

Three of those fields are the whole point.

context_before and context_after are real document text. A comment on its own is usually unreadable — "this is wrong" means nothing. The parser knows the passage a comment is anchored to; the sentences either side of it live in word/document.xml, which the C++ core reads for anchors and then discards. v1.5 reads it back, in Python, on demand and cached per document, so a model sees the comment in the document rather than as a fragment.

thread travels with every comment in it. Redundant on purpose: a retrieval system pulls back single records, and a record that cannot see the reply that resolved it will report an answered question as open.

category, priority, intent and action_required arrive already computed. Rule-based, local, microseconds, and every decision explains itself by naming the words that drove it. It means a prompt can say "here are the 12 compliance comments" instead of spending its budget rediscovering that.

Masking the people first

records = parser.to_llm_dataset(anonymize="strict")

Alice Tester   →  person-71e9b0
"Mail bob.reviewer@acme.example, key AKIAIOSFODNN7EXAMPLE, call +44 20 7946 0958."
               →  "Mail [EMAIL], key [API_KEY], call [PHONE]."

A .docx review is one of the most personal artefacts a company holds: every comment carries a named human, an opinion, a timestamp, and — genuinely, routinely — a key someone pasted in and forgot. Sending that to a hosted model is a disclosure, and "we only sent the comments" is not a defence.

So the redaction is real rather than decorative:

• Checksum-validated, not shape-matched. A sixteen-digit number is not a credit card; one that passes Luhn probably is. An IBAN has to pass mod-97. That is what keeps a redactor from mangling every build number in a document until people stop trusting it.
• Names come from the document, not from a gazetteer. The reviewers are known exactly — the parser read them out of the file — so recall on the people who matter is total, and a section about a Mark is not redacted because someone called Mark reviewed it.
• One person, one stand-in, everywhere. Alice, alice tester, @alice and the author column all become the same Reviewer 1, so "Reviewer 1 raised this three times and Reviewer 2 disagreed" is still a sentence the model can produce.
• The output is checked. After redacting, the result is scanned again with the same detectors, and anything still matching is reported. A redactor that cannot tell you whether it worked is one that will quietly stop working.

And because the mapping stays on your machine, the model's answer can be turned back:

dataset = parser.to_llm_records(anonymize="strict")
answer  = ask_your_model(dataset)                 # sees "person-71e9b0"
print(dataset.anonymizer.deanonymize(answer))     # you see "Alice Tester"

Five presets — names_only, balanced, secrets_only, strict, gdpr — and per-entity control over whether each kind is pseudonymised, redacted, masked, hashed, removed or kept.

Prompts that pack the corpus for you

The hard part of asking a model about a review is not the wording; it is which comments fit, in what order, with what context, serialised so the answer can be joined back to real comment ids. That is the part this library knows:

from docx_comment_parser.llm import create_action_items_prompt

prompt = create_action_items_prompt(dataset.records)
response = client.messages.create(**prompt.to_anthropic(), max_tokens=4096)

Six builders — summary, action_items, resolution, triage, risk, and diff_summary, which takes a v1.4 DiffResult directly. Each one asks for a named JSON schema, requires every claim to cite a comment id, and says out loud when the corpus did not fit rather than presenting a truncated review as a complete one.

Nothing is sent anywhere. There is no client, no key, no network call in this package — the builders return strings.

And the rest

parser.to_embeddings_input()        # one retrieval document per thread, for RAG
parser.export_jsonl("review.jsonl") # the format every batch API takes
parser.to_llm_chunks(max_tokens=100_000)  # never splits a thread in half
parser.privacy_scan()               # what is in here, changing nothing

From a terminal:

docx-comments llm spec.docx --anonymize strict -o review.jsonl
docx-comments prompt spec.docx --kind risk
docx-comments classify spec.docx --priority blocker
docx-comments anonymize contract.docx --scan --fail-on-secret   # a CI gate

Nothing got heavier. The base install still has zero dependencies — the context reader, the classifier and the whole privacy layer are standard library only. Token counting uses tiktoken if you have it and a calibrated estimate that errs high if you do not. Importing the library loads neither layer. The parser is untouched and just as fast; see Performance.

Full Changelog: v1.4.0...v1.5.0

LLM functionalities (optimised tokenisation, anonymisation)

What's new in v1.5

Everything from v1.4 still works exactly as before. v1.5 adds two things that belong together: it can hand the review to a language model, and it can do that without handing over the people in it.

The export a model can actually use

records = parser.to_llm_dataset()

{
  "comment_id": 4,
  "author": "Dave Architect",
  "context_before": "Section 5 describes data handling obligations under applicable law.",
  "referenced_text": "Records are retained for seven years.",
  "context_after": "Deletion requests are processed within 30 days.",
  "comment": "Why seven years? GDPR needs a documented lawful basis for that.",
  "thread": [...],
  "category": "Compliance",
  "priority": "high",
  "intent": "question",
  "action_required": true
}

Three of those fields are the whole point.

context_before and context_after are real document text. A comment on its own is usually unreadable — "this is wrong" means nothing. The parser knows the passage a comment is anchored to; the sentences either side of it live in word/document.xml, which the C++ core reads for anchors and then discards. v1.5 reads it back, in Python, on demand and cached per document, so a model sees the comment in the document rather than as a fragment.

thread travels with every comment in it. Redundant on purpose: a retrieval system pulls back single records, and a record that cannot see the reply that resolved it will report an answered question as open.

category, priority, intent and action_required arrive already computed. Rule-based, local, microseconds, and every decision explains itself by naming the words that drove it. It means a prompt can say "here are the 12 compliance comments" instead of spending its budget rediscovering that.

Masking the people first

records = parser.to_llm_dataset(anonymize="strict")

Alice Tester   →  person-71e9b0
"Mail bob.reviewer@acme.example, key AKIAIOSFODNN7EXAMPLE, call +44 20 7946 0958."
               →  "Mail [EMAIL], key [API_KEY], call [PHONE]."

A .docx review is one of the most personal artefacts a company holds: every comment carries a named human, an opinion, a timestamp, and — genuinely, routinely — a key someone pasted in and forgot. Sending that to a hosted model is a disclosure, and "we only sent the comments" is not a defence.

So the redaction is real rather than decorative:

• Checksum-validated, not shape-matched. A sixteen-digit number is not a credit card; one that passes Luhn probably is. An IBAN has to pass mod-97. That is what keeps a redactor from mangling every build number in a document until people stop trusting it.
• Names come from the document, not from a gazetteer. The reviewers are known exactly — the parser read them out of the file — so recall on the people who matter is total, and a section about a Mark is not redacted because someone called Mark reviewed it.
• One person, one stand-in, everywhere. Alice, alice tester, @alice and the author column all become the same Reviewer 1, so "Reviewer 1 raised this three times and Reviewer 2 disagreed" is still a sentence the model can produce.
• The output is checked. After redacting, the result is scanned again with the same detectors, and anything still matching is reported. A redactor that cannot tell you whether it worked is one that will quietly stop working.

And because the mapping stays on your machine, the model's answer can be turned back:

dataset = parser.to_llm_records(anonymize="strict")
answer  = ask_your_model(dataset)                 # sees "person-71e9b0"
print(dataset.anonymizer.deanonymize(answer))     # you see "Alice Tester"

Five presets — names_only, balanced, secrets_only, strict, gdpr — and per-entity control over whether each kind is pseudonymised, redacted, masked, hashed, removed or kept.

Prompts that pack the corpus for you

The hard part of asking a model about a review is not the wording; it is which comments fit, in what order, with what context, serialised so the answer can be joined back to real comment ids. That is the part this library knows:

from docx_comment_parser.llm import create_action_items_prompt

prompt = create_action_items_prompt(dataset.records)
response = client.messages.create(**prompt.to_anthropic(), max_tokens=4096)

Six builders — summary, action_items, resolution, triage, risk, and diff_summary, which takes a v1.4 DiffResult directly. Each one asks for a named JSON schema, requires every claim to cite a comment id, and says out loud when the corpus did not fit rather than presenting a truncated review as a complete one.

Nothing is sent anywhere. There is no client, no key, no network call in this package — the builders return strings.

And the rest

parser.to_embeddings_input()        # one retrieval document per thread, for RAG
parser.export_jsonl("review.jsonl") # the format every batch API takes
parser.to_llm_chunks(max_tokens=100_000)  # never splits a thread in half
parser.privacy_scan()               # what is in here, changing nothing

From a terminal:

docx-comments llm spec.docx --anonymize strict -o review.jsonl
docx-comments prompt spec.docx --kind risk
docx-comments classify spec.docx --priority blocker
docx-comments anonymize contract.docx --scan --fail-on-secret   # a CI gate

Nothing got heavier. The base install still has zero dependencies — the context reader, the classifier and the whole privacy layer are standard library only. Token counting uses tiktoken if you have it and a calibrated estimate that errs high if you do not. Importing the library loads neither layer. The parser is untouched and just as fast; see Performance.

Full Changelog: v1.4.0...1.5.0

v1.3 - Build HTML reports

What's new in v1.3

Everything from v1.2 still works exactly as before. v1.3 adds one thing: you can now hand your review to someone else.

Until now the library gave you data — rows, JSON, a DataFrame. Useful if you write code. Useless if the person who needs to see the comments is a manager, a client, or a lawyer.

parser.export_html_report("review.html")

That writes one HTML file. Double-click it and you get a page with:

• the headline numbers — how many comments, how many resolved, how many still open, who reviewed
• a per-reviewer table showing who is keeping up and who is not
• a chart of comment activity per day and per week
• every conversation, expandable, in reading order
• a search box and filters for author, status, keyword and date

It is one file. No folder of assets, no web server, no internet. Email it, put it on a USB stick, open it on a plane — it works, because the charts, the styling and the comments are all inside the file itself.

If you prefer text you can paste into a pull request or a ticket:

parser.export_markdown_report("review.md")

There is a terminal command too:

docx-comments report contract.docx -o review.html

Nothing got heavier. The base install still has zero dependencies, and importing the library does not load the reporting code at all — you only pay for a report when you ask for one. The parser is untouched and just as fast; see Performance.

1.2

Changelog

v1.2.0 — Structured export and a command-line tool

Public API: backward compatible. Existing code needs no changes. The test_core_regression.py suite exists to prove it.

New — export comments as data

• to_dataframe() (pandas), to_polars() (polars), to_dict(), to_json(), export_csv(), export_json() and to_comments() on both DocxParser and BatchParser.
• A new Comment dataclass: the flat, one-row-per-comment view. Uses __slots__, so 10,000 comments stay cheap.
• Computed columns the parser did not previously expose: thread_depth, root_id, reply_count, document_name, and date_parsed (a real datetime alongside the untouched original string).
• filter_comments() for author / keyword / resolved / thread filtering, shared with the CLI.
• CSV export streams to disk; DataFrame export builds column-first, keeping a 10,000-comment export at ~161 ms.

New — the docx-comments command

• parse, stats, export, unresolved and batch, built with Typer and Rich.
• Filters on every relevant command: --author, --contains, --resolved, --unresolved, --threads-only, --limit.
• unresolved exits 1 when open comments remain, so it works as a CI gate.
• export writes to stdout by default, so it pipes into jq.

New — BatchParser.parsed_files()

Returns the sorted list of files that parsed successfully and still hold results. This is what lets the batch exporters work without being handed the paths again.

Fixed — DocxFileError and DocxFormatError were unreachable

py::register_exception was called with the base class last, and pybind11 tries translators in reverse registration order — so DocxParserError caught every derived type first. Every failure surfaced as DocxParserError, and except dcp.DocxFileError silently never matched, despite being documented.

The three types are now created with PyErr_NewException and a tuple of bases, and dispatched by a single translator with most-derived-first clauses. DocxFileError is now both a DocxParserError and an OSError; DocxFormatError is both a DocxParserError and a ValueError. Code catching any of the old types keeps working; catching the specific types now works too.

Fixed — stale statistics after parsing a comment-free document

DocxParser::Impl::parse returned early when a document had no comments.xml, or an empty one, before reaching compute_stats(). Re-using a parser therefore left the previous document's totals and file_path visible:

parser.parse("has_comments.docx")
parser.parse("no_comments.docx")
parser.stats().file_path        # v1.1.2: "has_comments.docx"  ← wrong
                                # v1.2.0: "no_comments.docx"

Stats are now reset at the start of every parse().

Packaging

• The compiled extension moved from the top level to docx_comment_parser._core, inside a new pure-Python package. import docx_comment_parser as dcp is unchanged.
• Optional extras: [pandas], [polars], [cli], [all], [dev]. The base install still has zero dependencies.
• Ships py.typed and a _core.pyi stub; mypy --strict passes.

Testing

• 188 Python tests at 97% coverage, alongside the existing 66 C++ checks.
• Parser throughput verified against v1.1.2 with interleaved A/B runs: no regression (see Performance).

1.1.2

Changelog

v1.1.2 — Added multiple enconding support

Included multiple text enconding support for a wide range of encondings. Updated unit tests for the new text enconding functionality.

src/xml_parser.cpp — Added a complete encoding transcoding layer before the XML parser:

extract_xml_encoding_decl() — scans the XML prolog for encoding="..."

detect_encoding() — BOM detection (UTF-8/16/32 LE/BE) takes precedence, falls back to the XML declaration

utf16_to_utf8() / utf32_to_utf8() — built-in converters (no platform dependency) with correct surrogate-pair handling

Windows path: win_mbcs_to_utf8() via MultiByteToWideChar + WideCharToMultiByte; maps 60+ encoding names to Windows codepage numbers (all Windows-125x, ISO-8859-1..16, Asian, Cyrillic, Thai, OEM codepages)

Linux/macOS path: iconv_convert() via iconv(3) with the same name alias table; handles E2BIG/EILSEQ/EINVAL gracefully
transcode_to_utf8() — public entry point, called at the start of sax_parse() so all parsing paths (DOM and SAX) go through it automatically

include/xml_parser.h — Exposed transcode_to_utf8() as a public API with full docstring.

CMakeLists.txt — Added find_package(Iconv QUIET) for non-Windows targets; links Iconv::Iconv only when it's a separate library (not built into libc).

tests/test_docx_parser.cpp — Added 7 encoding tests (66 total, all green):

test_encoding_utf8_bom — UTF-8 BOM is silently stripped

test_encoding_utf16le / test_encoding_utf16be — BOM-detected UTF-16

test_encoding_utf32le — BOM-detected UTF-32

test_encoding_windows1252 — encoding="windows-1252" with ç, é, ä in content

test_encoding_iso8859_1 — encoding="ISO-8859-1" with é, ñ

test_encoding_numeric_entities — 中 (Chinese) and é (é) references

Release v1.1.1

updated python-publish workflow for multi envs

1.1.1

Release 1.1.1 - built with Mingw64 (MSYS) under Python 3.14 and Windows 11 x64