feat(chapters): a chunk is a markdown block, not a blank-line paragraph
Phases 7 and 8 of docs/chapter-uploads.md. Measured against markdown-it -- the parser mantra.press translates with -- this app's blank-line split disagreed on ten cases in eleven, and on the text as well as the boundaries. markdown-it's content is a leaf block's inline source with the BLOCK marker removed, so `# Chapter one` is "Chapter one" while inline markup is kept. That is the half that matters to a translator: handed "# Chapter one" they can break the document by mistyping the hash, and handed "Chapter one" they cannot. So splitBlocks adopts it, boundaries and stripped text both, with three deliberate departures. Code blocks yield no chunk -- markdown-it's fence and code_block carry content, so mantra.press asks a translator to translate `code();` and the group to sign it as prose. A table is one chunk of its own source rather than one per cell, because four envelopes to carry one word each is a poor trade and a table's alignment is structure per-cell translation cannot preserve. And a link reference definition yields nothing, which is free: it is markdown-it's own behaviour with the `reference` rule left enabled, where mantra.press disables it -- reaching for a block rule to stop links rendering -- and makes a translatable chunk of a URL. Hand-rolled, not org.jetbrains:markdown, for one reason specific to this function: its output is hashed into ids a quorum signs, so a version bump that moved one boundary would silently re-split every chapter proposed afterwards and nothing would notice. What pins it instead is thirty golden fixtures generated FROM markdown-it by docs/scripts/markdown-block-fixtures.js, which also encodes the three departures so the Kotlin test is plain equality. MarkdownProducer is new and is what makes stripping markers safe: nothing in the app could put a `#` back before. It substitutes by index, which is strictly better than the thing it mirrors -- mantra.press matches on text, so a chapter with two identical paragraphs puts the first one's translation into both. The scanner segments every line rather than only the chunks, which is what makes the byte-for-byte round trip exact by construction rather than by getting every markup rule right. ProposedEvent.summarize stops re-splitting a chapter to count its chunks and takes the count from the batch. That number is a count of the batch dressed as a fact about the text: across a splitter change the figure in the approval gate would disagree with the events being approved, and it re-scanned every open proposal's prose on every emission of the room's list. Phase 2's constant was flagged as provisional pending this, and measured it did not move: prose separated by blank lines splits the same under both rules, so the fixture pinning 24 KiB over forty paragraphs is the shape this does not touch. The narrower claim holds -- a heading- or list-heavy chapter now reaches both caps sooner, visibly, in the form's own counter. 987 tests, no failures; m3Audit's budgets all met. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -890,6 +890,37 @@ Either take this phase ahead of Phase 2, or treat Phase 2's constant as
|
||||
provisional and re-run its encoded-proposal measurement here. The second is
|
||||
cheaper and the measurement is the same test either way.
|
||||
|
||||
**Measured, and it did not move.** `ChapterProposalSizeTest`'s pinned boundary —
|
||||
24 KiB of markdown over forty paragraphs — is identical before and after, for a
|
||||
plain reason the warning above missed: prose separated by blank lines splits the
|
||||
same way under both rules, so the fixture that pins the number is exactly the
|
||||
shape the change does not touch. What *is* true is the narrower claim: a
|
||||
list-heavy or heading-heavy chapter now reaches both caps sooner than the same
|
||||
bytes of plain prose would, which is a real effect on real chapters and shows up
|
||||
in the form's own counter rather than in a constant.
|
||||
|
||||
### What else came out of it
|
||||
|
||||
Three things the plan had not separated out:
|
||||
|
||||
- **`splitBlocks` returns blocks, not strings.** The marker has to be kept
|
||||
*somewhere* for the producer, and the chapter's `original_text` is the wrong
|
||||
place to look it up from — matching text back to its marker is the
|
||||
string-equality bug this phase fixes elsewhere. So `MarkdownBlock(kind,
|
||||
markup, text, indent)`, and `splitChunks` for the callers that only want the
|
||||
words.
|
||||
- **The scanner segments the whole text, not just the chunks.**
|
||||
`MarkdownSegment` covers every line, with `block = null` for a fence, a
|
||||
thematic break, a reference definition or a blank line. That is what makes the
|
||||
byte-for-byte round trip possible at all: an untranslated chunk is reassembled
|
||||
from its own source lines rather than re-rendered from its parsed form, so the
|
||||
round trip is exact by construction rather than by getting every markup rule
|
||||
right.
|
||||
- **A blockquote containing a blank `>` line is two chunks, not one.**
|
||||
markdown-it reads it as two inline runs, so the scanner re-segments it per
|
||||
inner paragraph — and the fixture for it is the one that would have caught
|
||||
this being missed.
|
||||
|
||||
### Tests
|
||||
|
||||
`commonTest`, all of it pure:
|
||||
@@ -900,10 +931,13 @@ cheaper and the measurement is the same test either way.
|
||||
block yield no chunk, a table yields one, a reference definition yields none.
|
||||
- inline markup survives: `*italic*` and `[text](url)` are in the chunk's text.
|
||||
- `MarkdownProducerTest`: round-trip with no translations reproduces the input
|
||||
byte for byte (the assertion that catches every markup-reassembly bug at
|
||||
once), and a chapter with two identical paragraphs translated differently
|
||||
comes back with both translations in the right places — the case mantra.press
|
||||
gets wrong.
|
||||
byte for byte **over all thirty fixtures** (the assertion that catches every
|
||||
markup-reassembly bug at once), and a chapter with two identical paragraphs
|
||||
translated differently comes back with both translations in the right places —
|
||||
the case mantra.press gets wrong. Plus the markers going back on: hashes, list
|
||||
markers and indents, a blockquote re-quoted line by line so a translation that
|
||||
grew a line is still quoted, and a setext underline re-cut to the length of
|
||||
the translated title.
|
||||
- `ChapterChunkSplitTest` keeps its existing round-trip and id-stability
|
||||
assertions, now with the converter's output shape as a fixture.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user