mantra/mantra-kmp
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: a phased plan for chapter uploads, group-signed

Browse Source 
Mirrors mantra.press's multi-file chapter upload under this app's rule that
the group, not the uploader, authors the work.

The web app's shape -- N files in, N chapters out, one transaction, one
redirect -- is the one shape this app cannot have. A signing session is one
quorum, all-or-nothing, 64 items and about 64 KiB on the wire, so twelve
chapters in one batch does not fit, should not fit, and would ask every
member to read twelve chapters in a single approval gate. One upload is N
sessions, one per chapter, and that cost belongs in front of whoever is
picking files.

Reading both trees turned up four things the single-chapter path gets away
with at N = 1 and an upload does not: the index is read from rows and no
proposed chapter is a row yet, so twelve proposals all sign the same index;
nothing checks the proposal's size, which puts a silent ceiling near 28 KiB
of markdown; there is no file picker in the app at all; and translation
scaffolding opens a session per chapter where one batch would do.

The splitter is measured rather than assumed. Run side by side, markdown-it's
block tokens and this app's blank-line rule disagree on ten cases in eleven,
and on the text as well as the boundaries -- markdown-it strips the block
marker, so `# Chapter one` is `"Chapter one"`. Phases 7 and 8 adopt it, with
three deliberate departures (no chunk for code blocks, one chunk per table,
none for a link reference definition) and a producer, which the app has never
had.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kgothatso Ngako
2026-10-02 18:29:04 +02:00
parent 76055bc0a8
commit 6ab4f0e557
2 changed files with 926 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/README.md
View File

@@ -21,6 +21,7 @@ silent, or a decision that looked arbitrary and was not.
| [multiple-profiles.md](./multiple-profiles.md) | several profiles on one device — why a profile is a credential and a seed is a wallet attached to one, why a switch is a restart rather than a swap, the two entrances the app lacks, and the inbox a switch would silently lose |
| [jvm-target.md](./jvm-target.md) | what desktop support cost, phased — why the native chain was already done, why an empty source set in our phoenix fork was the real blocker, and why DAO tests need none of it |
| [npub-profile-preview.md](./npub-profile-preview.md) | showing the person before a direct message is started from a pasted npub — why the preview is a screen keyed by public key, what "found" means when a row can be a placeholder, and the phase that makes the button honest about a key package that never comes |
| [chapter-uploads.md](./chapter-uploads.md) | picking files off a device and having the group sign each one in as a chapter — why N files can never be one signing session, and the four things the single-chapter path gets away with |
| [material-design-conformance.md](./material-design-conformance.md) | what the M3 foundations actually require, measured against all 43 screens — the colour pairing that renders the app's own proposals invisible, and eight phases that put the decisions back in the theme |
| [curated-to-mantra.md](./curated-to-mantra.md) | pulling the Curated fork's thirty-nine commits back under Mantra's names — which lines of work to take, the three decisions, and a measured way to replay a twice-rebranded history without touching seven hundred files by hand |
| [curated-to-mantra-profiles.md](./curated-to-mantra-profiles.md) | the second pull: the fork's several-profiles line, ten commits and its first schema migration — which of it is a fix Mantra has today, who allocates a schema version number when two trees share one history, and the one conflict, which was Mantra's own |
@@ -72,3 +73,12 @@ smallest of the plans: it changes the entrance to the direct-message flow and
nothing past it, so read it with `StartDirectMessageToNpubOrNip05Dialog` and
`ChatRoomMessagingViewModel.initiateNewChat` open, its four decisions before its
four phases, and its table of where the build chose differently first.
The chapter uploads note is a phased plan that has not been built, and is the
only one that starts from another codebase: it mirrors the mantra.press web app's
multi-file upload under this app's rule that the group, not the uploader, is the
author. Read the batch-signing note first — its three limits are the whole shape
of this one — and read its four-defects section even if the feature is never
built, because each of those is wrong in the single-chapter path today. Its two
splitter phases are separable from the rest and worth reading on their own: what
a chunk is, measured against the parser the web app translates with rather than
assumed.

916
docs/chapter-uploads.md Normal file
View File

@@ -0,0 +1,916 @@
# Chapter uploads
Picking files off a device and having a group sign each one into an artifact as a
`MantraChapter` with its chunks — the way
[mantra.press](https://code.sigidli.com/mantra/mantra.press) does it over one
HTTP POST, except that here nobody is authoring anything on their own.
**Not built.** This is a plan. The constraint that shapes every phase is stated
first, because the web app's shape — N files in, N chapters out, one
transaction, one redirect — is the one shape this app cannot have, and every
phase below is a consequence of that rather than a feature.
---
## What mantra.press does
One route, `POST /v/:id/chapters/upload`, in
`server/router/versions/index.js`. A multipart form with `multiple` on the file
input (`server/views/chapters-upload.pug`), and for each file:
```js
const fileText = file.data.toString()
const chunks = md.parse(fileText).filter(t => t.content).map(t => t.content)
db.Chapter.create({
creatorId: userId,
name: file.name,
originalText: fileText,
wordCount: chunks.reduce(wordCounter, 0),
characterCount: chunks.reduce(characterCounter, 0),
index: chapterIndex,
artifactVersionId: artifactVersion.id,
chunks: chunks.map((chunk, index) => ({ ... })),
}, { include: [{ association: db.Chapter.Chunks }] })
```
Six decisions in there worth naming, because this plan either keeps or
deliberately drops each one:
| mantra.press | here |
|---|---|
| the chapter's name is the **filename** | kept as the default, editable before proposing (§ Phase 4) |
| the index is `chapters.length + 1`, re-read from the version | allocated once for the whole upload, counting proposals as well as rows (§ Phase 3) |
| chunks are **markdown-it block tokens**, with the block marker stripped | adopted, with three deliberate departures (§ Phase 7, built in § Phase 8) |
| the uploader is the author (`creatorId`) | the **group** is the author, once a quorum signs |
| all files succeed or the request 500s | one session per chapter, each failing on its own (§ The constraint) |
| `file.data.toString()`, whatever the bytes are | UTF-8 decoded strictly, non-text refused (§ Phase 1) |
The web app has two small bugs in that loop that are worth knowing about so they
are not faithfully reproduced. `request.files.chapters.length > 0` is `undefined`
for a single file, so one file always takes the `else` branch — harmless, but it
means the multi-file path has had less use than it looks. And
`Number(chapterIndex) + Number(index)` starts from a `chapterIndex` that is
already `length + 1`, so a second file lands at `length + 2` and the first
upload into an empty version starts at index 1 while the add-chapter form starts
at `length`. Indices here are 0-based (`ChapterEvent.build(index = …)`, written
by `AddChapterViewModel` as `getChaptersForArtifactVersion(...).size`), and this
plan keeps them 0-based.
## What is already here
Nearly all of it, for one chapter at a time. This plan adds no event kind, no
tag and no schema version to the chapter path.
- [`ChapterEvent`](../composeApp/src/commonMain/kotlin/press/mantra/compose/nostr/nip30303/ChapterEvent.kt)
(30302) carries the whole markdown in an `original_text` tag;
[`ChunkEvent`](../composeApp/src/commonMain/kotlin/press/mantra/compose/nostr/nip30303/ChunkEvent.kt)
(30303) carries one paragraph, and `ChunkEvent.splitOf(chapter)` cuts the
chunks from a chapter *after* it is authored, which is the only place the
chapter id exists to be named.
- [`AddChapterViewModel.addChapter`](../composeApp/src/commonMain/kotlin/press/mantra/compose/ui/view/model/AddChapterViewModel.kt)
proposes the chapter and its chunks as one batch via
`proposeSigningBatch(lead = …, dependents = ChunkEvent::splitOf)`, then opens
a second session scaffolding the chapter into every translation of the
version.
- `ChatMessage.applySignedEvent` has both branches (`ChapterEvent.KIND`,
`ChunkEvent.KIND`), writes `groupSignedEventId` provenance, and
`GroupSignedEvent` keeps the event as the group signed it.
- [`ChapterBatchProposalJvmTest`](../composeApp/src/jvmTest/kotlin/press/mantra/compose/managers/ChapterBatchProposalJvmTest.kt)
and `SignedChapterTest` already cover the single-chapter batch end to end.
So the work is **not** "make a chapter group-signed". It is: get bytes off three
platforms, turn a pile of files into a reviewable list of proposals, and deal
with the four things a single hand-typed chapter gets away with and a twelve-file
upload does not.
---
## The constraint
A signing session is **one quorum, all-or-nothing, at most 64 items, and about
64 KiB on the wire**.
Three numbers, from three places, and the plan is shaped by all three:
- `FrostSigningManager.MAX_BATCH_SIZE = 64`, checked in `proposeSigningBatch`
*and* independently in `acceptProposal` — see
[frost-batch-signing.md](./frost-batch-signing.md).
- *All-or-nothing*: "a batch is only as available as its worst item. If any
event cannot be aggregated the session fails and none of them are applied."
- *Bytes*: the proposal carries `k` whole events, which is the term that binds.
`ChronicleEvent.MAX_PAGE_BYTES = 64 * 1024` is this codebase's measured proxy
for what a kind:445 clears "with room for the MLS framing and the NIP-44
expansion that sit outside it".
### The shape that follows: one session per chapter
The web app's single transaction has no analogue. Twelve chapters in one batch
is wrong three times over:
1. **It does not fit.** A chapter takes one item and its paragraphs take the
rest (`MAX_CHUNKS_PER_CHAPTER = 63`). Two chapters of thirty paragraphs each
already overflow.
2. **It should not fit.** A batch is only as available as its worst item, so one
chapter that fails to reach quorum would take the other eleven with it. A
book is precisely a set of things that do *not* need to arrive together.
3. **It is the wrong prompt.** Every member would be asked to read twelve
chapters' full text in one approval gate, and the gate's own rule is that it
"must show every event, not a count".
So: **one upload is N sessions, one per chapter.** That is the honest cost and
it has to be in front of whoever is picking files, not discovered afterwards —
N proposals, each needing a quorum, arriving in the room at once.
What it buys: a chapter that fails is re-proposable on its own, chapters land
and become readable as each quorum arrives rather than all at the end, and the
approval gate stays one chapter long.
---
## Four things the single-chapter path gets away with
Each is a real defect today, each is invisible at N = 1, and each is a phase.
**The index is read from rows, and no chapter is a row until its quorum lands.**
`addChapter` does `getChaptersForArtifactVersion(artifactVersion.id).size` at
propose time. Propose twelve chapters back to back and all twelve read the same
size, so all twelve are signed with the same index. The same collision already
exists between two members adding a chapter at the same moment; an upload just
guarantees it. **Phase 3.**
**There is no byte check at all.** The paragraph-count cap is enforced and shown
as you type; the size of the proposal is not checked anywhere. A proposal
carries the chapter (whole markdown) *plus* a chunk per paragraph (the same
words again), so it is roughly 2× the text before framing — which puts the
ceiling near 28–30 KiB of markdown, well inside the range of a real book
chapter. Today that is an `IllegalArgumentException`-free failure: the proposal
goes out, the group event is over size, and the session stalls. **Phase 2.**
**There is no file picker in the app.** Nothing in `composeApp/src` references
`rememberLauncherForActivityResult`, `OpenDocument`, FileKit or any equivalent;
the only platform `expect` declarations are for the database builder, context,
theme and motion. Three targets need it: `androidTarget`, `jvm`, and the two ios
targets gated behind macOS in `composeApp/build.gradle.kts`. **Phase 1.**
**Translation scaffolding fans out per chapter.** `addChapter` opens a second
session per chapter for the translation chapters. Twelve chapters in a group
with three dialects is twelve chapter sessions plus twelve scaffolding sessions
= twenty-four proposals, where the scaffolding is 36 tiny events that fit in one
batch. `TranslationScaffold.chaptersOf` already takes a *list* of chapters.
**Phase 5.**
---
## Phase 1 — a file in, as text
**Two days, most of it the three actuals. No protocol change, no new event.**
Ship the picker on its own, before anything knows about multiple chapters, and
prove it on all three platforms. It lands as a **"load from file" button on the
existing `AddChapterScreen`** that fills `originalTextFieldState` and the name
field: one file, straight into the form that already works, so the picker is
exercised and reviewable without the batching argument attached to it.
### The API
```kotlin
// commonMain
data class PickedFile(val name: String, val bytes: ByteArray)
@Composable
expect fun rememberFilePicker(onPicked: (List<PickedFile>) -> Unit): FilePicker
interface FilePicker { fun launch(multiple: Boolean) }
```
Returned as bytes rather than a path or a stream. A path is meaningless on iOS
once the picker's security scope closes, and a stream would make the decode
decision below somebody else's problem on a background dispatcher. Chapters are
tens of kilobytes; this is not where to be clever.
### The three actuals, or one dependency
**Built with the three actuals, not FileKit.** The plan recommended the
dependency and the implementation went the other way, for three reasons that
only became clear with the tree open: this is the app's only dialog, the three
calls are a dozen lines each, and the project already keeps four platform seams
of exactly this shape (`PlatformContext`, `PlatformDatabaseBuilder`,
`platformReducedMotion`, `platformThemeContrast`) — so the seam was free and the
library's Compose-version coupling was not. The ios target cannot be compiled on
a linux host either way, so a dependency would not have bought verification
where verification is actually missing.
**The three actuals**, which were a day's work and no mystery:
| target | call |
|---|---|
| `androidMain` | `rememberLauncherForActivityResult(ActivityResultContracts.OpenMultipleDocuments())` |
| `jvmMain` | `java.awt.FileDialog` with `isMultipleMode = true` |
| `iosMain` | `UIDocumentPickerViewController(forOpeningContentTypes:)`, `allowsMultipleSelection = true` |
Android needs no manifest permission for `OpenDocument` — it is a SAF handoff.
iOS needs the document types declared; `.md` has no registered UTI, so
`UTTypePlainText` plus `UTTypeData` and sniff afterwards.
Three things the build added that the plan had not thought of:
- **Both android contracts are registered**, `OpenDocument` and
`OpenMultipleDocuments`, and `launch(multiple)` picks. Registering only the
multiple one would offer a six-file picker to a form that keeps one file.
- **Every actual stops reading one byte past the cap**, not at it, so an
over-size file arrives over-size rather than silently truncated — a truncated
chapter decodes cleanly and would be proposed with its ending missing.
- **The dialog title is a parameter**, read from the catalogue by the caller,
because `stringResource` is composable and a `FileDialog` is not. Only desktop
shows it; android's and iOS's pickers title themselves. Taken on all three so
that the one platform showing text to a reader gets it from the catalogue like
every other string in the app.
**The ios actual is unverified.** The two ios targets are declared only on macos,
so `FilePicker.ios.kt` is not compiled on a linux host and has never run. It is
written to the shape of the eight other `.ios.kt` actuals beside it and says so
in its own KDoc. A stub returning no files would have been worse: it reads as a
working picker that a reader's chapters vanish into.
### Decoding, which is where the web app is wrong
`file.data.toString()` on a `.docx` produces a chapter of zip header garbage,
and the group would be asked to sign it. Here:
```kotlin
// commonMain
object ChapterText {
const val MAX_FILE_BYTES = 1 * 1024 * 1024
sealed interface Result {
data class Text(val markdown: String) : Result
data object NotText : Result
data object TooLarge : Result
}
fun decode(bytes: ByteArray): Result
}
```
Four rules, each with a reason:
- **Strict UTF-8.** `decodeToString(throwOnInvalidSequence = true)` and refuse on
failure. A replacement character signed into an `original_text` tag is a
permanent `U+FFFD` in the group's record.
- **Strip the BOM.** `U+FEFF` at the head of the first paragraph is invisible in
the form and changes the chunk's text, so it changes the chunk's id.
- **Normalise CRLF to LF before hashing.** `Markdown.PARAGRAPH_BREAK` is
`(?:\r?\n){2,}`, so splitting already tolerates CRLF — but the `original_text`
tag would carry the `\r`s, and the same chapter uploaded from Windows and from
Linux would be two different events. Normalise once, at the door.
- **Sniff, do not trust the extension.** A NUL byte in the first 8 KiB means
`NotText`. Cheap, catches every office format and every image.
`MAX_FILE_BYTES` is a crash guard, not the real limit — the real one is Phase 2
and is twenty times tighter.
**Tests** (`commonTest`): a `ChapterTextTest` over valid UTF-8, invalid UTF-8,
BOM, CRLF-vs-LF producing identical text, a NUL-bearing payload, and the size
cap. All pure; no picker involved. 13 of them, plus 9 in a `ChapterNamingTest`
that the plan had folded into Phase 4 and which belongs here — the default name
is derived from the file, so it arrives with the file.
`ChapterNaming` turned out to deserve its own object rather than a line in the
upload screen. The web app has two answers and uses a different one per route —
`file.name` on the upload, `chunks[0].trim()` on the paste form — and both are
right some of the time. The rule built here is: the filename with its extension
and ordinal prefix removed, falling through to the text's first heading and then
its first line when the filename is only an ordinal. `01.md` is not a chapter
title; `03 - The Wreck.md` is.
---
## Phase 2 — the byte cap, in front of the user
**Half a day. A fix to the path that exists, and a prerequisite for Phase 4.**
A chapter whose markdown is over roughly 28 KiB cannot be proposed today and
nothing says so. Make the limit explicit and check it where the paragraph cap is
already checked.
```kotlin
// ChapterEvent companion, beside build()
/**
* The most markdown a chapter may carry.
*
* A chapter is proposed with a chunk per paragraph, so the proposal carries its
* text roughly twice -- whole in the chapter's `original_text` tag, and again
* split across the chunks -- before any framing. The group event that carries
* the proposal is sized by ChronicleEvent.MAX_PAGE_BYTES; half of it, less room
* for k event envelopes, is what is left for prose.
*
* Measure it against a real proposal rather than trusting this arithmetic, the
* way MAX_PAGE_BYTES itself says to.
*/
const val MAX_ORIGINAL_TEXT_BYTES = 24 * 1024
```
24 KiB rather than 28: the envelopes are not free (an id, a pubkey and a
signature are ~370 bytes before an event says anything, times `k`), and a cap
that is comfortably under the wire limit fails in the form instead of on the
wire. It should be arrived at by measuring an encoded proposal for a 63-chunk
chapter, not by this paragraph.
**And the number is provisional until Phase 8 lands.** Block splitting produces
more chunks than blank-line splitting, so the same prose carries more envelopes;
whichever of the two phases goes second re-runs the measurement.
Where it goes:
- `AddChapterViewModel.addChapter` — beside the
`paragraphs.size > MAX_CHUNKS_PER_CHAPTER` guard, same early `onFailure`.
- `AddChapterScreen` — the counter line already reads
`"$wordCount words · $characterCount characters · $paragraphCount chunks of N"`;
it gains the byte figure against the cap and turns `error`-coloured on the
same rule the chunk count does.
- The two existing hardcoded English literals on that screen (the no-shared-key
message and the too-many-chunks message) move into
`composeApp/src/commonMain/composeResources/values/strings.xml` while the file
is open. They are UI text in a composable and `CLAUDE.md`'s budget is zero
title-case strings with literals reported; a third one added by this work
would be the wrong direction.
**Test:** extend `ChapterBatchProposalJvmTest`, which already asserts the
paragraph cap's boundary (`MAX_BATCH_SIZE - 1` proposes, `MAX_BATCH_SIZE`
throws). Add the byte boundary the same way: one chapter at the cap proposes,
one a byte over does not. And one that measures the *encoded proposal* for a
cap-sized, 63-chunk chapter and asserts it is under `MAX_PAGE_BYTES` — that is
the assertion that makes the constant honest, and the only one that will catch
it drifting when a tag is added to either kind.
---
## Phase 3 — an index that counts proposals, not rows
**A day. A fix to the path that exists.**
The index is signed into the event and cannot be recounted on arrival — the
comment in `addChapter` says exactly why: "a device applying it cannot recount
from a version other members may have added to in a different order". So the
number has to be right at proposal time, and "rows in the table" is not it,
because a proposed chapter is not a row for as long as the quorum takes.
The information is already available. `FrostSigningRepository` exposes
`observeSessionsForChatRoom`, `getItems(sessionId)` and `unsignedEvents(items)`,
and the last of these exists precisely for this: "A caller wanting an id … needs
it while the first session is still running, and the id is settled at proposal
time."
```kotlin
// MantraRepository, or a small ChapterIndexing object beside TranslationScaffold
/**
* The index the next chapter of [artifactVersionId] takes, counting chapters
* the room has proposed but not yet signed.
*
* Rows alone are wrong by however many proposals are in flight, which at N = 1
* is almost never and during an upload is always. A proposed chapter's index is
* already fixed -- it is signed into the unsigned event the session will sign --
* so it counts.
*/
suspend fun nextChapterIndex(chatRoomId: String, artifactVersionId: String): Int
```
Implemented as: rows for the version, plus the unsigned `ChapterEvent`s of every
open session in the room whose `artifactVersionId()` matches, de-duplicated by
event id, and `maxOf(index) + 1` rather than a count — a count is wrong the
moment a gap exists, and after a chapter fails to reach quorum a gap is exactly
what there is.
Then `AddChapterViewModel` calls it instead of `.size`, and the upload allocates
`base + i` for its i-th file **once**, before proposing anything, rather than
re-asking between sessions. Re-asking would work and is worse: it makes the
upload's numbering depend on how fast the sessions are written.
**What this does not fix.** Two *devices* proposing at the same moment still
collide — neither sees the other's proposal until it arrives. That is a
distributed-allocation problem and the honest answer is that index collisions
are tolerable: the index orders reading, two chapters at index 4 sort
arbitrarily between themselves and nothing else breaks. Worth a sentence in the
code so the next reader does not take the local fix for a global one.
**Test** (`jvmTest`, beside `ChapterBatchProposalJvmTest`): propose three
chapters back to back without completing any session, and assert the three
signed indices are 0, 1, 2 — which is the test that fails today.
---
## Phase 4 — the upload screen
**Three days. The feature.**
`UploadChaptersRoute` / `UploadChaptersUIState` / `UploadChaptersViewModel` /
`UploadChaptersScreen`, registered in `MantraNavHost`, reached from
`ArtifactDetailScreen` next to the existing "add chapter" `TextButton`
(`ArtifactDetailScreen.kt:165`), with the same `enabled = versions.isNotEmpty()`
condition.
### It is a review step, not a submit button
This is the one place the plan departs visibly from the web app, and the reason
is that a signature is not a submission. mantra.press can afford to post twelve
files and redirect, because what it created is the uploader's and the uploader
can fix it. Here, twelve files become twelve requests for the group's own
signature, and anything wrong in them — a filename for a title, a file that
decoded to nothing, a chapter over the byte cap — is wrong in something a quorum
is about to put its name to.
So picking files does not propose anything. It produces a list, one row per
file:
| shown | from |
|---|---|
| name, **editable**, defaulting to the filename without its extension | `PickedFile.name` |
| the index it will take | `base + i` from Phase 3 |
| chunks, against `MAX_CHUNKS_PER_CHAPTER` | `Markdown.splitParagraphs(...).size` |
| words, characters, bytes against `MAX_ORIGINAL_TEXT_BYTES` | `Markdown`, Phase 2 |
| a per-row problem, when there is one | below |
| a remove action | — |
Four per-row problems, each blocking that row only — a bad file should not
cost the other eleven:
- not text, or not UTF-8 (`ChapterText.Result.NotText`)
- empty after trimming, which is a chapter with nothing to translate
- over the chunk cap, with the advice to split it
- over the byte cap
- **duplicate**: its normalised text matches an existing chapter of this version
or another row in this list. Not blocking — a book can repeat a page — but
warned, because the common cause is picking the same directory twice, and two
chapters with the same text are two different event ids that no later
de-duplication can merge.
A per-file name field rather than one derived from the text, because the
filename is right often enough to be the default and wrong often enough
(`chapter-01.md`, `ch1_final_v2.md`) that it must be fixable before a quorum
sees it. Where it is empty, fall back to the first markdown heading in the text,
then to the first paragraph trimmed to a line — which is what the web app's
*other* route does (`name: chunks[0].trim()`), and is better than either
extreme.
### Proposing
The FAB — `ExtendedFab`, as every primary action in the app is — reads
"propose N chapters", is disabled when any included row has a blocking problem
or `canSign` is false, and on tap:
```
for each row, in index order:
propose(chapter + chunks) // the existing batch, one session per chapter
record the session id
then:
propose the scaffolding for all N chapters at once // Phase 5
then:
navigate to the proposal list, not to a signing screen
```
**Sequentially, not concurrently.** Each `proposeSigningBatch` writes a session
and publishes group events; N at once is N interleaved MLS epochs, and
[mls-skipped-keys.md](./mls-skipped-keys.md) is about what a group event that
arrives a moment out of order costs. The upload is not in a hurry.
**Partial success is the expected outcome, not an error.** If the fourth
proposal fails, the first three are real proposals that the group can still
sign, and nothing can retract them. So the screen does not roll back and does
not navigate away on a failure: it marks the rows that were proposed, leaves the
rest listed with their reason, and offers "propose the remaining". Phase 3's
`nextChapterIndex` means a retry re-reads the base and does not re-use the
indices already proposed.
**It navigates to `ProposalListScreen`, not to `FrostSigningRoute`.** The single
chapter path goes to the signing screen because there is one session to watch,
and the reasoning in `addChapter` is sound for it ("landing on the list it is not
in would read as a failure"). At N there is no one session, and the list is the
screen that exists for "two proposals need a place that shows both". A snackbar
via `rememberNotifier` carries the count, read above the handler as `CLAUDE.md`
requires.
### Scope, and what it must say out loud
Before the picker opens, the screen says what an upload will cost the room: one
proposal per file, each needing a quorum. This is not a nicety. The app's
approval gate is deliberately per-session and deliberately shows every event,
and twelve of them arriving at once is a real imposition on every other member
that the person picking files is the only one in a position to weigh.
A soft cap belongs here too — refuse to propose more than, say, 20 files in one
go, and say why. Not a protocol limit; a limit on how much of the room's
attention one tap may spend.
### M3, per `CLAUDE.md`
`NavigateBackButton` in the app bar's leading slot (it is a pushed screen);
`readableContent()` on the content root; all four states, with `EmptyState`
carrying its own message for "no files picked yet" and `ErrorState` taking a
real `onRetry`; `ScreenStateTransition` wrapping the `when` as the composable's
whole body; spacing from `MaterialTheme.spacing`; every string in `strings.xml`
in sentence case with `%1$s`/`%1$d` arguments; every icon with a decided
`contentDescription`. Finish with `./gradlew :composeApp:m3Audit`.
---
## Phase 5 — scaffold once, not once per chapter
**Half a day. A fix to the path that exists, and an N-fold saving.**
`AddChapterViewModel.scaffoldIntoTranslations` is right about needing its own
session — a chapter's paragraph budget should not be spent on the group's
dialects — and wrong about needing one *per chapter*, which only ever happens
during an upload.
`TranslationScaffold.chaptersOf(translationArtifactVersionIds, chapters,
createdAt)` already takes a list and already builds the full cross product in
translation-then-reading order. So lift the scaffolding out of the per-chapter
loop: collect each chapter's `SourceChapter(id, index)` from its session's
unsigned lead event, and after the last chapter is proposed, propose the whole
cross product in `chunked(MAX_BATCH_SIZE)` batches — which is what `addChapter`
already does across *translations* and what
`TranslationArtifactVersionDetailViewModel` does across chapters.
Twelve chapters × three dialects: from 12 scaffolding sessions to 1.
Keep the existing failure policy verbatim, including its reasoning: scaffolding
failure is logged and swallowed, because the chapters are the thing that was
asked for, and a scaffolding row naming a chapter that never reached quorum
fails a foreign key, is logged by `applySignedEvent`, and never becomes a row.
The catch-up on the translation fills the gap afterwards. At N that argument gets
*stronger*, not weaker — one cross-product batch covering twelve chapters is
only as available as its worst item, and some of those chapters may genuinely
fail.
`AddChapterViewModel` keeps its one-chapter call into the same helper, so there
is one piece of code deciding what a chapter's scaffolding is.
---
## Phase 6 — N proposals, in the room
**A day. No new screens.**
The transcript and the proposal list were built for two concurrent sessions —
`ChatMessage.frostSigningSessionId` exists because matching a request against
"any later line of the fulfilling type" broke the moment a chapter and its
scaffolding were open together. Twelve is the same problem at a scale that will
find whatever that fix missed.
What to check rather than assume:
- `ProposalListScreen` with twenty open sessions: the ones waiting on the reader
gather at the top, and `ProposedEvent.summarize` gives a chapter a name rather
than "Event of kind 30302". Chapters are summarised today; confirm the
summary is the chapter's *name*, since twelve rows reading "Chapter" are
twelve rows a member cannot choose between.
- Twelve "Review" lines in the transcript, each resolving its own session.
`answeredRequests`/`settledRequests` read `frostSigningSessionId` when both
rows have one; twelve is where a fallback-to-clock path would show.
- Twelve approval gates. No batching of the gate — that is the one thing the
batch-signing note refuses, and for the same reason: "a member who cannot
scroll the whole list should not be able to approve it". What can be done is
make the *list* cheap to work through, which is the point above.
No new `ChatMessage` type, no new set to add to.
---
## Phase 7 — the splitter, which is a decision about translation
**Half a day, and it is a measurement rather than code.**
mantra.press splits with markdown-it block tokens; this app splits on blank
lines.
```js
// mantra.press: one chunk per block-level token that carries content
md.parse(text).filter(t => t.content).map(t => t.content)
```
```kotlin
// here: one chunk per blank-line-separated block
text.split(Regex("(?:\\r?\\n){2,}")).map { it.trim() }.filter { it.isNotEmpty() }
```
They were assumed to agree on ordinary prose and to disagree only on converted
files. Run side by side over a corpus — the reference parser as
mantra.press configures it, against the Kotlin rule — they disagree on **almost
everything**, and on two separate axes:
| input | markdown-it | `splitParagraphs` |
|---|---|---|
| `# Chapter one` ⏎ `First paragraph.` (the converter's shape) | 3 chunks, heading apart | 2 chunks, heading glued to the prose |
| `# Chapter one` ⏎⏎ `First paragraph.` | `"Chapter one"` | `"# Chapter one"` |
| a tight three-item list | 4 chunks — the lead-in and each item | 2 — the lead-in and the whole list |
| `> line one` ⏎ `> line two` | `"line one\nline two"` | `"> line one\n> line two"` |
| a fenced code block | `"code();\n"`, fence markers gone | ``"```\ncode();\n```"`` |
| a 2 × 2 table | 4 chunks — one per cell | 1 — the table's source |
| `---` between two paragraphs | 2 chunks, the rule is not one | 3 — the rule becomes a chunk |
| `Title` ⏎ `=====` (setext) | `"Title"` | `"Title\n====="` |
| a nested list | 2 chunks, one per item | 1 |
| a paragraph with a lazy continuation line | 1 chunk | 1 chunk — agree |
One row in eleven agrees. The table is reproducible in a dozen lines of node
against `mantra.press/node_modules/markdown-it`, and Phase 8 turns that into
checked-in fixtures.
**Axis one is boundaries**: how many chunks, which is the obvious half.
**Axis two is the text itself**, which is the half that was missed: markdown-it's `content`
is the leaf block's inline source with the *block* marker removed — no `#`, no
`> `, no `- `, no fence, no setext underline — while inline markup (`*italic*`,
a link's brackets) is kept verbatim. So even the rows where the counts agree
produce different chunk text, and therefore different chunk ids.
A chunk is the unit a translation is of and the thing the group signs. Axis two
is the one that matters to a translator: handed `"# Chapter one"` they can break
the document by mistyping the `#`, and handed `"Chapter one"` they cannot.
**The decision: adopt markdown-it's block tokens, boundaries and stripped text
both, for one splitter shared by the upload and the paste form.** The
alternative — markdown-it semantics for uploaded files, blank lines for typed
ones — means the same text yields different chunks depending on how it arrived,
which is indefensible the moment two members do it differently.
### Three places to diverge from mantra.press on purpose
Faithful is not the goal; the right unit of translation is. Three of
markdown-it's answers are wrong here, and each should be decided now rather than
inherited.
**Code blocks should not be chunks.** `fence` and `code_block` carry content, so
mantra.press turns `code();` into a chunk — with a trailing `\n` in its text —
and asks a translator to translate it and the group to sign it as prose. Emit no
chunk for either; the code stays in the chapter's `original_text` and the
producer passes it through untouched.
**A table should be one chunk, not one per cell.** Four chunks for a 2 × 2 table
is markdown-it's honest reading — cells do contain prose — but each costs an
event with a 64-character id, a 64-character pubkey and a 128-character
signature to carry one word, and a table's alignment is structure that per-cell
translation cannot preserve anyway. One chunk of table source, and the
translator keeps the pipes.
**A link reference definition should not be a chunk.** mantra.press's parser is
`markdown-it().disable(['link', 'reference'])`. `link` is an *inline* rule, so it
changes neither the block tokens nor an inline token's content; `reference` is a
*block* rule, and disabling it stops `[ref]: https://example.com` being consumed
as a definition, so it falls through to `paragraph` and becomes a translatable
chunk of a URL. That is a side effect of reaching for the wrong rule name, not a
decision. Recognise the form and emit nothing.
### Why the change is safe to make at any time
No device ever re-derives chunks from a chapter's text. `ChunkEvent.splitOf` is
called at proposal time only — `AddChapterViewModel` and tests, nowhere on the
apply path — and the chunks travel as their own signed events;
`ChatMessage.applySignedEvent` writes the rows it is given, and the chronicle's
fallback rebuilds them from rows via `toXEvent()` rather than by splitting
anything. So a device on an older splitter receiving a newer-split chapter
applies the chunks it was handed, and there is no skew to negotiate, no
migration, and nothing to re-sign.
What does *not* change retroactively: chapters already signed keep the chunks
they were signed with. Re-splitting one later means re-proposing it as a new
chapter with new ids. So this lands before the first real upload, not after.
---
## Phase 8 — a block splitter that agrees with markdown-it
**Three days, and the fixtures are most of the value.**
The implementation of Phase 7's decision. One new function, five call sites
moved together, one thing the app has never had, and an oracle instead of an
opinion.
### The function
```kotlin
// press.mantra.compose.text.Markdown
/**
* The blocks a chapter's markdown splits into, one chunk each.
*
* A chunk is a *leaf* block's inline text with its block marker removed: the
* words a translator is given and the words the group signs. Containers --
* lists, blockquotes, tables -- contribute nothing of their own; a thematic
* break and a code block contribute nothing at all.
*
* This is markdown-it's block token stream, which is what mantra.press
* translates against, with the three deliberate departures in
* docs/chapter-uploads.md. Inline markup is kept verbatim, because emphasis and
* links are the translator's to carry; block markers are not, because a `#`
* mistyped into a translation breaks the document.
*
* Its output is hashed into event ids a quorum signs, so it is pinned by the
* golden fixtures in MarkdownBlockFixtures rather than by this comment.
*/
fun splitBlocks(text: String): List<Block>
data class Block(
val kind: Kind, // Paragraph, Heading(level), ListItem, Quote, Table
/** The block marker, kept so a document can be put back together. */
val markup: String,
/** The inline source, marker removed. This is the chunk's text. */
val text: String,
)
```
A `Block` rather than a `String`, because Phase 7's stripping decision means the
`#` and the `- ` have to be kept *somewhere* for the producer below, and the
chapter's `original_text` is the wrong place to look them up from — matching
text back to its marker is the string-equality bug described two sections down.
The block starts to recognise, which is markdown-it's block ruler minus the
rules Phase 7 drops: ATX heading, setext heading, fenced code (skipped),
indented code (skipped), blockquote, bullet and ordered list items including
nesting, thematic break (skipped), table (one block), link reference definition
(skipped), paragraph with lazy continuation.
### Hand-rolled, with the reference implementation as the oracle
`org.jetbrains:markdown` is multiplatform and CommonMark-compliant and would do
the parsing. It is the wrong dependency *here*, for one reason that is specific
to this function: its output is hashed into ids that a quorum signs, so a
version bump that moves one boundary silently re-splits every chapter proposed
afterwards, and nothing in the app would notice. A hand-rolled scanner over the
block starts above is a few hundred lines whose behaviour is ours to pin.
That is only defensible with an oracle, so:
**`MarkdownBlockFixtures` — input/expected pairs generated by markdown-it
itself, checked in as test data.** A script under `docs/scripts/` runs the
reference parser over a corpus (the converter's output, a hand-typed chapter, the
pathological cases in Phase 7's table) and emits the expected blocks; the
`commonTest` suite asserts `splitBlocks` reproduces them, with the three
departures listed as explicit exceptions rather than silently absent. Fidelity
is then measured against the implementation it claims to agree with, which is
the same standard `MAX_PAGE_BYTES` sets for itself.
Being hand-rolled, it is a documented *subset* of CommonMark and should say so:
no HTML blocks (markdown-it's default is `html: false` and the `<div>` in the
corpus comes back as a paragraph, so this matches), no footnotes, no nested
blockquote-in-list beyond one level. Each omission is a fixture asserting what
happens instead, not a gap.
### The producer, which the app has never had
Stripping block markers from chunks is only safe if a translated chapter can be
put back together, and nothing in `composeApp/src` does that today — there is no
equivalent of mantra.press's `lib/markdown-producer/index.js`. It is part of this
phase, because Phase 7's decision is incomplete without it.
```kotlin
object MarkdownProducer {
/** The chapter's markdown with each chunk's translation in place of its text. */
fun produce(originalText: String, translations: Map<Int, String>): String
}
```
Walk `splitBlocks(originalText)`, substitute by **index**, and reassemble with
each `Block.markup`. Substituting by index rather than by text is the one place
this is strictly better than the thing it mirrors: mantra.press matches with
`translationChunks.find(tc => tc.text == token.content)`, so a chapter with two
identical paragraphs puts the first one's translation into both. Chunks here
already carry an `IndexTag`, so position is available and exact.
Two outputs, as mantra.press has: markdown, and whatever the reader needs. Only
markdown is in scope here — the app renders chunks itself today — but the walk
is shared.
### Five call sites, moved together
`splitParagraphs` is called from five places and they have to agree, or the
count shown to a member differs from the chunks in the batch they are approving:
| call site | change |
|---|---|
| `ChunkEvent.splitOf` | splits into `Block`s, takes `.text` for the chunk |
| `AddChapterViewModel.addChapter` | the cap check counts blocks |
| `AddChapterScreen` | the live counter counts blocks |
| `ProposedEvent.summarize` | **stops splitting** — see below |
| `ChapterChunkSplitTest` | its parity assertion still holds, against the new function |
`splitParagraphs` goes rather than delegating. A second splitter left in the
codebase is a second answer to "what is a chunk", and the next caller will pick
whichever they find first.
**`ProposedEvent.summarize` has a latent bug this phase should fix rather than
port.** It reports a proposed chapter's chunk count by re-splitting the
chapter's text (`ProposedEvent.kt:77`), so the number a member reads in the
approval gate is derived rather than observed — and across a splitter change it
is simply wrong, since the batch was built by the proposer's splitter and
counted by the reader's. The count is sitting right there: the session's items
minus the lead. Count those.
### What it costs, and the phase it moves
Block splitting produces **more** chunks than blank-line splitting — every
heading and every list item is its own now — so both of Phase 2's limits bind
sooner. A list-heavy chapter can double or triple its chunk count against
`MAX_CHUNKS_PER_CHAPTER = 63`, and every extra chunk adds an event envelope
(~370 bytes before it says anything) to the proposal.
So **`MAX_ORIGINAL_TEXT_BYTES` must be measured after this lands, not before.**
Either take this phase ahead of Phase 2, or treat Phase 2's constant as
provisional and re-run its encoded-proposal measurement here. The second is
cheaper and the measurement is the same test either way.
### Tests
`commonTest`, all of it pure:
- `MarkdownBlockTest` against the generated fixtures — the bulk of it, and the
part worth reviewing.
- the three departures, asserted as departures: a fenced block and an indented
block yield no chunk, a table yields one, a reference definition yields none.
- inline markup survives: `*italic*` and `[text](url)` are in the chunk's text.
- `MarkdownProducerTest`: round-trip with no translations reproduces the input
byte for byte (the assertion that catches every markup-reassembly bug at
once), and a chapter with two identical paragraphs translated differently
comes back with both translations in the right places — the case mantra.press
gets wrong.
- `ChapterChunkSplitTest` keeps its existing round-trip and id-stability
assertions, now with the converter's output shape as a fixture.
---
## Phase 9 — tests and rollout
**A day, and the list is short because most of it is in the phases.**
New, per phase: `ChapterTextTest` (decode), the byte-cap boundary and the
encoded-proposal measurement in `ChapterBatchProposalJvmTest`, the
three-proposals-in-flight index test, `MarkdownBlockTest` against the generated
fixtures and `MarkdownProducerTest`'s byte-for-byte round trip, and a
`ChapterUploadJvmTest` beside the
existing batch tests that drives the view model over a real in-memory database
and asserts the whole shape: N files in, N sessions out with one lead chapter
each, indices `base..base+N-1`, chunks matching each file's paragraphs, one
scaffolding session for the lot, and — the one that matters — that a failure on
file 4 leaves files 1–3 proposed and recoverable rather than rolled back.
`jvmTest`, not `commonTest`, because it is Room-backed; `./gradlew
:composeApp:jvmTest`, per the project's build notes.
Rollout needs no flag. The chapter and chunk kinds, their tags and their apply
branches are unchanged, so an older build receives an uploaded chapter as the
chapter it is. Two things do change what a peer sees, and neither needs
negotiation: a room will see N proposals where it used to see one (which an
older build handles — it is N ordinary sessions), and Phases 7–8 change chunk
boundaries and chunk text for new chapters only — safely, because nothing
re-derives chunks from a chapter's text: `splitOf` runs at proposal time and the
chronicle rebuilds from rows. The byte cap of Phase 2 is the only
change that makes the app *refuse* something it used to attempt, and what it
used to attempt was a session that stalled.
Order: 1 → 2 → 3 before 4; 5 and 6 after 4; 7 → 8 whenever, but before the
first real upload, since a chapter already signed keeps the chunks it was signed
with. Phases 2, 3, 5 and 8 are each shippable alone and each fix something that
is wrong today at N = 1 — the index collision, the unchecked proposal size, the
scaffolding fan-out and the splitter — which is the argument for landing them as
their own commits rather than inside the feature.
---
## Appendix — considered and rejected
**One session for the whole upload.** The three reasons are in § The constraint.
The size cap alone kills it at two chapters.
**One chapter event per upload, with the files concatenated.** Fits, signs once,
and destroys the thing being modelled: a chapter is the unit of work assigned to
a translator, and `ChapterEvent` carries one index.
**A manifest: sign one event committing to N chapter ids, derive the chapters
from it.** Cheapest option on this page, and already argued down in
[frost-batch-signing.md](./frost-batch-signing.md) — each chapter stops carrying
its own verifiable signature, and the point of
[shared-key-derivation.md](./shared-key-derivation.md) is that a reader holding
one signed thing can check it against the room it was found in without a lookup.
Chapters are read apart; they get their own signatures.
**Derive the chunks on arrival instead of signing them.** This is what the app
did before batch signing, and it was deliberately changed: "a chunk that can be
checked on its own is worth more than one that can only be checked by
re-deriving it". It would double the number of uploadable paragraphs, which is
not worth it. Note the asymmetry, though — it *is* the right answer for the
translation scaffolding, which is why that is a separate session rather than
separate events.
**A remote upload: post the files somewhere and have the group sign a URL.** No.
An artifact's `url` tag is a reference to a published work; the text the group
signs is the text, and a chapter whose words live behind a link is a chapter the
group has not actually agreed to.
**An uploads table to make an interrupted upload resumable.** A schema version
for state that Phase 3 can read off the open sessions instead. The proposals are
the record of what was uploaded; `unsignedEvents` is how to read it. If resume
ever needs more than "re-pick the files and the duplicate check tells you what
already went", revisit — but it will need a reason that is not available today.
**Mirroring mantra.press's xlsx import** (`POST /projects/:id/import`,
`server/router/projects/index.js:1565`) — a spreadsheet of translations rather
than chapter markdown. A different feature on the translation side of the
hierarchy, with the same batching argument and a different parser. Out of scope
here, and the plan above is the shape it would follow.