commit 7a77a1dc023e9db208e6dd7a22e0ed1a4d939c5e
parent 7cd316d1ab9164c525858de4e7d6bd4546d4ad76
Author: Michael Camilleri <[email protected]>
Date: Sun, 2 Aug 2026 14:53:23 +0900
Add notes about custom XD decorations section
Diffstat:
| A | Notes/Decorations.md | | | 194 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 194 insertions(+), 0 deletions(-)
diff --git a/Notes/Decorations.md b/Notes/Decorations.md
@@ -0,0 +1,194 @@
+# Decorations — Design
+
+Crossmate supports a `## Decorations` section in `.xd`, used to bake an
+External provider's overlay art into the puzzle source. The 2026-07-23 Thursday
+puzzle hides `ENERGY` in its **black squares**, revealed on completion — the
+revealer is `DARKENERGY` at row 13. The provider ships that as a transparent PNG
+overlay (`overlays.afterSolve`), not as puzzle data; the blocks are literally
+`{}` in the JSON.
+
+The section name is deliberately **not** `## Design`, which is what xdformat v4
+is standardising (century-arcade/xdformat#8, still open). Crossmate's syntax
+differs, so squatting that name would break a conforming reader. Sections are
+looked up by name and unknown named sections are ignored, so a future
+`## Design` can coexist.
+
+Written 2026-07-28; rendering and `beforeStart` support landed 2026-07-29 and
+this note was brought up to date on 2026-08-02.
+
+## Format
+
+```
+## Decorations
+
+<design grid — one char per cell, `.` = none>
+
+<char>. <kind>=<value> [before|after]
+```
+
+- Kinds: `mark` (`circle`/`shaded`), `bg`, `fg`, `text`, `data`.
+- `bg`/`fg` values: `#RRGGBB` or `<light>;<dark>`.
+- `data` values: `<mime-type>[;<encoding>],<payload>`.
+- Phase defaults to `before`; `after` is revealed only once solved.
+- Stack layers by repeating the character on more lines. Line order = paint
+ order.
+- Grid lines and definition lines are told apart by whitespace (definition lines
+ always have some), so their order doesn't matter.
+
+## What is built
+
+1. **Read** — `XD.parseDecorations`, `Puzzle.Decoration`, section names honoured
+ by `splitIntoSections`.
+2. **Write** — both converters emit `## Decorations`; `Specials:`/`Special:` are
+ now read-only legacy.
+3. **Images** — `NYTOverlaySlicer` cuts an overlay into per-cell PNG tiles;
+ `NYTPuzzleFetcher` fetches the asset; the converter emits `data=`.
+4. **Render** — `GridView` draws all five kinds, phase-gated on
+ `game.completionState == .solved && !isReplaying`.
+
+Rendering was refined afterwards: block-square overlay letters are recognised,
+letters render legibly in every square, and a tile that is a single flat colour
+is emitted as `bg=` rather than a PNG payload
+(`NYTOverlaySlicer.uniformBackgroundHex`, used at
+`Crossmate/Services/NYTToXDConverter.swift:781`). `beforeStart` overlays are
+also fetched and emitted now, so a puzzle can carry art for both phases —
+2021-02-14 pairs red outlines before start with letters after solve on the same
+cells.
+
+Files: `Crossmate/Models/XD.swift`, `Crossmate/Models/Puzzle.swift`,
+`Crossmate/Models/XDDecorationWriter.swift`,
+`Crossmate/Models/GridPosition.swift` (moved out of `Crossmate/Sync/Moves.swift`),
+`Crossmate/Services/NYTOverlaySlicer.swift`,
+`Crossmate/Services/NYTToXDConverter.swift`,
+`Crossmate/Services/PUZToXDConverter.swift`,
+`Crossmate/Services/NYTPuzzleFetcher.swift`,
+`Crossmate/Views/Puzzle/GridView.swift`,
+`Crossmate/Views/Puzzle/DecorationImages.swift`. Tests:
+`Tests/Unit/XDDecorationTests.swift`, `Tests/Unit/NYTOverlaySlicerTests.swift`,
+`Tests/Unit/DecorationRenderingTests.swift`.
+
+## Settled: OCR must not be used to emit `text=`
+
+Experiment run 2026-07-28. **Verdict: do not build this.**
+
+The idea was to recognise letter art in an overlay tile and emit `text=` instead
+of `data=`, mainly so VoiceOver could read the reveal. The stated acceptance
+criteria passed, but only because they tested text art against text art.
+Widening the negative set to non-letter art breaks the idea, and the proposed
+mitigation turns out not to work at all.
+
+### What passed
+
+- 2026-07-23 `afterSolve`: all six tiles read `E N E R G Y` correctly. Needs
+ `.fast`; `.accurate` intermittently returns nothing for the two `E`s.
+- 2021-02-14 `afterSolve`: zero confident misreads, in **all 144** preprocessing
+ configurations swept (canvas × inset × level × min-height × rendering). Vision
+ reads the whole `RedR` as one four-character region, so the single-character
+ rule rejects it. This failure mode is solidly handled.
+
+### What killed it
+
+1. **The confidence floor does no work.** Confidence is quantised to roughly
+ {0.3, 0.5, 1.0} and is *anti-correlated* with correctness on isolated
+ glyphs: correct reads of `E` and `I` come back at 0.50, while wrong reads
+ come back at 1.00 (`•`@1.00 for a circle, `+`@1.00 for a plus, `t`@1.00 for
+ an up arrow). "Require a high confidence floor" was the headline mitigation
+ and it filters nothing.
+2. **Case isn't recovered, and fixing that opens leaks.** With no surrounding
+ text Vision can't infer case: a capital `O` comes back `"o"`, a capital `X`
+ comes back `"x"`. So a strict A–Z-capital rule *rejects two of the commonest
+ crossword letters*. Relaxing to case-insensitive then turns crescent→`C`,
+ heart→`V`, up arrow→`T`, digit `0`→`O`, saltire→`X` into confident letter
+ misreads. There is no setting that both accepts a capital `O` and rejects a
+ crescent.
+3. **A plain vertical bar reads as `I`** — a leak that survives even the strict
+ rule, with no knob left to close it.
+4. **It would run on device, unreviewed.** Conversion is *not* only the offline
+ authoring script. `Crossmate/CrossmateApp.swift:966` re-converts through
+ `NYTPuzzleUpgrader` whenever a game is opened whose stored source predates
+ `XD.currentConverterVersion`, and `structuralDivergence` guards only geometry
+ and the solution — a changed decoration passes straight through and replaces
+ the persisted `puzzleSource`. A misread is therefore not a diff an author
+ eyeballs before shipping; it silently rewrites a live game. Worse, Vision's
+ output is revision- and OS-dependent, so the same puzzle need not convert
+ identically on two players' devices in a shared game.
+
+The synthetic motif set used for (1)–(3): circle, ring, square, diamond, 5- and
+6-point star, heart, plus, saltire, vertical bar, triangle, arrow, crescent,
+chequer, digits 0 and 1, drawn at the real 106px cell size, with letters as
+controls.
+
+### What to do instead
+
+The whole argument for OCR was VoiceOver. But `text=` can only ever help
+*letter* art — a puzzle whose reveal is hearts stays unreadable no matter how
+good recognition gets. The accessibility fix and the OCR idea are therefore
+separable, and the accessibility fix is the one carrying the value.
+
+1. **Give decorations author-supplied alt text and wire it into
+ `GridAccessibility`.** An `alt=` kind, or an optional trailing description on
+ any decoration line, describes *all* art rather than only letters, is always
+ correct, and needs no recognition. This closes the accessibility gap below.
+2. **Write `text=` by hand where the art really is letters.** The format already
+ supports it; for 2026-07-23 that is six lines, and it gets the full
+ size/dark-mode/Dynamic Type/VoiceOver win with none of the risk.
+3. If OCR ever comes back, it belongs **only** in `Scripts/nyt-to-xd.sh` as a
+ printed suggestion for a human to paste — never automatic, and never in code
+ that `NYTPuzzleUpgrader` can reach.
+
+### Regenerating the experiment data
+
+```bash
+bash Scripts/fetch-nyt.sh 2026-07-23 puzzle-2026-07-23.json
+bash Scripts/fetch-nyt.sh 2021-02-14 puzzle-2021-02-14.json
+# Resolve each overlay URI from the JSON rather than constructing it (see the
+# one-based indexing note below); the assets themselves need no cookie:
+jq -r '.assets[(.body[0].overlays.afterSolve - 1)].uri' puzzle-2026-07-23.json
+curl -sSL -o solve-2026.png "<resolved URI>"
+# full conversion, overlay baked in:
+bash Scripts/nyt-to-xd.sh --date 2026-07-23 --output puzzle.xd
+```
+
+The overlays are transparent with white or coloured glyphs — they look blank
+until composited onto a mid-grey background.
+
+## Hard-won facts worth not rediscovering
+
+- **`body[0].overlays.afterSolve` is a ONE-BASED index into the root `assets`
+ array.** Read the `uri` from there; never construct it. Both the host and the
+ filename shape have changed over the years — the 2021 asset is served from a
+ general object-storage host, the 2026 one from the provider's own domain, with
+ differently shaped filenames. Probing guessed filenames is useless as a survey
+ method.
+- **Cell size varies with grid size.** 15×15 uses 33-unit cells in a 501
+ viewBox; 21×21 uses **23**-unit cells in a **489** viewBox. Derive geometry
+ from `body[0].board`'s viewBox and first cell path — a hardcoded 33 misaligns
+ every tile on a Sunday, and shows up as a nonsense tile count such as "20
+ tiles for 8 boxes".
+- **Overlays are cell-aligned.** An earlier claim to the contrary was a geometry
+ bug, not a property of the data.
+- **The unit suite runs tests in parallel**, and `PuzzleSessionTests` uses
+ `waitUntil` helpers with 5-second deadlines. A CPU-saturating test fixture
+ will make those fail from starvation, looking like unrelated flakiness. Keep
+ test images small.
+- `Scripts/nyt-to-xd.sh` compiles real sources rather than stubbing `XD`. It
+ pulls in `Puzzle.swift` and therefore SwiftUI via `XDMarkup`. Cutting that
+ cone would mean extracting `Decoration`/`Special` into a dependency-free file
+ — deliberately deferred as its own change.
+- No CloudKit schema change was needed: `puzzleSource` is already a `CKAsset`.
+- **Conversion is not authoring-only.** `NYTToXDConverter` runs on device (see
+ item 4 above), and its result replaces the persisted `puzzleSource`. Anything
+ nondeterministic or unreviewable in the converter ships straight into live
+ games — assume no human is in the loop.
+- **The 2021-02-14 tiles are fully opaque**, not transparent glyph art: each is
+ a white square holding red `Red` plus a black letter. Flattening a tile by its
+ alpha channel — right for the 2026 white-on-transparent letters — turns them
+ into solid black squares, which looks like a clean recognition failure but is
+ really the harness destroying the input. Composite over a backdrop instead.
+
+## Known gaps
+
+- VoiceOver doesn't see decorations at all: `GridAccessibility` reads
+ `Puzzle.Cell` and has no decoration awareness, so the reveal is sighted-only.
+ **This is the next piece of work** — see "What to do instead" above.
+ Author-supplied alt text, not OCR.
