Notes/Decorations.md (10179B)
1# Decorations — Design
2
3Crossmate supports a `## Decorations` section in `.xd`, used to bake an
4External provider's overlay art into the puzzle source. The 2026-07-23 Thursday
5puzzle hides `ENERGY` in its **black squares**, revealed on completion — the
6revealer is `DARKENERGY` at row 13. The provider ships that as a transparent PNG
7overlay (`overlays.afterSolve`), not as puzzle data; the blocks are literally
8`{}` in the JSON.
9
10The section name is deliberately **not** `## Design`, which is what xdformat v4
11is standardising (century-arcade/xdformat#8, still open). Crossmate's syntax
12differs, so squatting that name would break a conforming reader. Sections are
13looked up by name and unknown named sections are ignored, so a future
14`## Design` can coexist.
15
16Written 2026-07-28; rendering and `beforeStart` support landed 2026-07-29 and
17this note was brought up to date on 2026-08-02.
18
19## Format
20
21```
22## Decorations
23
24<design grid — one char per cell, `.` = none>
25
26<char>. <kind>=<value> [before|after]
27```
28
29- Kinds: `mark` (`circle`/`shaded`), `bg`, `fg`, `text`, `data`.
30- `bg`/`fg` values: `#RRGGBB` or `<light>;<dark>`.
31- `data` values: `<mime-type>[;<encoding>],<payload>`.
32- Phase defaults to `before`; `after` is revealed only once solved.
33- Stack layers by repeating the character on more lines. Line order = paint
34 order.
35- Grid lines and definition lines are told apart by whitespace (definition lines
36 always have some), so their order doesn't matter.
37
38## What is built
39
401. **Read** — `XD.parseDecorations`, `Puzzle.Decoration`, section names honoured
41 by `splitIntoSections`.
422. **Write** — both converters emit `## Decorations`; `Specials:`/`Special:` are
43 now read-only legacy.
443. **Images** — `NYTOverlaySlicer` cuts an overlay into per-cell PNG tiles;
45 `NYTPuzzleFetcher` fetches the asset; the converter emits `data=`.
464. **Render** — `GridView` draws all five kinds, phase-gated on
47 `game.completionState == .solved && !isReplaying`.
48
49Rendering was refined afterwards: block-square overlay letters are recognised,
50letters render legibly in every square, and a tile that is a single flat colour
51is emitted as `bg=` rather than a PNG payload
52(`NYTOverlaySlicer.uniformBackgroundHex`, used at
53`Crossmate/Services/NYTToXDConverter.swift:781`). `beforeStart` overlays are
54also fetched and emitted now, so a puzzle can carry art for both phases —
552021-02-14 pairs red outlines before start with letters after solve on the same
56cells.
57
58Files: `Crossmate/Models/XD.swift`, `Crossmate/Models/Puzzle.swift`,
59`Crossmate/Models/XDDecorationWriter.swift`,
60`Crossmate/Models/GridPosition.swift` (moved out of `Crossmate/Sync/Moves.swift`),
61`Crossmate/Services/NYTOverlaySlicer.swift`,
62`Crossmate/Services/NYTToXDConverter.swift`,
63`Crossmate/Services/PUZToXDConverter.swift`,
64`Crossmate/Services/NYTPuzzleFetcher.swift`,
65`Crossmate/Views/Puzzle/GridView.swift`,
66`Crossmate/Views/Puzzle/DecorationImages.swift`. Tests:
67`Tests/Unit/XDDecorationTests.swift`, `Tests/Unit/NYTOverlaySlicerTests.swift`,
68`Tests/Unit/DecorationRenderingTests.swift`.
69
70## Settled: OCR must not be used to emit `text=`
71
72Experiment run 2026-07-28. **Verdict: do not build this.**
73
74The idea was to recognise letter art in an overlay tile and emit `text=` instead
75of `data=`, mainly so VoiceOver could read the reveal. The stated acceptance
76criteria passed, but only because they tested text art against text art.
77Widening the negative set to non-letter art breaks the idea, and the proposed
78mitigation turns out not to work at all.
79
80### What passed
81
82- 2026-07-23 `afterSolve`: all six tiles read `E N E R G Y` correctly. Needs
83 `.fast`; `.accurate` intermittently returns nothing for the two `E`s.
84- 2021-02-14 `afterSolve`: zero confident misreads, in **all 144** preprocessing
85 configurations swept (canvas × inset × level × min-height × rendering). Vision
86 reads the whole `RedR` as one four-character region, so the single-character
87 rule rejects it. This failure mode is solidly handled.
88
89### What killed it
90
911. **The confidence floor does no work.** Confidence is quantised to roughly
92 {0.3, 0.5, 1.0} and is *anti-correlated* with correctness on isolated
93 glyphs: correct reads of `E` and `I` come back at 0.50, while wrong reads
94 come back at 1.00 (`•`@1.00 for a circle, `+`@1.00 for a plus, `t`@1.00 for
95 an up arrow). "Require a high confidence floor" was the headline mitigation
96 and it filters nothing.
972. **Case isn't recovered, and fixing that opens leaks.** With no surrounding
98 text Vision can't infer case: a capital `O` comes back `"o"`, a capital `X`
99 comes back `"x"`. So a strict A–Z-capital rule *rejects two of the commonest
100 crossword letters*. Relaxing to case-insensitive then turns crescent→`C`,
101 heart→`V`, up arrow→`T`, digit `0`→`O`, saltire→`X` into confident letter
102 misreads. There is no setting that both accepts a capital `O` and rejects a
103 crescent.
1043. **A plain vertical bar reads as `I`** — a leak that survives even the strict
105 rule, with no knob left to close it.
1064. **It would run on device, unreviewed.** Conversion is *not* only the offline
107 authoring script. `Crossmate/CrossmateApp.swift:966` re-converts through
108 `NYTPuzzleUpgrader` whenever a game is opened whose stored source predates
109 `XD.currentConverterVersion`, and `structuralDivergence` guards only geometry
110 and the solution — a changed decoration passes straight through and replaces
111 the persisted `puzzleSource`. A misread is therefore not a diff an author
112 eyeballs before shipping; it silently rewrites a live game. Worse, Vision's
113 output is revision- and OS-dependent, so the same puzzle need not convert
114 identically on two players' devices in a shared game.
115
116The synthetic motif set used for (1)–(3): circle, ring, square, diamond, 5- and
1176-point star, heart, plus, saltire, vertical bar, triangle, arrow, crescent,
118chequer, digits 0 and 1, drawn at the real 106px cell size, with letters as
119controls.
120
121### What to do instead
122
123The whole argument for OCR was VoiceOver. But `text=` can only ever help
124*letter* art — a puzzle whose reveal is hearts stays unreadable no matter how
125good recognition gets. The accessibility fix and the OCR idea are therefore
126separable, and the accessibility fix is the one carrying the value.
127
1281. **Give decorations author-supplied alt text and wire it into
129 `GridAccessibility`.** An `alt=` kind, or an optional trailing description on
130 any decoration line, describes *all* art rather than only letters, is always
131 correct, and needs no recognition. This closes the accessibility gap below.
1322. **Write `text=` by hand where the art really is letters.** The format already
133 supports it; for 2026-07-23 that is six lines, and it gets the full
134 size/dark-mode/Dynamic Type/VoiceOver win with none of the risk.
1353. If OCR ever comes back, it belongs **only** in `Scripts/nyt-to-xd.sh` as a
136 printed suggestion for a human to paste — never automatic, and never in code
137 that `NYTPuzzleUpgrader` can reach.
138
139### Regenerating the experiment data
140
141```bash
142bash Scripts/fetch-nyt.sh 2026-07-23 puzzle-2026-07-23.json
143bash Scripts/fetch-nyt.sh 2021-02-14 puzzle-2021-02-14.json
144# Resolve each overlay URI from the JSON rather than constructing it (see the
145# one-based indexing note below); the assets themselves need no cookie:
146jq -r '.assets[(.body[0].overlays.afterSolve - 1)].uri' puzzle-2026-07-23.json
147curl -sSL -o solve-2026.png "<resolved URI>"
148# full conversion, overlay baked in:
149bash Scripts/nyt-to-xd.sh --date 2026-07-23 --output puzzle.xd
150```
151
152The overlays are transparent with white or coloured glyphs — they look blank
153until composited onto a mid-grey background.
154
155## Hard-won facts worth not rediscovering
156
157- **`body[0].overlays.afterSolve` is a ONE-BASED index into the root `assets`
158 array.** Read the `uri` from there; never construct it. Both the host and the
159 filename shape have changed over the years — the 2021 asset is served from a
160 general object-storage host, the 2026 one from the provider's own domain, with
161 differently shaped filenames. Probing guessed filenames is useless as a survey
162 method.
163- **Cell size varies with grid size.** 15×15 uses 33-unit cells in a 501
164 viewBox; 21×21 uses **23**-unit cells in a **489** viewBox. Derive geometry
165 from `body[0].board`'s viewBox and first cell path — a hardcoded 33 misaligns
166 every tile on a Sunday, and shows up as a nonsense tile count such as "20
167 tiles for 8 boxes".
168- **Overlays are cell-aligned.** An earlier claim to the contrary was a geometry
169 bug, not a property of the data.
170- **The unit suite runs tests in parallel**, and `PuzzleSessionTests` uses
171 `waitUntil` helpers with 5-second deadlines. A CPU-saturating test fixture
172 will make those fail from starvation, looking like unrelated flakiness. Keep
173 test images small.
174- `Scripts/nyt-to-xd.sh` compiles real sources rather than stubbing `XD`. It
175 pulls in `Puzzle.swift` and therefore SwiftUI via `XDMarkup`. Cutting that
176 cone would mean extracting `Decoration`/`Special` into a dependency-free file
177 — deliberately deferred as its own change.
178- No CloudKit schema change was needed: `puzzleSource` is already a `CKAsset`.
179- **Conversion is not authoring-only.** `NYTToXDConverter` runs on device (see
180 item 4 above), and its result replaces the persisted `puzzleSource`. Anything
181 nondeterministic or unreviewable in the converter ships straight into live
182 games — assume no human is in the loop.
183- **The 2021-02-14 tiles are fully opaque**, not transparent glyph art: each is
184 a white square holding red `Red` plus a black letter. Flattening a tile by its
185 alpha channel — right for the 2026 white-on-transparent letters — turns them
186 into solid black squares, which looks like a clean recognition failure but is
187 really the harness destroying the input. Composite over a backdrop instead.
188
189## Known gaps
190
191- VoiceOver doesn't see decorations at all: `GridAccessibility` reads
192 `Puzzle.Cell` and has no decoration awareness, so the reveal is sighted-only.
193 **This is the next piece of work** — see "What to do instead" above.
194 Author-supplied alt text, not OCR.