Decorations.md (10179B)
1 # Decorations — Design 2 3 Crossmate supports a `## Decorations` section in `.xd`, used to bake an 4 External provider's overlay art into the puzzle source. The 2026-07-23 Thursday 5 puzzle hides `ENERGY` in its **black squares**, revealed on completion — the 6 revealer is `DARKENERGY` at row 13. The provider ships that as a transparent PNG 7 overlay (`overlays.afterSolve`), not as puzzle data; the blocks are literally 8 `{}` in the JSON. 9 10 The section name is deliberately **not** `## Design`, which is what xdformat v4 11 is standardising (century-arcade/xdformat#8, still open). Crossmate's syntax 12 differs, so squatting that name would break a conforming reader. Sections are 13 looked up by name and unknown named sections are ignored, so a future 14 `## Design` can coexist. 15 16 Written 2026-07-28; rendering and `beforeStart` support landed 2026-07-29 and 17 this note was brought up to date on 2026-08-02. 18 19 ## Format 20 21 ``` 22 ## Decorations 23 24 <design grid — one char per cell, `.` = none> 25 26 <char>. <kind>=<value> [before|after] 27 ``` 28 29 - Kinds: `mark` (`circle`/`shaded`), `bg`, `fg`, `text`, `data`. 30 - `bg`/`fg` values: `#RRGGBB` or `<light>;<dark>`. 31 - `data` values: `<mime-type>[;<encoding>],<payload>`. 32 - Phase defaults to `before`; `after` is revealed only once solved. 33 - Stack layers by repeating the character on more lines. Line order = paint 34 order. 35 - Grid lines and definition lines are told apart by whitespace (definition lines 36 always have some), so their order doesn't matter. 37 38 ## What is built 39 40 1. **Read** — `XD.parseDecorations`, `Puzzle.Decoration`, section names honoured 41 by `splitIntoSections`. 42 2. **Write** — both converters emit `## Decorations`; `Specials:`/`Special:` are 43 now read-only legacy. 44 3. **Images** — `NYTOverlaySlicer` cuts an overlay into per-cell PNG tiles; 45 `NYTPuzzleFetcher` fetches the asset; the converter emits `data=`. 46 4. **Render** — `GridView` draws all five kinds, phase-gated on 47 `game.completionState == .solved && !isReplaying`. 48 49 Rendering was refined afterwards: block-square overlay letters are recognised, 50 letters render legibly in every square, and a tile that is a single flat colour 51 is emitted as `bg=` rather than a PNG payload 52 (`NYTOverlaySlicer.uniformBackgroundHex`, used at 53 `Crossmate/Services/NYTToXDConverter.swift:781`). `beforeStart` overlays are 54 also fetched and emitted now, so a puzzle can carry art for both phases — 55 2021-02-14 pairs red outlines before start with letters after solve on the same 56 cells. 57 58 Files: `Crossmate/Models/XD.swift`, `Crossmate/Models/Puzzle.swift`, 59 `Crossmate/Models/XDDecorationWriter.swift`, 60 `Crossmate/Models/GridPosition.swift` (moved out of `Crossmate/Sync/Moves.swift`), 61 `Crossmate/Services/NYTOverlaySlicer.swift`, 62 `Crossmate/Services/NYTToXDConverter.swift`, 63 `Crossmate/Services/PUZToXDConverter.swift`, 64 `Crossmate/Services/NYTPuzzleFetcher.swift`, 65 `Crossmate/Views/Puzzle/GridView.swift`, 66 `Crossmate/Views/Puzzle/DecorationImages.swift`. Tests: 67 `Tests/Unit/XDDecorationTests.swift`, `Tests/Unit/NYTOverlaySlicerTests.swift`, 68 `Tests/Unit/DecorationRenderingTests.swift`. 69 70 ## Settled: OCR must not be used to emit `text=` 71 72 Experiment run 2026-07-28. **Verdict: do not build this.** 73 74 The idea was to recognise letter art in an overlay tile and emit `text=` instead 75 of `data=`, mainly so VoiceOver could read the reveal. The stated acceptance 76 criteria passed, but only because they tested text art against text art. 77 Widening the negative set to non-letter art breaks the idea, and the proposed 78 mitigation turns out not to work at all. 79 80 ### What passed 81 82 - 2026-07-23 `afterSolve`: all six tiles read `E N E R G Y` correctly. Needs 83 `.fast`; `.accurate` intermittently returns nothing for the two `E`s. 84 - 2021-02-14 `afterSolve`: zero confident misreads, in **all 144** preprocessing 85 configurations swept (canvas × inset × level × min-height × rendering). Vision 86 reads the whole `RedR` as one four-character region, so the single-character 87 rule rejects it. This failure mode is solidly handled. 88 89 ### What killed it 90 91 1. **The confidence floor does no work.** Confidence is quantised to roughly 92 {0.3, 0.5, 1.0} and is *anti-correlated* with correctness on isolated 93 glyphs: correct reads of `E` and `I` come back at 0.50, while wrong reads 94 come back at 1.00 (`•`@1.00 for a circle, `+`@1.00 for a plus, `t`@1.00 for 95 an up arrow). "Require a high confidence floor" was the headline mitigation 96 and it filters nothing. 97 2. **Case isn't recovered, and fixing that opens leaks.** With no surrounding 98 text Vision can't infer case: a capital `O` comes back `"o"`, a capital `X` 99 comes back `"x"`. So a strict A–Z-capital rule *rejects two of the commonest 100 crossword letters*. Relaxing to case-insensitive then turns crescent→`C`, 101 heart→`V`, up arrow→`T`, digit `0`→`O`, saltire→`X` into confident letter 102 misreads. There is no setting that both accepts a capital `O` and rejects a 103 crescent. 104 3. **A plain vertical bar reads as `I`** — a leak that survives even the strict 105 rule, with no knob left to close it. 106 4. **It would run on device, unreviewed.** Conversion is *not* only the offline 107 authoring script. `Crossmate/CrossmateApp.swift:966` re-converts through 108 `NYTPuzzleUpgrader` whenever a game is opened whose stored source predates 109 `XD.currentConverterVersion`, and `structuralDivergence` guards only geometry 110 and the solution — a changed decoration passes straight through and replaces 111 the persisted `puzzleSource`. A misread is therefore not a diff an author 112 eyeballs before shipping; it silently rewrites a live game. Worse, Vision's 113 output is revision- and OS-dependent, so the same puzzle need not convert 114 identically on two players' devices in a shared game. 115 116 The synthetic motif set used for (1)–(3): circle, ring, square, diamond, 5- and 117 6-point star, heart, plus, saltire, vertical bar, triangle, arrow, crescent, 118 chequer, digits 0 and 1, drawn at the real 106px cell size, with letters as 119 controls. 120 121 ### What to do instead 122 123 The whole argument for OCR was VoiceOver. But `text=` can only ever help 124 *letter* art — a puzzle whose reveal is hearts stays unreadable no matter how 125 good recognition gets. The accessibility fix and the OCR idea are therefore 126 separable, and the accessibility fix is the one carrying the value. 127 128 1. **Give decorations author-supplied alt text and wire it into 129 `GridAccessibility`.** An `alt=` kind, or an optional trailing description on 130 any decoration line, describes *all* art rather than only letters, is always 131 correct, and needs no recognition. This closes the accessibility gap below. 132 2. **Write `text=` by hand where the art really is letters.** The format already 133 supports it; for 2026-07-23 that is six lines, and it gets the full 134 size/dark-mode/Dynamic Type/VoiceOver win with none of the risk. 135 3. If OCR ever comes back, it belongs **only** in `Scripts/nyt-to-xd.sh` as a 136 printed suggestion for a human to paste — never automatic, and never in code 137 that `NYTPuzzleUpgrader` can reach. 138 139 ### Regenerating the experiment data 140 141 ```bash 142 bash Scripts/fetch-nyt.sh 2026-07-23 puzzle-2026-07-23.json 143 bash Scripts/fetch-nyt.sh 2021-02-14 puzzle-2021-02-14.json 144 # Resolve each overlay URI from the JSON rather than constructing it (see the 145 # one-based indexing note below); the assets themselves need no cookie: 146 jq -r '.assets[(.body[0].overlays.afterSolve - 1)].uri' puzzle-2026-07-23.json 147 curl -sSL -o solve-2026.png "<resolved URI>" 148 # full conversion, overlay baked in: 149 bash Scripts/nyt-to-xd.sh --date 2026-07-23 --output puzzle.xd 150 ``` 151 152 The overlays are transparent with white or coloured glyphs — they look blank 153 until composited onto a mid-grey background. 154 155 ## Hard-won facts worth not rediscovering 156 157 - **`body[0].overlays.afterSolve` is a ONE-BASED index into the root `assets` 158 array.** Read the `uri` from there; never construct it. Both the host and the 159 filename shape have changed over the years — the 2021 asset is served from a 160 general object-storage host, the 2026 one from the provider's own domain, with 161 differently shaped filenames. Probing guessed filenames is useless as a survey 162 method. 163 - **Cell size varies with grid size.** 15×15 uses 33-unit cells in a 501 164 viewBox; 21×21 uses **23**-unit cells in a **489** viewBox. Derive geometry 165 from `body[0].board`'s viewBox and first cell path — a hardcoded 33 misaligns 166 every tile on a Sunday, and shows up as a nonsense tile count such as "20 167 tiles for 8 boxes". 168 - **Overlays are cell-aligned.** An earlier claim to the contrary was a geometry 169 bug, not a property of the data. 170 - **The unit suite runs tests in parallel**, and `PuzzleSessionTests` uses 171 `waitUntil` helpers with 5-second deadlines. A CPU-saturating test fixture 172 will make those fail from starvation, looking like unrelated flakiness. Keep 173 test images small. 174 - `Scripts/nyt-to-xd.sh` compiles real sources rather than stubbing `XD`. It 175 pulls in `Puzzle.swift` and therefore SwiftUI via `XDMarkup`. Cutting that 176 cone would mean extracting `Decoration`/`Special` into a dependency-free file 177 — deliberately deferred as its own change. 178 - No CloudKit schema change was needed: `puzzleSource` is already a `CKAsset`. 179 - **Conversion is not authoring-only.** `NYTToXDConverter` runs on device (see 180 item 4 above), and its result replaces the persisted `puzzleSource`. Anything 181 nondeterministic or unreviewable in the converter ships straight into live 182 games — assume no human is in the loop. 183 - **The 2021-02-14 tiles are fully opaque**, not transparent glyph art: each is 184 a white square holding red `Red` plus a black letter. Flattening a tile by its 185 alpha channel — right for the 2026 white-on-transparent letters — turns them 186 into solid black squares, which looks like a clean recognition failure but is 187 really the harness destroying the input. Composite over a backdrop instead. 188 189 ## Known gaps 190 191 - VoiceOver doesn't see decorations at all: `GridAccessibility` reads 192 `Puzzle.Cell` and has no decoration awareness, so the reveal is sighted-only. 193 **This is the next piece of work** — see "What to do instead" above. 194 Author-supplied alt text, not OCR.