15 — Self-healing and failure recovery
Headline
OSL does not currently self-heal a protected carrier send: the canonical tree has useful fail-closed carrier probes and window repair, but its shared send transaction is not implemented by any production carrier, has no durable attempt record, and cannot recover an unknown send after a carrier exit, network loss or OSL crash. [integration/full:apps/osl-hub/src/protected_carrier_transaction.rs:91-105,262-354; integration/full:apps/osl-hub/tests/task_7760_protected_carrier_transaction.rs:151-244; integration/full:apps/osl-hub/tests/task_7762_telegram_protected_carrier.rs:218-272]
This audit inspected canonical integration/full at 4d13b8a04. A checked task is not treated as proof: TASK 7760's evidence is a compiled fake adapter using account-7760, room-7760 and provider-row-7760, while TASK 7761's own evidence says packaged Discord was not observed. [OSL-AUDITS/evidence/7760.md:18-42; OSL-AUDITS/evidence/7761.md:3-10,83-100]
The two hard lines survive every recommendation here: tests never message a person the owner does not control, and the product never labels an unknown outcome Sent or Not sent. [RULING D38; RULINGS.txt:1915-1930; 01-FULL-SPEC.md:15,35]
What exists now
There are four separate pieces which must not be mistaken for one working recovery path:
1. protected_carrier_transaction.rs defines a good fail-closed ordering in memory: bind the scope, place, read the composer back, trigger Send, inspect receive, restore, and issue a receipt last. It persists none of the attempt, phase, stable operation ID or outcome. [integration/full:apps/osl-hub/src/protected_carrier_transaction.rs:262-354]
2. No production type implements ProtectedCarrierAdapter; the only implementations are AdapterHost and Host under the TASK 7760 and 7762 test executables. telegram_protected_carrier.rs calls the shared function, but no production caller or production driver implements the required trait. [integration/full:apps/osl-hub/src/protected_carrier_transaction.rs:91-105; integration/full:apps/osl-hub/src/telegram_protected_carrier.rs:203-264; integration/full:apps/osl-hub/tests/task_7760_protected_carrier_transaction.rs:151-244; integration/full:apps/osl-hub/tests/task_7762_telegram_protected_carrier.rs:218-272]
3. Native adapters contain useful local protections. Telegram re-reads the accessibility tree immediately before writing, rebinds a uniquely moved composer, and refuses moved or ambiguous surfaces; it also excludes Phone number, Search and other login/navigation fields. [integration/full:apps/osl-hub/src/native_telegram_adapter.rs:148-177,429-493] Discord's borrowed-window tether revalidates PID, process creation, session and path before mutation and repairs window presentation while that exact process remains alive. [integration/full:apps/osl-hub/src/native_window_host.rs:226-241,4197-4207,4580-4837]
4. OSL Chats has a durable encrypted queue only for the half-finished relay-notice window after the wrapped key was accepted. It does not authorize an ordinary offline outbox. [integration/full:apps/osl-hub/src/osl_chat_queue.rs:8-28,54-78; RULING Q2; RULINGS.txt:624-630]
5. A reusable committed-send journal already exists in crates/transport, including durable pre-I/O state and authoritative recovery, but send_packaged and recover_once have no production caller in apps/osl-hub; only the crate test exercises them. [integration/full:crates/transport/src/committed_send.rs:218-343; integration/full:crates/transport/tests/task_5195_crash_reconciliation.rs:283-307]
That is partial defensive plumbing, not product self-healing.
Failure matrix
| Break | What the code does now | What it should do | Gap |
|---|---|---|---|
| Carrier app closes or restarts mid-send | The in-memory transaction returns an adapter or restoration error when its next carrier call fails. It has no durable phase to distinguish before placement, placed but unsent, Send invoked, or provider accepted. Discord's suspended native draft and typed-carrier record are explicitly RAM-only, so an OSL/carrier restart loses them. [protected_carrier_transaction.rs:287-347; native_discord_adapter.rs:545-578,667-686,990-1036] The native host can cold-launch or adopt a carrier on a later explicit host call, but its continuous Discord tether repairs only the still-valid borrowed process; it does not relaunch a dead process and resume a send. [native_window_host.rs:5897-6064,8888-8898] | Before the irreversible trigger: refuse, preserve the private draft, remove only text OSL can prove it owns, and require a fresh send gesture. After the trigger may have fired: reopen/rebind read-only, find the stable attempt/provider row, and resolve to Sent or Not sent; if neither can be proved, show Unknown and do not resend. This is the Q2 committed boundary, not a general retry permission. [RULING Q2; TASK 5195; todo/28-seven-acceptance-rulings.txt:160-180] | Dangerous. Today an accepted send can be reported only as a generic failure, while a retry can duplicate it; an existing Discord draft can also be lost when the process dies. [native_discord_adapter.rs:667-686; apps/osl-hub-ui/src/overlay.ts:1781-1847] TASK 3585 is wrong to require “nothing was sent” at every close point; that statement is unprovable after carrier acceptance. [TASK 3585; todo/12-final-audit.txt:2061-2072; TASK 5195] |
| Carrier account is signed out | Telegram and WhatsApp discovery refuse known Phone number/login fields instead of treating them as composers. That protects those measured shapes, but there is no production shared-transaction implementation or mid-transaction authentication state that pauses and later resumes safely. [native_telegram_adapter.rs:148-177; native_whatsapp_adapter.rs:844-872,918-934; protected_carrier_transaction.rs:91-105] | Revalidate process, signed-in account, conversation and recipient immediately before placement and again immediately before Send. A signed-out result blocks with zero write, preserves the draft, and exposes Open carrier/Sign in. After sign-in, reacquire every binding and require a fresh user send gesture; never automate credentials. [01-FULL-SPEC.md:69-75,98; RULING D38] | Dangerous at the boundary; cosmetic after safe refusal. Writing into a login/search field or a different account is harmful. Failing to resume automatically after a correct refusal is only availability. [RULING “SELF-HEALING IS UNIVERSAL”; RULINGS.txt:129-146] |
| Network drops | The shared carrier transaction has no network state. It invokes the carrier's Send trigger and then waits for receive evidence; any later error is folded into SelectedSendTrigger, Adapter or ProtectedReceiveAbsent, with no persisted indication of whether the carrier queued or accepted the message. [protected_carrier_transaction.rs:293-334] OSL Chats separately persists only a relay notice after the first remote effect succeeded. [broker.rs:4818-4866] | Pre-acceptance network failure is Not sent, leaves no delayed work and requires a fresh gesture. Lost acknowledgement after remote acceptance keeps one stable ID and performs read-only reconciliation to exactly one outcome. Never blindly replay through the carrier because the carrier may have its own outbox. [RULING Q2; TASK 5194; TASK 5195; todo/28-seven-acceptance-rulings.txt:135-180] | Dangerous. The current carrier path cannot tell rejection from accepted-but-unobserved and therefore cannot make a truthful terminal claim. [protected_carrier_transaction.rs:303-334; 01-FULL-SPEC.md:15,75] |
| OSL is killed mid-operation | Tauri's single-instance callback can reveal the surviving main window or request an app restart when the retained process has no usable main UI. A separate guardian restores a borrowed carrier window's presentation after OSL dies. Neither mechanism records or reconciles the protected send. [integration/full:apps/osl-hub/src/main.rs:10704-10725; integration/full:apps/osl-hub/src/native_window_host.rs:419-423,3778-3827,3952-4020] protected_carrier_transaction.rs holds all send progress on the stack; Discord's RAII cleanup explicitly does not run on process kill, and its draft/carrier recovery state is RAM-only. [protected_carrier_transaction.rs:262-354; native_discord_adapter.rs:545-578,667-686,17714-17739] A correct committed-send journal exists but is not wired into apps/osl-hub. [crates/transport/src/committed_send.rs:218-343] | Fsync a small encrypted attempt record before each irreversible boundary; on startup, reconcile provider/service facts before enabling Retry. Only post-acceptance attempts may complete automatically, and only under the same stable ID; pre-acceptance attempts return to Draft. [RULING Q2; TASK 5195] | Dangerous. A crash can lose the private draft and the distinction between no effect and a real carrier message. The full spec contains no carrier-send crash state machine even though it requires exact readback before success. [native_discord_adapter.rs:667-686; 01-FULL-SPEC.md:69-75; TASK 6214] |
| Carrier changes its UI | Telegram has real bounded rediscovery of a uniquely moved composer and refuses ambiguity. Discord has substantial live window geometry/presentation repair with fresh identity checks. The shared transaction also defines geometry/appearance repair, but only fake test adapters implement it, so the universal product claim is not wired. [native_telegram_adapter.rs:429-493; native_window_host.rs:4197-4207,4580-4837; protected_carrier_transaction.rs:232-257; TASK 7760 evidence, evidence/7760.md:18-42] | Ship detect-and-refuse first for every actionable carrier. Re-find a composer only from a measured, signed adapter profile and only when account/conversation/recipient remain exact. Defer automatic appearance reconstruction until one carrier works end to end; refusal already prevents wrong-target placement and a mismatched privacy overlay. [RULING “SELF-HEALING IS UNIVERSAL”; RULING “CARRIER VISUAL FIDELITY IS A PASS BAR”; RULINGS.txt:129-146,187-216] | Dangerous if detection/refusal is absent; cosmetic availability if safe refusal works. The existing spec turns automatic repair into the release bar, but the harm-preventing part is detection plus refusal. [01-FULL-SPEC.md:53,98,328] |
| Message is partially placed | The shared transaction treats place_text as one opaque call. If it errors after writing bytes, it does not inspect how many bytes landed or clear an owned span; it merely attempts generic surface restoration. [protected_carrier_transaction.rs:287-345] Discord revalidates each chunk and refuses an uncertain chunk, but its cleanup deletes only an exact full carrier; the code admits a partial prefix can remain after interruption, and hard kill skips Drop cleanup entirely. [native_discord_adapter.rs:17714-17739,17828-17904,18272-18430] Telegram's probe-only path clears after a readback failure, but the file says that path is not a send path. [native_telegram_adapter.rs:485-515] | If the composer is unsent and the exact inserted span is still provably OSL-owned, remove that span and restore the private draft. If ownership is uncertain, touch nothing and show Manual cleanup required. If Send may have fired, reconcile the provider row and never offer a blind Retry. [01-FULL-SPEC.md:74-75; TASK 5631] | Dangerous. A blind retry can append or send a duplicate/corrupt cover, and a partial prefix can be mistaken for the person's draft on the next run. [native_discord_adapter.rs:14317-14378,18417-18430] The WhatsApp error design compounds this: it shows an inert Retry with no zero/partial/unknown distinction. [design/UI-FINAL-INSTRUCTIONS/WhatsApp Overlay Error.dc.html:22-34] |
| Queue drain crashes | The local queue removes a record only after its delivery callback returns success and then saves the shortened queue, so a local save failure retains the item. [integration/full:crates/ipc/src/offline_send_queue.rs:166-195] The promised idempotency is not carried to the actual ordinary relay API: PendingSend.idempotency_key never reaches post_control_inbox, and the retry signs a fresh timestamp. [offline_send_queue.rs:21-29; apps/osl-hub/src/broker.rs:5085-5103; crates/keystore/src/client.rs:1891-1899,1956-1981] The server deduplicates the canonical request digest, which includes that timestamp, while ordinary rows have no collapse key; only revocations upsert on a collapse key. [integration/full:keyserver-cf/src/endpoints/control-inbox.ts:363-396,478-573,622-650] | Make the remote ordinary-send endpoint accept a stable operation key and atomically return the existing result on replay. The drain sequence is durable claim, idempotent remote effect, durable applied marker, then dequeue. Crash at every boundary must converge to zero or one remote row, never two. [TASK 0866b; TASK 5195] | Dangerous and concrete. If the server commits but the response is lost, the queue retains the item; after restart the fresh request digest can append a second ordinary row. [broker.rs:5089-5104; keyserver-cf/src/endpoints/control-inbox.ts:387-396,561-573] |
| Carrier profile is locked by another instance; lock record names a reused PID | tauri-plugin-single-instance prevents a second process for the same bundle and focuses/restarts the existing OSL process. [integration/full:Cargo.toml:85-92; integration/full:apps/osl-hub/src/main.rs:10704-10725] That is not a carrier-profile lease. Discord's persistent claim file binds a channel to an OSL owner but contains no live process owner; the same owner can reopen it, and prepare_profile only creates/validates directories. [native_window_host.rs:3004-3111,7504-7542] Dedicated launch retries and returns ProfileInitializationFailed when it cannot obtain a usable Telegram window; it neither locates the profile owner nor reclaims a stale carrier lock. [native_window_host.rs:6075-6107] PID plus creation-time checks protect borrowed HWND mutation, not carrier-profile ownership. [native_window_host.rs:226-241,4197-4207] | Hold one OS kernel/file lease per carrier profile. Record PID, process creation identity, executable and bundle/owner identity for diagnosis, but use the held lease—not the PID file—as authority. Focus the verified live owner; if the lease is free, reclaim stale metadata without killing any process. Serialize all profile writes and make different QA bundle IDs contend on the same lease. | Dangerous when two writers open one carrier profile; otherwise an availability failure. The current Tauri and stale-PID protections solve different problems and do not establish profile recovery. No owner ruling, full-spec clause or product task covers this case; TASK 7505 is only a QA helper identity check. [TASK 7505; todo/46-capability-routes.txt:69-79; 01-FULL-SPEC.md has no profile-lock contract] |
The UI does not supply the missing recovery
The rendered design has one correct fail-closed state: when composer reach is false, Strip says OSL cannot reach the composer and protected sends would go in the clear, then blocks the private composer. [design/UI-FINAL-INSTRUCTIONS/Strip.dc.html:158-177,469-475,651-667] The apparent reconnect copy in renderVals is dead data: the template never renders stripReachDetail or stripReachAction, and the visible lock click only toggles sliceOn. [Strip.dc.html:54,838-850]
There is no rendered state for carrier process exit/restart, carrier sign-out, network loss, startup send reconciliation, queue recovery or another-instance profile ownership. Settings Account Locked is an OSL identity-unlock page, not a carrier-profile lock page. [design/UI-FINAL-INSTRUCTIONS/Settings Account Locked.dc.html:34-36; design/UI-FINAL-INSTRUCTIONS/README.md:34]
The Proof dialog is a contract, not evidence: it contains hard-coded demo receipts such as carrier accepted/read back/confirmed and Nothing was sent. Production must never select one after an interrupted attempt unless persisted evidence establishes it. [design/UI-FINAL-INSTRUCTIONS/Strip.dc.html:256-266,613-617,760-764; design/UI-FINAL-INSTRUCTIONS/README.md:73-74]
Requirements that are poisoning the working path
1. Cut TASK 5633 from V1. It builds a second recovery worker, full carrier-state snapshot, consent/version-token protocol, recovery journal, state lease, boot recovery and 30-second relaunch guarantee for an OSL-requested carrier restart. OSL does not need to restart carriers to send a message; V1 should never initiate that restart. Detect a carrier exit, preserve/reconcile the attempt, and let the person reopen it. The giant restart subsystem buys little safety once automatic carrier restart is removed. [TASK 5633; todo/11-typing-into-other-apps.txt:713-734]
2. Trim TASK 6214 to the dangerous core. Keep encrypted durable attempt state, one stable ID, post-acceptance reconciliation, truthful Sent/Not sent/Unknown, and an effective Cancel. Defer the 15-point campaign, 2,000/10,000 item limits, 128 MiB/1 GiB limits, disk-floor formula and two-writer edge matrix until the basic path works. [TASK 6214; todo/12-final-audit.txt:8220-8244]
3. Correct TASK 6214's retry rule. “Never discard or terminally fail Pending automatically; deliver exactly once on recovery unless Cancel won” wrongly authorizes pre-acceptance sends to leave later. Q2 forbids that, and TASK 5195 requires every pre-acceptance kill to produce zero later placement/delivery. Pending before provider acceptance must return to Draft and require a fresh gesture; only an already accepted operation may reconcile automatically. [RULING Q2; RULINGS.txt:624-630; TASK 6214; todo/12-final-audit.txt:8227-8230; TASK 5195; todo/28-seven-acceptance-rulings.txt:160-180]
4. Defer automatic runtime appearance repair. Keep runtime drift detection and visible refusal now. Perfect automatic restyling is valuable later, but it does not stop harm beyond what a correct refusal already stops, and it must not block making one carrier actually send and recover. [RULING “CARRIER VISUAL FIDELITY IS A PASS BAR”; RULINGS.txt:187-216; owner working-product-first instruction supplied for this audit]
5. Delete the false universal proof claim, not the requirement. TASK 7760 proves a fake adapter can satisfy the trait; TASK 7761 admits Discord was not observed; TASK 7762's printed C:/OSL/telegram/alice and bob facts come from a test host, not Telegram Desktop. Keep these as unit tests, but do not call them live carrier proof. [evidence/7760.md:18-53; evidence/7761.md:83-100; evidence/7762.md:19-40; integration/full:apps/osl-hub/tests/task_7762_telegram_protected_carrier.rs:218-272]
The spec also miscites universal self-healing as D30 at lines 98 and 328. D30 is unrelated; the authority is the unnumbered 9 August universal-self-healing ruling. Ordinary offline refusal at spec lines 25 and 90 likewise omits Q2, losing Q2's crucial exception for an already committed crash retry. [01-FULL-SPEC.md:25,90,98,328; RULINGS.txt:129-146,624-630,1648-1685]
Three highest-value fixes
1. Put one real carrier behind a durable send state machine
Wire the production carrier driver—not a fake host—through one state machine with an encrypted fsynced attempt record and stable ID: Draft → PlacedUncommitted → CommitUnknown → Sent, plus NotSent and Cancelled. Persist the suspended native draft and confirmed placement prefix as part of that record. Reuse the existing committed_send journal rather than inventing another test-only model. Revalidate carrier process/account/conversation immediately before placement and Send. On startup or carrier return, reconcile read-only before permitting Retry. Never auto-replay Draft, PlacedUncommitted or CommitUnknown; only finish an effect the provider has already authoritatively accepted. [protected_carrier_transaction.rs:262-354; native_discord_adapter.rs:545-578,667-686,18417-18430; crates/transport/src/committed_send.rs:218-343; RULING Q2; TASK 5195]
Start with Discord or Telegram, whichever can produce a real two-owner-identity receipt fastest. Do not attempt all carriers at once. A single real path exercised at carrier-exit, network-loss, partial-placement and process-kill boundaries is more product than the current universal fake conformance test. [evidence/7760.md:18-42; evidence/7761.md:83-100; 01-FULL-SPEC.md:384]
2. Make recovery effects genuinely idempotent
Carry the stable operation ID into the remote ordinary inbox API and enforce a unique/upsert result server-side. Persist the local applied marker before dequeue. Use the same primitive for carrier reconciliation where the provider exposes a stable row ID; otherwise leave the result Unknown and require manual inspection. This fixes the concrete queue-drain duplication window and supplies the foundation for crash recovery. [offline_send_queue.rs:166-195; broker.rs:5085-5104; keyserver-cf/src/endpoints/control-inbox.ts:387-396,561-573; TASK 5195]
3. Add a real cross-instance carrier-profile lease
Key an OS-held lease by canonical carrier-profile path, not Tauri bundle ID. Verify a live owner by PID plus creation time and signed executable, focus it when appropriate, and reclaim stale metadata only after the lease is free. Test two differently bundled OSL instances and forced PID-record reuse. This directly closes the case that Tauri single-instance and HWND PID checks do not cover. [Cargo.toml:85-92; main.rs:10704-10725; native_window_host.rs:226-241,3004-3111,6075-6107]
Release judgment
Self-healing is not established for any production carrier send on canonical integration/full. Safe local refusal exists for some measured surfaces, and OSL Chats has a narrow durable half-send queue, but there is no production carrier transaction with durable recovery and the queue's advertised idempotency does not reach the ordinary remote write. [protected_carrier_transaction.rs:91-105,262-354; osl_chat_queue.rs:8-28; broker.rs:5085-5104; keyserver-cf/src/endpoints/control-inbox.ts:387-396,561-573]
The ship bar should therefore be small and honest: one real carrier, one durable attempt journal, one stable remote ID, four forced breaks, zero duplicates, zero wrong-target writes, and no Sent/Not sent claim without observed proof. [RULING D38; RULING Q2; TASK 5195; owner working-product-first instruction supplied for this audit]