4. When Things Go Wrong

Where we left off

Doc 3 described the happy path: files finish, a counter climbs, the disk check passes, and the playlists are written. Real downloads rarely go that smoothly:

a mobile signal drops for a minute
the phone locks and the operating system freezes the app
a CDN node returns an error
a task never makes it into the native queue
the app is swiped away halfway through a film

This doc covers the layers that turn those events into “it finished anyway”, and the four user-visible actions: pause, resume, retry and cancel.

The shape of the recovery system

Recovery is layered. Each layer catches what the one inside it cannot:

 ┌───────────────────────────────────────────────────────────────────┐
 │ 6. COLD START        app process starts: rebuild state from Hive,  │
 │                      the package database and the disk; re-queue   │
 │ ┌───────────────────────────────────────────────────────────────┐ │
 │ │ 5. APP RESUME      app returns to the foreground: recount the  │ │
 │ │                    disk, finalize whatever finished meanwhile  │ │
 │ │ ┌───────────────────────────────────────────────────────────┐ │ │
 │ │ │ 4. SETTLEMENT    compare the plan with the disk; re-queue  │ │ │
 │ │ │                  only missing files; give up after 3       │ │ │
 │ │ │                  fruitless rounds                          │ │ │
 │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │
 │ │ │ │ 3. STALL WATCHDOG   no progress for 2 min → settle    │ │ │ │
 │ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ │
 │ │ │ │ │ 2. MISS SET     every task has finished, some     │ │ │ │ │
 │ │ │ │ │                 failed → settle now               │ │ │ │ │
 │ │ │ │ │ ┌───────────────────────────────────────────────┐ │ │ │ │ │
 │ │ │ │ │ │ 1. PACKAGE RETRIES   3 retries per file,      │ │ │ │ │ │
 │ │ │ │ │ │                      after 2 s, 4 s, 8 s      │ │ │ │ │ │
 │ │ │ │ │ └───────────────────────────────────────────────┘ │ │ │ │ │
 │ │ │ │ └───────────────────────────────────────────────────┘ │ │ │ │
 │ │ │ └───────────────────────────────────────────────────────┘ │ │ │
 │ │ └───────────────────────────────────────────────────────────┘ │ │
 │ └───────────────────────────────────────────────────────────────┘ │
 └───────────────────────────────────────────────────────────────────┘

Layers 2 to 4 all feed the same routine, settlement, so there is one implementation of “figure out what is missing and fix it”.

Layer 1: package retries

Every task is created with retries: 3. When a file fails, the package:

reports waitingToRetry
waits 2 s, then 4 s, then 8 s
enqueues the task again

Only after the fourth failure does it report failed.

These waits are Dart timers inside the package. They only fire while the app’s Dart code is running:

Android. When a file fails, the package deletes its WorkManager job natively so the OS does not reschedule it, and leaves the retry to Dart. If the Flutter engine has been destroyed, nobody retries.
iOS. Dart is suspended in the background. Failures that happen while the phone is locked wait until the app is opened.

That gap is why the outer layers exist, and why Android keeps the Dart side alive during downloads (doc 5).

Layer 2: the miss set

When a counted task (video, audio or key) reports a final failed, notFound or canceled, _onStatusUpdate adds its task ID to _missedTaskIds[contentId], then calls _maybeSettle:

if counter + misses ≥ totalSegments:   every task has reached an end state
    → settle now                         (do not wait for the watchdog)

During a healthy download there are no misses, so this never fires early. It exists so that a download whose last few files failed does not sit at “downloading” for two minutes before anything happens. A later complete for the same task removes it from the set.

Layer 3: the stall watchdog

A periodic timer runs _runStallWatchdog every 30 seconds. For each row it checks:

status == "downloading"
and totalSegments > 0
and completedSegments < totalSegments
and now − lastAdvance ≥ 2 minutes          → settle this row

lastAdvance is refreshed:

whenever the counter moves
whenever a recovery pass queues work
when the app comes back to the foreground (Dart was asleep, not stalled)

A settlement pass that finds everything still queued natively also resets it, so a slow but healthy download is checked at most once every two minutes.

Layer 4: settlement

_settleDownload is a per-title mutex around _settleImpl. Concurrent triggers wait for the pass already running.

_settleImpl(contentId)
  │
  ├─ row is "completed", "cancelling" or "failed"?     → nothing to do
  │
  ├─1─ DISK TRUTH
  │      reconcileNow   (refresh the counter from the disk)
  │      missingOnDisk  (name-by-name check against the plan)
  │      nothing missing? → _safeFinalize, done
  │
  ├─2─ BUDGET
  │      settlementAttempts ≥ 3?
  │        → row "failed": "N segments could not be downloaded. Please try again."
  │        → cancelAll(group)      nothing may keep trickling
  │        → "Download failed" notification, refresh/stop the progress notification
  │        → done
  │
  └─3─ RE-QUEUE
         clear the miss set
         reenqueueMissing(row)
         reset the stall clock
         enqueued anything?  → settlementAttempts += 1 (saved)
         log "[dl-mgr] settlement: X missing, Y re-enqueued (attempt Z)"

The budget, and why it refills

settlementAttempts limits how many fruitless recovery rounds a title gets:

Event	Effect on `settlementAttempts`
A settlement pass queues at least one file	+1
A settlement pass finds everything already queued	unchanged: waiting is not failing
The counter moves forward	reset to 0
`start` on an existing row, `retry`	reset to 0
Cold start resumes a `failed` row	reset to 0

The refill is what lets a two-hour download survive a flaky evening. Each recovery that leads to real progress earns another three. A file that can never be fetched, such as a permanent 404, still ends the download:

Everything else finishes.
Progress stops.
Three rounds in a row re-queue the same file and get nowhere.
The row becomes failed.

`reenqueueMissing`: re-queue only what is missing

Every recovery path queues work through DownloadOrchestrator.reenqueueMissing(row), never through a full start:

reenqueueMissing(row)
  │
  ├─ no saved plan?  → full orchestrator.start(row)    returns missing = −1
  │
  ├─ walk the disk against the plan → set of missing names
  │     none missing                                   returns missing = 0
  │
  ├─ buildMissing: tasks for exactly those files (same deterministic IDs)
  │
  ├─ remove tasks whose destination file the package is already working on:
  │     FileDownloader().allTasks(group) → the files of tasks queued,
  │     running, held in Android's holding queue, or waiting to retry
  │     (if that query throws, keep all of them: a duplicate beats a stall)
  │
  └─ _enqueueChecked(the rest)                          returns (missing, enqueued)

Result	Meaning for the caller
`missing = 0`	Complete on disk: finalize.
`missing = −1`	The plan was gone, so a full start was done instead.
`missing > 0`, `enqueued = 0`	Everything outstanding is already queued. Wait.
`missing > 0`, `enqueued > 0`	Real recovery happened.

Filtering by active task ID is essential. A file still waiting in the queue is also “missing on disk”, and queueing it again would create a second native job for the same file: double the bandwidth, and a complete event twice.

Layer 5: returning to the foreground

DownloadManagerImpl registers an AppLifecycleListener. On every resume, _onAppResume does the following for each downloading or paused row with a known total:

reset the stall clock                    (the silence was Dart sleeping)
reconcileNow                             (files may have finished natively)
disk count ≥ total → _safeFinalize       (which still checks name by name)

It then forces a notification sync. That restarts the Android foreground service if the OS refused a start while the app was in the background, or stopped it at the 6-hour limit (doc 5).

Layer 6: cold start

When the app process starts, DownloadManager.initialize() rebuilds everything from persistent state (doc 7 has the startup sequence):

initialize()
  │
  ├─ listen to FileDownloader().updates           (before start, see doc 3)
  ├─ FileDownloader().start(autoCleanDatabase: true, doRescheduleKilledTasks: false)
  │     • track tasks in the package's own database
  │     • replay status updates the native side could not deliver
  │     • mark tasks whose destination file already exists as complete
  │     • clean old database records (keeps ≤ 500 records, ≤ 10 days old)
  ├─ ask for notification permission if not yet granted
  ├─ initialize the Android notifier
  │
  ├─ _coldStartReconcile
  │     1. count "complete" records per title in the package database
  │     2. for each Hive row: load it into memory; seed the tracker with that
  │        count; if the row's saved progress is lower, raise and save it
  │     3. reconcile rows two at a time, in the background:
  │
  │        _reconcileSingleRow(row)
  │          paused or cancelling          → leave alone
  │          reconcileNow (disk count)
  │          completed with a local master → leave alone
  │          disk count ≥ total            → mark 100%, _safeFinalize
  │          otherwise:
  │            reenqueueMissing(row)
  │            not "downloading"? → "downloading" (a failed row also gets
  │                                  a fresh budget and loses its error)
  │            nothing missing → _safeFinalize
  │            else            → force the progress notification
  │          reenqueueMissing threw (e.g. no network yet)?
  │            → still "downloading"; the watchdog and settlement decide later
  │
  ├─ publish all records; watch the language for notification text
  ├─ force the progress notification; start the 30 s watchdog
  └─ register the lifecycle listener (layer 5)

Cold start deliberately resumes failed downloads. A download marked failed on a bad night gets another chance the next time the app opens, without the user tapping Retry.

Why doRescheduleKilledTasks: false. The package has its own cold-start recovery. It re-enqueues tasks its database lists as enqueued or running that the native side no longer holds. The app already rebuilds its queue from the disk, so running both would enqueue the same files twice. That is especially true on Android, where tasks waiting in the holding queue are forgotten when the process dies.

The user’s actions

Action	What the manager does	Files on disk
Pause `pause(id)`	Sets the row to `paused` first, so the `canceled` updates that follow are not treated as misses. Calls `cancelAll(group)`, shows “Download paused” on Android, refreshes the progress notification.	Kept. In-flight partial files are discarded.
Resume `resume(id)`	Sets `downloading` and clears the error. Dismisses the paused notice, forces the progress notification, recounts the disk, `reenqueueMissing`, finalizes if nothing is missing.	Reused. Nothing is downloaded twice.
Retry `retry(id)`	Sets `queued`, resets the budget and the miss set, forces the notification. Recounts, `reenqueueMissing`, sets `downloading`, dismisses the failed notice, finalizes if nothing is missing. If this throws, the row becomes `failed` again.	Reused.
Cancel / Remove `cancel(id)`	Optimistic. Forgets the tracker state, cached plan, save timer, misses and stall clock. Dismisses notices, deletes the Hive row, emits `null` to watchers, refreshes the notification. Then `cancelAll(group)`, deletes the package’s database records for the group, deletes `<Documents>/<slug>/`.	Deleted.

A few facts about these actions are not obvious:

Pause is implemented as cancel. The package’s own pause() only affects tasks that are already running. The thousands still queued would carry on downloading. Cancelling the group really stops everything, and resume works because the queue is rebuilt from the disk.
Pause comes from two places. The Download bottom sheet offers “Pause download” while a title is downloading or queued, and the Wi-Fi-only rule below pauses automatically. Tapping a paused title resumes it.
Cancel removes the row before the slow native cleanup finishes. The item disappears the instant the user confirms. Any task update that arrives during cleanup finds no row and is dropped.

Wi-Fi-only downloads

When Download over Wi-Fi only is on, ConnectivityCubit enforces it:

connectivity report: primary network is not Wi-Fi (and not a VPN)
   └─ _pauseAllInFlight: pause every "downloading"/"queued" row,
                         remember their IDs

connectivity report: Wi-Fi again
   └─ _resumeAutoPaused: resume each remembered row that is still paused

The remembered IDs live in memory, so an app restart forgets them. The rule also only runs while the app process is running. On Android, pausing stops the foreground service, so if the phone later rejoins Wi-Fi while the app sits frozen in the background, the resume happens the next time the app runs. The tap-time check (doc 2) prevents starting a download on mobile data at all.

How the system avoids downloading a file twice

Mechanism	Where
`start` returns early for a row already `queued` or `downloading`	`DownloadManagerImpl.start`
Every re-queue is filtered against the destination files the package is already working on	`DownloadTaskFactory.excludingActive`, `DownloadOrchestrator._withoutActiveTasks`
The package’s own cold-start reschedule is off	`FileDownloader().start(doRescheduleKilledTasks: false)`
A `complete` for an already-counted file is ignored	`AssetProgressTracker.onSegmentCompleted`
Leftover tasks are cancelled after completion	`_safeFinalize` → `cancelAll(group)`
Completion requires a name-by-name disk check	`_resumeIfIncomplete`

What “failed” means

A row becomes failed for one of three reasons:

Cause	Error message on the row
Three settlement rounds in a row queued files without any progress	“N segments could not be downloaded. Please try again.” (localized)
Writing the playlists threw	the exception text
`start` or `retry` threw before anything was queued, e.g. the manifest could not be fetched	the exception text

In every case the title’s native tasks are cancelled or were never queued, the Android progress notification stops if nothing else is active, and a “Download failed” notice appears. The user recovers by tapping the title (which sends a retry) or simply by reopening the app (cold start resumes failed rows).

A flaky evening, as a timeline

 21:00  tap Download → row queued → service on → 1,804 tasks queued
 21:00–21:40  files complete steadily; counter 0 → 1,050
 21:41  signal drops; 8 in-flight files fail and wait 2 s, 4 s, 8 s
 21:41  the retries sit in the native queue waiting for a network
 21:44  watchdog: 2 min with no progress → settlement:
          disk check → 754 missing, all 754 already queued → budget untouched
 21:52  signal back; files complete again; counter keeps climbing
 22:30  3 segments failed four times (CDN hiccup) → miss set
 22:31  counter + misses = total → settlement → 3 re-queued (budget 1)
 22:31  they complete → counter moves → budget back to 0
 22:31  counter = total → disk check: 0 missing → playlists written → completed

Where this lives in the code

lib/data/services/download_manager_impl.dart: _maybeSettle, _settleDownload, _settleImpl, _runStallWatchdog, _onAppResume, _resyncOnResume, _coldStartReconcile, _reconcileRowsWithLimit, _completedSegmentsByContentFromPkgDb, _reconcileSingleRow, pause, resume, retry, cancel.
lib/data/services/hls/download_orchestrator.dart: reenqueueMissing, _withoutActiveTasks, missingOnDisk.
lib/presentation/bloc/connectivity/connectivity_cubit.dart: _pauseAllInFlight, _resumeAutoPaused.
lib/presentation/widgets/bottomsheet_widget/download_bottomsheet.dart: the actions the user can take.

The takeaway

The system never tries to remember exactly which native task did what. Its recovery question is always the same: according to the saved plan, which files are not on disk yet, and which of those is nobody already working on? Every trigger (a failure, a silence, a return to the app, a fresh process) asks that question, and the answer comes from local, durable state. A bounded budget, refilled by real progress, keeps it from asking forever.

4. When Things Go Wrong

4. When Things Go Wrong

Where we left off

The shape of the recovery system

Layer 1: package retries

Layer 2: the miss set

Layer 3: the stall watchdog

Layer 4: settlement

The budget, and why it refills

reenqueueMissing: re-queue only what is missing

Layer 5: returning to the foreground

Layer 6: cold start

The user’s actions

Wi-Fi-only downloads

How the system avoids downloading a file twice

What “failed” means

A flaky evening, as a timeline

Where this lives in the code

The takeaway

`reenqueueMissing`: re-queue only what is missing