8. Operating and Extending

8. Operating and Extending

Where we left off

Docs 1 to 7 explained how the system works. This last doc is for the person who has to debug it on a real phone, tune it, or change it. It covers:

• what the logs mean
• how to reproduce the hard cases
• what the tests cover and what they don’t
• the constants worth knowing
• the known gaps
• step-by-step recipes for common changes

Reading the logs

Every download log line starts with a component tag:

Tag	Component
[dl-mgr]	DownloadManagerImpl
[orch]	DownloadOrchestrator
[apt]	AssetProgressTracker
[dl-notif]	DownloadProgressNotifier (Android notifications and the service channel)
[dl-tip]	the OEM protection tip
[connectivity]	ConnectivityCubit (Wi-Fi-only pause/resume)
[bgdl], [startup]	configure() and DownloadManager.initialize() at launch
[migrate], [hive]	storage migration and box opening
[proxy]	the offline playback server
BackgroundDownloader, TaskRunner	the package’s native Android logs

The lines that tell the story of a download:

Log line	Meaning
[orch] <id>: N missing on disk, M already active, K re-enqueued	A re-queue pass ran. K = 0 with N > 0 means everything is still waiting in the native queue.
[orch] X/Y tasks failed to enqueue after retry; settlement will recover them	The package rejected some tasks twice. Harmless unless it repeats.
[dl-mgr] watchdog: <id> stalled (c/t) — settling	No progress for 2 minutes.
[dl-mgr] settlement: X missing, Y re-enqueued for <id> (attempt Z)	A recovery round. Z climbing to 3 without progress ends in failed.
[dl-mgr] finalize deferred for <id>: N files missing on disk	The counter said done; the disk disagreed. The download continues.
[dl-mgr] finalize failed for <id>: …	Writing the playlists threw, so the row is failed.
[dl-mgr] cold-start re-orchestrate failed for <id>: …	Launch-time recovery could not queue work, often because there is no network yet. The watchdog tries again later.
[dl-notif] fg update failed: …	The progress service could not be updated.
[startup] DownloadManager.initialize() failed/timed out	Initialization threw or took longer than 15 s. ready still completes.

On Android, filter logcat to just the download story:

adb logcat | grep -E "\[dl-mgr\]|\[orch\]|\[apt\]|\[dl-notif\]|\[connectivity\]|BackgroundDownloader|TaskRunner"

Reproducing the hard cases (Android)

All commands assume the package com.prothomalo. run-as only works on debuggable builds.

Scenario	How to reproduce	What should happen
6-hour dataSync limit	adb shell device_config put activity_manager data_sync_fgs_timeout_duration 120000, then start a download, press Home and lock the phone. Undo with adb shell device_config delete activity_manager data_sync_fgs_timeout_duration.	After 2 minutes the progress notification is replaced by “Download paused / Open Chorki to continue downloading”. No crash in logcat. Opening the app restarts the service and the download continues.
Background job quotas	adb shell am set-standby-bucket com.prothomalo rare, then kill the process (below). Check with am get-standby-bucket.	With the process gone, only jobs already in WorkManager trickle. Opening the app resumes at full speed.
Process death	Background the app, then adb shell am kill com.prothomalo.	Nothing re-downloads on the next launch. [orch] … re-enqueued shows only missing files.
OEM-style force stop	adb shell am force-stop com.prothomalo	Everything stops until the app is opened. Then cold start recovers.
Back button	Press Back twice on Home while downloading.	The app goes to the background, and the notification keeps advancing.
Flaky network	Airplane mode for about 3 minutes, several times during one title.	The row never becomes failed. The title completes and plays offline.
Wi-Fi only	Turn the setting on, then switch Wi-Fi off and on.	Downloads pause, then resume.
Doze	adb shell dumpsys deviceidle force-idle (undo: unforce)	The foreground service keeps network access, so downloading continues.

Inspecting state on the device:

# files for one title (count should approach totalSegments + playlists + art)
adb shell run-as com.prothomalo sh -c 'ls app_flutter/<slug> | wc -l'

# package task map size: should stay small with the holding queue
adb shell run-as com.prothomalo ls -l shared_prefs/

# is the download service running as a foreground service?
adb shell dumpsys activity services com.prothomalo | grep -i -A3 DownloadForegroundService

# WorkManager's own view (written to logcat under the WM-Diagnostics tag)
adb shell am broadcast -a "androidx.work.diagnostics.REQUEST_DIAGNOSTICS" -p com.prothomalo

# JobScheduler's view, including quota state
adb shell dumpsys jobscheduler | grep -A5 com.prothomalo

On iOS, reproduce slow networks with Network Link Conditioner (Settings → Developer). To see the resource timeout in action, temporarily set Config.resourceTimeout to a few minutes: queued segments then fail with NSURLErrorTimedOut (-1001).

Tests

fvm flutter test test/data/services/hls/
fvm flutter test test/presentation/bloc/download_screen/download_screen_bloc_test.dart

Test	Covers
test/data/services/hls/asset_progress_tracker_test.dart	Duplicate complete events count once; the counter never passes the total; per-title isolation; forget; resetCounter moving backwards.
test/data/services/hls/hls_manifest_parser_test.dart	variantsFromMaster and pickVariant: each quality tier, falling back below the target, smallest-when-all-taller, widescreen ladders, keeping the #EXT-X-STREAM-INF line.
test/data/services/hls/download_task_factory_test.dart	Task IDs stay unique across audio renditions, build and buildMissing agree on them, and filtering matches by destination file.
test/data/services/download_visibility_test.dart	Who may see a download: signed out, another account, rows with no owner, maturity levels, unrated titles.
test/data/helpers/content_helper_download_test.dart	The storage estimate and human-readable sizes.
test/presentation/bloc/download_screen/download_screen_bloc_test.dart	The bloc’s translation of events into DownloadManager calls, with the manager mocked.

Not covered by automated tests:

• DownloadManagerImpl and DownloadOrchestrator. They call the FileDownloader() singleton and Hive directly.
• All native code. The Android service, the channel and the package.

Those need the device scenarios above. If you change settlement, finalize or cold start, run at least process death, flaky network and the 6-hour limit.

Tuning knobs

Constant	Value	Where	Trade-off
Holding queue size (Android)	8	lib/main.dart _initBackgroundDownloader	Higher can use fast Wi-Fi better, but costs more package bookkeeping, more files broken by each signal drop, and more load on the CDN.
Request (connect / idle) timeout	60 s	same	Shorter fails faster on dead links, but fails more on slow ones.
iOS resource timeout	7 days	same	Must comfortably exceed the slowest title download.
Enqueue chunk	200 tasks	DownloadOrchestrator._enqueueChecked	Platform-channel payload size vs number of calls.
Package retries per file	3 (2 s, 4 s, 8 s)	DownloadTaskFactory._buildTask	Retries run in Dart only.
Stall threshold	2 min	DownloadManagerImpl._stallThreshold	Shorter recovers sooner, but runs more disk walks on slow links.
Watchdog period	30 s	DownloadManagerImpl._initialize
Settlement budget	3 rounds without progress	DownloadManagerImpl._maxSettlementAttempts	Refilled by any progress.
Disk recount debounce	15 s	AssetProgressTracker.scheduleReconcile
Row save debounce	1 s	DownloadManagerImpl._scheduleSave
All-records stream coalescing	250 ms	DownloadManagerImpl._emitAll
Cold-start reconcile concurrency	2 titles	DownloadManagerImpl._coldStartReconcile
Finalize re-parse	2 attempts × 30 s	DownloadOrchestrator._parseWithRetry	Only when no plan was saved.
Startup limits	configure 6 s, initialize 15 s	lib/main.dart
Default download quality	720	AppBootstrapImpl._restoreDownloadPreferences, Download settings
Download limit	50 titles	ContentHelper.maxDownloads	Client-side rule; counts the account’s downloads that have not failed.
Storage safety margin	50 MB	ContentHelper._storageMarginBytes	Added to every size estimate.
Manifest fetch for the estimate	3 s	ContentHelper._estimatedDownloadBytes	Longer delays the tap; shorter skips the check more often.

Known limitations

These are real gaps in the current code, listed so nobody mistakes them for intended behaviour:

1. Visibility is enforced in the app, not in the files. A download hidden from an account or a restricted profile keeps its files on disk, and if it is still running it keeps downloading. The Android progress notification counts it too, because stopping the service would stall the transfer.
2. Downloads made before owners were recorded have no owner. They stay visible to any signed-in account until someone re-downloads that title, which records the new owner.
3. The storage check is an estimate. It uses the rendition’s peak BANDWIDTH and the title’s duration, and it deliberately lets the download proceed whenever it cannot work the size out. A disk that fills up mid-download still shows up as failing files.
4. The 50-download limit is a client-side rule with no backend field behind it. It counts the downloads the signed-in account can see.
5. A restricted profile cannot see unrated downloads. That mirrors the catalog, so a title with no contentRatingMinAge is hidden from a profile with a maturity level.
6. Android has no read timeout. The package sets only a connect timeout, so a dead socket holds one of the eight slots until the 9-minute worker limit.
7. Android loses the holding queue on process death. This was an accepted trade-off (doc 5): the rest resumes when the app is opened.
8. OEM force-stops halt everything until the app is opened again.
9. Wi-Fi-only auto-resume lives in memory and only runs while the app process is running.
10. On iOS, retries and recovery need the app to run. What swiping the app away does to transfers has not been verified on a device.
11. Short-lived key tokens would break downloads. If the proposed KMS authorization ships, the key task re-queued hours later would carry an expired token (../playback/06-the-new-kms-authorization.md, section 3).
12. The key is stored in the clear next to the segments it unlocks (../playback/05-threat-model-and-gaps.md).
13. Some sheet and dialog text is hard-coded in English: “Waiting For WiFi”, “Cancel Download”, “Remove from download”, “Download Paused”, “Downloads upgraded”.
14. DownloadStatusV2.cancelling is never assigned. It is dead code in switches and guards.

The long-term direction for Android is one user-initiated data transfer job per title, implemented natively. Those jobs are exempt from background quotas and the 6-hour dataSync limit, and Google recommends them for exactly this kind of transfer on Android 15 and 16. That would replace the per-file WorkManager jobs and the app’s foreground service.

Recipes

Change how many files Android downloads at once

1. Edit the first number in (Config.holdingQueue, (8, null, null)) in lib/main.dart.
2. Test on a low-end phone on mobile data, not just office Wi-Fi.
3. Watch for CDN errors in TaskRunner logs and for more [orch] … re-enqueued lines after signal drops.

Add a new kind of file to a download

1. Add a constant to TaskKinds in download_task_factory.dart.
2. Build its task in DownloadTaskFactory.build, with a unique, stable index per file of that kind.
3. Decide whether it is required for playback:
   • Required:
     • add it to HlsParsedPlan.totalRequiredCount
     • add it to DownloadOrchestrator._findMissingOnDisk
     • add it to DownloadTaskFactory.buildMissing
     • add it to DownloadManagerImpl._isCountedKind
     • make sure its file extension is counted by AssetProgressTracker._diskCount
   • Side file (like the poster): add a nullable Hive field with a new index for its local path, and set it in DownloadManagerImpl._onStatusUpdate on complete.
4. If it must appear in a playlist, update LocalPlaylistWriterImpl.
5. Regenerate adapters: dart run build_runner build --delete-conflicting-outputs.

Change the Hive row

• Adding a field: use the next free @HiveField index (currently 22), give it a default, and regenerate the adapters. Existing downloads keep working.
• Renaming, retyping or reusing an index is a breaking change. Bump _currentDownloadsSchemaVersion in lib/main.dart, which deletes every user’s downloads on update and shows the “Downloads upgraded” notice. Avoid it unless unavoidable.

Change the download limit

• The number is ContentHelper.maxDownloads; the sheet’s wording is download_limit_body in the ARB files.
• The count comes from DownloadManager.list(), so it only counts what the signed-in account can see.

Change which quality is downloaded

• The rendition choice is HlsManifestParserImpl.pickVariant; the target and account cap are DownloadOrchestrator._downloadResolution.
• Add a case to test/data/services/hls/hls_manifest_parser_test.dart for any new rule.
• Saved plans keep their quality, so the change affects new downloads only.

Change notification text

1. Edit the download_notification_* keys in lib/l10n/intl_en.arb and lib/l10n/intl_bn.arb.
2. Run fvm flutter gen-l10n. l10n.yaml points at intl_en.arb.
3. On iOS the text must stay constant for the whole download (doc 6).

Change the Android foreground service

• Keep the ordering rules in doc 5: stop as a start command, startForeground in every branch, stopSelf(startId) when stopping, and stopSelf() in onTimeout.
• Test:
  • cancel immediately followed by re-download of the same title
  • download completing while another is queued
  • the 6-hour limit via device_config

Upgrade background_downloader

The design relies on these package behaviours. Re-check each against the new version’s source before upgrading:

Assumption	Relied on by
pause() only affects running tasks	pause implemented as cancelAll
Retries are scheduled in Dart	the recovery layers in doc 4
FileDownloader().start() replays undelivered updates; listeners must attach first	DownloadManagerImpl._initialize ordering
enqueueAll returns per-task success flags	_enqueueChecked
Enqueues are not deduplicated by task ID, and neither is the holding queue	the active-task filter, and start returning early for an active row
allTasks(group:) includes held and waiting-to-retry tasks	_withoutActiveTasks
Holding queue config can be Android-only	androidConfig in main.dart
iOS group notification with complete: null strands a stale banner	_configureGroupNotification
Android keeps all WorkManager-enqueued tasks in one SharedPreferences map	the holding queue’s size (doc 5)

Where this lives in the code

• Every file listed in the README’s “Map to the code” table.
• test/data/services/hls/ and test/presentation/bloc/download_screen/.

The takeaway

The download system is small in files but large in edge cases. The logs narrate each recovery decision, the adb recipes reproduce the conditions users hit on the train and overnight, and the tables above record every constant and assumption the design depends on. Change anything in the recovery or Android service paths only with the device scenarios at hand.