Phase 4 — the E2E suite

Phases 0–3 covered the app from the inside: unit, widget, golden and contract tests, all of them running in flutter test with mocked platform channels and no network. Phase 4 is the other half — the real app, the real API, on a real simulator, doing what a viewer does.

That machinery already exists and works. Phase 4 is not “build an E2E harness”; it is “write the flows it is waiting for”, plus put Android into the matrix. This document is the walkthrough for doing that from a cold start.

Written 2026-09-18, at the end of Phase 3. The branch to work on is test/phase-4-e2e, stacked on test/phase-3-quality.

1. What is already built

  e2e/matrix.sh ─┬─ slot 0 ─ workspace clone ─ e2e/run.sh ─ "E2E iPhone 17"          ┐
                 ├─ slot 1 ─ workspace clone ─ e2e/run.sh ─ "E2E iPhone 17e"         │ at once
                 ├─ slot 2 ─ workspace clone ─ e2e/run.sh ─ "E2E iPhone 18 Pro Max"  │
                 └─ slot 3 ─ workspace clone ─ e2e/run.sh ─ "E2E iPad mini"          ┘
                                    │
       each run.sh:  check session ─► patrol test (CHORKI_E2E=true) ─► capture
                                    │        ├─ PlayerProbe    player state over time
                                    │        ├─ BugRecorder    findings + repro steps
                                    │        └─ ErrorWidget    grey/red boxes
                                    │
        e2e/tools/analyze_run.dart ◄┘   one report, bugs grouped across devices

Piece	Where	State
Runner, one device	`e2e/run.sh`	built
Runner, parallel matrix	`e2e/matrix.sh`	built
Report generator	`e2e/tools/analyze_run.dart`	built
Session capture	`e2e/scripts/session_from_token.sh`, `capture_session.sh`, `check_session.sh`	built
Patrol injection wrapper	`e2e/scripts/with_patrol.sh`	built
Test helpers	`patrol_test/support/`	built
`launch` suite	`patrol_test/app_launch_test.dart`	built
`smoke` suite	`patrol_test/smoke/playback_smoke_test.dart`	built
`lifecycle`, `network`, `visual`, `monkey` suites	—	this phase
Android in the matrix	—	this phase

e2e/README.md is the operating manual: how to run it, what the exit codes mean, what counts as a bug, how the report groups things. Read it before writing a test. This document does not repeat it.

The helpers you will write tests against

File in `patrol_test/support/`	Gives you
`app_harness.dart`	`chorkiTest(name, ($, bugs) async {...})`, `startApp($)`
`content_navigator.dart`	`waitForHome()`, `openContentPage(fixture)`, `tryOpenContentPage(...)`, `tapPlayOnContentPage()`, `openPlayerDirectly(fixture)`, `waitForPlayerScreen()`, `leaveCurrentScreen()`
`player_probe.dart`	`waitForStatus(...)`, `expectProgressAdvances(...)`, `expectStopsWithin(...)`, `expectNotPlayingFor(...)`, `nativeStatus()`, `sawStatus(...)`, `expectNoPlayerError()`, the timeline
`bug_recorder.dart`	`bugs.step('...')` (becomes a repro step), `bugs.record(...)` (becomes a finding; `fatal: true` fails the test)
`fixtures.dart`	the five content fixtures below

Current fixtures, all real titles on the live API:

Constant	Title	Notes
`freeMovieWithTrailer`	2nd Chorki Carnival	free, has a trailer, 6415 s
`freeShortTitle`	Sagorika	free, 401 s — use where a test needs to reach the end
`premiumMovieWithSubtitles`	The University of Chankharpul	needs the subscription, 1 subtitle
`freeSeriesEpisodeOne`	Star Ship Fusion Chef Ep 01	free episode, season of 8
`premiumEpisodeWithIntro`	Cactus Ep 1	has intro markers at 633–706 s

2. Before you start: three things that will bite

2.1 The session token, and why runs die at exit code 3

e2e/scripts/session_from_token.sh              # → e2e/.secrets/session.json
e2e/scripts/session_from_token.sh --name qa2   # a second account, for the matrix
e2e/scripts/check_session.sh                   # 0 valid · 1 rejected · 2 missing · 3 unreachable

Switching profile anywhere — the phone in your pocket, the website, a tap on a simulator — issues a new token and revokes the old one. So does logging out. When that happens every run stops immediately with exit code 3 and the report opens with “Blocked by account state”. There is no way to recover but a fresh token.

So, in order:

Get a fresh token from the user before a session of E2E work.
Do not use that QA account anywhere else while tests run.
Never write a test that taps a profile, unless the test is about profiles.

Account limits, which cap how wide the matrix can go:

   per account:  2 concurrent streams          7 registered devices
   → max parallel devices = 2 × (number of accounts with a valid session)
   → every E2E simulator burns one of the 7 device slots, per account

2.2 Xcode 27 breaks two-architecture simulator builds

patrol test and flutter build ios --simulator fail inside debug_unpack_ios with “does not contain architectures arm64 x86_64”. flutter run is unaffected, which is why this only shows up here.

Work around it before the first build of a session, and restore ios/Podfile.lock after any Patrol build that failed — a half-finished build leaves it rewritten.

2.3 Patrol is not a committed dependency

Flutter does not strip dev-dependency plugins on iOS, so while patrol sat in pubspec.yaml it shipped patrol.framework inside every Runner.app, including App Store builds. run.sh therefore re-execs under e2e/scripts/with_patrol.sh, which adds patrol, runs, and restores pubspec.yaml / pubspec.lock on every exit path including Ctrl-C.

Driving patrol by hand goes through the same wrapper:

e2e/scripts/with_patrol.sh patrol develop --target patrol_test/app_launch_test.dart

flutter analyze excludes patrol_test/ for the same reason — package:patrol does not resolve on a normal tree, so the analyzer and the Phase 3 contract tests never see these files. Patrol analyses them itself during a run.

3. The work, in the order to do it

Each step is one commit. The suites are folders under patrol_test/, and run.sh --suite <name> maps straight onto patrol_test/<name>/, so a new suite is a new folder — no wiring needed.

Step 1 — new fixtures (`patrol_test/support/fixtures.dart`)

The flows below need titles the current five do not cover. Add them first, because a fixture that has quietly disappeared from the catalogue is reported as fixture_unavailable, not as a player bug, and you want that failure early.

Needed for	What to find
search	a title whose name is unambiguous in search results
watchlist	any free title; must not already be in the QA account’s watchlist
download	a free, short title — the download has to finish inside a test
deep links	one title per live URL shape (`/watch/...`, `/clip/...`, `/metas/...`, a tab)
language	nothing new; the switch is in Preferences

Pull the routes from the live site rather than inventing them, and keep durationSec honest — several probe helpers derive their timeouts from it.

Step 2 — `patrol_test/flows/` (the eight flows)

A new folder, run with e2e/run.sh --suite flows.

patrol_test/flows/
  search_to_play_test.dart
  watchlist_test.dart
  download_offline_test.dart
  deep_link_test.dart
  subscription_test.dart
  language_test.dart
  shorts_test.dart
  offline_guard_test.dart

1. Search → open a result → play. Type into search, wait past the 600 ms debounce, open a result, press Play, assert the player reaches playing and progress advances. This is the flow the widget tests can only half-cover: the real debounce, the real API, the real player.

2. Watchlist add and remove. Add a title from its content page, go to Library → Watchlist, assert it is there, remove it, assert it is gone. Leaves the account as it found it, which matters because the QA account is shared.

3. Download → offline playback. Download a free short title, wait for it to finish, force the app offline, open Downloads, play it. This needs a new hook:

lib/core/e2e/e2e_config.dart gains CHORKI_E2E_OFFLINE, read by Reachability so a test build can be told the network is down without touching the simulator’s network. Follow the shape of the existing isE2E / e2eSessionJson / e2eMediaProxy defines, and make sure the non-E2E path is unchanged.

4. Cold-start deep links. xcrun simctl openurl for each live URL shape, from a cold start, asserting the screen that opens. The unit tests in test/domain/services/deeplink_services_test.dart already pin the routing table; this checks the other half — that iOS hands the URL to the app at all, that the host gate lets it through, and that a cold start does not lose it.

5. The subscription screen lists plans. Open it, assert plans are listed with prices. Buy nothing. The purchase sheet is StoreKit’s, not the app’s, and a sandbox purchase changes the QA account’s entitlement for every other run.

6. Switch language en ↔ bn. Preferences → language, then walk home, a content page and the player, checking flutter_error never appears. Bengali is the locale the widget tests check for overflows; this is the one that checks the app still works in it.

7. Shorts. Swipe through several bytes and assert exactly one player is active. The shorts_player_pool unit tests cover the pool’s bookkeeping; only a device run shows two videos actually playing at once.

8. Offline banner and the offline guard. With CHORKI_E2E_OFFLINE on: the banner appears, a route that needs the network is refused, and everything recovers when it is turned back off.

Rules for all of them — these are in e2e/README.md but they are the ones people get wrong:

Every confirmed bug gets its smallest reproducing test.
bugs.step(...) at every user-visible action; those lines become the repro steps in the report, and a report without them is much harder to act on.
Never switch or lock a profile. It rotates the shared token (§2.1).
Open a page the way the app does. The app opens an episode by its series route; opening it by the episode’s own route makes the app request <episode>/seasons, which the API answers 404.
Never assert on the player in the same frame you acted — use the probe’s waits.

Step 3 — Android in the matrix

The Android side does not exist yet: there is no android/app/src/androidTest/ and no patrol block in the Gradle files.

Add the MainActivityTest instrumentation runner under android/app/src/androidTest/, per the Patrol docs for the pinned patrol_cli version (4.7.0).
Add the patrol Gradle configuration. Keep it inside the same with_patrol.sh toggle as iOS if it changes the shipped APK at all.
Add emulator chorki_phone_api36 to e2e/matrix.sh’s device list and teach it the Android device-naming and port scheme.
Check the report generator: analyze_run.dart reads iOS crash reports (Runner-*.ips) and the dev.khaled.kvideo device log. Android produces tombstones and a different logcat tag, so the crash and native-stall detection need an Android branch.

Worth knowing before you start: the app runs on 32-bit-only Fire Stick hardware in another repo, and Android Studio’s bundled JDK 25 breaks this project’s Gradle — Flutter is pinned to JBR 17 via flutter config --jdk-dir. If Gradle fails with flutter-plugin-loader > 25.0.3, that is what it is.

Step 4 — run the matrix and triage

e2e/run.sh --suite flows                      # one device first
e2e/matrix.sh --suite flows --repeat 3        # then the matrix

Read e2e/reports/<timestamp>-matrix/report.html. For each entry:

“Blocked by account state” → not an app bug; see §2.1.
A new app bug → fix it, add the smallest reproducing test, log it in docs/testing/bug-log.md with the next number. Phase 3 ended at #26.
A flaky finding → tighten the test’s waits before blaming the app.

4. Done when

The eight flows exist under patrol_test/flows/ and pass on one device.
e2e/matrix.sh --suite flows is green across the four iOS devices.
Android is in the matrix and --suite launch passes on it.
CHORKI_E2E_OFFLINE exists, is read by Reachability, and changes nothing in a non-E2E build.
Every bug the run found is fixed, tested and in the bug log.
tool/quality.sh full passes — it runs the whole Phase 0–3 gate and then e2e/run.sh smoke.
e2e/README.md’s roadmap table is updated (phase 1c → built).

5. Where Phases 0–3 left things

Context for anyone picking this up cold.


Tests	2,297, all green
Coverage	logic 67.0% · player 75.4% · UI 62.0% · overall 65.5%
Analyzer	1,643 issues, down from 1,939; ratcheted per rule in `tool/baselines/analyzer.json`
Errors, not warnings	`use_build_context_synchronously`, `avoid_print`, `unused_import`, `unnecessary_import` — all at zero and held there by `analysis_options.yaml`
Goldens	19 images under `test/goldens/goldens/`
Contract tests	layer boundaries, DI graph, Hive schema, API fixtures, l10n, no-print, image error widgets
Gate	`tool/quality.sh` (fast) and `tool/quality.sh full` (adds E2E smoke)
Bugs found and fixed	26, in `docs/testing/bug-log.md`

Two behaviours are pinned but deliberately unchanged, both recorded under “Flagged, not changed” in the bug log — read that section before an E2E run turns one of them up and it looks like a new bug:

ContentHelper.isUserHasPermission refuses every title to a signed-in viewer with no profile loaded, to a profile whose maturityLevel is 0, and to a title the API sent no content_rating for.
Two BlocProviders on the content page are lazy and never read, so the GetDownloadList / CheckContentEvent in their create never fire.

Branches, stacked, none pushed:

main
 └─ test/phase-0-infra
     └─ test/phase-1-logic
         └─ test/phase-2-ui
             └─ test/phase-3-quality
                 └─ test/phase-4-e2e   ← start here

Phase 4 — the E2E suite

Phase 4 — the E2E suite

1. What is already built

The helpers you will write tests against

2. Before you start: three things that will bite

2.1 The session token, and why runs die at exit code 3

2.2 Xcode 27 breaks two-architecture simulator builds

2.3 Patrol is not a committed dependency

3. The work, in the order to do it

Step 1 — new fixtures (patrol_test/support/fixtures.dart)

Step 2 — patrol_test/flows/ (the eight flows)

Step 3 — Android in the matrix

Step 4 — run the matrix and triage

4. Done when

5. Where Phases 0–3 left things

Step 1 — new fixtures (`patrol_test/support/fixtures.dart`)

Step 2 — `patrol_test/flows/` (the eight flows)