HAQQ Mobile Experiments

The two arms

Both arms start at the same decision: a guided tour finishes or is skipped, and the user is on the Free plan, in an individual workspace, with a loaded store offering, and has not had their turn on this install. Both arms mark the user as having had their turn.

control: nothing is scheduled. No sheet, no keyboard dismissal. Every other paywall (limit hit, settings, chat usage, credit top-ups) is unchanged, so a control user still meets the paywall when they run out.

paywall: after 700 ms the keyboard is dismissed and the paywall sheet opens with trigger: 'onboarding', selecting the plan and billing period in the flag payload, { "plan": "starter" | "pro", "cycle": "monthly" | "yearly" }. A missing or unreadable payload opens on the onboarding default, Starter monthly.

There is no *-control sibling file: both arms are a branch in runOnboardingPaywallDecision in src/lib/onboarding-paywall-decision.ts, called from use-onboarding-paywall.ts.

Assignment and exposure

• experiment.exposed fires once per user, at the decision, in both arms, with experiment: 'onboarding-paywall', variant and $feature/onboarding-paywall. It uses the manual mode of useExperiment: the deciding hook is mounted for every signed-in user, so a render-time exposure would enrol everyone at app start.
• Unbucketed users (flag not loaded, false because it is missing, disabled or not launched, or an undeclared variant such as skipped or PostHog's default test) see no paywall, fire no exposure and store no variant.
• The arm is stored per user on the device (onboarding.paywall_variant.<userId>, write-once) and stamped as experiment_variant on every later chat.prompt_submitted from that user.
• Users are identified long before the decision, so assignment is per person.
• Builds before ENG-4373 read the arms the old way: control there opens the paywall and fires an exposure for control. The flag must only enrol app versions with ENG-4373 (release condition on $app_version, see PostHog configuration), or the control arm mixes in users who saw a paywall.
• Like today's paywall, the decision can follow a Settings tour replay or a returning user's guided tour.

Metrics

Every metric step except the exposure is filtered on platform = mobile. The exposure event is filtered on experiment = onboarding-paywall and $environment = production.

The table is the design from ENG-3808. With the arms reversed, Sandro sets the primary metric before launch: purchases (7d or 14d) fit the "monthly Starter vs no paywall" question better than 24-hour prompts, which become the guardrail. The control baseline for 24-hour prompts was about 24% on 1.0.8, before the profile moved to the second send; expect it to change.

Purchases use Growth's existing action Purchase — any surface (canonical) (credits.topup_payment_success, paywall.purchase_completed, subscription.purchased, subscription.payment_success), filtered on platform = mobile in the metric. Its overlapping events do not double count: a funnel counts users, not events.

Purchase events carry no trigger, so the limit-hit purchase metric is a time-windowed approximation, not attribution.

Breakdowns when reading: $os, country (Lebanon against the rest), persona, and whether the user registered within 24 hours before exposure.

Calling it

Run until 1,000 exposed per arm and at least 3 full weeks. Do not call it early.

Ship rule: to be agreed before launch (Steve and Sandro). The 2026-10-02 rule was written for the old arms (remove the paywall if skipped raised 24-hour prompts) and no longer applies.

Limits worth knowing before reading the result:

• At about 3.5 purchases per 1,000 viewers, each arm sees single-digit purchases at 1,000 exposed. The purchase leg of the ship rule is a directional check, not a significant one.
• 1,000 per arm at a 24% baseline detects roughly a 5-point absolute lift. The 1.0.8 drop was about 19 points, so a large paywall effect will show; a small one may not.

PostHog configuration

1. Feature flag onboarding-paywall: multiple variants, user-level, control 50% / paywall 50%. Rename the old skipped variant to paywall; the app treats skipped as unbucketed. Set the paywall payload, e.g. {"plan":"starter","cycle":"monthly"}. Add a release condition $app_version at or above the first build with ENG-4373, so older builds stay unbucketed. Before launch, release conditions only for test accounts, each with a variant override.
2. Experiment Onboarding: paywall after the tour, linked to the existing flag. Exposure: custom event experiment.exposed, filtered as above. Exclude multivariate users, filter out internal and test users, Bayesian at 95%, Replay Vision off. Metrics as in the table. Save as draft.
3. Verify on a device with each test account: one exposure per arm, control shows nothing, paywall opens on the payload's plan and billing period, later prompts carry experiment_variant; with no override, nothing shows and nothing is exposed.
4. Launch once the build is live in both stores: replace the test release conditions with one set for all users at 100%, launch, tell Steve the date, and tell Growth the flag key (their Experimentation Health dashboard lists only exp_ flags).

Kill switch: set the flag to 100% control, or turn it off. Every user not yet at the decision gets no onboarding paywall, with no release.

The two arms

control — unchanged. ProfileCreate (name, country, phone, role, referral) and, for legal professionals, ProfileProfessional (position, org name, type, size, referral).

stepped — one question per step, ordered so friction rises only as commitment does:

The referral disclosure stays folded into the final step of whichever path the user is on, collapsed, exactly as ENG-2708 shipped it. It does not get its own step: it is optional, most users skip it, and a step most people skip is a step most people can quit on.

Path lengths differ by design — 3 steps for a help seeker, 4 for a professional. Asking role early is what buys that; asking it late would mean showing professionals' fields to everyone.

Because role is step 2, the total is not known while the user is on steps 1–2. The indicator assumes the shorter path (3) and animates a fourth dot in if the user picks legal professional. It moves at most once, and only ever grows — a count that shrank would read as the flow being cut short.

Metrics

Primary — decides the winner

Onboarding completion rate: onboarding.profile_completed ÷ experiment.exposed.

This is the whole point of the change and the tightest loop around it. Everything else is diagnostic or a guardrail.

Diagnostic — tells you why, does not decide

Step funnel: onboarding.step_viewed broken down by step_id. Stepped arm only — the control has no equivalent step boundaries, so this diagnoses where the stepped flow bleeds and can never compare the two arms. Cross-arm comparison happens on profile_completed.

This is the most valuable thing on the dashboard. If stepped loses, this says exactly which screen bled users; if it wins, it says which step to attack next. Break down by step_id, never by index — the seeker path and the professional path have different lengths, so index 3 is contact for one population and work for another, and a funnel built on the index silently averages two different flows together. (Same reasoning as tour_step_viewed.)

Also worth watching per step:

• onboarding.step_back — returning to a previous step signals a step that asked something unclear, or one the user wasn't ready for.
• Validation errors per step — a step with a high error rate is a step with a bad field, not a bad position in the flow.

Guardrails — can veto a win

A completion win that costs any of these is not a win:

Calling it

Ship stepped only if all of:

1. Completion rate is higher, and PostHog reports significance (do not call it early — the number swings hardest in the first days).
2. No guardrail regressed materially.
3. The step funnel shows no single step bleeding badly enough to be worth fixing before rolling out.

If completion is flat but the step funnel is clean and time-to-complete is equal, prefer stepped anyway — it is the better platform for future onboarding changes. Say so explicitly in the decision rather than calling a tie.

Assignment caveat: PostHog initialises regardless of App Tracking Transparency — only identify() is ATT-gated. So a user who denies tracking is still bucketed and still fires exposure, but on an anonymous per-install distinct id rather than their user id. Assignment for them is per install, not per person: reinstalling or using a second device can land them in the other arm. Exposure counts should still track signups closely; a large shortfall means flags are failing to load, not that ATT is filtering people out.

Genuinely unbucketed users — PostHog disabled by env, or flags not yet loaded — render control and fire no exposure, so they land in neither arm.

PostHog configuration

Events this experiment reads

Event names are namespace.event — the adapter joins them, so search PostHog for the joined form.

variant is the authoritative record of what was rendered, not $feature/.... The SDK appends its flag properties after the ones we pass, so it overwrites our explicit $feature/onboarding-stepped-form with whatever the flag says at capture time. Our variant property survives, and it holds the frozen value the user actually saw. If the two ever disagree — possible across an anonymous → identified transition — variant is the truth and PostHog's attribution is not. Break down on it when auditing.

step_viewed fires in the stepped arm only. The step funnel is therefore a diagnostic for that arm, not a cross-arm comparison — the control has no equivalent step boundaries. Cross-arm comparison happens on profile_completed.

Wizard settings

1. Description — name Onboarding: stepped form, flag key onboarding-stepped-form.
2. Variant rollout — rename PostHog's default test to stepped; leave control. User-level participants, 100% rollout, 50/50 split.
3. Analytics —
4. Include people when: custom event experiment.exposed, not the default "Feature flag is called". Ours fires on render, which is the denominator the arms actually share.
5. Filter that event to experiment = onboarding-stepped-form using the filter control beside the event picker. This is required, not optional: the SDK stamps $feature/<flag> onto every event for every loaded flag, so once a second experiment also emits experiment.exposed, its exposures would carry this flag's property too and be counted as exposures here. The experiment property is what scopes the event to one experiment.
6. Multiple variant handling: Exclude multivariate users (the default). A person exposed anonymously and again after identification can land in different arms; someone who saw both tells us nothing, so drop them rather than pick one.
7. Filter out internal and test users: on.
8. Replay Vision: leave off. Session replay is already installed, so replays can be filtered by experiment.exposed → variant by hand. The scanner costs credits and keeps running after the experiment ends, which is an easy thing to forget.

Where the platform filter goes — and where it does not.

Filters belong in the metric definition, never applied ad hoc while reading results: PostHog derives significance from the definition, so a filter added at viewing time changes the chart but not the maths behind it.

Filter every step of every metric on the platform under test:

The experiment touches no web surface, so cross-platform behaviour is identical in both arms and the comparison is unaffected either way — what filtering buys is that each number means what its label says. A dashboard where some steps are scoped and others are not is the kind of thing that gets misread months later, when whoever reads the result was not in this conversation.

The one thing given up: a user who completes onboarding on mobile and sends their first prompt on web reads as un-activated. Rare, and equally rare in both arms.

Filter value — a time-bounded exception. The rule forbids $lib filters, and the mobile client now registers { platform: 'mobile' }. But that property only exists on events from a build carrying it, so until the release that ships this PR reaches users, platform matches nothing and $lib = posthog-react-native is the only filter that works.

Use $lib to configure and verify before launch. Once the release is out, and before reading any result, swap the $lib filters on the shared-map steps — the three marked "Yes" in the table above — to platform = mobile.

experiment.exposed is not part of that swap: it stays on its experiment filter and takes no platform filter at all, as the table says. Delete this paragraph once the swap is done.

Metrics to attach

Attach guardrails as secondary metrics on the experiment rather than building standalone insights: PostHog then does variant attribution off the exposure event itself, instead of relying on you to break every insight down by flag correctly.

Two guardrails named above are deliberately not attached as experiment metrics:

• Time to complete — PostHog experiment metrics have no duration type. Read it as a standalone insight: median time from experiment.exposed to onboarding.profile_completed, broken down by variant.
• Data quality (org-field completeness) — read as a standalone insight too: onboarding.profile_completed where persona = legal_pro, comparing the share with organization_name set, by variant.

Both are eyeball checks before calling the result, not automated vetoes. Anything listed as a veto criterion that has no definition here is not enforceable — say so rather than implying it is being watched.

Diagnostic insight (not an experiment metric)

One funnel, stepped arm only, to find the bleeding step:

onboarding.step_viewed where step_id = name → role → contact → work

Build it as a funnel with one step per step_id value, not a breakdown on step_index.

Build it as two funnels, split by path, or the numbers lie: help seekers never reach work by design, so a single name → role → contact → work funnel scores every one of them as a drop-off at the last step.

• Help seekers: name → role → contact, filtered to persona = legal_seeker
• Professionals: name → role → contact → work, filtered to persona = legal_pro

Alternatively filter the work step to professionals only. Add a second insight trending onboarding.step_back by step_id — a step people reverse out of is a step that asked something unclear.