Android test plan

What to test on a Android release so the baseline still holds: the checks of Google Android 17 COBO Y26M06, by area

Derive from STIG Include CAT III Owner's templates

The release under test Version Hardware

Areas to test Hardware root of trust and secure enclave 6 Platform SSO and identity 5 Authentication 12 MDM enrolment 4 Network 4 Browsers 1 Logging and audit 2 Everything else 20 Untick an area to leave it out of the exports; it stays on the page, marked as not tested this time.

Export the kept checks Report look Untick a check, then Rebuild or export; the form carries the state and nothing is stored. One area is drawn at a time; what you untick in one is kept when you open another, and every export covers the whole plan: the checks kept, what was left out and why, a findings table and a sign-off.

The report looks

Plain: Calibri and grey tables, nothing else: the look before styles.
Formal assessment: A government assessment report in the SP 800-53A manner: serif headings, numbered sections, cover page. Cover page. Numbered sections.
Engineering runbook: For the engineer at the keyboard: monospaced headings, dark command blocks. Numbered sections.
Executive: Navy and teal with a cover page: for the people who sign it off. Cover page.
Compact checklist: Black on white, small type, no cover: prints the most checks per page.
Modern: A clean product look in indigo and slate.
Dark console: Light text on a dark page, terminal colours: for screens, not paper.
Board: Issue-tracker blues and lane greys, as the decisions board draws them.
Audit workpaper: Ledger green and auditor blue, with room for initials and references. Cover page. Numbered sections.
Markdown-native: How the report reads on GitHub: system fonts, ruled headings, grey code.
Word: Office's own blues: the house style of a document written in Word. Cover page. Numbered sections.

The look applies to Word, PDF, LaTeX and the printable checklist; Markdown, CSV and JSON stay plain. Each follows one of the mockups in docs/mockups/exports.

Link to this plan: the release, hardware and areas as a URL to bookmark or send; unticked checks do not travel with it.

Hardware root of trust and secure enclave 6 kept

owner's template Run adb shell getprop ro.boot.verifiedbootstate and adb shell getprop ro.boot.flash.locked.
Android Open Source Project, "Boot flow" (Verified Boot) (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: green and 1: the bootloader is locked and the device booted an unmodified, vendor-signed image.
If it does not: yellow means a user-set root of trust, orange an unlocked bootloader, and a red screen at boot a corrupted partition; any of them is a device outside the managed fleet until it is reflashed and locked.
owner's template On a Samsung Galaxy, check the Knox warranty bit (Download mode shows KNOX WARRANTY VOID, and Knox attestation reports warrantyFuseState).
Samsung Knox, "Knox Attestation v3 tutorial" (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: The fuse is 0 (never tripped); Knox attestation's verdict is Yes with warrantyFuseState 0 and trustBootState 0.
If it does not: A tripped warranty bit is permanent: the device can never run the Knox workspace again and leaves the fleet.
owner's template Run adb shell pm has-feature android.hardware.strongbox_keystore.
Android Developers, "Android Keystore system" (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: true on Pixel and Galaxy devices (a separate secure processor with its own storage and random number generator holds the keys); false on devices with only a TEE-backed keystore, which is the floor Android requires.
If it does not: A device without StrongBox still has a hardware-backed keystore; record it, and keep high-value keys (certificate-based Wi-Fi, derived credentials) on StrongBox devices only.
owner's template Read the Play Integrity verdict the MDM collected for the device (Intune's compliance policy with "Check strong integrity using hardware-backed security features").
Android Developers, "Play Integrity verdicts" (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: MEETS_DEVICE_INTEGRITY and MEETS_STRONG_INTEGRITY: a certified device with a locked bootloader whose OS and vendor partitions have security updates from the last year.
If it does not: Device integrity without strong integrity means the device is genuine but its patches are older than a year; patch it or retire it. Basic integrity only, or none, means an uncertified or modified device: block it.
owner's template Run adb shell getprop ro.build.version.security_patch and compare with the current Android Security Bulletin (and, on a Galaxy, Samsung's monthly SMR).
Android Open Source Project, "Android Security Bulletins" (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: The patch level is the current month's or the previous month's; a Galaxy on the previous Android version with the current patch level passes.
If it does not: A patch level older than two months on a device the vendor still supports is a finding against the fleet's update policy; a device the vendor no longer updates leaves the fleet.
owner's template In the MDM, confirm the device's enrolment mode and the STIG's device-level settings: fully managed or work profile on company-owned, developer options and USB debugging off, USB mass storage off, removable media encrypted or disabled, no unknown sources, and an app allowlist.
DISA, Samsung Android 16 with Knox 3.x STIG (COPE) (Verified boot, locked bootloader, hardware keystore and attestation on Android)
Success looks like: Every setting reads as the STIG requires; a Galaxy also shows Common Criteria mode on and the work-profile audit log collecting.
If it does not: A setting the device reports differently from the policy means the policy did not apply (sync and re-check) or the device is in a mode the policy does not reach (a personal work profile where the STIG expects company-owned).

Any release

Choose a version and hardware above and the plan leaves out what does not apply to them, saying why. Every release and the hardware it runs on is on the Fleet page.

Derived from

STIG: Google Android 17 COBO Y26M06
Checks: 54 tested: 37 from the publishers' procedures, 17 from the owner's templates
By area: Hardware root of trust and secure enclave 6, Platform SSO and identity 5, Authentication 12, MDM enrolment 4, Network 4, Browsers 1, Logging and audit 2, Everything else 20

Templates 4

Conditional Access holds on the new release 3 steps · Authentication · conditional-access.yaml
Whatever the platform, the new OS release must still satisfy the compliance and device-state signals the Conditional Access policies ask for.
Entra ID single sign-on on iPhone, iPad and Android through the Microsoft brokers 5 steps · Platform SSO and identity · entra-sso-mobile.yaml
On Apple devices the Microsoft Enterprise SSO plug-in rides Microsoft Authenticator (extension com.microsoft.azureauthenticator.ssoextension) and is turned on by an Extensible Single Sign-on payload of type Redirect; on Android the broker is Authenticator or Company Portal. Test with a test account under the fleet's Conditional Access policies, on a device enrolled the way the fleet is enrolled.
Verified boot, locked bootloader, hardware keystore and attestation on Android 6 steps · Hardware root of trust and secure enclave · hardware-root-android.yaml
The chain from the bootloader up. Run with USB debugging turned on only for the test and off again after; the STIG forbids it in service. Google's verdicts come through Play Integrity (any device) and Samsung's through Knox attestation (Galaxy only); the hardware you chose rules the Samsung steps in or out.
Automated enrolment and the baseline profiles on a wiped device 3 steps · MDM enrolment · mdm-enrolment.yaml
The first thing to test on a new release is that a wiped device enrols without a hand on it and receives the baseline.

Templates are YAML files under data/test-plans in the repository; the owner extends them by adding a file. Each says which releases and hardware it applies to, what success looks like and what to do when a step fails, and cites the document it follows. They are labelled as the owner's, not the publisher's.

How the areas are chosen

Each check is filed by the words in its title (then its procedure): Secure Enclave, TPM, secure boot and attestation under the hardware root of trust; Platform SSO, Entra, Kerberos and passkeys under SSO; BitLocker, FileVault and LUKS under disk encryption; passwords, smart cards, session lock and sudo under authentication; audit and logs under logging; firewall, SMB, SSH, TLS, Bluetooth and sharing under network; updates and releases under updates; enrolment and management under MDM; browsers by name; meeting rooms under collaboration. The rest is "everything else". The words are the publishers'; only the filing is this product's.

The Android baseline · Fleet · All baselines