Add hidden App Doctor commands

[jplhomer]

Problem

Shopify app developers need a Shopify-specific security review workflow that combines deterministic checks with structured prompts for deeper review by their coding agent. The coding-agent procedure should come from the same Shopify CLI build that owns the review artifacts rather than requiring developers to install AI Toolkit or copy a separately versioned skill.

Unsupported or incomplete analysis must never be presented as a clean security grade. Submitted agent findings are validated before they can affect the compiled trace.

Solution

Add hidden shopify app doctor commands directly to Shopify CLI.

shopify app doctor

• discovers the app root from --path or a nested working directory;
• runs the current 14-check deterministic catalog across supported Shopify React Router apps, theme app extensions, and structured Shopify configuration;
• uses structured TOML parsing, narrow JavaScript/TypeScript analysis, Theme Check Liquid AST analysis, and isolated package-manager audits;
• writes app-doctor-review.json with versioned semantic checks for coding-agent follow-up;
• writes an unsigned App Doctor trace using schema version 2;
• accepts --findings <file> to validate agent results and compile them into the trace;
• supports human and JSON output plus high | medium | low | none CI blocking;
• records per-check status, implementation provenance, inspected files, skipped inputs, and explicit coverage gaps;
• withholds the grade when required coverage is unsupported, unresolved, unreadable, or rejected;
• redacts matched secrets, including complete and malformed private-key material, from findings, evidence, output, and traces.

Security boundaries

• nested apps, dependency trees, generated output, tests, and fixtures are excluded recursively;
• generated scan artifacts are excluded from subsequent scan inputs;
• dependency audits run in a private temporary sandbox containing only sanitized manifest data and the selected lockfile, with scripts/plugins/local dependencies removed, a trusted registry, a sanitized environment, and an explicitly selected Yarn Berry version;
• submitted agent and external findings receive strict runtime shape, provenance, path, and scanned-input validation;
• standalone instructions never infer completion from a persisted review pack, and agent fallback never instructs the agent to execute repository-controlled package-manager code.

shopify app doctor instructions

• prints instructions to stdout by default;
• supports --path, --copy, and --write <path>;
• always starts from a fresh shopify app doctor invocation;
• embeds adjacent Markdown as the source of truth, so the published CLI does not depend on runtime instruction assets.

Both commands remain callable but hidden from normal command listings while the workflow is in development. The engine lives inside @shopify/app; shopify app doctor submit remains reserved for future authenticated upload.

Risk

This adds a substantial local scanner and a new trace schema. Static analysis cannot prove an app secure, so the implementation explicitly records incomplete coverage and withholds a grade rather than reporting a false clean result. The command does not reuse the unrelated DoctorSuite release-test harness.

Top-hatting

pnpm install --frozen-lockfile
pnpm --filter @shopify/app type-check
pnpm --filter @shopify/app lint
pnpm --filter @shopify/app build
pnpm --filter @shopify/cli bundle

Run a scan:

node packages/cli/bin/run.js app doctor --path /path/to/app --skip-instructions
CI=1 node packages/cli/bin/run.js app doctor --path /path/to/app --json --skip-instructions

Print, copy, or write coding-agent instructions:

node packages/cli/bin/run.js app doctor instructions --path /path/to/app
node packages/cli/bin/run.js app doctor instructions --path /path/to/app --copy
node packages/cli/bin/run.js app doctor instructions --path /path/to/app --write /tmp/app-doctor.md

After completing the generated review pack:

node packages/cli/bin/run.js app doctor --path /path/to/app   --findings /path/to/app/findings.json   --skip-instructions

Verification

• Focused App Doctor engine, command, and service suite: 15 files, 161 tests passing.
• @shopify/app command registration test passing.
• App type-check and lint pass.
• Knip passes.
• App build and CLI bundle pass.
• Prompt embedding and OCLIF manifest regeneration are clean.
• Bundled help confirms shopify app doctor and shopify app doctor instructions.
• git diff --check passes.
• All actionable findings from iterative Binks reviews were addressed.

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260828224943

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260831163348

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260831165144

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[nickwesselman]

Can we make the scan command just shopify app doctor for simplicity? Commands can have subcommands so this doesn't preclude shopify app doctor submit

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260901125419

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260901133028

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260901155310

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jplhomer]

Architecture split: Resolved by e5ef447

[jplhomer]

/snapit

[github-actions]

🫰✨ Thanks @jplhomer! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260902180008

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[jek]

I focused on frictions for doctor as a CI and adhoc tool. No blocking concerns for this iteration, but we could do a couple passes as separate PRs because this one is pretty large:

• a correctness pass specifically for Windows & Windows CI execution (I tagged the subprocess spawn that popped out at me most)
• a general optimization pass on the overall scanning & metadata collection with an eye to keeping memory usage constant & generally being resource friendly on larger code bases.

[jek]

Immediately cleaning the sandbox may be flaky here if the spawned command or its unreaped subprocesses still have files open on Windows.

[jek]

This looks like it can exit success when a run failed with incomplete coverage.

[dmerand]

Thanks for all of the changes! This is close to good enough for hidden/early-access. I have a few more notes, which I've made explicitly verbose. I'll provide some other notes for follow-up.

[dmerand]

This command scans the app again. The findings document does not identify the scan that created its review pack. If a file path still exists, the command can accept a finding for old file contents. The final trace then records the new input hash for that old finding.

Please add a schema version to the findings document. Add the scan input hash or a generated review ID to review.json and findings.json. Before validation or merge, reject findings when this value does not match. Add a test that changes a scanned file after review generation and before compilation.

[dmerand]

The current checks examine only the final artifact file. mkdir() can follow a repository-controlled .shopify or .shopify/app-doctor symbolic link or Windows junction. The command can then write outside the app root.

Please reject symbolic-link or junction ancestors in the artifact directory path. Make sure that the final directory stays in the app root. Add an external-target test for a symbolic link. Add a Windows junction test.

[dmerand]

App Doctor constructs this command more than once. Presentation selects shell syntax from process.platform. The terminal report omits the resolved --path. The agent instructions include it.

The CLI already uses a shell-free pattern. packageManagerBinaryCommand() returns separate command and args values in packages/cli-kit/src/public/node/node-package-manager.ts. runGraphqlCodegen() passes these values to exec() in packages/app/src/cli/services/function/build.ts.

Please create one resolved action with separate command and args values. Use this action for terminal output and agent instructions. Render shell-specific text only at the final presentation boundary. Add a test in which the current directory differs from --path. Use a Windows path that contains spaces and %.