Regression automation should explain what a change affects, which checks actually ran, and who owns remaining risk. This guide uses a fictional city photography-exhibition ticket page with date filtering and a new cancellation entry. Planning, locator repair, and reporting create different evidence. No example was executed, and full automation is not the success criterion.
Connect changes to acceptance
Collect the diff, product rules, existing cases, environment, and past defects. Separate filtering, cancellation, shared navigation, and booking APIs. Changed files are not the entire impact surface; shared state, access, and caching can matter. Mark uncertainty rather than using filenames alone to exclude testing.
We searched GitHub for open-source approaches and consulted Playwright's retry, trace, and reporting guidance. Distinguishing first-attempt success, success after retry, and persistent failure helps avoid hiding instability. Check version, license, browser, and dependencies before adoption. Framework capability does not establish that the reader's system passed. No tools were installed or run here.
Scenario one: maintain a risk-based regression pack
Map each rule to evidence of covered, partial, or missing behavior. A test file does not prove coverage. A clickable filter button is only partial evidence when result dates are the actual requirement. Cancellation needs identity, state, and failure-feedback checks, not merely a visible icon.
Use smoke, targeted, and justified broader regression. Smoke checks essential entry points and dependencies; targeted cases cover changed rules and related defects; cross-system or consequential changes justify expansion. Select automation by stable interfaces, frequency, risk, and maintenance. Complex infrequent cases can use human exploration. Common login or payment paths do not justify unrestricted real-account operations.
Each case needs an owner, fixtures, environment, expectation, artifacts, and execution strategy. Use fictional exhibitions, isolated bookings, and notification doubles rather than production identities. Label mock and real test-environment evidence explicitly. Reduced scope needs a rationale and mitigation rather than silent omission.
A bounded request is: “Review date-filter changes against existing assertions and produce a coverage matrix. List smoke, targeted, and exploratory work, explaining exclusions. Use only the specified test environment and fictional data.” Review behavior and risk before deciding which scripts deserve ongoing maintenance.
Scenario two: inspect state before repairing locators
When local success becomes CI failure, compare code, framework, browser, OS, viewport, language, timezone, data, and network. Confirm readiness, identity, destination, and state. A missing button may reflect product behavior, access, missing fixtures, or a brittle selector. Automatically replacing a locator is not necessarily a fix.
Use source to understand semantics, then observe rendered DOM. Prefer stable unique accessible controls or agreed test identifiers. Verify the intended action: cancellation of a booking is different from clearing a filter. Scope duplicate text to a meaningful container rather than using first() to hide ambiguity. Identify dynamic records through stable IDs or explicit conditions instead of position.
Wait for meaningful state such as visible results, confirmed storage, or enabled actions. Fixed sleeps can conceal timing issues. Before increasing a timeout, check that the awaited condition is correct. Preserve screenshots, logs, traces, actions, and network context. Fix product defects in the product and mistaken assumptions in the test without deleting valid assertions for a green result.
Mini Program WXML and browser DOM require different runners, snapshots, and locator APIs. Playwright capability does not automatically cover WXML repair. Do not infer compatibility from an unverified package name. After repair, run focused and related cases, retaining the first failure. Limited retry can collect evidence, but success after retry remains flaky. Quarantine needs an owner, reason, deadline, and alternative critical-path coverage.
A verified Skill for local UI checks
Webapp Testing fits local ticket-page interactions, screenshots, logs, and reproducible findings. Its public SkillPI detail, pinned Anthropic source, and Apache-2.0 license record were checked. Prepare Python, Playwright, suitable browser binaries, startup instructions, a port, and approved fictional fixtures.
Use it for DOM observation and selected journeys, not as a complete strategy, WXML fixer, security scan, or universal browser certificate. Source review and execution are separate evidence levels; no tests ran for the reader. Refresh associations when verified candidates match planning, repair, or reporting responsibilities.
Scenario three: turn artifacts into decision evidence
Before parsing JUnit, Jest, Pytest, Allure, or another format, verify schema/version, run ID, revision, time, shards, and environment. Failure, error, skip, and retry semantics vary. Similar field names do not establish equivalence. Keep parse failures visible. Missing artifacts and empty suites are not zero failures or success.
Separate unique cases from attempts and first-attempt passes from retry passes. State the denominator for pass rates, listing skipped and unexecuted work separately. Code and requirements coverage are different. Merge shards by stable case identity; browsers and parameters may represent distinct configurations, while retries must not inflate case counts.
For each failure, give expectation, observation, category, evidence location, impact, and owner. Compare trends only with compatible scope and environment. A smaller suite or many new easy cases can change pass rates without better quality. HTML and spreadsheets are presentation formats; conclusions need traceable raw results. Report generation does not prove complete parsing.
Artifacts can expose accounts, tokens, page data, or private code. Use approved storage, access, and retention rather than default public sharing. Scheduled regression also needs evidence that the schedule triggered, execution completed, and the report is fresh. Missing or stale results need explicit status, not yesterday's green display.
Make an acceptance decision and maintain it
Report accepted, additional-validation-needed, or blocked status with evidence, scope, gaps, and ownership. A total score cannot override a critical failure, and an unexecuted critical path cannot be called complete regression. Test acceptance is separate from deployment authorization. Inspect uncertain external state before repeating actions.
Maintain the matrix, fixture construction, dependencies, retries, and retirement criteria. Rule changes need updated expectations, not only new snapshots. Framework upgrades require locator and parser checks. Before removing redundancy, confirm no unique risk is lost and keep recovery. Automation proportion follows risk and maintainability, not a universal percentage.
FAQ and sources
Planning or locator repair first? Establish scope and expectation, then investigate the failure; the activities can iterate but cannot replace one another. Can reports be automatic? Yes, from structured evidence with checked totals, semantics, and links. Why local green and CI red? Compare environment and data while preserving first-failure evidence rather than assuming the test is wrong.
Does free software mean no cost? Account for frameworks, models, runners, and maintenance separately from licenses and service fees. Start with one change, matrix, and fixture set before scaling. Deliver the plan, repair evidence, results, report, acceptance decision, and recovery instructions, stating what was actually executed.
This original exhibition example covers the bilingual topics' planning, locator repair, and reporting/acceptance scenarios without adopting testimonials, downloads, or blanket safety claims. References are Playwright's retry guidance and trace documentation.