Choose one independently verifiable change, such as a status filter on a project list. Pin the commit, fixture, and acceptance criteria. Start each workflow from the same baseline and record time spent briefing, revising, and reviewing.
Give each workflow the same brief
With inline completion, locate the component and tests yourself, then use suggestions as you type. In chat, ask for an explanation of the data flow before requesting small edits. For agent execution, give the allowed directories, existing test commands, and stopping conditions. Ask for the diff and observed results at handoff.
Run the same regression checks
Prepare a fixture with open and closed items. Use Webapp Testing to check the initial list, filtered results, clearing the filter, and the empty state. Check that pagination still works. Record time to the first passing run, human corrections, files changed, and maintenance work introduced.
Choose based on observed cost
For a small function, see whether inline completion is enough. For a change spanning multiple files, compare the review cost of chat and agent execution. Preserve the brief and results so the team can repeat the comparison on a different task size.
Start from isolated branches
Open the baseline project list and record the default ordering and pagination behavior. Prepare three branches from the same revision, install the locked dependencies, and seed the same anonymous fixtures. Keep viewport, browser, and account permissions consistent. If the tools use different models or execution permissions, record that difference alongside the outcome.
A brief for this example might read as follows. Add a status filter to the project list. The initial view shows all projects. Closed shows only closed projects. A filter change returns to the first page, and clearing the filter restores all items. Use the existing list endpoint and preserve the existing permission checks. Find the current list and pagination tests before choosing where to extend coverage. Decide whether a reload should preserve the filter before treating it as a test expectation.
Run three versions of the same task
Inline completion leaves navigation with the developer
Read the list component and data loading function first. Decide where filter state belongs and write a small change. Review each suggested line against the existing query contract. Check whether the suggestion resets pagination and sends the filter to the right layer. If the endpoint returns only one page at a time, filtering that page locally can miss matching records on later pages. Save the final diff and record how much correction each accepted suggestion needed.
Chat collaboration adds one small change at a time
Give the assistant the entry files and existing pagination tests. Ask it to explain how the component, request parameters, and response fit together. Verify that explanation against the repository. Then ask for the selector, the request and pagination change, and the empty state in separate steps. Read the diff and run the relevant check after each one. If a proposal adds a component library, ask for an implementation using the controls the app already ships. Record the back-and-forth needed to convey project conventions and the cost of maintaining either approach.
Agent execution needs an explicit boundary and handoff
List editable directories, permitted commands, and available test resources. Ask the agent to inspect repository instructions and identify entry points before editing. Have it stop on missing authorization, a connection to a non-test environment, or a new service dependency. At handoff, require changed files, commands run, failing output, and cases it could not check. Review whether the diff introduced a new shared state mechanism, changed another list, or rewrote unrelated configuration. Match any claims of passing tests against the actual run output.
Compare outcomes using a fixed results sheet
Seed at least two pages with a closed item beyond the first page. Check the default list, the closed filter, an empty result, clearing the filter, and switching conditions from page two. Webapp Testing helps structure browser actions and visible assertions once the local application is running. For each case, record the input, expected entries, page number, and observed result. For a failure, keep the address and smallest fixture needed to reproduce it. If the endpoint accepts a status parameter, use the existing server-side checks to verify the behavior for an invalid value.
Separate machine time from time a person spends briefing, reviewing, explaining failures, fixing fixtures, and revising the change. Record generation, build, and test duration separately. Ask a second reviewer to check one diff without relying on the tool's summary. Note which outcomes they could establish from the changes and tests. A single comparison measures this fixture and task. Repeat it with a bug fix or a cross-component change to see which parts of the workflow still help.
Questions teams run into
Can the modes be combined?
Yes. A developer can use chat to understand the repository, completion for a local edit, and an agent to prepare repeatable checks. If you want to measure the combination, run it as a fourth case and log what each handoff needed.
Why inspect the fixture after the agent reports success?
A visual success message may only show that the interface updated. Put a matching item on a later page and check that filtering returns it. Verify that changing the filter from page two resets pagination. Attach each result to the underlying test or observation.
How do we pick a team default?
Sample tasks the team actually does, such as form validation, a cross-component interaction, and an endpoint change. Apply the same timing categories and acceptance rules. Keep the failures and follow-up edits. Use the observations to choose a default for similar tasks, then rerun the comparison when tooling or project constraints change.