A property management team builds a repair assistant. In the demo, a resident reports a leaking washing machine and the assistant prepares a ticket. In the pilot, a returning resident gets details from the wrong ticket, an apparently failed request creates two drafts, and the duty officer misses an escalation. Reproduce each failure with a synthetic ticket, then exercise the whole path from the resident's first message to the officer's handover.
Give the conversation and ticket separate identifiers
Write down which fields the resident has confirmed, which ticket the current conversation concerns, and whether each fact came from a user message or a system response. Keep recent dialogue available while the session is active. Store the ticket identifier, current status, and confirmed address in the service that owns the ticket. When a resident returns, fetch only tickets available to the authenticated account. If the request could refer to more than one ticket, show enough information to distinguish them and ask which one they mean.
Keep a field map with the authority for each value, its lifetime, the permission needed to read it, and how to resolve disagreements. For example, the address on the selected ticket is the current address for that job. A change requires confirmation and a draft update. A summary can keep confirmed facts, unanswered questions, and the next action. Compress a long conversation in a test and start a fresh session. Check that the same ticket can be recovered and that unconfirmed details remain unconfirmed. Repeat the query with a different test account to verify that access is checked again.
Turn tool failures into cases the team can replay
Separate reading a ticket, creating a draft, and submitting a request. Give each operation an input schema, a defined response, and expected error categories. Use the same fixture to try success, a missing field, insufficient permission, timeout, and an incompatible response. Capture a request identifier, operation name, elapsed time, status code, and redacted validation result. A duty officer should be able to locate the failed operation by identifier without opening the entire conversation.
When a create call times out, check whether the ticket service received and processed the request. If the service supports idempotency keys, use the same key for retries of the same action. When the service already created the draft, read it back and show its actual state. If the outcome cannot be resolved, put the case in a reconciliation queue for a human. Use bounded retries for reads and define recovery for writes explicitly. The resident should hear that the submission status is being checked while the backend outcome is unknown.
Replay a realistic failure in a preview environment
Prepare two tickets for one test account, a leaking washer and an entry buzzer that does not ring. Create a draft for the washer, then ask about the buzzer. Return to the washer conversation and confirm that the assistant uses its ticket and address. Simulate a successful server write followed by a client timeout, then replay the original request. Check the service for exactly one draft and compare its status with the resident-facing message. Ask a different account for the same draft and verify that the reply contains neither the address nor contact details.
Save the initial fixture, action, expected result, observed result, and evidence for each case. If a valid tool response leads to an incorrect reply, inspect the mapping between internal states and wording. A draft, a submitted ticket, and a dispatched job represent different outcomes. Exercise all three states in regression tests and verify that the assistant never promises a dispatch based on a draft response. Keep the version of the tool contract with the evidence so the same test can be rerun after an interface change.
Set conditions for widening the pilot
Start with a limited set of buildings and an officer who can take over during operating hours. Agree on a measurement window for ticket accuracy, duplicates, escalations that reach an officer, and response time. Define the unit being counted and how a retry contributes. Include the sample size and examples of critical failures. A cross-account read, duplicate submission, or incorrect dispatch warrants a fix and a focused retest before the pilot grows.
Show the officer the ticket ID, confirmed fields, failure category, and decision needed. Test an officer rejecting a draft. The assistant should explain the next step to the resident and leave the draft unsubmitted. Rehearse switching off automatic submission while keeping lookup and manual intake available. Reconcile work already in flight before re-enabling writes. The pilot report should name the owner, the allowed scope, the evidence supporting the decision, and the next review date.
Pick a skill for a specific check
Context review needs a capability that can trace field ownership, access, and session recovery. Provide the field map and two sample conversations and ask for stale-data and conflict cases. Tool debugging needs a capability that can compare redacted requests and responses and document a failure the team can replay. End-to-end acceptance needs a capability that can exercise the preview with test accounts and preserve evidence against named assertions. If a skill only writes a test plan, run that plan separately and record the observations. Check host support, local dependencies, network requirements, and access before installing anything. Keep the old version and its passing fixture when trying an upgrade.
Common questions during rollout
Why test network failures after the demo passes? The demo normally covers a clean run. A timeout after a successful write requires a different recovery path from a timeout before the request reaches the service. Reproduce both and inspect the resulting ticket count.
Does memory require storing every message? Start with the facts that a future session needs and determine where to recover them. A ticket ID and current status can often come from the service under the resident's existing permissions. For any retained summary, specify its purpose, lifetime, and deletion rules, then check that expired facts cannot reappear in a draft.
How can the team diagnose an intermittent failure? Match the assistant trace and the service log by request ID. Preserve the interface version and error category. Replay the same safe fixture, checking parameters and permissions before testing transport and response parsing. After the fix, rerun both the isolated call and the resident-to-officer flow.
What happens if automatic dispatch stops during the pilot? Route incoming cases to the manual queue with their current status and request IDs. Have the officer reconcile existing tickets before creating replacements. Restore automatic dispatch after verifying recovery on the same failure fixture.
Use the linked Webapp Testing skill for the browser portion of acceptance. Point it at the preview and verify the resident's visible ticket state, the permission error for the second account, and the officer's escalation screen. Use the service fixture to inject the timeout and check the stored record count independently.