A repair cooperative wants to let branch shops reserve spare parts from a central warehouse. A shop requests a transfer, the warehouse reserves stock and dispatches it, and the receiving shop records delivery. Finance needs a read-only view of the movement. Older branch clients are still in service, and nightly jobs update the same inventory. Use two synthetic shops and ten spare parts to rehearse a release that changes the API, database and notification worker together.
Agree on state transitions and inventory invariants
Start with an approved table of actors, operations, preconditions and results. A shop may request parts, but the request does not consume stock. A warehouse worker may approve an open request if enough stock remains. Dispatch moves the reserved quantity into transit, and receipt transfers it to the receiving shop. Define who can cancel at each step. If the cooperative accepts partial deliveries, record whether each partial receipt reduces the in-transit count and when the transfer becomes complete. Resolve that rule with warehouse staff before implementing it. The resulting table is a contract both developers and operational reviewers can use.
Write down the external request and response fields, supported error codes and retry behavior. Use a stable transfer identifier across API responses, events and operator tools. When adding a new state, feed a sample response to the actual old client. It may reject an unfamiliar enum value even if the transport and JSON are valid. If compatibility requires mapping a new state to an older one, document the mapping and give consumers a version or a capability check that lets them adopt the richer state deliberately. Include pagination behavior in the read contract, especially whether an item can move between pages as its state changes.
Turn business invariants into assertions. Available stock never falls below zero. A reserved quantity belongs to a specific transfer. Confirming or retrying a transfer creates only the inventory movements allowed by its current state. In the ten-part fixture, make two shops request seven parts each and confirm concurrently. Capture both responses and verify that only one is successful, that three parts remain available, and that the losing transfer has no reservation movement or notification job. Run the same case with the successful identity reversed. If a test assumes a particular request wins, it may hide a race behind runner order.
Implement one slice and trace it across boundaries
Begin with a read endpoint and confirm that the intended callers can see the correct source, destination and status. Then implement approval with a transaction that changes the transfer state and available inventory together. Add an idempotency key to the client request where the operation can be retried. Define the response when a retry uses the same key and payload, and what happens when the payload changes. If the client times out after the transaction commits, resending the same key should recover the original outcome without allocating more stock. Keep the expected record and corresponding movements in the test fixture so a reviewer can check the outcome independently.
Record a notification event within the transaction and let a worker deliver it after commit. Give each event a durable identifier. Test a scenario in which the transport accepts the message but the worker loses its connection before marking it as delivered. Confirm that the retry uses the same logical notification and examine how the recipient or provider handles a repeated delivery. Where the external system cannot guarantee deduplication, record that limitation and choose a reconciliation procedure with its owner. Verify that the warehouse can still see a confirmed transfer while the notification is pending. Notification status and transfer status should have their own meanings in the operator view.
Use a common correlation identifier to join the request, stock movement and worker receipt in logs. Keep tokens and sensitive payload fields out of that record. Provide the on-call operator with an approved read-only path that can answer whether the inventory was reserved, which event was queued and when it last attempted delivery. Ask an operator who did not implement the feature to use only the runbook and a synthetic transfer identifier to explain an apparent timeout. If the only way to answer is an undocumented database query, make the supported diagnostic path part of the release.
Rehearse database changes while old and new code coexist
Suppose the old transfer stores destination as free text and the new release needs a constrained branch identifier. Start by adding the new nullable field while keeping the old field. Measure the schema change on an isolated copy with representative table size, writes and index distribution. Record how long locks are held, the impact on ordinary writes and whether replication falls behind. Test the old application against the expanded schema and the new application against rows written by the old version. Decide which field each version reads and writes during the overlap and how mismatches are detected.
Backfill in batches from a stable cursor. Update only rows that still need the new value and have an unambiguous mapping. Keep a report of missing or ambiguous destinations for the data owner. Record batch size, progress, rows examined, rows updated, failures and elapsed time. Pause after a few batches while new transfers are still being written, then resume and confirm that the job handles both newly created records and earlier incomplete records. If a branch was renamed or merged, test the historical mapping against the approved business rule before moving those records into the new format.
Compare old and new reads on a sample and count differences across the whole rehearsal data set. An exit code can tell you a job ran to completion. The reconciliation gives you evidence that all expected rows were populated and that open exceptions have an owner. Switch reads after old instances are retired and the remaining difference is understood. Enforcing a new constraint and dropping the old field should happen in a separate release after the compatibility window closes. The Prisma project documents this staged migration pattern. Apply the idea to the database and deployment tools the cooperative actually uses.
Test application rollback independently of a data correction. After the new code has written records, prove that the intended rollback build can still read them and that queued work can complete. If a database change created incompatible values, agree on a reversible correction for the affected rows or keep the compatibility build running while repairing them. Rehearse with a transfer already in transit, since a rollback rehearsal with an empty database cannot show whether real ongoing work survives.
Apply least privilege to people and background jobs
Create a matrix for branch users, warehouse operators, finance readers and worker identities. A branch user can read its own requests. A warehouse operator can approve requests for the warehouse they manage. Finance may read approved reporting fields. Use two branches and run an allowed path as the first branch before trying to read the second branch's transfer with the same account. Repeat on list, export and asynchronous download endpoints. Inspect returned rows and file contents as well as status codes. If approval queues an export, confirm that the identity and scope are checked again when the file is retrieved, according to the agreed policy.
Keep service permissions narrow enough to diagnose mistakes. The notification worker needs access to queued events and delivery status, and it can operate without the ability to modify stock counts. An inventory reconciliation job can use a dedicated reader and report discrepancies to an authorized operator. In rehearsal, verify that each job still works under its intended account and that a denied write leaves inventory and queued events unchanged. Supply sanitized logs and test data to any review assistant. Include the route, expected policy and supporting evidence with each finding so the owner can reproduce it without granting broad access.
Measure performance before tuning
Make the rehearsal workload resemble real operation. Branches list transfers throughout the day, warehouse workers approve them in bursts, and the backfill runs alongside both. Populate a few busy branches with many transfers, include a large transfer with many line items, and test the agreed pagination size. Record the release, traffic mix, dataset shape and measurement window. Capture request rate, latency percentiles, error rate, query counts, lock waits and queue age. For an apparent regression, separate application processing time, database wait and external response time before choosing a change.
If list latency rises with the number of line items, compare queries for a short and a full page and inspect the database plan using the rehearsal dataset. If approval slows during the backfill, examine whether the batch job and approval path contend for the same rows. Try one change at a time, such as reducing transaction duration or changing batch size, and rerun under the same conditions. Recheck permissions, totals and worker behavior after tuning. A faster endpoint that occasionally returns another branch's rows or loses a reservation fails the release criteria.
Choose tools and deliver evidence for a release decision
Use a design assistant to review the state table and identify transaction boundaries. Give an implementation skill the approved contract and a bounded set of files so its patch can be reviewed independently. A database tool can apply the migration in an isolated environment and report locks and batch timings. A test assistant can draft fixtures and checks, provided you can inspect which endpoints and database state it will touch. Assess each on a small example, including its inputs, required permissions and output, before relying on it for broader work. Keep human approval for policy decisions, migration exceptions and release readiness.
Combine unit checks for state transitions, consumer contract checks for old and new clients, and integration tests using the same database engine family and actual migration scripts. Include the background worker, alert rules and retry configuration in the rehearsal. Save an evidence packet with a deployment identifier, fixture seed, request IDs and the status of each check. Distinguish a failed business rule from a case that did not run because a dependency was down. Assign each unresolved item to someone who can produce the missing evidence.
Enable the feature for a limited set of shops first. Establish a baseline for inventory discrepancies, request failures and queue age. Agree on an observation period and stop conditions such as unexplained negative stock, cross-branch exposure or sustained growth in queued work. Practice disabling the new entry point while preserving already accepted transfers. Verify the chosen rollback build can read records written by the new version before switching traffic. Review residual transfers and migration exceptions with the operations owner after the exercise.
Questions from the rehearsal
What if the database migration passed but branch totals disagree?
Compare the source and destination records for a specific transfer and follow its movement ledger. Reconcile old and new identifiers and confirm that any rows created during the backfill were included. Record the first point where totals diverge. If a row has an unresolved branch mapping, route it to the data owner and keep the legacy representation until the correction is accepted.
What if the old client rejects a response after rollout?
Capture the response shape and identify the consumer version. Compare it to the accepted compatibility sample. If the new feature can be turned off independently, restore the supported response path and confirm that in-flight transfers remain visible. Add the exact rejected response to the consumer regression suite.
Who should be able to reproduce the release evidence?
The client owner should be able to replay the contract cases, the warehouse owner should be able to reconcile the ten-part fixture, and operations should be able to replay the failure and rollback exercise. Give each person the build and fixture identifier and access to the relevant sanitized results. Run the handoff once with someone who was not present during implementation.