MimicClose the loop: let your coding agent test what it builds
A coding agent can read the code, write a change, and get it to build. It cannot see how the app handles errors when the only backend it can reach answers 200 every time.
A declined card, a payments service that is busy twice before it recovers, a session that expires between the cart and the receipt: these are in the API spec, and nothing the agent can run will produce them. So the agent reports the feature as done without having run a 503.
Mimic can produce them. It sits between the app and its backends, records what the real servers send, and plays back any sequence of answers you script, from a command line an agent can drive. With it, the agent runs the app against each case it found in the spec, reads what happened in the request log, fixes what broke, and checks again. This post walks through that loop on a small app.
The example: Acme Store
Acme Store is a small Mac shop app written for this post. It signs in against an Auth service, loads products from a Catalog service and pays through a Payments service. That is three backends, each on its own host in production. The app, a stand-in for the three services, and the OpenAPI spec are in the Mimic repository, under docs/examples/acme-store.
The first version of its checkout has a bug, and we left it in so the loop has something to catch. The spec says that when Payments answers 503, the app should try three times in total. The app tries twice.
Step 2 Record one real journey
One Mimic project stands in for all three services, one local port each. Each port forwards to its real service and saves the first complete text reply for each route as a mock. The app only changes which host it calls.
# one Mimic project, three local servers, each in front of a real service mimic server configure --name Auth --upstream http://127.0.0.1:9080 --pass-through true --capture-responses true mimic server backend add --name Catalog --port 8081 --upstream http://127.0.0.1:9081 --pass-through true --capture-responses true mimic server backend add --name Payments --port 8082 --upstream http://127.0.0.1:9082 --pass-through true --capture-responses true mimic server start # one real checkout, through all three mimic/run.sh --none
After one checkout, Mimic holds a recorded response for every call on the happy path, filed under the backend that sent it. The real services can go away now.
Step 3 Read three sources, write the test cases
The agent gets three inputs, and each answers a different question.
- The request log, from
mimic log list, says what the server actually sent. openapi.yamlsays what the server is allowed to send: every documented error, and the rules a client must follow, such as three attempts on a 503.- The app's own networking code says what the app does with each answer.
From those, it writes a table of checkout cases.
| # | Case | What Mimic answers | The app should show |
|---|---|---|---|
| 1 | Happy path | recorded responses only | Receipt |
| 2 | Card declined, then another card | POST /orders 402, then 201 | Declined message, card switched, receipt |
| 3 | Payments busy, then paid | POST /orders 503 ×2 (Retry-After: 1), then 201 | Two retries, receipt |
| 4 | Session expires at payment | POST /orders 401, POST /auth/refresh 200, POST /orders 201 | Silent refresh, receipt |
| 5 | Item sells out in the cart | POST /orders 409 out_of_stock, then 201 | Item removed with a message, rest paid |
| 6 | Slow catalog | GET /products after 2.5 s | Loading grid, then receipt |
| 7 | Catalog offline, then back | GET /products times out, then 200 | Offline view, Retry, receipt |
Step 4 Run each case
Each row becomes a journey: a short script of answers that Mimic serves in order. Calls a journey does not mention fall through to the recording, so a case only spells out what is different. Case 3 sends the same request three times and gets three answers.
# $PAYMENTS is the Payments server's ID, from mimic server status J="Checkout: payments busy, then paid" mimic journey create "$J" --group "Checkout QA" mimic journey step add "$J" POST /orders --backend $PAYMENTS \ --status 503 -H 'Retry-After: 1' --repeat-count 2 mimic journey step add "$J" POST /orders --backend $PAYMENTS \ --status 201 --body-file paid.json mimic/run.sh "$J"
run.sh rewinds Mimic, plays the app through a checkout, and prints three things: what the app showed, which journey steps were served, and Mimic's request log for the run.
Step 5 Watch it fail
Six cases pass. “Payments busy” does not. The app shows “Payments is unavailable”. The journey's last step, the 201, is never served, and the request log has two POST /orders where the spec requires three. The agent does not have to guess from a screenshot. The evidence is text it can read.
Step 6 Fix, then run everything again
The agent traces the retry count to one condition in the checkout code.
// Store.swift, checkout() case 503: - if attempt < maxPaymentAttempts - 1 { + if attempt < maxPaymentAttempts { attempt += 1
Then it runs all seven cases again, not only the one that failed, because a change to retry logic can break the declined-card case just as easily. All seven pass.
Why this suits an agent
- Every operation is a CLI command with JSON output.
- A journey makes a sequence repeatable: 503, 503, 201 arrives in the same order on every run, so the agent can check a fix by running it again.
- The request log is evidence per request: which backend, which status, and which journey step answered it.
- The app under test is unchanged. It calls local ports instead of production hosts, and there are no test doubles in its code.
Try it
You need Mimic, Xcode or the Swift toolchain, and an agent that can run shell commands. The prompt the agent gets is in qa/QA_PROMPT.md.
git clone https://github.com/srpadrono/mimic cd mimic && git checkout 772386583828 # the example's commit cd docs/examples/acme-store python3 backend/acme_backend.py & # the real API mimic/setup.sh --fresh # three servers, recording mimic/run.sh --none # one real checkout claude "$(cat qa/QA_PROMPT.md)" # hand checkout to the agent
The same loop works on your own app: point it at Mimic, record one real run, and give your agent the spec.