One test setup took me five minutes by hand. My performance test needed 100,000 of them. The skill I built to get out of that became my biggest AI win in QA, and I didn’t see it coming. Here’s why it worked, what went wrong, and a prompt to build your own.
It started with a performance test
Creating test data for a normal test is already a pain, especially when it has to fit a specific scenario. At Choco, restaurants order from their suppliers through a chat in our app. So one usable scenario means a supplier with its catalog, a restaurant as the buyer, a few users on each side, and the chat that connects them. By hand, one set took me at least 5 minutes, and that was on a good day.
For a performance test with 100,000 users, all set up correctly, that isn’t slow. It simply doesn’t happen. So I started writing scripts, and not long after, I wrapped them in a skill. Today a complete set takes seconds instead of minutes.
If you haven’t used one: a skill is a folder of instructions and scripts for an AI agent like Claude Code or Cursor. The agent only reads a short description of each skill up front, and loads the rest when your request matches it. So you just ask for test data, and the skill kicks in.
Scripts were never the hard part
What surprised me was that scale turned out to be the easy bit. A loop solves it. The hard part was knowing which calls to make, in which order, and how to connect the results. You either had to know a dozen commands by heart, or write one huge script that does everything.
And nobody knew all of it. One team had scripts to set up specific product catalogs, another for particular buyer or chat settings. The more I looked, the more of these scripts I found in different corners of our codebase. Each one solved its own piece of the puzzle. Meanwhile the same question kept coming up in our team channels: how do I create data for this?
Everyone asks for something different
The skill became the place where all those pieces landed. You ask for what you need (“a buyer with two users, connected to an existing supplier”, “20 accounts for a bug bash”) and the AI picks the right commands. It works the other way round too. “Which supplier is this buyer connected to?” turned out to be almost as useful as creating things.
The requests are more varied than I expected. A developer wants a quick fixture, QA a dataset, automation a set of users, and a performance test thousands of them. For a bug bash or acceptance testing, everyone in the room needs a working account. Each of those needs different settings, so the skill first asks what the data is for.
At first, though, hardly anyone used it. Every time I saw someone ask how to create test data, I pointed them to the skill. I think it usually clicked when someone needed a lot of test data at once, and doing it by hand was taking too long. Moving it to our central repo for agents and skills helped too. And once people tried it, developers started adding their own scripts and test data setups to it, without me asking.
The connections are the data
In August I spent a day in a virgin forest in the Făgăraș mountains in Romania. It’s a forest that has never been touched by humans, which is rare in Europe. The ground was full of dead trunks, home to insects and fungi. Our guide explained that underneath us ran a network of fungal threads, connecting the roots of trees standing metres apart. From above, you only see separate trees.
Our test data works the same way. A supplier, a buyer, users and a chat are useless until something connects them. So was the knowledge of how to create them, spread over teams. The skill didn’t replace what each team knew. It became the network underneath.
Let the AI choose and the scripts do the work
Looking back, I think it paid off because creating test data is a deterministic process. The same input should give the same kind of setup every time. So the AI only does the fuzzy part: turning a request into the right combination of commands. The scripts do the exact part. The AI never has to understand our data model, because the scripts already encode it.
That’s also why I built it on plain scripts instead of an MCP server, a plug-in that hands the agent a set of tools. Every MCP tool takes up space in the agent’s working memory all the time, while a script costs nothing until it runs. The scripts also work without AI. They live in a versioned repo together with the skill, reviewed like any other code, and work for Claude Code and Cursor users alike.
One option per entity or connection (like --buyer-id, --role or --phone) gives the AI fewer ways to be creative with your database. And make the scripts reject options they don’t know. A typo that gets silently ignored doesn’t fail. It quietly does the wrong thing.
Test data at scale is its own performance test
Even with all of this in place, the 100,000 users took several days. The scripts weren’t slow. I kept tuning batches and parallel runs so the test environment, with its limited resources, could keep up.
The ugliest part came afterwards. I had created thousands of entities, and only then found out the performance test needed a specific ID I hadn’t saved. I had to write lookup scripts to go back and find them. So learn from my mistake, and decide what output you need before you create anything. Prove the whole flow on a small batch first. Then run the big job in a cloud agent (an AI session running on a remote server), instead of babysitting your laptop for days.
Safe by default
The skill writes to shared test environments, so the safe option is always the default. Everything it creates is marked as test data with a (TEST) suffix, and edits are a dry run until you pass --apply. For access, I kept three rules:
- Borrow the human’s session. The agent uses a short-lived login someone opened, never stored keys.
- Block production in the script, not the prompt. A rule in a prompt is a request. A check in code is a wall.
- Log in again every session and say which environment you’re in. A stale credentials file is how a run lands somewhere nobody meant.
Build your own
My first version was full of 🚨 NEVER rules in capitals, written to keep Claude in line. Today the skill mostly explains why things break in our domain. Some of those early rules may not even be needed with newer models, so revisit your skill when you upgrade. Making the AI behave turned out to be the easy part. The real work was gathering the rules of our data model, which lived in people’s heads and in scripts all over the codebase.
You don’t need my scripts to try this. Give your agent something like this:
Help me build a test data skill for my project. The goal: anyone on my
team can ask for test data in plain language and get a correctly
connected setup, built on scripts that also work without AI.
1. Interview me first: which entities do we have, how do they connect,
and which use cases do I need (manual QA, automation, performance,
acceptance testing, bug bashes)?
2. Find the scripts, API calls and fixtures that already exist in this
repo before writing anything new.
3. Build one command with a specific option per entity or connection.
Reject unknown options.
4. Mark everything it creates as test data. Make edits a dry run unless
I pass --apply. Refuse to run against production, in the code itself.
5. Add read commands, so I can look up how existing data is connected.
6. Write the skill file so it explains why each rule exists, and what a
generated setup does NOT give you.
7. When it's done, review the skill with a skill reviewer if your agent
has one, and test it on three real requests from my team.Start with one scenario you build by hand every week. Then let the people who keep asking for test data help you grow it.
In the end, the time was never going into writing scripts. It went into finding pieces scattered across teams and repos that nobody had connected. The skill connects them, the same way those threads connect trees under the forest floor.