AI Augmented Testing: Best Practices on How to Control AI in Writing Test Cases

There is a familiar trade-off in QC work: the more thoroughly we want to test, the more test cases we need. But as the number of test cases grows, so does the time required to write and maintain them.

According to the traditional testing process, QC engineers often go through the same series of tasks: reading the spec, breaking down requirements, identifying business rules, defining validations and error cases, thinking through possible scenarios, and converting everything into the team’s test case format. For features with complex workflows or interconnected rules, turning requirements into test cases can take considerable effort.

Although AI helps fasten lots of manual tasks, we must be careful enough to ask how much AI should be involved without sacrificing the quality of testing. 

After some trial and error, it becomes apparent to us that AI cannot replace humans in the entire process. So the goal is not to remove QC engineers from the process, but to design an AI-augmented testing workflow where AI handles repetitive scopes, people guide, review, and approve the work at the points where judgment matters most. Only when we successfully implement that workflow, will AI augmented testing become a useful, reusable know-how of the company. 

Can AI handle the end-to-end testing process by itself?

In theory, processing large amounts of information, structuring the content, and generating artifacts in a predefined format is well suited to AI, because AI can complete these tasks much faster than a person.

However, another problem quickly appeared during our experiments.

If AI misunderstands the requirements from the beginning, speed is no longer an advantage. It only helps us produce mistakes faster.

One misunderstood requirement can be expanded into dozens of incorrect test cases. At that point, the question is no longer: Can AI write good test cases?

It becomes: How can QC engineers verify how AI understands a requirement before that interpretation is multiplied across dozens of outputs?

To address this problem, our team began experimenting with a different approach. Instead of asking one AI agent to handle everything from reading the spec to generating test cases, we divided the process into smaller steps. We also added a checkpoint where a human could review how AI had interpreted the requirements before allowing it to continue.

This article shares what our team learned while building a small pipeline to explore that approach. We are not claiming that this is the optimal solution. It is simply one set of lessons from our implementation process and one small part of the broader picture as AI gradually becomes part of the QC workflow.

Why AI Augmented Testing Needs More Than One Agent

The most natural approach is to combine everything into one prompt: read the spec, understand it, standardize it, identify business rules, brainstorm scenarios, write test cases, and export them as a CSV file.

This approach is not wrong. However, when the output does not meet expectations, it becomes difficult to identify where the problem occurred. Did AI misunderstand the requirement, or was the original spec already unclear? Debugging the entire process can take a significant amount of time.

Our team divided the pipeline into smaller steps, with each step responsible for one task:

The main benefit is not only easier maintenance. Dividing the pipeline creates a clear checkpoint where a person can review how AI understands the spec before that understanding is expanded into dozens of test cases. This checkpoint is what makes the approach AI augmented testing rather than fully automated test generation.

Of course, this pipeline is not a fixed structure. It is still being developed and will continue to change as we gain new insights and encounter new use cases.

A Practical Example: Why the Standardization Step Matters More Than It Seems

Consider a simple Forgot Password feature. The shortened raw spec contains only a few lines:

Users can reset their password using their registered email address. Email is required and must follow a valid format. If the email does not exist, display ‘Account not found.’ For a valid email, the system sends a reset link. The reset link expires after 30 minutes.

If we give this paragraph directly to AI and ask it to “generate test cases” the result may still look reasonable. However, it is difficult to verify which specific rules AI extracted from the spec.

The Reformat step addresses this problem by requiring AI to separate the spec into clear units before writing any test cases:

ID	Object	Type	Condition	Expected Result
RS-01	Email	Required	Email is empty	Reject the request
RS-02	Email	Validation	Invalid email format	Reject the request
RS-03	Account	Error	Email format is valid, but the account does not exist	Display ‘Account not found’
RS-04	Reset Request	Behavior	Email is valid	Send a reset link
RS-05	Reset Link	Business Rule	The link is more than 30 minutes old	The link expires

The Reformat step provides another benefit. It standardizes specs from different projects and source formats into one consistent structure.

Giving AI input in a familiar and predictable format helps it perform more consistently across multiple runs. It also makes the pipeline easier to apply to other projects without redesigning it from the beginning.

At this point, the reviewer does not need to inspect a long list of test cases. They only need to compare these five rows with the original spec and confirm whether they match.

Once approved, the Generate Test Case step can expand each rule into specific scenarios, such as:

• Empty email
• Invalid email format
• Account does not exist
• Reset link is still valid
• Reset link has expired

The same table can also be used to generate a testing checklist. However, the checklist only needs to identify what should be tested. It does not require the same level of detail as a full test case.

Human Review Is the Core of AI Augmented Testing

The most important part of the pipeline is the review checkpoint. There is also one detail that can easily be overlooked: review is not only about checking whether AI produced the correct result.

This is where human effort and experience enter the process, including strengths that AI cannot replace:

• User experience knowledge
• Domain knowledge
• Bugs previously found in production
• Legacy system behavior
• Risks specific to a module
• Special integrations
• Other project-specific context

After the Structured Spec is approved and the test cases or checklist have been generated, one lighter review step remains before test execution.

This quick review is used to identify issues that only become visible after AI expands the approved rules into dozens of cases.

This Structure Makes It Easier to Adjust the Pipeline and Isolate Issues

Dividing the pipeline into phases does more than simplify the review process. It also becomes useful after the Structured Spec has been approved:

• It is easier to identify where an error occurred. Each step is separated, so the team does not need to investigate the entire process from the beginning.
• Repeated errors in the generation step point to a specific problem. This is a sign that the prompt or agent used in that step should be reviewed and improved.
• If the root cause is in the Structured Spec, fix it there. Then generate the output again instead of editing the final test cases.
• Both sides of the workflow can be improved. This includes the quality of the input and output, as well as the agent or prompt used at each step. The team does not have to keep fixing isolated outputs without understanding the cause.

One principle I try to maintain when operating this pipeline is simple: when a test case is wrong, do not edit the test case first. Find out what is wrong in the Structured Spec.

If someone edits the test case file directly, that manual correction will disappear the next time the spec changes and the test cases are generated again.

Issue Type	Where to Fix It	Where Not to Fix It
Test case is missing or contains incorrect logic	Structured Spec, then generate again	Test case file
Structured Spec is missing information	Clarify with the original spec or BA, then generate again	Do not guess
Output file has formatting issues	Fix the generation step, then run it again	Do not edit manually in Excel
Coverage is insufficient	Add the missing information to the spec or test strategy, then generate again	Do not add isolated test cases

Following this principle ensures that every output can be reproduced from the beginning. Returning to the same spec should produce the same result without depending on scattered manual edits. It also keeps the AI augmented testing workflow maintainable as requirements change.

Lessons Learned from Running the Pipeline in Practice

• Divide the spec by feature or module instead of putting everything into one session. If each section is too small, connections between different parts may be lost. If it is too large, AI may overlook information near the end. A practical unit is one that still contains enough context to understand the complete business flow.
• Classify data before sending it to AI. Treat AI the same way you would treat any other external tool. Define which data can be submitted, which data must be anonymized, and which data must never leave an approved environment. AI is not an exception to the existing security policy.
• Convert the spec to Markdown before feeding it into the pipeline when the original input is a PDF, DOCX file, or scanned image. This requires some initial effort, but it significantly reduces extraction errors. It also makes debugging much easier when something unexpected occurs.
• A stronger model can reduce the risk of misunderstanding, especially when the spec contains many cross-references or dependent business rules. However, it cannot replace the review step. A better model may produce a stronger first draft, but a person must still confirm whether it reflects the actual business requirements.

Conclusion

The main lesson I learned from using AI in this part of the QC process was not how much faster AI could write test cases.

The real lesson was that reliable AI augmented testing requires a clear checkpoint where people can verify how AI understands the problem before that interpretation is expanded into multiple outputs.

AI is good at:

• Reading information
• Breaking down requirements
• Standardizing content
• Generating artifacts in a defined format

Human experience is still needed for:

• Test coverage
• Business accuracy
• Risk assessment
• Final approval

AI can identify more possible angles in a short time than one person may be able to consider alone. Many of those suggestions can be reasonable and worth reviewing. We should use this ability to brainstorm areas that people might otherwise overlook.

However, that does not mean AI should make every decision. Regardless of how many ideas AI suggests, people still need to decide which ones to keep and which ones to remove.

AI can help us think more broadly. It should not replace human judgment entirely.

The question is no longer: “Can AI write better test cases?”

The more useful question is: What other pain points in the QC workflow need to change so that teams can make a real breakthrough as AI continues to advance?

FAQs Section