AI-Assisted Testing and Production Monitoring

Chapter 1: Build the First Test Set with AI

• Where AI fits in the testing workflow
• How to give AI enough context to produce useful test work
• Moving from feature request to test scenarios
• Normal paths, boundary paths, edge cases, and negative paths
• Expected results and test intent
• Keeping the engineer responsible for final test decisions

Hands-On Exercise 1: Use AI to interpret a support-assistant feature request, generate test scenarios, create normal, boundary, edge-case, and negative tests, add expected results, and organise the tests into a usable test set.

Chapter 2: Generate Test Data and Automated Test Examples

• Generating realistic test data
• Creating boundary and edge-case data
• Creating negative and invalid data
• Avoiding unsafe or unrealistic synthetic data
• Generating API or functional test examples
• Adding assertions
• Making tests readable and maintainable
• Using AI to explain test failures and suggest next steps

Hands-On Exercise 2: Use AI to generate JSON test data, create API or functional test examples, add assertions and expected outcomes, run or inspect the test logic, and improve the test set so it would be useful to a real team.

Chapter 3: Test AI-Enabled Behavior

• Why AI-enabled features need additional evaluation
• Testing prompts and expected behavior
• Testing retrieved context and grounded answers
• Testing tool-call selection and tool-call arguments
• Testing uncertain or unsupported answers
• Checking for hallucination, poor grounding, and unsafe responses
• Testing prompt injection and data exposure risks
• Creating simple evaluation checks for AI behavior

Hands-On Exercise 3: Use AI to create evaluation examples for the support assistant, define expected answer qualities, create checks for grounded responses, create checks for tool-call behavior, and create checks for refusal, escalation, or uncertainty handling.

Chapter 4: Prepare the Release with AI-Assisted Quality Gates

• Why passing tests alone is not enough
• Release confidence as a body of evidence
• Quality gates for traditional application behavior
• Quality gates for AI-enabled behavior
• Regression risk and change risk
• Prompt injection, data leakage, weak tool controls, and unsafe automation behavior
• Canary releases, feature flags, and rollback planning
• Monitoring readiness before release
• AI-assisted release summaries
• Human approval and accountability

Hands-On Exercise 4: Use AI to create a release-readiness checklist, define quality-gate criteria, identify required release evidence, prepare a deployment risk summary, and decide what monitoring must be in place before production release.

Chapter 5: Monitor Production and Investigate an Incident with AI

• Monitoring known failures versus investigating unknown behavior
• Logs, metrics, traces, events, dashboards, and alerts
• SLOs, SLIs, and error budgets
• Monitoring AI-enabled applications
• Latency, cost, token usage, failed responses, grounding, tool calls, and user feedback
• Model drift, data drift, prompt drift, and evaluation drift
• Using AI to summarise telemetry
• Using AI to suggest likely causes
• Validating AI-assisted incident findings
• Turning incidents into future tests and release improvements

Hands-On Exercise 5: Use AI to review simulated production telemetry, summarise symptoms from logs, metrics, and traces, identify likely causes, recommend immediate response actions, create follow-up test cases based on the incident, and improve the release gate or monitoring plan.