Run Tests via Copilot Studio Kit

Run a batch test suite against a published Copilot Studio agent using the Power CAT Copilot Studio Kit.

Prerequisites

The user must have:

1. The Copilot Studio Kit installed in their Power Platform environment
2. Published their agent in the Copilot Studio UI
3. Created a test set in the Copilot Studio Kit
4. An Azure App Registration with Dataverse permissions

Phase 1: Configure Settings

1. Read tests/settings.json (relative to the user’s project CWD) and check for missing or placeholder values (containing YOUR_).

2. If the file doesn’t exist, create it from the template:

   cp ${CLAUDE_SKILL_DIR}/../../tests/settings-example.json ./tests/settings.json
   

3. If values are missing, ask the user for each missing value. Explain where to find each one:

   • Environment URL (dataverse.environmentUrl): “What is your Dataverse environment URL? Find it in Power Platform admin center or Copilot Studio > Settings > Session Details. It looks like https://orgXXXXXX.crm.dynamics.com”
   • Tenant ID (dataverse.tenantId): “What is your Azure tenant ID? Find it in Azure Portal > Microsoft Entra ID > Overview. It’s a GUID like c87f36f7-fc65-453c-9019-0d724f21bc42”
   • Client ID (dataverse.clientId): “What is your App Registration client ID? Find it in Azure Portal > App Registrations > your app > Application (client) ID. It’s a GUID.”
   • Agent Configuration ID (testRun.agentConfigurationId): “What is your agent configuration ID? In Copilot Studio, go to your agent > Tests tab. The ID is a GUID found in the URL or test configuration.”
   • Test Set ID (testRun.agentTestSetId): “What is your test set ID? In Copilot Studio, go to your agent > Tests tab > select your test set. The ID is a GUID found in the URL.”

   Ask for ALL missing values at once (don’t ask one at a time).

4. Write tests/settings.json with the collected values:

   {
     "dataverse": {
       "environmentUrl": "<value>",
       "tenantId": "<value>",
       "clientId": "<value>"
     },
     "testRun": {
       "agentConfigurationId": "<value>",
       "agentTestSetId": "<value>"
     }
   }
   

5. If all values are already configured and valid, proceed to Phase 2.

Phase 2: Run Tests

1. Ensure tests/package.json exists in the user’s project. If not, copy it:

   cp ${CLAUDE_SKILL_DIR}/../../tests/package.json ./tests/package.json
   

2. Install dependencies if tests/node_modules/ doesn’t exist:

   npm install --prefix tests
   

3. Run the test script in the background with a 100-minute timeout (6000000ms):

   node ${CLAUDE_SKILL_DIR}/../../tests/run-tests.js --config-dir ./tests
   

   Use run_in_background: true for this command. Save the returned task ID.

4. Wait 10 seconds, then check the background task output (non-blocking check).

5. Detect the authentication state from the output:

   • If the output contains “Using cached token”: Authentication succeeded automatically. Tell the user: “Authentication successful (cached credentials). Tests are running, this may take several minutes…”

   • If the output contains “use a web browser to open the page”: Extract the URL and device code from the message. Present this prominently to the user:

     Authentication Required

     Open your browser to: https://microsoft.com/devicelogin Enter the code: XXXXXXXXX (extract the actual code from the output)

     After signing in, the tests will continue automatically.

   • If the output contains an error: Report the error to the user and stop.

   • If the output is empty or incomplete: Wait another 10 seconds and check again (retry up to 3 times).

6. Wait for the background task to complete (blocking). The script polls every 20 seconds until all tests finish and downloads results as a CSV.

7. Read the final output to get the success rate and CSV filename.

8. Proceed to Phase 3.

Phase 3: Analyze Results

1. Get the results: Glob: tests/test-results-*.csv — read the most recent CSV file (newest by modification time).

2. Parse the CSV columns: | Column | Meaning | |——–|———| | Test Utterance | The user message that was tested | | Expected Response | What the test expected | | Response | What the agent actually responded | | Latency (ms) | Response time | | Result | Success, Failed, Unknown, Error, or Pending | | Test Type | Response Match, Topic Match, Generative Answers, Multi-turn, Plan Validation, or Attachments | | Result Reason | Why the test passed or failed |

3. Focus on failed tests (Result = Failed or Error). For each failure, analyze:
   • Test Type = Topic Match: The wrong topic was triggered, or no topic matched. Check trigger phrases and model descriptions.
   • Test Type = Response Match: The response didn’t match expected. Check SendActivity messages, instructions, or generative answer config.
   • Test Type = Generative Answers: The generative answer was incorrect or missing. Check knowledge sources, SearchAndSummarizeContent, and agent instructions.
   • Test Type = Plan Validation: The orchestrator’s plan was wrong. Check topic descriptions and agent-level instructions.
   • Test Type = Multi-turn: A multi-turn conversation failed. Check topic flow, variable handling, and conditions.
4. Proceed to Phase 4 (Propose Fixes).

Phase 4: Propose Fixes

1. For each failure, identify the relevant YAML file(s):
   • Auto-discover the agent: Glob: **/agent.mcs.yml
   • Find the relevant topic by matching the test utterance against trigger phrases and model descriptions
   • Read the topic file to understand the current flow
2. Propose specific YAML changes to fix each failure. Present them to the user as a summary:
   • Which test(s) failed and why
   • Which file(s) need changes
   • What the proposed change is (show the diff)
3. Wait for user decision. The user can:
   • Accept all — apply all proposed changes
   • Accept partially — apply only some changes (ask which ones)
   • Reject — discard proposed changes and discuss alternative approaches
4. Apply accepted changes using the Edit tool. After applying, remind the user to push and publish again before re-running tests.

Test Result Codes Reference

Result: 1=Success, 2=Failed, 3=Unknown, 4=Error, 5=Pending
Test Type: 1=Response Match, 2=Topic Match, 3=Attachments, 4=Generative Answers, 5=Multi-turn, 6=Plan Validation
Run Status: 1=Not Run, 2=Running, 3=Complete, 4=Not Available, 5=Pending, 6=Error