[bot] Merge master/a0e06bc5 into rel/dev - #1754
Conversation
… failed one _extract_metric_result (and conversation.py's own copy of it) returned on the first create_metric tool call in a turn, regardless of isError. When the agent self-corrects an invalid MAQL by retrying create_metric within the same turn, this captured the failed first attempt's error payload instead of the successful retry's payload: - In conversation.py, turn_outputs then had no metric_id, so any later fixture turn referencing $ref:create_metric.* raised inside _resolve_refs and was silently [SKIP]ped -- never sent to the server. - In metric_skill.py, the run reported maql_correct=False even though the metric was created, and metric_id_to_delete stayed unset, so the metric the successful retry created leaked into the shared workspace. Fix, consolidated into one implementation that conversation.py now imports instead of keeping its own copy: - Skip a create_metric result when the decoded payload is not a dict, is an error, or is empty -- callers unconditionally call .get() on it. - Prefer the most recent successful call in the turn. - conversation._check_output_present's metric branch now requires an actually-extracted result, not just any create_metric call having been made -- a turn where every attempt failed no longer reports success while the real failure surfaces as the next turn being skipped. - metric_skill._execute_single_metric_run now tracks every metric id any create_metric call in the run produced (via the existing _extract_created_metric_ids) and deletes all of them, instead of only the one derived from the (possibly failed) primary candidate. evaluators/metric_skill.py's _find_create_metric has the same first-match pattern but is only reachable from gooddata-eval's standalone CLI, not from the ai-agent-tests-staging.yml pipeline -- left as-is. Verified locally against gdc-nas: ran the full agent_metric_skill and agent_conversations suites (18 fixtures) in parallel the way daily CI does; the workspace's metric list matched its pre-run baseline afterward. JIRA: QA-29053 risk: low
…ction fix(gooddata-eval): keep retrying create_metric result, not the first failed one
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1754 +/- ##
===========================================
+ Coverage 80.16% 80.20% +0.03%
===========================================
Files 272 272
Lines 19120 19117 -3
===========================================
+ Hits 15327 15332 +5
+ Misses 3793 3785 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
🚀 Automated PR to perform merge from master into rel/dev with changes up to a0e06bc (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/32703319578).