[bot] Merge master/bf0e8581 into rel/dev - #1755
Merged
Merged
Conversation
…guardrail guardrail was one of the agentic kinds whose evaluate_agentic_* function returned None instead of an AgenticEvalOutcome, so its reasoning_steps/ conversation_id/response_id were silently dropped -- confirmed live: a real eval run produced 0/30 reasoning sidecars for agentic_guardrail questions, versus 14/15 for the same questions under the old single-shot guardrail evaluator. Mirrors the existing metric_skill/alert_skill/ conversation pattern exactly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…search_tool Same fix as guardrail's -- this kind's reasoning/conversation/response data was captured by the SSE client but never returned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…general_question Same fix as guardrail's/search_tool's. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…visualization Same fix, but multi-turn: reasoning_steps/response_id accumulate across every simulated-user iteration in _execute_single_run, not just the final turn -- mirrors metric_skill's existing accumulation pattern. Covers both vis_agentic and agentic_visualization dispatch entries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…kda_skill Three-part fix, since this kind was missing more than the other four: (1) agent_id was never threaded through run_agentic_kda_skill/ evaluate_agentic_kda_skill/its ChatClient construction at all, unlike every other agentic kind; (2) reasoning_steps/response_id accumulate across iterations, same as the other four kinds; (3) cli/agentic_runner.py's agentic_kda_skill dispatch branch called evaluate_agentic_kda_skill without `return` or `agent_id=` -- a real bug independent of this fix that silently dropped its outcome even once (2) was in place. Extends test_agentic_runner.py's parametrized dispatch coverage to every kind in AGENTIC_TEST_KINDS (was missing agentic_metric_skill/ agentic_alert_skill/agentic_kda_skill) and adds a new parametrized test asserting every kind returns the exact AgenticEvalOutcome its evaluator produced -- confirmed this catches both bugs fixed in this series (reverted the kda_skill `return` fix locally and confirmed the new test fails, then re-applied). Removes the now-false test_run_agentic_items_defaults_reasoning_steps_to_empty_for_untouched_kinds, which had asserted the old bug's behavior (guardrail returning None) as the intended, permanent contract. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- kda_skill.py: the send_message except-branch recovered tool calls from a ChatError's partial_result but dropped its reasoning_steps/response_id, unlike the success path right below it. Extend/retain them the same way, plus a regression test asserting the resulting KdaSkillAssertionError keeps them. - ruff format guardrail.py and test_agentic_search_tool.py (CI's lint-and-format-check was failing on these two).
Companion gap to the reasoning_steps fix above: run_agentic_items never
copied any per-kind diagnostic info into ItemReport.best_detail, so every
agentic result's "detail" field was always {} -- undiagnosable, unlike
the single-shot path which gets this for free from ItemEvaluation.detail.
AgenticEvalOutcome gains a `detail: dict` field, populated by each
evaluate_agentic_* function from whatever its own Result/Evaluation class
already tracks (no new capture needed) -- e.g. actual_maql for
metric_skill, the full visualization per-check breakdown (now shared via
evaluation_result_detail() between the single-shot and agentic paths so
both report the identical shape), judge_reasoning/actual_output for the
LLM-judge kinds (guardrail, general_question). Same *AssertionError
idiom as reasoning_steps: attached as .detail on failure, returned on
the AgenticEvalOutcome on success. cli/agentic_runner.py copies it onto
item_report.best_detail in both branches.
create_metric_alert's tool schema declares internal_recipients as `anyOf: [array of string, string, null]` -- a single id passed as a bare string is schema-legal, not a malformed call. _check_recipients only accepted a list, so a correct alert with a single internal recipient silently failed recipients_correct. Normalize the same way recipients/external_recipients already are just above it. Confirmed live: gpt-5.5 passed internal_recipients as a bare string for a single-recipient alert; every other field matched but recipients_correct still failed.
…-remaining-kinds feat(gooddata-eval): return AgenticEvalOutcome from every agentic kind, not just 3 of 8
…ients-string Fix: accept internal_recipients as a bare string in agentic_alert_skill eval
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1755 +/- ##
===========================================
+ Coverage 80.20% 80.58% +0.38%
===========================================
Files 272 272
Lines 19117 19218 +101
===========================================
+ Hits 15332 15487 +155
+ Misses 3785 3731 -54 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to bf0e858 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/32703665995).