Skip to content

Commit 3df7ad4

Browse files
mtodorgithub-actions[bot]
authored andcommitted
Update model evaluations 2026-08-18
1 parent fe76289 commit 3df7ad4

1 file changed

Lines changed: 15 additions & 15 deletions

File tree

docs/model-evaluation.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -39,27 +39,27 @@ A task passes when **all** its assertions pass **and** the LLM judge approves th
3939

4040
<!-- model:gpt-5-mini start -->
4141

42-
### gpt-5-mini — 2026-08-04
42+
### gpt-5-mini — 2026-08-18
4343

44-
**Overall: 11/11 tasks passed (100%)**
44+
**Overall: 10/11 tasks passed (90%)**
4545

4646
#### Task Results
4747

4848
| # | Task | Result | toolsUsed | minCalls | maxCalls | Input Tokens | Output Tokens |
4949
|---|------|--------|-----------|----------|----------|--------------|---------------|
50-
| 1 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 507 | 847 |
51-
| 2 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 1496 | 1355 |
52-
| 3 | cve-nonexistent | Pass | Pass | Pass | Pass | 561 | 1569 |
53-
| 4 | rhsa-not-supported | Pass | | Pass | Pass | 618 | 1670 |
54-
| 5 | cve-detected-clusters | Pass | Pass | Pass | Pass | 1513 | 1566 |
55-
| 6 | cve-log4shell | Pass | Pass | Pass | Pass | 1976 | 2654 |
56-
| 7 | cve-clusters-general | Pass | Pass | Pass | Pass | 764 | 2361 |
57-
| 8 | cve-multiple | Pass | Pass | Pass | Pass | 1110 | 2810 |
58-
| 9 | list-clusters | Pass | Pass | Pass | Pass | 668 | 700 |
59-
| 10 | cve-cluster-list | Pass | Pass | Pass | Pass | 674 | 2040 |
60-
| 11 | cve-detected-workloads | Pass | Pass | Pass | Pass | 533 | 1712 |
61-
62-
**Total input tokens**: 10420 | **Total output tokens**: 19284
50+
| 1 | cve-cluster-list | Pass | Pass | Pass | Pass | 456 | 1011 |
51+
| 2 | cve-clusters-general | Pass | Pass | Pass | Pass | 484 | 1350 |
52+
| 3 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 2520 | 1191 |
53+
| 4 | cve-detected-workloads | Pass | Pass | Pass | Pass | 1557 | 1415 |
54+
| 5 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 1531 | 1294 |
55+
| 6 | rhsa-not-supported | Pass | | Pass | Pass | 2120 | 3042 |
56+
| 7 | cve-multiple | Pass | Pass | Pass | Pass | 2134 | 2670 |
57+
| 8 | cve-nonexistent | **Fail** | Pass | Pass | Pass | 561 | 1682 |
58+
| 9 | cve-log4shell | Pass | Pass | Pass | Pass | 3024 | 2123 |
59+
| 10 | list-clusters | Pass | Pass | Pass | Pass | 668 | 490 |
60+
| 11 | cve-detected-clusters | Pass | Pass | Pass | Pass | 703 | 2168 |
61+
62+
**Total input tokens**: 15758 | **Total output tokens**: 18436
6363

6464
<!-- model:gpt-5-mini end -->
6565

0 commit comments

Comments
 (0)