fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation - #7050
fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation#7050pabloDeputter wants to merge 6 commits into
Conversation
- merge Sentry baggage with existing vendor (e.g. Datadog) baggage in botocore's`before-sign` hook; avoiding post-sign header tampering that invalidates the SigV4 signature. - Skip propagation for presigned requests Fixes: #7031 & PY-2667
Codecov Results 📊✅ 99287 passed | ⏭️ 6494 skipped | Total: 105781 | Pass Rate: 93.86% | Execution Time: 354m 38s 📊 Comparison with Base Branch
➖ Removed Tests (1)View removed tests
All tests are passing successfully. ✅ Patch coverage is 95.06%. Project has 2508 uncovered lines. Files with missing lines (2)
Coverage diff@@ Coverage Diff @@
## main #PR +/-##
==========================================
+ Coverage 89.95% 89.98% +0.03%
==========================================
Files 193 193 —
Lines 24955 25029 +74
Branches 9004 9038 +34
==========================================
+ Hits 22448 22521 +73
- Misses 2507 2508 +1
- Partials 1437 1440 +3Generated by Codecov Action |
| # do not mutate headers already signed by SigV4. | ||
| if normalized_header in existing_headers and ( | ||
| normalized_header != BAGGAGE_HEADER_NAME | ||
| or normalized_header in signed_headers | ||
| ): | ||
| continue |
There was a problem hiding this comment.
Bug: The logic to avoid overwriting signed headers is too broad. It skips injecting sentry-trace if it already exists, even if it's unsigned, causing stale traces to be propagated.
Severity: LOW
Suggested Fix
The condition should be simplified to only skip headers that are present in signed_headers. The fix is to change the conditional to if normalized_header in signed_headers: continue. This ensures Sentry only avoids mutating headers that are actually part of a SigV4 signature, and will correctly overwrite any stale, unsigned headers.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: sentry_sdk/integrations/stdlib.py#L228-L233
Potential issue: The conditional logic in the patched `endheaders` method incorrectly
skips injecting the `sentry-trace` header if any pre-existing, unsigned `sentry-trace`
header is found. The logic was intended to avoid overwriting SigV4-signed headers, but
the condition `normalized_header != BAGGAGE_HEADER_NAME` causes it to always skip any
pre-existing `sentry-trace` header, regardless of whether it is signed. This can lead to
stale trace information being propagated if user code or middleware adds a
`sentry-trace` header before `endheaders` is called in a non-boto3 context.
Did we get this right? 👍 / 👎 to inform future reviews.
| return rv | ||
|
|
||
| def endheaders(self: "HTTPConnection", *args: "Any", **kwargs: "Any") -> "Any": | ||
| trace_headers = getattr(self, "_sentrysdk_trace_headers", ()) |
There was a problem hiding this comment.
Does calling sentry_sdk.get_current_scope().iter_trace_propagation_headers() here instead of in putrequest() work?
It would be best to avoid stashing stuff on the HTTPConnection instance if we can help it.
Description
Summary of issue
baggagewas not included inSignedHeaders. Any later modifications to the value did not invalidate the request.before-signevent. It addsbaggage, ... andx-datadog-*before signing. Any later modifications to the value DO invalidate the request, thus later HTTP-client injection is suppressed to avoid duplicate headers.before-signhandler writes the baggage to the AWS requestbaggagein the SigV4 signaturebaggagevalue403 ForbiddenorSignatureDoesNotMatch.Changes
before-signhandler, so finalbaggageandsentry-tracevalues are created before SigV4 signing.http.clientpropagation is delayed untilendheaders(), when the complete request headers and SigV4SignedHeadersare available. Existingbaggageheader is never mutated after it already was signed.Issues
Resolves: #7031 & PY-2667
Reminders
uv run ruff.feat:,fix:,ref:,meta:)