aiohttp integration corrupts SigV4-signed propagation headers, breaking aiobotocore S3 calls (since 2.69.0)

[Robinbinu]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.69.0

Steps to Reproduce

Since 2.69.0, S3 calls made through aiobotocore fail with SignatureDoesNotMatch whenever a Sentry transaction is active. Synchronous boto3 is not affected.

aiobotocore sends botocore's already-signed requests through aiohttp. #7050 made the Boto3 integration add sentry-trace and baggage at before-sign, so both are named in SignedHeaders. The aiohttp client integration then rewrites them after signing, appending to baggage:

if key == BAGGAGE_HEADER_NAME and params.headers.get(BAGGAGE_HEADER_NAME):
    params.headers[key] += "," + value

The sent value no longer matches the signed value, so the request is rejected.

stdlib already skips propagation headers named in SignedHeaders, which is why sync boto3 is fine. The aiohttp path never got the same guard.

Reproduction against any S3-compatible endpoint:

import asyncio, sentry_sdk
import aiobotocore.session as abcs

async def main():
    sentry_sdk.init(dsn="<dsn>", traces_sample_rate=1.0)
    session = abcs.get_session()
    async with session.create_client(
        "s3", region_name="auto", endpoint_url="<endpoint>",
        aws_access_key_id="<key>", aws_secret_access_key="<secret>",
    ) as client:
        with sentry_sdk.start_transaction(op="t", name="r2"):
            await client.list_objects_v2(Bucket="<bucket>", MaxKeys=1)

asyncio.run(main())

Expected Result

The request is sent with the headers botocore signed, and succeeds.

Actual Result

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the
ListObjectsV2 operation: The request signature we calculated does not match the signature you
provided. Check your secret access key and signing method.

Isolation, against a real Cloudflare R2 bucket, with credentials, botocore 1.43.75 and aiobotocore 3.9.1 held constant:

aiobotocore, all default integrations       SignatureDoesNotMatch
aiobotocore, AioHttpIntegration disabled    OK
aiobotocore, Boto3Integration disabled      OK
synchronous boto3, all integrations         OK

By SDK version, same setup:

2.63.0  OK
2.66.0  OK
2.67.0  OK
2.68.0  OK
2.69.0  SignatureDoesNotMatch

To rule out the endpoint being strict about what may be signed: with no SDK involved, botocore signing baggage, sentry-trace, an arbitrary x-koodle-test or x-amz-meta-* is accepted every time. The problem is only that the header changes after signing.

Two notes for anyone hitting this. Calls fail only inside a transaction, so anything running outside one keeps working and it presents as intermittent. And the workarounds are trace_propagation_targets set to exclude the endpoint, or pinning sentry-sdk<2.69.0.

Fix in #7427.

[linear-code]

PY-2769

[Robinbinu]

Updated the description above. My first analysis blamed the endpoint for rejecting signed baggage and sentry-trace, which was wrong: signing those is fine. The cause is the aiohttp integration rewriting them after botocore signed, on a path stdlib already guards. Fix in #7427.

[pabloDeputter]

Thank you for the report! I’ll take a look at the proposed fix.