Skip to content

Feature request: aws_ses source, sign POST requests and add the POST-only tables #86804

Description

@jakesciotto

Feature request

Sign POST requests in the aws_ses client, then add the seven SESv2 read operations that only POST reaches.

Current state: the client is GET-only by construction. send_request builds AWSRequest(method="GET", url=url) and signs it with SigV4Auth, and TRANSPORT_RETRY sets allowed_methods=frozenset(["GET"]). Query parameters go into the URL, and the code encodes them to match the SigV4 canonical query string byte for byte.

SESv2 puts 7 of its 44 read operations on POST, because each one takes a request body. So the client reaches none of them today. This is the reason the source shipped without them, not a scoping preference.

Why now: a customer runs the source against a live AWS account and asks for the widest table coverage we can offer. A separate issue covers the GET-reachable tables.

Every shape below comes from botocore's bundled service model, sesv2/2019-09-27.

Part 1: sign a request body

SigV4Auth.add_auth already signs a body when the AWSRequest carries one. It computes the payload hash and sets x-amz-content-sha256. So the work is to let the client build and send a POST.

  1. Give send_request an optional JSON body, and a method argument that defaults to GET.
  2. Build the AWSRequest with that method and the serialized body. Serialize once, then sign and send the identical bytes. A re-serialization between signing and sending breaks the signature, exactly as a query-string mismatch would.
  3. Set Content-Type: application/json.
  4. Add POST to TRANSPORT_RETRY.allowed_methods. These operations are reads, so a retry is safe despite the verb.
  5. Move NextToken and PageSize into the body for these endpoints. _walk_pages puts them in the query string today, so it needs a per-endpoint switch between query parameters and body fields.
  6. Check the data-imports semgrep transport rules still pass. They constrain how a source builds its session.

Part 2: the tables

Table Path result_key Required input Notes
reputation_entities /v2/email/reputation/entities ReputationEntities none Per-identity and per-configuration-set reputation status. The highest value table in this group for a transactional sender.
import_jobs /v2/email/import-jobs/list ImportJobs none Contact and suppression list import history.
export_jobs /v2/email/list-export-jobs ExportJobs none Metrics and message export history.
tenants /v2/email/tenants/list Tenants none Multi-tenancy. Returns an empty list on an account that uses no tenants.
contacts /v2/email/contact-lists/{ContactListName}/contacts/list Contacts ContactListName Two-level. Fans out from ListContactLists, then paginates inside each list.
tenant_resources /v2/email/tenants/resources/list TenantResources TenantName Two-level. Fans out from ListTenants.
recommendations /v2/email/vdm/recommendations Recommendations none Needs Virtual Deliverability Manager. Report a clear reason when it is off.

contacts and tenant_resources each need pagination inside a fan-out. The current _fanout_page_rows issues exactly one detail request per listed item, so it cannot walk a second NextToken. Extend it, or give these two endpoints their own walk.

ListResourceTenants is the inverse of ListTenantResources. It requires a ResourceArn, and no SESv2 list operation returns the ARNs to feed it. tenant_resources already covers the tenant-to-resource mapping, so leave ListResourceTenants out.

Sync behavior

None of these endpoints accepts a server-side time filter, so every table stays full refresh. Keep sort_mode="desc".

contacts is the one table that can grow large, because it holds one row per subscriber. Confirm the resume path works there. A saved page token inside a fan-out must not restart the outer walk.

Definition of done

  1. The client signs and sends a POST with a request body, and a unit test asserts the signature over that body.
  2. The seven tables appear in the schema picker and sync rows against a live AWS account.
  3. canonical_descriptions.py and ENDPOINT_DESCRIPTIONS carry a description for every new table and column.
  4. probe_endpoint_permissions reports a clear reason for each new table when the IAM policy denies it, and when VDM is off.
  5. SOURCES.md and the posthog.com source doc report the new tables.

Implementation guide: .agents/skills/implementing-warehouse-sources/SKILL.md.

Related: #86803 covers the GET-reachable tables.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions