Feature request
Sign POST requests in the aws_ses client, then add the seven SESv2 read operations that only POST reaches.
Current state: the client is GET-only by construction. send_request builds AWSRequest(method="GET", url=url) and signs it with SigV4Auth, and TRANSPORT_RETRY sets allowed_methods=frozenset(["GET"]). Query parameters go into the URL, and the code encodes them to match the SigV4 canonical query string byte for byte.
SESv2 puts 7 of its 44 read operations on POST, because each one takes a request body. So the client reaches none of them today. This is the reason the source shipped without them, not a scoping preference.
Why now: a customer runs the source against a live AWS account and asks for the widest table coverage we can offer. A separate issue covers the GET-reachable tables.
Every shape below comes from botocore's bundled service model, sesv2/2019-09-27.
Part 1: sign a request body
SigV4Auth.add_auth already signs a body when the AWSRequest carries one. It computes the payload hash and sets x-amz-content-sha256. So the work is to let the client build and send a POST.
- Give
send_request an optional JSON body, and a method argument that defaults to GET.
- Build the
AWSRequest with that method and the serialized body. Serialize once, then sign and send the identical bytes. A re-serialization between signing and sending breaks the signature, exactly as a query-string mismatch would.
- Set
Content-Type: application/json.
- Add
POST to TRANSPORT_RETRY.allowed_methods. These operations are reads, so a retry is safe despite the verb.
- Move
NextToken and PageSize into the body for these endpoints. _walk_pages puts them in the query string today, so it needs a per-endpoint switch between query parameters and body fields.
- Check the data-imports semgrep transport rules still pass. They constrain how a source builds its session.
Part 2: the tables
| Table |
Path |
result_key |
Required input |
Notes |
reputation_entities |
/v2/email/reputation/entities |
ReputationEntities |
none |
Per-identity and per-configuration-set reputation status. The highest value table in this group for a transactional sender. |
import_jobs |
/v2/email/import-jobs/list |
ImportJobs |
none |
Contact and suppression list import history. |
export_jobs |
/v2/email/list-export-jobs |
ExportJobs |
none |
Metrics and message export history. |
tenants |
/v2/email/tenants/list |
Tenants |
none |
Multi-tenancy. Returns an empty list on an account that uses no tenants. |
contacts |
/v2/email/contact-lists/{ContactListName}/contacts/list |
Contacts |
ContactListName |
Two-level. Fans out from ListContactLists, then paginates inside each list. |
tenant_resources |
/v2/email/tenants/resources/list |
TenantResources |
TenantName |
Two-level. Fans out from ListTenants. |
recommendations |
/v2/email/vdm/recommendations |
Recommendations |
none |
Needs Virtual Deliverability Manager. Report a clear reason when it is off. |
contacts and tenant_resources each need pagination inside a fan-out. The current _fanout_page_rows issues exactly one detail request per listed item, so it cannot walk a second NextToken. Extend it, or give these two endpoints their own walk.
ListResourceTenants is the inverse of ListTenantResources. It requires a ResourceArn, and no SESv2 list operation returns the ARNs to feed it. tenant_resources already covers the tenant-to-resource mapping, so leave ListResourceTenants out.
Sync behavior
None of these endpoints accepts a server-side time filter, so every table stays full refresh. Keep sort_mode="desc".
contacts is the one table that can grow large, because it holds one row per subscriber. Confirm the resume path works there. A saved page token inside a fan-out must not restart the outer walk.
Definition of done
- The client signs and sends a POST with a request body, and a unit test asserts the signature over that body.
- The seven tables appear in the schema picker and sync rows against a live AWS account.
canonical_descriptions.py and ENDPOINT_DESCRIPTIONS carry a description for every new table and column.
probe_endpoint_permissions reports a clear reason for each new table when the IAM policy denies it, and when VDM is off.
SOURCES.md and the posthog.com source doc report the new tables.
Implementation guide: .agents/skills/implementing-warehouse-sources/SKILL.md.
Related: #86803 covers the GET-reachable tables.
Feature request
Sign POST requests in the
aws_sesclient, then add the seven SESv2 read operations that only POST reaches.Current state: the client is GET-only by construction.
send_requestbuildsAWSRequest(method="GET", url=url)and signs it withSigV4Auth, andTRANSPORT_RETRYsetsallowed_methods=frozenset(["GET"]). Query parameters go into the URL, and the code encodes them to match the SigV4 canonical query string byte for byte.SESv2 puts 7 of its 44 read operations on POST, because each one takes a request body. So the client reaches none of them today. This is the reason the source shipped without them, not a scoping preference.
Why now: a customer runs the source against a live AWS account and asks for the widest table coverage we can offer. A separate issue covers the GET-reachable tables.
Every shape below comes from botocore's bundled service model,
sesv2/2019-09-27.Part 1: sign a request body
SigV4Auth.add_authalready signs a body when theAWSRequestcarries one. It computes the payload hash and setsx-amz-content-sha256. So the work is to let the client build and send a POST.send_requestan optional JSON body, and a method argument that defaults toGET.AWSRequestwith that method and the serialized body. Serialize once, then sign and send the identical bytes. A re-serialization between signing and sending breaks the signature, exactly as a query-string mismatch would.Content-Type: application/json.POSTtoTRANSPORT_RETRY.allowed_methods. These operations are reads, so a retry is safe despite the verb.NextTokenandPageSizeinto the body for these endpoints._walk_pagesputs them in the query string today, so it needs a per-endpoint switch between query parameters and body fields.Part 2: the tables
result_keyreputation_entities/v2/email/reputation/entitiesReputationEntitiesimport_jobs/v2/email/import-jobs/listImportJobsexport_jobs/v2/email/list-export-jobsExportJobstenants/v2/email/tenants/listTenantscontacts/v2/email/contact-lists/{ContactListName}/contacts/listContactsContactListNameListContactLists, then paginates inside each list.tenant_resources/v2/email/tenants/resources/listTenantResourcesTenantNameListTenants.recommendations/v2/email/vdm/recommendationsRecommendationscontactsandtenant_resourceseach need pagination inside a fan-out. The current_fanout_page_rowsissues exactly one detail request per listed item, so it cannot walk a secondNextToken. Extend it, or give these two endpoints their own walk.ListResourceTenantsis the inverse ofListTenantResources. It requires aResourceArn, and no SESv2 list operation returns the ARNs to feed it.tenant_resourcesalready covers the tenant-to-resource mapping, so leaveListResourceTenantsout.Sync behavior
None of these endpoints accepts a server-side time filter, so every table stays full refresh. Keep
sort_mode="desc".contactsis the one table that can grow large, because it holds one row per subscriber. Confirm the resume path works there. A saved page token inside a fan-out must not restart the outer walk.Definition of done
canonical_descriptions.pyandENDPOINT_DESCRIPTIONScarry a description for every new table and column.probe_endpoint_permissionsreports a clear reason for each new table when the IAM policy denies it, and when VDM is off.SOURCES.mdand the posthog.com source doc report the new tables.Implementation guide:
.agents/skills/implementing-warehouse-sources/SKILL.md.Related: #86803 covers the GET-reachable tables.