Feature request: aws_ses source, add the remaining GET-reachable tables

[jakesciotto]

Feature request

Expand the aws_ses source from 4 tables to 11. Every new table uses a GET operation, so the existing SigV4 client already reaches all of them.

Current state: #79766 shipped the source on 2026-08-10 with account, configuration_sets, email_identities and suppressed_destinations. That was the starting scope named in #79107. The build environment held no AWS credentials, so nobody ran a live sync before the merge, and the source shipped as ALPHA.

Why now: a customer runs the source against a live AWS account today, and the four tables sync. They ask for the widest table coverage we can offer. SESv2 exposes 44 read operations. We use 5 of them.

Every shape below comes from botocore's bundled service model, sesv2/2019-09-27.

Tables to add

Six tables are one new entry each in AWS_SES_ENDPOINTS (settings.py). The existing pagination and fan-out code handles them with no change.

Table	List path	result_key	Detail path	item_name_key	name_column	primary_key
email_templates	/v2/email/templates	TemplatesMetadata	/v2/email/templates/{name}	TemplateName	template_name	["template_name"]
contact_lists	/v2/email/contact-lists	ContactLists	/v2/email/contact-lists/{name}	ContactListName	contact_list_name	["contact_list_name"]
dedicated_ip_pools	/v2/email/dedicated-ip-pools	DedicatedIpPools	/v2/email/dedicated-ip-pools/{name}	None	pool_name	["pool_name"]
dedicated_ips	/v2/email/dedicated-ips	DedicatedIps	none	n/a	n/a	["ip"]
custom_verification_email_templates	/v2/email/custom-verification-email-templates	CustomVerificationEmailTemplates	/v2/email/custom-verification-email-templates/{name}	TemplateName	template_name	["template_name"]
multi_region_endpoints	/v2/email/multi-region-endpoints	MultiRegionEndpoints	none	n/a	n/a	["endpoint_id"]

Per-table notes:

1. email_templates: the list response carries only TemplateName and CreatedTimestamp. GetEmailTemplate adds TemplateContent, which holds Subject, Text and Html. The fan-out is what makes this table useful. Set timestamp_columns=("created_timestamp",).
2. contact_lists: GetContactList adds Topics, Description, CreatedTimestamp and Tags. Topics and Tags are lists, so _flatten keeps them whole. Set timestamp_columns=("created_timestamp", "last_updated_timestamp").
3. dedicated_ip_pools: the list returns plain strings, exactly like ListConfigurationSets, so item_name_key stays None. GetDedicatedIpPool returns a nested DedicatedIpPool structure, which flattens to dedicated_ip_pool_* columns.
4. dedicated_ips: one row per IP, with Ip, WarmupStatus, WarmupPercentage and PoolName. An account with no dedicated IPs returns an empty list. That is not an error, and the table must stay reachable in the schema picker.
5. custom_verification_email_templates: the list already carries every field except TemplateContent. Keep the fan-out anyway, because the template body is the point of the table.
6. multi_region_endpoints: PageSize here is PageSizeV2, which AWS bounds at 1000. Every other endpoint uses an unbounded MaxItems. Set page_size=1000 for this table and 100 for the rest. Regions is a list and stays whole. Set timestamp_columns=("created_timestamp", "last_updated_timestamp").

The seventh table needs a small code change

configuration_set_event_destinations does not fit the current fan-out shape.

• List: /v2/email/configuration-sets, result_key=ConfigurationSets. This reuses the walk that configuration_sets already performs.
• Detail: /v2/email/configuration-sets/{name}/event-destinations.
• The detail response is {"EventDestinations": [...]}, which is a list. Today _fanout_page_rows runs row.update(normalize_row(endpoint_config, detail)), so the whole list would land in one event_destinations column, with one row per configuration set.
• Wanted: one row per configuration set and event destination pair. Set primary_key=["configuration_set_name", "name"].
• Change: add a detail_result_key field to AwsSesEndpointConfig. When a config sets it, _fanout_page_rows iterates detail[detail_result_key] and emits one row per element. When a config leaves it None, the current single-row merge behavior stays.
• Each EventDestination carries Name, Enabled, MatchingEventTypes, and one of KinesisFirehoseDestination, CloudWatchDestination, SnsDestination, EventBridgeDestination or PinpointDestination. _flatten turns those into columns such as sns_destination_topic_arn, which is the wanted shape.

This table earns its place. It reports where SES publishes per-message events today, per configuration set. SESv2 serves no per-message data itself, so this table is how a customer finds the pipeline that does.

Sync behavior

None of the seven endpoints accepts a server-side time filter. So all seven stay full refresh, like the three existing non-suppression tables. Keep sort_mode="desc", and add no INCREMENTAL_FIELDS entries.

SESv2 read APIs throttle at roughly one request per second, and TRANSPORT_RETRY already gives 429 extra headroom. Four of the new tables fan out at one detail request per item. These lists are small in practice, so the request cost is acceptable.

Deliberately excluded

• ListTagsForResource requires a ResourceArn, so it fans out over every identity, configuration set, template and pool. The request cost is high and the value is low.
• GetEmailIdentityPolicies adds nothing. GetEmailIdentity already returns Policies, and email_identities stores it whole through raw_keys.
• GetDedicatedIp and GetSuppressedDestination are single-item reads of rows the list tables already return.
• The POST-shaped list operations need request-body signing first. Feature request: aws_ses source, sign POST requests and add the POST-only tables #86804 tracks them.
• The VDM sending metrics need both body signing and a synthesized table design. Feature request: aws_ses source, add a sending_metrics table from BatchGetMetricData #86806 tracks them.
• The Deliverability Dashboard family needs the paid AWS subscription, at $1,250 per month. Those tables return nothing on a normal account, so they read as broken.
• Per-message send, bounce, open and click stay impossible. SESv2 has no list operation for messages. SES publishes those events to an event destination instead.

Definition of done

1. The seven tables appear in the schema picker and sync rows against a live AWS account.
2. canonical_descriptions.py and ENDPOINT_DESCRIPTIONS carry a description for every new table and column.
3. SOURCES.md reports the new table count.
4. probe_endpoint_permissions reports a clear reason for each new table when the IAM policy denies it.
5. The source doc at contents/docs/cdp/sources/aws-ses.md on posthog.com lists the new tables.

Implementation guide: .agents/skills/implementing-warehouse-sources/SKILL.md.