Skip to content

refactor: split the Gmail and Outlook clients (MI campaign 3c) - #703

Merged
andris9 merged 64 commits into
masterfrom
refactor/mi-3c-api-clients
Nov 16, 2026
Merged

andris9 merged 64 commits into
masterfrom
refactor/mi-3c-api-clients

Conversation

@andris9

@andris9 andris9 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Sixth of seven PRs in the maintainability campaign. Stacked on #702. Behavior-preserving apart from the two fix: commits at the end. The goldens are unchanged.

What changes

  • Gmail and Graph init() are split into shared BaseClient helpers (probeApiAuth, flagAuthenticationFailure and friends) plus per-client steps. They stay plain helpers, not a template method, because Gmail renews its watch before the profile probe and Outlook creates its subscription after the connected flip. The login-success webhook is still sent before the account is marked connected.
  • Gmail listMessages() is split by listing shape: single message, thread, query, batched fetch.
  • Graph listMessages() is split into scope, paging, select, search and request (with the InefficientFilter fallback), plus cursor steps.
  • The Pub/Sub pull loop run() is split into its steps.

Tests first. Neither init() nor the Outlook listing had a unit test.

  • The order of side effects in both init() methods is pinned, not just the end state.
  • Both listings are covered.
  • The Pub/Sub token refresh before a late ACK is covered.

Fixes

  • c7cb6459 Gmail init() threw a TypeError when the profile carried no historyId and no checkpoint was stored, so the account failed every connection attempt. It now skips the seed with a warning; the next init() seeds it once the profile has an ID.
  • ac2e0d31 The Gmail listing flushed its fetch batch at LIST_BATCH_SIZE + 1 requests. It now flushes at LIST_BATCH_SIZE, the same as executeBatchRequests().

Numbers (on top of #702)

  • Repository MI: 46.92 -> 47.18
  • Low-MI share: 2.50% -> 1.90%
  • Own functions below MI 20: 7 -> 4

Checks

  • unit 6328/6328
  • test:cjs 6328/6328
  • integration 264/264
  • /simplify applied
  • /security-review found nothing. It covered URL encoding, auth-failure classification, the state-change order and Pub/Sub ownership checks.

Merge in its own release cycle, after #702.

Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account
read and write. Compare whole outputs, the stored key order, the logged
failures and the webhookErrorFlag reset in both key orders, so that the
table-driven rewrite has to reproduce them exactly.
…ents

Both run on every listed or fetched message and only single fields were
asserted. Compare whole outputs for Gmail API and Graph payloads (full,
listing, truncated text, minimal format, a bare message), including keys
that hold undefined and the key order, and pin the one provider difference
the split has to keep: Graph checks for an auto-reply only when headers
are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were
asserted. Compare whole outputs for an extended fetch and a listing entry,
and pin the draft detected from labels, the removed \Recent flag, the
fallback to the internal date for an invalid envelope date, the stripped
undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over
two dozen field names inside a forEach callback. The per-field decoding
and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and
FIELD_ENCODERS, one small function per kind of field, and both methods
become a lookup per key. Each entry writes into the result itself, so the
output keeps the input key order the webhookErrorFlag reset depends on,
and encryptConnectionSecrets() replaces the loop that shadowed the outer
key variable.
The minimal-format shape, the date parsing, the header map, the snippet
decoding and the special-use lookup (which existed twice) become module
functions, and the auto-reply and OTP-Token tail moves to
BaseClient.finalizeApiMessage() so the Graph client can share it. The
tail takes the one provider difference as an explicit argument: Gmail
checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection,
the recipient lists and the attachment list become module functions, and
the auto-reply and OTP-Token tail runs through the shared
BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply
only when headers are present. The helpers allocate what the inline code
did: the text maps are filled in place and the attachment list is built
with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error
message), the date fallback, the draft detection, the convenience flag
booleans and the undefined stripping become module functions. They take
the mailbox as an argument rather than living on the class, so the tests
that call getMessageInfo() on a plain context keep working, and compact()
loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into
setStoredJson(), write the Gmail special-use lookup once as
setLabelSpecialUse(), drop the per-call arrays in two type checks and the
Graph text type check, use filter(Boolean) for the Graph recipient
lists, and await packUid() inline in getMessageInfo() so only the failure
path goes through a helper, without an extra async frame per message.
- getText() and getRawMessage() use bodyPreferHeader() and
  mapGetMessageError(), the last two copies of the Prefer header and the
  400/404 mapping
- the Graph $select value is joined once instead of per batch and call
- the OAuth2 clients use the shared USER_AGENT for token requests as well
- Mail.ru describes a request through its own logContext() like the others
- encodeRequest() tests the payload once
- fetchReferencedMessage() builds ReferenceNotFound in one place: a
  permanent 404 falls through to the check that already throws it
- parseGraphHeaders() and resolveReference() lose guards that could
  never fire
…essageEntry()

queueMessageEntry() is about to be split into ordered steps. These pin the
paths the existing suite did not reach: a missing gateway failing the
submission, a base URL with no host falling back to the host name for the
List-ID, the preview text being rendered with the other template fields, and
the generated Message-ID using the punycode form of an IDN sender domain.
submitMessage() is about to be split into steps. These pin a missing job
resolving false without sending, the account penalty after a send that ran
into the deadline (and only then), and that every step after the server
accepted the message keeps its own failure: a throw there would make the
submit worker retry and deliver the message twice.
The header rewriter of getRawEmail() is about to be split into helpers. These
pin that an invalid X-EE-Send-At is dropped without replacing the request's
send time, and that the Date header follows a future send time of the request
but not a past one.
The list unsubscribe headers, the template rendering and the open and click
tracking move into their own methods, and the template context, the List-ID
domain, the sender domain of a generated Message-ID and the reference part of
the response (built twice, for a dry run and a queued message) into module
functions. queueMessageEntry() keeps the order of the steps, since each one
updates the same data object the next reads, and keeps both early returns.
The SMTP settings, the send itself, the success bookkeeping and the actions
after a delivery move into buildSmtpContext(), sendOnce(),
recordSubmitSuccess() and runPostSendActions(). Only sendOnce() may throw;
every step after it keeps its own try/catch, since a throw once the server
accepted the message would make the submit worker deliver it again.
The header rewriter of a raw message becomes extractControlHeaders(),
applyAddressHeaders(), buildEnvelope(), ensureStandardHeaders() and
applyCustomHeaders(), each returning what the closure used to write into the
variables of processMessage(). The closure keeps their order: the envelope is
built from the rewritten address headers, and a generated Message-ID uses the
envelope sender's domain.

The structured path of getRawEmail() becomes buildComposerOptions(),
composeMime() and envelopeFromData(), and the returnObject parse of a raw
message parseRawToObject().
The post-send helpers read the queue id, Message-ID, envelope and reference
from the submission data and the routing from the SMTP context they already
get, instead of a separate parameter object. senderDomain() takes the
address it uses, buildComposerOptions() reads the send time from the data it
is given, and the render condition is checked once instead of per field. The
always true baseDomain check in applyListHeaders() goes (listBaseDomain()
falls back to localhost), and the Subject override moves out of
ensureStandardHeaders() into the rewriter at the same point in the sequence.
Covers the replication-lag retry and messageMissing, the notifyFrom and
reconnect skips, the Gmail category, seemsLikeNew, and that every arrival
check rethrows a gone connection. These are what the split of processNew()
must keep. withInstantSleeps() skips the retry waits of one module so the
tier does not sleep through them.
…rocessNew()

The ARF path from the attachment download to the complaint event, the reuse
of the delivery report download by the bounce check, the calendar step, and
the notification target with the path cleared off the payload, which the
Outlook client relies on.
The growing wait per batch, syncError recorded and cleared, the abort when
another folder is selected or the connection goes away during the wait, the
status refresh that moves where the batches end, the connection re-read per
attempt, the rows of a failed attempt dropped, and the time bound.
…sage()

The text parts read per textType, preProcessHtml, the release of the FETCH
lock on every way out, and the second lock the inline image download takes.
The complaint, delivery report and bounce checks were the same sequence in
Mailbox.processNew() and BaseClient.processNew(), differing only in how the
content is downloaded and in IMAP letting a gone connection propagate.
runArrivalChecks() in new-message-steps.js now runs them once, with the
download callbacks, the rethrow rule and the expunge handling supplied per
backend; runLlmSummary() takes the AI summary step the same way.

Mailbox.processNew() also hands the replication-lag retry, the skip
filters, the Gmail category, the seen-messages HyperLogLog and the web-safe
rendering to helpers of their own, and BaseClient.processNew() its web-safe
rendering. Nothing takes a lock of its own, the order of every step is
unchanged and header narrowing is still the last step before notify.
fetchUnderLock() is the whole FETCH under the message lock, released on
every way out as before; attachTextContent() reads the text parts the
caller asked for; embedInlineImagesWithLock() downloads the inline images
under the second lock of its own. Nothing moved across the lock boundary.
The two suites that drive getMessage() on a hand-built context now hand it
fetchUnderLock() as well.
fetchBatchOnce() runs one FETCH of a batch with its own pending chunk, so a
retry still refetches the rows that were not compared; classifyFetchRow()
sorts the rows; handleFetchFailure() keeps the order of the error branch
(interrupt check, syncError, wait, interrupt check, folder check, status
refresh, time bound) and returns the refreshed status, which still moves
where the batches end; finishFullSync() is the tail. The position stays
shared by every batch and the connection is still read per attempt.
classifyFetchRow() counts the row itself and answers whether to compare it,
so its kinds no longer have to match the counter keys, and fetchBatchOnce()
builds the query object for its logs once per FETCH instead of once per
row. fetchBatchOnce() takes the retry state and the status like
handleFetchFailure(), arrivalCheckFailed() takes only what it reads, and
the last hand-written hidden-content property uses setHiddenContent().
Neither API client's init() had a unit test. Both connect an account
through an ordered sequence whose order is the behavior: the park check
before any token work, Gmail renewing its watch before the profile probe,
the first-login announcement before the connected flip, Graph starting
its subscription and timers only after it. Every collaborator records
itself into one log so the tests assert the sequence, not just the end
state, ahead of splitting both functions.
Graph listMessages() had no unit test and Gmail's covered none of its
four shapes. Pins the Graph folder and \All scopes, the three field
sets, offset paging and the skiptoken carried through the cursor, and
the InefficientFilter retry without ordering; and on Gmail the refused
page number, the emailId shortcut, local thread paging, metadataOnly
and the per-message fetch that skips a message deleted meanwhile.
A pull that took more than four minutes asks for a fresh service token
before acknowledging, and falls back to the original one when that
fails. Nothing exercised it; Date.now() is stubbed to move the clock
while the pull is in flight.
The two API clients connected an account through two long functions
that repeated their first steps, the auth-error catch around the
profile probe and their last steps word for word. Those become plain
BaseClient helpers each init() calls in its own order: beginApiInit()
(state, account, park check), computeSendOnly(), probeApiAuth(),
oauthUserUpdate() and finishApiInit() (first-login announcement before
the connected flip, then the change event). It is not a template
method because the clients differ in where the rest goes: Gmail renews
its watch before the profile probe and Graph starts its subscription
after the flip.

The catch in probeApiAuth() cannot be reportTokenRefreshFailure()
itself, which also posts a metric and logs a transient failure, but
both now share flagAuthenticationFailure() for the refused case, which
differs only in the reported server response code.

Gmail keeps loadInitAccountData(), collectProfileUpdates(),
ensurePubSubBinding(), syncHistoryOnConnect() and armBackgroundTimers();
Graph keeps checkScopes(), collectProfileUpdates() and
startFullAccessServices().
The emailId shortcut and the thread listing each answered from the
middle of listMessages() with their own paging. They become
listSingleMessage() and listThreadMessages(); the query of a regular
listing is built by buildGmailListQuery() and the per-message fetch,
with its not-found skip, is fetchMessagesBatched(). The label lookup,
the order of the checks and the batch flush condition are unchanged.
…quest steps

listMessages() resolved the scope, decoded the cursor, chose one of
three field lists, built the search, retried a refused category filter
and computed the cursors in one function. Those become
resolveListingScope(), parseListPaging(), applyListSearch(),
requestListingWithFallback() and buildListCursors(); the field lists
and the InefficientFilter classification are pure helpers next to the
other Graph message helpers (buildListSelect(), isInefficientFilterError()).
The request, its query and the error logged on failure are unchanged.
run() checked the subscription, pulled, routed every message, ACKed
with a possible token refresh and classified the pull failure in one
function. Those become ensureSubscriptionConfigured(), pullMessages(),
processPulledBatch(), ackMessages() and handlePullError(), called from
run() in the same order and inside the same try, so a failure anywhere
in the pull still reaches the same classification.
computeSendOnly() reads the scope check off each client's own
checkScopes() instead of taking it as a callback, Graph's
resolveListingScope() drops a temporary and an explicit undefined, and
the thread page slice drops a third argument slice() ignores.
When an account had no stored history checkpoint and the Gmail profile
answered without a historyId, init() seeded the checkpoint with
historyId.toString() on null and threw a TypeError, so the account
failed every connection attempt and the worker kept retrying it.

There is nothing to seed from in that case, so the seed is skipped with
a warning and the account connects. A missing checkpoint is a state the
rest of the client already handles: externalNotify() syncs from a
missing one, and the next init() seeds it once the profile carries an
ID. Writing a placeholder instead would have stored a value that is not
a Gmail history ID and that history requests would then start from.
The per-message fetch of a Gmail listing flushed its batch once it held
more than LIST_BATCH_SIZE requests, so every batch ran eleven requests
at once. LIST_BATCH_SIZE is documented as the maximum number of
concurrent listing requests, and executeBatchRequests() in gmail-api.js,
its other user, flushes at exactly that size. The listing now does the
same; the regression test pins the batch sizes of a 25-message page.
@andris9
andris9 changed the base branch from refactor/mi-3b-arrival to master November 16, 2026 21:16
@andris9
andris9 merged commit ac2e0d3 into master Nov 16, 2026
31 checks passed
@andris9
andris9 deleted the refactor/mi-3c-api-clients branch November 17, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant