Repository navigation
refactor: split the Gmail and Outlook clients (MI campaign 3c) - #703
Merged
Merged
Conversation
Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account read and write. Compare whole outputs, the stored key order, the logged failures and the webhookErrorFlag reset in both key orders, so that the table-driven rewrite has to reproduce them exactly.
…ents Both run on every listed or fetched message and only single fields were asserted. Compare whole outputs for Gmail API and Graph payloads (full, listing, truncated text, minimal format, a bare message), including keys that hold undefined and the key order, and pin the one provider difference the split has to keep: Graph checks for an auto-reply only when headers are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were asserted. Compare whole outputs for an extended fetch and a listing entry, and pin the draft detected from labels, the removed \Recent flag, the fallback to the internal date for an invalid envelope date, the stripped undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over two dozen field names inside a forEach callback. The per-field decoding and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and FIELD_ENCODERS, one small function per kind of field, and both methods become a lookup per key. Each entry writes into the result itself, so the output keeps the input key order the webhookErrorFlag reset depends on, and encryptConnectionSecrets() replaces the loop that shadowed the outer key variable.
The minimal-format shape, the date parsing, the header map, the snippet decoding and the special-use lookup (which existed twice) become module functions, and the auto-reply and OTP-Token tail moves to BaseClient.finalizeApiMessage() so the Graph client can share it. The tail takes the one provider difference as an explicit argument: Gmail checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection, the recipient lists and the attachment list become module functions, and the auto-reply and OTP-Token tail runs through the shared BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply only when headers are present. The helpers allocate what the inline code did: the text maps are filled in place and the attachment list is built with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error message), the date fallback, the draft detection, the convenience flag booleans and the undefined stripping become module functions. They take the mailbox as an argument rather than living on the class, so the tests that call getMessageInfo() on a plain context keep working, and compact() loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into setStoredJson(), write the Gmail special-use lookup once as setLabelSpecialUse(), drop the per-call arrays in two type checks and the Graph text type check, use filter(Boolean) for the Graph recipient lists, and await packUid() inline in getMessageInfo() so only the failure path goes through a helper, without an extra async frame per message.
- getText() and getRawMessage() use bodyPreferHeader() and mapGetMessageError(), the last two copies of the Prefer header and the 400/404 mapping - the Graph $select value is joined once instead of per batch and call - the OAuth2 clients use the shared USER_AGENT for token requests as well - Mail.ru describes a request through its own logContext() like the others - encodeRequest() tests the payload once - fetchReferencedMessage() builds ReferenceNotFound in one place: a permanent 404 falls through to the check that already throws it - parseGraphHeaders() and resolveReference() lose guards that could never fire
…essageEntry() queueMessageEntry() is about to be split into ordered steps. These pin the paths the existing suite did not reach: a missing gateway failing the submission, a base URL with no host falling back to the host name for the List-ID, the preview text being rendered with the other template fields, and the generated Message-ID using the punycode form of an IDN sender domain.
submitMessage() is about to be split into steps. These pin a missing job resolving false without sending, the account penalty after a send that ran into the deadline (and only then), and that every step after the server accepted the message keeps its own failure: a throw there would make the submit worker retry and deliver the message twice.
The header rewriter of getRawEmail() is about to be split into helpers. These pin that an invalid X-EE-Send-At is dropped without replacing the request's send time, and that the Date header follows a future send time of the request but not a past one.
The list unsubscribe headers, the template rendering and the open and click tracking move into their own methods, and the template context, the List-ID domain, the sender domain of a generated Message-ID and the reference part of the response (built twice, for a dry run and a queued message) into module functions. queueMessageEntry() keeps the order of the steps, since each one updates the same data object the next reads, and keeps both early returns.
The SMTP settings, the send itself, the success bookkeeping and the actions after a delivery move into buildSmtpContext(), sendOnce(), recordSubmitSuccess() and runPostSendActions(). Only sendOnce() may throw; every step after it keeps its own try/catch, since a throw once the server accepted the message would make the submit worker deliver it again.
The header rewriter of a raw message becomes extractControlHeaders(), applyAddressHeaders(), buildEnvelope(), ensureStandardHeaders() and applyCustomHeaders(), each returning what the closure used to write into the variables of processMessage(). The closure keeps their order: the envelope is built from the rewritten address headers, and a generated Message-ID uses the envelope sender's domain. The structured path of getRawEmail() becomes buildComposerOptions(), composeMime() and envelopeFromData(), and the returnObject parse of a raw message parseRawToObject().
The post-send helpers read the queue id, Message-ID, envelope and reference from the submission data and the routing from the SMTP context they already get, instead of a separate parameter object. senderDomain() takes the address it uses, buildComposerOptions() reads the send time from the data it is given, and the render condition is checked once instead of per field. The always true baseDomain check in applyListHeaders() goes (listBaseDomain() falls back to localhost), and the Subject override moves out of ensureStandardHeaders() into the rewriter at the same point in the sequence.
Covers the replication-lag retry and messageMissing, the notifyFrom and reconnect skips, the Gmail category, seemsLikeNew, and that every arrival check rethrows a gone connection. These are what the split of processNew() must keep. withInstantSleeps() skips the retry waits of one module so the tier does not sleep through them.
…rocessNew() The ARF path from the attachment download to the complaint event, the reuse of the delivery report download by the bounce check, the calendar step, and the notification target with the path cleared off the payload, which the Outlook client relies on.
The growing wait per batch, syncError recorded and cleared, the abort when another folder is selected or the connection goes away during the wait, the status refresh that moves where the batches end, the connection re-read per attempt, the rows of a failed attempt dropped, and the time bound.
…sage() The text parts read per textType, preProcessHtml, the release of the FETCH lock on every way out, and the second lock the inline image download takes.
The complaint, delivery report and bounce checks were the same sequence in Mailbox.processNew() and BaseClient.processNew(), differing only in how the content is downloaded and in IMAP letting a gone connection propagate. runArrivalChecks() in new-message-steps.js now runs them once, with the download callbacks, the rethrow rule and the expunge handling supplied per backend; runLlmSummary() takes the AI summary step the same way. Mailbox.processNew() also hands the replication-lag retry, the skip filters, the Gmail category, the seen-messages HyperLogLog and the web-safe rendering to helpers of their own, and BaseClient.processNew() its web-safe rendering. Nothing takes a lock of its own, the order of every step is unchanged and header narrowing is still the last step before notify.
fetchUnderLock() is the whole FETCH under the message lock, released on every way out as before; attachTextContent() reads the text parts the caller asked for; embedInlineImagesWithLock() downloads the inline images under the second lock of its own. Nothing moved across the lock boundary. The two suites that drive getMessage() on a hand-built context now hand it fetchUnderLock() as well.
fetchBatchOnce() runs one FETCH of a batch with its own pending chunk, so a retry still refetches the rows that were not compared; classifyFetchRow() sorts the rows; handleFetchFailure() keeps the order of the error branch (interrupt check, syncError, wait, interrupt check, folder check, status refresh, time bound) and returns the refreshed status, which still moves where the batches end; finishFullSync() is the tail. The position stays shared by every batch and the connection is still read per attempt.
classifyFetchRow() counts the row itself and answers whether to compare it, so its kinds no longer have to match the counter keys, and fetchBatchOnce() builds the query object for its logs once per FETCH instead of once per row. fetchBatchOnce() takes the retry state and the status like handleFetchFailure(), arrivalCheckFailed() takes only what it reads, and the last hand-written hidden-content property uses setHiddenContent().
Neither API client's init() had a unit test. Both connect an account through an ordered sequence whose order is the behavior: the park check before any token work, Gmail renewing its watch before the profile probe, the first-login announcement before the connected flip, Graph starting its subscription and timers only after it. Every collaborator records itself into one log so the tests assert the sequence, not just the end state, ahead of splitting both functions.
Graph listMessages() had no unit test and Gmail's covered none of its four shapes. Pins the Graph folder and \All scopes, the three field sets, offset paging and the skiptoken carried through the cursor, and the InefficientFilter retry without ordering; and on Gmail the refused page number, the emailId shortcut, local thread paging, metadataOnly and the per-message fetch that skips a message deleted meanwhile.
A pull that took more than four minutes asks for a fresh service token before acknowledging, and falls back to the original one when that fails. Nothing exercised it; Date.now() is stubbed to move the clock while the pull is in flight.
The two API clients connected an account through two long functions that repeated their first steps, the auth-error catch around the profile probe and their last steps word for word. Those become plain BaseClient helpers each init() calls in its own order: beginApiInit() (state, account, park check), computeSendOnly(), probeApiAuth(), oauthUserUpdate() and finishApiInit() (first-login announcement before the connected flip, then the change event). It is not a template method because the clients differ in where the rest goes: Gmail renews its watch before the profile probe and Graph starts its subscription after the flip. The catch in probeApiAuth() cannot be reportTokenRefreshFailure() itself, which also posts a metric and logs a transient failure, but both now share flagAuthenticationFailure() for the refused case, which differs only in the reported server response code. Gmail keeps loadInitAccountData(), collectProfileUpdates(), ensurePubSubBinding(), syncHistoryOnConnect() and armBackgroundTimers(); Graph keeps checkScopes(), collectProfileUpdates() and startFullAccessServices().
The emailId shortcut and the thread listing each answered from the middle of listMessages() with their own paging. They become listSingleMessage() and listThreadMessages(); the query of a regular listing is built by buildGmailListQuery() and the per-message fetch, with its not-found skip, is fetchMessagesBatched(). The label lookup, the order of the checks and the batch flush condition are unchanged.
…quest steps listMessages() resolved the scope, decoded the cursor, chose one of three field lists, built the search, retried a refused category filter and computed the cursors in one function. Those become resolveListingScope(), parseListPaging(), applyListSearch(), requestListingWithFallback() and buildListCursors(); the field lists and the InefficientFilter classification are pure helpers next to the other Graph message helpers (buildListSelect(), isInefficientFilterError()). The request, its query and the error logged on failure are unchanged.
run() checked the subscription, pulled, routed every message, ACKed with a possible token refresh and classified the pull failure in one function. Those become ensureSubscriptionConfigured(), pullMessages(), processPulledBatch(), ackMessages() and handlePullError(), called from run() in the same order and inside the same try, so a failure anywhere in the pull still reaches the same classification.
computeSendOnly() reads the scope check off each client's own checkScopes() instead of taking it as a callback, Graph's resolveListingScope() drops a temporary and an explicit undefined, and the thread page slice drops a third argument slice() ignores.
When an account had no stored history checkpoint and the Gmail profile answered without a historyId, init() seeded the checkpoint with historyId.toString() on null and threw a TypeError, so the account failed every connection attempt and the worker kept retrying it. There is nothing to seed from in that case, so the seed is skipped with a warning and the account connects. A missing checkpoint is a state the rest of the client already handles: externalNotify() syncs from a missing one, and the next init() seeds it once the profile carries an ID. Writing a placeholder instead would have stored a value that is not a Gmail history ID and that history requests would then start from.
The per-message fetch of a Gmail listing flushed its batch once it held more than LIST_BATCH_SIZE requests, so every batch ran eleven requests at once. LIST_BATCH_SIZE is documented as the maximum number of concurrent listing requests, and executeBatchRequests() in gmail-api.js, its other user, flushes at exactly that size. The listing now does the same; the regression test pins the batch sizes of a 25-message page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixth of seven PRs in the maintainability campaign. Stacked on #702. Behavior-preserving apart from the two
fix:commits at the end. The goldens are unchanged.What changes
init()are split into shared BaseClient helpers (probeApiAuth,flagAuthenticationFailureand friends) plus per-client steps. They stay plain helpers, not a template method, because Gmail renews its watch before the profile probe and Outlook creates its subscription after the connected flip. The login-success webhook is still sent before the account is marked connected.listMessages()is split by listing shape: single message, thread, query, batched fetch.listMessages()is split into scope, paging, select, search and request (with the InefficientFilter fallback), plus cursor steps.run()is split into its steps.Tests first. Neither
init()nor the Outlook listing had a unit test.init()methods is pinned, not just the end state.Fixes
c7cb6459Gmailinit()threw a TypeError when the profile carried no historyId and no checkpoint was stored, so the account failed every connection attempt. It now skips the seed with a warning; the nextinit()seeds it once the profile has an ID.ac2e0d31The Gmail listing flushed its fetch batch atLIST_BATCH_SIZE + 1requests. It now flushes atLIST_BATCH_SIZE, the same asexecuteBatchRequests().Numbers (on top of #702)
Checks
Merge in its own release cycle, after #702.