Skip to content

refactor: remove duplicated send, OAuth2 and Graph code (MI campaign 2) - #700

Merged
andris9 merged 39 commits into
masterfrom
refactor/mi-2-dedupe
Nov 16, 2026
Merged

andris9 merged 39 commits into
masterfrom
refactor/mi-2-dedupe

Conversation

@andris9

@andris9 andris9 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Third of seven PRs in the maintainability campaign. Stacked on #699. Behavior-preserving; the goldens are unchanged.

What changes

  • Reply/forward reference handling. About 210 of its 244 lines were the same in prepareRawMessage() and queueMessageEntry(). It is now shared as resolveReference(), resolveInlineOptions() and resolveAttachmentReferences(). The three real differences, all used only by the queueing path, become explicit options:
    • checkConnection: the IMAP connection check before the fetch
    • trackUpdate: setting reference.update from the referenced message's flags
    • copyThreadId: copying the thread id unless the caller set one
  • OAuth2 request(). Gmail, Outlook and Mail.ru now share lib/oauth/http-request.js. Each client keeps its own method casing, abort signal or header merge, and way of reading a successful response. The error objects and log entries are pinned field by field and in key order.
  • Outlook getMessage() / getMessages(). They share the Graph field lists, Prefer header, header parsing, error mapping and batch helpers in lib/email-client/outlook/graph-message.js.
  • IMAP mailbox. getMessage() and processNew() share collectCidParts().

Tests first. prepareRawMessage() had no direct test; every existing test stubbed it. It is now pinned against the real implementation, including the whole raw MIME. The new tests also cover:

  • the reference path of queueMessageEntry()
  • every OAuth2 request error shape
  • Outlook message fetch and batch export

Every new test was shown to fail under at least one deliberate mutation: 93 in total.

Numbers (code-health metrics, on top of 1b)

  • Repository MI: 46.02 -> 46.51
  • Low-MI share: 4.40% -> 3.29%
  • Highest CC: 123 -> 103
  • prepareRawMessage: MI 10.7 -> 47.3
  • queueMessageEntry: MI 0 -> 16.5. Its full split is the next PR.

Checks

  • unit 6136/6136
  • test:cjs 6136/6136
  • integration 264/264
  • /simplify applied
  • /security-review found nothing at confidence 8 or above. It checked that the partial-secret guard, token placement and the send-only refusal are unchanged.

Preserved as-is, worth a look later

  • On Outlook with no client secret, formatPartialSecretKey(undefined) throws inside a swallowed path, so that auth flag is never raised.
  • Outlook treats only a lowercase 'get' as GET.

Merge after the release that ships #698 and #699.

Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account
read and write. Compare whole outputs, the stored key order, the logged
failures and the webhookErrorFlag reset in both key orders, so that the
table-driven rewrite has to reproduce them exactly.
…ents

Both run on every listed or fetched message and only single fields were
asserted. Compare whole outputs for Gmail API and Graph payloads (full,
listing, truncated text, minimal format, a bare message), including keys
that hold undefined and the key order, and pin the one provider difference
the split has to keep: Graph checks for an auto-reply only when headers
are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were
asserted. Compare whole outputs for an extended fetch and a listing entry,
and pin the draft detected from labels, the removed \Recent flag, the
fallback to the internal date for an invalid envelope date, the stripped
undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over
two dozen field names inside a forEach callback. The per-field decoding
and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and
FIELD_ENCODERS, one small function per kind of field, and both methods
become a lookup per key. Each entry writes into the result itself, so the
output keeps the input key order the webhookErrorFlag reset depends on,
and encryptConnectionSecrets() replaces the loop that shadowed the outer
key variable.
The minimal-format shape, the date parsing, the header map, the snippet
decoding and the special-use lookup (which existed twice) become module
functions, and the auto-reply and OTP-Token tail moves to
BaseClient.finalizeApiMessage() so the Graph client can share it. The
tail takes the one provider difference as an explicit argument: Gmail
checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection,
the recipient lists and the attachment list become module functions, and
the auto-reply and OTP-Token tail runs through the shared
BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply
only when headers are present. The helpers allocate what the inline code
did: the text maps are filled in place and the attachment list is built
with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error
message), the date fallback, the draft detection, the convenience flag
booleans and the undefined stripping become module functions. They take
the mailbox as an argument rather than living on the class, so the tests
that call getMessageInfo() on a plain context keep working, and compact()
loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into
setStoredJson(), write the Gmail special-use lookup once as
setLabelSpecialUse(), drop the per-call arrays in two type checks and the
Graph text type check, use filter(Boolean) for the Graph recipient
lists, and await packUid() inline in getMessageInfo() so only the failure
path goes through a helper, without an extra async frame per message.
Every existing test stubbed prepareRawMessage(), so the reference block it
shares with queueMessageEntry() had no direct coverage on this side. The new
suite runs the real method against stubbed collaborators and pins the whole
returned object and the raw MIME, plus the ways it differs from the queueing
path: the send-only refusal, a sender defaulted only for a reference action,
no flag update or thread id copied onto the reference.
The reference block moves into shared helpers next. Pin what only the queueing
path does with it (the connection check before the fetch, the flag update
decision, a caller-supplied thread id winning) and the transient-error retry,
with and without ignoreMissing, on mocked timers.
…ueMessageEntry()

About 210 of the 244 lines that resolve a reply or forward were copies of each
other. They now live in resolveReference() and its helpers, with the three
things only the queueing path does passed as options: the connection check
before the fetch, the flag update decision and copying the thread id.
prepareRawMessage() also hands its send-only refusal and its final composer
options to helpers. queueMessageEntry() loses only the shared blocks here.
Gmail, Outlook and Mail.ru each carry a copy of the same request code with
differences that matter: method casing, header merging, how the body is
parsed, which fields an error carries and what is logged. They move into a
shared module next, so pin what each one sends, throws and logs first,
including key order, since both reach API callers and operators.
Gmail, Outlook and Mail.ru each carried a copy of the same request()
code. The shared parts (body and query encoding, the timed fetch, the
failure error with its Retry-After, reading the error body and raising its
flag, parsing a JSON body read as text) move to lib/oauth/http-request.js.
Each client keeps what differs: method casing, header merging, where the
token goes, a logContext() of the fields its errors and logs carry, and
how it reads a successful answer.
Both carry their own copy of the Graph field list, the header parsing and
the formatting, which move into shared helpers next. Pin the request each
sends, the 400 and 404 mapping, marking as seen and inline image embedding
of the single fetch, and the batch retry with its backoff, per-message
errors and the reconciliation of messages a batch answer left out.
…es()

The field list, the attachment expansion, the Prefer header and the header
parsing existed in both, and the expansion a third time in the listing.
They move to lib/email-client/outlook/graph-message.js with the 400/404
mapping, the batch request builder, the retry decision and the
reconciliation of missing responses. getMessages() hands the retry loop to
postBatchWithRetry(), whose wait can be injected, and the response loop to
processBatchResponse(); getMessage() hands inline image embedding to
embedInlineAttachments().
…sNew()

Both looped over the attachments the html refers to by content id, set the
non-enumerable IMAP body part on each and listed the parts to download.
collectCidParts() now does it once; processNew() passes reuseLoaded, which
keeps content notifyAttachments already loaded and skips oversized parts,
the only way the two copies differed.
- getText() and getRawMessage() use bodyPreferHeader() and
  mapGetMessageError(), the last two copies of the Prefer header and the
  400/404 mapping
- the Graph $select value is joined once instead of per batch and call
- the OAuth2 clients use the shared USER_AGENT for token requests as well
- Mail.ru describes a request through its own logContext() like the others
- encodeRequest() tests the payload once
- fetchReferencedMessage() builds ReferenceNotFound in one place: a
  permanent 404 falls through to the check that already throws it
- parseGraphHeaders() and resolveReference() lose guards that could
  never fire
@andris9
andris9 changed the base branch from refactor/mi-1b-shaping to master November 16, 2026 21:16
@andris9
andris9 merged commit db589f0 into master Nov 16, 2026
56 of 59 checks passed
@andris9
andris9 deleted the refactor/mi-2-dedupe branch November 17, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant