Skip to content

refactor: hot-path data shaping (MI campaign 1b) - #699

Merged
andris9 merged 30 commits into
masterfrom
refactor/mi-1b-shaping
Nov 16, 2026
Merged

andris9 merged 30 commits into
masterfrom
refactor/mi-1b-shaping

Conversation

@andris9

@andris9 andris9 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Second of seven PRs in the maintainability campaign. Stacked on #698. Behavior-preserving; the goldens are unchanged.

These functions run on every account read or every message, so whole-output fixtures land first:

  • Account hash codec. Covers the exact stored hash, its key order, the webhookErrorFlag reset in both key orders, encryption, and the round trip.
  • Gmail and Graph formatMessage. Compares the whole output, including keys that hold undefined and the JSON key order.
  • IMAP getMessageInfo.

Each new test was shown to fail under deliberate mutations: 40 in total.

What changes

  • lib/account.js: both serializer callbacks become FIELD_DECODERS / FIELD_ENCODERS tables in the new lib/account/field-codec.js. This also removes the shadowed loop variable in the encrypt step.
  • Gmail and Graph formatMessage are split into module helpers.
    • The special-use loop that existed twice is now one function.
    • Both clients share finalizeApiMessage(message, autoReplyNeedsHeaders), which keeps the existing difference: Outlook checks for auto-replies only when headers exist.
  • IMAP getMessageInfo is split into module helpers.
  • No per-message allocations were added. The tables are module-level, and codec lookups go through a Map.

Numbers (code-health metrics, on top of 1a)

  • Repository MI: 45.78 -> 46.02
  • Low-MI share: 4.80% -> 4.40%

Checks: unit 6040/6040, test:cjs 6040/6040, integration 264/264. /simplify was applied. /security-review found nothing at confidence 8 or above.

Preserved as-is

  • A path or webhooksCustomHeaders set to undefined is still stored from JSON.stringify(undefined).
  • An object on the plain-string path is still stored as '[object Object]'.

Merge together with #698, after the next release is cut.

Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account
read and write. Compare whole outputs, the stored key order, the logged
failures and the webhookErrorFlag reset in both key orders, so that the
table-driven rewrite has to reproduce them exactly.
…ents

Both run on every listed or fetched message and only single fields were
asserted. Compare whole outputs for Gmail API and Graph payloads (full,
listing, truncated text, minimal format, a bare message), including keys
that hold undefined and the key order, and pin the one provider difference
the split has to keep: Graph checks for an auto-reply only when headers
are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were
asserted. Compare whole outputs for an extended fetch and a listing entry,
and pin the draft detected from labels, the removed \Recent flag, the
fallback to the internal date for an invalid envelope date, the stripped
undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over
two dozen field names inside a forEach callback. The per-field decoding
and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and
FIELD_ENCODERS, one small function per kind of field, and both methods
become a lookup per key. Each entry writes into the result itself, so the
output keeps the input key order the webhookErrorFlag reset depends on,
and encryptConnectionSecrets() replaces the loop that shadowed the outer
key variable.
The minimal-format shape, the date parsing, the header map, the snippet
decoding and the special-use lookup (which existed twice) become module
functions, and the auto-reply and OTP-Token tail moves to
BaseClient.finalizeApiMessage() so the Graph client can share it. The
tail takes the one provider difference as an explicit argument: Gmail
checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection,
the recipient lists and the attachment list become module functions, and
the auto-reply and OTP-Token tail runs through the shared
BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply
only when headers are present. The helpers allocate what the inline code
did: the text maps are filled in place and the attachment list is built
with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error
message), the date fallback, the draft detection, the convenience flag
booleans and the undefined stripping become module functions. They take
the mailbox as an argument rather than living on the class, so the tests
that call getMessageInfo() on a plain context keep working, and compact()
loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into
setStoredJson(), write the Gmail special-use lookup once as
setLabelSpecialUse(), drop the per-call arrays in two type checks and the
Graph text type check, use filter(Boolean) for the Graph recipient
lists, and await packUid() inline in getMessageInfo() so only the failure
path goes through a helper, without an extra async frame per message.
@andris9
andris9 changed the base branch from refactor/mi-1a-tables to master November 16, 2026 21:16
@andris9
andris9 merged commit dc33cb7 into master Nov 16, 2026
57 of 59 checks passed
@andris9
andris9 deleted the refactor/mi-1b-shaping branch November 17, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant