Repository navigation
refactor: split the ordering-critical functions (MI campaign 4) - #704
Merged
Merged
Conversation
Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account read and write. Compare whole outputs, the stored key order, the logged failures and the webhookErrorFlag reset in both key orders, so that the table-driven rewrite has to reproduce them exactly.
…ents Both run on every listed or fetched message and only single fields were asserted. Compare whole outputs for Gmail API and Graph payloads (full, listing, truncated text, minimal format, a bare message), including keys that hold undefined and the key order, and pin the one provider difference the split has to keep: Graph checks for an auto-reply only when headers are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were asserted. Compare whole outputs for an extended fetch and a listing entry, and pin the draft detected from labels, the removed \Recent flag, the fallback to the internal date for an invalid envelope date, the stripped undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over two dozen field names inside a forEach callback. The per-field decoding and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and FIELD_ENCODERS, one small function per kind of field, and both methods become a lookup per key. Each entry writes into the result itself, so the output keeps the input key order the webhookErrorFlag reset depends on, and encryptConnectionSecrets() replaces the loop that shadowed the outer key variable.
The minimal-format shape, the date parsing, the header map, the snippet decoding and the special-use lookup (which existed twice) become module functions, and the auto-reply and OTP-Token tail moves to BaseClient.finalizeApiMessage() so the Graph client can share it. The tail takes the one provider difference as an explicit argument: Gmail checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection, the recipient lists and the attachment list become module functions, and the auto-reply and OTP-Token tail runs through the shared BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply only when headers are present. The helpers allocate what the inline code did: the text maps are filled in place and the attachment list is built with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error message), the date fallback, the draft detection, the convenience flag booleans and the undefined stripping become module functions. They take the mailbox as an argument rather than living on the class, so the tests that call getMessageInfo() on a plain context keep working, and compact() loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into setStoredJson(), write the Gmail special-use lookup once as setLabelSpecialUse(), drop the per-call arrays in two type checks and the Graph text type check, use filter(Boolean) for the Graph recipient lists, and await packUid() inline in getMessageInfo() so only the failure path goes through a helper, without an extra async frame per message.
The per-message fetch of a Gmail listing flushed its batch once it held more than LIST_BATCH_SIZE requests, so every batch ran eleven requests at once. LIST_BATCH_SIZE is documented as the maximum number of concurrent listing requests, and executeBatchRequests() in gmail-api.js, its other user, flushes at exactly that size. The listing now does the same; the regression test pins the batch sizes of a 25-message page.
The api-token strategy decision is now a flat sequence of awaited steps in the order the contract names: session token branch, token lookup, audit entry, scope, permission, account binding, restrictions (address, referrer, rate limit last), and the allow row. Each step keeps its own log line, audit write and error shape; the ordering tests in test/api-token-auth-test.js pass unchanged.
Before ensurePubsubLocked is split, pin what each of its six Google requests does on failure: the 403 auth flags (including the disabled API message), the per-phase network warnings and rethrows, other errors rethrown untouched, the ttlWarning written after a failed TTL PATCH, and the IAM etag and existing bindings passed through to setIamPolicy.
The topic, subscription, TTL reconciliation, publisher binding and final metadata write are separate methods over a small run object, and the five copies of the error classification (transient rethrow, 403 auth flag, 404/409 continuation) are one handlePubsubError(). Each phase keeps its own log messages and flags, the auth flag write stays fire-and-forget, and the app record is still read once before any adoption, so a resource adopted in the same run counts as unmanaged.
Before the history page recursion becomes a loop, pin what crosses pages: every page is fetched from the run's start with the next page token, the context is resolved once per run, the cursor ends on the newest history id any page reported, an entry failing on a later page keeps the cursor on the last good entry, and a 404 on a later page moves to the target without a profile lookup. Also pin the profile lookup failing and a non-404 history error ending the run with the marker kept.
getHistoryPage() returns the next page token and processHistory() loops on it, so a long history no longer nests one call frame per page. The expired-cursor branch returns no token, which ends the run where the recursion unwound. Nothing else moves: the in-flight marker is still written before an entry and the cursor advanced after it.
The history run keeps its cursor state on a small run object, and the page fetch, the expired-cursor handling, the guarded processing of one entry and the poison-entry skip become methods. processHistory itself is the page loop. The order is unchanged: the in-flight marker is written before an entry is processed, the cursor advances only after it went through, and a failing entry still ends the run with the marker kept.
Which dashboard counters a worker's metrics message bumps was decided inline in the server.js message handler, which no test can load. The decision and the day and minute buckets move to a pure module with tests; the handler keeps the Redis writes, the API call duration observation and the guarded Prometheus call, in the same order.
The listener now counts the message and dispatches on its cmd to named functions: handleRpcResponse, handleRpcCall, recordWorkerMetrics, forwardSettings, forwardStateChange, and the onWorkerReady closure, which keeps isOnline and resolve in spawnWorker and hands an IMAP worker to onImapWorkerReady for the reassignment bookkeeping. The old fall-through between kinds only ever reached the ready check, which none of them matched, so each kind now returns. The Prometheus try/catch stays where a throw would otherwise kill every worker.
… expiry Before the classification is pulled out, pin the three outcomes by time left: inside the clock-skew buffer (or past expiry) a new subscription is created, inside the renewal window it is renewed under the held lock (and created anew when the renewal finds it expired), and beyond that, or with an expiry that does not parse, it is left alone with any failure cleared.
…elper classifyExistingSubscription() in lib/email-client/outlook/subscription-expiry.js decides expired, renew or valid from the parsed expiry and the clock, with its boundaries tested. Only the decision moves: the lock, the save of the creating state before the POST and the retry scheduling stay in ensureSubscriptionLocked().
…ndler needsVersionRecheck(), subscriptionCheckState() and workerSuspensionChange() join the other subscription rules in lib/license-subscription.js, with tests: whether the validation record still covers this release, which license the online check concerns and the suspension a passed deadline implies, and whether the licensed workers are suspended or resumed. The re-entrancy guard, the subcheck claim, the online call and the worker respawn loop stay in the handler, in the same order.
Before the update builders are pulled out of the account edit POST handler, pin the SMTP section, an untouched save leaving the stored imap and smtp objects as they were, empty password inputs keeping the stored passwords, submitted passwords and checkboxes replacing them, and the form rendered again from the submitted values (passwords blanked) when the save fails.
buildBaseUpdates, buildImapUpdates and buildSmtpUpdates move from the account edit POST handler to lib/ui-routes/account-edit-form.js, with tests. They work on a copy of the stored imap and smtp objects (auth and tls included) instead of writing into the loaded account; every comparison reads the same object as before, so the update is the same. The load, the save and the error re-render stay in the handler. account-routes.js drops to 517 statements and the new module gets its own budget.
From the simplify pass: the poison-entry skip takes the entry id its caller already computed, the Gmail publisher binding is built by one helper instead of three literals, needsVersionRecheck() returns early instead of flipping a flag, classifyExistingSubscription() checks for a missing expiry once, statBuckets() formats the date once, and two comments now say why the SMTP comparison and the two IMAP ready checks are shaped the way they are.
An Outlook API request that failed with an error description blaming the client secret called formatPartialSecretKey() on the configured secret. With no secret configured that threw inside the error handling, which swallowed it, so the OUTLOOK_CLIENT_SECRET_INVALID flag was never raised and the error carried no code. The shared handler now only shows part of the secret when one is configured, the same guard the Gmail client already applied on its own (now redundant there and removed), and raises the flag either way. The test in test/oauth-request-shapes-test.js that pinned the missing flag now asserts the flag and the error code.
With tokens required (disableTokens off), the api-token validator called token.startsWith() on whatever it was given, so a call without a token threw a TypeError that would surface as a 500. The bearer scheme never calls the validator without one today, but the validator is the strategy's decision and must not depend on that. A missing or empty token is now refused with the same 401 "Bad token" (Bearer challenge) an unknown token gets, before any token lookup and without an audit row, like the other refusals made before a token is resolved. The assertion in test/api-token-auth-test.js that pinned the TypeError now asserts the 401.
The admin account edit form always submits the "IMAP disabled" checkbox, and the update copied it into the imap object as is. For an account that never stored imap.disabled, the first save wrote `disabled: false`, which counts as an IMAP change and forced a reconnect even though nothing was edited. A missing key and false now mean the same thing: `disabled` is written only when the effective state changes. Checking the box still writes true, and unchecking it on a disabled or parked account still writes false, which lifts the park. test/ui-routes-handlers-test.js stored the key explicitly to work around this; its account is now created without it, and test/account-edit-form-test.js covers the three cases.
arfDetect() collects recipients from the original-message copies (X-HmXmrOriginalRecipient) and is meant to accumulate them across attachments: copies that come after the message/feedback-report part add to the report's Original-Rcpt-To list. A report part that came after the copies replaced the list instead, so which recipients a complaint named depended on attachment order. The report's own recipients now come first, followed by the ones collected before it that it does not list. Every other report field still overwrites what the copies filled in. The "report last" case in test/complaint-test.js pinned the dropped recipients and now asserts the accumulated list.
…payload The shared request encoder only recognised the exact string 'get', and the Outlook client passes the method through as the caller spelled it. An Outlook request made as 'GET' with a payload therefore had the payload JSON-encoded into a body, which undici refuses for a GET, so the request failed before reaching Graph. Every current caller happens to pass lowercase 'get', so this was a trap for the next one. The method is now compared case-insensitively, so any spelling of GET puts the payload in the query string. The Gmail client already lowercased the method first and is unaffected. The test in test/oauth-request-shapes-test.js that pinned the refusal now asserts the query string.
On the Gmail and Graph arrival path, a getRawMessage() that answered null (Gmail returns null when the provider's answer has no raw message) was handed to the delivery report parser anyway, which failed and logged "Failed to process DSN", and the bounce check then downloaded the same message a second time. The IMAP path already treated an empty source as a message that is gone and skipped both. An empty source now means the same on every backend: nothing is parsed, nothing is logged as an error, and the bounce check does not ask again. The missingIsGone option that only IMAP set is removed, since the rule no longer differs. Covered by a new case in test/base-client-process-new-test.js.
unserializeAccountData() builds the counters object from hash fields named stats:count:<type>:<counter>. A type of __proto__ resolved to Object.prototype and the counter was written onto it for the whole process, constructor wrote onto the Object function, and an inherited name such as toString wrote onto the shared method. Nothing writes these fields today, so this was a hazard rather than a reachable bug. Names that are prototype keys are now skipped, and a type is created as an own property whenever the object does not already own it. Covered in test/account-serialize-test.js.
serializeAccountData() handed a key holding undefined to its field
encoder. The plain path and most encoders dropped it, but `path`,
`webhooksCustomHeaders` and the JSON fields (subconnections,
watchResponse, watchFailure) stored the undefined that
JSON.stringify(undefined) returns, which hmset writes as ''. For `path`
that is the cleared marker and reads back as "every folder", silently
widening what the account syncs; for the JSON fields it fails to parse
and is logged on every read. outlookSubscription stored '{}' and an
undefined connection object logged a stringify failure.
A key holding undefined now counts as not given and is skipped before
any encoder runs, the way the plain fields already behaved. No current
caller passes one. The assertions in test/account-serialize-test.js
that pinned the stored undefined, and the stringify log for an
undefined oauth2, now assert that nothing is stored or logged.
serializeAccountData() falls back to toString() for a field with no encoder, so an object given there was stored as '[object Object]'. That included the stored-state fields the decoder reads as JSON (lastErrorState, syncError, smtpStatus, webhookErrorFlag, imapServerInfo, smtpServerEhlo): they had a decoder but no encoder, so an object written through the account update could never be read back and its parse failure was logged on every load. The sync clients write these fields with their own hset today, so no current caller hit it. Those six fields now have an encoder that stores an object as JSON and keeps a string, which is that JSON already, as it is. The plain fallback refuses an object with no string form of its own and logs it like any other value that cannot be stringified, instead of storing '[object Object]'. The assertion in test/account-serialize-test.js that pinned the '[object Object]' value now asserts the JSON, and a new case covers the refusal.
…ody structure
Mailbox.getAttachmentList() answers { attachments, textId,
encodedTextSize }, but returned the bare attachments array when the
message had no body structure. Its one caller destructures the object,
so the fields came out undefined and the message info happened to be
right; any caller reading .attachments, or iterating encodedTextSize,
would have got the wrong thing.
It now returns the same object with an empty attachment list, no text
id and no text sizes, which leaves the message info unchanged. Covered
in test/mailbox-get-message-info-test.js.
The fix that writes imap.disabled only when its effective state changes added a separate statement after the host/port loop, which took lib/ui-routes/account-edit-form.js two statements over its budget in test/ui-routes-size-test.js. The check is now the loop's condition for that one key, with the same behavior and within the budget.
The feedback report and the original-message copies each merged addresses into original-rcpt-to with their own concat-and-filter. Both now go through addRecipients(), which adds each address once, before the listed ones for the report and after them for the copies. Same output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of seven PRs in the maintainability campaign. Stacked on #703. Behavior-preserving; the goldens are unchanged.
Full splits
validateApiToken()is now a flat sequence of steps that each throw, in the original order: session token, token lookup, audit entry, scope, permission (deferred at the MCP endpoint), account binding, IP, referrer, rate limit, then the allow row. The ordering tests from refactor: tables and pure helpers (MI campaign 1a) #698 pass with no change to their assertions. Every refusal from the scope check on still writes its audit row first.ensurePubsubLocked()is now one method per step, andhandlePubsubError()replaces the five copies of the error classification. The app record is still read once, before any adoption.processHistory(). The recursive page fetch became a loop, in its own commit with the history tests green before and after. It was then split into page, expired-cursor and guarded-entry steps. The marker is still written before an entry is processed, and the cursor still advances after.server.jsworker message handler is split by message kind, with pureresolveMetricKeysandstatBucketsinlib/worker-metrics.js. The Prometheus try/catch stays.Pure pieces only, with tests
ensureSubscriptionLockedgainsclassifyExistingSubscription.licenseCheckHandlergainsneedsVersionRecheck,subscriptionCheckStateandworkerSuspensionChange.lib/ui-routes/account-edit-form.js. They now work on a copy instead of writing into the loaded account. Locks, the re-entrancy guard and the respawn loop stay where they were.Tests first. Cases that land before their code moves:
ensurePubsub: 403 flags for all six phases, network rethrows, TTL PATCH failure, IAM etag and bindings.Every test was shown to fail under deliberate mutations.
Numbers (on top of #703)
validateApiTokenMIensurePubsubLockedMIChecks
Preserved as-is, worth a look
imap_disabled. The first save of an account that never had that key therefore writesdisabled: false, which forces a reconnect.Merge in its own release cycle, after #703.