Skip to content

refactor: split the ordering-critical functions (MI campaign 4) - #704

Merged
andris9 merged 90 commits into
masterfrom
refactor/mi-4-ordering
Nov 16, 2026
Merged

andris9 merged 90 commits into
masterfrom
refactor/mi-4-ordering

Conversation

@andris9

@andris9 andris9 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Last of seven PRs in the maintainability campaign. Stacked on #703. Behavior-preserving; the goldens are unchanged.

Full splits

  • validateApiToken() is now a flat sequence of steps that each throw, in the original order: session token, token lookup, audit entry, scope, permission (deferred at the MCP endpoint), account binding, IP, referrer, rate limit, then the allow row. The ordering tests from refactor: tables and pure helpers (MI campaign 1a) #698 pass with no change to their assertions. Every refusal from the scope check on still writes its audit row first.
  • ensurePubsubLocked() is now one method per step, and handlePubsubError() replaces the five copies of the error classification. The app record is still read once, before any adoption.
  • Gmail processHistory(). The recursive page fetch became a loop, in its own commit with the history tests green before and after. It was then split into page, expired-cursor and guarded-entry steps. The marker is still written before an entry is processed, and the cursor still advances after.
  • The server.js worker message handler is split by message kind, with pure resolveMetricKeys and statBuckets in lib/worker-metrics.js. The Prometheus try/catch stays.

Pure pieces only, with tests

  • ensureSubscriptionLocked gains classifyExistingSubscription.
  • licenseCheckHandler gains needsVersionRecheck, subscriptionCheckState and workerSuspensionChange.
  • The account edit POST handler's update builders move to the new lib/ui-routes/account-edit-form.js. They now work on a copy instead of writing into the loaded account. Locks, the re-entrancy guard and the respawn loop stay where they were.

Tests first. Cases that land before their code moves:

  • ensurePubsub: 403 flags for all six phases, network rethrows, TTL PATCH failure, IAM etag and bindings.
  • Gmail history paging and the expired-cursor paths.
  • Outlook subscription expiry.
  • The account edit form's SMTP section, an untouched save, password handling and the error re-render.

Every test was shown to fail under deliberate mutations.

Numbers (on top of #703)

Before After
validateApiToken MI 12.6 44.2
ensurePubsubLocked MI 5.5 49.3
Worker message handler MI 12 43.7
Repository MI 47.18 47.60
Low-MI share 1.90% 0.98%

Checks

  • unit 6393/6393
  • test:cjs 6393/6393
  • integration 264/264, run alone at the final commit
  • /simplify applied
  • /security-review, scoped to this PR's diff, found nothing. It checked auth step order and polarity, audit rows before refusals, the session-token fall-through, and password blanking.

Preserved as-is, worth a look

  • The edit form always submits imap_disabled. The first save of an account that never had that key therefore writes disabled: false, which forces a reconnect.

Merge in its own release cycle, after #703.

Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account
read and write. Compare whole outputs, the stored key order, the logged
failures and the webhookErrorFlag reset in both key orders, so that the
table-driven rewrite has to reproduce them exactly.
…ents

Both run on every listed or fetched message and only single fields were
asserted. Compare whole outputs for Gmail API and Graph payloads (full,
listing, truncated text, minimal format, a bare message), including keys
that hold undefined and the key order, and pin the one provider difference
the split has to keep: Graph checks for an auto-reply only when headers
are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were
asserted. Compare whole outputs for an extended fetch and a listing entry,
and pin the draft detected from labels, the removed \Recent flag, the
fallback to the internal date for an invalid envelope date, the stripped
undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over
two dozen field names inside a forEach callback. The per-field decoding
and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and
FIELD_ENCODERS, one small function per kind of field, and both methods
become a lookup per key. Each entry writes into the result itself, so the
output keeps the input key order the webhookErrorFlag reset depends on,
and encryptConnectionSecrets() replaces the loop that shadowed the outer
key variable.
The minimal-format shape, the date parsing, the header map, the snippet
decoding and the special-use lookup (which existed twice) become module
functions, and the auto-reply and OTP-Token tail moves to
BaseClient.finalizeApiMessage() so the Graph client can share it. The
tail takes the one provider difference as an explicit argument: Gmail
checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection,
the recipient lists and the attachment list become module functions, and
the auto-reply and OTP-Token tail runs through the shared
BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply
only when headers are present. The helpers allocate what the inline code
did: the text maps are filled in place and the attachment list is built
with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error
message), the date fallback, the draft detection, the convenience flag
booleans and the undefined stripping become module functions. They take
the mailbox as an argument rather than living on the class, so the tests
that call getMessageInfo() on a plain context keep working, and compact()
loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into
setStoredJson(), write the Gmail special-use lookup once as
setLabelSpecialUse(), drop the per-call arrays in two type checks and the
Graph text type check, use filter(Boolean) for the Graph recipient
lists, and await packUid() inline in getMessageInfo() so only the failure
path goes through a helper, without an extra async frame per message.
The per-message fetch of a Gmail listing flushed its batch once it held
more than LIST_BATCH_SIZE requests, so every batch ran eleven requests
at once. LIST_BATCH_SIZE is documented as the maximum number of
concurrent listing requests, and executeBatchRequests() in gmail-api.js,
its other user, flushes at exactly that size. The listing now does the
same; the regression test pins the batch sizes of a 25-message page.
The api-token strategy decision is now a flat sequence of awaited steps in
the order the contract names: session token branch, token lookup, audit
entry, scope, permission, account binding, restrictions (address, referrer,
rate limit last), and the allow row. Each step keeps its own log line,
audit write and error shape; the ordering tests in
test/api-token-auth-test.js pass unchanged.
Before ensurePubsubLocked is split, pin what each of its six Google
requests does on failure: the 403 auth flags (including the disabled API
message), the per-phase network warnings and rethrows, other errors
rethrown untouched, the ttlWarning written after a failed TTL PATCH, and
the IAM etag and existing bindings passed through to setIamPolicy.
The topic, subscription, TTL reconciliation, publisher binding and final
metadata write are separate methods over a small run object, and the five
copies of the error classification (transient rethrow, 403 auth flag,
404/409 continuation) are one handlePubsubError(). Each phase keeps its own
log messages and flags, the auth flag write stays fire-and-forget, and the
app record is still read once before any adoption, so a resource adopted
in the same run counts as unmanaged.
Before the history page recursion becomes a loop, pin what crosses pages:
every page is fetched from the run's start with the next page token, the
context is resolved once per run, the cursor ends on the newest history id
any page reported, an entry failing on a later page keeps the cursor on the
last good entry, and a 404 on a later page moves to the target without a
profile lookup. Also pin the profile lookup failing and a non-404 history
error ending the run with the marker kept.
getHistoryPage() returns the next page token and processHistory() loops
on it, so a long history no longer nests one call frame per page. The
expired-cursor branch returns no token, which ends the run where the
recursion unwound. Nothing else moves: the in-flight marker is still
written before an entry and the cursor advanced after it.
The history run keeps its cursor state on a small run object, and the
page fetch, the expired-cursor handling, the guarded processing of one
entry and the poison-entry skip become methods. processHistory itself is
the page loop. The order is unchanged: the in-flight marker is written
before an entry is processed, the cursor advances only after it went
through, and a failing entry still ends the run with the marker kept.
Which dashboard counters a worker's metrics message bumps was decided
inline in the server.js message handler, which no test can load. The
decision and the day and minute buckets move to a pure module with tests;
the handler keeps the Redis writes, the API call duration observation and
the guarded Prometheus call, in the same order.
The listener now counts the message and dispatches on its cmd to named
functions: handleRpcResponse, handleRpcCall, recordWorkerMetrics,
forwardSettings, forwardStateChange, and the onWorkerReady closure, which
keeps isOnline and resolve in spawnWorker and hands an IMAP worker to
onImapWorkerReady for the reassignment bookkeeping. The old fall-through
between kinds only ever reached the ready check, which none of them
matched, so each kind now returns. The Prometheus try/catch stays where a
throw would otherwise kill every worker.
… expiry

Before the classification is pulled out, pin the three outcomes by time
left: inside the clock-skew buffer (or past expiry) a new subscription is
created, inside the renewal window it is renewed under the held lock (and
created anew when the renewal finds it expired), and beyond that, or with
an expiry that does not parse, it is left alone with any failure cleared.
…elper

classifyExistingSubscription() in lib/email-client/outlook/subscription-expiry.js
decides expired, renew or valid from the parsed expiry and the clock, with
its boundaries tested. Only the decision moves: the lock, the save of the
creating state before the POST and the retry scheduling stay in
ensureSubscriptionLocked().
…ndler

needsVersionRecheck(), subscriptionCheckState() and workerSuspensionChange()
join the other subscription rules in lib/license-subscription.js, with
tests: whether the validation record still covers this release, which
license the online check concerns and the suspension a passed deadline
implies, and whether the licensed workers are suspended or resumed. The
re-entrancy guard, the subcheck claim, the online call and the worker
respawn loop stay in the handler, in the same order.
Before the update builders are pulled out of the account edit POST
handler, pin the SMTP section, an untouched save leaving the stored imap
and smtp objects as they were, empty password inputs keeping the stored
passwords, submitted passwords and checkboxes replacing them, and the form
rendered again from the submitted values (passwords blanked) when the
save fails.
buildBaseUpdates, buildImapUpdates and buildSmtpUpdates move from the
account edit POST handler to lib/ui-routes/account-edit-form.js, with tests.
They work on a copy of the stored imap and smtp objects (auth and tls
included) instead of writing into the loaded account; every comparison
reads the same object as before, so the update is the same. The load, the
save and the error re-render stay in the handler. account-routes.js drops
to 517 statements and the new module gets its own budget.
From the simplify pass: the poison-entry skip takes the entry id its
caller already computed, the Gmail publisher binding is built by one
helper instead of three literals, needsVersionRecheck() returns early
instead of flipping a flag, classifyExistingSubscription() checks for a
missing expiry once, statBuckets() formats the date once, and two
comments now say why the SMTP comparison and the two IMAP ready checks
are shaped the way they are.
An Outlook API request that failed with an error description blaming the
client secret called formatPartialSecretKey() on the configured secret.
With no secret configured that threw inside the error handling, which
swallowed it, so the OUTLOOK_CLIENT_SECRET_INVALID flag was never raised
and the error carried no code.

The shared handler now only shows part of the secret when one is
configured, the same guard the Gmail client already applied on its own
(now redundant there and removed), and raises the flag either way. The
test in test/oauth-request-shapes-test.js that pinned the missing flag
now asserts the flag and the error code.
With tokens required (disableTokens off), the api-token validator called
token.startsWith() on whatever it was given, so a call without a token
threw a TypeError that would surface as a 500. The bearer scheme never
calls the validator without one today, but the validator is the
strategy's decision and must not depend on that.

A missing or empty token is now refused with the same 401 "Bad token"
(Bearer challenge) an unknown token gets, before any token lookup and
without an audit row, like the other refusals made before a token is
resolved. The assertion in test/api-token-auth-test.js that pinned the
TypeError now asserts the 401.
The admin account edit form always submits the "IMAP disabled" checkbox,
and the update copied it into the imap object as is. For an account that
never stored imap.disabled, the first save wrote `disabled: false`,
which counts as an IMAP change and forced a reconnect even though
nothing was edited.

A missing key and false now mean the same thing: `disabled` is written
only when the effective state changes. Checking the box still writes
true, and unchecking it on a disabled or parked account still writes
false, which lifts the park. test/ui-routes-handlers-test.js stored the
key explicitly to work around this; its account is now created without
it, and test/account-edit-form-test.js covers the three cases.
arfDetect() collects recipients from the original-message copies
(X-HmXmrOriginalRecipient) and is meant to accumulate them across
attachments: copies that come after the message/feedback-report part
add to the report's Original-Rcpt-To list. A report part that came after
the copies replaced the list instead, so which recipients a complaint
named depended on attachment order.

The report's own recipients now come first, followed by the ones
collected before it that it does not list. Every other report field
still overwrites what the copies filled in. The "report last" case in
test/complaint-test.js pinned the dropped recipients and now asserts
the accumulated list.
…payload

The shared request encoder only recognised the exact string 'get', and
the Outlook client passes the method through as the caller spelled it.
An Outlook request made as 'GET' with a payload therefore had the
payload JSON-encoded into a body, which undici refuses for a GET, so the
request failed before reaching Graph. Every current caller happens to
pass lowercase 'get', so this was a trap for the next one.

The method is now compared case-insensitively, so any spelling of GET
puts the payload in the query string. The Gmail client already
lowercased the method first and is unaffected. The test in
test/oauth-request-shapes-test.js that pinned the refusal now asserts
the query string.
On the Gmail and Graph arrival path, a getRawMessage() that answered
null (Gmail returns null when the provider's answer has no raw message)
was handed to the delivery report parser anyway, which failed and logged
"Failed to process DSN", and the bounce check then downloaded the same
message a second time. The IMAP path already treated an empty source as
a message that is gone and skipped both.

An empty source now means the same on every backend: nothing is parsed,
nothing is logged as an error, and the bounce check does not ask again.
The missingIsGone option that only IMAP set is removed, since the rule
no longer differs. Covered by a new case in
test/base-client-process-new-test.js.
unserializeAccountData() builds the counters object from hash fields
named stats:count:<type>:<counter>. A type of __proto__ resolved to
Object.prototype and the counter was written onto it for the whole
process, constructor wrote onto the Object function, and an inherited
name such as toString wrote onto the shared method. Nothing writes these
fields today, so this was a hazard rather than a reachable bug.

Names that are prototype keys are now skipped, and a type is created as
an own property whenever the object does not already own it. Covered in
test/account-serialize-test.js.
serializeAccountData() handed a key holding undefined to its field
encoder. The plain path and most encoders dropped it, but `path`,
`webhooksCustomHeaders` and the JSON fields (subconnections,
watchResponse, watchFailure) stored the undefined that
JSON.stringify(undefined) returns, which hmset writes as ''. For `path`
that is the cleared marker and reads back as "every folder", silently
widening what the account syncs; for the JSON fields it fails to parse
and is logged on every read. outlookSubscription stored '{}' and an
undefined connection object logged a stringify failure.

A key holding undefined now counts as not given and is skipped before
any encoder runs, the way the plain fields already behaved. No current
caller passes one. The assertions in test/account-serialize-test.js
that pinned the stored undefined, and the stringify log for an
undefined oauth2, now assert that nothing is stored or logged.
serializeAccountData() falls back to toString() for a field with no
encoder, so an object given there was stored as '[object Object]'. That
included the stored-state fields the decoder reads as JSON
(lastErrorState, syncError, smtpStatus, webhookErrorFlag, imapServerInfo,
smtpServerEhlo): they had a decoder but no encoder, so an object written
through the account update could never be read back and its parse
failure was logged on every load. The sync clients write these fields
with their own hset today, so no current caller hit it.

Those six fields now have an encoder that stores an object as JSON and
keeps a string, which is that JSON already, as it is. The plain fallback
refuses an object with no string form of its own and logs it like any
other value that cannot be stringified, instead of storing
'[object Object]'. The assertion in test/account-serialize-test.js that
pinned the '[object Object]' value now asserts the JSON, and a new case
covers the refusal.
…ody structure

Mailbox.getAttachmentList() answers { attachments, textId,
encodedTextSize }, but returned the bare attachments array when the
message had no body structure. Its one caller destructures the object,
so the fields came out undefined and the message info happened to be
right; any caller reading .attachments, or iterating encodedTextSize,
would have got the wrong thing.

It now returns the same object with an empty attachment list, no text
id and no text sizes, which leaves the message info unchanged. Covered
in test/mailbox-get-message-info-test.js.
The fix that writes imap.disabled only when its effective state changes
added a separate statement after the host/port loop, which took
lib/ui-routes/account-edit-form.js two statements over its budget in
test/ui-routes-size-test.js. The check is now the loop's condition for
that one key, with the same behavior and within the budget.
The feedback report and the original-message copies each merged
addresses into original-rcpt-to with their own concat-and-filter. Both
now go through addRecipients(), which adds each address once, before the
listed ones for the report and after them for the copies. Same output.
@andris9
andris9 changed the base branch from refactor/mi-3c-api-clients to master November 16, 2026 21:16
@andris9
andris9 merged commit 48f2fd0 into master Nov 16, 2026
32 checks passed
@andris9
andris9 deleted the refactor/mi-4-ordering branch November 17, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant