Repository navigation
refactor: split the submit path (MI campaign 3a) - #701
Merged
Merged
Conversation
Covers every entry of the chain through resolver() with stand-in DNS and fetch, including the two that delegate to an autoconfig file, the first-match order, the two exact matches and the fall-through to the racing resolvers, so the chain can become a table without a change going unnoticed.
The provider chain in resolveUsingMX was 230 lines of if blocks that each returned the same object shape. It is now the MX_PROVIDERS table, checked in the same order, with the two exact string matches kept exact and staticConfig() building a fresh object per lookup.
validateApiToken documents its order (token, scope, permissions, account, restrictions as address, referrer, rate limit) but the tests only pinned scope before permission. Each new case fails two checks at once and asserts which refusal wins, which audit rows are written and which lookups never happen, plus where a session token, a missing token and a session token on a non-api route or the MCP door go. The strategy is split in a later change; these have to pass unchanged across it.
Only the Gmail paths of getClient() were reached by a test. This covers the 400 for every missing credential of each provider, the 404 and unknown-provider refusals, the Mail.ru client, and that caller options win the merge over the stored record, before the per-provider branches become a table.
getClient() repeated the same shape five times: read the record, refuse missing credentials with a 400, build the client with the flag setter and the caller's options last. CLIENT_BUILDERS holds what differs per provider and assertCredentials() the shared refusal. The awaits stay in the same order per provider and the caller's options still win the merge.
prepareFilterQuery() had tests for its dates and labels only. These cover the refused terms (by presence, whatever the value), the boolean-only flags, the from expression, the falsy skips, the message-id header in any case and the refused headers, and the order the parts are joined in.
prepareFilterQuery() was one branch per search term. The terms are now four ordered tables (refused, boolean, string and date terms) with labelFilters() and headerFilters() for the two that loop, so the parts come out in the same order and the same terms are refused with the same error.
Only a refused connection was covered, through the integration tier. The route handler is now driven directly with verifyAccountInfo stood in, so every IMAP and SMTP code reaches its translation, the OpenSSL text fallbacks keep their case rules, and the signature gate is shown to run before any connection.
The connection check handler carried two long switch statements mapping error codes to messages. They are now IMAP_ERROR_MESSAGES and SMTP_ERROR_MESSAGES in the new lib/ui-routes/account-page.js, built per call with the request's translator so every gettext() argument stays a literal. The signature gate is still the handler's first statement, and the account-routes.js budget drops to its new statement count.
The hosted form's connection check messages moved to lib/ui-routes/account-page.js, so the catalog has to name that module for them. The set of strings is unchanged; only their source references move.
buildToolRegistry() refuses a malformed plugins.mcp block at build time. A duplicate tool name, a forced value naming no argument or an offered one, and the plural forms of the unknown-option and hidden-required messages had no test, so the split of the builder could drop one unnoticed.
buildToolRegistry() derived each tool in one 200-line loop body. Each step is now its own function (validateMcpOptions, resolveRenames, partitionArguments, applyForce, applyBounds, buildAnnotations, buildToolEntry), called in the same order so the build-time refusals keep their precedence. The tool manifest golden is unchanged.
The fixture tests check single fields of real complaints. These assert the whole report for a feedback report plus two copies of the original headers in both orders: report fields overwrite what original headers filled in, original headers only fill in what is missing, recipients accumulate without duplicates, the last Return-Path names the Hotmail sender, and the Hotmail defaults sit under the report.
arfDetect() parsed both attachment kinds inline in one switch. The feedback report is now parseFeedbackReport() over normalizeArfField(), and a copy of the original headers is readOriginalHeaders() plus parseOriginalHeaders() with a HEADER_EXTRACTORS table for the six copied headers. First-wins for the copied headers, last-wins for report fields and Return-Path, and the size caps are unchanged.
The API pass-through, a LIST failure with no status or code, a disconnected account served from (or missing) the stored listing, a live entry without a stored counterpart, and the 503 for a failure while merging had no test.
The listing read did the state gate, the live LIST, the per-folder status pipeline and the merge in one function, and built the same Boom error for a failed LIST twice. listingError() builds that error once; fetchLiveListing(), indexLiveListing(), readMailboxStatuses() and mergeMailboxEntry() are the steps, called in the same order, and the merge failure is still caught and reported per folder in the caller.
updateQueueCounters() set every gauge in one function. It now runs setConfigMetrics(), setThreadMetrics(), setQueueMetrics() and setRedisMetrics() in the same order, and the two values it computed inline, the license days left and the thread counts, are pure functions in lib/metric-values.js with their own tests, since server.js cannot be loaded by one.
The account page handler had no test. It is now driven with a stood-in account record, pinning the SMTP auth mechanisms read out of the EHLO lines, the support URL check, the Gmail watch card, the precedence of the top-of-page alert, the webhook routing card and the path list, before the handler's pieces move out.
The account page handler built the SMTP mechanism list, the Gmail watch card, the top-of-page alert and the webhook routing card inline. They are now describeSmtpAuthMechanisms() over an SMTP_AUTH_INFO table, buildGmailWatchView(), buildAccountAlert() and loadWebhookRouting(), called at the same points. The support URL check stays in the handler, and the account-routes.js budget drops to its new statement count.
The view context lived inside workers/api.js, which a test cannot load, so the system alerts every admin page shows had no test. It is now buildViewContext() with resolveAuthDisplay() over a provider table, a pure buildSystemAlerts() over an ALERT_RULES array in display order, and licenseDetailsView(). The move is mechanical; the old and new context were compared key for key and value for value over 3840 combinations of settings, license, session and auth flag state, and the new module has its own tests.
Review pass over this change: the OData filter refusals use the shared statusError() instead of a local copy, the two not-ready refusals of the mailbox listing share notReadyError(), the ARF from extractor reuses addressList(), and a comment the filter tables had separated from LISTING_REFRESH_INTERVAL is back above it.
The pattern was neither anchored nor had the dot before com escaped, so an exchange that merely contained a WorkMail host (inbound-smtp.<region>.amazonaws.com.attacker.example, or a different domain in place of amazonaws.com) was offered WorkMail's IMAP and SMTP hosts. Harmless in practice, the captured region cannot contain a dot, but the pattern now matches only the real exchange name and such a lookalike falls through to the other resolvers.
serializeAccountData() and unserializeAccountData() run on every account read and write. Compare whole outputs, the stored key order, the logged failures and the webhookErrorFlag reset in both key orders, so that the table-driven rewrite has to reproduce them exactly.
…ents Both run on every listed or fetched message and only single fields were asserted. Compare whole outputs for Gmail API and Graph payloads (full, listing, truncated text, minimal format, a bare message), including keys that hold undefined and the key order, and pin the one provider difference the split has to keep: Graph checks for an auto-reply only when headers are present, Gmail checks every message.
Every IMAP message is shaped by it and only its address fields were asserted. Compare whole outputs for an extended fetch and a listing entry, and pin the draft detected from labels, the removed \Recent flag, the fallback to the internal date for an invalid envelope date, the stripped undefined fields and the packUid failure message and log entry.
unserializeAccountData() and serializeAccountData() each switched over two dozen field names inside a forEach callback. The per-field decoding and encoding now lives in lib/account/field-codec.js as FIELD_DECODERS and FIELD_ENCODERS, one small function per kind of field, and both methods become a lookup per key. Each entry writes into the result itself, so the output keeps the input key order the webhookErrorFlag reset depends on, and encryptConnectionSecrets() replaces the loop that shadowed the outer key variable.
The minimal-format shape, the date parsing, the header map, the snippet decoding and the special-use lookup (which existed twice) become module functions, and the auto-reply and OTP-Token tail moves to BaseClient.finalizeApiMessage() so the Graph client can share it. The tail takes the one provider difference as an explicit argument: Gmail checks every message for an auto-reply, Graph only one with headers.
The minimal-fields shape, the flag mapping, the body text collection, the recipient lists and the attachment list become module functions, and the auto-reply and OTP-Token tail runs through the shared BaseClient.finalizeApiMessage(), told that Graph checks for an auto-reply only when headers are present. The helpers allocate what the inline code did: the text maps are filled in place and the attachment list is built with a loop instead of a per-message map callback.
The packUid failure path (stored status lookup, log entry, error message), the date fallback, the draft detection, the convenience flag booleans and the undefined stripping become module functions. They take the mailbox as an argument rather than living on the class, so the tests that call getMessageInfo() on a plain context keep working, and compact() loops over the keys instead of allocating a callback per message.
Fold the repeated parse-then-set step of the account decoders into setStoredJson(), write the Gmail special-use lookup once as setLabelSpecialUse(), drop the per-call arrays in two type checks and the Graph text type check, use filter(Boolean) for the Graph recipient lists, and await packUid() inline in getMessageInfo() so only the failure path goes through a helper, without an extra async frame per message.
Every existing test stubbed prepareRawMessage(), so the reference block it shares with queueMessageEntry() had no direct coverage on this side. The new suite runs the real method against stubbed collaborators and pins the whole returned object and the raw MIME, plus the ways it differs from the queueing path: the send-only refusal, a sender defaulted only for a reference action, no flag update or thread id copied onto the reference.
The reference block moves into shared helpers next. Pin what only the queueing path does with it (the connection check before the fetch, the flag update decision, a caller-supplied thread id winning) and the transient-error retry, with and without ignoreMissing, on mocked timers.
…ueMessageEntry() About 210 of the 244 lines that resolve a reply or forward were copies of each other. They now live in resolveReference() and its helpers, with the three things only the queueing path does passed as options: the connection check before the fetch, the flag update decision and copying the thread id. prepareRawMessage() also hands its send-only refusal and its final composer options to helpers. queueMessageEntry() loses only the shared blocks here.
Gmail, Outlook and Mail.ru each carry a copy of the same request code with differences that matter: method casing, header merging, how the body is parsed, which fields an error carries and what is logged. They move into a shared module next, so pin what each one sends, throws and logs first, including key order, since both reach API callers and operators.
Gmail, Outlook and Mail.ru each carried a copy of the same request() code. The shared parts (body and query encoding, the timed fetch, the failure error with its Retry-After, reading the error body and raising its flag, parsing a JSON body read as text) move to lib/oauth/http-request.js. Each client keeps what differs: method casing, header merging, where the token goes, a logContext() of the fields its errors and logs carry, and how it reads a successful answer.
Both carry their own copy of the Graph field list, the header parsing and the formatting, which move into shared helpers next. Pin the request each sends, the 400 and 404 mapping, marking as seen and inline image embedding of the single fetch, and the batch retry with its backoff, per-message errors and the reconciliation of messages a batch answer left out.
…es() The field list, the attachment expansion, the Prefer header and the header parsing existed in both, and the expansion a third time in the listing. They move to lib/email-client/outlook/graph-message.js with the 400/404 mapping, the batch request builder, the retry decision and the reconciliation of missing responses. getMessages() hands the retry loop to postBatchWithRetry(), whose wait can be injected, and the response loop to processBatchResponse(); getMessage() hands inline image embedding to embedInlineAttachments().
…sNew() Both looped over the attachments the html refers to by content id, set the non-enumerable IMAP body part on each and listed the parts to download. collectCidParts() now does it once; processNew() passes reuseLoaded, which keeps content notifyAttachments already loaded and skips oversized parts, the only way the two copies differed.
- getText() and getRawMessage() use bodyPreferHeader() and mapGetMessageError(), the last two copies of the Prefer header and the 400/404 mapping - the Graph $select value is joined once instead of per batch and call - the OAuth2 clients use the shared USER_AGENT for token requests as well - Mail.ru describes a request through its own logContext() like the others - encodeRequest() tests the payload once - fetchReferencedMessage() builds ReferenceNotFound in one place: a permanent 404 falls through to the check that already throws it - parseGraphHeaders() and resolveReference() lose guards that could never fire
…essageEntry() queueMessageEntry() is about to be split into ordered steps. These pin the paths the existing suite did not reach: a missing gateway failing the submission, a base URL with no host falling back to the host name for the List-ID, the preview text being rendered with the other template fields, and the generated Message-ID using the punycode form of an IDN sender domain.
submitMessage() is about to be split into steps. These pin a missing job resolving false without sending, the account penalty after a send that ran into the deadline (and only then), and that every step after the server accepted the message keeps its own failure: a throw there would make the submit worker retry and deliver the message twice.
The header rewriter of getRawEmail() is about to be split into helpers. These pin that an invalid X-EE-Send-At is dropped without replacing the request's send time, and that the Date header follows a future send time of the request but not a past one.
The list unsubscribe headers, the template rendering and the open and click tracking move into their own methods, and the template context, the List-ID domain, the sender domain of a generated Message-ID and the reference part of the response (built twice, for a dry run and a queued message) into module functions. queueMessageEntry() keeps the order of the steps, since each one updates the same data object the next reads, and keeps both early returns.
The SMTP settings, the send itself, the success bookkeeping and the actions after a delivery move into buildSmtpContext(), sendOnce(), recordSubmitSuccess() and runPostSendActions(). Only sendOnce() may throw; every step after it keeps its own try/catch, since a throw once the server accepted the message would make the submit worker deliver it again.
The header rewriter of a raw message becomes extractControlHeaders(), applyAddressHeaders(), buildEnvelope(), ensureStandardHeaders() and applyCustomHeaders(), each returning what the closure used to write into the variables of processMessage(). The closure keeps their order: the envelope is built from the rewritten address headers, and a generated Message-ID uses the envelope sender's domain. The structured path of getRawEmail() becomes buildComposerOptions(), composeMime() and envelopeFromData(), and the returnObject parse of a raw message parseRawToObject().
The post-send helpers read the queue id, Message-ID, envelope and reference from the submission data and the routing from the SMTP context they already get, instead of a separate parameter object. senderDomain() takes the address it uses, buildComposerOptions() reads the send time from the data it is given, and the render condition is checked once instead of per field. The always true baseDomain check in applyListHeaders() goes (listBaseDomain() falls back to localhost), and the Subject override moves out of ensureStandardHeaders() into the rewriter at the same point in the sequence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth of seven PRs in the maintainability campaign. Stacked on #700. Behavior-preserving; the goldens are unchanged.
What changes
queueMessageEntry()is now an ordered list of awaited steps:applyListHeaders,renderTemplateFields, the reference resolvers from refactor: remove duplicated send, OAuth2 and Graph code (MI campaign 2) #700,applyTrackingandbuildReferenceResponse. Both early returns (skipped and dry run) stay.submitMessage()is split intobuildSmtpContext,sendOnce(the only step allowed to throw),recordSubmitSuccessandrunPostSendActions. Every step after the send keeps its own try/catch, so no failure there can cause a redelivery.get-raw-email.js. The header rewriter becomesextractControlHeaders,applyAddressHeaders,buildEnvelope,ensureStandardHeadersandapplyCustomHeaders, in the same order. Each returns what the closure used to write.getRawEmail()becomesparseRawToObject,buildComposerOptions,composeMimeandenvelopeFromData.Tests first
queueMessageEntry: a missing gateway, preview text, the base URL fallback, a punycode sender domain.submitMessage: a missing job, the deadline penalty, and nine subtests. Each makes one post-send step throw and asserts that the call still resolves with the send result and sends exactly once.getRawEmail: an invalidX-EE-Send-At, and the Date update for a future send time.Every test was shown to fail under a deliberate mutation.
Numbers (on top of #700)
queueMessageEntrysubmitMessagegetRawEmailRepository MI went from 46.51 to 46.68, and the low-MI share from 3.29% to 3.08%.
Checks
Merge in its own release cycle, after #700.