Repository navigation
fix(lakebase): deliver getLakebaseOrmConfig() pool settings to TypeORM and Sequelize - #731
billmcmillin wants to merge 4 commits into
Conversation
… options getLakebasePgConfig() built the pg.Pool config from a fixed list of 11 keys, so maxLifetimeSeconds, application_name, keepAlive, query_timeout and options were silently dropped. Without a lifetime, pool connections were never recycled. Access granted through Unity Catalog claims in a Lakebase OAuth token stops working 4 hours after the token is minted while the connection stays open, so a busy pool kept failing with "permission denied" until the app restarted. - Pass the five options through to pg.Pool, keeping the explicit key list. - Default maxLifetimeSeconds to 1800 (30 minutes); an explicit 0 is kept. - Warn when maxLifetimeSeconds is 0 or above 9000 (2.5 hours): the 4-hour lapse, less up to an hour of token age at login, less a 30-minute margin. - Ignore a negative or non-finite maxLifetimeSeconds, with a warning, and use the default. pg-pool would otherwise close every connection as soon as it opens (negative) or never recycle (NaN). - Document maxLifetimeSeconds, including that held clients are only recycled on release, and regenerate the API docs. - Add unit tests for the default, the warnings and the pass-through. - Add tools/lakebase-soak.ts, a soak harness that runs a real pool against a sandbox endpoint and checks connection age, token age at login, application_name and table access. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Bill McMillin <bill.mcmillin@databricks.com>
pg-pool passes maxLifetimeSeconds * 1000 to setTimeout. Above 2^31 - 1 ms (about 24.8 days), Node fires the timer after 1 ms, so every connection closed after a single use. Treat those values like Infinity: warn and use the default. Compare in milliseconds so 2147483 s is kept and fractional values just past the limit are rejected. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Bill McMillin <bill.mcmillin@databricks.com>
…M and Sequelize getLakebaseOrmConfig() returned pool and client settings at the top level, where neither ORM reads them, so TypeORM and Sequelize users silently lost connection recycling, the pool size and timeouts, and (Sequelize) SSL. - TypeORM: pool size, connect timeout and application name go under its own poolSize, connectTimeoutMS and applicationName, so caller values still win; lifetime, idle and statement timeouts go under extra. - Sequelize: SSL and client timeouts go under dialectOptions, pool size and idle under pool. Its pool has no lifetime option, so connections are recycled by age with pool.validate and an afterConnect hook. - Log only when the caller passes a logger, so nothing new is printed by default. Fixes databricks#730 Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Bill McMillin <bill.mcmillin@databricks.com>
Live check against a real Lakebase endpointI ran TypeORM 0.3.28 and Sequelize 6.37.7 against a Lakebase Autoscaling sandbox endpoint as a service principal. I used Every 5 seconds for 6 minutes, each run asked the server about the connection its query ran on: SELECT a.pid, a.backend_start, a.application_name, s.ssl,
current_setting('statement_timeout') AS statement_timeout, now()
FROM pg_stat_activity a JOIN pg_stat_ssl s USING (pid)
WHERE a.pid = pg_backend_pid();At the end, each run tried
This checks that the settings reach the server; it doesn't reproduce the 4-hour Unity Catalog access lapse. That lapse only affects tokens minted with claims, which |
Fixes #730.
Stacked on #719. This branch is based on #719, so the diff below also shows #719's two commits (
0ab349f3,888448ff) until it merges. Only the last commit,2e44157b, belongs to this PR.Problem
getLakebaseOrmConfig()is documented for TypeORM and Sequelize, but it returned pool and client settings at the top level, where neither ORM reads them. Unknown keys are silently ignored, so TypeORM and Sequelize users got:42501 permission denied). fix(lakebase): recycle pool connections and pass through dropped pool options #719 fixes this forpg.Poolusers; this PR fixes it for ORM users.statement_timeout,query_timeoutand the other client timeouts were ignored.sslonly fromdialectOptions, sosslModewas ignored. Connections used TLS only ifPGSSLMODEhappened to be set in the environment.Why each ORM ignored the settings, from their source:
PostgresDriver.js:1185-1196) buildspg.Pooloptions from its own fields plusextra, and ignores top-level pg options.user,password,host,databaseandporttopg.Clientfrom the top level; everything else comes fromdialectOptions(postgres/connection-manager.js:72-96). Its pool, sequelize-pool, has no lifetime option.Fix
getLakebaseOrmConfig()now also returns each setting where each ORM reads it. The existing top-level keys are kept, so current callers don't break.poolSizepool.maxconnectTimeoutMSdialectOptions.connectionTimeoutMillisapplicationNamedialectOptions.application_nameextra.maxLifetimeSecondspool.validate+hooks.afterConnect(age check)extra.idleTimeoutMillispool.idlekeepAlive;optionsextra.*dialectOptions.*ssl(unchanged)dialectOptions.sslextralast, over its own options. Pool size, connect timeout and application name therefore go under TypeORM's own names, so a caller'spoolSizeorconnectTimeoutMSset after the spread still wins.afterConnectrecords when each connection opens. sequelize-pool callsvalidatebefore reusing an idle connection, and a connection older than the lifetime is destroyed and replaced.validatealso repeats Sequelize's built-in health check (_invalid/_ending), which a customvalidatereplaces. As with pg-pool, a connection that's checked out isn't closed until it's released.extra,pool,dialectOptionsorhooksmust merge with the returned values; the README and JSDoc show how.hooksis replaced, lifetimes count from a connection's first reuse rather than when it opened, and a warning is logged once.logger, so nothing new is printed by default. With a logger configured, fix(lakebase): recycle pool connections and pass through dropped pool options #719's lifetime warnings now reach ORM users too.getLakebaseOrmConfig, which it used without importing.Proof
Before and after against a real Postgres server
I ran
embedded-postgreswith the real TypeORM 0.3.28 and Sequelize 6.37.7, using@databricks/lakebasebuilt from #719's tip (before) and from this branch (after).maxLifetimeSeconds: 2, idle 3 s, thenpg_backend_pid()statement_timeout: 500, thenpg_sleep(1)sslMode: "require"against a server without TLSpoolSize: 20, connectTimeoutMS: 2000max=20, 2000→max=20, 2000sslMode: "disable", create/insert/selectTests
packages/lakebase/src/__tests__/orm-config.test.tshas 15 tests. They run the real ORMs against a fakepgmodule, which records what the ORM passes topg.Pool/pg.Client, so no database is needed.pg.Pool;poolSize/connectTimeoutMS/applicationNamewin;extrakeeps both sides.sslModeand the client timeouts reachpg.Client;hooksstill recycle and warn once;consolewithout alogger; warnings and token errors reach a configured logger.Against the original
getLakebaseOrmConfig(), 12 of these 15 tests fail. The other three guard the new code: lifetime 0, invalid connections, and logging with a configured logger.typeormandsequelizeare added as dev dependencies of@databricks/lakebase, pinned to the versionsdev-playgroundalready uses. They're imported only by this test file, so the published package is unchanged and the lockfile gains no new packages.Checks run locally:
pnpm build,pnpm docs:build,pnpm -r typecheck,pnpm checkandpnpm knippass.testing/tests/mock-workspace-client.test.ts("formatting and structural equality neither throw nor recurse"), and it fails the same way without this change.Behavior changes for Sequelize users
These are the intended effect of the fix, but they change what existing Sequelize apps get:
max10 (was 5), idle 30 s (was 10 s), and a 10 s connect timeout (was none). Expect up to twice as many connections per app instance.sslMode. The default,require, means verified TLS. A local Postgres without TLS now needssslMode: "disable"(orPGSSLMODE=disable); before, it worked without TLS because the setting was ignored.PGSSLMODE=preferno longer verifies certificates on Sequelize. It now maps to{ rejectUnauthorized: false }, the same as forpg.Pooland TypeORM users. Before,pg's ownPGSSLMODEhandling verified them. Flagging this in case maintainers wantpreferto verify on this path.Unaffected:
createLakebasePool,getLakebasePgConfig, Drizzle, Prisma and rawpg, which use the unchangedpg.Poolpath.Known limitations
getOrmConfig()rebuilds the config on every call, so the lifetime warning repeats per call. This only applies when a logger is configured.getLakebaseOrmConfigshows internal variable names as default values (for example= pgConfig.max). That predates this PR (username = user); fixing it needs an exported return type.This pull request and its description were written by Isaac.