[pffigueiredo]

Why

A chart on the Arrow path fails with HTTP 500 "Query execution failed" when workspace storage refuses the app server's Cloud Fetch download. One example is a storage firewall that doesn't allow the serverless network the app runs on (Jira LA-2169). On pods with internet off, a blocked host can hang instead, until the route's 120 s timer returns a 503 that blames the warehouse. With this PR, the chart renders in both cases.

What happens now

A Cloud Fetch download that fails before its first byte falls back to an inline JSON_ARRAY run, rebuilt as Arrow. Each warehouse remembers what worked.

flowchart TD
  Q["Arrow chart query"] --> M{"Warehouse memo"}
  M -- "links-denied" --> I1["INLINE + JSON_ARRAY, rebuilt as Arrow"]
  I1 -- "fits" --> OK1["Serve"]
  I1 -- "over 25 MiB" --> CF
  M -- "inline / unknown" --> P["INLINE + ARROW_STREAM probe"]
  P -- "accepted (Reyden)" --> OK2["Serve, remember inline"]
  P -- "rejected" --> CF
  M -- "external" --> CF["EXTERNAL_LINKS: download the first chunk"]
  CF -- "first byte" --> OK3["Serve, remember external"]
  CF -- "fails before the first byte" --> F{"Fits inline?"}
  F -- "yes" --> I2["INLINE + JSON_ARRAY, rebuilt as Arrow"]
  I2 -- "fits" --> OK4["Serve"]
  I2 -- "over 25 MiB" --> R
  F -- "no" --> R{"Refused?"}
  R -- "yes" --> D["EXTERNAL_LINKS_DENIED"]
  R -- "no" --> RT["Cloud Fetch once more, full header wait"]
  RT -- "first byte" --> OK5["Serve"]
  RT -- "refused or no answer" --> D
  RT -- "other failure" --> E["That failure's own error"]

Loading

• Refused means a 400 or 403 after one re-mint. It's remembered as "links-denied" straight away.
• No answer means no response headers, or no connection, within 10 s. It's remembered once the inline result has been served, or once the retry also fails.
• "links-denied" lasts 10 minutes, then reads as "external", so Cloud Fetch is tried again.

The download

for each chunk
  header wait = first chunk ? 10 s (30 s on the retry) : 30 s
  GET with the link's httpHeaders
    400 or 403   → re-mint the link once, retry at once; then the failure is "refused"
    other 4xx    → stop (408 and 429 retry)
    no headers, or Node's 10 s connect timeout
                 → first chunk: stop as "no answer"; later chunks: retry
    5xx, network → retry after 1 s, then 2 s
  stream the body; every read must arrive within 30 s

Only the first chunk gets the short wait, because only a failure before the first byte can still fall back. Every failure carries EXTERNAL_LINK_DOWNLOAD_FAILED.

Where it lives

AnalyticsPlugin._handleArrowStreamQuery          plugins/analytics/analytics.ts
  ArrowCapabilityMemo.get / set                  plugins/analytics/result-delivery.ts
  deliverArrowBytesWithFallback                  plugins/analytics/result-delivery.ts
    deliverArrowBytes          INLINE probe, then EXTERNAL_LINKS
      ArrowStreamProcessor.streamChunks          stream/arrow-stream-processor.ts
    inlineJsonAsArrow          treats "Inline byte limit exceeded" as too large
      SQLWarehouseConnector.executeStatement({ asArrow: true })
                                                 connectors/sql-warehouse/client.ts
        _allInlineRows         follows nextChunkIndex with getResultData
        buildArrowIPCFromRows                    connectors/sql-warehouse/arrow-schema.ts
          parseJsonArrayCell                     connectors/sql-warehouse/json-array-cells.ts

• The rebuild. The connector returns the rebuilt Arrow as an attachment, like an INLINE + ARROW_STREAM result. So the route and @databricks/appkit-ui don't change.
• Cell parsing. One codec table in json-array-cells.ts reads every cell. It uses the platform's parsers, like the Databricks Python connector. A cell that doesn't fit its column type throws ARROW_REBUILD_FAILED.
• Caching. The route caches every INLINE request, keyed by its format parameters.

Rebuilt Arrow vs. native

This was checked against a captured warehouse response (tests/fixtures/typed-values.json: 17 columns, nulls in every column). The rebuild matches native Arrow in 14 columns, names and nulls included. The other 3 are known limits:

- ts        Timestamp<MICROSECOND, America/Los_Angeles>
+ ts        Timestamp<MICROSECOND, UTC>      same instants; appkit-ui doesn't read the label
- ts_micro  1767290400123.456
+ ts_micro  1767290400123                    JSON_ARRAY text carries milliseconds only
- arr       List<Int64>  [0, 1]
+ arr       Utf8         ["0","1"]           nested, interval and geospatial columns arrive as text

Tradeoffs

• A slow store can be skipped. A download that never answers is remembered as denied, because blocked hosts on internet-off pods hang instead of refusing. App Builder pods have no SEG, so their storage traffic doesn't go through the node proxy. The cost is that a slow but healthy store can be skipped for up to 10 minutes. Small results still render inline, and large ones still go back to Cloud Fetch.
• One new limit on the healthy path. The bytes and executions are unchanged, apart from the 10 s wait for the first chunk's headers.

Scope

• Covered: refused, reset, stalled and unanswered downloads that fail before any byte reaches the browser.
• Not covered: a chunk that fails after bytes were sent.
• Results over 25 MiB under a refusal still fail, but with EXTERNAL_LINKS_DENIED and a message that names the storage download.

Verification

This ran on a real SQL warehouse, using an app rendered from template/ and viewed in headless Chrome. A preload answered the storage downloads with 403, 400, or no response.

Storage answers	0.89.0	This branch	Next load
normally	renders	renders via Cloud Fetch
403	"Query execution failed"	renders in 2.9 s	3 ms
400	"Query execution failed"	renders in 2.8 s	3 ms
never	503 after 120 s	renders in 12.9 s	3 ms

• Unroutable storage host, result too large for inline: fails in 20 s with EXTERNAL_LINKS_DENIED and the storage message. Node's own connect timeout counts as no answer, so the full-wait retry doesn't fall back to three network retries.
• Multi-chunk inline result: with storage answering 403, a 10,000-row result that the warehouse returns in 2 inline chunks (10 MB of Arrow) is served whole.
• Tests: vitest run packages/appkit passes 4372, with 1 skipped. Typecheck, pnpm check and knip are clean.

On the App Builder

I ran the builder's own evals, headless, on a local App Builder stack where the workspace storage host resolves to an unroutable address. That's what a pod with internet off and no SEG sees: Cloud Fetch downloads never answer. The prompt asked for a chart of samples.nyctaxi.trips daily trips and average fare, fetched with format "arrow_stream". The builder's template pins AppKit 0.90.0, so the patched runs used 0.90.0 plus this PR's source changes, with the same dependencies; this PR's tests pass on that base too.

AppKit	Runs	Chart in the preview	Time	Screenshots
0.90.0	1	Never loaded. The agent told the user it would work once deployed.	11.2 min	5
0.90.0 + this PR	3	Rendered every time, still on arrow_stream	3.2–8.3 min	1–3

The builder preview at the end of each run:

AppKit 0.90.0	AppKit 0.90.0 + the fix

The 0.90.0 run logged Failed to download chunk 0 after 3 attempts: fetch failed six times. A patched run's dev server logged:

[appkit:analytics:delivery] ARROW_STREAM INLINE rejected by warehouse, falling back to EXTERNAL_LINKS: …
[appkit:analytics:delivery] Cloud Fetch download failed (unanswered), serving the result inline: Failed to download chunk 0: no response within 10000ms

In the slowest patched run, the agent's first screenshot landed during the 10 s header wait and showed the loading skeleton. Its next screenshot showed the chart. A date-range dashboard prompt was also run on both versions, but the agent fetched it with useAnalyticsQuery as JSON, so it never touched Cloud Fetch.

[github-actions]

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+12 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	1.3 MB (+14 KB)	467 KB (+4.9 KB)
Type declarations	468 KB (-41 B)	169 KB (-20 B)
Source maps	2.6 MB (+26 KB)	871 KB (+8.0 KB)
Other	11 KB	3.7 KB
Total	4.3 MB (+40 KB)	1.5 MB (+13 KB)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
.	110 KB (+2.4 KB)	2.5 KB (+1 B)	112 KB (+2.4 KB)	external	354 KB (+6.3 KB)
./beta	105 KB (+1.9 KB)	485 B	106 KB (+1.9 KB)	external	316 KB (+4.7 KB)
./testing	46 KB	33 KB (-4 B)	79 KB (-4 B)	external	229 KB
./tsdown	520 B	0 B	520 B	external	813 B
./type-generator	26 KB	0 B	26 KB	external	74 KB

Chunks:

Entry	Chunk	Load	Size (gz)
.	index.js	initial	105 KB
.	utils.js	initial	4.6 KB
.	remote-tunnel-manager.js	lazy	2.5 KB
./beta	beta.js	initial	85 KB
./beta	modular.js	initial	6.0 KB
./beta	stream-manager.js	initial	5.9 KB
./beta	service-context.js	initial	4.3 KB
./beta	databricks.js	initial	3.3 KB
./beta	client.js	initial	649 B
./beta	index.js	initial	20 B
./beta	supervisor-api.js	lazy	193 B
./beta	databricks.js	lazy	177 B
./beta	index.js	lazy	115 B
./testing	manifest.js	initial	33 KB
./testing	index.js	initial	10 KB
./testing	wide-event-emitter.js	initial	2.9 KB
./testing	index.js	lazy	29 KB
./testing	remote-tunnel-manager.js	lazy	2.5 KB
./testing	utils.js	lazy	1.8 KB
./tsdown	index.js	initial	520 B
./type-generator	index.js	initial	26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	429 KB	147 KB
Type declarations	267 KB	98 KB
Source maps	848 KB	285 KB
CSS	16 KB	3.2 KB
Total	1.5 MB	533 KB

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
./js	5.3 KB	49 KB	55 KB	208 KB	14 KB
./js/beta	2.1 KB	0 B	2.1 KB	0 B	5.0 KB
./react	440 KB	49 KB	489 KB	1.4 MB	177 KB
./react/beta	4.9 KB	0 B	4.9 KB	0 B	12 KB

Chunks:

Entry	Chunk	Load	Size (gz)
./js	index.js	initial	5.2 KB
./js	chunk	initial	120 B
./js	apache-arrow	lazy	49 KB
./js/beta	beta.js	initial	2.1 KB
./react	index.js	initial	438 KB
./react	tslib	initial	2.1 KB
./react	apache-arrow	lazy	49 KB
./react/beta	beta.js	initial	4.9 KB

[github-actions]

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 42998813156 -R databricks/appkit -n appkit-template-0.92.0-pr.e8d277a-fix-la-2169-cloud-fetch-fallback-723 -D appkit-pr-723 \
  && unzip -o "appkit-pr-723/appkit-template-0.92.0-pr.e8d277a-fix-la-2169-cloud-fetch-fallback-723.zip" -d "appkit-pr-723" \
  && databricks apps init --template "appkit-pr-723"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.