[Bug] Python 3.13 Windows proxy worker can crash after evicting and reimporting live Protobuf native modules

[TsuyoshiUshio]

Summary

On Windows, the Python 3.13 proxy worker can exit with 0xC0000005 during indexing/logging after dependency isolation evicts Protobuf modules while retaining worker-generated RPC message classes. Reimporting the same google._upb._message binary and then accessing a retained generated type is sufficient to reproduce a native access violation. Different Protobuf versions, A2A, Durable, gRPC, and the Functions host are not required for the smallest reproduction.

Originally investigated through Azure/azure-functions-agents-runtime#211. The application's earlier physical-versus-editable packaging explanation was superseded by controlled reproductions. This issue tracks the worker/native-module lifetime defect, not an A2A implementation bug.

Still reproducible in the actual host bundled with Core Tools 4.14.0, using an empty Python 3.13 venv without customer Protobuf. Upgrading does improve a different dependency-resolution path; see below.

Environment and release chain

Component	Original	Latest Core Tools comparison (2026-09-09)
Core Tools	4.10.0	4.14.0
Host	4.1048.200.26180	4.1052.200.26352
Proxy worker	4.43.0	4.45.1
V2 library worker	1.1.0	1.1.1
Python	3.13.15, Windows x64	Same
Bundled Protobuf for Python 3.13	5.29.6	5.29.6
Bundled azure-functions for Python 3.13	1.25.0b3	2.2.0b5

The host versions' eng/build/Workers.Python.props specify the worker releases. Because that package reference excludes Windows, the actual downloaded Windows Core Tools distribution and startup logs were also inspected. Worker 4.46.0 is a separate release, not the worker bundled in Core Tools 4.14.0; a 4.46.0 host comparison has not been performed.

Observed failure chain

1. The proxy worker imports its own Protobuf runtime and proxy_worker.protos, retaining generated RPC message classes.
2. DependencyManager.prioritize_customer_dependencies() removes/re-adds the worker dependency path. _remove_module_cache() exempts proxy_worker.*, but evicts google.protobuf.* and google._upb._message resolved from that path.
3. Customer import of google.protobuf.timestamp_pb2 (or the native module itself) can reload the same worker _message.pyd if no separate customer Protobuf wins resolution.
4. The worker still holds its original RPC types. Dispatcher.on_logging() reads protos.RpcLog.Error and the process crashes inside _message.pyd.
5. Host reports the child process failure. Its subsequent Value cannot be null. (Parameter 'provider') is secondary, not evidence of a model-provider configuration error.

Relevant tagged sources: dependency.py, dispatcher.py.

The original application also encountered ImportError: cannot import name 'df_dumps' from 'azure.functions._durable_functions': new Durable dependencies were resolved against an older worker SDK. Logging that ordinary Python exception exposed the secondary native crash. SDK selection and native-module lifetime are separate problems.

Reproduction A: actual worker cache transition, without host

Run each case in a disposable subprocess: the failing case intentionally terminates Python. Do not run this in a live worker or shared interpreter. No installed package modifications are required.

Save as worker_cache_repro.py:

import argparse
import faulthandler
import importlib
import sys

parser = argparse.ArgumentParser()
parser.add_argument("worker_directory")
parser.add_argument("--clear", action="store_true")
args = parser.parse_args()
faulthandler.enable(all_threads=True)
sys.path.insert(0, args.worker_directory)

from proxy_worker import protos
from proxy_worker.utils.dependency import DependencyManager

print("Retained RpcLog class:", protos.RpcLog, flush=True)
print("Native before:", sys.modules["google._upb._message"].__file__, flush=True)
if args.clear:
    DependencyManager._remove_from_sys_path(args.worker_directory)
    DependencyManager._add_to_sys_path(args.worker_directory, True)
    print("Native evicted:", "google._upb._message" not in sys.modules, flush=True)

importlib.import_module("google")
module = importlib.import_module("google._upb._message")
print("Native after:", module.__file__, flush=True)
print("Reading retained RpcLog.Error", flush=True)
print("RpcLog.Error =", protos.RpcLog.Error, flush=True)

PowerShell, using the Python 3.13 bundle from an extracted Core Tools ZIP:

$worker = 'C:\path\to\core-tools\workers\python\3.13\WINDOWS\X64'
python -I .\worker_cache_repro.py $worker
# Control: RpcLog.Error = 4; exit 0
python -I .\worker_cache_repro.py $worker --clear
$LASTEXITCODE
# Failing released worker: -1073741819 (0xC0000005)

On both inspected worker releases, the failing case reports native module eviction and the same native file before/after, then faults when reading the enum.

Reproduction B: reduce further to Protobuf only

In the same isolated Python process setup, prepend a directory containing one tested Protobuf distribution to sys.path, then execute:

import faulthandler
import importlib
import sys
from google.protobuf import descriptor_pb2

faulthandler.enable(all_threads=True)
message_type = descriptor_pb2.FieldDescriptorProto
print(sys.modules["google._upb._message"].__file__, flush=True)
del sys.modules["google._upb._message"]
importlib.import_module("google._upb._message")
print(message_type.TYPE_DOUBLE, flush=True)

Omitting the delete/reimport yields 1. The failing variant also reproduces with python -I and while keeping an extra strong reference to the original native module.

Python	Protobuf	Control	Delete/reimport
3.13.15	5.29.6	Exit 0	Native access violation
3.13.15	6.33.6	Exit 0	Native access violation
3.14.0	6.33.5	Exit 0	Native access violation

The Python 3.14 row is a standalone native reproduction, not a claim that every 3.14 Functions app crashes. It also does not imply arbitrary native-module unloading must be supported by Protobuf; the worker must avoid invalidating its own live objects regardless.

Actual host observations and controls

A minimal HTTP application imports google.protobuf.timestamp_pb2 at module scope and calls logging.error(...) before defining its route. No A2A, Durable, model connection, or customer Protobuf installation is needed in the empty-venv case.

Scenario	Result
4.10.0, ordinary global interpreter, no PYTHONPATH override	Native crash; Protobuf resolves to worker bundle
4.14.0, same app/interpreter environment	HTTP 200; Protobuf resolves to global customer site-packages
4.14.0, empty venv, no customer Protobuf or PYTHONPATH override	Native crash; Protobuf resolves to worker bundle
Earlier baseline app: logging without native-loading import	Succeeds (two runs)
Earlier timestamp import + logging	Crashes (two runs)
Earlier plain import google.protobuf + logging	Succeeds
Earlier PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python control	No native crash

For the empty-venv case, create a venv with python -m venv --without-pip, activate/select its interpreter, and point Core Tools at a minimal HTTP app. Keep PYTHONPATH and PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION unset. Confirm the selected interpreter and module paths in diagnostics; merely having a venv directory is insufficient.

A native debugger identified a null-pointer read at address 0x18 inside _message.pyd for the 5.29.6 reproduction. No private symbols were available, so no exact internal C function/source line is asserted.

Upgrade / workaround status

• Core Tools 4.14.0 contains fix: fix default cx deps path for 3.13 #1833, which prioritizes local interpreter site-packages when the customer path was not otherwise found. This avoids the original symptom when a compatible customer dependency closure exists. It does not fix the empty-venv native fallback case.
• With 4.14.0, the A2A application factory returned Agent Card HTTP 200 both in the existing global environment and in a separate external venv exposing existing physical non-editable dependencies, without PYTHONPATH/source injection. The latter used a directory junction to an existing package target, not a fresh pip-install run. No model call or Durable workflow execution was performed in this comparison.
• Selecting pure-Python Protobuf avoided the native crash in a diagnostic control, but is process-wide, has unmeasured performance/compatibility impact, and does not independently resolve the SDK mismatch. Treat as a temporary diagnostic option, not a qualified application fix.
• Do not patch shared Core Tools files or ship arbitrary dependency pins as the solution.

Candidate permanent fix (proposal, not validated implementation)

Maintain one coherent worker-owned Protobuf family for the entire lifetime of worker RPC objects while allowing customer packages to resolve independently.

A candidate to prototype is a private, worker-owned pure-Python Protobuf runtime plus generated RPC/well-known types under the proxy worker namespace. Rewrite static/dynamic imports consistently, choose the implementation only inside that private family, and keep it out of customer cache eviction. Preserve the same family across the proxy-to-library protos interface, RPC serialization, descriptors, and callbacks. Do not alias it into public google.protobuf, shadow the google namespace, or force the customer's implementation via a process-wide environment variable.

Measure RPC serialization throughput, large-payload latency, CPU, and memory before approving pure Python. Performance budgets require maintainer agreement. If unsuitable, consider genuinely independent native-module state/build isolation or a larger process boundary. Renaming/copying a .pyd alone does not prove isolation.

Prior art #1873/#1876 applies to Python <=3.12, not this 3.13 proxy path. In particular, public aliases/process-wide settings in that fallback should not be copied blindly into a customer-independent design.

Experiments in an isolated worker copy explain why smaller patches are insufficient: preserving only the native module produced Python Message identity errors; preserving the entire public Protobuf family avoided the crash but broke customer Protobuf 6 generated code against worker runtime 5.29.6. Conditional preservation based on file existence was diagnostic only, not a portable fix.

Related #1906 concerns a different asymmetric namespace eviction/AttributeError in the classic worker. Regression coverage should include that namespace integrity constraint; purging an entire family is not safe if live worker native types remain.

Acceptance Criteria

• A released-worker regression demonstrates the native fault in a bounded subprocess/actual host, without A2A, Durable, cloud credentials, or editable imports.
• On the corrected worker, dependency switching never invalidates the Protobuf family backing retained worker RPC objects. Indexing, logging, enum/descriptor access, and RPC serialization succeed in the formerly failing host scenario. Tests should assert the lifetime invariant, not require a particular vendoring mechanism.
• Cover no customer Protobuf, same-version customer Protobuf, newer customer runtime/gencode, and explicitly selected customer pure-Python implementation. A base HTTP app must not need to install Protobuf just to prevent a worker crash.
• Customer public package resolution and native implementation remain independent; no Message identity/descriptor-pool/version errors, and no google.cloud/google.api_core namespace regression.
• App-local packages, external venvs, explicit customer paths, startup, specialization, and reload are covered across supported Python versions/platforms. Windows 3.13 is mandatory for this regression; do not claim Linux/macOS reproduction until tested.
• Ordinary application import/configuration errors are reported with useful diagnostics instead of native exit or silent empty indexing. Track incompatible Functions SDK selection separately.
• Proxy/library integration, base HTTP, physical-wheel A2A startup/request execution, and Durable workflow behavior are exercised; preserve the host wire contract.
• Benchmark the chosen implementation and meet maintainer-agreed resource/latency budgets before release.
• Validate packaged worker artifacts and the consuming Host/Core Tools release, not just an editable checkout.

The standalone Protobuf-only delete/reimport reproducer may remain an upstream Protobuf concern even if the worker stops taking that unsafe path. Fixing the worker does not require promising arbitrary customer sys.modules manipulation is safe.

Suggested handoff sequence

1. Reproduce against the exact 4.45.1 bundle and read the tagged dependency-manager/proxy-library interfaces.
2. Add focused subprocess regressions; retain the original reproducer as the failing baseline.
3. Review/prototype coherent private isolation in a dedicated worker worktree, not by modifying the shared CLI installation.
4. Resolve SDK selection as a separate track, then expand lifecycle/platform/performance coverage.
5. Coordinate release integration and update Windows Python worker startup crash: Protobuf cache eviction, SDK selection, and Core Tools upgrade mitigation azure-functions-agents-runtime#211 with the first qualified fixed release.

No permanent worker fix has been implemented or benchmarked yet.

[TsuyoshiUshio]

Alternatives analysis: this is not simply a gRPC version mismatch

Follow-up investigation, 2026-09-09. This supplements the diagnosis and broadens the solution space; the private pure-Python worker design is a candidate, not a requirement to implement that one mechanism.

Three different layers

Layer	Current behavior
Functions Host <-> Python worker	Worker-owned gRPC and generated RPC types
External A2A caller <-> P3 application	JSON-RPC over HTTP, not gRPC
A2A SDK 1.x internal message representation	Protobuf-generated classes, including on the HTTP path

Removing sys.modules entries does not retire all live generated types or reliably unload a native DLL. Reimport can occur while old worker objects remain alive. The same-file, same-version Protobuf-only reproduction is why version skew cannot be the full explanation. A2A does not have to require a newer gRPC for this native failure to occur.

Dependency facts

• a2a-sdk[http-server]==1.1.2 requires protobuf >=5.29.5,<7. grpcio is in its optional grpc/all extras (>=1.60), not the base/HTTP-server requirements. We already select HTTP-server.
• Installed a2a.types imports a2a_pb2; the P3 registration code also uses google.protobuf.json_format.MessageToDict. Simply choosing HTTP or removing a gRPC extra does not remove Protobuf.
• azure-functions-durable==2.0.0b2 -> durabletask[opentelemetry]>=1.9.0. Installed durabletask 1.10.1 unconditionally requires grpcio and protobuf; its gRPC generated-code version check names 1.65.4, below worker grpcio 1.70.0. This does not establish full compatibility, but the resolved application grpcio 1.83.1 is not shown to be an A2A-specific minimum.
• Installed google-api-core==2.35.0 and googleapis-common-protos==1.75.2 require protobuf >=6.33.5. These transitive constraints explain the Protobuf 6 floor in that closure.
• agent-framework-hosting-a2a==1.0.0a260730 requires a2a-sdk >=1,<2.

Alternatives and tradeoffs

Option	Assessment
Upgrade Core Tools / customer dependency resolution	Observed mitigation: 4.14.0 starts the tested physical-package A2A factory without PYTHONPATH injection. Empty-venv native fallback still crashes.
Upgrade worker grpcio/Protobuf and qualify regenerated RPC code	Sensible compatibility maintenance, but not sufficient by itself: same-version Protobuf 5.29.6 and 6.x already reproduce. A future release with a demonstrated native-lifetime fix would be different; no such fixed version is established here.
Align customer dependency versions with worker versions	Resolver-feasible candidate, described below. Not a native-lifetime fix; requires actual host/request/workflow and support/security qualification.
Preserve a coherent public worker Protobuf family, explicitly validate compatibility	Possible constrained design, not an unconditional fix. Earlier preservation broke newer customer generated code against the older worker runtime. Any rejection must be explicit, not silently override customer semantics.
Private worker Protobuf family	Correctness-oriented candidate. Pure Python is one prototype; genuinely independent native state/build isolation is another. Benchmark and audit proxy/library interfaces. Copying/aliasing a .pyd alone is not proven isolation.
Fix native-module reinitialization in upstream Protobuf	Potential fundamental native fix; the small reproducer is available. Exact C-level solution is still unknown, and worker Python/native-family consistency also needs qualification.
Explicit process-wide pure-Python Protobuf	Avoided the crash in a diagnostic control. Global impact/performance and the separate SDK mismatch remain. Opt-in temporary measure only, not a silent worker override.
Downgrade A2A to 1.0.x	1.0.0 still requires Protobuf/Google API dependencies; trigger removal is not demonstrated. Requires API/bug-fix compatibility review.
Downgrade A2A to 0.3.x	Not drop-in: current MAF hosting excludes it, and the protocol/API/model representation changes. 0.3.0 pins protobuf 5.29.5; 0.3.26 still requires protobuf >=5.29.5. An older matching hosting/client stack would need qualification.
Build a narrow HTTP/JSON-only A2A adapter or decouple SDK types upstream	Plausible application-level trigger reduction. Replace the present Protobuf/MAF A2A adapter hooks while retaining MAF execution where possible. Must preserve A2A 1.0 negotiation, JSON mapping, errors, auth, limits, and conformance; explicit design approval required.
Replace OpenAI/Google model SDK	Different layer: not a substitute for the A2A endpoint. Current runtime already uses OpenAI through MAF. Default OpenAI/Google GenAI metadata has no grpcio/Protobuf requirement, but leaving A2A/Durable in place retains those paths.
Switch to Google ADK A2A	Does not independently remove this dependency family: google-adk 2.8.0's a2a extra depends on a2a-sdk[http-server]>=0.3.4,<2.
Separate A2A/execution service or process	Can avoid the shared Functions-interpreter collision for moved code. Larger operational cost: identity, deployment, cancellation, tracing, scaling, and failure handling. Remaining Functions-side Protobuf paths still matter.

Making Durable optional and lazy is also useful: it removes a real grpcio/Protobuf dependency path for non-Durable apps. It does not remove A2A 1.x's own Protobuf usage. Combining optional Durable with a genuinely JSON-only adapter could remove more application-side triggers, but neither fixes a generic worker lifetime defect.

Concrete alignment feasibility result (not recommended pins)

A Python 3.13 pip dry-run, without installing packages, resolved the selected A2A 1.1.2 / MAF hosting 1.0.0a260730 / core 1.13.0 / OpenAI 1.10.2 / Foundry 1.10.3 / Durable 2.0.0b2 tuple with:

protobuf==5.29.6
grpcio==1.70.0
google-api-core==2.24.2
googleapis-common-protos==1.70.0

Pip selected durabletask 1.10.1 and proto-plus 1.28.2. This shows the selected tuple does not intrinsically require grpcio 1.83.1 or Protobuf 6. Pinning only common-protos would be insufficient if google-api-core 2.35.0 remained selected.

This is dependency-resolution evidence only, not a complete application/dev/monitor lock, runtime test, security approval, or release recommendation. No packages/project manifests were changed. A separate customer binary may avoid the same-file reimport path, but version equality itself does not prove that; the same-version native reproduction remains the required negative control.

Additional acceptance / decision criteria

• Compare viable lifetime/isolation alternatives rather than making vendoring the required implementation.
• Document supported version combinations and explicit rejection behavior if choosing a constrained shared-family design.
• Qualify aligned-version candidates in actual host startup, RPC serialization, A2A requests, and Durable workflows; keep resolver success distinct from runtime success.
• Do not silently change A2A protocol version, MAF integration, or model provider to conceal a worker defect.
• Benchmark the chosen native/pure-Python/process-isolation approach and preserve independent customer dependencies.

Suggested order: qualify the current Core Tools mitigation first; use aligned constraints only as a bounded fallback experiment; pursue a worker lifetime/isolation fix independently. Consider custom adapters or service separation only if their maintenance/operational costs are justified.

Further sources: A2A 1.0 migration guide, A2A 0.3.26 metadata, Google API Core 2.24.2, common-protos 1.70.0, Google ADK 2.8.0, Google GenAI 2.22.0, OpenAI 2.54.0.

[hallvictoria]

Hey Tsuyoshi Ushio (@TsuyoshiUshio), thanks for reporting.

As noted, there was a fix in identifying local customer package paths present in Core Tools 4.14.0. That's why the original A2A/workflow app failed in Core Tools v4.10.0 but indexed & executed successfully in Core Tools v4.14.0 (as per the table on the linked AFAR issue).

Regarding the empty virtual environment + Core Tools 4.14.0 scenario:
The main issue is that the example function app is importing a generated module from protobuf via:
from google.protobuf import timestamp_pb2

I would consider this an app error. If the app imports protobuf, it needs to bring in protobuf as a dependency - ie, the virtual environment should not be empty. It's not a best practice or recommended to implicitly take a dependency on the worker's internal packages. This specific scenario should be comparable to a ModuleNotFoundError or ImportError instead.

That being said, I agree that a crash is not the expected outcome here. I'll keep this issue open for now while we iterate and see if we want to improve the behavior or just consider it an unsupported scenario. This shouldn't be a blocker to your work though, as you mentioned that the A2A app + Core Tools v4.14.0 was successful.