Keep Workflow Streams publishing after transient errors

[1fanwang]

After one transient signal error, a long-lived Workflow Streams publisher stopped sending later items. Delivery resumed only when callers flushed explicitly or exited the client context.

The flusher now retries service RPC statuses that SDK Core classifies as retryable, so buffered items resume delivery on the next interval without caller action. Payload conversion failures, non-retryable RPC statuses, oversized-message errors, and an expired retry window still propagate. The pending batch remains ordered ahead of newly buffered items.

Testing

uv run pytest tests/contrib/workflow_streams/test_workflow_streams.py -q \
  -k 'background_flusher_retries_failed_signal or background_flusher_propagates_payload_codec_error or background_flusher_propagates_message_too_large'
uv run pytest tests/contrib/workflow_streams/test_workflow_streams.py -q

Raw logs

# d0e075c1e25b3371f0e0f5116d9c6626892e3eab
# First signal raises RPCStatusCode.UNAVAILABLE
RuntimeError: simulated delivery failure
1 failed

# Current branch
The preserved batch was delivered on the next interval.

# 4eb9b5c3ebb60e2a63eb589b93b977f4cd0db2bf
# PayloadCodec.encode raises before the RPC
TimeoutError
1 failed

# Current branch
3 passed
42 deselected

# Workflow Streams suite
45 passed

[brianstrauch]

Two findings from my review:

1. Please avoid classifying every signal-path exception as retryable.

   _pending is set before WorkflowHandle.signal(), but signal() performs the envelope DataConverter.encode step—including payload codecs and external storage—before issuing the RPC. Consequently, a codec, configuration, external-storage, or interceptor error occurs while _pending is non-None, and the new catch suppresses it and retries until max_retry_duration. With the default settings, this hides the original diagnostic for ten minutes and ultimately replaces it with TimeoutError.

   I reproduced this with a PayloadCodec.encode() implementation that raises: the background flusher remained pending on this PR's head, while the same probe propagated the error with the prior implementation. Could the suppression be restricted to genuinely retryable delivery/RPC failures, with a regression test confirming that codec failures still propagate?

2. Please add an Unreleased / Fixed changelog entry.

   This fixes user-visible Workflow Streams delivery behavior, and the repository guidance requires a high-level changelog entry for user-facing changes.

Validation on the PR head: the focused regression passed, and the complete Workflow Streams test file passed (43 passed).

[1fanwang]

Done in 7213ce2. Retry scope now matches SDK Core, codec and oversized-message failures propagate, and the changelog is updated.