Checklist
CPython versions tested on:
3.8, 3.9, 3.10, 3.11, 3.12
Operating systems tested on:
Linux, macOS
Description:
Occasionaly since last summer, I am working on this issue #93122 to figure out why behaviors are different in asyncio.gather when there is at least one task passed to this function, and when one of them, raises an SystemExit or KeybordInterrupt.
And I found odd behavior when there is only one task that raises SystemExit (or KeybordInterrupt), as below:
import asyncio
async def sub_task():
raise SystemExit
async def asyncio_gather():
try:
await asyncio.gather(
sub_task(),
)
except BaseException as ee:
print(f'{ee = }, id={hex(id(ee))}')
raise ee
if __name__ == '__main__':
try:
asyncio.run(asyncio_gather())
except BaseException as ee:
print(f'END {ee = }, id={hex(id(ee))}')Result is:
ee = CancelledError(), id=0x102c01e40
END ee = SystemExit(), id=0x102c01d80
Raising CancelledError exception in asyncio_gather coroutine is really unexpected. IMO it should be SystemExit. Just to check, I created a new example with a TaskGroup where the same sub_task()coroutine is created in.
import asyncio
async def sub_task():
raise SystemExit
async def asyncio_taskgroup():
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(sub_task())
except BaseException as ee:
print(f'{ee = }, id={hex(id(ee))}')
raise ee
if __name__ == '__main__':
try:
asyncio.run(asyncio_taskgroup())
except BaseException as ee:
print(f'END {ee = }, id={hex(id(ee))}') Result is:
ee = SystemExit(), id=0x103174520
END ee = SystemExit(), id=0x103174520
Task exception was never retrieved
future: <Task finished name='Task-1' coro=<asyncio_taskgroup() done, defined at /Users/yves/Desktop/Bugs/test_ghxxxxx.py:31> exception=SystemExit()>
Traceback (most recent call last):
...
...
And to complete the problem analysis, from the first example, when asyncio.gather is called with 2 tasks as await asyncio.gather(sub_task(), asyncio.sleep(0.5)), SystemExit is raised, not CancelledError. It seems really inconsistent to raise a CancelledError exception.
Analysis:
I tried to understand why CancelledError is raised in the first example.
When the sub_task coroutine raises systemExit, this exception is propagated up to the RunnerContext Manager created in asyncio.run and finally treated in the __exit__ method of this CM.
Through the Runner.close method, there is a call to cancel_all_tasks. In this function, only one task is pending and candidate to cancel: asyncio_gather. The sub_task coroutine which raised SystemExit is finsihed. This part is processing in the call to asyncio.run_until_complete(asyncio.gather(*to_cancel, return_exceptions=True)).
At this point, asyncio_gather task waits for its GatheringFuture future ending. Later in the processing of event loop, two callbacks are executed:
gather.<locals>._done_callback on the pending GatheringFuture future, setting its attribut _exception to SystemExit as result of sub_task coroutine.Task.__wakeup on asyncio_gather pending task, where a call to future.result() cause an SystemExit exception that is going to pass as argument to self.__step.
In this _step method, exc is changed to CancelledError, and passed to self.__step_run_and_handle_result(exc). That triggers the CancelledError exception.
Question:
Is this really an issue ? Could a cancelling task be raised CancelledError when its _exception attribute is already set to SystemExit or KeybordInterrupt ?
Maybe we need to discuss this before validating the problem and finding a fix?
Possible fix (if this is really an issue):
In case of issue/feature, a fix is to change the behavior of Task.__step as follow:
when a task is 'to cancel', the exc variable could be overwrite to CancelledError, only if exc is not already an instance of CancelledError, SystemExit or KeybordInterrupt, as below:
index 8d5bde09ea..07f916dc6d 100644
--- a/Lib/asyncio/tasks.py
+++ b/Lib/asyncio/tasks.py
@@ -283,7 +283,12 @@ def __step(self, exc=None):
raise exceptions.InvalidStateError(
f'_step(): already done: {self!r}, {exc!r}')
if self._must_cancel:
- if not isinstance(exc, exceptions.CancelledError):
+ if not isinstance(exc, (exceptions.CancelledError,
+ SystemExit, KeyboardInterrupt
+ )):
exc = self._make_cancelled_error()
self._must_cancel = False
self._fut_waiter = Noneif we validate this feature, the fix should also be made to the asyncio module: _asynciomodule.c to validate test.test_asyncio.
Linked PRs
Checklist
and am confident this bug has not been reported before
CPython versions tested on:
3.8, 3.9, 3.10, 3.11, 3.12
Operating systems tested on:
Linux, macOS
Description:
Occasionaly since last summer, I am working on this issue #93122 to figure out why behaviors are different in
asyncio.gatherwhen there is at least one task passed to this function, and when one of them, raises anSystemExitorKeybordInterrupt.And I found odd behavior when there is only one task that raises
SystemExit(orKeybordInterrupt), as below:Result is:
Raising
CancelledErrorexception inasyncio_gathercoroutine is really unexpected. IMO it should beSystemExit. Just to check, I created a new example with aTaskGroupwhere the samesub_task()coroutine is created in.Result is:
And to complete the problem analysis, from the first example, when
asyncio.gatheris called with 2 tasks asawait asyncio.gather(sub_task(), asyncio.sleep(0.5)),SystemExitis raised, notCancelledError. It seems really inconsistent to raise aCancelledErrorexception.Analysis:
I tried to understand why
CancelledErroris raised in the first example.When the
sub_taskcoroutine raisessystemExit, this exception is propagated up to theRunnerContext Manager created inasyncio.runand finally treated in the__exit__method of this CM.Through the
Runner.closemethod, there is a call tocancel_all_tasks. In this function, only one task is pending and candidate to cancel:asyncio_gather. Thesub_taskcoroutine which raisedSystemExitis finsihed. This part is processing in the call toasyncio.run_until_complete(asyncio.gather(*to_cancel, return_exceptions=True)).At this point,
asyncio_gathertask waits for itsGatheringFuturefuture ending. Later in the processing of event loop, two callbacks are executed:gather.<locals>._done_callbackon the pendingGatheringFuturefuture, setting its attribut_exceptiontoSystemExitas result ofsub_taskcoroutine.Task.__wakeuponasyncio_gatherpending task, where a call tofuture.result()cause anSystemExitexception that is going to pass as argument toself.__step.In this
_stepmethod,excis changed toCancelledError, and passed toself.__step_run_and_handle_result(exc). That triggers theCancelledErrorexception.Question:
Is this really an issue ? Could a cancelling task be raised
CancelledErrorwhen its_exceptionattribute is already set toSystemExitorKeybordInterrupt?Maybe we need to discuss this before validating the problem and finding a fix?
Possible fix (if this is really an issue):
In case of issue/feature, a fix is to change the behavior of
Task.__stepas follow:when a task is 'to cancel', the
excvariable could be overwrite toCancelledError, only ifexcis not already an instance ofCancelledError,SystemExitorKeybordInterrupt, as below:if we validate this feature, the fix should also be made to the asyncio module:
_asynciomodule.cto validatetest.test_asyncio.Linked PRs
asyncio, do not cancel the current task inTask.__stepwhen aSystemExitexception was just raised #156309