!OqDVTrmPstKzivLwZW:gitter.im

python-trio/general

770 Members
Discussion of Trio, a friendly Python library for async concurrency and I/O18 Servers

Load older messages


SenderMessageTime
25 Apr 2024
@oremanj-5abea8c4d73408ce4f93d0be:gitter.imoremanj (Joshua Oreman)e.g. imagine that the context manager starts up a background task and wants to allow you to implicitly communicate with it19:00:21
@oremanj-5abea8c4d73408ce4f93d0be:gitter.imoremanj (Joshua Oreman)

If you do:

async with trio.open_nursery() as n:
    async with my_special_scope():
        n.start_soon(some_task)
    await something_else()

then it is actively unhelpful for some_task to see the new value, because that would make it try to communicate with a task that it will outlive

19:01:25
2 May 2024
@mikenerone:matrix.orgmikenerone

I want a try/except specifically to detect an error during entry into an async context manager. Due to syntactic nesting, I don't see how to do this with the usual async with syntax.

try:
    async with my_cm():
        raise SomeError  # I don't want an error within the block like this to count for this try/except, but it does.
except ...:
    ...

The obvious solution is to use an AsyncExitStack:

async with AsyncExitStack() as stack:
    try:
        await stack.enter_async_context(my_cm())
    except ...:
        ...
   raise SomeError  # There, now this is outside the try/except.

I know Trio goes to some great pains to prevent misnesting of CMs. I just wanted to double check that this try/except approach, which is similarly not possible with the normal nesting syntax, doesn't similarly violate any rules.

14:01:09
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)I don't see a problem with this14:03:56
@mikenerone:matrix.orgmikeneroneCool. I didn't think so. Thanks!14:06:34
@teamspen210-5734b2c8c43b8c6019728f66:gitter.imTeamSpen210 (Spencer Brown)That would be fine, since if the try succeeds, you entered the CM and then the exit stack will ensure you exit in the right order. If it fails, you didn't enter the CM at all. 21:58:39
3 May 2024
@arthur-tacca:matrix.orgarthur-taccaRedacted or Malformed Event12:59:26
@arthur-tacca:matrix.orgarthur-tacca It's equivalent to this scope-respecting snippet (which you could equally use instead) so it doesn't violate any nesting rules. The reason it's equivalent boils down to what @TeamSpen210 (Spencer Brown) said: if the call to my_cm() or cm.__enter__() fails then .__exit__() does not get called regardless of whether using async with directly or AsyncExitStack like your snippet.

entered_cm = False
try:
    async with my_cm():
        entered_cm = True
        await foo()
except SomeError as e:
    if not entered_cm:
        raise
    # ...
12:59:55
@arthur-tacca:matrix.orgarthur-tacca
In reply to@mikenerone:matrix.org

I want a try/except specifically to detect an error during entry into an async context manager. Due to syntactic nesting, I don't see how to do this with the usual async with syntax.

try:
    async with my_cm():
        raise SomeError  # I don't want an error within the block like this to count for this try/except, but it does.
except ...:
    ...

The obvious solution is to use an AsyncExitStack:

async with AsyncExitStack() as stack:
    try:
        await stack.enter_async_context(my_cm())
    except ...:
        ...
   raise SomeError  # There, now this is outside the try/except.

I know Trio goes to some great pains to prevent misnesting of CMs. I just wanted to double check that this try/except approach, which is similarly not possible with the normal nesting syntax, doesn't similarly violate any rules.

It's equivalent to this scope-respecting snippet (which you could equally use instead) so it doesn't violate any nesting rules. The reason it's equivalent boils down to what @TeamSpen210 (Spencer Brown) said: if the call to my_cm() or cm.__enter__() fails then .__exit__() does not get called regardless of whether using async with directly or AsyncExitStack like your snippet.

entered_cm = False
try:
    async with my_cm():
        entered_cm = True
        await foo()
except SomeError as e:
    if not entered_cm:
        raise
    # ...
13:00:40
@arthur-tacca:matrix.orgarthur-tacca
In reply to@arthur-tacca:matrix.org
It's equivalent to this scope-respecting snippet (which you could equally use instead) so it doesn't violate any nesting rules. The reason it's equivalent boils down to what @TeamSpen210 (Spencer Brown) said: if the call to my_cm() or cm.__enter__() fails then .__exit__() does not get called regardless of whether using async with directly or AsyncExitStack like your snippet.

entered_cm = False
try:
    async with my_cm():
        entered_cm = True
        await foo()
except SomeError as e:
    if not entered_cm:
        raise
    # ...
Actually the above is not true anyway if you want to suppress the error - with the original snippet, if you don't re-raise the exception (or any other exception or return) then body of the context manager will still execute, but just with the context manager not active
13:02:46
@mikenerone:matrix.orgmikeneroneAh, yes. In my case, I don't want to suppress the error, so the effect is the same.14:35:03
@mikenerone:matrix.orgmikeneroneAm I the only one that feels like the inability to syntactically wrap the entry of a CM with a try/except (and by "syntactically" I mean with syntax only, no separate state variable) is a bit of a design oversight? That's not to say I have a better idea off the top of my head.15:18:40
@mikenerone:matrix.orgmikenerone At least it's easy to work around (either of the above approaches, or by wrapping in a custom CM of your own (and not even that if the target CM is under your control)), but it feels like it should be able to be done syntactically. 15:34:52
@daniel-alp:gitter.imDaniel Alp joined the room.18:11:45
@arthur-tacca:matrix.orgarthur-tacca
In reply to@mikenerone:matrix.org
At least it's easy to work around (either of the above approaches, or by wrapping in a custom CM of your own (and not even that if the target CM is under your control)), but it feels like it should be able to be done syntactically.
The same is true of stack variable constructors in C++. You can work around that by using the heap and std:: unique_ptr, which is obviously very different from AsyncExitStack but acts a bit like it in this context. (I'm not disagreeing, it's just a point of composition.)
20:00:33
@arthur-tacca:matrix.orgarthur-tacca The same is true of stack variable constructors in C++. You can work around that by using the heap and std::unique_ptr, which is obviously very different from AsyncExitStack but acts a bit like it in this context. (I'm not disagreeing, it's just a point of comparison) 20:02:35
4 May 2024
@drfrogman:matrix.orgPeter SuttonIs there a good pattern for ensuring progress. I've got a collections of tasks that go async a lot then loop. In the middle of the loop, the task does a very expensive task (ffmpeg video encode). I'd like tasks to take turns at this point, round robin style, so that the tasks are all outputting files at roughly the same rate.20:07:23
@drfrogman:matrix.orgPeter Sutton * Is there a good pattern for ensuring progress. I've got a collection of tasks. Each task goes async a lot then loops. In the middle of the loop, the task does a very expensive task (ffmpeg video encode). I'd like tasks to take turns at this point, round robin style, so that the tasks are all outputting files at roughly the same rate.20:08:10
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)how are you launching the ffmpeg thing? if you do it with async directives, as you should be, they would all run concurrently20:10:01
@drfrogman:matrix.orgPeter Suttonffmpeg saturates all CPUs so I can only run one at once (I'm using a CapacityLimiter).20:14:46
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)so what is it that you're aiming for here? that it would suspend one of the subprocesses somehow, in favor of another?20:16:01
@drfrogman:matrix.orgPeter SuttonOnly one task can run ffmpeg at once. Each task should only be allowed to encode again once every other task waiting to encoding has encoded.20:17:57
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)oh, okay, so you don't want to suspend the subprocesses, but instead let another task launch a new subprocess20:18:39
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)that's perfectly doable20:18:56
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm) you need to run these tasks concurrently, and use a Lock 20:19:26
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)each task acquires a lock before running the subprocess, and then releases it20:19:43
@drfrogman:matrix.orgPeter SuttonDo locks act like FIFO queuea? I.e., the lock with be given to tasks in the order they attempt to acquire it.20:21:35
@drfrogman:matrix.orgPeter SuttonI guess they are: https://trio.readthedocs.io/en/stable/reference-core.html#fairness20:28:21
@agronholm-58ef83f0d73408ce4f5824c3:gitter.imagronholm (Alex Grönholm)on trio, you'll want this: https://trio.readthedocs.io/en/stable/reference-core.html#trio.StrictFIFOLock20:28:58
@drfrogman:matrix.orgPeter Suttonoh yer. Thanks that's exactly what I need.20:33:16

There are no newer messages yet.


Back to Room ListRoom Version: 6