Sender | Message | Time |
---|---|---|
10 Jan 2025 | ||
Michel Lind 🎩 UTC-6 joined the room. | 20:27:07 | |
Michel Lind 🎩 UTC-6 | Hi everyone. is cfp.fedoraproject.org maintained by people here? | 20:27:51 |
Michel Lind 🎩 UTC-6 | I don't see it listed in apps.fedoraproject.org somehow | 20:29:27 |
Michel Lind 🍥🇧🇪 UTC+1 until Feb 4 changed their display name from Michel Lind 🍥 UTC-6 to Michel Lind 🍥⛱️ UTC-8. | 23:18:31 | |
Michel Lind 🍥🇧🇪 UTC+1 until Feb 4 changed their display name from Michel Lind 🍥⛱️ UTC-8 to Michel Lind 🍥⛱️ UTC-8 until Jan 19. | 23:19:09 | |
Michel Lind 🎩 UTC-6 changed their display name from Michel Lind 🎩 UTC-6 to Michel Lind 🎩⛱️ UTC-8 until Jan 19. | 23:20:01 | |
12 Jan 2025 | ||
mattia | if anyone has some spare cycles, I'm stuck on debugging bodhi crash on F41: https://github.com/fedora-infra/bodhi/issues/5841 | 14:16:58 |
13 Jan 2025 | ||
Jeremy Cline | I've not used SQLAlchemy in years but it sounds like something changed in it or in the web framework bodhi uses (Pyramid?) between F40 and F41. I think here the post-request handler is attempting to finalize the SQL transaction by committing it (or rolling back if the request handler raised an exception), and SQLAlchemy is saying "no no, this transaction has already been committed". I'd look at SQLAlchemy changes first around transaction commits happening, and then possibly at Pyramid middleware getting run more than once somehow if the SQLAlchemy stuff doesn't turn up anything. You could also downgrade each to the F40 version to see which one causes the failure. | 15:06:52 |
Jeremy Cline | Actually the log The way the process used to work (from my hazy memory) was when a request came in, before the handler ran middleware opened a new SQLAlchemy session and somehow ensured the handler used it, and then the session got closed and committed after the request handler returned in the post-request middleware. This sounds like during the request handler the operations aren't happening in that session at all, or the session already got closed somehow. "autoflush" sounds like a likely candidate. | 15:11:39 |
Jeremy Cline | Although according to src.fedoraproject.org both Pyramid and SQLAlchemy are the same version in F40 and F41, so maybe all the above is nonsense. | 15:13:37 |
mattia | From what I checked, all involved components are the same versions. What it changes is Python (from 3.12 to 3.13) | 16:20:29 |
mattia | I think the warnings about SAWarning: Object of type <Update> not in session are not directly related to the crash (although I'm preparing a fix for that too) | 16:21:15 |
mattia | the crash seems to happen when a notification is published to fedora-messaging, and sqlalchemy complains about the session is already committed... but that was never a problem before and according to docs it should work (the session should automatically be reopened) | 16:22:57 |
mattia | I think the real crash is just before when python threads module outputs RuntimeError: thread already started and then everything fails after that | 16:24:39 |
mattia | that seems to be caused by crochet, but again we have the same version in F40 and F41 and I can't find any bug report upstream | 16:27:44 |
Jeremy Cline | Oh, from crochet? I've seen that when twisted is already started and then you use fedora-messaging which uses crochet to provide a blocking api on top of twisted. Everything still seems to work so I've not bothered looking at it harder. | 16:28:37 |
Jeremy Cline | This is what I've seen elsewhere:
| 16:31:17 |
Jeremy Cline | Oh I see. Try disabling message publishing and see if it's still reproducible (drop https://github.com/fedora-infra/bodhi/blob/2c109c8d1ec2f698212da4550780215c34fc86b7/bodhi-server/bodhi/server/notifications.py#L35 or make the function a no-op or something). If it's not reproducible it's some interaction between crochet raising an exception and the SQLAlchemy session handling, I guess. | 16:40:24 |
mattia | let's try. I didn't thought of that | 16:42:32 |
mattia | BTW, checking crochet F41 build logs shows many tests are failing, but still reported passing... I don't think it works well with Python 3.13. cc Aurélien B as crochet and fedora-messaging maintainer | 17:07:43 |
mattia | so, disabling sending messages to fedora-message gives some improvements. But now the call to celery task fails with some error about ssl handshaking... https://github.com/fedora-infra/bodhi/issues/5841#issuecomment-2587766155 | 17:42:17 |
mattia | * so, disabling sending messages to fedora-messaging gives some improvements. But now the call to celery task fails with some error about ssl handshaking... https://github.com/fedora-infra/bodhi/issues/5841#issuecomment-2587766155 | 17:42:40 |
mattia | But I think that's a separate issue, it happens later in Bodhi code | 17:43:38 |
Jeremy Cline | That does seem like perhaps kombu just not gracefully handling a connection timeout | 18:26:47 |
15 Jan 2025 | ||
Roman Epo joined the room. | 13:01:22 | |
16 Jan 2025 | ||
nirik | REMINDER: There will be a Fedora Infrastructure meeting today (2025-01-16 at 17:00 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org). Please check the agenda at https://board.net/p/fedora-infra | 16:47:00 |
nirik | REMINDER: There is a Fedora Infrastructure meeting starting right now: (2025-01-16 at 17:00 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org). Please check the agenda at https://board.net/p/fedora-infra | 17:01:01 |
ffmancera | Hello Fedora Apps people! The CFP for Google Summer of Code at Fedora is now open! https://gitlab.com/fedora/mentoring/home/-/issues/149 - let me know any question you might have :) | 19:09:34 |
17 Jan 2025 | ||
Aurélien B | Join me on https://www.twitch.tv/ohwellien to look at Fedora infrastructure applications! And maybe fix bugs! | 15:38:43 |
19 Jan 2025 | ||
brozic joined the room. | 20:28:23 |