!etqgAEaQcsBuGJpEvE:matrix.org

Fedora Apps

208 Members
Please register to speak - https://da.gd/register | Fedora Infrastructure applications | https://apps.fedoraproject.org - bodhi, fas, mirrormanager, pagure etc11 Servers

Load older messages


SenderMessageTime
10 Jan 2025
@salimma:fedora.imMichel Lind 🎩⛱️ UTC-8 until Jan 19 joined the room.20:27:07
@salimma:fedora.imMichel Lind 🎩⛱️ UTC-8 until Jan 19Hi everyone. is cfp.fedoraproject.org maintained by people here?20:27:51
@salimma:fedora.imMichel Lind 🎩⛱️ UTC-8 until Jan 19I don't see it listed in apps.fedoraproject.org somehow20:29:27
@michel:one.ems.hostMichel Lind 🍥⛱️ UTC-8 until Jan 19 changed their display name from Michel Lind 🍥 UTC-6 to Michel Lind 🍥⛱️ UTC-8.23:18:31
@michel:one.ems.hostMichel Lind 🍥⛱️ UTC-8 until Jan 19 changed their display name from Michel Lind 🍥⛱️ UTC-8 to Michel Lind 🍥⛱️ UTC-8 until Jan 19.23:19:09
@salimma:fedora.imMichel Lind 🎩⛱️ UTC-8 until Jan 19 changed their display name from Michel Lind 🎩 UTC-6 to Michel Lind 🎩⛱️ UTC-8 until Jan 19.23:20:01
12 Jan 2025
@mattia:fedora.immattiaif anyone has some spare cycles, I'm stuck on debugging bodhi crash on F41: https://github.com/fedora-infra/bodhi/issues/584114:16:58
13 Jan 2025
@jcline:fedora.imJeremy ClineI've not used SQLAlchemy in years but it sounds like something changed in it or in the web framework bodhi uses (Pyramid?) between F40 and F41. I think here the post-request handler is attempting to finalize the SQL transaction by committing it (or rolling back if the request handler raised an exception), and SQLAlchemy is saying "no no, this transaction has already been committed". I'd look at SQLAlchemy changes first around transaction commits happening, and then possibly at Pyramid middleware getting run more than once somehow if the SQLAlchemy stuff doesn't turn up anything. You could also downgrade each to the F40 version to see which one causes the failure. 15:06:52
@jcline:fedora.imJeremy Cline

Actually the log SAWarning: Object of type <Update> not in session, add operation along 'User.updates' will not proceed (This warning originated from the Session 'autoflush' process, which was invoked automatically in response to a user-initiated operation.) makes me particularly suspicious of SQLAlchemy.

The way the process used to work (from my hazy memory) was when a request came in, before the handler ran middleware opened a new SQLAlchemy session and somehow ensured the handler used it, and then the session got closed and committed after the request handler returned in the post-request middleware. This sounds like during the request handler the operations aren't happening in that session at all, or the session already got closed somehow. "autoflush" sounds like a likely candidate.

15:11:39
@jcline:fedora.imJeremy ClineAlthough according to src.fedoraproject.org both Pyramid and SQLAlchemy are the same version in F40 and F41, so maybe all the above is nonsense.15:13:37
@mattia:fedora.immattiaFrom what I checked, all involved components are the same versions. What it changes is Python (from 3.12 to 3.13)16:20:29
@mattia:fedora.immattia I think the warnings about SAWarning: Object of type <Update> not in session are not directly related to the crash (although I'm preparing a fix for that too) 16:21:15
@mattia:fedora.immattiathe crash seems to happen when a notification is published to fedora-messaging, and sqlalchemy complains about the session is already committed... but that was never a problem before and according to docs it should work (the session should automatically be reopened)16:22:57
@mattia:fedora.immattia I think the real crash is just before when python threads module outputs RuntimeError: thread already started and then everything fails after that 16:24:39
@mattia:fedora.immattiathat seems to be caused by crochet, but again we have the same version in F40 and F41 and I can't find any bug report upstream16:27:44
@jcline:fedora.imJeremy ClineOh, from crochet? I've seen that when twisted is already started and then you use fedora-messaging which uses crochet to provide a blocking api on top of twisted. Everything still seems to work so I've not bothered looking at it harder. 16:28:37
@jcline:fedora.imJeremy Cline

This is what I've seen elsewhere:

[2024-12-13 10:36:42,990 twisted INFO] Log opened.
Exception in thread CrochetReactor:
[2024-12-13 10:36:42,991 twisted INFO] Log opened.
Traceback (most recent call last):
  File "/usr/lib64/python3.13/threading.py", line 1041, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/usr/lib64/python3.13/threading.py", line 992, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.13/site-packages/crochet/_eventloop.py", line 360, in <lambda>
    target=lambda: self._reactor.run(installSignalHandlers=False),
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.13/site-packages/twisted/internet/asyncioreactor.py", line 252, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.13/site-packages/twisted/internet/base.py", line 924, in startRunning
    raise error.ReactorAlreadyRunning()
twisted.internet.error.ReactorAlreadyRunning
16:31:17
@jcline:fedora.imJeremy ClineOh I see. Try disabling message publishing and see if it's still reproducible (drop https://github.com/fedora-infra/bodhi/blob/2c109c8d1ec2f698212da4550780215c34fc86b7/bodhi-server/bodhi/server/notifications.py#L35 or make the function a no-op or something). If it's not reproducible it's some interaction between crochet raising an exception and the SQLAlchemy session handling, I guess.16:40:24
@mattia:fedora.immattialet's try. I didn't thought of that16:42:32
@mattia:fedora.immattia BTW, checking crochet F41 build logs shows many tests are failing, but still reported passing... I don't think it works well with Python 3.13. cc Aurélien B as crochet and fedora-messaging maintainer 17:07:43
@mattia:fedora.immattiaso, disabling sending messages to fedora-message gives some improvements. But now the call to celery task fails with some error about ssl handshaking... https://github.com/fedora-infra/bodhi/issues/5841#issuecomment-258776615517:42:17
@mattia:fedora.immattia* so, disabling sending messages to fedora-messaging gives some improvements. But now the call to celery task fails with some error about ssl handshaking... https://github.com/fedora-infra/bodhi/issues/5841#issuecomment-258776615517:42:40
@mattia:fedora.immattiaBut I think that's a separate issue, it happens later in Bodhi code17:43:38
@jcline:fedora.imJeremy ClineThat does seem like perhaps kombu just not gracefully handling a connection timeout 18:26:47
15 Jan 2025
@romanepo:fedora.imRoman Epo joined the room.13:01:22
16 Jan 2025
@nirik:matrix.scrye.comnirik REMINDER: There will be a Fedora Infrastructure meeting today (2025-01-16 at 17:00 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org). Please check the agenda at https://board.net/p/fedora-infra 16:47:00
@nirik:matrix.scrye.comnirik REMINDER: There is a Fedora Infrastructure meeting starting right now: (2025-01-16 at 17:00 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org). Please check the agenda at https://board.net/p/fedora-infra 17:01:01
@ffmancera:fedora.imffmanceraHello Fedora Apps people! The CFP for Google Summer of Code at Fedora is now open! https://gitlab.com/fedora/mentoring/home/-/issues/149 - let me know any question you might have :)19:09:34
17 Jan 2025
@abompard:fedora.imAurélien BJoin me on https://www.twitch.tv/ohwellien to look at Fedora infrastructure applications! And maybe fix bugs!15:38:43
19 Jan 2025
@brozic:matrix.orgbrozic joined the room.20:28:23

There are no newer messages yet.


Back to Room ListRoom Version: 5