| 19 Apr 2024 |
 FrostyX | Kai: I'd probably ask on https://matrix.to/#/#devel:fedoraproject.org | 12:43:20 |
| FrostyX | This is more of a Copr and Mock related channel | 12:43:50 |
|  zbyszek joined the room. | 14:52:44 |
| zbyszek | Hi, I'm seeing the following issue, it seems always only in the i386 repo of a multi-arch copr project:
Errors during downloading metadata for repository 'local':
- Status code: 404 for https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/229b34fcd39b88532a400a0264a7bc62a532af4c0567b0a4a4e6fc111afd307c-primary.xml.gz (IP: 38.145.60.21)
| 14:54:53 |
| zbyszek | * Hi, I'm seeing the following issue, it seems always only in the i386 repo of a multi-arch copr project:
Errors during downloading metadata for repository 'local':
- Status code: 404 for https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86\_64/repodata/229b34fcd39b88532a400a0264a7bc62a532af4c0567b0a4a4e6fc111afd307c-primary.xml.gz (IP: 38.145.60.21)
| 14:55:01 |
| zbyszek | For example systemd CI build: https://download.copr.fedorainfracloud.org/results/packit/systemd-systemd-32361/fedora-rawhide-i386/07328466-systemd/builder-live.log.gz | 14:55:27 |
| zbyszek | Something similar in my copr repo zbyszek/merged-sbin:
https://download.copr.fedorainfracloud.org/results/zbyszek/merged-sbin/fedora-rawhide-i386/07309235-postfix/builder-live.log.gz | 14:56:56 |
 Jiri Kyjovsky | seems like fedora-rawhide-i386 chroot is broken, idk why. I'll have a closer look on it later | 15:06:35 |
| zbyszek | OK, I'm told that it's because the i386 repos are not mirrored. | 15:10:08 |
| zbyszek | Maybe the retry limit should be increased (with a delay)? | 15:10:45 |
 music | I can confirm I’ve been seeing flaky failures due to similar 404’s on i386 builds in COPR. It seems to have started, or at least become relatively common (maybe as much as 5% of builds?) a few months ago. | 15:14:03 |
| 20 Apr 2024 |
 Michael J Gruber | In reply to @frostyx:matrix.org
I agree with you that my expectation as a user is - when using all the default options, if something builds in Copr, it should build in Koji as well. I don't think we will change the default but I will ask other folks what they think of having some options Just to understand - what is/was the reasoning to have copr defaults different from koji? | 08:20:58 |
| FrostyX | Michael J Gruber: This definitely predates my time in the Copr team, so I cannot speak to how it actually happened. But I think the default makes sense. Copr uses the default upstream Mock configs, that's a feature, not a bug. And Mock is a general tool that supports multiple linux distributions, so its goal isn't to provide a local Koji environment with all Koji quirks. I'd say it's the other way around - Koji uses Mock and when they step out of the default configuration, they have a good reason to do so.
In this case, Mock (and therefore Copr) uses the official Fedora repositories that you and I have enabled on our laptops, and it seems, that Koji disables them and uses their local repositories. I guess by setting config_opts['mirrored'] = False .
| 10:28:36 |
 Stephen J Smoogen | It is hard to set up the same defaults as koji because those are set by the Fedora release engineers and the current version of koji installed. Koji basically pulls a config out of its .. settings and gives it to a builder to point that systems mock at it. What is in that config depends on various things like 'what is in the koji buildroot, what kind of builder is this set up in the database, do I need to use a particular buildroot or side, etc'. Some of those are ephemeral | 20:02:38 |
| 21 Apr 2024 |
| Michael J Gruber | I see. So copr builds are much more like local mock builds, and koji is just different. There is no strict substitute for koji --scratch. | 11:01:36 |
 Penguinpee | In reply to @mjg:fedora.im
I see. So copr builds are much more like local mock builds, and koji is just different. There is no strict substitute for koji --scratch. You can mimic that with --config-opts mirrored=False added to mock or fedpkg mockbuild -- --config-opts mirrored=False, I believe. | 15:16:07 |
| 22 Apr 2024 |
 praiskup | Michael J Gruber: Penguinpee ftr: https://github.com/fedora-copr/copr/issues/2643 | 11:03:23 |
| 23 Apr 2024 |
| *  knurd wonders a bit why some copr builds that apparently finished more than an hour ago remain in the running state: https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/mainline-wo-mergew/build/7338589/ | 10:45:53 |
| * knurd suspects there is some problem detecting or downloading the packages that were built | 10:46:39 |
| Jiri Kyjovsky | aha so this is why we have such large queue right now! | 11:33:35 |
| Jiri Kyjovsky | thank you for the ping | 11:33:40 |
| Jiri Kyjovsky | I'll try to restart the backendm hopefully it will work (this is not first time this hapenned so we should probably investigate what's going on) | 11:34:56 |
| knurd | many thx; yeah, I noticed something similar a few weeks ago, but it was not that bad back then | 11:35:34 |
| knurd | in case I notice something like this again I'll ping earlier next time | 11:35:56 |
| knurd | first I thought it was just something random that would resolve itself after some time | 11:36:17 |
| Jiri Kyjovsky | some builds that were stuck in the running state are finishing. So it should recover itself after a while | 11:41:42 |
| knurd | yeah, a few that were kinda stalled for a while are now finished; thx again! | 11:47:08 |
| * knurd sees that the build in https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/next/build/7338891/ are still not processed and wonders if backendm is still processing the backlog or slow/stuck again | 12:51:54 |
| Jiri Kyjovsky | it's getting slowly better but the backend is under huge load so we decreased max concurent running builds and the queue and load is getting better, but it is slow process :/ | 12:53:14 |
| Jiri Kyjovsky | hopefully in another 30 min max 1 hour it will be fine | 12:53:39 |
