| 25 Apr 2024 |
 nirik | no, I guess it still is. ;) | 15:53:01 |
| nirik | anyhow, I think it's compeltely normal | 15:53:14 |
 Aurélien B | alright | 15:53:39 |
| Aurélien B | OK I've stopped the scripts and killed the postgresql connections coming from badges-backend01, load is starting to go down, let's see if that's enough to empty the badges queue | 15:57:31 |
| nirik | if you like I can remove old badges from ansible and take the vm's down (but save them in case we need them back) | 16:03:03 |
| nirik | huh... did bugzilla2fedmsg get an upgrade? :) It's sending fedora-messages now where I don't think it was before... | 16:16:30 |
 Non Bot |
PROBLEM
[noc01]:
Check queue bodhi
on
rabbitmq01.iad2.fedoraproject.org
is
CRITICAL
RABBITMQ_QUEUE CRITICAL - messages CRITICAL (781), messages_ready OK (771) messages_unacknowledged OK (10) consumers OK (1) | 16:17:33 |
| nirik | thats likely the f40 kde update stable causing a flurry of messages. | 16:18:04 |
 Zlopez | nirik: Looking at the fedmsg -> fedora messaging tracking ticket, it seems that it was removed from the list 2 years ago https://pagure.io/fedora-infrastructure/issue/8213#comment-745345 | 16:27:26 |
| nirik | odd. I don't recall getting any from the new FMN ever... perhaps the change was there? | 16:28:08 |
| Zlopez | It could be | 16:32:36 |
| Non Bot |
RECOVERY
[noc01]:
Disk_Space_/
on
pagure02.fedoraproject.org
is
OK
DISK OK - free space: / 169262 MiB (16.87% inode=98%): | 16:32:43 |
| Non Bot |
ACKNOWLEDGEMENT
by Kevin Fenzi
[noc01]:
Disk_Space_/
on
proxy06.fedoraproject.org
is
WARNING
DISK WARNING - free space: / 6295 MiB (14.96% inode=76%): | 16:32:55 |
| Non Bot |
ACKNOWLEDGEMENT
by Kevin Fenzi
[noc01]:
Check queue bodhi
on
rabbitmq01.iad2.fedoraproject.org
is
CRITICAL
RABBITMQ_QUEUE CRITICAL - messages CRITICAL (600), messages_ready OK (590) messages_unacknowledged OK (10) consumers OK (1) | 16:33:07 |
| Non Bot |
RECOVERY
[noc01]:
Check queue bodhi
on
rabbitmq01.iad2.fedoraproject.org
is
OK
RABBITMQ_QUEUE OK - messages OK (0) messages_ready OK (0) messages_unacknowledged OK (0) consumers OK (1) All queues under the thresholds | 16:57:33 |
| nirik | NOTICE: In about an hour there may be a slight delay in package signing to allow upgrading the signing infra. Signing should resume after things are back up. | 18:57:39 |
| nirik | so... on the logs being larger on those US proxies... there's really a lot of epel-7 requests from GCP... odd. Would someone have spun up a bunch of new rhel7? why? | 19:29:52 |
| Non Bot |
New Issue created by andreashaerter in fedora-infrastructure
#11898 SSH into homedir on fedorapeople.org: Login does not work
| 19:51:53 |
| Non Bot |
kevin pushed 1 commit(s)
by dustymabe
to main branch of fedora-infra/ansible
| 19:56:05 |
| Non Bot |
PROBLEM
[noc01]:
on
worker05.ocp.stg.iad2.fedoraproject.org
is
DOWN
PING CRITICAL - Packet loss = 100% | 19:59:36 |
| nirik | hum? | 20:10:55 |
| nirik | The console has on it: | 20:12:05 |
| nirik | Red Hat Enter | 20:12:08 |
| nirik | seems locked. | 20:12:14 |
| nirik | power cycled it. | 20:12:57 |
| Non Bot |
PROBLEM
[noc01]:
on
worker05.ocp.stg.iad2.fedoraproject.org
is
DOWN
PING CRITICAL - Packet loss = 100% | 20:14:48 |
| Non Bot |
RECOVERY
[noc01]:
on
worker05.ocp.stg.iad2.fedoraproject.org
is
UP
PING OK - Packet loss = 0%, RTA = 1.14 ms | 20:19:37 |
|  Paul Maconi joined the room. | 20:49:03 |
| nirik | ok, sadly, the sigul upgrade didn't go well... we are back on the old one until I can sort out why it was erroring. ;( | 21:22:35 |
| nirik | I've got to go run into town to pick up a desk in a bit here. Will be back after a few hours. | 21:23:11 |
