!QzQwlBsGqgNDqvlZwU:matrix.org

Wikimedia SRE

196 Members
📟 https://icinga.wikimedia.org/alerts 📟 https://www.mediawiki.org/wiki/SRE | Channel is logged at https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-sre/2 Servers

Load older messages


SenderMessageTime
22 Oct 2021
@cdanis:libera.chatcdanis I'm remembering that time that rzl baited me into making my already-disgusting shell one-liner we were using to troubleshoot some issue on commons into producing output formatted as Phab table markup 13:31:19
@robh:libera.chat@robh:libera.chat joined the room.14:17:23
@robh:libera.chat@robh:libera.chat left the room.14:17:24
@dwisehaupt:libera.chatdwisehaupt joined the room.15:47:43
@tn:libera.chattn joined the room.16:31:23
@tn:libera.chattn left the room.16:37:15
@TNT[irccloud]:libera.chat@TNT[irccloud]:libera.chat joined the room.16:37:16
@TNT[irccloud]:libera.chat@TNT[irccloud]:libera.chat left the room.16:37:55
@tn:libera.chattn joined the room.16:49:23
@tn:libera.chattn left the room.17:17:33
@tn:libera.chattn joined the room.18:19:19
@Ales:libera.chat@Ales:libera.chat joined the room.18:54:40
@Ales:libera.chat@Ales:libera.chat left the room.18:54:57
@mutante:libera.chatmutanteicinga claims puppet-board is down since 6 hours with connection refused.. but my browers says puppet-board works fine19:53:15
@mutante:libera.chatmutante hosts end in 2. I bet it's new hosts with a new role applied that adds monitoring.. common issue 19:53:50
@bblack:libera.chatbblackno, those have been around a while20:05:53
@bblack:libera.chatbblack(dns2002 and others)20:06:00
@bblack:libera.chatbblacksomething else changed in puppet most likely20:06:07
@bblack:libera.chatbblackthere was this several hours ago which touched related bits: https://gerrit.wikimedia.org/r/c/operations/puppet/+/73295420:08:08
@bblack:libera.chatbblack (added drmrs, which may have caused some impact on the other sites' peers settings) 20:08:24
@mutante:libera.chatmutantethe puppetboard1002/2002 are new hosts, it is running on 1001/2001. but the puppet fails on dns hosts are separate 20:09:10
@bblack:libera.chatbblackyeah20:09:16
@bblack:libera.chatbblack I think the change I referenced was what affected dns[12]00[12], I'm just not exactly sure of the details yet 20:09:35
@bblack:libera.chatbblack yeah, I'm gonna roll it back. in theory it can be fixed, but I'm starting to get out in the weeds of it, and don't want to leave timesync or puppet on those hosts in a bad state for the weekend. 20:13:44
@bblack:libera.chatbblack mutante: those dns hosts are clean now with the revert, we'll follow up next week at the traffic level since those patches and the servers are all ours :) 21:05:47
@mutante:libera.chatmutantebblack: :) thanks! great21:06:12
@mutante:libera.chatmutanteand now .. to the Bolivian Cooking video :022:38:06
23 Oct 2021
@cormacparle:libera.chatcormacparle left the room.02:19:48
@cormacparle:libera.chatcormacparle joined the room.02:20:01
@Guest987:libera.chatGuest987 joined the room.02:33:53

There are no newer messages yet.


Back to Room List