| 22 Oct 2021 |
 cdanis | I'm remembering that time that rzl baited me into making my already-disgusting shell one-liner we were using to troubleshoot some issue on commons into producing output formatted as Phab table markup | 13:31:19 |
|  @robh:libera.chat joined the room. | 14:17:23 |
| @robh:libera.chat left the room. | 14:17:24 |
|  dwisehaupt joined the room. | 15:47:43 |
|  tn joined the room. | 16:31:23 |
| tn left the room. | 16:37:15 |
| ![@TNT[irccloud]:libera.chat](./avatar/%40TNT%5Birccloud%5D%3Alibera.chat) @TNT[irccloud]:libera.chat joined the room. | 16:37:16 |
| @TNT[irccloud]:libera.chat left the room. | 16:37:55 |
| tn joined the room. | 16:49:23 |
| tn left the room. | 17:17:33 |
| tn joined the room. | 18:19:19 |
|  @Ales:libera.chat joined the room. | 18:54:40 |
| @Ales:libera.chat left the room. | 18:54:57 |
 mutante | icinga claims puppet-board is down since 6 hours with connection refused.. but my browers says puppet-board works fine | 19:53:15 |
| mutante | hosts end in 2. I bet it's new hosts with a new role applied that adds monitoring.. common issue | 19:53:50 |
 bblack | no, those have been around a while | 20:05:53 |
| bblack | (dns2002 and others) | 20:06:00 |
| bblack | something else changed in puppet most likely | 20:06:07 |
| bblack | there was this several hours ago which touched related bits: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732954 | 20:08:08 |
| bblack | (added drmrs, which may have caused some impact on the other sites' peers settings) | 20:08:24 |
| mutante | the puppetboard1002/2002 are new hosts, it is running on 1001/2001. but the puppet fails on dns hosts are separate | 20:09:10 |
| bblack | yeah | 20:09:16 |
| bblack | I think the change I referenced was what affected dns[12]00[12], I'm just not exactly sure of the details yet | 20:09:35 |
| bblack | yeah, I'm gonna roll it back. in theory it can be fixed, but I'm starting to get out in the weeds of it, and don't want to leave timesync or puppet on those hosts in a bad state for the weekend. | 20:13:44 |
| bblack | mutante: those dns hosts are clean now with the revert, we'll follow up next week at the traffic level since those patches and the servers are all ours :) | 21:05:47 |
| mutante | bblack: :) thanks! great | 21:06:12 |
| mutante | and now .. to the Bolivian Cooking video :0 | 22:38:06 |
| 23 Oct 2021 |
|  cormacparle left the room. | 02:19:48 |
| cormacparle joined the room. | 02:20:01 |
|  Guest987 joined the room. | 02:33:53 |
