| 14 May 2021 |
 Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
Very possible it's a PEBKAC/ID10T error. When the Server showed up I ran a patch to set accepted: true but also set bmc user\pass to one I set up manually specifically for sidero... I later saw the patch notes about Auto BMC config, and it's using Secret refs for the creds, but I think my manual setups may have clashed at some point The sidero/timniverse is the user/pass I set up manually on the IPMI interface for sidero, and the one I put into the Server spec at /spec/bmc/user and /spec/bmc/pass | 20:04:29 |
 andrey | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
The sidero/timniverse is the user/pass I set up manually on the IPMI interface for sidero, and the one I put into the Server spec at /spec/bmc/user and /spec/bmc/pass it very well might be it, Sidero is also provisioning sidero user | 20:09:29 |
| andrey | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
it very well might be it, Sidero is also provisioning sidero user so you can try removing your bmc user/pass but keeping the reference | 20:09:41 |
| Tim Jones | In reply toundefined
also, any way to control the namespace the Secret is generated in? I don't like stuff in default at all!! 🙂 | 20:13:35 |
| Tim Jones | In reply toundefined
(edited) ... at all¬¬ 🙂 => ... at all!! 🙂 | 20:13:40 |
| andrey | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
also, any way to control the namespace the Secret is generated in? I don't like stuff in default at all!! 🙂 rsmitty has more details, but this basically boils down to the cluster-api limitations | 20:16:12 |
| andrey | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
rsmitty has more details, but this basically boils down to the cluster-api limitations we can't put it to sidero-system | 20:16:21 |
| Tim Jones | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
we can't put it to sidero-system no? shame... | 20:16:37 |
| andrey | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
no? shame... user secrets can be anywhere, but auto-generated ones need some namespace | 20:16:44 |
| andrey | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
user secrets can be anywhere, but auto-generated ones need some namespace sidero-system is owned by clusterctl install/upgrade process, so it gets cleaned completely on upgrade | 20:17:07 |
| Tim Jones | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
sidero-system is owned by clusterctl install/upgrade process, so it gets cleaned completely on upgrade yeah, just wondering how that namespace could be provided to sidero... yet another env var in the templates? | 20:17:24 |
 rsmitty | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
yeah, just wondering how that namespace could be provided to sidero... yet another env var in the templates? Yeah it’d have to be something like that | 20:17:44 |
| Tim Jones | In reply to@_slack_taloscommunity_UH9V1FL0K:matrix.org
Yeah it’d have to be something like that can default to default, but allows override... | 20:17:46 |
| Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
can default to default, but allows override... cool | 20:17:52 |
| andrey | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
cool yep, please log an issue to GitHub so that we don't forget it please 😉 | 20:17:56 |
| Tim Jones | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
yep, please log an issue to GitHub so that we don't forget it please 😉 sure thing 👍 | 20:18:34 |
| Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
sure thing 👍 https://github.com/talos-systems/sidero/issues/396 | 20:21:51 |
| Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
https://github.com/talos-systems/sidero/issues/396 Might even be something I can take a look at, but I should be heading to bed already as it is!! Maybe during the little ones nap-time tomorrow 😉 | 20:22:44 |
| Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
Might even be something I can take a look at, but I should be heading to bed already as it is!! Maybe during the little ones nap-time tomorrow 😉 I also have to ask, is IPMI really so insecure as to allow adding an admin-level user account without prior credentials?? WTF?? 😆 | 20:25:11 |
| rsmitty | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
I also have to ask, is IPMI really so insecure as to allow adding an admin-level user account without prior credentials?? WTF?? 😆 lol yeah I was kind of surprised that we could just do that | 20:25:39 |
| rsmitty | In reply to@_slack_taloscommunity_UH9V1FL0K:matrix.org
lol yeah I was kind of surprised that we could just do that but as long as you're on the box itself, it's pretty straightforward | 20:25:54 |
 Seán C McCord | In reply to@_slack_taloscommunity_UH9V1FL0K:matrix.org
but as long as you're on the box itself, it's pretty straightforward You can't do unauthenticated except via local access... but yeah, it's kind of crazy-powerful-insecure | 20:26:24 |
| Seán C McCord | In reply to@_slack_taloscommunity_UJYGJM76W:matrix.org
You can't do unauthenticated except via local access... but yeah, it's kind of crazy-powerful-insecure Fundamentally, never expose IPMI to an untrusted network. | 20:26:42 |
| Tim Jones | In reply to@_slack_taloscommunity_UJYGJM76W:matrix.org
Fundamentally, never expose IPMI to an untrusted network. Guess I should turn off port-forwarding from my public interface then 😉 | 20:27:54 |
| Seán C McCord | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
Guess I should turn off port-forwarding from my public interface then 😉 Eh, if you want to. It can be fun watching what a hacked box can do, too. Especially since BMC hacks are harder to detect and can do much fancier things than mere OS hacks. 🙂 | 20:31:39 |
| Tim Jones | In reply to@_slack_taloscommunity_UJYGJM76W:matrix.org
Eh, if you want to. It can be fun watching what a hacked box can do, too. Especially since BMC hacks are harder to detect and can do much fancier things than mere OS hacks. 🙂 Yeah, I used to enjoy setting up honeypots open to 22 & the like, but really they were just logging everything peeps triedf to do... | 20:39:35 |
| Tim Jones | In reply to@_slack_taloscommunity_UG8G8UMMG:matrix.org
Yeah, I used to enjoy setting up honeypots open to 22 & the like, but really they were just logging everything peeps triedf to do... But it was only cool when it was real people, when the botnets really started taking off it was just the same script scraping the box and got boring | 20:40:21 |
| 15 May 2021 |
 Alex Szakaly | In reply to@_slack_taloscommunity_UGL0YU56H:matrix.org
https://www.talos.dev/docs/v0.9/guides/upgrading-kubernetes/#kubelet I forgot to thank you, sorry Andrey. 🎉 | 05:23:20 |
 born2bake | In reply to@_slack_taloscommunity_U01L24AR5MG:matrix.org
and then it's added as a worker node Hi andrey , I ve managed to restore the masters and I can see all of them as etcd members. However, once they were restored, all services/workload are unavailable, nginx-ingress shows 503 errors so it looks like the masters cant see the workers in the cluster. Any suggestions if I can restore worker nodes somehow?
The way I restore masters - master-3 was bootstrapped as a worker node, I ve run etcd restore from backup snapshot and then master-3 instantly became a master, then I ran reset ephemeral on that node and restarted it. Then I did restore etcd for master-2 and reset it as well. So all 3 now in a quorum | 08:43:59 |
| born2bake | In reply to@_slack_taloscommunity_U01L24AR5MG:matrix.org
Hi andrey , I ve managed to restore the masters and I can see all of them as etcd members. However, once they were restored, all services/workload are unavailable, nginx-ingress shows 503 errors so it looks like the masters cant see the workers in the cluster. Any suggestions if I can restore worker nodes somehow?
The way I restore masters - master-3 was bootstrapped as a worker node, I ve run etcd restore from backup snapshot and then master-3 instantly became a master, then I ran reset ephemeral on that node and restarted it. Then I did restore etcd for master-2 and reset it as well. So all 3 now in a quorum I just rebooted worker nodes and all services were restored 🙌 thew | 10:27:00 |
