In reply to @katie-dfinity:matrix.org
Our emails are FirstName.LastName@dfinity.org

In reply to @katie-dfinity:matrix.org
Thanks! Sven is looking at it today, and I'm trying to understand the ins and outs of reward proposals better.

Hey Bhavik Vadhia! It looks like all of these nodes are back online. Could you share any steps you took, if any, to change this?

From the outside, I can confirm that we had no visibility to these nodes, I wasn't able to reach them when investigating this morning. For any sort of network issues like this in the future, it is really helpful to us if you can share any view of the console.

Now that the nodes have recovered and logs are reachable again, they have all made it to the correct version and nothing really jumps out as wrong. We've seen this sort of thing a few times in the past where networking equipment between the nodes and the internet (routers, firewalls, etc.) get a bit hung up after redeployments or upgrades. Restarting any of these has cleared up problems in the past. When you say you were able to ping them, was this from their internal network?

In reply to @marianostoll:matrix.org
There was a massive internet outage in Buenos Aires that lasted less than 5 minutes. We are awaiting the result report from the IPX, but our 4 nodes degraded during this situation. Should we completely reinstall the nodes or can we save them without reinstalling? Also, what is the possible recovery flow in such cases?

In reply to @_slack_dfinity_B067Y8TK4LD:matrix.org
It's a known software issue on our side, we've been analyzing this most of the day today. The ongoing rollout seems to be fixing the problem. We haven't seen the same problem on the nodes running the new version. We are keeping a close eye on the node health.

In reply to @malithh:matrix.org
Hi Saša Tomić [DFINITY]
Our ISP currently provides IPv4 and IPv6 addresses from India, but with local loop/breakout, therefor latency and IPs act like local IPs. We want to switch to local IPs for better geolocation accuracy and future-proofing, ensuring access to more local IPv4 and IPv6 addresses. Is there a procedure to change IPs without redeploying the nodes?

In reply to @sasa-tomic:matrix.org
Thats sadly not possible at the moment. The ipv6 address is read only for the node, so the only way is to redeploy a node. You will need to create a new config file in the installation usb. However you will need to manually remove the old node records from the registry since the ipv6 address will be different when a node is redeployed. Also, the removal of nodes that are currently in subnets can't work. So we'll need to remove nodes from the subnets first. This can be done with the DRE tool. I'll share the details when you are ready

Seems like a long process from what I understand
5/6 Nodes are in the subnet.

1. Remove Node from Subnet via NNS proposal ?
2. Remove nodes from the registry
3. Redeploy as new nodes via standard USB installation

I think it will be worth it. I will share once we are ready.

I'd actually recommend to:

1. disconnect nodes from the internet to prevent them from re-registering
2. remove nodes assigned to subnets via NNS proposals, using DRE tool
3. remove all unassigned nodes from the registry by following https://wiki.internetcomputer.org/wiki/Removing_a_Node_From_the_Registry
4. redeploy all nodes as new nodes via standard USB installation.
   There is no need to change the rewards configuration if you're keeping the same node count.

I'd actually recommend to:

1. disconnect nodes from the internet to prevent them from re-registering
2. remove nodes assigned to subnets via NNS proposals, using DRE tool
3. remove all unassigned nodes from the registry by following https://wiki.internetcomputer.org/wiki/Removing_a_Node_From_the_Registry
4. redeploy all nodes as new nodes via standard USB installation.

There is no need to change the rewards configuration if you're keeping the same node count.