Sender | Message | Time |
---|---|---|
11 Jul 2024 | ||
Stollmar | In reply to @katie-dfinity:matrix.orgJust sent | 10:30:55 |
Katie Peters (DFINITY) | Thanks! Sven is looking at it today, and I'm trying to understand the ins and outs of reward proposals better. | 11:11:19 |
Stollmar | In reply to @katie-dfinity:matrix.orgPlease reach out with any news! hoping to get the reward this week | 11:24:23 |
Bhavik Vadhia | One of the node- 6syks-la2v3-ua4aj-iyt6f-oivnq-6yosi-cqxwo-ahzno-ieuvx-bl7pf-vqe got awaiting status after few hours of reboot. Rest nodes are still offline. Can you guide us to view the logs so that we can take necessary action and set them up and running. | 13:38:13 |
Saša Tomić [DFINITY] | andrewbattat [DFINITY]: just finished developing a new feature for checking logs in such cases. I'll let him comment and provide instructions. | 13:39:39 |
Charles Witte | Hello. I am configuring HU1s RMU device and I could use some help filling in some blanks. Can someone from DFINITY with RMU configuration knowledge reach out to me if you have time for some troubleshooting? | 15:00:11 |
eero [DFINITY] changed their display name from eerokelly to eero. | 18:12:38 | |
eero [DFINITY] changed their display name from eero to eero [DFINITY]. | 18:13:28 | |
eero [DFINITY] | Hey Bhavik Vadhia! It looks like all of these nodes are back online. Could you share any steps you took, if any, to change this? From the outside, I can confirm that we had no visibility to these nodes, I wasn't able to reach them when investigating this morning. For any sort of network issues like this in the future, it is really helpful to us if you can share any view of the console. Now that the nodes have recovered and logs are reachable again, they have all made it to the correct version and nothing really jumps out as wrong. We've seen this sort of thing a few times in the past where networking equipment between the nodes and the internet (routers, firewalls, etc.) get a bit hung up after redeployments or upgrades. Restarting any of these has cleared up problems in the past. When you say you were able to ping them, was this from their internal network? | 18:42:04 |
12 Jul 2024 | ||
Stollmar | There was a massive internet outage in Buenos Aires that lasted less than 5 minutes. We are awaiting the result report from the IPX, but our 4 nodes degraded during this situation. Should we completely reinstall the nodes or can we save them without reinstalling? Also, what is the possible recovery flow in such cases? | 01:43:29 |
Stollmar | Download image.png | 01:43:33 |
Stollmar | all of our nodes are back to normal no | 03:34:14 |
Stollmar | now* | 03:34:20 |
Saša Tomić [DFINITY] | In reply to @marianostoll:matrix.orghttps://wiki.internetcomputer.org/wiki/Node_Provider_Troubleshooting | 05:28:07 |
Stollmar | Download image.png | 06:55:47 |
Bhavik Vadhia | There was power fluctuation observed because of which the switch was reset. This caused the same IPs assigned to alternate machines. The IP were able to ping but the assignment was in correct. while analysing the issue, the incorrect assignment was observed. Later ip with respective port was assigned and system were UP. | 06:59:28 |
13 Jul 2024 | ||
Bhavik Vadhia | @nabdullin-dfinity:matrix.org, Saša Tomić [DFINITY] This server is still in degraded mode. Any know steps to get it active? It’s been a week now. | 05:25:30 |
14 Jul 2024 | ||
marioliang | god.. we should try this as well! since ours been version behind for more than a month | 05:00:57 |
17 Jul 2024 | ||
Charles Witte | Redacted or Malformed Event | 18:33:29 |
Charles Witte | Redacted or Malformed Event | 18:34:25 |
18 Jul 2024 | ||
alexa-zhang410 | In reply to @_slack_dfinity_B067Y8TK4LD:matrix.orgHi there,One of our nodes encountered the "IC_Networking_StateSyncLoop" error today https://dashboard.internetcomputer.org/node/c3syk-l56ky-owcvg-y4ogf-3qfti-elv4e-2bqp7-k5v32-ibzm6-o4k6s-eqe . We want to confirm if this is still considered a software issue. I've already checked and confirmed that there are no issues with the network links. Is there anything specific we need to do on our end? Thanks | 02:44:25 |
19 Jul 2024 | ||
Malith Hatanachchige | Hi Saša Tomić [DFINITY] Our ISP currently provides IPv4 and IPv6 addresses from India, but with local loop/breakout, therefor latency and IPs act like local IPs. We want to switch to local IPs for better geolocation accuracy and future-proofing, ensuring access to more local IPv4 and IPv6 addresses. Is there a procedure to change IPs without redeploying the nodes? | 05:08:43 |
Saša Tomić [DFINITY] | In reply to @malithh:matrix.orgThats sadly not possible at the moment. The ipv6 address is read only for the node, so the only way is to redeploy a node. You will need to create a new config file in the installation usb. However you will need to manually remove the old node records from the registry since the ipv6 address will be different when a node is redeployed. Also, the removal of nodes that are currently in subnets can't work. So we'll need to remove nodes from the subnets first. This can be done with the DRE tool. I'll share the details when you are ready | 06:59:29 |
Malith Hatanachchige | In reply to @sasa-tomic:matrix.org Seems like a long process from what I understand
I think it will be worth it. I will share once we are ready. | 07:02:13 |
Malith Hatanachchige | Also do we have to submit a reward proposal ? | 07:03:17 |
Malith Hatanachchige | * Also do we have to submit a reward proposal ? I assume not | 07:03:34 |
Saša Tomić [DFINITY] | I'd actually recommend to:
| 07:08:35 |
Saša Tomić [DFINITY] | * I'd actually recommend to:
There is no need to change the rewards configuration if you're keeping the same node count. | 07:08:45 |
Saša Tomić [DFINITY] | For reference, I've added the process to the wiki. Please feel free to make changes: https://wiki.internetcomputer.org/wiki/Changing_IPv6_addresses_of_nodes | 07:19:24 |
Malith Hatanachchige | Wonderful, I'll keep you posted when we have the IPs ready. They will be all static and we can scale up to 500+ v4 on top of the v6 | 07:23:53 |