Just following up on our brief chat earlier – last Friday Tristan and I noticed that an instance of the Cylc8 GUI that I had had running for ~45 days to monitor and interact daily with my reanalysis download suites had started to use up a fair chunk of the available resource on w-nwp01.maui.niwa.co.nz. See the top line in the image below. w-nwp01 fell over early evening on Friday, but we’re not sure if this process was the cause. The HPC team just side it’s like because the machine ran out of memory.

Is this an expected footprint for the Cylc8 GUI in your experience? We were wondering what would be the outcome once everyone is running Cylc8 and perhaps running their own instance of the GUI. Should these be running on the Cylc servers instead? Or is the recommended practise that the GUI is started/shutdown regularly rather than being left running in the background?

I see that there are now more recent versions of Cylc8 available on the NeSI software stack so I’ll update very soon. Perhaps this is a known issue that has since been solved that I’d missed a note for in the Teams channel?

I need to restart the GUI later today. To do the monitoring that you mentioned earlier, would you like me to start the GUI again at vn8.1.1 or shall I upgrade everything to vn8.2.4 before restarting it all?

Going back to this from Ronnie Dutta -

Nevermind, looks like the memory leak is only in whatever hooks are injected into the code in the development build only. In the production build there does not seem to be a leak:

Turns out the above user was running 8.1.1

I've asked him to upgrade.

In reply to @oliver-sanders2:matrix.org
Out of interest, do we know if this UIS had active cylc-ui session(s) connected?

User reponse:

if i close the browser tab that I was using the CPU usage stays high

So it turns out the UIS is still subscribing to all data from all active workflows. I think this was a temporary solution that became permanent but no issue for prioritisation so it slipped the net.

As such high memory usage for large numbers of running workflows is expected and unavoidable.

I've raised an issue for this:

https://github.com/cylc/cylc-uiserver/issues/568

It's relatively straight forward to adjust what the UIS subscribes to, protobuf "topics" correspond to GraphQL "fields", but working out what topics we need for which workflows to satisfy the active subscriptions is a trickier problem that doesn't neatly fit any existing interface.

Given that we have reports of high memory usage and that this is kinda fundamental to UIS scalability, suggest bumping this up the priority ladder :(

David Sutherland would be good to get your ideas on this.

So it turns out the UIS is still subscribing to all data from all active workflows. I think this was a temporary solution that became permanent but no issue for prioritisation so it slipped the net.

As such high memory usage for large numbers of running workflows is expected and unavoidable (exactly what "large" means is up for grabs though, it also depends on the complexity of the workflows themselves). I've raised an issue for this:

https://github.com/cylc/cylc-uiserver/issues/568

It's relatively straight forward to adjust what the UIS subscribes to, protobuf "topics" correspond to GraphQL "fields", but working out what topics we need for which workflows to satisfy the active subscriptions is a trickier problem that doesn't neatly fit any existing interface.

Given that we have reports of high memory usage and that this is kinda fundamental to UIS scalability, suggest bumping this up the priority ladder :(

David Sutherland would be good to get your ideas on this.

On the subject of UIS resource usage...

So it turns out the UIS is still subscribing to all data from all active workflows. I think this was a temporary solution that became permanent but no issue for prioritisation so it slipped the net.

As such high memory usage for large numbers of running workflows is expected and unavoidable (exactly what "large" means is up for grabs though, it also depends on the complexity of the workflows themselves). I've raised an issue for this:

https://github.com/cylc/cylc-uiserver/issues/568

It's relatively straight forward to adjust what the UIS subscribes to, protobuf "topics" correspond to GraphQL "fields", but working out what topics we need for which workflows to satisfy the active subscriptions is a trickier problem that doesn't neatly fit any existing interface.

Given that we have reports of high memory usage and that this is kinda fundamental to UIS scalability, suggest bumping this up the priority ladder :(

David Sutherland would be good to get your ideas on this.

On the subject of UIS resource usage...

So it turns out the UIS is still subscribing to all data from all active workflows. I think this was a temporary solution that became permanent but no issue for prioritisation so it slipped the net.

As such high memory usage for large numbers of running workflows is expected and unavoidable (exactly what "large" means is up for grabs though, it also depends on the complexity of the workflows themselves). CPU usage may also be high as a result of the cost of reconciling large numbers of updates. I've raised an issue for this:

https://github.com/cylc/cylc-uiserver/issues/568

It's relatively straight forward to adjust what the UIS subscribes to, protobuf "topics" correspond to GraphQL "fields", but working out what topics we need for which workflows to satisfy the active subscriptions is a trickier problem that doesn't neatly fit any existing interface.

Given that we have reports of high memory usage and that this is kinda fundamental to UIS scalability, suggest bumping this up the priority ladder :(

David Sutherland would be good to get your ideas on this.

FYI there is a new VSCode extension for Vitest: https://marketplace.visualstudio.com/items?itemName=vitest.explorer

I had to switch to the pre-release version otherwise all the tests that use string substitution in their name fail (e.g. it(`returns ${expected}`)), but seems pretty nifty

FYI there is a new VSCode extension for Vitest: https://marketplace.visualstudio.com/items?itemName=vitest.explorer

I had to switch to the pre-release version otherwise all the tests that use string substitution in their name fail (e.g. it(`returns ${expected}`), but seems pretty nifty

UIS efficiency

We don't yet know the cause of the high memory usage BoM has reported, hopefully they will investigate this and share the results.

However, I suspect that the cause is likely data accumulation at the UIS. At present, data only leaves the store when the workflow is removed (i.e. "unregistered" in UIS speak). This means that data for old, stopped workflows remains in the store. On top of this we are subscribing to all data from all active workflows whether it is needed or not exacerbating the issue.

Although the store data is (remarkably) small thanks to the protobuf serialisation, it can still accumulate and (library bugs aside) I can't think what other data might be accumulating at the UIS.

So I think these two issues should move up the priority ladder:

• https://github.com/cylc/cylc-uiserver/issues/568
• https://github.com/cylc/cylc-uiserver/issues/585