cylc web architecture

9 Members
architecture supporting the new Cylc web GUI1 Servers

Load older messages

Timestamp Message
18 Dec 2018
01:03:58@revilo666:matrix.orgHilary Oliver (I mean this Room right here, not the new Room!)
05:24:14@revilo666:matrix.orgHilary OliverFinal (for now) Cylc-8 Architecture write-up is now available: https://cylc.github.io/cylc-admin/cylc-8-architecture
05:26:21@revilo666:matrix.orgHilary OliverI'll be sending out a link this, for BoM for review. Please do so at your own sites if you have anyone who can usefully comment. (We are of course hoping for no objections at all, but if there are any, better to know sooner than later!).
2 Jan 2019
04:18:24@kinow:matrix.orgBruno P. Kinoshita

Tornado 6.0a1: https://groups.google.com/d/msg/python-tornado/mSeG7Kc6z4o/mKCgR9UuFQAJ

I've just uploaded the first alpha release of Tornado 6.0. This release completes the transition that began in version 5.0: Support for Python 2 has been dropped and Tornado only supports versions of Python with asyncio and native coroutines. A lot of legacy callback-based interfaces have been removed as well. Finally, Tornado now includes full mypy-compatible type hints.

Don't know how long - or if - 5.x will receive updates, but given what we have is still in the works, might be better to stick with 6.x, so that we use the most up to date when cylc 8.x is ready πŸ‘

04:26:52@dwsutherland:matrix.orgDavid SutherlandπŸ‘
08:03:21@matthewrmshin:matrix.orgMatt ShinπŸ‘
10:26:36@oliver-sanders:matrix.orgOliver Sanders joined the room.
8 Jan 2019
21:56:16@revilo666:matrix.orgHilary Oliver

Cross-posting from BoM Slack:

regarding the new Cylc-8 proposed architecture. There are some concerns here about the privileged server model (that in our environment would presumably spawn services run as our various service accounts). The question has been asked if this is a feature that can be disabled (allowing us to manage the invocation of these services ourselves, that might then somehow "hook-back" in/communicate back to the Hub).
I can see how the "spawn" model neatly lets you manage everything, and it aligns well with most other environments.
I suppose, we are interested in the possibility of running the Hub in an unprivileged mode that would just relay comms around (and presumably keep track of state, provide some discovery etc)

21:57:27@revilo666:matrix.orgHilary Oliver

My reply (let me know if you disagree or have other relevant points to make):

I think a web UI with single point of access, to control (including start-up) an evolving pool of services over multiple users and hosts, requires a privileged "hub" of some kind. (Although we hope to keep things nearly as simple as now for single users on their own laptops, who will presumably be able to run the Hub under their own user account). However, we will certainly make a point of documenting exactly what the consequences are of not enabling spawning privileges for the Hub (which would have to be specifically enabled by system admins if not running the Hub as root, of course) and how to deal with the resulting inconvenience. I think (at this stage) this will be:You will have to manually log in to service accounts and use the Cylc CLI to:

  • start Workflow Services (fka "suite daemons")
  • start UI Servers (one per service account) - these handle suite discovery and presentation of the UI, including for stopped suites
22:00:16@kinow:matrix.orgBruno P. KinoshitaMaybe we could add this option... In Jenkins, not sure if it changed, but you could have either the master spawning slaves via SSH/Docker/AWS/etc... or Jenkins listening to either a) HTTP requests to join the cluster, or b) UDP multicast messages with a certain header... then depending on the configuration slaves would be either automatically added to the cluster, or added but disabled until some admin enabled them.
22:00:47@kinow:matrix.orgBruno P. KinoshitaPerhaps... we could have the hub spawning UI servers... or... an option to let something spawn the UI servers elsewhere and let them "join" the hub
22:01:04@kinow:matrix.orgBruno P. KinoshitaThough I suspect we would be breaking away from the current model in JupyterHub
22:06:13@revilo666:matrix.orgHilary Oliver That's a good point Bruno P. Kinoshita, something to keep in mind (although it would be harder to implement than the spawner model, and it may be arguable that it's any more secure than a non-root Hub that spawns into user accounts only via specific sudo-enabled commands?)
22:23:41@kinow:matrix.orgBruno P. KinoshitaAgreed. We should probably start simple, and only then try exploring an alternative like this
22:25:33@revilo666:matrix.orgHilary OliverActually (according to my own diagram) the UI Server spawns suites, so only the second bullet point above is needed.
22:26:10@matthewrmshin:matrix.orgMatt Shin I guess the other model would allow a human admin to do some cross check? This would feel more secure (although not really as most insecurity comes from human interactions). The spawning system with accounting trail reduce the chance of human errors, and should be as secure as any other corporate services.
22:27:39@matthewrmshin:matrix.orgMatt Shin

Actually (according to my own diagram) the UI Server spawns suites, so only the second bullet point above is needed.

Back to the same debate again. I believe we still need to be able to start suites from non interactive environments.

22:29:28@revilo666:matrix.orgHilary Oliver Yes, Matt Shin I agree - but I'm just describing (to BoM staff) what needs to be done to use the new UI without spawn privileges. In which case, manually starting a UI Server (per service account) would be enough, right?
22:29:41@revilo666:matrix.orgHilary Oliver(And of course we still want the ability to run suites without the UI at all).
22:34:36@matthewrmshin:matrix.orgMatt ShinYes.
22:40:20@matthewrmshin:matrix.orgMatt Shin(SSL certificate comes to mind immediately when we are running in this mode.)
9 Jan 2019
07:48:50@davidmatthews:matrix.orgDavid MatthewsI'm confused. Surely they just need to start a hub per service account and then everything should work fine? Why would they need to use the CLI?
23:42:12@revilo666:matrix.orgHilary Oliver David Matthews: I just meant use the CLI to start UI Servers on each account (as opposed to having the Hub automatically spawn UI Servers). I guess an alternative would be to start a Hub per service account, as you suggest, but that would require the same amount of manual work, and you would no longer have a single point of access to all service accounts.
23:43:55@revilo666:matrix.orgHilary Oliver

Some feedback on cylc-8 architecture from a NIWA HPC admin:

This is really nice Hilary,

I did not got super familiarized with the Cylc-7 architecture, but I have the general idea.

My opinions/ideas on this architecture:

Good effort updating the languages to accommodate new technology.
I would have thought that completely leaving root space would be amazing too, this is a big jump already, and if only the spawner is the dependency, maybe it could be the only thing using a SUID for root (in case that is not harmful when executed by malicious user). Being completely independent from root provides some advantages, but also creates more complexity on the Cylc Hub area. NodeJS should definitively run as non-root if possible and the rest as far I know can run as dedicated service users. This way of setting up can even be optional (provided the spawner runs independent of other Cylc Hub components ownerships, as long there are permissions).
Alternative to above, would be to use containers for the Cylc Hub (either as a hole, or for each of the components).
Since most of the load, looks like being on the suite communication, I am wondering if the actions/spawns > against suites and jobs, could have/support alternative remote shells. This could potentially allow faster > performances for sites that don’t have security problems, and have huge workloads.

Nevertheless, looks great!

23:47:15@revilo666:matrix.orgHilary Oliver(That last sentence was supposed to be in-quote too)
23:52:32@kinow:matrix.orgBruno P. Kinoshita

NodeJS should definitively run as non-root if possible

+1 to that. Even though the nodejs reverse proxy is maintained by jupyterhub (https://github.com/jupyterhub/configurable-http-proxy).

Running npm install now by default does a security audit on the dependencies too... just grabbed their master branch, and it found 2 low severity vulnerabilities. While I believe these 2 should be easily mitigated/fixed... having to keep an eye on python tools/libraries/apps is much easier IMO than doing the same for most nodejs apps

23:53:22@kinow:matrix.orgBruno P. Kinoshita(plus what's not audited by npm install, that could be a security bug, but just never used/tested... not sure how production-ready & battle-proofed that proxy is)
10 Jan 2019
07:38:01@davidmatthews:matrix.orgDavid Matthews
In reply to @revilo666:matrix.org
David Matthews: I just meant use the CLI to start UI Servers on each account (as opposed to having the Hub automatically spawn UI Servers). I guess an alternative would be to start a Hub per service account, as you suggest, but that would require the same amount of manual work, and you would no longer have a single point of access to all service accounts.
Hilary Oliver: Ah OK. Not sure how the trust between the Ui servers and hub woud work in this case.
09:02:28@davidmatthews:matrix.orgDavid Matthews(but I'm not yet clear how that works anyway)
16 Jan 2019
04:59:07@martinryan:matrix.orgmartin ryan changed their profile picture.

There are no newer messages yet.

Back to Room List