log verbosity

After code review comments I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though: it would be really nice to have at least one "verbose" logging level between INFO and DEBUG.

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't necessarily suggest contiguous graph nodes
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't necessarily suggest contiguous graph nodes, which is good

• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't necessarily suggest contiguous graph nodes, which is good
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't necessarily suggest contiguous graph nodes, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't really suggest contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Looking at the code, it is a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool, "health" messages detail next polling times, etc.
• DEBUG: plus the extreme verbosity, line numbers, and tracebacks, needed for literally debugging the scheduler (which I find pretty painful to read under most circumstances - it shouldn't be needed for just following events in more detail)

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is quite nice in some ways, considering the scheduler alone, but:
  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't really suggest contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is quite nice in some ways, considering the scheduler alone, but:

  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't really suggest contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is quite nice in some ways, considering the scheduler alone, but:
  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't really suggest contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is quite nice in some ways, considering the scheduler alone, but:

  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't really suggest contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

Finally, how should we refer to tasks entering and exiting the scheduler task pool? We've been using "spawn" and "remove" for these forever. I don't think "spawn" is a bad term for that (upcoming tasks get "spawned" into the scheduler's awareness on demand). But I suppose we could decide to say "enter" and "exit" in user docs. (cylc remove causes a task to exit the task pool...)

Thoughts? Flame-wars? 😁

log verbosity

After code review comments on https://github.com/cylc/cylc-flow/pull/6067 I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though: it would be really nice to have at least one "verbose" logging level between INFO and DEBUG.

log verbosity

In response to review of https://github.com/cylc/cylc-flow/pull/6067 I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though: it would be really nice to have at least one "verbose" logging level between INFO and DEBUG.

log verbosity

In response to review of [https://github.com/cylc/cylc-flow/pull/6067](cylc-flow PR 6067) I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though: it would be really nice to have at least one "verbose" logging level between INFO and DEBUG.

log verbosity

In response to review of cylc-flow PR 6067 I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though: it would be really nice to have at least one "verbose" logging level between INFO and DEBUG.

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is quite nice in some ways, considering the scheduler alone, but:
  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't imply contiguous graph, which is good (e.g. you could trigger a task very far away on the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but

• this is quite nice in some ways, considering the scheduler alone, but:

  • it includes some tasks that are not "active"
  • it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't imply contiguous graph (good - you can trigger a task very far away in the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of / currently managing" vs the rest of the (abstract) workflow graph

Finally, how should we refer to tasks entering and exiting the scheduler task pool? We've been using "spawn" and "remove" for these forever. I don't think "spawn" is a bad term for that (upcoming tasks get "spawned" into the scheduler's awareness on demand). But I suppose we could decide to say "enter" and "exit" in user docs, to avoid bespoke terminology as far as possible. (cylc remove causes a task to exit the task pool...)

Thoughts? Flame-wars? 😁

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window". This is quite nice in some ways, considering the scheduler alone, but:

• it includes some tasks that are not "active"
• it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't imply contiguous graph (good - you can trigger a task very far away in the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of / currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window". This is quite nice in some ways, considering the scheduler alone, but:

• it includes some tasks that are not "active"
• it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't imply contiguous graph (good - you can trigger a task very far away in the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of / currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window". This is quite nice in some ways, considering the scheduler alone, but:

• it includes some tasks that are not "active", which is potentially confusing, and
• it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)

  • we could say "active task pool" but then we have to qualify the word "active" very carefully

• "pool" doesn't imply contiguous graph (good - you can trigger a task very far away in the graph, to enter the pool)

• it kinda works as "the pool of tasks that the scheduler is currently aware of / currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window". This is quite nice in some ways, considering the scheduler alone, but:

• it includes some tasks that are not "active", which is potentially confusing, and
• it is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it avoids the word "active" (good - it contains some non-active tasks)
  • we could say "active task pool" but then we have to qualify the word "active" very carefully
• "pool" doesn't imply contiguous graph (good - you can trigger a task very far away in the graph, to enter the pool)
• it kinda works as "the pool of tasks that the scheduler is currently aware of / currently managing" vs the rest of the (abstract) workflow graph

I have thought for some time that this would be useful - the flip side is a question in my head about whom it's useful to: I think if we called the two states (user) DEBUG and DEV (what we currently call debug) that might better reflect what we are trying to do.

I think it's worth considering as a sort of proposal how you'd document who should use each when - i.e. (using your labels)

VERBOSE:
Who: Cylc learners
When: You want to understand what a workflow feature is doing.

Who: Workflow developers
When: You want to check what effect you change has.

DEBUG:

Who: Cylc Developers
When: Developing Cylc

Who: Cylc users
When: Asking Cylc developers for help with a bug.

I have thought for some time that this would be useful - the flip side is a question in my head about whom it's useful to: I think if we called the two states (user) DEBUG and DEV (what we currently call debug) that might better reflect what we are trying to do.

I think it's worth considering as a sort of proposal how you'd document who should use each when - i.e. (using your labels)

INFO:

Who: All Cylc users
When: Checking that the workflow ran OK.

VERBOSE:

Who: Cylc learners
When: You want to understand what a workflow feature is doing.

Who: Workflow developers
When: You want to check what effect you change has.

DEBUG:

Who: Cylc Developers
When: Developing Cylc

Who: Cylc users
When: Asking Cylc developers for help with a bug.

I've been talking (hopefully not erroneously) using phrases like "teed up in memory", and "tasks the scheduler is thinking about".

"pool" doesn't imply contiguous graph

If it's a typing/taxi/subprocess pool as opposed to a filing system (the rest of the graph) this works well - if it's a pool of water then it's not unreasonable to expect it to be a contiguous thing, and it's not: Extending that analogy leads to task puddles. For Pythony people this is probably not a problem, but I don't know if other languages use the word pool in the same way.

We've been using "spawn" and "remove" for these forever.

How about loaded/archived?

You get a 🐸 for mention spawning to a pool.

I've been talking (hopefully not erroneously) using phrases like "teed up in memory", and "tasks the scheduler is thinking about".

"pool" doesn't imply contiguous graph

If it's a typing/taxi/subprocess pool as opposed to a filing system (the rest of the graph) this works well - if it's a pool of water then it's not unreasonable to expect it to be a contiguous thing, and it's not: Extending that analogy leads to task puddles. For Pythony people this is probably not a problem, but I don't know if other languages use the word pool in the same way.

We've been using "spawn" and "remove" for these forever.

How about loaded/archived ? I think this hints at the relationship between the pool and the graph...

You get a 🐸 for mention spawning to a pool.

I've been talking (hopefully not erroneously) using phrases like "teed up in memory", and "tasks the scheduler is thinking about".

"pool" doesn't imply contiguous graph

If it's a typing/taxi/subprocess pool as opposed to a filing system (the rest of the graph) this works well - if it's a pool of water then it's not unreasonable to expect it to be a contiguous thing, and it's not: Extending that analogy leads to task puddles. For Pythony people this is probably not a problem, but I don't know if other languages use the word pool in the same way.

We've been using "spawn" and "remove" for these forever.

How about loaded/archived ? I think this hints at the relationship between the pool and the graph...

You get a 🐸 for mention spawning to a pool: Presumably 🐸 + 🔥 (flame wars) = 🦎 (salamander)