small functional test included, which just runs this:

[meta]
    title = "Checks task state of alternate run dir"
[scheduler]
    UTC mode = True
   [[events]]
      inactivity timeout = PT1M
      abort on inactivity timeout = True
[scheduling]
    initial cycle point = 20240101T00
    [[xtriggers]]
        up_dir_foo = workflow_state(\
            workflow=other-run-dir, \
            task=foo, \
            point=20240101T0000Z, \
            cylc_run_dir=~/cylc-run-other-test \
        ):PT10S
    [[graph]]
        R1 = """
foo
@up_dir_foo => bar
"""
[runtime]
    [[foo]]
        script = """
mkdir -pv ~/cylc-run-other-test
cd ~/cylc-run-other-test
echo 'Current directory:'
pwd
echo 'Creating link to original run dir:'
ln -sfv ${CYLC_RUN_DIR}/${CYLC_WORKFLOW_NAME} other-run-dir
"""
    [[bar]]
        script = """
cylc workflow-state --debug other-run-dir \
    --run-dir=~/cylc-run-other-test \
    --task=foo \
    --point=20240101T0000Z
rm -rfv ~/cylc-run-other-test
"""

First task (foo) creates an alternate run dir containing the same workflow using a link, second task (bar) and xtrigger checks task completion of the first task via the alternate run dir.

In reply to @dwsutherland:matrix.org
why would it be looking there ... hmm

David Sutherland: what's the context here? Did you (re)discover this bug yourself, or did you see the issue raised 2 weeks ago by Tom Coleman (BOM) on GitHub?

Either way, I put a fix PR up 2 weeks ago 🙂

I didn't add a test yet though, so we perhaps we can combine forces.

https://github.com/cylc/cylc-flow/issues/6030

https://github.com/cylc/cylc-flow/pull/6031

Done - assigned you as a review David Sutherland

Can someone from the UK side pile in on the review plz.

Note this is an urgent migration-blocker bug fix (certainly at NIWA and BOM).

https://github.com/cylc/cylc-flow/pull/6031

Done - assigned you as a reviewer David Sutherland

Can someone from the UK side pile in on the review plz.

Note this is an urgent migration-blocker bug fix (certainly at NIWA and BOM).

https://github.com/cylc/cylc-flow/pull/6031

In reply to @revilo666:matrix.org
(Currently assigned to 8.3.0 milestone, but it could be retargetted at 8.2.x if not too late for the final 8.2. maintenance release).

Ah, looks like it is too late for that, so 8.3.0 is fine (we're not planning to migrate NIWA ops till then anyway).

(I was away over the Easter long weekend so was out of the loop for a bit).

log verbosity

After code review comments I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though:

1. I don't think we have agreed yet on what exactly to call the scheduler task pool in user docs - as it does still have some kind of user-facing importance
2. it would be nice to have at least one "verbose" logging level between INFO and DEBUG

log verbosity

After code review comments I've demoted added to/ removed from active task pool messages to DEBUG level.

I also demoted the task event manager health messages, which are pretty distracting in the log.

I was planning to argue against the first demotion, because the presence or absence of tasks in the scheduler task pool still matters to users in several important ways. But I've changed my mind, for now.

Relatedly though:

It would be nice to have at least one "verbose" logging level between INFO and DEBUG

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Although looking at the code, it is quite a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool
• DEBUG: plus the extreme verbosity needed for literally debugging the scheduler

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Looking at the code, it is a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool
• DEBUG: plus the extreme verbosity needed for literally debugging the scheduler

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Looking at the code, it is a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool, "health" messages detail next polling times, etc.
• DEBUG: plus the extreme verbosity needed for literally debugging the scheduler

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Looking at the code, it is a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool, "health" messages detail next polling times, etc.
• DEBUG: plus the extreme verbosity needed for literally debugging the scheduler (which I find pretty painful to see, under most circumstances - it shouldn't be needed for just following events in more detail)

There is a 3rd party package out there that does this: https://pypi.org/project/verboselogs and https://github.com/xolox/python-verboselogs

Looking at the code, it is a trivial extension of Python's logging, which we could easily do ourselves to avoid the dependency.

I'd suggest then, something like this:

• INFO: minimal trace of normal task events and state changes
• VERBOSE: plus additional informative details such as tasks entering and exiting the scheduler task pool, "health" messages detail next polling times, etc.
• DEBUG: plus the extreme verbosity needed for literally debugging the scheduler (which I find pretty painful to read under most circumstances - it shouldn't be needed for just following events in more detail)

scheduler task pool - what to call it?

Like it or not, the scheduler task pool still has some kind of user-facing importance. We need to refer to it somehow, e.g. to describe:

• the scheduler's moving window of awareness on the potentially infinite graph
• stall by finished tasks with incomplete outputs
• the difference in task-matching behaviour in and out of the pool

scheduler task pool - what to call it in user docs?

Like it or not, the scheduler task pool still has some kind of user-facing importance. We need to refer to it somehow, e.g. to describe:

• the scheduler's moving window of awareness on the potentially infinite graph
• stall by finished tasks with incomplete outputs
• the difference in task-matching behaviour in and out of the pool

I think I've raised this before, but it got glossed over. Anyway, it's crunch time.

The original idea was "active window" but:

• this is too easily confused with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks
• "pool" doesn't necessarily suggest contiguous graph nodes
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

I think I've raised this before, but it got glossed over and we currently have some inconsistent and varied terminology in the docs.

The original idea was "active window" but:

• this is too easily confused (as already demonstrated by some users) with the related-but-different GUI n-window

My preference, I think, is just "scheduler task pool"

• it contains some non-active tasks
• "pool" doesn't necessarily suggest contiguous graph nodes
• it kinda works as "the pool of tasks that the scheduler is currently aware of, or currently managing" vs the rest of the (abstract) workflow graph

Finally, how should we refer to tasks entering and exiting the scheduler task pool? We've been using "spawn" and "remove" for these forever. I don't think "spawn" is a bad term for that (upcoming tasks get "spawned" into the scheduler's awareness on demand). But I suppose we could decide to say "entering" and "exiting" (the task pool) in user docs.

Thoughts? Flame-wars? 😁