In reply to @acuteenvy:matrix.org

I tested all other official clients:

• the Python client fails one test - it doesn't support macos as an alias for osx,
• the C client fails all language tests (because it doesn't support translations) + it doesn't convert uppercase letters to lowercase,
• the Rust client passes.

In reply to @kbdk:matrix.org
Regarding this, since we don't require clients to implement language flags do we need to have it in main compliance testing? Seth: or is it part of optional checks?

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envcar LANG=nl is set, it must be respected unless specified otherwise.

In reply to @kbdk:matrix.org
Regarding this, since we don't require clients to implement language flags do we need to have it in main compliance testing? Seth: or is it part of optional checks?

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envvar LANG=nl is set, it must be respected unless specified otherwise.

In reply to @kbdk:matrix.org
Regarding this, since we don't require clients to implement language flags do we need to have it in main compliance testing? Seth: or is it part of optional checks?

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envvar LANG=nl is set, it must be respected unless specified otherwise.

If a client has access to environment variables, it MUST use them to derive the preferred user language as described in the next paragraphs.

— https://github.com/tldr-pages/tldr/blob/main/CLIENT-SPECIFICATION.md#language

In reply to @kbdk:matrix.org
Regarding this, since we don't require clients to implement language flags do we need to have it in main compliance testing? Seth: or is it part of optional checks?

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envvar LANG=nl is set, it must be respected unless specified otherwise.

If a client has access to environment variables, it MUST use them to derive the preferred user language as described in the next paragraphs.
…
Clients SHOULD offer options to configure or override the language using configuration files or even command-line options

— https://github.com/tldr-pages/tldr/blob/main/CLIENT-SPECIFICATION.md#language

In reply to @sethi:one.ems.host

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envcar LANG=nl is set, it must be respected unless specified otherwise.

In reply to @sethi:one.ems.host

My interprettation is that the language flag is optional, but respecting the language envvars is mandatory.

For example, a client doesn't have to accept -L nl, but if the envcar LANG=nl is set, it must be respected unless specified otherwise.

If it's alright with others, I can transfer it to the tldr-pages repo now. Seems like the right time now that I'm more openly prompting discussion.

Edit: https://github.com/tldr-pages/tldr-pages-test-harness

Anyone is welcome to yell at me if there are problems with it, or if in practice it's not something we're keen to maintain in the org. I don't mean to add unneccesary maintenence burden on anyone else.

In reply to @sethi:one.ems.host

If it's alright with others, I can transfer it to the tldr-pages repo now. Seems like the right time now that I'm more openly prompting discussion.

Edit: https://github.com/tldr-pages/tldr-pages-test-harness

Anyone is welcome to yell at me if there are problems with it, or if in practice it's not something we're keen to maintain in the org. I don't mean to add unneccesary maintenence burden on anyone else.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
seems fine for now! is it a work in progress, or a 'completed' thing?

It's still a WIP overall, but I don't expect the structure change much unless I receive substantial feedback. The things I'm actively addressing are:

1. Working with a designer for a better compliance badges.
2. As Dharun said, include Dockerfiles for other popular clients.
3. Then just acting on feedback if anyone wants to comment. (Issues/PRs/complaints welcome!)

Transfering it to the org wasn't an indicator that it's done by any means, but just that to me it seems succesful as a proof-of-concept (both self-review and based on feedback in the chat) and so I wanted to move it to a shared space to encourage feedback/contributions.

In reply to @sethi:one.ems.host

It's still a WIP overall, but I don't expect the structure change much unless I receive substantial feedback. The things I'm actively addressing are:

1. Working with a designer for a better compliance badges.
2. As Dharun said, include Dockerfiles for other popular clients.
3. Then just acting on feedback if anyone wants to comment. (Issues/PRs/complaints welcome!)

Transfering it to the org wasn't an indicator that it's done by any means, but just that to me it seems succesful as a proof-of-concept (both self-review and based on feedback in the chat) and so I wanted to move it to a shared space to encourage feedback/contributions.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently only runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs, or to run it locally.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note: While tests are defined for optional/recommends, CI currently only runs tests for required specifications. Happy to change that… I made it lenient during the proof-of-concept and figured I'd just leave it until we badges to separate different compliance levels. ^-^')

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs, or to run it locally.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note: While tests are defined for optional/recommends, CI currently only runs tests for required specifications. Happy to change that, but I made it lenient during the proof-of-concept and figured I'd just leave it until we have badges to separate different compliance levels.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs, or to run it locally.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note: While tests are defined for optional/recommends, CI currently only runs tests for required specifications. Happy to change that, but I made it lenient during the proof-of-concept and figured I'd just leave it until we have badges to separate different compliance levels.)

I could split the action up too, so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical to read if people want to review it via the GitHub Action.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs, or to run it locally.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note: While tests are defined for optional/recommends, CI currently only runs tests for required specifications. Happy to change that, but I made it lenient during the proof-of-concept and figured I'd just leave it until we have badges to separate different compliance levels.)

I could split the action up too so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical to read if people want to review it via the GitHub Action.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, I can update the README to clarify both ways to run it locally. That might be easier that digging through the CI logs for now.

If you have the client installed, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed, you can do:

# specify which client based on the directory names in dockerfiles/
CLIENT_NAME=tldr-c-client
docker build -t tldr-test:$CLIENT_NAME -f dockerfiles/$CLIENT_NAME/Dockerfile .
docker run -it tldr-test:$CLIENT_NAME make validate

You can replace validate with validate-level-2 or validate-level-3 to turn on optional tests.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, I can update the README to clarify both ways to run it locally. That might be easier that digging through the CI logs for now.

If you have the client installed, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed, you can do:

# specify which client based on the directory names in dockerfiles/
CLIENT_NAME=tldr-c-client
docker build -t tldr-test:$CLIENT_NAME -f dockerfiles/$CLIENT_NAME/Dockerfile .
docker run -it tldr-test:$CLIENT_NAME make validate

(I should convert the Docker version to a make command. ^-^')

You can replace validate with validate-level-2 or validate-level-3 to turn on optional tests.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, I can update the README to clarify both ways to run it locally. That might be easier that digging through the CI logs for now.

If you have the client installed, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed, you can do:

# specify which client based on the directory names in dockerfiles/
CLIENT_NAME=tldr-c-client
docker build -t tldr-test:$CLIENT_NAME -f dockerfiles/$CLIENT_NAME/Dockerfile .
docker run -it tldr-test:$CLIENT_NAME make validate

(I should convert the Docker way to a make command. ^-^')

You can replace validate with validate-level-2 or validate-level-3 to turn on optional tests.

In reply to @sbrl-57247531659847a7aff542b6:gitter.im
sounds good!
is there any way to tell why the failing clients failed at all?

Currently, the only way would be to check the last CI run and review the logs, or to run it locally.

They're all run in a single action, so you'd have to scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note: While tests are defined for optional/recommends, CI currently only runs tests for required specifications. Happy to change that, but I made it lenient during the proof-of-concept and figured I'd just leave it until we have badges to separate different compliance levels.)

I could split the action up too so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical to read if people want to review it via the GitHub Action.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, I can update the README to clarify both ways to run it locally. That might be easier that digging through the CI logs for now.

If you have the client installed, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed but have Docker setup, you can do:

CLIENT={client-name} make docker-validate

You can append validate/docker-validate with -level-2 or -level-3 to turn on optional tests.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, I can update the README to clarify both ways to run it locally. That might be easier that digging through the CI logs for now.

If you have the client installed, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed but have Docker setup, you can check the list of clients supported in the dockerfiles/ directory, then do:

CLIENT={client-name} make docker-validate

You can append validate/docker-validate with -level-2 or -level-3 to turn on optional tests.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, that might be easier that digging through the CI logs for now.

If you have the client installed locally, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed but have Docker setup, you can check the list of clients supported in the dockerfiles/ directory, then do:

CLIENT={client-name} make docker-validate

Demo: https://github.com/tldr-pages/tldr-pages-test-harness/pull/3

You can append validate/docker-validate with -level-2 or -level-3 to turn on optional tests.

In reply to @sethi:one.ems.host

Currently, the only way would be to check the last CI run and review the logs.

They're all run in a single action, so you'd have to just scroll/search for the client you're interested in:
https://github.com/tldr-pages/tldr-pages-test-harness/actions/runs/8068177164/job/22040305500

(Note, while tests are defined for optional/recommends, CI currently on runs tests for required specifications.)

I could split that up so it's easier to read logs by client. I did it this way to avoid maintaining the list of clients in another place, but it's not very practical.

If you're alright with checking it locally, that might be easier than digging through the CI logs for now.

If you have the client installed locally, you can do so with:

PATH_TO_TLDR_CLIENT={{path/to/binary}} make validate

If you don't have it installed but have Docker setup, you can check the list of clients supported in the dockerfiles/ directory, then do:

CLIENT={client-name} make docker-validate

Demo: https://github.com/tldr-pages/tldr-pages-test-harness/pull/3

You can append validate/docker-validate with -level-2 or -level-3 to turn on optional tests.