Test doesn't assert for correctness

[Avasam]

AFAICT, tests/test_pymonctl.py is a visual only test, running it in the CI can ensure nothing hangs or crashes, but there's no value validation.

If you are fine with the tests staying simply smoke tests, feel free to close this.

[Kalmat]

You're absolutely right. It is a visual test that only makes sense when you run it on your own system, and especially if you have 2 or more monitors attached. I don't see any way to actually test this module's features in a CI environment.

Furthermore, and eventhough you are testing in an actual system, it is not recommendable to enforce the results of the modifications made to the monitors (e.g. adding assert sentences), since there are many caveats when dealing with monitors in every platform (for more info, check the Monitor Class section in README.md, if you haven't yet).

In short, it would be a good thing to have a better test environment, but I don't know how in this case.

[Avasam]

Well, speaking of caveats: Tests fail if you have no primary monitors! 😆 #13 (comment)

it would be a good thing to have a better test environment, but I don't know how in this case.

We're already using virtual monitors and launching session buses just to have a testable environment. Surely it's possible to design a test suite (at least linux-only) that consists ob spawning subprocesses with their own tailor-made virtual desktop for testing various scenarios.

Though I'll be honest, that's more than I'm willing to throw myself at ^^"

[Kalmat]

Though I'll be honest, that's more than I'm willing to throw myself at ^^"

Of course! you have gone much further than one could ask for! Thanks a lot for your work and for pointing out all these issues and improvements.

I will try to find a way to improve the tests so they are meaningful. I don't know how CI internally works nor how it manages displays, roots and screens (monitors). So, by the way, do you know if it is possible to configure more screens (monitors) in CI environment? Do you know if there is any information source in relation to this subject I can dig into?

EDIT: Do you know if it is possible to clone the GitHub CI environment locally? This would allow me to tweak everything before uploading anything.

[Avasam]

Do you know if it is possible to clone the GitHub CI environment locally?

You could technically replicate the GitHub env (the linux one at least) through docker.
There is information out there about pre-installed software on the runners, and such:

• https://docs.github.com/en/actions/concepts/runners/github-hosted-runners#preinstalled-software-for-github-owned-images
• https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Arm64-Readme.md

There's also tools that claim being able to run GitHub actions locally (so you can test them in a similar environment and reduce back&forth with CI). This is NOT an endorsement, I've never tried it, but I've seen https://github.com/nektos/act mentioned in blog posts and on stackoverflow.

---

There's also some VSCode extensions that claim being able to run github actions locally, but VSCode actions have been a minefield of malware recently, so I'll avoid making any recommendations I don't trust.

[Kalmat]

Thank you!

[Avasam]

I just thought of something, if you can't assert expectations (due to wildly different setups the tests can run on), you could use snapshots!
Basically, save all the print outputs from the CI (1 per OS) into a file (the snapshot). Then a CI test can compare its output to ensure it matches the snapshot 1-to-1.
Whenever it differs (due to CI environment update, or due to code change) the CI test would fail. You can review the new output, ensure all code paths are still taken, and replace the snapshot if the new output is correct.

This would at least ensure consistency without having to emulate full environments. (if you change any behavior that results in a different test output, it won't go by unnoticed). And without forcing specific hardcoded values out of the test.

Locally you could create a temporary snapshot file before working on a feature to help spot any diff before it even goes to CI.

[Kalmat]

That sounds really well, in deed. I'll give it a try as soon as I can. Thank you so much again!

I will also work on the suggestions you made, as well as other minor improvements I detected when reviewing your PRs. This will take some additional time since I will need to deeply test everything before uploading a new version (to be honest, I have to re-build my whole test environment since I don't use it like 2 years ago).