Manual testing expectations for PRs

[xuang7]

Hi community,

I would like to start a discussion about our expectations for manual testing in PRs, especially around what should be tested and how testing responsibilities should be shared.

Currently, we rely heavily on automated checks, which are helpful and should remain the main baseline. We are also continuing to improve test coverage. At the same time, some issues may only show up in specific manual workflows or environments, such as local setup, Docker, or Kubernetes. For example, during RC2 validation, we found an environment-specific issue related to the new PVE feature (#5963). We have also seen cases during workshops or demos where some issues were only discovered during actual usage.

We probably do not need to require every PR to be tested in every environment, since that would add a lot of overhead. Instead, it may be helpful to define a lightweight standard for manual testing based on the type and risk of the PR.

Some questions I would like to discuss:

1. What should the manual testing expectations be for different types of PRs?

Examples:

• Frontend behavior changes
• Backend/API changes
• Database or schema changes
• Authentication/authorization changes
• Deployment, Docker, Kubernetes, or CI changes
• Large feature PRs

2. What should PR authors be expected to include?

Examples:

• What environment was tested
• What manual workflow was tested
• Whether screenshots, logs, or videos are useful
• Known limitations or untested areas

3. What should reviewers be expected to check?

Examples:

• Whether the testing section is clear enough
• Whether the testing level matches the scope/risk of the change
• Whether additional manual testing should be requested before merge
• Whether reviewers are expected to manually test the PR as well, and under what circumstances

4. How can we make testing across different environments more practical?

Examples:

• Are there ways to provide easier setup instructions for Docker or Kubernetes testing?
• For environments that are time-consuming to set up, can maintainers or reviewers help verify them when needed?
• Should certain high-risk PRs request help from someone who already has the relevant environment available?

The goal is not to increase overhead unnecessarily, but to develop a shared expectation for when manual testing is needed beyond automated checks. This can help clarify what information authors should provide and what reviewers should check, while improving feature reliability in real-world usage.

Please feel free to share your thoughts on what would be practical and reasonable for different types of PRs. Thanks!

[gracecluvohio]

For testing across environments, I'm not sure if it is necessary to target every single environment. We have limited time and lots of work, so I think we should only target the ones that are relevant. For instance, for frontend changes, testing across common browser; for backend, testing on similar environments to our cloud-based service/releases. Perhaps we can invest some time in creating dev containers like some other communities have done, to streamline setup for development and unify testing environments.

I think the burden of testing should fall on the PR author, rather than the reviewer. They should be responsible for testing on the necessary environments and cases and documenting it appropriately. Reviewers should not have to thoroughly test the code but should rather mostly check for general code quality. If they wish to, they can also manually test the PR, but I think that requiring manual checking could discourage reviewers from reviewing PRs and also block them from doing their own dev work. Furthermore, the author is the most familiar with their own PR, so they would have the most knowledge of what to test.

The fourth issue particularly resonates with me. #5961 definitely helps with dev work a lot, and I'd love to see more work to streamline development even more. Streamlining development would help make the feedback loop between testing and development a lot faster. Having better documentation for Docker and Kubernetes setup and keeping it updated would also help streamline the development process for everybody.

[carloea2]

1. What should the manual testing expectations be for different types of PRs? I prefer all of the PRs should be manually tested, 1 setting at least should be good enough.
2. What should PR authors be expected to include? I think current template is good but the points you mentioned will be helpful.
3. What should reviewers be expected to check? I prefer reviewer test the PR manually in 1 setting at least.
4. How can we make testing across different environments more practical? I have not proposal for this point.

Thanks

[Yicong-Huang]

1. What should the manual testing expectations be for different types of PRs?

This is a very big question. But generally speaking, the amount of testing is related to the risk of the change.

I prefer all of the PRs should be manually tested, 1 setting at least should be good enough.

hmm, how to manually test a PR with doc only change? how to manually test a performance change? how to manually test a race condition that only happens 1 out of 1000 times?

If I tested on windows, local dev mode, is it sufficient to say my change works on ubuntu with k8s as well?

2. What should PR authors be expected to include? I think current template is good but the points you mentioned will be helpful.

I think we can improve our template though. for instance, we should ask authors to provide more explicit evidences, for example, screenshots/gifs for UI change.

3. What should reviewers be expected to check? I prefer reviewer test the PR manually in 1 setting at least.

I really think that would be a huge burden on the reviewer. How many PRs can a reviewer work on in a day if we require a reviewer to manually verify a PR? 1, 2, or 10? Taking myself as an example, I have so far reviewed 998 PRs in apache/texera in total, if every PR needs manual tests, say let's 10 minutes, that's 9980 minutes, or about 166 hours, more than 20 full workdays spent on manual testing alone without a break. Not to mention I need time to write comments, discuss with author, and many PRs needs multiple rounds of reviews and testing. The burden to review is too much to afford.

With today's setup, only I and @aglinxinyuan reviewed more than 300 PRs and still being active on reviews. with more burden on reviewer, I doubt anyone would spend more time to review. At least I won't be able to help given this is only an open source project that I contribute to voluntarily.

I truly think reviewers should check the evidence provided by author, PR should be self-contained to explain the change, risk, and evidence why it is good to be merged. Of course, I don't want to prevent reviewers from manually verify the change, just to be safer, and I do test manually a lot during review. but requiring reviewers to manually test is too harsh. We need more help on review, and I want to encourage more ppl to help on review.

4. How can we make testing across different environments more practical? I have not proposal for this point.

This can only be done with CI/CD. it is not feasible for every developer to own/maintain multiple envs.

[SarahAsad23]

I think it's important that every PR includes some level of manual testing. The extent of the testing should depend on the scope and risk of the change, but I don't think we should rely solely on automated tests.

For features that introduce new API endpoints or make backend changes, I think it's especially important to test in our deployment environment, not just locally. I've personally run into cases where a feature worked correctly in my local environment but failed after deployment because the required Kubernetes configuration wasn't set up correctly. Those deployment specific issues cannot be caught with automated tests alone, so having at least some deployment level validation for higher risk changes would be good.

[Yicong-Huang]

but I don't think we should rely solely on automated tests.

Manual tests can verify even less than automated tests...

a feature worked correctly in my local environment but failed after deployment because the required Kubernetes configuration wasn't set up correctly.

This is a real pain. and we need to solve it. but I don't think manual test on your local env can help, right? because it is not the same setup/env.

Those deployment specific issues cannot be caught with automated tests alone

why not improve automated tests? we could also have k8s setup for testing.

[zyratlo]

As I am currently working on merging a new feature with significant amounts of new and modified code, I can share my experience with manual testing. In addition to automated tests I've written to ensure good code coverage, I've extensively inspected and manually ran many of my new API endpoints and services. Of course all of this was done on a local development environment, so I cannot say they will work flawlessly on deployment servers and Kubernetes environments. I will indeed need to do more manual testing on different environments to ensure the features I am adding work as intended.

Another concern is edge case coverage. We cannot expect every developer to manually test every edge case that can possibly be encountered, as it will be too much overhead and impossible to predict every edge case. As such, the developer should at least share what manual tests were performed and what possible cases were excluded.

[Yicong-Huang]

Of course all of this was done on a local development environment, so I cannot say they will work flawlessly on deployment servers and Kubernetes environments.

That is a good point, we need to make our automative (CI) testing environment more equal to production. It is impossible to require developer to manually test a feature against multiple environments, for instance I don't have a windows machine. I also don't have a linux server with k8s deployed ;(

We cannot expect every developer to manually test every edge case that can possibly be encountered, as it will be too much overhead and impossible to predict every edge case.

Exactly, it is just not feasible for developers to manually verify all cases.

[Yicong-Huang]

Hey guys, I find myself could not sleep on this question. Let me mention a few points about manual tests before I layout my opinion on how should we expect manual tests.

1. First of all, manual test can only verify at the exact moment when you tested, on the exact environment you used, the code was working as expected, exactly once. However, how do we know a code is working as expected, after 3 months? How do I know the feature will still work when any OS, env, deployment method changed? Next week a new dependency upgraded, the feature may just stopped working because the lib API changed. For some flaky cases caused by race condition, which may not trigger every time, how do you know if you are not just lucky to not encounter it during your manual test?
   On the other hand, automated test in CI, once you write it out, it will be executed every time when another PR is raised, another commit is submitted, the same code path will be verified thousands of times. Regression can be detected when time passes, dependencies are updated, and flakiness can be exposed by repeated testing.

2. Secondly, there are many aspects of "expectations", and many of them are not feasible to be manually tested, or easily eyeballed. For instance, success path needs to be verified, error cases needs to be verified, code performance is important for hot path, and stability is key for anything connecting to external resources like db, network, etc. Many of those cannot be possibly done manually: how do you expect a human to differentiate a debounce time of 300ms, vs 500ms? how do you expect a human to feel the difference between a network request which had 3 retries and 5 retries? how do you expect a human to read a json output from an API and check if any of the fields are incorrect? what about a feature that is not exposed easily on the UI or user-facing API, how to "manually" test it? What's more, as project goes bigger, there are too many features connected with each other. How do you expect a human to manually verify introducing a new feature did not silently killed some features he/she was not even aware of existing?
   Automated tests can put those expectations programmatically, if an API lowers than 300ms, timeout error can happen, if a method becomes slow, it ran report, if network retries more than 5 times to succeed, it can alert. It's much easier to quantify those expectations in automated tests.

3. Thirdly, human is extremely slow and developers' time is precious. For me, when I manually test a feature, I need to switch to the branch, start the deployment, click my mouse on browser, try for a particular sequence of operations, and eye ball the result. This could easily cost me 10+ minutes. With the same 10 minutes, automation test can run thousands of cases, with different OS, env, different inputs, scenarios. It is not scalable with manual testing, and it eats up developers' time, which we could have saved for many many other aspects.

So as a conclusion, manual test 1) is single time verification, 2) can cover very little expectations, and 3) can be costly and waste developers' time. If your impression is "automated tests cannot cover all cases, I always find more issues when I manually test it," it is because you did not have enough automated tests written, you should go back write more automated tests. Remember human cannot beat a machine's speed to verify things, if you find something machine forget to check, just ask the machine to check it for you in the future, instead of doing it by yourself manually. Of course, you may need to first manually test it, to discover the gap that machine forget to test.

But with all of those stated, I am NOT saying manual test is not important. Especially many UI features needs to be manually tried on a browser, to see if it is easy to use, good looking, etc. My view is, we should still do manual testing, but we need to

1. do it on top of automated tests, as additional verifications;
2. do it in a smarter way to save time.

So here is my proposal:

1. Not every single PR needs manual tests. Not worth the time. It is a balance of risk and efficiency. If you think some change is risky, and you think automated tests may have a gap, yes go ahead deploy everything and verify it. If you know some automation tests are verifying it, it is fine to minimize manual tests, just to save time. After a PR is merged, other developers while developing their features, if they encounter issues related to your change, can raise issues, this servers as post merge manual test as well. If so, go ahead fix it with new automated tests.

2. Go easy on development and be harsh on release. Main branch serves as the "developing branch", any code change goes onto it, will not affect users immediately, until a release is made. So, it is fine if main branch has bugs and issues to be fixed. It is fine if main branch contains half-baked features. It is fine if a merged PR is found problematic later: just revert it or fix forward, no big deal. When we cut a release, we can be more strict. In addition to automation tests, we can start to do more manual tests to verify the release, that's why we have release candidates for you to test, verify and vote, and people vote -1 in case any issues are found during RC preparation phase. We could also organize bug bash events during release windows, and use a dedicated 1 hour to manually verify things together, and we only need to do this before release, not every day.

3. PR authors do manual testing, reviewers optional. By default, PR authors should write automated tests, and in addition do some manual tests when preparing a PR, so that you know your code works as expected. PR reviewers should verify "evidence", instead of verify from manual tests. For instance, reviewer should check if automated tests are written, from the tests, are the source code behaving as described? If author did some manual tests, what are the evidence (e.g., urls for testing conducted, screenshots/gifs of the feature in action)? Reviewers review the produced artifacts, evaluate the risk of the change vs the benefit of the change. Reviewers can push back to ask authors provide more evidences, including do more manual tests or write more automated tests. Again, I want to reduce reviewers' burden, and encourage more ppl to help review each other's code. If we require reviewer to manually verify every PR, then it is a discouragement, a waste of reviewer's time, and it is too slow to review. The process is just too heavy to afford, and eventually there will be no one willing to review any code, and the community will stop moving.

4. Automation tests should cover more env, and closer to production. We need to make automated tests more similar or even equal to production environment, so that there will be less scenarios that "in my local environment but failed after deployment". Align with this purpose, we need staging server, which mimics the setting (as close as possible) of production env, and write some tests against staging environment, or conduct manual bug bash on the staging server.

5. Improve our automation tests/CI. We need to increase unit test coverage, add more integration tests, performance test, pressure tests, etc. We need to make things automated, instead of fallback to manual effort. I know we are far away from it, which is a fundamental reason why we are still having a lot of manual test need. Again, if you find a gap, write a script to verify it repeatedly. Every line of test you write, can save other developers hours to verify manually. '

6. Let users be a source of tester as well! In software development, there are phases like alpha tests, beta tests. Many bugs are indeed to be found by corner cases where only to be discovered in extensive usage (by actual users). Software has bugs, and users encounter and report them. It is natural to have something does not work, there is no perfect software in the world. developers' time are precious. we are only a community of 20ish human beings. How to use our time efficiently is important.

All in all, I hope we can elevate our overall productivity, and I believe manual testing is one giant burden we have had in the past that prevent us from doing things quicker. I know we had been doing only manual tests for a long time, and we are eating our own debt by moving ahead very very slowly: with the complexity of the project goes up, more features and more environments to support, it is harder and harder to ship new features and manually verify it works everywhere, every time, and did not break any existing features. Frankly speaking, with the old pace, this project will not last long and will soon be outdated.

It's 5 am in the morning, I typed all of those word by word, 0 AI used. I really mean it with my heart.

[chenlica]

Thanks to @Yicong-Huang for sharing these thought at 5 am in the morning :-)

I agree with those points. In general, we need to balance between development speed and quality control. AI tools make it easier to raise PRs, and we don't want the review to slow down the process. AI tools also make it easier to fix bugs. We need to gradually improve the automated testing framework. The "main" branch can go fast, and we use releases to do more thorough testing before shipping the binaries to users.

Since we have community contributors who are not committers, some of them (especially those new to open source development) need to be taught how to do testing properly. The corresponding committers can ask the authors to do testing properly. In this way, reviews don't become a huge burden for the reviewers.

We will constantly learn and improve our project management.