Staging openQA instances
Recently, some breakage has been introduced in openqa.opensuse.org by externally contributed tests ("externally" meaning me in this case ;-) ). External contributors cannot afford to verify the test in all the scenarios run by openqa.opensuse.org, so this will happen in the future again. It would be useful to use tortuga or any other machine to run a subset of the scenarios run in the official instance to make sure the new tests can be deployed with some degree of confidence.
Updated by okurz over 7 years ago
- Status changed from New to Resolved
any "staging openQA instance" would not be able to run everything which is run in production. It just does not scale. So anyway only a subset can be run and there can be always something missing. Also, we don't have the hardware capacity to cover everything twice and also consider SLE plus openSUSE. Our DOD should cover some important steps so that external contributors are motivated to test something locally first. We have a good test review process and it has to be decided by the reviewer if he accepts the risk of a new test with or without a local verification and covering which scenarios. Depending on the contributors it might make sense to setup a staging server with a subset of tests which is used by multiple test developers to share the burden of openQA setup and administration. E.g. since recently also the YaST team has one available: https://wiki.microfocus.net/index.php/YAST/openQA
I added a paragraph on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Test-development-instances-staging-openQA-instances to describe this (mainly same text as this comment). With this, I would like to resolve this ticket.