action #60173
openImprove the openQA documentation
0%
Description
The current openQA documentation is imho still lacking in some regards, as I (as a total beginner) am feeling completely lost when I need to get something done in openQA.
My problems with the documentation:
It needs a better introduction that explains the basics, a bit like http://open.qa/docs/#_basic_concepts but with less details (the needle description doesn't belong there, the glossary is just far too long).
It needs comprehensive tutorials for various use cases:
1) I just want to modify a single test, how do I run it?
2) I want to add a new test suite (or a test group, or something that shows up as a single item on o3), how can I do that and what do I need to modify?
3) I want to write actual GUI tests and create new needles, let openQA click things & read stuff from the screen.It is completely intransparent how tests are actually run: there are job groups and test suites. Test suites appear to contain job groups, but how is it decided which tests are run in a job group? E.g. I tried to find out what runs
create_hdd_gnome. But grepping the whole source tree of os-autoinst-distri-opensuse yields 0 hits.At least the openSUSE tests appear to be using a lot of "magic variables" where it is not apparent whether they are required, how they are set and what actually uses them. For example the
REPO_nvariables are automagically set on o3, but not if you start local tests. But your local tests just die if you don't have them set (or maybe it's because of something else, who knows).
My recommendation:
- Get someone fresh (= has never worked with openQA), give them a link to the docs and watch them read. Then after 10-20 minutes ask them whether they have any clue and if not, what's missing. Then try to address that, because most people will give up in far less than 10-20 min.
- Reorder the docs: http://open.qa/docs/#get-testing belongs definitely after http://open.qa/docs/#installing; http://open.qa/docs/#_triggering_tests_based_on_an_any_remote_git_refspec_or_open_github_pull_request is completely buried deep down in the docs and no one knows about it unless told by someone in
#opensuse-factory. - If possible, get a technical writer to help.
Updated by okurz about 5 years ago
- Priority changed from High to Low
And I thought I nailed it with the last doc rework ;P You are certainly right in most points. For a start, we will most likely not consider this as high priority so please understand when I set it to "Low" now. You have certain specific suggestions which one can start to work on right away, that helps. Thanks. Some topics we want to address by rethinking how tests are scheduled so it would not only be "documentation improvements".
Updated by MSirringhaus almost 4 years ago
I agree with the points raised here.
- Get someone fresh (= has never worked with openQA), give them a link to the docs and watch them read. Then after 10-20 minutes ask them whether they have any clue and if not, what's missing. Then try to address that, because most people will give up in far less than 10-20 min.
That is/was basically me today. What I wanted to do was to "quickly get an openQA instance running locally to write ONE test".
After spending the morning in the docs, all I can give you is a blank stare, I'm sorry to say.
It suffers from what I call "man page syndrome". It describes every little option and switch in great detail. But after reading that I still have no idea whats going on.
Each installation method is described in detail, each configuration option as well. But not how they interact with each other.
Which config steps are already done in the container bootstrap?
Do I have to call /usr/share/openqa/script/fetchneedles after bootstrapping or not?
Etc.
Once I had the web-UI running on http://localhost, I had no idea what to do next. Everything in the UI was empty.
After some heavy scrolling in the docs, I figured out that I probably needed ISOs.
Doc says "However, openQA also supports a remote-download feature of assets from trusted domains.", but not how (its in there somewhere, I'm sure. Described in great detail :-) ).
I also found myself going in an infinite loop between different sections, which referenced each other with "here are more infos on this".
At some point I just gave up.
Sorry, I can't give you a nice list of things to improve. This rather incoherent comment is all I can manage right now.
Dans initial list of improvements already sound like they might have helped me.
Updated by okurz almost 4 years ago
MSirringhaus wrote:
...
Thanks for your feedback. Appreciated :)
Updated by rainerkoenig over 3 years ago
dancermak wrote:
- Get someone fresh
I can relate to that. Started at SUSE beginning of August. Tried to get into openQA and there are so many frustrations and misunderstandings.
The current documentation is a reference material for those who know what is going on. But it is not a textbook to learn from.
First attempt was to bootstrap an systemd-nspawn container. Worked fine, was able to clone a job as my colleauges suggested. Seemed to work, but on the next day when restarting the machine I was there with questions "how do I start this container?" Even when I found a way to get it going again tests didn't work because "qemu exited unexpectedly". Ok, I "rm -rf /var/lib/machines/openqa1" and started over. Until the next day.
Then I tried a local installation without containers. This works so far, but I had to see all the videos that Dan Cermak did to find out how to populate test groups, test suites and so on.
What I'm really missing is a beginners tutorial. At the moment I feel like in pilot school and starting on a Boeing 747 instead of a Cessna.
Change the perspective. I have a system under test (SUT). Basic questions:
- Where does it come from? (Some installation tests create it from scratch, but anyway, where is defined what goes on it?) OpenQA has so many parameters and variables and I still have no clue what controls what will be tested on my SUT.
- Have a procedure how to "prepare the stage" for testing on the SUT.
- Then provide a simple test case that explains how openQA interacts with the SUT, how we send keys or clicks and how this needle thing is working. My first idea was that every test defines exactly what needle to use and then I learned, that needles are selected by tags. BTW: Do we have a tool where I can select a tag and then see all needles that fit?
- "reverse engineer" that basic test case. Tell the people what testapi calls you're doing and how this test is supposed to act.
- Give them a basic tour through the WebUI. How do I cancel jobs? How do I restart jobs? How can I look for failures? How can I adjust the Cleanup task so that the small hard disk in my computer does not get flooded.
- Provide a FAQ with some references about known issues. Example: I spent an afternoon in getting mad because my cloned SLE needles didn't work. At the end I could find out that the problem was, that I downloaded them to my /home and made a symlink under /var/lib/openqa... and it turned out that there is a problem when following symlinks over filesystem boundaries (thanks to the btrfs subvolumes).
I still feel that I'm making progress, but its awfully slow and difficult, too many secrets that you have to learn the hard way.