Log per test
|Target version:||QA - future|
All the logs are into one file, called autoinst-log.txt. This single file contains all the logs for the tests that took part during a build. As you can probably imagine this file is rather big, sometimes up to 20K lines long. Inside of it, you can find all sorts of messages; no doubt about it. But ... I would like to propose to create log per test (aka
*.pm) which is imho much more helpful and meaningful.
This can be done by extracting the text between the
||| starting and
||| finished placeholders.
10:44:49.1221 26680 ||| starting boot_to_desktop tests/boot/boot_to_desktop.pm at 2016-11-18 10:44:49 ... (extract) ... (this) ... (part) 10:45:21.5714 26680 ||| finished boot_to_desktop boot at 2016-11-18 10:45:21 (32 s)
As soon as you have successfully extracted the correct portion of the log file, you can save it as
$test.log (in that case it should be:
Then repeat the same procedure for all the tests which run during the build and save each log separately.
Attention : When a test fails, the placeholder for signaling the end of the test, is not the same. Expected keywords are
starting=1; # Number of the line that the test log starts finished=1 # Number of the line that the test log finishes while IFS='' read -r line || [[ -n "$line" ]]; do if [[ $line == *"starting $testname "* ]] then line_start=$starting fi if [[ $line == *"finished $testname "* ]] || [[ $line == *"$testname died"* ]] || [[ $line == *"$testname failed"* ]] then line_finish=$finished fi starting=$[$starting +1] finished=$[$finished +1] done < "$file"
PS: Most probably, the back-end developers have another (better) way of doing this, without looking for
placeholders, since this kind of wording can be changed anytime.
#5 Updated by okurz about 3 years ago
please keep in mind the glossary of openQA as defined here: https://progress.opensuse.org/projects/openqav3/wiki#Glossary
You are talking about a "test module", not a test. In general what one wants to conduct as "test case" is either a "test module" which is executed as a "test step" or a "test suite" which is conducted as scenarios.
Please think about if you can restructure your feature request based on the template proposal in https://progress.opensuse.org/projects/openqav3/wiki#feature-requests
I for myself have no problems searching for specific keywords within the big logfile. Splitting the logfile per module would not provide a big benefit but would make understanding the full flow of scenario way harder, especially because the test steps are not independant.
I suggest to close as "rejected" but I leave that to you for consideration for now. Thank you for understanding.
#6 Updated by pgeorgiadis about 3 years ago
@okurz thanks for the links! From now on I will remember to use the correct terminology ;) FYI: I tried to edit the current feature request, but I see no 'edit' button :/
Please try to see this from a tester's point of view. As a tester, I have not the luxury to search into a big log file, plus the fact that there are also irrelevant information (internal engine logs, backend mechanism). From the other hand, these data are relevant to you as the developer of the tool. So having two different aspects, I propose to keep them both, so every one is satisfied: one big file, and isolated logs per test. If I may, a good place to put them would be under the name of each test-module, but that's just a stylistic reference.
As for the flow of the scenarios, I don't see any impact on that. If someone would like to see the test plan, he could use the webui to see what test-modules are included and which job-groups are related to them. Looking at the logs for understanding the test-plan and the full flow of a scenario is wrong. Furthermore, the vast majority of test steps for QAM tests-modules are completely independent -- apart from the installation.
#7 Updated by szarate about 3 years ago
@okurz, @panos, please see the linked task and my last comment on #14972. Which is the way this can be addressed, i see value in the feature, but it will need to be discussed further to take into account the linked feature requests created and related to this one.
#10 Updated by pgeorgiadis about 3 years ago
On Tuesday, 24 January 2017 11:03:26 CET you wrote:
@okurz thanks for the links! From now on I will remember to use the correct
terminology ;) FYI: I tried to edit the current feature request, but I see
no 'edit' button :/
try the little pencil icon
I've tried but it prompts me to a new message, while I want to edit the already posted one.
The link of the little pencil icon is this: https://progress.opensuse.org/issues/16166/edit
I can edit only my replies to that issue, not the original (first) message that describes the issue :/
#12 Updated by email@example.com about 3 years ago
On Tuesday, 24 January 2017 16:41:31 CET you wrote:
I've tried but it prompts me to a new message, while I want to edit the
already posted one. The link of the little pencil icon is this:
I can edit only my replies to that issue, not the original (first)
message that describes the issue :/
because you were obviously never added as a member to the project. I thought
your QAM PM should do it? It might make sense to check with your colleagues if
they are also already member. If not ask to be added, e.g. to me.
#13 Updated by okurz about 3 years ago
- Status changed from New to Rejected
feel free to reopen with an updated description which describes the feature with a proper goal and user story. Otherwise I fear we might just be stuck in discussion about implementation details when we do not even understand the "why".