Project

General

Profile

action #93324

Exception in log parser leads to failed job with hard to guess root cause

Added by asmorodskyi about 2 months ago. Updated about 1 month ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Concrete Bugs
Target version:
Start date:
2021-06-01
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

http://autobot.qa.suse.de/tests/2436 - this job has state failed , and one of modules has result Passed another has result None.
Actual reason of failure is that log parsing for External Result tab has failed because of changes in input log file for parsing . in os-autoinst log you may see following error :

[0mUse of uninitialized value $t_name in pattern match (m//) at /usr/share/openqa/lib/OpenQA/Parser/Format/IPA.pm line 42.
    OpenQA::Parser::Format::IPA::parse(OpenQA::Parser::Format::IPA=HASH(0x55dc86d92e00), "{\"tests\": [{\"outcome\": \"passed\", \"test_index\": 0, \"nodeid\": \""...) called at /usr/share/openqa/lib/OpenQA/Parser.pm line 68
    OpenQA::Parser::load(OpenQA::Parser::Format::IPA=HASH(0x55dc86d92e00), "ulogs/img_proof-20210518103037.results") called at /usr/share/openqa/lib/OpenQA/Parser.pm line 43
    OpenQA::Parser::parser("IPA", "ulogs/img_proof-20210518103037.results") called at /usr/lib/os-autoinst/testapi.pm line 2108
    eval {...} called at /usr/lib/os-autoinst/testapi.pm line 2105
    testapi::parse_extra_log("IPA", "img_proof_results/ec2/ami-0ae7dcf6b1b1188dd/i-06bf1da3a81e2f0"...) called at sle/tests/publiccloud/img_proof.pm line 131
    img_proof::run(img_proof=HASH(0x55dc84eecc58)) called at /usr/lib/os-autoinst/basetest.pm line 356
    eval {...} called at /usr/lib/os-autoinst/basetest.pm line 350
    basetest::runtest(img_proof=HASH(0x55dc84eecc58)) called at /usr/lib/os-autoinst/autotest.pm line 374
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 374
    autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 242
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 242
    autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 298
    autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x55dc858a43a0)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x55dc858a43a0), CODE(0x55dc8563ad20)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 477

Suggestion

  • Check if img-proof 7.3.0 is broken or regressed somehow
  • See what else may have changed wrt generation of .results
  • Reproduce with other tests
autoinst-log.txt (470 KB) autoinst-log.txt asmorodskyi, 2021-06-01 07:50

Related issues

Copied to openQA Project - action #93342: Exception in log parser should Incomplete if the parser can't parse or throws an exceptionWorkable2021-06-01

History

#1 Updated by VANASTASIADIS about 2 months ago

  • Category set to Concrete Bugs
  • Target version set to Ready

#2 Updated by cdywan about 2 months ago

  • I would also expect it to be Incomplete if the parser throws an exception.
  • The parser should be able to parse img_proof-20210518103037.log

So that's two AC, maybe two tickets if it's more involved? I'd probably start the parser fix with a reproducer based on the log file

#3 Updated by cdywan about 2 months ago

It seems like name is not contained in the results:

{"tests": [{"outcome": "passed", "test_index": 0, "nodeid": "test_soft_reboot"}, {"nodeid": "test_sles_license.py::test_sles_license[paramiko://18.192.182.175]", "lineno": 3, "outcome": "passed", "keywords": ["test_sles_license[paramiko://18.192.182.175]", "SLES", "paramiko://18.192.182.175", "test_sles_license.py"], "setup": {"duration": 

I think we need to confirm what a successful run looks like. There's only 2 in total here and the previous one doesn't have a .results file The successful (but failed) run here has a name field in the JSON unlike the broken case: https://openqa.suse.de/tests/5988889

Also img-proof=7.0.0 vs img-proof=7.3.0 which presumably changed the format

#4 Updated by cdywan about 2 months ago

  • Copied to action #93342: Exception in log parser should Incomplete if the parser can't parse or throws an exception added

#5 Updated by cdywan about 2 months ago

  • Description updated (diff)
  • Status changed from New to Workable

#6 Updated by cdywan about 2 months ago

  • Priority changed from Normal to Low

#7 Updated by mkittler about 2 months ago

Note that the backtrace from the ticket description is actually just a warning. Of course the parser code shouldn't produce any warnings (regardless of the input). It should either cope with some data being missing or should emit a real error.

#8 Updated by okurz about 1 month ago

  • Target version changed from Ready to future

Also available in: Atom PDF