Project

General

Profile

Actions

coordination #68794

open

[qe-core][functional][epic] rework postfail hooks

Added by szarate over 3 years ago. Updated 4 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Refactor/Code Improvements
Target version:
-
Start date:
2020-03-31
Due date:
% Done:

71%

Estimated time:
(Total: 42.00 h)
Difficulty:
Sprint:
QE-Core: February Sprint (Feb 08 - Mar 08)

Description

From time to time, we end up having post fail hooks that take a long time, see kontact on live cd for instance:

[2020-06-19T07:53:36.894 CEST] [debug] ||| starting kontact tests/x11/kontact.pm
...
[2020-06-19T08:30:32.577 CEST] [debug] post_fail_hook failed: command 'curl --form upload=@problem_detection_logs.tar.xz --form upname=kontact-problem_detection_logs.tar.xz http://10.0.2.2:20103/H_2GxrpPuX6zL3oK/uploadlog/problem_detection_logs.tar.xz' timed out at /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/lib/opensusebasetest.pm line 212.

[2020-06-19T08:30:32.580 CEST] [debug] Loading a VM snapshot lastgood

If my math is right, that's 30 minutes spent on collecting logs, that for one instance could be greatly reduced by switching to serial terminal, and using scripts rather than typing all of the commands.

Other improvements would be needed, including making openQA display the time spent in a test, even if it fails, which we currently don't have.

Further read


Subtasks 8 (3 open5 closed)

action #65040: [sle][functiona][u] enhance post_fail_hook on OOM conditionResolveddheidler2020-03-31

Actions
action #112547: [qe-core] test fails in groupwise - make coredumps more visibleResolvedrfan12022-06-15

Actions
action #120720: [qe-core] Use Utils::Logging::save_ulog to redefine upload_logs in opensusebasetestNew2022-11-18

Actions
action #120723: [qe-core] Guarantee `show_tasks_in_blocked_state` will always be calledResolveddzedro2022-11-18

Actions
action #120729: [qe-core] *basetest::post_fail_hook: leave early if `NOLOGS` is setResolvedrfan12022-11-18

Actions
action #120735: [qe-core] PoC: Decrease typing on `post_fail_hooks` log collectionResolveddvenkatachala2022-11-18

Actions
action #120882: [qe-core] ensure `os-autoinst` reports properly the execution time of `post_fail_hooks`Blocked2023-01-19

Actions
action #123349: [qe-core] Find a better example of wrongly reported execution time of `post_fail_hooks`New2023-01-19

Actions

Related issues 4 (1 open3 closed)

Related to openQA Tests - action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some casesWorkable2022-02-18

Actions
Related to qe-yam - action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its actionRejected2022-01-19

Actions
Related to openQA Tests - action #109792: [qe-core] Offline extraction of logs from Serial consoleResolvedzluo2022-07-21

Actions
Related to qe-yam - action #124646: [ALP] consider using y2log-util for any yast testsRejected2023-02-15

Actions
Actions #1

Updated by szarate over 3 years ago

  • Related to action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition added
Actions #2

Updated by szarate over 3 years ago

  • Related to deleted (action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition)
Actions #3

Updated by szarate over 3 years ago

  • Related to action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some cases added
Actions #4

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
  • Difficulty deleted (medium)
Actions #6

Updated by tjyrinki_suse over 3 years ago

  • Subject changed from [functional][u][epic] rework postfail hooks to [qe-core][functional][epic] rework postfail hooks
Actions #7

Updated by okurz over 2 years ago

I came to this ticket due to periodically reviewing tickets as described on https://progress.opensuse.org/projects/openqatests/wiki#How-we-work-on-tickets

This ticket was set to "Normal" priority but was not updated within the SLO period for "Normal" tickets (365 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives

First reminder: Please consider picking up this ticket within the next 365 days or just set the ticket to the next lower priority of "Low" (no SLO related time period).

Actions #8

Updated by tjyrinki_suse over 2 years ago

  • Status changed from New to Resolved

All sub-tasks done, is this finished enough?

Actions #9

Updated by szarate over 2 years ago

  • Status changed from Resolved to Feedback

tjyrinki_suse wrote:

All sub-tasks done, is this finished enough?

Nah, we've done some stuff, and recently finally I had some time to add the time spent in failed tests but still there are too many post fail hooks that keep the console (mostly either log or root console), which uses considerably more time.

I'll look into refining better what this epic is about, and link subsequent related pr's

Actions #10

Updated by szarate over 1 year ago

  • Related to action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its action added
Actions #11

Updated by szarate over 1 year ago

  • Related to action #109792: [qe-core] Offline extraction of logs from Serial console added
Actions #12

Updated by szarate over 1 year ago

  • Sprint set to QE-Core: November Sprint (Nov 09 - Nov 22)

One example of part of those reworks: Ensure yas2-logs is always installed: https://suse.slack.com/archives/C02CSAZLAR4/p1668008178667569

Actions #14

Updated by szarate over 1 year ago

  • Description updated (diff)
  • Status changed from Feedback to Blocked
Actions #15

Updated by szarate over 1 year ago

  • Sprint changed from QE-Core: November Sprint (Nov 09 - Dec 07) to QE-Core: December Sprint (Dec 07 - Jan 04)
Actions #16

Updated by szarate about 1 year ago

  • Sprint changed from QE-Core: December Sprint (Dec 07 - Jan 11) to QE-Core: January Sprint (Jan 11 - Feb 08)
Actions #17

Updated by szarate about 1 year ago

  • Sprint changed from QE-Core: January Sprint (Jan 11 - Feb 08) to QE-Core: February Sprint (Feb 08 - Mar 08)
Actions #18

Updated by szarate about 1 year ago

  • Related to action #124646: [ALP] consider using y2log-util for any yast tests added
Actions #19

Updated by szarate 4 months ago

  • Subtask #112547 added
Actions #20

Updated by szarate 4 months ago

  • Parent task set to #124625
Actions #21

Updated by szarate 4 months ago

  • Tags set to platform-team
Actions

Also available in: Atom PDF