Project

General

Profile

coordination #68794

[qe-core][functional][epic] rework postfail hooks

Added by szarate over 2 years ago. Updated 11 days ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Refactor/Code Improvements
Target version:
-
Start date:
2020-03-31
Due date:
% Done:

17%

Estimated time:
(Total: 42.00 h)
Difficulty:

Description

From time to time, we end up having post fail hooks that take a long time, see kontact on live cd for instance:

[2020-06-19T07:53:36.894 CEST] [debug] ||| starting kontact tests/x11/kontact.pm
...
[2020-06-19T08:30:32.577 CEST] [debug] post_fail_hook failed: command 'curl --form upload=@problem_detection_logs.tar.xz --form upname=kontact-problem_detection_logs.tar.xz http://10.0.2.2:20103/H_2GxrpPuX6zL3oK/uploadlog/problem_detection_logs.tar.xz' timed out at /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/lib/opensusebasetest.pm line 212.

[2020-06-19T08:30:32.580 CEST] [debug] Loading a VM snapshot lastgood

If my math is right, that's 30 minutes spent on collecting logs, that for one instance could be greatly reduced by switching to serial terminal, and using scripts rather than typing all of the commands.

Other improvements would be needed, including making openQA display the time spent in a test, even if it fails, which we currently don't have.

Further read


Subtasks

action #65040: [sle][functiona][u] enhance post_fail_hook on OOM conditionResolveddheidler

action #120720: [qe-core] Use Utils::Logging::save_ulog to redefine upload_logs in opensusebasetestWorkable

action #120723: [qe-core] Guarantee `show_tasks_in_blocked_state` will always be calledWorkable

action #120729: [qe-core] *basetest::post_fail_hook: leave early if `NOLOGS` is setIn Progressrfan1

action #120735: [qe-core] PoC: Decrease typing on `post_fail_hooks` log collectionIn Progresspdostal

action #120882: [qe-core] ensure `os-autoinst` reports properly the execution time of `post_fail_hooks`Workable


Related issues

Related to openQA Tests - action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some casesWorkable2022-02-18

Related to qe-yam - action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its actionNew2022-01-19

Related to openQA Tests - action #109792: [qe-core] Offline extraction of logs from Serial consoleResolved2022-07-21

History

#1 Updated by szarate about 2 years ago

  • Related to action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition added

#2 Updated by szarate about 2 years ago

  • Related to deleted (action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition)

#3 Updated by szarate about 2 years ago

  • Related to action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some cases added

#4 Updated by szarate about 2 years ago

  • Tracker changed from action to coordination
  • Difficulty deleted (medium)

#6 Updated by tjyrinki_suse about 2 years ago

  • Subject changed from [functional][u][epic] rework postfail hooks to [qe-core][functional][epic] rework postfail hooks

#7 Updated by okurz about 1 year ago

I came to this ticket due to periodically reviewing tickets as described on https://progress.opensuse.org/projects/openqatests/wiki#How-we-work-on-tickets

This ticket was set to "Normal" priority but was not updated within the SLO period for "Normal" tickets (365 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives

First reminder: Please consider picking up this ticket within the next 365 days or just set the ticket to the next lower priority of "Low" (no SLO related time period).

#8 Updated by tjyrinki_suse 12 months ago

  • Status changed from New to Resolved

All sub-tasks done, is this finished enough?

#9 Updated by szarate 12 months ago

  • Status changed from Resolved to Feedback

tjyrinki_suse wrote:

All sub-tasks done, is this finished enough?

Nah, we've done some stuff, and recently finally I had some time to add the time spent in failed tests but still there are too many post fail hooks that keep the console (mostly either log or root console), which uses considerably more time.

I'll look into refining better what this epic is about, and link subsequent related pr's

#10 Updated by szarate 4 months ago

  • Related to action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its action added

#11 Updated by szarate 4 months ago

  • Related to action #109792: [qe-core] Offline extraction of logs from Serial console added

#12 Updated by szarate 24 days ago

  • Sprint set to QE-Core: November Sprint (Nov 09 - Nov 22)

One example of part of those reworks: Ensure yas2-logs is always installed: https://suse.slack.com/archives/C02CSAZLAR4/p1668008178667569

#14 Updated by szarate 13 days ago

  • Description updated (diff)
  • Status changed from Feedback to Blocked

Also available in: Atom PDF