Project

General

Profile

Actions
Copy link

coordination #68794

open

[qe-core][functional][epic] rework postfail hooks

Added by szarate over 4 years ago. Updated 12 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
szarate
Category:
Refactor/Code Improvements
Target version:
-
Start date:
2020-03-31
Due date:
% Done:

71%

Estimated time:
(Total: 42.00 h)
Difficulty:
Sprint:
QE-Core: February Sprint (Feb 08 - Mar 08)
Tags:
platform-team

Description

From time to time, we end up having post fail hooks that take a long time, see kontact on live cd for instance:

[2020-06-19T07:53:36.894 CEST] [debug] ||| starting kontact tests/x11/kontact.pm
...
[2020-06-19T08:30:32.577 CEST] [debug] post_fail_hook failed: command 'curl --form upload=@problem_detection_logs.tar.xz --form upname=kontact-problem_detection_logs.tar.xz http://10.0.2.2:20103/H_2GxrpPuX6zL3oK/uploadlog/problem_detection_logs.tar.xz' timed out at /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/lib/opensusebasetest.pm line 212.

[2020-06-19T08:30:32.580 CEST] [debug] Loading a VM snapshot lastgood

If my math is right, that's 30 minutes spent on collecting logs, that for one instance could be greatly reduced by switching to serial terminal, and using scripts rather than typing all of the commands.

Other improvements would be needed, including making openQA display the time spent in a test, even if it fails, which we currently don't have.

Further read

Subtasks 8 (3 open — 5 closed)

action #65040: [sle][functiona][u] enhance post_fail_hook on OOM conditionResolveddheidler2020-03-31

Actions
action #112547: [qe-core] test fails in groupwise - make coredumps more visibleResolvedrfan12022-06-15

Actions
action #120720: [qe-core] Use Utils::Logging::save_ulog to redefine upload_logs in opensusebasetestNew2022-11-18

Actions
action #120723: [qe-core] Guarantee `show_tasks_in_blocked_state` will always be calledResolveddzedro2022-11-18

Actions
action #120729: [qe-core] *basetest::post_fail_hook: leave early if `NOLOGS` is setResolvedrfan12022-11-18

Actions
action #120735: [qe-core] PoC: Decrease typing on `post_fail_hooks` log collectionResolveddvenkatachala2022-11-18

Actions
action #120882: [qe-core] ensure `os-autoinst` reports properly the execution time of `post_fail_hooks`Blocked2023-01-19

Actions
action #123349: [qe-core] Find a better example of wrongly reported execution time of `post_fail_hooks`New2023-01-19

Actions

Related issues 4 (1 open — 3 closed)

Related to openQA Tests - action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some casesFeedback2022-02-18

Actions
Related to qe-yam - action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its actionRejected2022-01-19

Actions
Related to openQA Tests - action #109792: [qe-core] Offline extraction of logs from Serial consoleResolvedzluo2022-07-21

Actions
Related to qe-yam - action #124646: [ALP] consider using y2log-util for any yast testsRejected2023-02-15

Actions
Actions
Copy link
 #1

Updated by szarate about 4 years ago

  • Related to action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition added
Actions
Copy link
 #2

Updated by szarate about 4 years ago

  • Related to deleted (action #65040: [sle][functiona][u] enhance post_fail_hook on OOM condition)
Actions
Copy link
 #3

Updated by szarate about 4 years ago

  • Related to action #71602: [qe-core][functional][osd] default "gnome" test suite on osd has no video due to max_job_time above default which is only necessary in some cases added
Actions
Copy link
 #4

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
  • Difficulty deleted (medium)
Actions
Copy link
 #6

Updated by tjyrinki_suse about 4 years ago

  • Subject changed from [functional][u][epic] rework postfail hooks to [qe-core][functional][epic] rework postfail hooks
Actions
Copy link
 #7

Updated by okurz about 3 years ago

I came to this ticket due to periodically reviewing tickets as described on https://progress.opensuse.org/projects/openqatests/wiki#How-we-work-on-tickets

This ticket was set to "Normal" priority but was not updated within the SLO period for "Normal" tickets (365 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives

First reminder: Please consider picking up this ticket within the next 365 days or just set the ticket to the next lower priority of "Low" (no SLO related time period).

Actions
Copy link
 #8

Updated by tjyrinki_suse almost 3 years ago

  • Status changed from New to Resolved

All sub-tasks done, is this finished enough?

Actions
Copy link
 #9

Updated by szarate almost 3 years ago

  • Status changed from Resolved to Feedback

tjyrinki_suse wrote:

All sub-tasks done, is this finished enough?

Nah, we've done some stuff, and recently finally I had some time to add the time spent in failed tests but still there are too many post fail hooks that keep the console (mostly either log or root console), which uses considerably more time.

I'll look into refining better what this epic is about, and link subsequent related pr's

Actions
Copy link
 #10

Updated by szarate over 2 years ago

  • Related to action #105085: [Research:24h] Identify which post_fail_hook is used by each module and document its action added
Actions
Copy link
 #11

Updated by szarate over 2 years ago

  • Related to action #109792: [qe-core] Offline extraction of logs from Serial console added
Actions
Copy link
 #12

Updated by szarate about 2 years ago

  • Sprint set to QE-Core: November Sprint (Nov 09 - Nov 22)

One example of part of those reworks: Ensure yas2-logs is always installed: https://suse.slack.com/archives/C02CSAZLAR4/p1668008178667569

Actions
Copy link
 #14

Updated by szarate about 2 years ago

  • Description updated (diff)
  • Status changed from Feedback to Blocked
Actions
Copy link
 #15

Updated by szarate almost 2 years ago

  • Sprint changed from QE-Core: November Sprint (Nov 09 - Dec 07) to QE-Core: December Sprint (Dec 07 - Jan 04)
Actions
Copy link
 #16

Updated by szarate almost 2 years ago

  • Sprint changed from QE-Core: December Sprint (Dec 07 - Jan 11) to QE-Core: January Sprint (Jan 11 - Feb 08)
Actions
Copy link
 #17

Updated by szarate almost 2 years ago

  • Sprint changed from QE-Core: January Sprint (Jan 11 - Feb 08) to QE-Core: February Sprint (Feb 08 - Mar 08)
Actions
Copy link
 #18

Updated by szarate over 1 year ago

  • Related to action #124646: [ALP] consider using y2log-util for any yast tests added
Actions
Copy link
 #19

Updated by szarate about 1 year ago

  • Subtask #112547 added
Actions
Copy link
 #20

Updated by szarate about 1 year ago

  • Parent task set to #124625
Actions
Copy link
 #21

Updated by szarate about 1 year ago

  • Tags set to platform-team
Actions
Copy link

Also available in: Atom PDF