Project

General

Profile

Actions

action #18558

closed

[sle][functional][u][hard][investigation] Snapper tests run too long

Added by mkravec over 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Category:
Enhancement to existing tests
Target version:
SUSE QA (private) - Milestone 17
Start date:
2017-04-13
Due date:
2018-07-03
% Done:

0%

Estimated time:
Difficulty:

Description

Snapper cleanup runs for 54 minutes.
Snapper create runs for 18 minutes.

Combined they take too long time and cause job timeouts after 2 hours.

Acceptance criteria

  • AC1: snapper cleanup + create runs together on ppc64le take less time than ~50m

Reproducible

openQA test in scenario:

It failed for SP3 on Build 0304

But now the test module is only scheduled for test suite: osd#extra_tests_filesystem#latest

Further details


Related issues 1 (0 open1 closed)

Related to openQA Tests (public) - action #31351: [functional][u][medium] force_cron_run does not actually run any crons (occasionally)Resolvedzluo2018-02-032018-07-03

Actions
Actions #1

Updated by dgutu over 7 years ago

I do consider to increase MAX_JOB_TIME. Right now I don't see any possible way to changes the test logic.
If this fails on the ppc64le and not on other platform than maybe its not the test case a problem.

Opinions are welcome :)

Actions #2

Updated by mkravec over 7 years ago

Snapper cleanup runs about 40 minutes on x86_64 - maybe using virtio_terminal it could be finished much faster.

Actions #3

Updated by dgutu over 7 years ago

mkravec wrote:

Snapper cleanup runs about 40 minutes on x86_64 - maybe using virtio_terminal it could be finished much faster.

Sorry, no idea what is virtio_terminal. Paste some more info please, thx.

Actions #4

Updated by okurz@suse.de over 7 years ago

I do consider to increase MAX_JOB_TIME. Right now I don't see any possible
way to changes the test logic. If this fails on the ppc64le and not on
other platform than maybe its not the test case a problem.

Opinions are welcome :)

short-term: make tests work again -> increase timeout
mid-term: simplify tests as one module for such a long time is certainly not a
good idea

Actions #5

Updated by dgutu over 7 years ago

Applied short-term solution.

Actions #6

Updated by okurz over 7 years ago

In the the meantime we split extra tests into "extra_tests_filesystem" and "extra_tests_in_textmode"

Actions #7

Updated by dgutu over 7 years ago

  • Assignee changed from dgutu to okurz
Actions #8

Updated by okurz over 7 years ago

  • Category changed from Bugs in existing tests to Enhancement to existing tests
  • Status changed from New to In Progress
  • Assignee deleted (okurz)

next steps: simplify tests as one module for such a long time is certainly not a
good idea

Actions #9

Updated by okurz over 7 years ago

  • Subject changed from Snapper tests run too long to [sle][functional]Snapper tests run too long
Actions #10

Updated by SLindoMansilla over 7 years ago

Two months ago the module snapper_cleanup was not included anymore on this test:

https://openqa.suse.de/tests/957546

Actions #11

Updated by okurz about 7 years ago

  • Target version set to Milestone 16
Actions #12

Updated by riafarov almost 7 years ago

  • Status changed from In Progress to Workable
Actions #13

Updated by okurz over 6 years ago

  • Subject changed from [sle][functional]Snapper tests run too long to [sle][functional][u] Snapper tests run too long
  • Description updated (diff)
  • Due date set to 2018-07-03
  • Target version changed from Milestone 16 to Milestone 17
Actions #14

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 17
Actions #15

Updated by riafarov over 6 years ago

  • Subject changed from [sle][functional][u] Snapper tests run too long to [sle][functional][u][hard][investigation] Snapper tests run too long
Actions #16

Updated by SLindoMansilla over 6 years ago

  • Assignee set to SLindoMansilla
Actions #17

Updated by SLindoMansilla over 6 years ago

  • Related to action #31351: [functional][u][medium] force_cron_run does not actually run any crons (occasionally) added
Actions #18

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
  • Status changed from Workable to In Progress
Actions #19

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
Actions #20

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
Actions #21

Updated by SLindoMansilla over 6 years ago

  • Status changed from In Progress to Resolved

The test module is only scheduled for test suites

The issue didn't happen since at least 7 months:

We have also improvements in the old "force_cron_run" (see poo#31351)
which is going to be renamed (see poo#38093)
And this kind of performance issues will be handle on system performance scenarios (see poo#38090)

Actions #22

Updated by okurz over 6 years ago

  • Status changed from Resolved to In Progress

So how long do the test modules take? What performance issue do you mean?

Actions #23

Updated by SLindoMansilla over 6 years ago

With performance issue I refer each time some background tasks affect the current test.


  • [2018-07-02T19:47:33.0300 CEST] [debug] ||| starting snapper_cleanup
  • [2018-07-02T19:56:44.0716 CEST] [debug] ||| finished snapper_cleanup

~9m11s

  • [2018-07-02T20:01:07.0820 CEST] [debug] ||| starting snapper_create
  • [2018-07-02T20:04:12.0381 CEST] [debug] ||| finished snapper_create

~3m5s


Wait, I realized I am checking for x86_64, but AC mentions ppc64le and that scenario is not active. Activated scenario on Test Development: SLE_12_SP4 and tracking.

Actions #24

Updated by okurz over 6 years ago

good. With https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5222 snapper_cleanup might have changed its behaviour a lot so it might be that actually the goal is fulfilled already without further code changes but you are right to crosscheck this. Before closing this ticket please make sure we do not have "lost" snapper_cleanup on our relevant products. Could be that we have other tickets for that. I just want to make sure we don't forget something :)

Actions #25

Updated by okurz over 6 years ago

  • Status changed from In Progress to Resolved

I crosschecked that we have the test module "snapper_cleanup" executed on all current relevant products, e.g.:

Actions

Also available in: Atom PDF