Project

General

Profile

coordination #34357

[epic] Improve openQA performance

Added by mkittler over 3 years ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2017-08-25
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

This is a general topic. The purpose of this ticket is mainly identifying performance issues which are slowing down work with openQA. It also already gives a few hints how we could deal with them (feel free to extend the list).

Tasks

  1. Identify performance issues in openQA.
    • Consider the 'test writer workflow', which involves restarting jobs quite often. Just try to create a new test and pay attention to performance issues in particular.
    • Beside the web UI, also scheduler and worker performance are important (eg. for restarting a job).
  2. Investigate the issues in detail.
    • Measure SQL performance.
    • Enable nytprof for server-side profiling. (Be careful, this tool generates lots of data!)
    • Use web-developer tools to for client-side profiling.
    • ...
  3. Fix issues
    • Avoid polling. (We already use websockets and similar ways to avoid it, but not in all places where it might make sense.)
    • Optimize database queries.
    • Make use of lazy loading where it makes sense.
    • ...

It is likely most appropriate to create separate sub tickets for tasks 2. and 3. when concrete issues are identified.

Issues I'm already aware of

  • DONE: The markdown renderer is very slow. It is the main reason pages containing lots of comments are slow. Adding a pagination on the group overview already provides a workaround. But still, why is it so horribly show? -> https://github.com/os-autoinst/openQA/pull/2232
  • DONE: Loading the 'All tests' page is also a bit slow. Note that enabling server-side rendering for this full-featured table didn't help much (when I tried it).
  • DONE: The time to restart a job has gotten worse since recent changes. This is annoying when having to restart a job quite often (locally). Definitely needs investigation.
  • DONE: The interactive mode is a mix of being broken and just slow.

Feel free to add more from your experience.


Subtasks

coordination #65402: [epic] Revamp test details page to improve loading times and prevent timeoutsResolvedmkittler

action #25814: load job page, e.g. test details, only on demandResolvedmkittler

action #32611: job details in browser windows do not automatically jump from "assigned" to "running" when they start - take 2Resolvedmkittler

action #66622: Live view - 'Autoscroll log' check box has no effect anymoreResolvedmkittler


Related issues

Related to openQA Project - coordination #64746: [saga][epic] Scale up: Efficient handling of large storage to be able to run current tests efficiently but keep big archives of old resultsBlocked2020-03-182021-10-28

History

#1 Updated by okurz over 2 years ago

  • Category set to Feature requests

#2 Updated by okurz over 1 year ago

  • Subject changed from Improve openQA performance to [epic] Improve openQA performance

#3 Updated by okurz over 1 year ago

  • Status changed from New to Blocked
  • Assignee set to okurz

I think we have improved the markdown rendering some months ago. IMHO it makes sense to work on #25814 first before continuing with other parts.

#4 Updated by mkittler over 1 year ago

  • Due date set to 2020-04-07

due to changes in a related task

#5 Updated by mkittler over 1 year ago

  • Due date set to 2017-08-25
  • Start date changed from 2020-04-07 to 2017-08-25

Grund: Änderung an zugehörigem Ticket: #65402

#6 Updated by kraih over 1 year ago

Yes, Markdown rendering was fixed last year and should not be causing any performance issues anymore. https://github.com/os-autoinst/openQA/pull/2232

#7 Updated by okurz over 1 year ago

  • Related to coordination #64746: [saga][epic] Scale up: Efficient handling of large storage to be able to run current tests efficiently but keep big archives of old results added

#8 Updated by okurz over 1 year ago

  • Description updated (diff)

#9 Updated by okurz over 1 year ago

  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)
  • Target version set to future

All present subtasks are fixed. We use more "lazy loading" a lot which makes the UI more responsive. I now in particular wonder about "Avoid polling" where we might see potential for this. Any specific ideas anyone?

#10 Updated by szarate about 1 year ago

  • Tracker changed from action to coordination
  • Status changed from Workable to New

Also available in: Atom PDF