Project

General

Profile

Actions

action #77842

closed

openQA Project - coordination #103938: [saga][epic] Scale up: Efficient handling of large storage on o3

coordination #68923: [epic] Use external videoencoder in production auto_review:"External encoder not accepting data"

Use external videoencoder in production on all o3 machines size:M

Added by okurz over 3 years ago. Updated 12 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2020-11-13
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Motivation

See #68923

Acceptance criteria

  • AC1: All o3 workers run the same external video encoder configured (exceptions allowed if required and documented) as in #77839
  • AC2: All o3 workers are not overstressed by the external video encoder, i.e. not more jobs failing or incompleting due to an overstressed worker

Suggestions

  • Research what has been done in #77839
  • Ensure ffmpeg is installed on all worker machines (should be done on all of them as as of #77842#note-6)
  • Configure external video encoder as documented in https://progress.opensuse.org/issues/68923#note-17, just see in #77842-10
  • Monitor the CPU usage of ffmpeg e.g. via htop and also the overall performance of each worker, e.g. check for incomplete jobs due to performance issues
  • Wait for feedback for some days

Related issues 3 (1 open2 closed)

Related to openQA Project - action #129673: Change the default of the video encoder options in os-autoinstNew2023-05-22

Actions
Copied from openQA Infrastructure - action #77839: Use external videoencoder in production on one o3 workerResolvedokurz2020-11-13

Actions
Copied to openQA Infrastructure - action #77845: Use external videoencoder in production on all osd machines size:MResolvedokurz2020-11-13

Actions
Actions #1

Updated by okurz over 3 years ago

  • Copied from action #77839: Use external videoencoder in production on one o3 worker added
Actions #2

Updated by okurz over 3 years ago

  • Copied to action #77845: Use external videoencoder in production on all osd machines size:M added
Actions #3

Updated by okurz over 3 years ago

  • Tracker changed from coordination to action
Actions #4

Updated by okurz almost 3 years ago

  • Status changed from Workable to New

moving all tickets without size confirmation by the team back to "New". The team should move the tickets back after estimating and agreeing on a consistent size

Actions #5

Updated by okurz almost 3 years ago

  • Target version changed from Ready to future
Actions #6

Updated by mkittler over 2 years ago

  • Description updated (diff)

I needed to install a new package on o3 workers anyways so I went ahead and installed ffmpeg-4 on all workers.

Actions #7

Updated by mkittler over 2 years ago

  • Description updated (diff)
Actions #8

Updated by okurz about 1 year ago

  • Tags set to infra
Actions #9

Updated by okurz about 1 year ago

  • Target version changed from future to Ready
Actions #10

Updated by okurz 12 months ago

$ ssh o3
Last login: Fri May 12 06:59:23 2023 from scar.infra.opensuse.org
okurz@ariel:~> for i in aarch64 openqaworker4 openqaworker7 openqaworker19 openqaworker20 qa-power8-3 rebel; do echo $i && ssh root@$i "grep EXTE /etc/openqa/workers.ini" ; done
aarch64
openqaworker4
openqaworker7
openqaworker19
EXTERNAL_VIDEO_ENCODER_CMD=ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -pix_fmt yuv420p -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0
EXTERNAL_VIDEO_ENCODER_OUTPUT_FILE_EXTENSION=webm
#EXTERNAL_VIDEO_ENCODER_ADDITIONALLY=1
openqaworker20
EXTERNAL_VIDEO_ENCODER_CMD=ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -pix_fmt yuv420p -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0
EXTERNAL_VIDEO_ENCODER_OUTPUT_FILE_EXTENSION=webm
#EXTERNAL_VIDEO_ENCODER_ADDITIONALLY=1
qa-power8-3
rebel

so we already use the external video encoder on w19+20, just use the same on the other machines.

Actions #11

Updated by okurz 12 months ago

  • Subject changed from Use external videoencoder in production on all o3 machines to Use external videoencoder in production on all o3 machines size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #12

Updated by dheidler 12 months ago

  • Related to action #129673: Change the default of the video encoder options in os-autoinst added
Actions #13

Updated by mkittler 12 months ago

  • Assignee set to mkittler
Actions #14

Updated by mkittler 12 months ago

  • Status changed from Workable to Feedback

I've ensured ffmpeg-4 is installed on all workers (it was missing on some) and configured the external video encoder on all of them (using the command line that has already been configured on workers 19 and 20). I'll checkout whether new jobs look still good.

Actions #15

Updated by mkittler 12 months ago

It looks generally still good.

Actions #16

Updated by mkittler 12 months ago

It looks still good.

I've just fixed the worker aarch64 where one of the cache services failed and thus no tests ran. So I'll keep this a little bit longer in feedback to wait until more jobs on that worker are executed.

Note that all the PowerPC jobs I've found have NOVIDEO=1 set. That means this change does not make a difference for the architecture/qa-power8-3 (for the better or worse).

Actions #17

Updated by mkittler 12 months ago

  • Status changed from Feedback to Resolved

This seems to work. If there were bigger problems we would know by now.

Actions

Also available in: Atom PDF