Project

General

Profile

action #76987

openQA Project - coordination #64746: [saga][epic] Scale up: Handle large storage efficiently to be able to run current tests efficiently but keep big archives of old results

re-encode some videos from existing results to save space

Added by okurz 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2020-11-04
Due date:
% Done:

0%

Estimated time:

Description

mkittler suggested:

"If we use av1 with the ffmpeg command mentioned in the ticket it will take forever to see any improvement. It is just very slow. So we could use the ffmpeg command for VP9:
ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -pix_fmt yuv420p -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0
Likely we don't need the color space conversion if we encode the existing theora steam."


Related issues

Related to openQA Infrastructure - coordination #68923: [epic] Use external videoencoder in production auto_review:"External encoder not accepting data"Blocked2020-11-13

History

#1 Updated by okurz 3 months ago

  • Due date set to 2020-11-10
  • Status changed from In Progress to Feedback

I picked an arbitrary example on osd: /results/testresults/04941/04941557-sle-15-SP3-Online-E-Staging-x86_64-BuildE.65.1-minimal+base@64bit

But first installed "ffmpeg" on osd. After some minutes the video in the above job seems to have vanished :D

Picking another example /results/testresults/04941/04941850-sle-15-SP3-JeOS-for-kvm-and-xen-x86_64-Build20.107-jeos-filesystem@uefi-virtio-vga and starting:

time ffmpeg -i video.ogv -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0 video.webm

this ran with around 1.7fps so quite low but ok for trying to optimize as much as possible. With nice -n 19 ionice -c 3 … -cpu-used 8 I get around 3-5fps.

Thinking of running the following:

find -name 'video.ogv' -exec nice -n 19 ionice -c 3 sh -c 'ffmpeg -i {} -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 8 ${1%.ogv}.webm && rm {}' _ {} \;

Running it now in a screen session of user "geekotest" on osd. It is obviously not occupying more than one core, could be done in parallel as well.

#2 Updated by okurz 2 months ago

openqa:/var/lib/openqa/testresults # find -wholename '049*/*/*video.webm' could not find any hits. I guess re-encoding to VP9 in a single process on osd in the background is slower than we clean out older jobs :D

So I call this experiment successfully failed ;)

#3 Updated by okurz 2 months ago

  • Related to coordination #68923: [epic] Use external videoencoder in production auto_review:"External encoder not accepting data" added

#4 Updated by okurz 2 months ago

  • Status changed from Feedback to Resolved
  • Parent task set to #64746

#5 Updated by okurz 2 months ago

  • Due date deleted (2020-11-10)

Also available in: Atom PDF