QA » openQA Project » openQA Infrastructure

Careful, weekend incoming ;) So depending on your availability it can be safer to apply changes whenever you are available to monitor the effect more closely. Keep in mind that openSUSE never sleeps! :)

I know. I plan to check the job history of these workers tomorrow. (Should only take one minute and disabling the setting again is easy as well.)

Looks like apparmor (which is running on imagetester) prevented ffmpeg to start:

[2020-07-25T08:23:17.559 UTC] [debug] Launching external video encoder: ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0 '/var/lib/openqa/pool/1/video.webm'
sh: /usr/bin/ffmpeg: Permission denied

PR: https://github.com/os-autoinst/openQA/pull/3292

I'm currently running a job with the mentioned settings on a dedicated worker slot on imagetester and it works. So the apparmor PR worked. Whether there are performance problems is yet to be determined (and likely depends on the worker host). Due to my extra worker slot there are currently 3 tests running in imagetester and one of them encodes VP9. The CPU usage of the ffmpeg process ranges from 0 to 100 percent. So far it seems it is fast enough and doesn't disturb the actual jobs.

After the test has finished with no further problems I'm going to enable the VP9 encoder on both regular worker slots and disable the extra worker slot again. The video is truncated so I'm not enabling the encoder. (see https://openqa.opensuse.org/tests/1353163)

The truncated video can be locally reproduced as well. It can happen when the test execution just ends (isotovideo terminates itself; it is not aborted by the worker or Ctrl+C).

The problem is caused by not explicitly stopping (and waiting for) ffmpeg in that case. Of course I've thought about this problem when introducing the external video encoder and have even already implemented it: https://github.com/os-autoinst/os-autoinst/compare/master...Martchus:terminate-video-encoder?expand=1

However, I didn't propose that change after all because:

1. When isotovideo is stopped by the worker or Ctrl+C SIGTERM/SIGINT is sent to the video encoder process without the extra need to do it from our side.
2. When ffmpeg receives a 2nd signal to exit it will immediately exit without further finalizing the file.

These points together create an annoying problem: When isotovideo terminates itself (and therefore sends SIGTERM to the backend) the backend would need to send SIGTERM to ffmpeg and wait for its termination. When isotovideo is stopped by the worker or Ctrl+C we must not send SIGTERM to ffmpeg because that would mean ffmpeg receives 2 signals to exit (because of 1.) and would possibly leave the file unfinalized (because of 2.).

It is currently not clear to me how to solve this. I haven't found anything in ffmpeg's documentation to prevent 2. (which would be the easiest solution). The backend process would need to determine whether it received SIGINT/SIGTERM from isotovideo simply because the test execution ended or from the worker/shell which sends the exit signal to the entire process group. Only in the first case it would terminate ffmpeg.

Maybe it also works to close the pipe we use to send data to ffmpeg. Closing the pipe might cause ffmpeg to exist but would not interfere with a signal sent to ffmpeg by the worker/shell.

Another solution would be to explicitly terminate (and wait for) the process group within the worker - even if isotovideo itself terminated on its own. (That would at least help when isotovideo is executed within the worker. Getting rid of any potential leftover process is likely a good idea anyways.)

According to a reply in #ffmpeg on freenode closing the pipe is supposed to work and there's really no option to change ffmpeg's behavior (2.). So I'll try closing the pipe. Nevertheless I'll also improve the worker code to take care of the whole process group in any case.

PR for closing pipes: https://github.com/os-autoinst/os-autoinst/pull/1503

Improving the worker code to take care of the whole process group would likely be a little bit more work because it currently lacks a way to track the process group ID and only relies on calling getpgrp for the PID of the immediate child process. Likely one needed to use vfork() to keep track of the process group ID from the parent process side. (see https://en.wikipedia.org/wiki/Process_group)

The PR got merged. Do you have more items in mind for this?

Yes - the PR was just for fixing an issue which came up when testing this in production. What I actually want to do is finally enabling this in production. Therefore I'm trying my luck on imagetester again.

It seems to work nicely on both imagetester worker slots and my PR to fix warnings worked as well.

However, I've discovered that Firefox doesn't like the produced VP9 videos (although it is supposed to support VP9 within WebM in general). The videos themselves are definitely not truncated anymore. Maybe Firefox has problems with some particularity of the files produced via ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0.

Luckily it was just the color space which wasn't supported by Firefox. I extended the example command to add -pix_fmt yuv420p to the ffmpeg arguments: https://github.com/os-autoinst/os-autoinst/pull/1513

So since it generally works I suppose we can enable it on more workers.

For the record: To enable the external video encoder on further workers one has to install ffmpeg¹ and add the following two lines to /etc/openqa/workers.ini (as it is already configured on imagetester right now):

EXTERNAL_VIDEO_ENCODER_CMD=ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -pix_fmt yuv420p -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 0
EXTERNAL_VIDEO_ENCODER_OUTPUT_FILE_EXTENSION=webm

¹I've tested ffmpeg 3.x and 4.x as provided by Leap 15.1 and Tumbleweed.

The change takes effect after restarting the worker service(s). After that I would recommended to closely monitor the workers, e.g. checkout the job history for the worker slots on the web UI for incomplete jobs and also check whether the video looks good and is seekable. Also check for failed jobs because they might have failed due to a performance regression caused by the additional CPU load.

The ffmpeg parameters can be tweaked of course as needed. However, I'd like to stick with VP9 because it is currently the most modern, production-ready and royalty-free format which is available. It can be encoded and decoded under openSUSE without having to enable additional repositories. (See my previous comment for why -pix_fmt is required. More VP9 options are documented in the ffmpeg wiki.)