QA » openQA Project » openQA Infrastructure

Disabled most openqa worker instances to reduce the load

for i in aarch64 openqaworker4 openqaworker7 power8 imagetester rebel; do echo $i && ssh root@$i "systemctl stop openqa-worker-auto-restart@\*" ; done

This is how strace from openqa webui processes looks like:

strace: Process 30934 attached
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0;\0dbdpg_p30934_2\0\0\0\0\3\0\0\0\010202"..., 82, MSG_NOSIGNAL, NULL, 0) = 82
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\3\242\0\37id\0\0\0@\205\0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 24576
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "6_64-Build20210512-kde-live_inst"..., 32706, 0, NULL, NULL) = 32706
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "eap_15.2_gnome\0\0\0\10opensuse\0\0\0\nTu"..., 32605, 0, NULL, NULL) = 32605
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "Build20210512-gpt@64bit\0\0\0\1f\0\0\0\4"..., 32706, 0, NULL, NULL) = 15617
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\v\254\0dbdpg_p30934_1b\0\0\0\1\r\0\0\0\00717"..., 3011, MSG_NOSIGNAL, NULL, 0) = 3011
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\0\362\0\tid\0\0\0@*\0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 10560
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0+\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\vboo"..., 66, MSG_NOSIGNAL, NULL, 0) = 66
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 598
getpid()                                = 30934
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0)\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\tpoo"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 554
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0+\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\vboo"..., 66, MSG_NOSIGNAL, NULL, 0) = 66
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 568
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0)\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\tpoo"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 577
getpid()                                = 30934
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0)\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\tpoo"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 595
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0)\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\tpoo"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 565
getpid()                                = 30934
getpid()                                = 30934
getpid()                                = 30934
getpid()                                = 30934
sendto(13<socket:[747058]>, "B\0\0\0)\0dbdpg_p30934_1\0\0\0\0\1\0\0\0\tpoo"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
poll([{fd=13<socket:[747058]>, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=13, revents=POLLIN}])
recvfrom(13<socket:[747058]>, "2\0\0\0\4T\0\0\1X\0\rid\0\0\0@ \0\1\0\0\0\27\0\4\377\377\377\377\0"..., 32768, 0, NULL, NULL) = 572
getpid()                                = 30934
getpid()                                = 30934

I wonder where the many "sendto" with "boo" and "poo" come from, could be just tests overview page.

I disabled AMQP in o3 /etc/openqa/openqa.ini assuming that maybe it's also problematic as it involves remote communication.

Disabled most content in /usr/share/openqa/templates/webapi/test/overview.html.ep

Jane suggested https://metacpan.org/pod/App%3a%3aStacktrace . seems like we do not have that package within openSUSE Leap so I installed perl-App-cpanminus gcc and then cpanm --notest App::Stacktrace as okurz and called sudo sh -c 'PERL5LIB=/home/okurz/perl5/lib/perl5 /home/okurz/perl5/bin/perl-stacktrace 7278' which aborted with Cannot access memory at address 0xffffffffae9b5970 so not progressed much.

disabled /usr/share/openqa/templates/webapi/main/group_overview.html.ep as well

Enabled all pages in openQA again, enabled amqp, enabled job done hooks, started openqa-scheduler.

Started one pipeline in https://gitlab.suse.de/openqa/auto-review/-/pipelines/140956 to handle the accumulated incomplete and failed.

Completed all rollback steps. So we have a rewrite rule excluding many users from working with the o3 webUI normally which we need to improve. Also we should conduct a "Five Why" analysis and come up with improvements, e.g. rate limiting on multiple levels.

ideas:

• only allow certain routes or query parameters for logged in users
• disable more costly query parameters by default, e.g. no comment, tag, label parsing
• use external tools to enforce rate limiting, e.g. in the apache proxy

Setting due date based on mean cycle time of SUSE QE Tools

Disabled the block patterns again for now, looks ok so far, monitoring in htop. Removing auto_review string auto_review:"(?s)2021-05-1[78].*api failure: ([0-9]* response: (Request Timeout|timestamp mismatch|Proxy Error|Service Unavailable)|Temporary failure in name resolution)":retry to prevent overly heavy grep calls (as seen on o3 in htop as well).

I guess we will keep it as is and followup in #92854 with hopefully a better solution