Project

General

Profile

Actions

action #159639

closed

[alert] "web UI: Too many 5xx HTTP responses alert" size:S

Added by okurz 7 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-04-26
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=80&orgId=1&from=1714042970812&to=1714056541493
shows an alert condition (dashed red line)

https://mailman.suse.de/mlarch/SuSE/osd-admins/2024/osd-admins.2024.04/msg00148.html
is the corresponding alert which bundles two alerts and only the less significant one was commented on. We should still look into the 5xx HTTP response alert problem

Acceptance criteria

  • AC1: The error source is addressed and handled

Suggestions

  • Check the actual error about 5xx response and apply according mitigations and fixes or plan them in separate specific tickets if they are not trivial to fix
  • Maybe the alert is misconfigured but we should still look into the logs what the actual 5xx responses were about

Related issues 4 (0 open4 closed)

Related to openQA Infrastructure - action #122848: Configure grouped alerts in Grafana correctly size:MResolvedokurz2023-01-09

Actions
Related to openQA Infrastructure - action #138044: Grouped seemingly unrelated alert emails are confusing size:MRejectedokurz2023-10-09

Actions
Related to openQA Infrastructure - action #159396: Repeated HTTP Response alert for /tests and unresponsiveness due to potential detrimental impact of pg_dump (was: HTTP Response alert for /tests briefly going up to 15.7s) size:MResolveddheidler

Actions
Related to openQA Project - action #159792: Add better logging for 500 errors on websocket routes size:MRejectedokurz2024-04-26

Actions
Actions #1

Updated by okurz 7 months ago

  • Related to action #122848: Configure grouped alerts in Grafana correctly size:M added
Actions #2

Updated by okurz 7 months ago

  • Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
Actions #3

Updated by livdywan 7 months ago

I thought that was already filed as #159396

Actions #4

Updated by livdywan 7 months ago

  • Related to action #159396: Repeated HTTP Response alert for /tests and unresponsiveness due to potential detrimental impact of pg_dump (was: HTTP Response alert for /tests briefly going up to 15.7s) size:M added
Actions #5

Updated by okurz 7 months ago

  • Subject changed from has anyone of you already received an alert email about "web UI: Too many 5xx HTTP responses alert"? grafana says firing but I don't see an email about that yet to [alert] "web UI: Too many 5xx HTTP responses alert"
Actions #6

Updated by jbaier_cz 7 months ago

  • Subject changed from [alert] "web UI: Too many 5xx HTTP responses alert" to [alert] "web UI: Too many 5xx HTTP responses alert" size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #7

Updated by okurz 7 months ago

  • Priority changed from Normal to Urgent

another alert triggered pointing to that

Actions #8

Updated by dheidler 7 months ago

  • Assignee set to dheidler

Looks like it is mostly websocket routes:

> select * from apache_log WHERE host = 'openqa' AND time >= 1714042970812ms and time <= 1714056541493ms AND resp_code = '500';
name: apache_log
time                agent                                                                                                                   auth client_ip                             host   http_version ident origin path                        referrer request                                                   resp_bytes resp_code response_time_us verb
----                -----                                                                                                                   ---- ---------                             ----   ------------ ----- ------ ----                        -------- -------                                                   ---------- --------- ---------------- ----
1714043050068000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146323/developer/ws-proxy/status            500       29               GET
1714043084120000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146324/developer/ws-proxy/status            500       29               GET
1714043104010000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146324/developer/ws-proxy/status            500       39               GET
1714043119000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146324/developer/ws-proxy/status            500       204331           GET
1714043894074000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146365/developer/ws-proxy/status            500       25               GET
1714043934000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146365/developer/ws-proxy/status            500       191425           GET
1714044127061000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146365/developer/ws-proxy/status            500       64               GET
1714044174020000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146483/developer/ws-proxy/status            500       32               GET
1714044227026000000 Mojolicious (Perl)                                                                                                      -    2a07:de40:b203:12:3eec:efff:fefe:ac4  openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/mutex/support_server_ready?action=lock            865        500       41               POST
1714044325000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146483/developer/ws-proxy/status            500       81497            GET
1714047192000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10069            GET
1714047202000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10096            GET
1714047212000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10089            GET
1714047222000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10054            GET
1714047232000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10062            GET
1714047242000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10058            GET
1714047252000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10063            GET
1714047262000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10047            GET
1714047272001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10059            GET
1714047282000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10057            GET
1714047292000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       11920            GET
1714047304000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10080            GET
1714047314000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10072            GET
1714047324000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10079            GET
1714047334000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10060            GET
1714047344000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10076            GET
1714047354000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10094            GET
1714047354001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10095            GET
1714047365002000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10094            GET
1714047375000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10077            GET
1714047385000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10090            GET
1714047395000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10069            GET
1714047405000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10074            GET
1714047415000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10076            GET
1714047415001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10086            GET
1714047435000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10065            GET
1714047445000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10075            GET
1714047455000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10083            GET
1714047465000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10054            GET
1714047475000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10061            GET
1714047485000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10060            GET
1714047495000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10065            GET
1714047505000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10072            GET
1714047515000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10075            GET
1714047535000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10056            GET
1714047545001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10095            GET
1714047566000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10064            GET
1714047576000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10070            GET
1714047576001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10072            GET
1714047586000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10070            GET
1714047596000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10058            GET
1714047616000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10079            GET
1714047616001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10064            GET
1714047636000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10076            GET
1714047636001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10052            GET
1714047646000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10047            GET
1714047656000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10059            GET
1714047676000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10080            GET
1714047696000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10074            GET
1714047706000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10055            GET
1714047716000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714047726000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10053            GET
1714047736000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10063            GET
1714047747000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714047756000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10057            GET
1714047767000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10051            GET
1714047777000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10050            GET
1714047787000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10061            GET
1714047797000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10073            GET
1714047817000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10052            GET
1714047827000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10051            GET
1714047837000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10054            GET
1714047847000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10050            GET
1714047857000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10077            GET
1714047867000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10056            GET
1714047877000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10049            GET
1714047897000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10070            GET
1714047897001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10070            GET
1714047917000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10061            GET
1714047927001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10054            GET
1714047937000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10071            GET
1714047947000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10060            GET
1714047957000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10065            GET
1714047957001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10068            GET
1714047967000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10058            GET
1714047968000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10050            GET
1714047978000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10060            GET
1714047988000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10060            GET
1714047988001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10053            GET
1714047998000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10065            GET
1714048008000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10055            GET
1714048018000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10082            GET
1714048018001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10074            GET
1714048028000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10058            GET
1714048028001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10066            GET
1714048038000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10066            GET
1714048038001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10057            GET
1714048058000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10056            GET
1714048088000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10058            GET
1714048098000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10058            GET
1714048108000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10057            GET
1714048118000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10062            GET
1714048128000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10049            GET
1714048138000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10061            GET
1714048138001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10051            GET
1714048158000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10062            GET
1714048169001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714048179000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10065            GET
1714048179001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10062            GET
1714048189003000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10088            GET
1714048199001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10121            GET
1714048209000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10061            GET
1714048219000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10060            GET
1714048223011000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146684/developer/ws-proxy/status            500       24               GET
1714048239000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10054            GET
1714048249001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10068            GET
1714048259000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10050            GET
1714048269000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10067            GET
1714048279000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10063            GET
1714048289000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10056            GET
1714048299000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10071            GET
1714048299001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10070            GET
1714048309000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10080            GET
1714048309001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10083            GET
1714048319000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10075            GET
1714048319001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10080            GET
1714048319002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10074            GET
1714048329000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10064            GET
1714048339000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10068            GET
1714048360000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10088            GET
1714048380000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10101            GET
1714048400000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10072            GET
1714048400001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10066            GET
1714048410000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10074            GET
1714048420000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10058            GET
1714048420001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10061            GET
1714048430001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10054            GET
1714048440000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10054            GET
1714048460000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10060            GET
1714048470000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10055            GET
1714048470001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10063            GET
1714048490000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10062            GET
1714048500001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10070            GET
1714048510000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10096            GET
1714048510002000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10062            GET
1714048520000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10070            GET
1714048520001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10084            GET
1714048530000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10068            GET
1714048540000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10066            GET
1714048540001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10060            GET
1714048551000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10060            GET
1714048561000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10072            GET
1714048561001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10062            GET
1714048561002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10077            GET
1714048571000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10053            GET
1714048581001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10076            GET
1714048581002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10057            GET
1714048601000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10072            GET
1714048601001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10069            GET
1714048601002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10067            GET
1714048611000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10066            GET
1714048621000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10067            GET
1714048621001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10084            GET
1714048641000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10075            GET
1714048641001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10076            GET
1714048661000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10062            GET
1714048671000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10059            GET
1714048681000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10057            GET
1714048691001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10067            GET
1714048701000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10061            GET
1714048701001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10063            GET
1714048721000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10063            GET
1714048721001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10054            GET
1714048721002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714048731000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10061            GET
1714048741000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10054            GET
1714048752000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10070            GET
1714048762000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10064            GET
1714048782000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10085            GET
1714048782001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10103            GET
1714048792000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10066            GET
1714048802001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10098            GET
1714048802002000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10130            GET
1714048822001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10086            GET
1714048832000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10063            GET
1714048842000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10107            GET
1714048862000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10064            GET
1714048872000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10051            GET
1714048882000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10073            GET
1714048882001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10085            GET
1714048892000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10072            GET
1714048902000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10074            GET
1714048912000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10057            GET
1714048922000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10061            GET
1714048963000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10066            GET
1714048973000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10070            GET
1714048983000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10053            GET
1714048993000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10228            GET
1714049003000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10056            GET
1714049003001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10055            GET
1714049013000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10067            GET
1714049023000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10057            GET
1714049033000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10076            GET
1714049043000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10069            GET
1714049053000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       14599            GET
1714049068000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10073            GET
1714049078000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10094            GET
1714049088000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10069            GET
1714049108000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10071            GET
1714049118000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10054            GET
1714049128000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10047            GET
1714049138000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10070            GET
1714049148000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10066            GET
1714049168000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10050            GET
1714049189000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10053            GET
1714049209000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10073            GET
1714049219000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10058            GET
1714049219001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10056            GET
1714049229001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10073            GET
1714049239000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10068            GET
1714049249000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10066            GET
1714049249001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10085            GET
1714049269000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10060            GET
1714049269001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10068            GET
1714049279001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10062            GET
1714049289000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10061            GET
1714049289001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10054            GET
1714049299000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10070            GET
1714049309000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10082            GET
1714049329000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10085            GET
1714049329001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10085            GET
1714049349000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10065            GET
1714049369000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10075            GET
1714049379000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10094            GET
1714049389000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10079            GET
1714049399000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10072            GET
1714049400000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10067            GET
1714049410000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10068            GET
1714049430000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10074            GET
1714049450000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10051            GET
1714049460000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714049470000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10073            GET
1714049470001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10053            GET
1714049490000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10116            GET
1714049500000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10078            GET
1714049520000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10051            GET
1714049530000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10067            GET
1714049540000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10054            GET
1714049550000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10058            GET
1714049560000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10069            GET
1714049560001000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10066            GET
1714049570000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10058            GET
1714049581000000000 Mojolicious (Perl)                                                                                                      -    10.100.2.148                          openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3411                                                      500       10073            GET
1714049590000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3412                                                      500       10068            GET
1714049590001000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3413                                                      500       10062            GET
1714049591000000000 Mojolicious (Perl)                                                                                                      -    10.161.50.3                           openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/ws/3410                                                      500       10068            GET
1714049603000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146684/developer/ws-proxy/status            500       328846           GET
1714049936040000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 -    2a07:de40:b2bf:1b::1040               openqa 1.1          -     salt   /var/log/apache2/access_log -        /liveviewhandler/tests/14146684/developer/ws-proxy/status            500       36               GET
1714053645136000000 Mojolicious (Perl)                                                                                                      -    2a07:de40:b203:12:7ec2:55ff:fe24:de9c openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/mutex/support_server_ready?action=lock            865        500       45               POST
Actions #9

Updated by dheidler 7 months ago

1714044227026000000 Mojolicious (Perl)                                                                                                      -    2a07:de40:b203:12:3eec:efff:fefe:ac4  openqa 1.1          -     salt   /var/log/apache2/access_log -        /api/v1/mutex/support_server_ready?action=lock            865        500       41               POST
Apr 25 13:23:47 openqa openqa[13713]: [error] [ArL8TEjbUMIC] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b30d466458): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 67
Actions #10

Updated by livdywan 7 months ago

  • Status changed from Workable to In Progress

Let's assume this is in progress ๐Ÿ˜‰

Also relevant Slack thread: https://suse.slack.com/archives/C02AJ1E568M/p1714391662393459

Actions #11

Updated by dheidler 7 months ago

  • Status changed from In Progress to Feedback
Actions #12

Updated by livdywan 7 months ago

  • Status changed from Feedback to In Progress
Actions #13

Updated by livdywan 7 months ago

dheidler wrote in #note-11:

https://github.com/os-autoinst/openQA/pull/5604

Please mitigate the urgency if you want to move the ticket to feedback

Actions #14

Updated by dheidler 7 months ago

The issues with the apple useragent at /liveviewhandler/tests/14146323/developer/ws-proxy/status is a safari browser.
Maybe that is related.
The error is raised in lib/OpenQA/LiveHandler/Controller/LiveViewHandler.pm in thews_proxy function.

I found out the affected user and asked them if they had any issues and can provide more information.

Actions #15

Updated by dheidler 7 months ago

  • Priority changed from Urgent to High

I don't see why this was urgent in the first place.

Actions #16

Updated by okurz 7 months ago

dheidler wrote in #note-15:

I don't see why this was urgent in the first place.

It was urgent as at that time the alert was again firing as visible in https://monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=80&orgId=1&from=now-7d&to=now . That did not happen meanwhile probably because over the weekend no significant amount of 5xx errors showed up. I am ok with this being less than "Urgent" as long as there is no more firing alert. If it happens again and if it's only those above issues which look non-critical I suggest to add an according alert silence as mitigation.

Actions #17

Updated by openqa_review 7 months ago

  • Due date set to 2024-05-14

Setting due date based on mean cycle time of SUSE QE Tools

Actions #18

Updated by dheidler 7 months ago

  • Copied to action #159792: Add better logging for 500 errors on websocket routes size:M added
Actions #19

Updated by dheidler 7 months ago ยท Edited

  • Status changed from In Progress to Resolved

The Safari user didn't recall any issues.

Added a ticket to add better error messages for the websocket issues.

I was unable to check the error log on the worker side, as I couldn't login to that machines.
So I guess there is nothing left to do here.

Actions #20

Updated by livdywan 7 months ago

  • Status changed from Resolved to In Progress

Seems like it's still happening, so we should look into #159792 asap or find another way to catch the issue as it happens.
The alert was firing 8.10 Nrg time today.

Actions #21

Updated by dheidler 7 months ago

  • Status changed from In Progress to Blocked
Actions #22

Updated by dheidler 7 months ago

  • Copied to deleted (action #159792: Add better logging for 500 errors on websocket routes size:M)
Actions #23

Updated by dheidler 7 months ago

  • Blocked by action #159792: Add better logging for 500 errors on websocket routes size:M added
Actions #24

Updated by livdywan 7 months ago

This is still blocking on #159792 which we unfortunately didn't get to resolving last week.

Actions #25

Updated by livdywan 7 months ago

  • Due date deleted (2024-05-14)
Actions #26

Updated by okurz 7 months ago

  • Priority changed from High to Normal

We switched to nginx so maybe we can't even reproduce those errors and have not encountered more user-facing bigger problems. let's wait for #159654 first

Actions #27

Updated by livdywan 6 months ago

  • Status changed from Blocked to Workable

okurz wrote in #note-26:

We switched to nginx so maybe we can't even reproduce those errors and have not encountered more user-facing bigger problems. let's wait for #159654 first

Blocker was resolved.

Actions #28

Updated by okurz 6 months ago

  • Blocked by deleted (action #159792: Add better logging for 500 errors on websocket routes size:M)
Actions #29

Updated by okurz 6 months ago

  • Related to action #159792: Add better logging for 500 errors on websocket routes size:M added
Actions #30

Updated by okurz 6 months ago

  • Status changed from Workable to Resolved

https://monitor.qa.suse.de/d/WebuiDb/webui-summary looks good now with nginx data coming in. It seems so far no 5xx errors. Unless we can't reproduce we shouldn't need to invest more here.

Actions

Also available in: Atom PDF