Actions
action #159639
closed[alert] "web UI: Too many 5xx HTTP responses alert" size:S
Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-04-26
Due date:
% Done:
0%
Estimated time:
Description
Observation¶
https://monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=80&orgId=1&from=1714042970812&to=1714056541493
shows an alert condition (dashed red line)
https://mailman.suse.de/mlarch/SuSE/osd-admins/2024/osd-admins.2024.04/msg00148.html
is the corresponding alert which bundles two alerts and only the less significant one was commented on. We should still look into the 5xx HTTP response alert problem
Acceptance criteria¶
- AC1: The error source is addressed and handled
Suggestions¶
- Check the actual error about 5xx response and apply according mitigations and fixes or plan them in separate specific tickets if they are not trivial to fix
- Maybe the alert is misconfigured but we should still look into the logs what the actual 5xx responses were about
Updated by okurz 8 months ago
- Related to action #122848: Configure grouped alerts in Grafana correctly size:M added
Updated by okurz 8 months ago
- Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
Updated by livdywan 8 months ago
- Related to action #159396: Repeated HTTP Response alert for /tests and unresponsiveness due to potential detrimental impact of pg_dump (was: HTTP Response alert for /tests briefly going up to 15.7s) size:M added
Updated by dheidler 8 months ago
- Assignee set to dheidler
Looks like it is mostly websocket routes:
> select * from apache_log WHERE host = 'openqa' AND time >= 1714042970812ms and time <= 1714056541493ms AND resp_code = '500';
name: apache_log
time agent auth client_ip host http_version ident origin path referrer request resp_bytes resp_code response_time_us verb
---- ----- ---- --------- ---- ------------ ----- ------ ---- -------- ------- ---------- --------- ---------------- ----
1714043050068000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146323/developer/ws-proxy/status 500 29 GET
1714043084120000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146324/developer/ws-proxy/status 500 29 GET
1714043104010000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146324/developer/ws-proxy/status 500 39 GET
1714043119000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146324/developer/ws-proxy/status 500 204331 GET
1714043894074000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146365/developer/ws-proxy/status 500 25 GET
1714043934000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146365/developer/ws-proxy/status 500 191425 GET
1714044127061000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146365/developer/ws-proxy/status 500 64 GET
1714044174020000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146483/developer/ws-proxy/status 500 32 GET
1714044227026000000 Mojolicious (Perl) - 2a07:de40:b203:12:3eec:efff:fefe:ac4 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/mutex/support_server_ready?action=lock 865 500 41 POST
1714044325000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146483/developer/ws-proxy/status 500 81497 GET
1714047192000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10069 GET
1714047202000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10096 GET
1714047212000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10089 GET
1714047222000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10054 GET
1714047232000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10062 GET
1714047242000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10058 GET
1714047252000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10063 GET
1714047262000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10047 GET
1714047272001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10059 GET
1714047282000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10057 GET
1714047292000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 11920 GET
1714047304000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10080 GET
1714047314000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10072 GET
1714047324000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10079 GET
1714047334000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10060 GET
1714047344000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10076 GET
1714047354000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10094 GET
1714047354001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10095 GET
1714047365002000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10094 GET
1714047375000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10077 GET
1714047385000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10090 GET
1714047395000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10069 GET
1714047405000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10074 GET
1714047415000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10076 GET
1714047415001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10086 GET
1714047435000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10065 GET
1714047445000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10075 GET
1714047455000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10083 GET
1714047465000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10054 GET
1714047475000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10061 GET
1714047485000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10060 GET
1714047495000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10065 GET
1714047505000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10072 GET
1714047515000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10075 GET
1714047535000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10056 GET
1714047545001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10095 GET
1714047566000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10064 GET
1714047576000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10070 GET
1714047576001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10072 GET
1714047586000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10070 GET
1714047596000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10058 GET
1714047616000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10079 GET
1714047616001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10064 GET
1714047636000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10076 GET
1714047636001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10052 GET
1714047646000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10047 GET
1714047656000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10059 GET
1714047676000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10080 GET
1714047696000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10074 GET
1714047706000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10055 GET
1714047716000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714047726000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10053 GET
1714047736000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10063 GET
1714047747000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714047756000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10057 GET
1714047767000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10051 GET
1714047777000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10050 GET
1714047787000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10061 GET
1714047797000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10073 GET
1714047817000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10052 GET
1714047827000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10051 GET
1714047837000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10054 GET
1714047847000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10050 GET
1714047857000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10077 GET
1714047867000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10056 GET
1714047877000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10049 GET
1714047897000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10070 GET
1714047897001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10070 GET
1714047917000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10061 GET
1714047927001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10054 GET
1714047937000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10071 GET
1714047947000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10060 GET
1714047957000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10065 GET
1714047957001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10068 GET
1714047967000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10058 GET
1714047968000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10050 GET
1714047978000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10060 GET
1714047988000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10060 GET
1714047988001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10053 GET
1714047998000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10065 GET
1714048008000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10055 GET
1714048018000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10082 GET
1714048018001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10074 GET
1714048028000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10058 GET
1714048028001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10066 GET
1714048038000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10066 GET
1714048038001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10057 GET
1714048058000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10056 GET
1714048088000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10058 GET
1714048098000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10058 GET
1714048108000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10057 GET
1714048118000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10062 GET
1714048128000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10049 GET
1714048138000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10061 GET
1714048138001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10051 GET
1714048158000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10062 GET
1714048169001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714048179000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10065 GET
1714048179001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10062 GET
1714048189003000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10088 GET
1714048199001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10121 GET
1714048209000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10061 GET
1714048219000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10060 GET
1714048223011000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146684/developer/ws-proxy/status 500 24 GET
1714048239000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10054 GET
1714048249001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10068 GET
1714048259000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10050 GET
1714048269000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10067 GET
1714048279000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10063 GET
1714048289000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10056 GET
1714048299000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10071 GET
1714048299001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10070 GET
1714048309000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10080 GET
1714048309001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10083 GET
1714048319000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10075 GET
1714048319001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10080 GET
1714048319002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10074 GET
1714048329000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10064 GET
1714048339000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10068 GET
1714048360000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10088 GET
1714048380000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10101 GET
1714048400000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10072 GET
1714048400001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10066 GET
1714048410000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10074 GET
1714048420000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10058 GET
1714048420001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10061 GET
1714048430001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10054 GET
1714048440000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10054 GET
1714048460000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10060 GET
1714048470000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10055 GET
1714048470001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10063 GET
1714048490000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10062 GET
1714048500001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10070 GET
1714048510000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10096 GET
1714048510002000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10062 GET
1714048520000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10070 GET
1714048520001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10084 GET
1714048530000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10068 GET
1714048540000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10066 GET
1714048540001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10060 GET
1714048551000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10060 GET
1714048561000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10072 GET
1714048561001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10062 GET
1714048561002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10077 GET
1714048571000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10053 GET
1714048581001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10076 GET
1714048581002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10057 GET
1714048601000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10072 GET
1714048601001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10069 GET
1714048601002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10067 GET
1714048611000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10066 GET
1714048621000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10067 GET
1714048621001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10084 GET
1714048641000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10075 GET
1714048641001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10076 GET
1714048661000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10062 GET
1714048671000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10059 GET
1714048681000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10057 GET
1714048691001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10067 GET
1714048701000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10061 GET
1714048701001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10063 GET
1714048721000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10063 GET
1714048721001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10054 GET
1714048721002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714048731000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10061 GET
1714048741000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10054 GET
1714048752000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10070 GET
1714048762000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10064 GET
1714048782000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10085 GET
1714048782001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10103 GET
1714048792000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10066 GET
1714048802001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10098 GET
1714048802002000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10130 GET
1714048822001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10086 GET
1714048832000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10063 GET
1714048842000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10107 GET
1714048862000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10064 GET
1714048872000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10051 GET
1714048882000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10073 GET
1714048882001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10085 GET
1714048892000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10072 GET
1714048902000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10074 GET
1714048912000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10057 GET
1714048922000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10061 GET
1714048963000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10066 GET
1714048973000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10070 GET
1714048983000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10053 GET
1714048993000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10228 GET
1714049003000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10056 GET
1714049003001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10055 GET
1714049013000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10067 GET
1714049023000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10057 GET
1714049033000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10076 GET
1714049043000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10069 GET
1714049053000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 14599 GET
1714049068000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10073 GET
1714049078000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10094 GET
1714049088000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10069 GET
1714049108000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10071 GET
1714049118000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10054 GET
1714049128000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10047 GET
1714049138000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10070 GET
1714049148000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10066 GET
1714049168000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10050 GET
1714049189000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10053 GET
1714049209000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10073 GET
1714049219000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10058 GET
1714049219001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10056 GET
1714049229001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10073 GET
1714049239000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10068 GET
1714049249000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10066 GET
1714049249001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10085 GET
1714049269000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10060 GET
1714049269001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10068 GET
1714049279001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10062 GET
1714049289000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10061 GET
1714049289001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10054 GET
1714049299000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10070 GET
1714049309000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10082 GET
1714049329000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10085 GET
1714049329001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10085 GET
1714049349000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10065 GET
1714049369000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10075 GET
1714049379000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10094 GET
1714049389000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10079 GET
1714049399000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10072 GET
1714049400000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10067 GET
1714049410000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10068 GET
1714049430000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10074 GET
1714049450000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10051 GET
1714049460000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714049470000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10073 GET
1714049470001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10053 GET
1714049490000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10116 GET
1714049500000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10078 GET
1714049520000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10051 GET
1714049530000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10067 GET
1714049540000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10054 GET
1714049550000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10058 GET
1714049560000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10069 GET
1714049560001000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10066 GET
1714049570000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10058 GET
1714049581000000000 Mojolicious (Perl) - 10.100.2.148 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3411 500 10073 GET
1714049590000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3412 500 10068 GET
1714049590001000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3413 500 10062 GET
1714049591000000000 Mojolicious (Perl) - 10.161.50.3 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/ws/3410 500 10068 GET
1714049603000000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146684/developer/ws-proxy/status 500 328846 GET
1714049936040000000 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15 - 2a07:de40:b2bf:1b::1040 openqa 1.1 - salt /var/log/apache2/access_log - /liveviewhandler/tests/14146684/developer/ws-proxy/status 500 36 GET
1714053645136000000 Mojolicious (Perl) - 2a07:de40:b203:12:7ec2:55ff:fe24:de9c openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/mutex/support_server_ready?action=lock 865 500 45 POST
Updated by dheidler 8 months ago
1714044227026000000 Mojolicious (Perl) - 2a07:de40:b203:12:3eec:efff:fefe:ac4 openqa 1.1 - salt /var/log/apache2/access_log - /api/v1/mutex/support_server_ready?action=lock 865 500 41 POST
Apr 25 13:23:47 openqa openqa[13713]: [error] [ArL8TEjbUMIC] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b30d466458): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 67
Updated by livdywan 8 months ago
- Status changed from Workable to In Progress
Let's assume this is in progress 😉
Also relevant Slack thread: https://suse.slack.com/archives/C02AJ1E568M/p1714391662393459
Updated by dheidler 8 months ago
The issues with the apple useragent at /liveviewhandler/tests/14146323/developer/ws-proxy/status is a safari browser.
Maybe that is related.
The error is raised in lib/OpenQA/LiveHandler/Controller/LiveViewHandler.pm in thews_proxy function.
I found out the affected user and asked them if they had any issues and can provide more information.
Updated by okurz 8 months ago
dheidler wrote in #note-15:
I don't see why this was urgent in the first place.
It was urgent as at that time the alert was again firing as visible in https://monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=80&orgId=1&from=now-7d&to=now . That did not happen meanwhile probably because over the weekend no significant amount of 5xx errors showed up. I am ok with this being less than "Urgent" as long as there is no more firing alert. If it happens again and if it's only those above issues which look non-critical I suggest to add an according alert silence as mitigation.
Updated by openqa_review 8 months ago
- Due date set to 2024-05-14
Setting due date based on mean cycle time of SUSE QE Tools
Updated by dheidler 8 months ago
- Copied to action #159792: Add better logging for 500 errors on websocket routes size:M added
Updated by dheidler 8 months ago · Edited
- Status changed from In Progress to Resolved
The Safari user didn't recall any issues.
Added a ticket to add better error messages for the websocket issues.
I was unable to check the error log on the worker side, as I couldn't login to that machines.
So I guess there is nothing left to do here.
Updated by dheidler 7 months ago
- Copied to deleted (action #159792: Add better logging for 500 errors on websocket routes size:M)
Updated by dheidler 7 months ago
- Blocked by action #159792: Add better logging for 500 errors on websocket routes size:M added
Updated by okurz 6 months ago
- Blocked by deleted (action #159792: Add better logging for 500 errors on websocket routes size:M)
Updated by okurz 6 months ago
- Related to action #159792: Add better logging for 500 errors on websocket routes size:M added
Updated by okurz 6 months ago
- Status changed from Workable to Resolved
https://monitor.qa.suse.de/d/WebuiDb/webui-summary looks good now with nginx data coming in. It seems so far no 5xx errors. Unless we can't reproduce we shouldn't need to invest more here.
Actions