action #45029

error 502 when triggering products with rsync.pl

Added by okurz 9 months ago. Updated about 1 month ago.

Status:ResolvedStart date:12/12/2018
Priority:NormalDue date:
Assignee:mkittler% Done:

0%

Category:Concrete Bugs
Target version:-
Difficulty:
Duration:

Description

Observation

sudo -u geekotest perl -I /opt/openqa-scripts /opt/openqa-scripts/rsync.pl --host https://openqa.suse.de --verbose --add-existing sle15_sp1

caused to trigger some media but failed on x86_64 with

error scheduling SLE-15-SP1-Installer-DVD-x86_64-Build121.1-Media1.iso
502 Proxy Error
"<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>502 Proxy Error</title>\n</head><body>\n<h1>Proxy Error</h1>\n<p>The proxy server received an invalid\r\nresponse from an upstream server.<br />\r\nThe proxy server could not handle the request <em><a href=\"/api/v1/isos\">POST&nbsp;/api/v1/isos</a></em>.<p>\nReason: <strong>Error reading from remote server</strong></p></p>\n<p>Additionally, a 502 Bad Gateway\nerror was encountered while trying to use an ErrorDocument to handle the request.</p>\n<hr>\n<address>Apache/2.4.23 (Linux/SUSE) Server at openqa.suse.de Port 443</address>\n</body></html>\n"Can't locate object method "is_status_class" via package "Mojo::Message::Response" at /opt/openqa-scripts/rsync.pl line 1052.

Problem

I am not sure if we had that problem already in before and if coolo rejected it as a "openqa/scripts" issue. We could retry in rsync.pl on 502 of course.


Related issues

Related to openQA Project - action #49535: Improve time to schedule a build Resolved 21/03/2019

History

#1 Updated by okurz 9 months ago

Reproduced a second time after trying the same call again. However, it seems like the product was still correctly scheduled.

#2 Updated by coolo 9 months ago

that's an apache timeout you see.

#3 Updated by okurz 9 months ago

sure but it's because openQA did not respond in time.

/var/log/apache2/error_log tells me:

[Wed Dec 12 09:42:40.851969 2018] [proxy_http:error] [pid 30319] (70007)The timeout specified has expired: [client 2620:113:80c0:8080:10:160:0:207:46048] AH01102: error reading status line from remote server localhost:9526
[Wed Dec 12 09:42:40.855040 2018] [proxy:error] [pid 30319] [client 2620:113:80c0:8080:10:160:0:207:46048] AH00898: Error reading from remote server returned by /api/v1/isos
[Wed Dec 12 09:42:40.857616 2018] [negotiation:error] [pid 30319] [client 2620:113:80c0:8080:10:160:0:207:46048] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_BAD_GATEWAY.html.var

and /var/log/openqa says for the same time:

[2018-12-12T09:42:30.0421 CET] [info] not registering asset fixed/SLE-15-Installer-DVD-x86_64-GM-DVD1/ containing /
[2018-12-12T09:42:32.0544 CET] [info] not registering asset fixed/SLE-15-Installer-DVD-x86_64-GM-DVD1/ containing /
[2018-12-12T09:42:50.0502 CET] [warn] START_AFTER_TEST=textmode+role_kvm:64bit-ipmi not found - check for dependency typos and dependency cycles
…
[2018-12-12T09:42:50.0855 CET] [warn] textmode:uefi has no child, check its machine placed or dependency setting typos

So I see 18 seconds between the second and third line, I guess openQA is still busy processing the request?

Can you please tell the other teams to do less stuff on openQA because it's at the limits? ;)

#4 Updated by coolo 6 months ago

@andriinikitin could you pick all rsync.pl issues into its own category? There should be quite some.

#5 Updated by andriinikitin 6 months ago

@coolo sure, I will name it "obs integration" (will investigate how to create category or ask somebody to create it)

#6 Updated by andriinikitin 6 months ago

  • Category changed from Concrete Bugs to 443

#7 Updated by okurz 3 months ago

  • Category changed from 443 to Concrete Bugs

#8 Updated by okurz about 1 month ago

  • Related to action #49535: Improve time to schedule a build added

#9 Updated by okurz about 1 month ago

  • Status changed from New to Resolved
  • Assignee set to mkittler

probably solved with https://github.com/os-autoinst/openQA/pull/2036 . Didn't see the error again.

Also available in: Atom PDF