action #32602
closedHDD upload fails with 502: Proxy Error
0%
Description
On OSD we encounter upload errors for image creation jobs. Many (if not all) fail in the middle of uploading with a HTTP error 502.
Examples:
aarch64:
x86_64:
autoinst-log.txt:
[2018-03-01T13:11:14.0700 CET] [debug] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 25994 (/usr/bin/isotovideo: backen)
[2018-03-01T13:11:14.0700 CET] [debug] sending magic and exit
[2018-03-01T13:11:14.0700 CET] [debug] received magic close
[2018-03-01T13:11:14.0712 CET] [debug] preparing hdd 1 for upload as SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2 in qcow2
[2018-03-01T13:11:14.0712 CET] [debug] running nice ionice qemu-img convert -O qcow2 raid/l1 assets_public/SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2 -c
[2018-03-01T13:15:47.0752 CET] [debug] runcmd pid 35259 returned 35259
25957: EXIT 0
[2018-03-01T13:15:48.0563 CET] [info] +++ worker notes +++
[2018-03-01T13:15:48.0563 CET] [info] end time: 2018-03-01 12:15:48
[2018-03-01T13:15:48.0563 CET] [info] result: done
[2018-03-01T13:15:48.0563 CET] [info] uploading logs_from_installation_system-y2logs.tar.bz2
[2018-03-01T13:15:48.0727 CET] [info] uploading SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2
[2018-03-01T13:26:29.0611 CET] [error] Failed uploading chunk
[2018-03-01T13:26:29.0612 CET] [error] ERROR SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: 502 response: Proxy Error
worker-log.txt:
[2018-03-01T13:24:26.0299 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1783/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0490 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1784/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0712 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1785/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0946 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1786/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0217 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1787/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0433 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1788/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0714 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1789/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0903 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1790/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0145 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1791/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0371 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1792/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0629 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1793/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0841 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1794/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:29.0110 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1795/2117 avg speed ~976.562KB/s
[2018-03-01T13:26:29.0612 CET] [error] Failed uploading chunk
[2018-03-01T13:26:29.0612 CET] [error] ERROR SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: 502 response: Proxy Error
The chunk where this happens always differ so I guess with enough luck we can at least create some HDDs.
Updated by coolo almost 7 years ago
- Target version set to Current Sprint
The only hint I have is this at the timestamp of the errors:
Mar 01 13:26:29 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Updated by coolo almost 7 years ago
- Project changed from openQA Tests (public) to openQA Project (public)
- Category set to Regressions/Crashes
- Assignee set to EDiGiacinto
already working on retries
Updated by coolo almost 7 years ago
- Target version changed from Current Sprint to Done