action #32602

HDD upload fails with 502: Proxy Error

Added by nicksinger almost 2 years ago. Updated almost 2 years ago.

Status:ResolvedStart date:01/03/2018
Priority:UrgentDue date:
Assignee:EDiGiacinto% Done:

0%

Category:Concrete Bugs
Target version:Done
Difficulty:
Duration:

Description

On OSD we encounter upload errors for image creation jobs. Many (if not all) fail in the middle of uploading with a HTTP error 502.

Examples:

autoinst-log.txt:

[2018-03-01T13:11:14.0700 CET] [debug] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 25994 (/usr/bin/isotovideo: backen)
[2018-03-01T13:11:14.0700 CET] [debug] sending magic and exit
[2018-03-01T13:11:14.0700 CET] [debug] received magic close
[2018-03-01T13:11:14.0712 CET] [debug] preparing hdd 1 for upload as SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2 in qcow2
[2018-03-01T13:11:14.0712 CET] [debug] running nice ionice qemu-img convert -O qcow2 raid/l1 assets_public/SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2 -c
[2018-03-01T13:15:47.0752 CET] [debug] runcmd pid 35259 returned 35259
25957: EXIT 0
[2018-03-01T13:15:48.0563 CET] [info] +++ worker notes +++
[2018-03-01T13:15:48.0563 CET] [info] end time: 2018-03-01 12:15:48
[2018-03-01T13:15:48.0563 CET] [info] result: done
[2018-03-01T13:15:48.0563 CET] [info] uploading logs_from_installation_system-y2logs.tar.bz2
[2018-03-01T13:15:48.0727 CET] [info] uploading SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2
[2018-03-01T13:26:29.0611 CET] [error] Failed uploading chunk
[2018-03-01T13:26:29.0612 CET] [error] ERROR SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: 502 response: Proxy Error

worker-log.txt:

[2018-03-01T13:24:26.0299 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1783/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0490 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1784/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0712 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1785/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:26.0946 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1786/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0217 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1787/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0433 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1788/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0714 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1789/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:27.0903 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1790/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0145 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1791/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0371 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1792/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0629 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1793/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:28.0841 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1794/2117 avg speed ~976.562KB/s
[2018-03-01T13:24:29.0110 CET] [info] SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: Processing chunk 1795/2117 avg speed ~976.562KB/s
[2018-03-01T13:26:29.0612 CET] [error] Failed uploading chunk
[2018-03-01T13:26:29.0612 CET] [error] ERROR SLE-15-x86_64-Build489.1-sled-gnome+development.qcow2: 502 response: Proxy Error

The chunk where this happens always differ so I guess with enough luck we can at least create some HDDs.

History

#1 Updated by coolo almost 2 years ago

  • Target version set to Current Sprint

The only hint I have is this at the timestamp of the errors:
Mar 01 13:26:29 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.
Mar 01 13:29:00 openqa openqa[21365]: Mojo::Reactor::Poll: I/O watcher failed: Connection already closed at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Renderer.pm line 29.

#2 Updated by coolo almost 2 years ago

  • Project changed from openQA Tests to openQA Project
  • Category set to Concrete Bugs
  • Assignee set to EDiGiacinto

already working on retries

#3 Updated by coolo almost 2 years ago

  • Status changed from New to Resolved

merged

#4 Updated by coolo almost 2 years ago

  • Target version changed from Current Sprint to Done

Also available in: Atom PDF