action #39881

Worker dies during file upload (Mojo bug)

Added by szarate over 1 year ago. Updated over 1 year ago.

Status:ResolvedStart date:16/08/2018
Priority:NormalDue date:
Assignee:szarate% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

This is indeed related to https://github.com/kraih/mojo/commit/61f6cbf22c7bf8eb4787bd1014d91ee2416c73e7

    Aug 16 15:44:50 power8 worker[110215]: [info] opensuse-Tumbleweed-ppc64le-20180813-DVD@ppc64le-with-ltp.qcow2: Processing chunk 2272/2275 avg speed ~976.562KB/s                                                                        
    Aug 16 15:44:50 power8 worker[110215]: [info] opensuse-Tumbleweed-ppc64le-20180813-DVD@ppc64le-with-ltp.qcow2: Processing chunk 2273/2275 avg speed ~976.562KB/s                                                                        
    Aug 16 15:44:50 power8 worker[110215]: [info] opensuse-Tumbleweed-ppc64le-20180813-DVD@ppc64le-with-ltp.qcow2: Processing chunk 2274/2275 avg speed ~976.562KB/s                                                                        
    Aug 16 15:44:51 power8 worker[110215]: [info] opensuse-Tumbleweed-ppc64le-20180813-DVD@ppc64le-with-ltp.qcow2: Processing chunk 2275/2275 avg speed ~416.875KB/s                                                                        
    Aug 16 15:44:51 power8 worker[110215]: [info] uploading vars.json                                                                                                                                                                       
    Aug 16 15:44:51 power8 worker[110215]: [info] uploading serial0.txt                                                                                                                                                                     
    Aug 16 15:44:51 power8 worker[110215]: [info] uploading autoinst-log.txt                                                                                                                                                                
    Aug 16 15:44:51 power8 worker[110215]: [info] uploading worker-log.txt                                                                                                                                                                  
    Aug 16 15:44:51 power8 worker[110215]: Use of uninitialized value in hash element at /usr/share/openqa/script/../lib/OpenQA/Worker/Common.pm line 411.                                                                                  
    Aug 16 15:44:51 power8 worker[110215]: Use of uninitialized value $loop in hash element at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/UserAgent.pm line 208.                                                                                
    Aug 16 15:44:51 power8 worker[110215]: Mojo::Reactor::Poll: Timer failed: Can't call method "remove" on an undefined value at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/UserAgent.pm line 264.                                             
    Aug 16 15:44:54 power8 worker[110215]: [info] cleaning up 00734134-opensuse-Tumbleweed-DVD-ppc64le-Build20180813-install_ltp+opensuse+DVD@ppc64le                                                                                       
    Aug 16 15:54:56 power8 worker[110215]: Use of uninitialized value in hash element at /usr/share/openqa/script/../lib/OpenQA/Worker/Common.pm line 411.                                                                                  
    Aug 16 15:54:56 power8 worker[110215]: Use of uninitialized value $loop in hash element at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/UserAgent.pm line 208.                                                                                
    Aug 16 15:54:56 power8 worker[110215]: Mojo::Reactor::Poll: Timer failed: Can't call method "remove" on an undefined value at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/UserAgent.pm line 264.                                             
    Aug 16 16:04:53 power8 worker[110215]: Use of uninitialized value in hash element at /usr/share/openqa/script/../lib/OpenQA/Worker/Common.pm line 411.                                                                                  
    Aug 16 16:04:53 power8 worker[110215]: Use of uninitialized value $loop in hash element at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/UserAgent.pm line 208.                                                        

Related issues

Related to openQA Project - action #39743: [o3][tools] o3 unusable, often responds with 504 Gateway ... Resolved 15/08/2018

History

#1 Updated by szarate over 1 year ago

  • Description updated (diff)

#2 Updated by szarate over 1 year ago

This has been mitigated by running, for the time being, until new version of Mojo, hits the repos :)

while true; do salt '*' cmd.run 'for i in `seq 1 16`; do systemctl status openqa-worker@$i |  grep -q Mojo:: && (echo $(hostname) openqa-worker@$i; systemctl restart openqa-worker@$i); done'; sleep 1h; done; 

#3 Updated by szarate over 1 year ago

  • Related to action #39743: [o3][tools] o3 unusable, often responds with 504 Gateway Time-out added

#4 Updated by mkittler over 1 year ago

I assume if updating the worker to Mojo 7.92 or 7.93 would have helped, this would be resolved by now?

#5 Updated by szarate over 1 year ago

  • Category set to 168
  • Status changed from New to Resolved
  • Assignee set to szarate

Yes it is :)

#6 Updated by coolo over 1 year ago

  • Target version changed from Current Sprint to Done

Also available in: Atom PDF