Project

General

Profile

Actions

action #23934

closed

[tools][sprint 201709.1] Mojo::Reactor::Poll: I/O watcher failed: SSL_ca_file SCALAR(0x2490188) does not exist

Added by michalnowak over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
-
Start date:
2017-09-06
Due date:
% Done:

0%

Estimated time:

Description

With yesterday's openQA & Mojolicious update on Leap 42.2, openQA worker is not able to connect to openQA (it worked with the previous version):

systemd[1]: Starting openQA Worker #1...
systemd[1]: Started openQA Worker #1.
worker[17322]: [INFO] CACHE: caching is enabled, setting up /var/lib/openqa/cache/localhost
worker[17322]: [INFO] registering worker with openQA http://localhost...
worker[17322]: Mojo::Reactor::Poll: I/O watcher failed: SSL_ca_file SCALAR(0x2490188) does not exist at /usr/lib/perl5/vendor_perl/5.18.2/IO/Socket/SSL.pm line 1616.
openqa-scheduler[15775]: [scheduler:debug] [Congestion control] Resetting failures count and rescheduling if necessary
openqa-scheduler[15775]: [scheduler:debug] [rescheduling] Current tick is at 60000ms. New tick will be in: 9000ms
openqa-scheduler[15775]: [scheduler:debug] [Congestion control] Resetting no actions count and rescheduling if necessary
openqa-scheduler[15775]: [scheduler:debug] +=----------------=+
openqa-scheduler[15775]: [scheduler:debug] -> Scheduling new jobs.
openqa-scheduler[15775]: [scheduler:debug]          Free workers: 1/9
openqa-scheduler[15775]: [scheduler:debug]          Failure# 0
openqa-scheduler[15775]: [scheduler:debug] Attempt to find job 1/1
openqa-scheduler[15775]: [scheduler:debug] [Worker#1] Possible jobs to allocate: 0
openqa-scheduler[15775]: [scheduler:debug] Scheduler took 0.02474s to perform operations and allocated 0 jobs
openqa-scheduler[15775]: [scheduler:debug] [Congestion control] Calculated backoff: 9000ms
openqa-scheduler[15775]: [scheduler:debug] [Congestion control] Rounds with no actions performed: 1
openqa-scheduler[15775]: [scheduler:debug] [Congestion control] Failures# 0
openqa-scheduler[15775]: [scheduler:debug] [rescheduling] Current tick is at 9000ms. New tick will be in: 9000ms
openqa-scheduler[15775]: [scheduler:debug] +=----------------=+

openQA-4.5.1504621155.1c421a8e-64.1.noarch
perl-Mojo-RabbitMQ-Client-0.0.9-2.1.noarch
perl-Mojolicious-Plugin-AssetPack-1.45-2.1.noarch
perl-Mojolicious-7.44-2.1.noarch
perl-Mojolicious-Plugin-RenderFile-0.10-1.1.noarch
perl-IO-Socket-SSL-1.962-5.2.noarch


Related issues 1 (0 open1 closed)

Related to openQA Project - action #23940: [tools][sprint 201709.1] clone job from https://openqa.opensuse.org doesn't workRejectedszarate2017-09-06

Actions
Actions #1

Updated by EDiGiacinto over 6 years ago

  • Related to action #23940: [tools][sprint 201709.1] clone job from https://openqa.opensuse.org doesn't work added
Actions #2

Updated by EDiGiacinto over 6 years ago

AFAICS this looks like the culprit: https://github.com/kraih/mojo/commit/19502a0b0a63f8d28b1258caed3eb0248deb025e#diff-4ae7b7f1fbef554da39b1d125642c2c7L54

Can you try in your local machine to revert that commit?

Actions #3

Updated by szarate over 6 years ago

  • Assignee set to kraih

I can confirm that on zluo's machine, removing the reference to undef, works. For some reason on my machine with the same mojolicious version, this works (But I'm using perl v5.24.1).

Actions #4

Updated by michalnowak over 6 years ago

EDiGiacinto wrote:

AFAICS this looks like the culprit: https://github.com/kraih/mojo/commit/19502a0b0a63f8d28b1258caed3eb0248deb025e#diff-4ae7b7f1fbef554da39b1d125642c2c7L54

Can you try in your local machine to revert that commit?

Hunk reverted and it works fine with perl 5.18.2.

Actions #5

Updated by EDiGiacinto over 6 years ago

michalnowak wrote:

EDiGiacinto wrote:

AFAICS this looks like the culprit: https://github.com/kraih/mojo/commit/19502a0b0a63f8d28b1258caed3eb0248deb025e#diff-4ae7b7f1fbef554da39b1d125642c2c7L54

Can you try in your local machine to revert that commit?

Hunk reverted and it works fine with perl 5.18.2.

Thanks for opening the issue and confirming, it was caused by an ABI break with versions of IO::Socket::SSL greater than 1.965. We opened a PR for fixing this and was merged https://github.com/kraih/mojo/pull/1129

Actions #6

Updated by kraih over 6 years ago

I've released a new version of Mojolicious with the fix, it should reach devel:languages:perl shortly. https://metacpan.org/release/SRI/Mojolicious-7.45

Actions #7

Updated by okurz over 6 years ago

  • Status changed from New to In Progress
  • Priority changed from Normal to High

also hit me.

https://build.opensuse.org/request/show/522241 is the submission to openSUSE:Factory

I could not find any maintenance update submission for openSUSE Leap.

Also, so far I have been using perl-Mojolicious from devel:openQA:Leap:42.2. Should we also update that?

I installed 7.45 from devel:languages:perl but the error persists for me.

EDIT: I as well had to manually revert https://github.com/kraih/mojo/commit/19502a0b0a63f8d28b1258caed3eb0248deb025e#diff-4ae7b7f1fbef554da39b1d125642c2c7L54 even in 7.45 to make it work

Actions #8

Updated by EDiGiacinto over 6 years ago

okurz wrote:

also hit me.

https://build.opensuse.org/request/show/522241 is the submission to openSUSE:Factory

I could not find any maintenance update submission for openSUSE Leap.

Also, so far I have been using perl-Mojolicious from devel:openQA:Leap:42.2. Should we also update that?

I installed 7.45 from devel:languages:perl but the error persists for me.

EDIT: I as well had to manually revert https://github.com/kraih/mojo/commit/19502a0b0a63f8d28b1258caed3eb0248deb025e#diff-4ae7b7f1fbef554da39b1d125642c2c7L54 even in 7.45 to make it work

Can you report the version of IO-Socket-SSL installed from zypper?

Also, if possible, the output of:

perl -MIO::Socket::SSL -e 'print "$IO::Socket::SSL::VERSION\n";'    
Actions #9

Updated by binary_sequence over 6 years ago

It also happens to me on Leap 42.2.

openqa:~ # /var/lib/openqa/script/clone_job.pl --skip-deps --host localhost --from https://openqa.opensuse.org 483028 _GROUP=0
Mojo::Reactor::Poll: I/O watcher failed: SSL_ca_file SCALAR(0x10ae188) does not exist at /usr/lib/perl5/vendor_perl/5.18.2/IO/Socket/SSL.pm line 1616.
failed to get job:  Connect timeout at /var/lib/openqa/script/clone_job.pl line 197.
openqa:~ # perl -MIO::Socket::SSL -e 'print "$IO::Socket::SSL::VERSION\n";'
1.962
openqa:~ # zypper se --installed-only -s -x openQA
Loading repository data...
Reading installed packages...

S | Name   | Type    | Version                      | Arch   | Repository
--+--------+---------+------------------------------+--------+-----------
i | openQA | package | 4.5.1505139204.cc6c98a7-74.1 | noarch | openQA
openqa:~ # zypper se --installed-only -s -x perl-Mojolicious
Loading repository data...
Reading installed packages...

S | Name             | Type    | Version  | Arch   | Repository
--+------------------+---------+----------+--------+--------------------
i | perl-Mojolicious | package | 7.44-2.1 | noarch | openQA-perl-modules
openqa:~ # zypper lr --uri
#  | Alias                     | Name                                    | Enabled | GPG Check | Refresh | URI
---+---------------------------+-----------------------------------------+---------+-----------+---------+---------------------------------------------------------------------------------------
 1 | openQA                    | openQA                                  | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_42.2
 2 | openQA-perl-modules       | openQA-perl-modules                     | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/devel:/openQA:/Leap:/42.2/openSUSE_Leap_42.2
 3 | openSUSE-42.2-0           | openSUSE-42.2-0                         | No      | ----      | No      | cd:///?devices=/dev/disk/by-id/scsi-14d534654202020207305e3437703544694957d7ced624a7d
 4 | repo-debug                | openSUSE-Leap-42.2-Debug                | No      | ----      | Yes     | http://download.opensuse.org/debug/distribution/leap/42.2/repo/oss/
 5 | repo-debug-non-oss        | openSUSE-Leap-42.2-Debug-Non-Oss        | No      | ----      | Yes     | http://download.opensuse.org/debug/distribution/leap/42.2/repo/non-oss/
 6 | repo-debug-update         | openSUSE-Leap-42.2-Update-Debug         | No      | ----      | Yes     | http://download.opensuse.org/debug/update/leap/42.2/oss/
 7 | repo-debug-update-non-oss | openSUSE-Leap-42.2-Update-Debug-Non-Oss | No      | ----      | Yes     | http://download.opensuse.org/debug/update/leap/42.2/non-oss/
 8 | repo-non-oss              | openSUSE-Leap-42.2-Non-Oss              | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/distribution/leap/42.2/repo/non-oss/
 9 | repo-oss                  | openSUSE-Leap-42.2-Oss                  | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/distribution/leap/42.2/repo/oss/
10 | repo-source               | openSUSE-Leap-42.2-Source               | No      | ----      | Yes     | http://download.opensuse.org/source/distribution/leap/42.2/repo/oss/
11 | repo-source-non-oss       | openSUSE-Leap-42.2-Source-Non-Oss       | No      | ----      | Yes     | http://download.opensuse.org/source/distribution/leap/42.2/repo/non-oss/
12 | repo-update               | openSUSE-Leap-42.2-Update               | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/update/leap/42.2/oss/
13 | repo-update-non-oss       | openSUSE-Leap-42.2-Update-Non-Oss       | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/update/leap/42.2/non-oss/
Actions #10

Updated by kraih over 6 years ago

You need Mojolicious 7.45 or IO::Socket::SSL 1.965.

Actions #11

Updated by szarate over 6 years ago

So, currently since the 7.45 version of Mojolicious is still not available (but should be in the next couple of days), Installing the mojolicious rpm from devel:languages:perl or directly the rpm from the mojolicious devel project, could help to circumvent the problem.

https://software.opensuse.org//download.html?project=devel%3Alanguages%3Aperl&package=perl-Mojolicious

Actions #12

Updated by SLindoMansilla over 6 years ago

The report from yesterday was from computer club.

In the machine at work does not work either and I have:

geekotest@copland:~> perl -MIO::Socket::SSL -e 'print "$IO::Socket::SSL::VERSION\n";'
1.962
geekotest@copland:~> zypper se --installed-only -s -x openQA
Loading repository data...
Reading installed packages...

S  | Name   | Type    | Version                      | Arch   | Repository       
---+--------+---------+------------------------------+--------+------------------
i+ | openQA | package | 4.5.1505139204.cc6c98a7-74.1 | noarch | (System Packages)
geekotest@copland:~> zypper se --installed-only -s -x perl-Mojolicious
Loading repository data...
Reading installed packages...

S  | Name             | Type    | Version    | Arch   | Repository
---+------------------+---------+------------+--------+-----------
i+ | perl-Mojolicious | package | 7.45-176.1 | noarch | devel:perl
geekotest@copland:~> zypper lr --uri
Repository priorities are without effect. All enabled repositories share the same priority.

#  | Alias                     | Name                                              | Enabled | GPG Check | Refresh | URI                                                                                   
---+---------------------------+---------------------------------------------------+---------+-----------+---------+---------------------------------------------------------------------------------------
 1 | SUSE_CA                   | SUSE Internal CA Certificate (openSUSE_Leap_42.2) | Yes     | (r ) Yes  | Yes     | http://download.suse.de/ibs/SUSE:/CA/openSUSE_Leap_42.2/                              
 2 | Virtualization            | Virtualization                                    | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/Virtualization/openSUSE_Leap_42.2/          
 3 | devel:perl                | devel:perl                                        | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_Leap_42.2/  
 4 | openQA                    | openQA                                            | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_42.2            
 5 | openQA-perl-modules       | openQA-perl-modules                               | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/repositories/devel:/openQA:/Leap:/42.2/openSUSE_Leap_42.2
 6 | openSUSE-42.2-0           | openSUSE-42.2-0                                   | No      | ----      | ----    | http://dist.suse.de/SLP/openSUSE-Leap-42.2-GM/x86_64/DVD1/                            
 7 | repo-debug                | openSUSE-Leap-42.2-Debug                          | No      | ----      | ----    | http://download.opensuse.org/debug/distribution/leap/42.2/repo/oss/                   
 8 | repo-debug-non-oss        | openSUSE-Leap-42.2-Debug-Non-Oss                  | No      | ----      | ----    | http://download.opensuse.org/debug/distribution/leap/42.2/repo/non-oss/               
 9 | repo-debug-update         | openSUSE-Leap-42.2-Update-Debug                   | No      | ----      | ----    | http://download.opensuse.org/debug/update/leap/42.2/oss/                              
10 | repo-debug-update-non-oss | openSUSE-Leap-42.2-Update-Debug-Non-Oss           | No      | ----      | ----    | http://download.opensuse.org/debug/update/leap/42.2/non-oss/                          
11 | repo-non-oss              | openSUSE-Leap-42.2-Non-Oss                        | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/distribution/leap/42.2/repo/non-oss/                     
12 | repo-oss                  | openSUSE-Leap-42.2-Oss                            | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/distribution/leap/42.2/repo/oss/                         
13 | repo-source               | openSUSE-Leap-42.2-Source                         | No      | ----      | ----    | http://download.opensuse.org/source/distribution/leap/42.2/repo/oss/                  
14 | repo-source-non-oss       | openSUSE-Leap-42.2-Source-Non-Oss                 | No      | ----      | ----    | http://download.opensuse.org/source/distribution/leap/42.2/repo/non-oss/              
15 | repo-update               | openSUSE-Leap-42.2-Update                         | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/update/leap/42.2/oss/                                    
16 | repo-update-non-oss       | openSUSE-Leap-42.2-Update-Non-Oss                 | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/update/leap/42.2/non-oss/
Actions #13

Updated by kraih over 6 years ago

You are correct, the workaround in Mojolicious 7.45 does not work. We'll have to wait for 7.46.

Actions #14

Updated by szarate over 6 years ago

  • Subject changed from Mojo::Reactor::Poll: I/O watcher failed: SSL_ca_file SCALAR(0x2490188) does not exist to [tools][sprint 201709.1] Mojo::Reactor::Poll: I/O watcher failed: SSL_ca_file SCALAR(0x2490188) does not exist
  • Status changed from In Progress to Feedback

I will set this to resolved, as we know now the root cause of the problem, and a fix is there (but at least not in TW yet)

Actions #15

Updated by kraih over 6 years ago

The fix is already in devel:languages:perl and on its way to TW now. https://build.opensuse.org/request/show/524560

Actions #16

Updated by okurz over 6 years ago

szarate wrote:

I will set this to resolved […]

but you set it to "feedback"?!? Also, we should regard it as resolved when it is, as in: The problem can not appear anymore for at least the reporter. That would mean that an updated package is provided at least in devel:openQA:Leap:42.2 or as maintenance update to openSUSE:Leap:42.2

Actions #17

Updated by szarate over 6 years ago

Actually I meant feedback :P. Because since the fix is already there, but is not available in Tumbleweed, then we can't yet mark it as resolved.

Actions #18

Updated by szarate over 6 years ago

  • Status changed from Feedback to Resolved

I'm setting this to resolved, packages are already within repos, with fixes and all... and no reports have been done within the last few days. Please reopen if still valid.

Actions

Also available in: Atom PDF