Project

General

Profile

action #96899

jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" size:M

Added by okurz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
Start date:
2021-08-16
Due date:
2021-08-31
% Done:

0%

Estimated time:

Description

Observation

https://jenkins.qa.suse.de/ is unreachable. On jenkins.qa.suse.de the systemd service "jenkins" is running but /var/log/jenkins/jenkins.log shows:

2021-08-16 05:45:51.981+0000 [id=1]     SEVERE  hudson.WebAppMain#contextInitialized: Failed to initialize Jenkins
java.io.IOException: NSS initialization failed
        at sun.security.pkcs11.Secmod.initialize(Secmod.java:234)
        at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218)
Caused: java.security.ProviderException: Could not initialize NSS
        at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:223)
        at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:103)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224)
        at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
        at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
        at sun.security.jca.ProviderList.loadAll(ProviderList.java:282)
        at sun.security.jca.ProviderList.removeInvalid(ProviderList.java:299)
        at sun.security.jca.Providers.getFullProviderList(Providers.java:173)
        at java.security.Security.removeProvider(Security.java:475)
        at hudson.WebAppMain.contextInitialized(WebAppMain.java:179)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at winstone.Launcher.<init>(Launcher.java:192)
        at winstone.Launcher.main(Launcher.java:369)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at Main._main(Main.java:375)
        at Main.main(Main.java:151)
2021-08-16 05:45:51.983+0000 [id=1]     WARNING o.e.jetty.webapp.WebAppContext#doStart: Failed startup of context w.@c8c12ac{Jenkins v2.289.3,/,file:///var/cache/jenkins/war/,UNAVAILABLE}{/var/cache/jenkins/war}

Acceptance criteria

  • AC1: Machine is reachable over network again
  • AC2: https://jenkins.qa.suse.de is reachable again (might be movable into another ticket)

Suggestions

The machine has lost network because the package for "wicked" seems to have disappeared. I could not recover that from any older btrfs snapshots.

Suggestion: Access the machine over virt-manager on qamaster.qa.suse.de. Look up DHCP entry on qamaster.qa and assign manually using ip commands, then use zypper to install wicked again.

History

#1 Updated by okurz 4 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to Ready

triggered a reboot, machine is stuck without network. Connected over virt-manager to qamaster where jenkins.qa.suse.de is running. Trying to recover network.

#2 Updated by openqa_review 4 months ago

  • Due date set to 2021-08-31

Setting due date based on mean cycle time of SUSE QE Tools

#3 Updated by okurz 3 months ago

  • Status changed from In Progress to New
  • Assignee deleted (okurz)

Seems my ticket update was lost or I never sent it.

The machine has lost network because the package for "wicked" seems to have disappeared. I could not recover that from any older btrfs snapshots.

Suggestion: Access the machine over virt-manager on qamaster.qa.suse.de. Look up DHCP entry on qamaster.qa and assign manually using ip commands, then use zypper to install wicked again.

#4 Updated by dheidler 3 months ago

  • Assignee set to dheidler

#5 Updated by okurz 3 months ago

  • Subject changed from jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" to jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" size:M
  • Description updated (diff)
  • Status changed from New to In Progress

#6 Updated by dheidler 3 months ago

  • Status changed from In Progress to Resolved

Reinstalled wicked, updated and rebooted the machine.
Network came back automatically after reboot.

#7 Updated by okurz 3 months ago

  • Status changed from Resolved to Feedback

thanks but https://jenkins.qa.suse.de/ is not yet reachable again.

#8 Updated by dheidler 3 months ago

  • Status changed from Feedback to Resolved

After an update, jenkins is running again and after reinstalling nginx and recreating the nginx config it is reachable on port 80.

Also available in: Atom PDF