action #96899
closedjenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" size:M
0%
Description
Observation¶
https://jenkins.qa.suse.de/ is unreachable. On jenkins.qa.suse.de the systemd service "jenkins" is running but /var/log/jenkins/jenkins.log shows:
2021-08-16 05:45:51.981+0000 [id=1] SEVERE hudson.WebAppMain#contextInitialized: Failed to initialize Jenkins
java.io.IOException: NSS initialization failed
at sun.security.pkcs11.Secmod.initialize(Secmod.java:234)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218)
Caused: java.security.ProviderException: Could not initialize NSS
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:223)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:103)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224)
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
at sun.security.jca.ProviderList.loadAll(ProviderList.java:282)
at sun.security.jca.ProviderList.removeInvalid(ProviderList.java:299)
at sun.security.jca.Providers.getFullProviderList(Providers.java:173)
at java.security.Security.removeProvider(Security.java:475)
at hudson.WebAppMain.contextInitialized(WebAppMain.java:179)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at winstone.Launcher.<init>(Launcher.java:192)
at winstone.Launcher.main(Launcher.java:369)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Main._main(Main.java:375)
at Main.main(Main.java:151)
2021-08-16 05:45:51.983+0000 [id=1] WARNING o.e.jetty.webapp.WebAppContext#doStart: Failed startup of context w.@c8c12ac{Jenkins v2.289.3,/,file:///var/cache/jenkins/war/,UNAVAILABLE}{/var/cache/jenkins/war}
Acceptance criteria¶
- AC1: Machine is reachable over network again
- AC2: https://jenkins.qa.suse.de is reachable again (might be movable into another ticket)
Suggestions¶
The machine has lost network because the package for "wicked" seems to have disappeared. I could not recover that from any older btrfs snapshots.
Suggestion: Access the machine over virt-manager on qamaster.qa.suse.de. Look up DHCP entry on qamaster.qa and assign manually using ip commands, then use zypper to install wicked again.
Updated by okurz over 3 years ago
- Status changed from New to In Progress
- Assignee set to okurz
- Target version set to Ready
triggered a reboot, machine is stuck without network. Connected over virt-manager to qamaster where jenkins.qa.suse.de is running. Trying to recover network.
Updated by openqa_review over 3 years ago
- Due date set to 2021-08-31
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz about 3 years ago
- Status changed from In Progress to New
- Assignee deleted (
okurz)
Seems my ticket update was lost or I never sent it.
The machine has lost network because the package for "wicked" seems to have disappeared. I could not recover that from any older btrfs snapshots.
Suggestion: Access the machine over virt-manager on qamaster.qa.suse.de. Look up DHCP entry on qamaster.qa and assign manually using ip commands, then use zypper to install wicked again.
Updated by okurz about 3 years ago
- Subject changed from jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" to jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" size:M
- Description updated (diff)
- Status changed from New to In Progress
Updated by dheidler about 3 years ago
- Status changed from In Progress to Resolved
Reinstalled wicked, updated and rebooted the machine.
Network came back automatically after reboot.
Updated by okurz about 3 years ago
- Status changed from Resolved to Feedback
thanks but https://jenkins.qa.suse.de/ is not yet reachable again.
Updated by dheidler about 3 years ago
- Status changed from Feedback to Resolved
After an update, jenkins is running again and after reinstalling nginx and recreating the nginx config it is reachable on port 80.