action #23536
Updated by nicksinger almost 7 years ago
Since (rough estimation) the heavy modification of the scheduler we can regularly observe the following error appear in the openQA log files: [Wed Aug 23 09:56:24 2017] [11197:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. Context from the log file: [Wed Aug 23 09:56:21 2017] [websockets:error] Worker not found for given connection during connection close [Wed Aug 23 09:56:22 2017] [3069:info] Stopping worker 16795 gracefully (800 seconds) [Wed Aug 23 09:56:22 2017] [23576:info] Worker 23576 started [Wed Aug 23 09:56:22 2017] [23576:info] Connecting to AMQP server [Wed Aug 23 09:56:22 2017] [3069:info] Worker 16795 stopped [Wed Aug 23 09:56:22 2017] [23576:info] AMQP connection established [Wed Aug 23 09:56:24 2017] [3069:info] Stopping worker 16889 gracefully (800 seconds) [Wed Aug 23 09:56:24 2017] [23578:info] Worker 23578 started [Wed Aug 23 09:56:24 2017] [23578:info] Connecting to AMQP server [Wed Aug 23 09:56:24 2017] [23578:info] AMQP connection established [Wed Aug 23 09:56:24 2017] [11197:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. [Wed Aug 23 09:56:24 2017] [18942:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. [Wed Aug 23 09:56:24 2017] [13669:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. [Wed Aug 23 09:56:24 2017] [22897:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. [Wed Aug 23 09:56:25 2017] [7513:debug] removing screenshot 4fb/518/987743821945823012420a62bd.png [Wed Aug 23 09:56:25 2017] [7513:debug] removing screenshot 83c/f60/e85ad7da4f25b1eb96f0680aa9.png [Wed Aug 23 09:56:25 2017] [7513:debug] removing screenshot 581/fd4/bdf3965a15065f31523cef9463.png [Wed Aug 23 09:56:25 2017] [7513:debug] removing screenshot 7ad/564/ae42ed15f1806c65f71cbfd4f9.png [Wed Aug 23 09:56:25 2017] [7513:debug] removing screenshot 898/5a6/6807e3c39b97dd10458dc2b70d.png [Wed Aug 23 09:56:25 2017] [3069:info] Stopping worker 5110 gracefully (800 seconds) [Wed Aug 23 09:56:25 2017] [3069:info] Worker 5110 stopped [Wed Aug 23 09:56:25 2017] [23579:info] Worker 23579 started [Wed Aug 23 09:56:25 2017] [23579:info] Connecting to AMQP server Everything related to one of the workers who raised this message: [Wed Aug 23 08:57:57 2017] [11197:info] Worker 11197 started [Wed Aug 23 08:57:57 2017] [11197:info] Connecting to AMQP server [Wed Aug 23 08:57:57 2017] [11197:info] AMQP connection established [Wed Aug 23 09:13:00 2017] [7513:debug] removing screenshot a4f/fc9/e72271244111977be85ad8dcc1.png [Wed Aug 23 09:53:30 2017] [11197:info] Got status update for job 1125270 that does not belong to Worker 543 [Wed Aug 23 09:56:24 2017] [11197:error] org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. [Wed Aug 23 09:56:39 2017] [3069:info] Stopping worker 11197 gracefully (800 seconds) [Wed Aug 23 09:56:39 2017] [3069:info] Worker 11197 stopped Unfortunately I cannot see what exactly is causing the issue here. ### Suggestions on how to improve this message: * If possible, include more specific reasons for this (what is the context of the message? What did the worker try tried to do?) If this message is critical (not self recovering): * Add hints where an admin could look for more information * Expand message to explain the admin: "Hey, something just broke - you need to interact" If this message should just inform the admin: * Decrease log level to at max "warn"