Project

General

Profile

Actions

action #128591

closed

[openqa_logwarn] logwarn reports the same entry over and over size:M

Added by tinita 11 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-05-03
Due date:
% Done:

0%

Estimated time:

Description

[openqa_logwarn][timeboxed:4h] logwarn reports the same entry over and over

Observation

We got several emails today so far which are all referring to the same entry:

  • Date: Wed, 03 May 2023 09:10:03 +0000
  • Date: Wed, 03 May 2023 10:10:02 +0000
  • Date: Wed, 03 May 2023 11:10:02 +0000
  • Date: Wed, 03 May 2023 12:10:03 +0000
  • ...
  • Date: Wed, 03 May 2023 19:10:02 +0000
[2023-05-03T08:40:57.434993Z] [warn] [pid:14091]  42-inst-oninstallation-uefi-20150922.json          |  16 -----
 42-inst-oninstallation-uefi-20150922.png           | Bin 132801 -> 0 bytes
 ...tallation_overview-Staging_Update-20180208.json |  16 -----
 ...stallation_overview-Staging_Update-20180208.png | Bin 48575 -> 0 bytes
 DIALOG-packages-notifications-20190105.json        |  15 -----
 DIALOG-packages-notifications-20190105.png         | Bin 67666 -> 0 bytes

seems to have continued into the evening of 2023-05-03 and then stopped

Acceptance criteria

  • AC1: logwarn should not warn repeatedly about the exact same log part

Suggestions

  • Create a fake log file and try to recreate the problem with logwarn locally
  • Try to use logwarn from distribution instead of custom binary
  • See if the problem resolved itself somehow or if we need or can still do improvements

Related issues 2 (0 open2 closed)

Related to openQA Project - action #110677: Investigation page shouldn't involve blocking long-running API routes size:MResolvedtinita2022-02-03

Actions
Copied to openQA Project - action #130258: [openqa_logwarn] git warning: exhaustive rename detection (investigation tab) size:MResolvedtinita2023-06-22

Actions
Actions #1

Updated by okurz 11 months ago

  • Target version set to Ready
Actions #2

Updated by okurz 11 months ago

  • Subject changed from [openqa_logwarn] logwarn reports the same entry over and over to [openqa_logwarn] logwarn reports the same entry over and over size:M
  • Status changed from New to Workable
Actions #3

Updated by okurz 11 months ago

  • Description updated (diff)
Actions #4

Updated by livdywan 11 months ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

tinita wrote:

  • Try to use logwarn from distribution instead of custom binary

Perhaps we can unify versions with a makefile:

https://github.com/os-autoinst/openqa-logwarn/pull/45

Actions #5

Updated by openqa_review 11 months ago

  • Due date set to 2023-05-23

Setting due date based on mean cycle time of SUSE QE Tools

Actions #6

Updated by livdywan 11 months ago

cdywan wrote:

tinita wrote:

  • Try to use logwarn from distribution instead of custom binary

Perhaps we can unify versions with a makefile:

https://github.com/os-autoinst/openqa-logwarn/pull/45

The makefile idea wasn't liked by everyone so I switched to the openSUSE package. However we get 1.0.14 on Leap and 1.0.17 on Tumbleweed. Meaning this would be a step back on o3 and osd machines running Leap - I need to find out what installs it in /usr/local/bin and I guess update the URL.

Actions #7

Updated by livdywan 11 months ago

Resorting to manually building and installing on o3:

sudo zypper in autoconf automake
wget https://github.com/archiecobbs/logwarn/archive/refs/tags/1.0.17.tar.gz && tar xf 1.0.17.tar.gz && cd logwarn-1.0.17 && ./autogen.sh && ./configure --prefix=/usr/local && make && sudo make install

And I guess the version on osd is unused anyway.

Actions #8

Updated by livdywan 11 months ago

  • Status changed from In Progress to Feedback

tinita wrote:

  • Create a fake log file and try to recreate the problem with logwarn locally

I wasn't very successful trying to reproduce the issue. Since we don't know what exactly caused it I'm inclined to leave it at that for now. We do have an improvement as part of this ticket which is the version upgrade that may or may not help with this.

Actions #9

Updated by livdywan 11 months ago

  • Status changed from Feedback to Resolved

tinita wrote:

  • See if the problem resolved itself somehow or if we need or can still do improvements

It seems like this is no longer happening, hence resolving.

Actions #10

Updated by tinita 11 months ago

  • Status changed from Resolved to Workable

Seems to happen again today:

[2023-05-17T08:31:17.116281Z] [warn] [pid:22771]  42-inst-oninstallation-uefi-20150922.json          |  16 -----
 42-inst-oninstallation-uefi-20150922.png           | Bin 132801 -> 0 bytes
 ...tallation_overview-Staging_Update-20180208.json |  16 -----
 ...stallation_overview-Staging_Update-20180208.png | Bin 48575 -> 0 bytes

Two emails so far about the very same entry

Actions #11

Updated by tinita 11 months ago

Looks like the needles_diff_stat from https://openqa.opensuse.org/tests/3296915#investigation for example. Not sure why it appears as a warning. couldn't reproduce

Actions #12

Updated by tinita 11 months ago

Ah. The end of the log message says (roughly 14600 lines btw):

...
zypper-zdup-finish-20201012.png                    | Bin 4912 -> 0 bytes
 14597 files changed, 12160 insertions(+), 139070 deletions(-)
warning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 13439 and retry the command.

I remember we had something like this before...

Actions #13

Updated by tinita 11 months ago

  • Related to action #110677: Investigation page shouldn't involve blocking long-running API routes size:M added
Actions #14

Updated by livdywan 10 months ago

  • Due date deleted (2023-05-23)
  • Assignee deleted (livdywan)

I've not done any work here since the ticket was re-opened, hence resetting the assignee and due date.

Actions #15

Updated by livdywan 10 months ago

tinita wrote:

Ah. The end of the log message says (roughly 14600 lines btw):

...
zypper-zdup-finish-20201012.png                    | Bin 4912 -> 0 bytes
 14597 files changed, 12160 insertions(+), 139070 deletions(-)
warning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 13439 and retry the command.

I remember we had something like this before...

Reading some suggestions on stackoverflow... would this be coming from the self-update logic? https://github.com/os-autoinst/openqa-logwarn/blob/master/pretty_logwarn#L16 Or somewhere else where push or merge is done?

Actions #16

Updated by tinita 10 months ago

cdywan wrote:

Reading some suggestions on stackoverflow... would this be coming from the self-update logic? https://github.com/os-autoinst/openqa-logwarn/blob/master/pretty_logwarn#L16 Or somewhere else where push or merge is done?

No, this comes from our investigation tab, like I mentioned above.

I linked the related issue #110677 where a fix was made for git_log_diff. needles_diff_stat would be a bit more work.

Actions #17

Updated by tinita 10 months ago

  • Copied to action #130258: [openqa_logwarn] git warning: exhaustive rename detection (investigation tab) size:M added
Actions #18

Updated by okurz 9 months ago

  • Status changed from Workable to Resolved
  • Assignee set to tinita

We assume that we prevented that issue from hitting us again by improving the investigation git log lookup.

Actions

Also available in: Atom PDF