action #128591
closed[openqa_logwarn] logwarn reports the same entry over and over size:M
Description
[openqa_logwarn][timeboxed:4h] logwarn reports the same entry over and over
Observation¶
We got several emails today so far which are all referring to the same entry:
- Date: Wed, 03 May 2023 09:10:03 +0000
- Date: Wed, 03 May 2023 10:10:02 +0000
- Date: Wed, 03 May 2023 11:10:02 +0000
- Date: Wed, 03 May 2023 12:10:03 +0000
- ...
- Date: Wed, 03 May 2023 19:10:02 +0000
[2023-05-03T08:40:57.434993Z] [warn] [pid:14091] 42-inst-oninstallation-uefi-20150922.json | 16 -----
42-inst-oninstallation-uefi-20150922.png | Bin 132801 -> 0 bytes
...tallation_overview-Staging_Update-20180208.json | 16 -----
...stallation_overview-Staging_Update-20180208.png | Bin 48575 -> 0 bytes
DIALOG-packages-notifications-20190105.json | 15 -----
DIALOG-packages-notifications-20190105.png | Bin 67666 -> 0 bytes
seems to have continued into the evening of 2023-05-03 and then stopped
Acceptance criteria¶
- AC1: logwarn should not warn repeatedly about the exact same log part
Suggestions¶
- Create a fake log file and try to recreate the problem with logwarn locally
- Try to use logwarn from distribution instead of custom binary
- See if the problem resolved itself somehow or if we need or can still do improvements
Updated by okurz over 1 year ago
- Subject changed from [openqa_logwarn] logwarn reports the same entry over and over to [openqa_logwarn] logwarn reports the same entry over and over size:M
- Status changed from New to Workable
Updated by livdywan over 1 year ago
- Status changed from Workable to In Progress
- Assignee set to livdywan
tinita wrote:
- Try to use logwarn from distribution instead of custom binary
Perhaps we can unify versions with a makefile:
Updated by openqa_review over 1 year ago
- Due date set to 2023-05-23
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan over 1 year ago
cdywan wrote:
tinita wrote:
- Try to use logwarn from distribution instead of custom binary
Perhaps we can unify versions with a makefile:
The makefile idea wasn't liked by everyone so I switched to the openSUSE package. However we get 1.0.14 on Leap and 1.0.17 on Tumbleweed. Meaning this would be a step back on o3 and osd machines running Leap - I need to find out what installs it in /usr/local/bin and I guess update the URL.
Updated by livdywan over 1 year ago
Resorting to manually building and installing on o3:
sudo zypper in autoconf automake
wget https://github.com/archiecobbs/logwarn/archive/refs/tags/1.0.17.tar.gz && tar xf 1.0.17.tar.gz && cd logwarn-1.0.17 && ./autogen.sh && ./configure --prefix=/usr/local && make && sudo make install
And I guess the version on osd is unused anyway.
Updated by livdywan over 1 year ago
- Status changed from In Progress to Feedback
tinita wrote:
- Create a fake log file and try to recreate the problem with logwarn locally
I wasn't very successful trying to reproduce the issue. Since we don't know what exactly caused it I'm inclined to leave it at that for now. We do have an improvement as part of this ticket which is the version upgrade that may or may not help with this.
Updated by livdywan over 1 year ago
- Status changed from Feedback to Resolved
tinita wrote:
- See if the problem resolved itself somehow or if we need or can still do improvements
It seems like this is no longer happening, hence resolving.
Updated by tinita over 1 year ago
- Status changed from Resolved to Workable
Seems to happen again today:
[2023-05-17T08:31:17.116281Z] [warn] [pid:22771] 42-inst-oninstallation-uefi-20150922.json | 16 -----
42-inst-oninstallation-uefi-20150922.png | Bin 132801 -> 0 bytes
...tallation_overview-Staging_Update-20180208.json | 16 -----
...stallation_overview-Staging_Update-20180208.png | Bin 48575 -> 0 bytes
Two emails so far about the very same entry
Updated by tinita over 1 year ago
Looks like the needles_diff_stat from https://openqa.opensuse.org/tests/3296915#investigation for example. Not sure why it appears as a warning. couldn't reproduce
Updated by tinita over 1 year ago
Ah. The end of the log message says (roughly 14600 lines btw):
...
zypper-zdup-finish-20201012.png | Bin 4912 -> 0 bytes
14597 files changed, 12160 insertions(+), 139070 deletions(-)
warning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 13439 and retry the command.
I remember we had something like this before...
Updated by tinita over 1 year ago
- Related to action #110677: Investigation page shouldn't involve blocking long-running API routes size:M added
Updated by livdywan over 1 year ago
- Due date deleted (
2023-05-23) - Assignee deleted (
livdywan)
I've not done any work here since the ticket was re-opened, hence resetting the assignee and due date.
Updated by livdywan over 1 year ago
tinita wrote:
Ah. The end of the log message says (roughly 14600 lines btw):
... zypper-zdup-finish-20201012.png | Bin 4912 -> 0 bytes 14597 files changed, 12160 insertions(+), 139070 deletions(-) warning: exhaustive rename detection was skipped due to too many files. warning: you may want to set your diff.renameLimit variable to at least 13439 and retry the command.I remember we had something like this before...
Reading some suggestions on stackoverflow... would this be coming from the self-update logic? https://github.com/os-autoinst/openqa-logwarn/blob/master/pretty_logwarn#L16 Or somewhere else where push or merge is done?
Updated by tinita over 1 year ago
cdywan wrote:
Reading some suggestions on stackoverflow... would this be coming from the self-update logic? https://github.com/os-autoinst/openqa-logwarn/blob/master/pretty_logwarn#L16 Or somewhere else where push or merge is done?
No, this comes from our investigation tab, like I mentioned above.
I linked the related issue #110677 where a fix was made for git_log_diff. needles_diff_stat would be a bit more work.
Updated by tinita over 1 year ago
- Copied to action #130258: [openqa_logwarn] git warning: exhaustive rename detection (investigation tab) size:M added
Updated by okurz over 1 year ago
- Status changed from Workable to Resolved
- Assignee set to tinita
We assume that we prevented that issue from hitting us again by improving the investigation git log lookup.
Updated by tinita about 1 month ago
- Related to action #169690: logwarn repeatedly sending openssl related problems. Is the state file stuck? added