Project

General

Profile

Actions

action #49583

closed

[arm][fs] cannot access '/var/lib/openqa/pool/16/vm-snapshots': Structure needs cleaning on openqaworker-arm-1

Added by mloviska about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
2019-03-22
Due date:
% Done:

0%

Estimated time:

Description

There is a file system meta data error on openqaworker-arm-1.

  • symptom:
ls: cannot access '/var/lib/openqa/pool/16/vm-snapshots': Structure needs cleaning
total 16
39878663 drwxr-xr-x  3 _openqa-worker root    4096 Mar 22 11:38 .
39878657 drwxr-xr-x 22 root           root    4096 Mar 13 08:35 ..
39878682 -rw-r--r--  1 _openqa-worker nogroup    6 Mar 21 06:48 .locked
       ? d?????????  ? ?              ?          ?            ? vm-snapshots
  • dmesg
[1456716.329067] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1456750.854886] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1456958.168556] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457052.579780] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457134.765256] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457302.871089] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457333.829373] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457333.829629] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457333.829777] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457458.342387] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1457534.537729] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457596.499426] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457596.499710] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457596.499859] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457678.232496] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457691.495892] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1457797.299237] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457832.595849] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457832.596128] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457832.596268] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457892.239429] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457900.565394] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458069.934858] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458135.682733] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458135.682985] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458135.683110] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458220.156186] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1458244.782752] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458318.628631] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458395.620245] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458395.620466] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458395.620588] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458399.926864] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458450.555769] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458583.027893] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1458669.570026] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458678.944287] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458678.944558] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458678.944689] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458706.547520] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458753.568826] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458876.258932] /usr/bin/isotov[8103]: unhandled level 2 translation fault (11) at 0x0000055c, esr 0x92000006
[1458876.258941] pgd = ffff80030f02a000
[1458876.258945] [0000055c] *pgd=000000041abe6003, *pud=00000033f382f003, *pmd=0000000000000000
[1460873.846113] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm ls: deleted inode referenced: 39896145
[1460940.818650] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm ls: deleted inode referenced: 39896145
  • tune2fs report:
openqaworker-arm-1:~ # tune2fs -l /dev/nvme0n1p1
tune2fs 1.42.11 (09-Jul-2014)
Filesystem volume name:   <none>
Last mounted on:          /var/lib/openqa
Filesystem UUID:          d7eda95b-d486-4afe-8e4c-b890923672c0
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         not clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              48840704
Block count:              195352790
Reserved block count:     9767639
Free blocks:              160434233
Free inodes:              48765190
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      977
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Fri Feb 23 10:21:22 2018
Last mount time:          Tue Mar  5 14:20:55 2019
Last write time:          Fri Mar 22 12:10:35 2019
Mount count:              5
Maximum mount count:      -1
Last checked:             Wed Aug 22 14:25:35 2018
Check interval:           0 (<none>)
Lifetime writes:          80 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Directory Hash Seed:      ec057e81-f808-4b8e-aba6-eac81cb1134b
FS Error count:           2616
First error time:         Tue Feb 12 15:48:45 2019
First error function:     ext4_lookup
First error line #:       1606
First error inode #:      39878663
First error block #:      0
Last error time:          Fri Mar 22 12:10:35 2019
Last error function:      ext4_lookup
Last error line #:        1606
Last error inode #:       39878663
Last error block #:       0

Related issues 1 (0 open1 closed)

Is duplicate of openQA Infrastructure - action #47858: [u] test fails in first_boot - pflash overlay deleted causing: mkdir vm-snapshots: Structure needs cleaningRejected2019-02-13

Actions
Actions #2

Updated by nicksinger almost 5 years ago

  • Status changed from New to In Progress
  • Assignee set to nicksinger
Actions #3

Updated by nicksinger almost 5 years ago

I've used the most recent arm-1 outage to clean its FS:

openqaworker-arm-1:/var/lib # fsck.ext2 -p /dev/nvme0n1p1
/dev/nvme0n1p1 contains a file system with errors, check forced.
/dev/nvme0n1p1: Inode 39878678, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878680, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878681, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878682, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878684, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878686, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878689, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878690, i_blocks is 24, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878691, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878693, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878694, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Inode 39878697, i_blocks is 16, should be 8.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878701 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878702 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878706 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878709 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878738 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39878741 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39879427 has zero dtime.  FIXED.
/dev/nvme0n1p1: Deleted inode 39879428 has zero dtime.  FIXED.
/dev/nvme0n1p1: Entry 'vm-snapshots' in /pool/16 (39878663) has deleted/unused inode 39896145.  CLEARED.
/dev/nvme0n1p1: Entry 'boot_ltp-ver_linux_before.txt' in /pool/11/ulogs (39896140) has deleted/unused inode 39896149.  CLEARED.
/dev/nvme0n1p1: Inode 39878663 ref count is 3, should be 2.  FIXED.
/dev/nvme0n1p1: Inode 39878672 ref count is 4, should be 3.  FIXED.
/dev/nvme0n1p1: 50632/48840704 files (19.7% non-contiguous), 18187749/195352790 blocks

A find yields no errors anymore so I assume the fsck did its job.

Actions #4

Updated by nicksinger almost 5 years ago

  • Status changed from In Progress to Resolved

Worker back and running. Please feel free to report further issues. We might need to reconsider our FS choice.

Actions #5

Updated by SLindoMansilla almost 5 years ago

  • Is duplicate of action #47858: [u] test fails in first_boot - pflash overlay deleted causing: mkdir vm-snapshots: Structure needs cleaning added
Actions

Also available in: Atom PDF