Actions
action #49583
closed[arm][fs] cannot access '/var/lib/openqa/pool/16/vm-snapshots': Structure needs cleaning on openqaworker-arm-1
Start date:
2019-03-22
Due date:
% Done:
0%
Estimated time:
Description
There is a file system meta data error on openqaworker-arm-1.
- symptom:
ls: cannot access '/var/lib/openqa/pool/16/vm-snapshots': Structure needs cleaning
total 16
39878663 drwxr-xr-x 3 _openqa-worker root 4096 Mar 22 11:38 .
39878657 drwxr-xr-x 22 root root 4096 Mar 13 08:35 ..
39878682 -rw-r--r-- 1 _openqa-worker nogroup 6 Mar 21 06:48 .locked
? d????????? ? ? ? ? ? vm-snapshots
- dmesg
[1456716.329067] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1456750.854886] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1456958.168556] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457052.579780] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457134.765256] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457302.871089] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457333.829373] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457333.829629] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457333.829777] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457458.342387] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1457534.537729] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457596.499426] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457596.499710] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457596.499859] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457678.232496] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457691.495892] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1457797.299237] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1457832.595849] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457832.596128] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457832.596268] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1457892.239429] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1457900.565394] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458069.934858] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458135.682733] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458135.682985] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458135.683110] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458220.156186] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1458244.782752] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458318.628631] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458395.620245] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458395.620466] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458395.620588] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458399.926864] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458450.555769] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458583.027893] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm /usr/bin/isotov: deleted inode referenced: 39896149
[1458669.570026] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458678.944287] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458678.944558] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458678.944689] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm /usr/bin/isotov: deleted inode referenced: 39896145
[1458706.547520] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39896140: comm worker: deleted inode referenced: 39896149
[1458753.568826] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm worker: deleted inode referenced: 39896145
[1458876.258932] /usr/bin/isotov[8103]: unhandled level 2 translation fault (11) at 0x0000055c, esr 0x92000006
[1458876.258941] pgd = ffff80030f02a000
[1458876.258945] [0000055c] *pgd=000000041abe6003, *pud=00000033f382f003, *pmd=0000000000000000
[1460873.846113] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm ls: deleted inode referenced: 39896145
[1460940.818650] EXT4-fs error (device nvme0n1p1): ext4_lookup:1606: inode #39878663: comm ls: deleted inode referenced: 39896145
- tune2fs report:
openqaworker-arm-1:~ # tune2fs -l /dev/nvme0n1p1
tune2fs 1.42.11 (09-Jul-2014)
Filesystem volume name: <none>
Last mounted on: /var/lib/openqa
Filesystem UUID: d7eda95b-d486-4afe-8e4c-b890923672c0
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags: unsigned_directory_hash
Default mount options: user_xattr acl
Filesystem state: not clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 48840704
Block count: 195352790
Reserved block count: 9767639
Free blocks: 160434233
Free inodes: 48765190
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 977
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Filesystem created: Fri Feb 23 10:21:22 2018
Last mount time: Tue Mar 5 14:20:55 2019
Last write time: Fri Mar 22 12:10:35 2019
Mount count: 5
Maximum mount count: -1
Last checked: Wed Aug 22 14:25:35 2018
Check interval: 0 (<none>)
Lifetime writes: 80 TB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Default directory hash: half_md4
Directory Hash Seed: ec057e81-f808-4b8e-aba6-eac81cb1134b
FS Error count: 2616
First error time: Tue Feb 12 15:48:45 2019
First error function: ext4_lookup
First error line #: 1606
First error inode #: 39878663
First error block #: 0
Last error time: Fri Mar 22 12:10:35 2019
Last error function: ext4_lookup
Last error line #: 1606
Last error inode #: 39878663
Last error block #: 0
Updated by mloviska about 5 years ago
openQA failures found by jrivrain:
Updated by nicksinger almost 5 years ago
- Status changed from New to In Progress
- Assignee set to nicksinger
Updated by nicksinger almost 5 years ago
I've used the most recent arm-1 outage to clean its FS:
openqaworker-arm-1:/var/lib # fsck.ext2 -p /dev/nvme0n1p1
/dev/nvme0n1p1 contains a file system with errors, check forced.
/dev/nvme0n1p1: Inode 39878678, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878680, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878681, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878682, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878684, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878686, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878689, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878690, i_blocks is 24, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878691, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878693, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878694, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Inode 39878697, i_blocks is 16, should be 8. FIXED.
/dev/nvme0n1p1: Deleted inode 39878701 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39878702 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39878706 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39878709 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39878738 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39878741 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39879427 has zero dtime. FIXED.
/dev/nvme0n1p1: Deleted inode 39879428 has zero dtime. FIXED.
/dev/nvme0n1p1: Entry 'vm-snapshots' in /pool/16 (39878663) has deleted/unused inode 39896145. CLEARED.
/dev/nvme0n1p1: Entry 'boot_ltp-ver_linux_before.txt' in /pool/11/ulogs (39896140) has deleted/unused inode 39896149. CLEARED.
/dev/nvme0n1p1: Inode 39878663 ref count is 3, should be 2. FIXED.
/dev/nvme0n1p1: Inode 39878672 ref count is 4, should be 3. FIXED.
/dev/nvme0n1p1: 50632/48840704 files (19.7% non-contiguous), 18187749/195352790 blocks
A find
yields no errors anymore so I assume the fsck did its job.
Updated by nicksinger almost 5 years ago
- Status changed from In Progress to Resolved
Worker back and running. Please feel free to report further issues. We might need to reconsider our FS choice.
Updated by SLindoMansilla almost 5 years ago
- Is duplicate of action #47858: [u] test fails in first_boot - pflash overlay deleted causing: mkdir vm-snapshots: Structure needs cleaning added
Actions