Status RPi3 20200526

Build 11.51, May 26th 2020


# rpm-sortbysize |tail -20
5660408 libsolv-tools
5737343 raspberrypi-firmware
5916442 file-magic
5996273 kbd
6016237 rpm
6530502 glibc
7195074 openssh
7774733 zypper
8034287 udev
8375418 gpg2
8457236 xtables-plugins
8533749 libzypp
9183861 util-linux
9620460 coreutils
13150058 systemd
25159592 grub2
27441111 glibc-locale-base
27863675 ignition
346060194 kernel-default
511097221 kernel-firmware

Boot Times

Initial boot, network connected

# systemd-analyze
Startup finished in 42.032s (kernel) + 19.073s (initrd) + 24.977s (userspace) = 1min 26.084s
# systemd-analyze blame|head -20
         14.088s btrfsmaintenance-refresh.service
          7.666s wicked.service
          4.904s create-dirs-from-rpmdb.service
          4.554s dracut-initqueue.service
          3.808s dracut-pre-mount.service
          3.678s lvm2-monitor.service
          2.578s initrd-switch-root.service
          2.263s systemd-journal-flush.service
          2.023s systemd-udev-settle.service
          1.744s systemd-logind.service
          1.741s kbdsettings.service
          1.225s systemd-remount-fs.service
          1.118s initrd-parse-etc.service
          1.028s dracut-cmdline.service
           986ms wickedd-auto4.service
           975ms wickedd-dhcp4.service
           970ms wickedd-dhcp6.service
           925ms rebootmgr.service
           786ms systemd-user-sessions.service
           746ms systemd-sysctl.service
# du -sh /boot/initrd-5.3.18-lp152.16-default
61M     /boot/initrd-5.3.18-lp152.16-default

The initrd is too big with 61MB. Let's rebuild it as the first boot one was created using --no-hostonly.

# mkinitrd
# du -sh /boot/initrd-5.3.18-lp152.16-default
23M     /boot/initrd-5.3.18-lp152.16-default

Remove kiwi dracut hooks and ignition as they are not needed after first boot and see the impact on initrd

# zypper rm dracut-kiwi-lib ignition ignition-dracut
# mkinitrd
# du -sh /boot/initrd-5.3.18-lp152.16-default 
12M     /boot/initrd-5.3.18-lp152.16-default
# reboot
# systemd-analyze 
Startup finished in 3.784s (kernel) + 7.243s (initrd) + 36.189s (userspace) = 47.217s
# systemd-analyze blame|head -20
         20.997s btrfsmaintenance-refresh.service
         14.378s wicked.service
          6.367s create-dirs-from-rpmdb.service
          5.909s lvm2-monitor.service
          4.429s systemd-journal-flush.service
          3.412s systemd-remount-fs.service
          3.094s systemd-modules-load.service
          2.684s systemd-udev-settle.service
          2.221s initrd-switch-root.service
          1.613s kbdsettings.service
          1.303s wickedd-auto4.service
          1.291s wickedd-dhcp6.service
          1.283s wickedd-dhcp4.service
           862ms rebootmgr.service
           661ms sshd.service
           646ms initrd-parse-etc.service
           597ms systemd-udev-trigger.service
           460ms chronyd.service
           405ms boot-efi.mount
           357ms systemd-udevd.service

That helped a lot. Remove obvious offenders too

# rpm -e btrfsmaintenance lvm2 --nodeps
# sed -i -e '/WAIT_FOR_INTERFACES/s/30/1/' /etc/sysconfig/network/config
# reboot
# systemd-analyze 
Startup finished in 3.774s (kernel) + 7.262s (initrd) + 19.238s (userspace) = 30.275s
# systemd-analyze blame|head -20
          5.190s create-dirs-from-rpmdb.service
          3.098s systemd-journal-flush.service
          2.903s systemd-udev-settle.service
          2.753s systemd-remount-fs.service
          2.471s systemd-modules-load.service
          2.038s initrd-switch-root.service
          1.856s wickedd-dhcp4.service
          1.846s wickedd-dhcp6.service
          1.818s wickedd-auto4.service
          1.150s wicked.service
          1.003s health-checker.service
           954ms rebootmgr.service
           765ms kbdsettings.service
           723ms boot-efi.mount
           700ms sshd.service
           696ms chronyd.service
           631ms initrd-parse-etc.service
           567ms systemd-udev-trigger.service
           542ms systemd-udevd.service
           462ms systemd-logind.service
# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character. @19.111s
└─ @18.505s
  └─getty@tty1.service @18.504s
    └─systemd-user-sessions.service @18.353s +132ms
      └─ @17.112s
        └─wicked.service @15.960s +1.150s
          └─wickedd-nanny.service @15.853s +100ms
            └─wickedd.service @15.743s +104ms
              └─wickedd-dhcp6.service @13.891s +1.846s
                └─dbus.service @12.479s
                  └─ @12.464s
                    └─ @12.463s
                      └─ca-certificates.path @12.463s
                        └─ @12.459s
                          └─systemd-update-utmp.service @12.380s +77ms
                            └─auditd.service @12.169s +204ms
                              └─systemd-tmpfiles-setup.service @12.097s +68ms
                                └─create-dirs-from-rpmdb.service @6.902s +5.190s
                                  └─ @6.880s
                                    └─boot-efi.mount @6.125s +723ms
                                      └─dev-disk-by\x2duuid-34A1\x2d7474.device @5.877s

create-dirs-from-rpmdb in Factory uses a cookie to check if the rpmdb actually changed. In 15 it runs always. So let's disable it. Also syncing of the journal to the SD card.

# systemctl disable create-dirs-from-rpmdb.service
# rpm -e systemd-logger --nodeps
# rm -rf /var/log/journal/
# reboot
# systemd-analyze 
Startup finished in 3.764s (kernel) + 7.143s (initrd) + 10.558s (userspace) = 21.467s
# systemd-analyze blame|head -20
          2.454s systemd-udev-settle.service
          2.007s initrd-switch-root.service
          1.552s systemd-remount-fs.service
          1.238s systemd-sysctl.service
          1.198s health-checker.service
          1.147s wicked.service
           939ms rebootmgr.service
           722ms chronyd.service
           711ms sshd.service
           596ms initrd-parse-etc.service
           556ms boot-efi.mount
           519ms systemd-udev-trigger.service
           465ms opt.mount
           456ms home.mount
           444ms srv.mount
           416ms systemd-modules-load.service
           414ms tmp.mount
           401ms usr-local.mount
           400ms systemd-journal-flush.service
           386ms \x2esnapshots.mount
localhost:~ # systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character. @10.426s
└─ @9.607s
  └─serial-getty@ttyS0.service @9.604s
    └─systemd-user-sessions.service @9.425s +158ms
      └─ @8.252s
        └─wicked.service @7.103s +1.147s
          └─wickedd-nanny.service @6.992s +104ms
            └─wickedd.service @6.883s +103ms
              └─wickedd-dhcp4.service @6.633s +243ms
                └─dbus.service @6.458s
                  └─ @6.417s
                    └─ @6.416s
                      └─dbus.socket @6.416s
                        └─ @6.410s
                          └─systemd-update-utmp.service @6.256s +152ms
                            └─auditd.service @5.943s +305ms
                              └─systemd-tmpfiles-setup.service @5.803s +132ms
                                └─systemd-journal-flush.service @5.390s +400ms
                                  └─ @5.380s
                                    └─boot-efi.mount @4.763s +556ms
                                      └─dev-disk-by\x2duuid-34A1\x2d7474.device @4.546s

Let's get rid of wicked for the fun of it

# rpm -e wicked wicked-service --nodeps
# systemctl enable systemd-networkd
# cat > <<-EOF

# reboot
# systemd-analyze 
Startup finished in 3.784s (kernel) + 7.131s (initrd) + 13.681s (userspace) = 24.596s
# systemd-analyze blame|head -20
          8.018s systemd-networkd-wait-online.service
          2.020s initrd-switch-root.service
          1.968s systemd-remount-fs.service
          1.671s systemd-modules-load.service
          1.072s health-checker.service
           657ms sshd.service
           639ms initrd-parse-etc.service
           604ms systemd-udev-trigger.service
           493ms rebootmgr.service
           414ms systemd-sysctl.service
           362ms systemd-udevd.service
           347ms chronyd.service
           335ms systemd-journald.service
           324ms sys-kernel-debug.mount
           322ms home.mount
           318ms auditd.service
           316ms systemd-logind.service
           313ms dev-hugepages.mount
           311ms boot-writable.mount
           300ms opt.mount

OOps! Now this actually works correctly. wicked and network-online semantics do not work actually. Rebootmgr pulls in network-online. The previous wicked timeout reduction would have actually broken rebootmgr if it really required the network to be online. Let's assume it does not.

# sed -i -e '/' /usr/lib/systemd/system/rebootmgr.service
# systemd-analyze
Startup finished in 3.764s (kernel) + 7.078s (initrd) + 9.769s (userspace) = 20.612s
# systemd-analyze blame|head -20
          2.367s systemd-remount-fs.service
          2.105s systemd-sysctl.service
          2.008s initrd-switch-root.service
          1.374s health-checker.service
           983ms sshd.service
           679ms rebootmgr.service
           675ms systemd-udev-trigger.service
           592ms initrd-parse-etc.service
           516ms boot-efi.mount
           382ms chronyd.service
           359ms systemd-udevd.service
           320ms systemd-journald.service
           289ms systemd-modules-load.service
           286ms boot-grub2-arm64\x2defi.mount
           282ms \x2esnapshots.mount
           276ms systemd-networkd.service
           271ms boot-writable.mount
           266ms issue-add-ssh-keys.service
           256ms dracut-cmdline.service
           253ms home.mount
# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character. @9.653s
└─ @8.936s
  └─serial-getty@ttyS0.service @8.934s
    └─systemd-user-sessions.service @8.817s +103ms
      └─ @7.282s
        └─tallow.service @7.281s
          └─ @7.272s
            └─ @7.272s
              └─dbus.socket @7.272s
                └─ @7.267s
                  └─systemd-update-utmp.service @7.180s +86ms
                    └─auditd.service @6.960s +212ms
                      └─systemd-tmpfiles-setup.service @6.878s +77ms
                        └─systemd-journal-flush.service @6.693s +179ms
                          └─ @6.678s
                            └─boot-efi.mount @6.107s +516ms
                              └─dev-disk-by\x2duuid-34A1\x2d7474.device @5.655s

More than one minute boot time saved. Still not great. More than that would require deeper investigation. For sure the initrd size needs to be reduced first.
Not included yet is the time in grub. It also has a 10s timeout by default that should be mostly useless on such systems.