action #31351
closed[functional][u][medium] force_cron_run does not actually run any crons (occasionally)
0%
Description
Observation¶
openQA test in scenario opensuse-Tumbleweed-DVD-aarch64-lxde@aarch64
should run the cron jobs, but does not run any.
force_cron_run
For unknown reason, ${RUN} in /usr/lib/cron/run-crons is empty.
Reproducible¶
Circumstances leading to running crons not yet known, e.g. for xfce using
the same snapshot the cron scripts are run:
https://openqa.opensuse.org/tests/600008#step/force_cron_run/6
Expected result¶
Compare with:
https://openqa.opensuse.org/tests/600008#step/force_cron_run/6
which has
STATUS=
'[' '!' -z ' daily' ']'
...
echo 'running daily cronjob scripts'
The ' daily' is the value of the ${RUN} variable.
Last good: 20180201 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by StefanBruens over 6 years ago
- Related to action #25554: [functional][bsc#1063638][u] soft fail in force_cron_run is too strict added
Updated by okurz over 6 years ago
- Subject changed from force_cron_run does not actually run any crons (occasionally) to [functional]force_cron_run does not actually run any crons (occasionally)
- Target version set to Milestone 16
Updated by okurz over 6 years ago
- Subject changed from [functional]force_cron_run does not actually run any crons (occasionally) to [functional][u]force_cron_run does not actually run any crons (occasionally)
- Due date set to 2018-05-08
Updated by okurz over 6 years ago
- Related to action #34210: [functional][u][medium]updates_packagekit_gpk restarts the updater several times added
Updated by okurz over 6 years ago
- Related to deleted (action #34210: [functional][u][medium]updates_packagekit_gpk restarts the updater several times)
Updated by mgriessmeier over 6 years ago
- Subject changed from [functional][u]force_cron_run does not actually run any crons (occasionally) to [functional][u][medium] force_cron_run does not actually run any crons (occasionally)
- Status changed from New to Workable
Updated by zluo over 6 years ago
- Status changed from Workable to In Progress
checking at first since it is an occasional issue:
Updated by zluo over 6 years ago
Updated by zluo over 6 years ago
- Status changed from In Progress to Resolved
checked tests for 2 months ago and recent test run don't show up any issue with crons on o3.
Updated by StefanBruens over 6 years ago
- Status changed from Resolved to Workable
Have you even read the original issue description?
Both test runs you linked to shows no sign of any started cron jobs. If the jobs were started, there would be a line stating "Running cronjob scripts"
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-05-08 to 2018-05-22
Updated by zluo over 6 years ago
increase timeout to 120:
sub run {
select_console 'root-console';
# show dmesg output in console during cron run
assert_script_run "dmesg -n 7";
# Make sure there's no load before we trigger one via cron.
settle_load;
my $before = time;
assert_script_run "bash -x /usr/lib/cron/run-crons", 1000;
record_soft_failure 'bsc#1063638 - review I/O scheduling parameters of btrfsmaintenance' if (time - $before) > 120 && get_var('SOFTFAIL_BSC1063638');
sleep 3; # some head room for the load average to rise
settle_load;
# return dmesg output to normal
assert_script_run "dmesg -n 1";
}
got expected results (6 times):
http://e13.suse.de/tests/2448#step/force_cron_run/6
More test runs triggered now.
Updated by zluo over 6 years ago
checked cron job rules:
## Type: string
## Default: ""
#
# At which time cron.daily should start. Default is 15 minutes after booting
# the system. Example setting would be "14:00".
# Due to the fact that cron script runs only every 15 minutes,
# it will only run on xx:00, xx:15, xx:30, xx:45, not at the accurate time
# you set.
DAILY_TIME=""
## Type: integer
## Default: 5
#
# Maximum days not running when using a fixed time set in DAILY_TIME.
# 0 to skip this. This is for users who will power off their system.
#
# There is a fixed max. of 14 days set, if you want to override this
# change MAX_NOT_RUN_FORCE in /usr/lib/cron/run-crons
MAX_NOT_RUN="5"
--
this is the problem I'm afraid...
Updated by StefanBruens over 6 years ago
There is another aspect here:
The check was added, AFAIK, to "flush" any btrfs maintenance jobs, to avoid these influence any later running jobs.
Now, the btrfs cron jobs have been migrated to systemd timers:
$> systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Thu 2018-05-17 19:00:00 CEST 59min left Thu 2018-05-17 18:00:03 CEST 16s ago snapper-timeline.timer snapper-timeline.service
Fri 2018-05-18 00:00:00 CEST 5h 59min left Thu 2018-05-17 00:00:05 CEST 18h ago logrotate.timer logrotate.service
Fri 2018-05-18 00:05:01 CEST 6h left Thu 2018-05-17 01:59:50 CEST 16h ago backup-sysconfig.timer backup-sysconfig.service
Fri 2018-05-18 00:10:57 CEST 6h left Thu 2018-05-17 00:20:53 CEST 17h ago backup-rpmdb.timer backup-rpmdb.service
Fri 2018-05-18 01:32:24 CEST 7h left Thu 2018-05-17 01:11:32 CEST 16h ago check-battery.timer check-battery.service
Fri 2018-05-18 10:15:22 CEST 16h left Thu 2018-05-17 10:15:22 CEST 7h ago snapper-cleanup.timer snapper-cleanup.service
Fri 2018-05-18 10:20:22 CEST 16h left Thu 2018-05-17 10:20:22 CEST 7h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2018-05-21 00:00:00 CEST 3 days left Mon 2018-05-14 00:00:08 CEST 3 days ago btrfs-balance.timer btrfs-balance.service
Mon 2018-05-21 00:00:00 CEST 3 days left Mon 2018-05-14 00:00:08 CEST 3 days ago btrfs-trim.timer btrfs-trim.service
Mon 2018-05-21 00:00:00 CEST 3 days left Mon 2018-05-14 00:00:08 CEST 3 days ago fstrim.timer fstrim.service
Fri 2018-06-01 00:00:00 CEST 2 weeks 0 days left Tue 2018-05-01 00:00:08 CEST 2 weeks 2 days ago btrfs-scrub.timer btrfs-scrub.service
It should be comparatively easy to check for any soon-expiring timers (LEFT column), and trigger the the corresponding services immediately. Hopefully, retriggering the services short time after will be cheap.
Updated by zluo over 6 years ago
I run "bash -x /usr/lib/cron/run-crons" on just rebooted sles 15. No cron job got executed.
To me this command can work (it still depends on system timers.), but it has not to work as we assumed for executing cron jobs on a running system with very short online duration.
I'll ask coolo about this. We might think about to run each cron jobs separately. But at first I want to make sure about:
Summary: Avoid suprises later and run the cron jobs explicitly
Updated by zluo over 6 years ago
http://e13.suse.de/tests/2518#step/force_cron_run/6
my $run_daily_cronjobs = 'for file in /etc/cron.daily/*; do echo execute daily cron jobs: $file; $file; done';
assert_script_run "$run_daily_cronjobs", 1000;
Updated by zluo over 6 years ago
Updated by StefanBruens over 6 years ago
and once a week and once a months QA jobs still fail, as the weekly/monthly maintenance jobs suddenly run ...
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-05-22 to 2018-06-05
Updated by zluo over 6 years ago
updated:
find /etc/cron.{hourly,daily,weekly,monthly} -type f -executable -exec echo run cron job: {} \; -exec {} \;
Updated by zluo over 6 years ago
info:
customers need to provide their own scripts for weekly, monthly, we don't provide packages for them, for hourly we have acronie-anacron which needs to be installed extra. systemd timers will replace cron job anyway. We can now check systemctl list-timers for details. but the question what should we do at moment.
coolo suggests to keep calling run-cron-jobs, but on top of it disable all systemd timers
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-06-05 to 2018-06-19
- Status changed from In Progress to Feedback
- Target version changed from Milestone 16 to Milestone 17
PR not merged yet, setting to feedback
Updated by okurz over 6 years ago
- Target version changed from Milestone 17 to Milestone 17
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-06-19 to 2018-07-03
discussion in PR is ongoing - moving
Updated by SLindoMansilla over 6 years ago
- Related to coordination #35302: [qe-core][opensuse][functional][epic][sporadic] Various unstable tests on o3 added
Updated by SLindoMansilla over 6 years ago
- Blocks action #37354: [opensuse][functional][u][sporadic][medium] test fails in desktop_runner is unstable added
Updated by okurz over 6 years ago
- Has duplicate action #37916: [sle][functional][u] test fails in gnome_control_center - control center doesn't show up added
Updated by okurz over 6 years ago
- Related to action #36304: [opensuse][functional]sporadic][u] chromium test is unstable added
Updated by okurz over 6 years ago
- Priority changed from Normal to High
So now more and more tasks depend on this, bumping prio
Updated by zluo over 6 years ago
- Status changed from Feedback to In Progress
working again on this issue
Updated by riafarov over 6 years ago
- Blocks action #36730: [sle][functional][y][medium] no route to the host 10.0.2.1 iscsi_client added
Updated by okurz over 6 years ago
- Blocks action #37662: [opensuse][functional][u] test fails in multi_users_dm added
Updated by zluo over 6 years ago
verification run:
http://e13.suse.de/tests/6070#step/force_cron_run (tw)
http://e13.suse.de/tests/6072#step/force_cron_run (sles 12 sp4)
http://e13.suse.de/tests/6077#step/force_cron_run (leap 15.0)
Updated by zluo over 6 years ago
Updated by SLindoMansilla over 6 years ago
PR merged, creating two tickets to handle:
- system_performance scenarios: https://openqa.opensuse.org/tests?match=system_performance
- renaming of force_cron_run
Updated by SLindoMansilla over 6 years ago
- Related to action #38090: [sle][functional] Do not mask systemd timers on system_performance scenarios added
Updated by SLindoMansilla over 6 years ago
- Related to action #38093: [sle][functional] Rename the module force_cron_run, it is no more only for cron jobs added
Updated by SLindoMansilla over 6 years ago
- Status changed from In Progress to Resolved
Now that the following steps to be done are properly defined on follow up tickets (following agile management), I can consider this planned ticket as resolved.
Next tasks (that were not planned for this sprint) are ready to be refined and planned:
- Do not mask systemd timers on system_performance scenarios: https://progress.opensuse.org/issues/38090
- Rename the module force_cron_run, it is no more only for cron jobs: https://progress.opensuse.org/issues/38093
Updated by SLindoMansilla over 6 years ago
- Related to action #18558: [sle][functional][u][hard][investigation] Snapper tests run too long added
Updated by okurz over 6 years ago
- Related to action #38228: [sle12sp4][functional][u][fast] test fails in force_scheduled_tasks - need wait more time than 90s for the cron jobs finish added
Updated by riafarov about 6 years ago
- Blocks deleted (action #36730: [sle][functional][y][medium] no route to the host 10.0.2.1 iscsi_client )
Updated by okurz about 6 years ago
- Related to action #41459: [sle][functional][u] Explicit test module for btrfs snapshots cleanup performance added