Commit Graph

31 Commits

Author SHA1 Message Date
Girish Ramakrishnan
7047ee9391 shell: add timeout logic and rework error handling
what's important:

* if task code ran, it exits with 0. this code is regardless of (error, result)
  * when it exited cleanly, we will get the values from the database

* if task timed out, the box code kills it and it has a flag tracking timedOut. we can
  ignore exit code in this case.

* if task code was stopped, box code will send SIGTERM which ideally it will handle and end with 70.

* if task code crashed and it caught the exception, it will return 50

* if task code crashed and node nuked us, it will exit with 1

* if task code was killed with some unhandleabe signal, taskworker.sh will return the signal (9=SIGKILL)
2025-07-17 12:44:24 +02:00
Girish Ramakrishnan
6364ea8e57 grammar 2025-07-17 08:30:51 +02:00
Girish Ramakrishnan
fb39aa32bb tasks: fix update failed notification
https://forum.cloudron.io/topic/13408/update-to-cloudron-8.3-error

We get a Task xx crashed with code null in the notification.

The crux of the issue is that we use KillMode=control-group. This ends
up sending SIGTERM signal to box code and all the sudo in parallel. The box
code then sees the sudo die and records the task as failed.

To fix, we switch to KillMode=mixed. This gives box code a chance to handle SIGTERM
first. It cleans out its task list and kills all the sudo.
2025-06-17 23:47:04 +02:00
Girish Ramakrishnan
72d29030d9 tasks: remove redundant BindsTo= 2025-06-16 18:19:30 +02:00
Girish Ramakrishnan
5cf98922fb tasks: add note on why we use systemd-run 2025-06-16 18:14:21 +02:00
Girish Ramakrishnan
6a2f2b4efe task: do not use --pipe
--pipe make the spawned task inherit the systemd-run's stdio. if/when
system-run is killed, it might take out the spawned task as well with EPIPE.
2025-06-16 14:20:07 +02:00
Girish Ramakrishnan
69cb8c5a0a task: can use env -S in ubuntu 20 and above 2025-06-16 14:18:49 +02:00
Girish Ramakrishnan
0ebda97b03 remove various 18.04 specific code 2025-06-15 13:58:28 +02:00
Girish Ramakrishnan
e7b11a7ceb typo 2025-02-13 10:31:20 +01:00
Girish Ramakrishnan
a138425298 storage: start migration of s3 api 2025-02-12 23:04:37 +01:00
Girish Ramakrishnan
d5ea99603f backups: give is a low oomScoreAdjust to not get killed 2024-07-19 13:05:09 +02:00
Girish Ramakrishnan
450dd70ea2 backups: up min memory limit to 1GB 2024-02-19 17:02:14 +01:00
Girish Ramakrishnan
ec23c7d2b8 Suppress aws sdk warning
https://github.com/aws/aws-sdk-js/issues/4354#issuecomment-1664694545
2023-08-04 09:21:48 +05:30
Girish Ramakrishnan
817e950d47 Fix upstreamUri verification 2022-11-23 12:58:17 +01:00
Girish Ramakrishnan
8bc3b832e7 detect oom in tasks correctly 2022-11-02 22:39:25 +01:00
Girish Ramakrishnan
80a3ca0f46 remove 16.04 related task logic 2022-11-02 21:22:42 +01:00
Girish Ramakrishnan
0f0a98f7ac Add TimeoutStopSec=10s for systemctl kill to work faster 2022-11-02 18:46:20 +01:00
Girish Ramakrishnan
88a5526e9b starttask: set NODE_OPTIONS instead of env -S 2021-05-14 09:36:00 -07:00
Johannes Zellner
5e48b69d3b Revert "taskworker: Use --unhandled-rejections=strict"
Breaks the task argument count

This reverts commit b7643ae3b3.
2021-05-14 11:13:52 +02:00
Johannes Zellner
a43e804ee2 Revert "taskworker: put the arg in shebang line"
Not supported on ubuntu 18

This reverts commit e6edc4e999.
2021-05-14 10:51:37 +02:00
Girish Ramakrishnan
e6edc4e999 taskworker: put the arg in shebang line
otherwise, it gets passed as an arg to the script and is visible in process.argv!
2021-05-13 22:49:15 -07:00
Girish Ramakrishnan
b7643ae3b3 taskworker: Use --unhandled-rejections=strict
this way, those tasks crash and do not hang on bad code
2021-05-13 22:32:12 -07:00
Girish Ramakrishnan
5adfa722d4 Add some debug information 2020-09-01 12:35:31 -07:00
Girish Ramakrishnan
cf9439fb3b systemd 237 ignores --nice value in systemd-run 2020-08-26 17:30:47 -07:00
Girish Ramakrishnan
7039108438 pass memory limit as argument to starttask.sh 2020-08-10 21:53:07 -07:00
Girish Ramakrishnan
e85f0a4f52 Rename to box-task
this way we can do systemctl stop box*
2020-08-09 11:14:11 -07:00
Girish Ramakrishnan
da72597dd3 Fix start/stop task scripts for ubuntu 16 2020-08-08 11:10:02 -07:00
Girish Ramakrishnan
f8e22a0730 Fix tests 2020-08-07 00:21:15 -07:00
Girish Ramakrishnan
114b45882a Set memory limit to 400M for tasks 2020-08-07 00:21:15 -07:00
Girish Ramakrishnan
b1b6f70118 Kill all tasks on shutdown and startup
BindsTo will kill all the tasks when systemctl stop box is executed.
But when restarted, it keeps the tasks running. Because of this behavior,
we kill the tasks on startup and stop of the box code.
2020-08-06 23:47:40 -07:00
Girish Ramakrishnan
ac63d00c93 run tasks as separate cgroup via systemd
this allows us to adjust the nice value and memory settings per task

part of #691
2020-08-06 16:43:14 -07:00