Girish Ramakrishnan
07732310c1
backuptask: track copy and upload statistics
2025-10-20 14:09:12 +02:00
Girish Ramakrishnan
10cf8f1d0e
rsync: integrity fixes
2025-10-09 00:23:44 +02:00
Girish Ramakrishnan
43e426ab9f
Revert "Add no-use-before-define linter rule"
...
This reverts commit fdcc5d68a2
2025-10-08 21:17:52 +02:00
Girish Ramakrishnan
505c71855e
retry download only 3 times
2025-10-07 20:42:04 +02:00
Girish Ramakrishnan
ac7c32ad4c
backups: log encryption and format in upload
2025-10-07 17:42:51 +02:00
Girish Ramakrishnan
832eca2d9c
backup size: display stats correctly
2025-10-01 17:43:27 +02:00
Girish Ramakrishnan
c5b7264f1a
rename backupTargets to backupSites
2025-09-12 10:32:37 +02:00
Girish Ramakrishnan
31df40a841
storage: add copyDir
...
we changed listDir in c44863a9bb
2025-08-26 00:01:14 +02:00
Girish Ramakrishnan
1e4df539b7
improve the logs
2025-08-25 19:15:08 +02:00
Girish Ramakrishnan
20b75b4065
rsync: integrity check
2025-08-16 19:29:59 +05:30
Girish Ramakrishnan
e7fb444bb3
backuptask: save backupinfo in one place
2025-08-15 14:33:31 +05:30
Girish Ramakrishnan
12e073e8cf
use node: prefix for requires
...
mostly because code is being autogenerated by all the AI stuff using
this prefix. it's also used in the stack trace.
2025-08-14 12:55:35 +05:30
Girish Ramakrishnan
2607ad2e24
rsync: add fileCount to stats and fix integrity collection
2025-08-14 10:05:13 +05:30
Girish Ramakrishnan
d7b5d1f947
rename .checksum to .backupinfo
2025-08-13 21:29:41 +05:30
Girish Ramakrishnan
b5c9f034ca
Use hash stream instead of crypto.createHash
...
crypto.createHash is a write stream but not a PassThrough stream!
2025-08-13 18:48:22 +05:30
Girish Ramakrishnan
0911de205f
stringify the Map correctly
2025-08-13 17:21:46 +05:30
Girish Ramakrishnan
59b428f24b
decrease the JSON indent
2025-08-13 15:55:00 +05:30
Girish Ramakrishnan
54e8395789
add missing require
2025-08-13 15:25:23 +05:30
Girish Ramakrishnan
2fabfbe8f6
rsync: add integrity information
2025-08-13 14:43:05 +05:30
Girish Ramakrishnan
28ac9e153e
rsync: make processSyncerChange a local function
2025-08-13 09:09:48 +05:30
Girish Ramakrishnan
4fcaae8053
backup: fix listDir to return path instead of fullPath
2025-08-02 10:24:51 +02:00
Girish Ramakrishnan
c935744f4c
backups: root ~~canal~~ path surgery
...
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
2025-08-02 10:15:34 +02:00
Girish Ramakrishnan
3cabbc1328
backupformat: getFileExtension
2025-08-01 22:58:26 +02:00
Girish Ramakrishnan
7192439b2c
no need for format specific getBackupFilePath
2025-08-01 20:49:11 +02:00
Girish Ramakrishnan
ea419509f1
backups: add setup/teardown
...
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js
we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
2025-08-01 15:36:25 +02:00
Girish Ramakrishnan
ae3a34287a
backup target: create snapshot and cache files per target
...
snapshot file tracks the snapshot directory. when app gets deleted,
the cleaner will remove the upstream snapshot directory when it runs.
cache files are used in rsync logic to track what was uploading into
snapshot in the previous run without needing to rescan upstream.
2025-07-30 11:44:42 +02:00
Girish Ramakrishnan
5be03c7ab5
backups: fix app restore with rsync
2025-07-25 13:49:37 +02:00
Girish Ramakrishnan
8b5fe714e1
rsync: throw on del error
2025-07-14 21:59:56 +02:00
Girish Ramakrishnan
6f653a603b
rsync: first try with one file. if that works, go faster
...
this works better if encryption fails. intead of downloading many files
and erroring non-stop
2025-07-14 16:29:18 +02:00
Girish Ramakrishnan
15ac81c1cd
s3: format errors with .Code
2025-07-14 15:54:31 +02:00
Girish Ramakrishnan
26637c0f1c
backuptask: improve the debugs
2025-07-14 15:16:28 +02:00
Girish Ramakrishnan
c7572f0624
rename function
2025-02-13 17:07:34 +01:00
Girish Ramakrishnan
b94ce542c3
syncer: simply return the changes
...
this is easier to test. the initial code wanted to make the changes a stream.
but this never happenned since the need never arose
2025-02-13 17:06:24 +01:00
Girish Ramakrishnan
9888aa8c08
storage: rework listDir api to be a generator (like) function
2025-02-12 19:34:57 +01:00
Girish Ramakrishnan
b6f70e4bc0
rsync: increase empty dir limit
...
a mail backup of a mailbox with many folders can have many empty dirs
https://forum.cloudron.io/topic/13047/since-update-to-v8-2-1-backups-fail-with-too-many-empty-directories
2025-01-03 13:01:10 +01:00
Girish Ramakrishnan
fd96665e97
rsync: show better error message with too many empty dirs, symlinks or executables
2024-11-18 08:11:14 +05:30
Girish Ramakrishnan
6c3ca9c364
shell: rework code to use shell.spawn
...
spawn gives out streams and we have more control over the stdout/stderr
buffers. otherwise, we have to provide a max buffer capture size to exec
2024-10-15 12:13:46 +02:00
Girish Ramakrishnan
a9e1d7641d
shell: make require take a tag
2024-10-14 21:08:32 +02:00
Girish Ramakrishnan
d6bb32aead
syncer: expose as async
2024-07-10 19:10:55 +02:00
Girish Ramakrishnan
baf598099f
rsync: fix upload logic to match new upload api
2024-07-08 15:21:56 +02:00
Girish Ramakrishnan
7911780a16
const
2024-07-08 13:18:22 +02:00
Girish Ramakrishnan
8fd7daade6
rsync: empty check was removed by mistake
2024-02-22 14:47:44 +01:00
Girish Ramakrishnan
a6f078330f
shell: no need to promise scoping
2024-02-21 19:40:27 +01:00
Girish Ramakrishnan
14c9260ab0
shell: exec encoding is utf8 by default and no shell
...
explicitly mark calls that require the shell
2024-02-21 17:47:25 +01:00
Girish Ramakrishnan
9b94cf18d0
convert more execSync to async
2024-02-21 11:00:12 +01:00
Girish Ramakrishnan
1d1a7af48e
rsync: bump the buffer size to 80MB
2024-02-19 14:15:28 +01:00
Johannes Zellner
9d9a407c3d
Noop provider does not have a rootPath set
2023-10-16 16:36:31 +02:00
Girish Ramakrishnan
7fe2de448e
remove pipeline() chain
...
it cannot be chained afaict
2023-08-29 17:44:02 +05:30
Girish Ramakrishnan
aa8c23c8b3
rework backup root
...
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from same backupConfig but different format. we do this by saving the format in the backups table
fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
68b4bf1667
backupformat: print the backupFilePath
2023-08-02 09:50:34 +05:30
