Commit Graph

62 Commits

Author SHA1 Message Date
Girish Ramakrishnan
59b428f24b decrease the JSON indent 2025-08-13 15:55:00 +05:30
Girish Ramakrishnan
54e8395789 add missing require 2025-08-13 15:25:23 +05:30
Girish Ramakrishnan
2fabfbe8f6 rsync: add integrity information 2025-08-13 14:43:05 +05:30
Girish Ramakrishnan
28ac9e153e rsync: make processSyncerChange a local function 2025-08-13 09:09:48 +05:30
Girish Ramakrishnan
dadbf1de90 tgz: save integrity information as a Map
just for consistency because rsync will use a Map
2025-08-13 08:39:03 +05:30
Girish Ramakrishnan
2e16dd983f backups: stash the stats to the backups table 2025-08-12 20:21:54 +05:30
Girish Ramakrishnan
47fc9561ab backups (tgz): save integrity information
we generate a signing key pair for each target. Initially, I had this
as global. We needed a route to return the public key and putting it
under backup target seemed natural. Since we delete the backups when
we delete a target, we lose all the signing hashes. So, it's fine to lose
the key pair on target delete.
2025-08-12 19:00:29 +05:30
Girish Ramakrishnan
5992658164 restore: fix restore 2025-08-05 14:38:12 +02:00
Girish Ramakrishnan
4fcaae8053 backup: fix listDir to return path instead of fullPath 2025-08-02 10:24:51 +02:00
Girish Ramakrishnan
c935744f4c backups: root ~~canal~~ path surgery
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
2025-08-02 10:15:34 +02:00
Girish Ramakrishnan
3cabbc1328 backupformat: getFileExtension 2025-08-01 22:58:26 +02:00
Girish Ramakrishnan
7192439b2c no need for format specific getBackupFilePath 2025-08-01 20:49:11 +02:00
Girish Ramakrishnan
ff6cbf6628 backups: encode extension into the remotePath 2025-08-01 20:42:12 +02:00
Girish Ramakrishnan
ea419509f1 backups: add setup/teardown
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js

we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
2025-08-01 15:36:25 +02:00
Girish Ramakrishnan
ae3a34287a backup target: create snapshot and cache files per target
snapshot file tracks the snapshot directory. when app gets deleted,
the cleaner will remove the upstream snapshot directory when it runs.

cache files are used in rsync logic to track what was uploading into
snapshot in the previous run without needing to rescan upstream.
2025-07-30 11:44:42 +02:00
Girish Ramakrishnan
5be03c7ab5 backups: fix app restore with rsync 2025-07-25 13:49:37 +02:00
Girish Ramakrishnan
3aafbd2ccb backups: add backup multiple targets 2025-07-25 01:30:27 +02:00
Girish Ramakrishnan
8b5fe714e1 rsync: throw on del error 2025-07-14 21:59:56 +02:00
Girish Ramakrishnan
6f653a603b rsync: first try with one file. if that works, go faster
this works better if encryption fails. intead of downloading many files
and erroring non-stop
2025-07-14 16:29:18 +02:00
Girish Ramakrishnan
15ac81c1cd s3: format errors with .Code 2025-07-14 15:54:31 +02:00
Girish Ramakrishnan
26637c0f1c backuptask: improve the debugs 2025-07-14 15:16:28 +02:00
Girish Ramakrishnan
c7572f0624 rename function 2025-02-13 17:07:34 +01:00
Girish Ramakrishnan
b94ce542c3 syncer: simply return the changes
this is easier to test. the initial code wanted to make the changes a stream.
but this never happenned since the need never arose
2025-02-13 17:06:24 +01:00
Girish Ramakrishnan
9888aa8c08 storage: rework listDir api to be a generator (like) function 2025-02-12 19:34:57 +01:00
Girish Ramakrishnan
b6f70e4bc0 rsync: increase empty dir limit
a mail backup of a mailbox with many folders can have many empty dirs

https://forum.cloudron.io/topic/13047/since-update-to-v8-2-1-backups-fail-with-too-many-empty-directories
2025-01-03 13:01:10 +01:00
Girish Ramakrishnan
fd96665e97 rsync: show better error message with too many empty dirs, symlinks or executables 2024-11-18 08:11:14 +05:30
Girish Ramakrishnan
6c3ca9c364 shell: rework code to use shell.spawn
spawn gives out streams and we have more control over the stdout/stderr
buffers. otherwise, we have to provide a max buffer capture size to exec
2024-10-15 12:13:46 +02:00
Girish Ramakrishnan
a9e1d7641d shell: make require take a tag 2024-10-14 21:08:32 +02:00
Girish Ramakrishnan
083432cbfe test: add EnsureFileSizeStream test 2024-07-18 15:39:45 +02:00
Girish Ramakrishnan
dbbce4160d tgz: underflow/overflow proxy stream
In tar, the entry header contains the file size. If we don't provide it those many bytes, the tar will become corrupt
Linux provides no guarantee of how many bytes can be read from a file. This is the case with sqlite and log files
which are accessed by other processes when tar is in action. This class handles overflow and underflow
2024-07-18 15:13:38 +02:00
Girish Ramakrishnan
885aac69c5 tgz: handle addEntryToPack to error 2024-07-18 14:47:31 +02:00
Girish Ramakrishnan
b3c301fc2a lint 2024-07-18 13:31:29 +02:00
Girish Ramakrishnan
8754a208b1 tgz: preserve mode 2024-07-11 18:10:40 +02:00
Girish Ramakrishnan
d6bb32aead syncer: expose as async 2024-07-10 19:10:55 +02:00
Girish Ramakrishnan
3a21191fba tgz: fix error handling 2024-07-10 19:10:24 +02:00
Girish Ramakrishnan
baf598099f rsync: fix upload logic to match new upload api 2024-07-08 15:21:56 +02:00
Girish Ramakrishnan
7911780a16 const 2024-07-08 13:18:22 +02:00
Girish Ramakrishnan
1dc6b40a68 tgz: extract using tar-stream directly
we used have a fork of tar-fs. using tar-stream directly gives us
more control
2024-07-08 13:06:56 +02:00
Girish Ramakrishnan
b64b513b14 Revert "use node-tar for extract"
This reverts commit 285feb4f8b.
2024-07-05 09:26:38 +02:00
Girish Ramakrishnan
285feb4f8b use node-tar for extract
we will switch over our tgz module to node-tar. Main advantage is that
it is used by npm. Currently, we have our own fork to ignore stat errors
in the other module.

unfortunately, I cannot get this to work with the create logic. It doesn't
support path modification - https://github.com/isaacs/node-tar/issues/271
so, will revert this immediately and keep this for future
2024-07-05 09:26:28 +02:00
Girish Ramakrishnan
8fd7daade6 rsync: empty check was removed by mistake 2024-02-22 14:47:44 +01:00
Girish Ramakrishnan
a6f078330f shell: no need to promise scoping 2024-02-21 19:40:27 +01:00
Girish Ramakrishnan
14c9260ab0 shell: exec encoding is utf8 by default and no shell
explicitly mark calls that require the shell
2024-02-21 17:47:25 +01:00
Girish Ramakrishnan
9b94cf18d0 convert more execSync to async 2024-02-21 11:00:12 +01:00
Girish Ramakrishnan
1d1a7af48e rsync: bump the buffer size to 80MB 2024-02-19 14:15:28 +01:00
Johannes Zellner
340ea3fe9b Fix variable usage bug for noop backup provider 2023-12-18 13:23:40 +01:00
Johannes Zellner
9d9a407c3d Noop provider does not have a rootPath set 2023-10-16 16:36:31 +02:00
Girish Ramakrishnan
7fe2de448e remove pipeline() chain
it cannot be chained afaict
2023-08-29 17:44:02 +05:30
Girish Ramakrishnan
aa8c23c8b3 rework backup root
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from  same backupConfig but different format. we do this by saving the format in the backups table

fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
68b4bf1667 backupformat: print the backupFilePath 2023-08-02 09:50:34 +05:30