jjkiers/cloudron-box
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
19,043 Commits 33 Branches 583 Tags
2c10ceba5b80e84f417f6c61885a55df0e004036
Commit Graph

83 Commits

Author SHA1 Message Date
Girish Ramakrishnan
9e82839fb7 rsync: bump empty dir limit to 80k 2025-11-24 12:06:52 +01:00
Girish Ramakrishnan
e106dcd76a storage: pass limits object to backend 2025-11-14 13:18:21 +01:00
Girish Ramakrishnan
07732310c1 backuptask: track copy and upload statistics 2025-10-20 14:09:12 +02:00
Girish Ramakrishnan
10cf8f1d0e rsync: integrity fixes 2025-10-09 00:23:44 +02:00
Girish Ramakrishnan
43e426ab9f Revert "Add no-use-before-define linter rule" 
This reverts commit fdcc5d68a2.

Unfortunately, this requires us to move exports to the bottom.
This in turn causes circular dep issues and also access of
exports.GLOBAL_VAR in the global context
 2025-10-08 21:17:52 +02:00
Girish Ramakrishnan
505c71855e retry download only 3 times 2025-10-07 20:42:04 +02:00
Girish Ramakrishnan
ac7c32ad4c backups: log encryption and format in upload 2025-10-07 17:42:51 +02:00
Girish Ramakrishnan
832eca2d9c backup size: display stats correctly 2025-10-01 17:43:27 +02:00
Girish Ramakrishnan
c5b7264f1a rename backupTargets to backupSites 2025-09-12 10:32:37 +02:00
Girish Ramakrishnan
31df40a841 storage: add copyDir 
we changed listDir in c44863a9bb to list
a directory . this broke copy for files since a '/' is added when listing
the file.
 2025-08-26 00:01:14 +02:00
Girish Ramakrishnan
1e4df539b7 improve the logs 2025-08-25 19:15:08 +02:00
Girish Ramakrishnan
20b75b4065 rsync: integrity check 2025-08-16 19:29:59 +05:30
Girish Ramakrishnan
19682ec21b tgz: integrity check 2025-08-15 21:23:39 +05:30
Girish Ramakrishnan
e7fb444bb3 backuptask: save backupinfo in one place 2025-08-15 14:33:31 +05:30
Girish Ramakrishnan
12e073e8cf use node: prefix for requires 
mostly because code is being autogenerated by all the AI stuff using
this prefix. it's also used in the stack trace.
 2025-08-14 12:55:35 +05:30
Girish Ramakrishnan
2607ad2e24 rsync: add fileCount to stats and fix integrity collection 2025-08-14 10:05:13 +05:30
Girish Ramakrishnan
d7b5d1f947 rename .checksum to .backupinfo 2025-08-13 21:29:41 +05:30
Girish Ramakrishnan
0c79dcdf1b tgz: add fileCount to integrity 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
cb1bc6cfdf tgz: add fileCount to stats 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
b5c9f034ca Use hash stream instead of crypto.createHash 
crypto.createHash is a write stream but not a PassThrough stream!
 2025-08-13 18:48:22 +05:30
Girish Ramakrishnan
0911de205f stringify the Map correctly 2025-08-13 17:21:46 +05:30
Girish Ramakrishnan
59b428f24b decrease the JSON indent 2025-08-13 15:55:00 +05:30
Girish Ramakrishnan
54e8395789 add missing require 2025-08-13 15:25:23 +05:30
Girish Ramakrishnan
2fabfbe8f6 rsync: add integrity information 2025-08-13 14:43:05 +05:30
Girish Ramakrishnan
28ac9e153e rsync: make processSyncerChange a local function 2025-08-13 09:09:48 +05:30
Girish Ramakrishnan
dadbf1de90 tgz: save integrity information as a Map 
just for consistency because rsync will use a Map
 2025-08-13 08:39:03 +05:30
Girish Ramakrishnan
2e16dd983f backups: stash the stats to the backups table 2025-08-12 20:21:54 +05:30
Girish Ramakrishnan
47fc9561ab backups (tgz): save integrity information 
we generate a signing key pair for each target. Initially, I had this
as global. We needed a route to return the public key and putting it
under backup target seemed natural. Since we delete the backups when
we delete a target, we lose all the signing hashes. So, it's fine to lose
the key pair on target delete.
 2025-08-12 19:00:29 +05:30
Girish Ramakrishnan
5992658164 restore: fix restore 2025-08-05 14:38:12 +02:00
Girish Ramakrishnan
4fcaae8053 backup: fix listDir to return path instead of fullPath 2025-08-02 10:24:51 +02:00
Girish Ramakrishnan
c935744f4c backups: root ~~canal~~ path surgery 
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
 2025-08-02 10:15:34 +02:00
Girish Ramakrishnan
3cabbc1328 backupformat: getFileExtension 2025-08-01 22:58:26 +02:00
Girish Ramakrishnan
7192439b2c no need for format specific getBackupFilePath 2025-08-01 20:49:11 +02:00
Girish Ramakrishnan
ff6cbf6628 backups: encode extension into the remotePath 2025-08-01 20:42:12 +02:00
Girish Ramakrishnan
ea419509f1 backups: add setup/teardown 
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js

we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
 2025-08-01 15:36:25 +02:00
Girish Ramakrishnan
ae3a34287a backup target: create snapshot and cache files per target 
snapshot file tracks the snapshot directory. when app gets deleted,
the cleaner will remove the upstream snapshot directory when it runs.

cache files are used in rsync logic to track what was uploading into
snapshot in the previous run without needing to rescan upstream.
 2025-07-30 11:44:42 +02:00
Girish Ramakrishnan
5be03c7ab5 backups: fix app restore with rsync 2025-07-25 13:49:37 +02:00
Girish Ramakrishnan
3aafbd2ccb backups: add backup multiple targets 2025-07-25 01:30:27 +02:00
Girish Ramakrishnan
8b5fe714e1 rsync: throw on del error 2025-07-14 21:59:56 +02:00
Girish Ramakrishnan
6f653a603b rsync: first try with one file. if that works, go faster 
this works better if encryption fails. intead of downloading many files
and erroring non-stop
 2025-07-14 16:29:18 +02:00
Girish Ramakrishnan
15ac81c1cd s3: format errors with .Code 2025-07-14 15:54:31 +02:00
Girish Ramakrishnan
26637c0f1c backuptask: improve the debugs 2025-07-14 15:16:28 +02:00
Girish Ramakrishnan
c7572f0624 rename function 2025-02-13 17:07:34 +01:00
Girish Ramakrishnan
b94ce542c3 syncer: simply return the changes 
this is easier to test. the initial code wanted to make the changes a stream.
but this never happenned since the need never arose
 2025-02-13 17:06:24 +01:00
Girish Ramakrishnan
9888aa8c08 storage: rework listDir api to be a generator (like) function 2025-02-12 19:34:57 +01:00
Girish Ramakrishnan
b6f70e4bc0 rsync: increase empty dir limit 
a mail backup of a mailbox with many folders can have many empty dirs

https://forum.cloudron.io/topic/13047/since-update-to-v8-2-1-backups-fail-with-too-many-empty-directories
 2025-01-03 13:01:10 +01:00
Girish Ramakrishnan
fd96665e97 rsync: show better error message with too many empty dirs, symlinks or executables 2024-11-18 08:11:14 +05:30
Girish Ramakrishnan
6c3ca9c364 shell: rework code to use shell.spawn 
spawn gives out streams and we have more control over the stdout/stderr
buffers. otherwise, we have to provide a max buffer capture size to exec
 2024-10-15 12:13:46 +02:00
Girish Ramakrishnan
a9e1d7641d shell: make require take a tag 2024-10-14 21:08:32 +02:00
Girish Ramakrishnan
083432cbfe test: add EnsureFileSizeStream test 2024-07-18 15:39:45 +02:00