Commit Graph

47 Commits

Author SHA1 Message Date
Girish Ramakrishnan 07732310c1 backuptask: track copy and upload statistics 2025-10-20 14:09:12 +02:00
Girish Ramakrishnan 505c71855e retry download only 3 times 2025-10-07 20:42:04 +02:00
Girish Ramakrishnan ac7c32ad4c backups: log encryption and format in upload 2025-10-07 17:42:51 +02:00
Girish Ramakrishnan 832eca2d9c backup size: display stats correctly 2025-10-01 17:43:27 +02:00
Girish Ramakrishnan c5b7264f1a rename backupTargets to backupSites 2025-09-12 10:32:37 +02:00
Girish Ramakrishnan 31df40a841 storage: add copyDir
we changed listDir in c44863a9bb to list
a directory . this broke copy for files since a '/' is added when listing
the file.
2025-08-26 00:01:14 +02:00
Girish Ramakrishnan 19682ec21b tgz: integrity check 2025-08-15 21:23:39 +05:30
Girish Ramakrishnan e7fb444bb3 backuptask: save backupinfo in one place 2025-08-15 14:33:31 +05:30
Girish Ramakrishnan 12e073e8cf use node: prefix for requires
mostly because code is being autogenerated by all the AI stuff using
this prefix. it's also used in the stack trace.
2025-08-14 12:55:35 +05:30
Girish Ramakrishnan d7b5d1f947 rename .checksum to .backupinfo 2025-08-13 21:29:41 +05:30
Girish Ramakrishnan 0c79dcdf1b tgz: add fileCount to integrity 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan cb1bc6cfdf tgz: add fileCount to stats 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan b5c9f034ca Use hash stream instead of crypto.createHash
crypto.createHash is a write stream but not a PassThrough stream!
2025-08-13 18:48:22 +05:30
Girish Ramakrishnan 0911de205f stringify the Map correctly 2025-08-13 17:21:46 +05:30
Girish Ramakrishnan 59b428f24b decrease the JSON indent 2025-08-13 15:55:00 +05:30
Girish Ramakrishnan 2fabfbe8f6 rsync: add integrity information 2025-08-13 14:43:05 +05:30
Girish Ramakrishnan dadbf1de90 tgz: save integrity information as a Map
just for consistency because rsync will use a Map
2025-08-13 08:39:03 +05:30
Girish Ramakrishnan 2e16dd983f backups: stash the stats to the backups table 2025-08-12 20:21:54 +05:30
Girish Ramakrishnan 47fc9561ab backups (tgz): save integrity information
we generate a signing key pair for each target. Initially, I had this
as global. We needed a route to return the public key and putting it
under backup target seemed natural. Since we delete the backups when
we delete a target, we lose all the signing hashes. So, it's fine to lose
the key pair on target delete.
2025-08-12 19:00:29 +05:30
Girish Ramakrishnan 5992658164 restore: fix restore 2025-08-05 14:38:12 +02:00
Girish Ramakrishnan c935744f4c backups: root ~~canal~~ path surgery
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
2025-08-02 10:15:34 +02:00
Girish Ramakrishnan 3cabbc1328 backupformat: getFileExtension 2025-08-01 22:58:26 +02:00
Girish Ramakrishnan 7192439b2c no need for format specific getBackupFilePath 2025-08-01 20:49:11 +02:00
Girish Ramakrishnan ff6cbf6628 backups: encode extension into the remotePath 2025-08-01 20:42:12 +02:00
Girish Ramakrishnan ea419509f1 backups: add setup/teardown
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js

we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
2025-08-01 15:36:25 +02:00
Girish Ramakrishnan 3aafbd2ccb backups: add backup multiple targets 2025-07-25 01:30:27 +02:00
Girish Ramakrishnan 083432cbfe test: add EnsureFileSizeStream test 2024-07-18 15:39:45 +02:00
Girish Ramakrishnan dbbce4160d tgz: underflow/overflow proxy stream
In tar, the entry header contains the file size. If we don't provide it those many bytes, the tar will become corrupt
Linux provides no guarantee of how many bytes can be read from a file. This is the case with sqlite and log files
which are accessed by other processes when tar is in action. This class handles overflow and underflow
2024-07-18 15:13:38 +02:00
Girish Ramakrishnan 885aac69c5 tgz: handle addEntryToPack to error 2024-07-18 14:47:31 +02:00
Girish Ramakrishnan b3c301fc2a lint 2024-07-18 13:31:29 +02:00
Girish Ramakrishnan 8754a208b1 tgz: preserve mode 2024-07-11 18:10:40 +02:00
Girish Ramakrishnan 3a21191fba tgz: fix error handling 2024-07-10 19:10:24 +02:00
Girish Ramakrishnan 1dc6b40a68 tgz: extract using tar-stream directly
we used have a fork of tar-fs. using tar-stream directly gives us
more control
2024-07-08 13:06:56 +02:00
Girish Ramakrishnan b64b513b14 Revert "use node-tar for extract"
This reverts commit 285feb4f8b.
2024-07-05 09:26:38 +02:00
Girish Ramakrishnan 285feb4f8b use node-tar for extract
we will switch over our tgz module to node-tar. Main advantage is that
it is used by npm. Currently, we have our own fork to ignore stat errors
in the other module.

unfortunately, I cannot get this to work with the create logic. It doesn't
support path modification - https://github.com/isaacs/node-tar/issues/271
so, will revert this immediately and keep this for future
2024-07-05 09:26:28 +02:00
Johannes Zellner 340ea3fe9b Fix variable usage bug for noop backup provider 2023-12-18 13:23:40 +01:00
Johannes Zellner 9d9a407c3d Noop provider does not have a rootPath set 2023-10-16 16:36:31 +02:00
Girish Ramakrishnan aa8c23c8b3 rework backup root
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from  same backupConfig but different format. we do this by saving the format in the backups table

fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan 68b4bf1667 backupformat: print the backupFilePath 2023-08-02 09:50:34 +05:30
Girish Ramakrishnan 9a359a27f5 backups: download is now async 2023-07-25 10:33:03 +05:30
Girish Ramakrishnan c4f4f3e914 logs: use %o to format error
otherwise, they are printed as multi-line and this messes up tail+date formatting
2023-04-16 10:49:59 +02:00
Girish Ramakrishnan 962d7030bb replace progress-stream with our implementation
upstream is mostly unmaintained
2022-11-06 10:17:14 +01:00
Girish Ramakrishnan 8a5d4e2fb0 better debugs 2022-11-05 08:43:02 +01:00
Girish Ramakrishnan 074e9cfd93 rename getRootPath to getBackupRootPath 2022-10-02 16:26:27 +02:00
Girish Ramakrishnan 6e98f5f36c backuptask: make upload/download async 2022-04-30 16:42:14 -07:00
Girish Ramakrishnan 26cf084e1c tarPack/tarExtract do not need a callback 2022-04-28 21:58:00 -07:00
Girish Ramakrishnan 7123ec433c split up backupformat logic into separate files 2022-04-28 19:10:57 -07:00