Commit Graph

41 Commits

Author SHA1 Message Date
Girish Ramakrishnan
19682ec21b tgz: integrity check 2025-08-15 21:23:39 +05:30
Girish Ramakrishnan
e7fb444bb3 backuptask: save backupinfo in one place 2025-08-15 14:33:31 +05:30
Girish Ramakrishnan
12e073e8cf use node: prefix for requires
mostly because code is being autogenerated by all the AI stuff using
this prefix. it's also used in the stack trace.
2025-08-14 12:55:35 +05:30
Girish Ramakrishnan
d7b5d1f947 rename .checksum to .backupinfo 2025-08-13 21:29:41 +05:30
Girish Ramakrishnan
0c79dcdf1b tgz: add fileCount to integrity 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
cb1bc6cfdf tgz: add fileCount to stats 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
b5c9f034ca Use hash stream instead of crypto.createHash
crypto.createHash is a write stream but not a PassThrough stream!
2025-08-13 18:48:22 +05:30
Girish Ramakrishnan
0911de205f stringify the Map correctly 2025-08-13 17:21:46 +05:30
Girish Ramakrishnan
59b428f24b decrease the JSON indent 2025-08-13 15:55:00 +05:30
Girish Ramakrishnan
2fabfbe8f6 rsync: add integrity information 2025-08-13 14:43:05 +05:30
Girish Ramakrishnan
dadbf1de90 tgz: save integrity information as a Map
just for consistency because rsync will use a Map
2025-08-13 08:39:03 +05:30
Girish Ramakrishnan
2e16dd983f backups: stash the stats to the backups table 2025-08-12 20:21:54 +05:30
Girish Ramakrishnan
47fc9561ab backups (tgz): save integrity information
we generate a signing key pair for each target. Initially, I had this
as global. We needed a route to return the public key and putting it
under backup target seemed natural. Since we delete the backups when
we delete a target, we lose all the signing hashes. So, it's fine to lose
the key pair on target delete.
2025-08-12 19:00:29 +05:30
Girish Ramakrishnan
5992658164 restore: fix restore 2025-08-05 14:38:12 +02:00
Girish Ramakrishnan
c935744f4c backups: root ~~canal~~ path surgery
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
2025-08-02 10:15:34 +02:00
Girish Ramakrishnan
3cabbc1328 backupformat: getFileExtension 2025-08-01 22:58:26 +02:00
Girish Ramakrishnan
7192439b2c no need for format specific getBackupFilePath 2025-08-01 20:49:11 +02:00
Girish Ramakrishnan
ff6cbf6628 backups: encode extension into the remotePath 2025-08-01 20:42:12 +02:00
Girish Ramakrishnan
ea419509f1 backups: add setup/teardown
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js

we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
2025-08-01 15:36:25 +02:00
Girish Ramakrishnan
3aafbd2ccb backups: add backup multiple targets 2025-07-25 01:30:27 +02:00
Girish Ramakrishnan
083432cbfe test: add EnsureFileSizeStream test 2024-07-18 15:39:45 +02:00
Girish Ramakrishnan
dbbce4160d tgz: underflow/overflow proxy stream
In tar, the entry header contains the file size. If we don't provide it those many bytes, the tar will become corrupt
Linux provides no guarantee of how many bytes can be read from a file. This is the case with sqlite and log files
which are accessed by other processes when tar is in action. This class handles overflow and underflow
2024-07-18 15:13:38 +02:00
Girish Ramakrishnan
885aac69c5 tgz: handle addEntryToPack to error 2024-07-18 14:47:31 +02:00
Girish Ramakrishnan
b3c301fc2a lint 2024-07-18 13:31:29 +02:00
Girish Ramakrishnan
8754a208b1 tgz: preserve mode 2024-07-11 18:10:40 +02:00
Girish Ramakrishnan
3a21191fba tgz: fix error handling 2024-07-10 19:10:24 +02:00
Girish Ramakrishnan
1dc6b40a68 tgz: extract using tar-stream directly
we used have a fork of tar-fs. using tar-stream directly gives us
more control
2024-07-08 13:06:56 +02:00
Girish Ramakrishnan
b64b513b14 Revert "use node-tar for extract"
This reverts commit 285feb4f8b.
2024-07-05 09:26:38 +02:00
Girish Ramakrishnan
285feb4f8b use node-tar for extract
we will switch over our tgz module to node-tar. Main advantage is that
it is used by npm. Currently, we have our own fork to ignore stat errors
in the other module.

unfortunately, I cannot get this to work with the create logic. It doesn't
support path modification - https://github.com/isaacs/node-tar/issues/271
so, will revert this immediately and keep this for future
2024-07-05 09:26:28 +02:00
Johannes Zellner
340ea3fe9b Fix variable usage bug for noop backup provider 2023-12-18 13:23:40 +01:00
Johannes Zellner
9d9a407c3d Noop provider does not have a rootPath set 2023-10-16 16:36:31 +02:00
Girish Ramakrishnan
aa8c23c8b3 rework backup root
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from  same backupConfig but different format. we do this by saving the format in the backups table

fixes #819
2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
68b4bf1667 backupformat: print the backupFilePath 2023-08-02 09:50:34 +05:30
Girish Ramakrishnan
9a359a27f5 backups: download is now async 2023-07-25 10:33:03 +05:30
Girish Ramakrishnan
c4f4f3e914 logs: use %o to format error
otherwise, they are printed as multi-line and this messes up tail+date formatting
2023-04-16 10:49:59 +02:00
Girish Ramakrishnan
962d7030bb replace progress-stream with our implementation
upstream is mostly unmaintained
2022-11-06 10:17:14 +01:00
Girish Ramakrishnan
8a5d4e2fb0 better debugs 2022-11-05 08:43:02 +01:00
Girish Ramakrishnan
074e9cfd93 rename getRootPath to getBackupRootPath 2022-10-02 16:26:27 +02:00
Girish Ramakrishnan
6e98f5f36c backuptask: make upload/download async 2022-04-30 16:42:14 -07:00
Girish Ramakrishnan
26cf084e1c tarPack/tarExtract do not need a callback 2022-04-28 21:58:00 -07:00
Girish Ramakrishnan
7123ec433c split up backupformat logic into separate files 2022-04-28 19:10:57 -07:00