jjkiers/cloudron-box
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
19,644 Commits 33 Branches 583 Tags
01d0c738bc32bd0fd3dede81023cc8d6b2d49214
Commit Graph

55 Commits

Author SHA1 Message Date
Girish Ramakrishnan
01d0c738bc replace debug() with our custom logger 
mostly we want trace() and log(). trace() can be enabled whenever
we want by flipping a flag and restarting box
 2026-03-12 23:08:35 +05:30
Girish Ramakrishnan
d57554a48c backup logs: make them much terse and concise 
these are making the rsync logs massive. instead resort to reporting
progress based on file count. there is also a heartbeat timer for
"stuck" or "long downloading" files, every minute.
 2026-03-12 19:40:46 +05:30
Girish Ramakrishnan
b5a1554631 Fix various linter errors 2026-02-15 19:37:30 +01:00
Girish Ramakrishnan
36aa641cb9 migrate to "export default" 
also, set no-use-before-define in linter
 2026-02-14 15:43:24 +01:00
Girish Ramakrishnan
96dc79cfe6 Migrate codebase from CommonJS to ES Modules 
- Convert all require()/module.exports to import/export across 260+ files
- Add "type": "module" to package.json to enable ESM by default
- Add migrations/package.json with "type": "commonjs" to keep db-migrate compatible
- Convert eslint.config.js to ESM with sourceType: "module"
- Replace __dirname/__filename with import.meta.dirname/import.meta.filename
- Replace require.main === module with process.argv[1] === import.meta.filename
- Remove 'use strict' directives (implicit in ESM)
- Convert dynamic require() in switch statements to static import lookup maps
  (dns.js, domains.js, backupformats.js, backupsites.js, network.js)
- Extract self-referencing exports.CONSTANT patterns into standalone const
  declarations (apps.js, services.js, locks.js, users.js, mail.js, etc.)
- Lazify SERVICES object in services.js to avoid circular dependency TDZ issues
- Add clearMailQueue() to mailer.js for ESM-safe queue clearing in tests
- Add _setMockApp() to ldapserver.js for ESM-safe test mocking
- Add _setMockResolve() wrapper to dig.js for ESM-safe DNS mocking in tests
- Convert backupupload.js to use dynamic imports so --check exits before
  loading the module graph (which requires BOX_ENV)
- Update check-install to use ESM import for infra_version.js
- Convert scripts/ (hotfix, release, remote_hotfix.js, find-unused-translations)
- All 1315 tests passing

Migration stats (AI-assisted using Cursor with Claude):
- Wall clock time: ~3-4 hours
- Assistant completions: ~80-100
- Estimated token usage: ~1-2M tokens

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-14 15:11:45 +01:00
Girish Ramakrishnan
5276321ade integrity: add integrity check fields and initial UI 2026-02-08 23:26:57 +01:00
Girish Ramakrishnan
42cefd56eb Fix uploader API to handle write stream errors 
When the upload is aborted/abandoed because the source file is missing,
the write stream is never destroyed. This dangling write stream can
later error and cause a crash.

Instead, create the write stream on demand and nearer to pipeline() to
make sure we do 'error' handling.
 2026-01-20 22:26:08 +01:00
Girish Ramakrishnan
e106dcd76a storage: pass limits object to backend 2025-11-14 13:18:21 +01:00
Girish Ramakrishnan
07732310c1 backuptask: track copy and upload statistics 2025-10-20 14:09:12 +02:00
Girish Ramakrishnan
505c71855e retry download only 3 times 2025-10-07 20:42:04 +02:00
Girish Ramakrishnan
ac7c32ad4c backups: log encryption and format in upload 2025-10-07 17:42:51 +02:00
Girish Ramakrishnan
832eca2d9c backup size: display stats correctly 2025-10-01 17:43:27 +02:00
Girish Ramakrishnan
c5b7264f1a rename backupTargets to backupSites 2025-09-12 10:32:37 +02:00
Girish Ramakrishnan
31df40a841 storage: add copyDir 
we changed listDir in c44863a9bb to list
a directory . this broke copy for files since a '/' is added when listing
the file.
 2025-08-26 00:01:14 +02:00
Girish Ramakrishnan
19682ec21b tgz: integrity check 2025-08-15 21:23:39 +05:30
Girish Ramakrishnan
e7fb444bb3 backuptask: save backupinfo in one place 2025-08-15 14:33:31 +05:30
Girish Ramakrishnan
12e073e8cf use node: prefix for requires 
mostly because code is being autogenerated by all the AI stuff using
this prefix. it's also used in the stack trace.
 2025-08-14 12:55:35 +05:30
Girish Ramakrishnan
d7b5d1f947 rename .checksum to .backupinfo 2025-08-13 21:29:41 +05:30
Girish Ramakrishnan
0c79dcdf1b tgz: add fileCount to integrity 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
cb1bc6cfdf tgz: add fileCount to stats 2025-08-13 19:56:21 +05:30
Girish Ramakrishnan
b5c9f034ca Use hash stream instead of crypto.createHash 
crypto.createHash is a write stream but not a PassThrough stream!
 2025-08-13 18:48:22 +05:30
Girish Ramakrishnan
0911de205f stringify the Map correctly 2025-08-13 17:21:46 +05:30
Girish Ramakrishnan
59b428f24b decrease the JSON indent 2025-08-13 15:55:00 +05:30
Girish Ramakrishnan
2fabfbe8f6 rsync: add integrity information 2025-08-13 14:43:05 +05:30
Girish Ramakrishnan
dadbf1de90 tgz: save integrity information as a Map 
just for consistency because rsync will use a Map
 2025-08-13 08:39:03 +05:30
Girish Ramakrishnan
2e16dd983f backups: stash the stats to the backups table 2025-08-12 20:21:54 +05:30
Girish Ramakrishnan
47fc9561ab backups (tgz): save integrity information 
we generate a signing key pair for each target. Initially, I had this
as global. We needed a route to return the public key and putting it
under backup target seemed natural. Since we delete the backups when
we delete a target, we lose all the signing hashes. So, it's fine to lose
the key pair on target delete.
 2025-08-12 19:00:29 +05:30
Girish Ramakrishnan
5992658164 restore: fix restore 2025-08-05 14:38:12 +02:00
Girish Ramakrishnan
c935744f4c backups: root ~~canal~~ path surgery 
remove rootPath and getBackupFilePath from the backup target and
make this backend specific.
 2025-08-02 10:15:34 +02:00
Girish Ramakrishnan
3cabbc1328 backupformat: getFileExtension 2025-08-01 22:58:26 +02:00
Girish Ramakrishnan
7192439b2c no need for format specific getBackupFilePath 2025-08-01 20:49:11 +02:00
Girish Ramakrishnan
ff6cbf6628 backups: encode extension into the remotePath 2025-08-01 20:42:12 +02:00
Girish Ramakrishnan
ea419509f1 backups: add setup/teardown 
1. add setup, teardown hooks
2. move the managed mount setup and teardown to filesystem backend
3. remove this vague storage.js

we should convert storageApi into a real object, so we don't have to
keep passing apiConfig around
 2025-08-01 15:36:25 +02:00
Girish Ramakrishnan
3aafbd2ccb backups: add backup multiple targets 2025-07-25 01:30:27 +02:00
Girish Ramakrishnan
083432cbfe test: add EnsureFileSizeStream test 2024-07-18 15:39:45 +02:00
Girish Ramakrishnan
dbbce4160d tgz: underflow/overflow proxy stream 
In tar, the entry header contains the file size. If we don't provide it those many bytes, the tar will become corrupt
Linux provides no guarantee of how many bytes can be read from a file. This is the case with sqlite and log files
which are accessed by other processes when tar is in action. This class handles overflow and underflow
 2024-07-18 15:13:38 +02:00
Girish Ramakrishnan
885aac69c5 tgz: handle addEntryToPack to error 2024-07-18 14:47:31 +02:00
Girish Ramakrishnan
b3c301fc2a lint 2024-07-18 13:31:29 +02:00
Girish Ramakrishnan
8754a208b1 tgz: preserve mode 2024-07-11 18:10:40 +02:00
Girish Ramakrishnan
3a21191fba tgz: fix error handling 2024-07-10 19:10:24 +02:00
Girish Ramakrishnan
1dc6b40a68 tgz: extract using tar-stream directly 
we used have a fork of tar-fs. using tar-stream directly gives us
more control
 2024-07-08 13:06:56 +02:00
Girish Ramakrishnan
b64b513b14 Revert "use node-tar for extract" 
This reverts commit 285feb4f8b.
 2024-07-05 09:26:38 +02:00
Girish Ramakrishnan
285feb4f8b use node-tar for extract 
we will switch over our tgz module to node-tar. Main advantage is that
it is used by npm. Currently, we have our own fork to ignore stat errors
in the other module.

unfortunately, I cannot get this to work with the create logic. It doesn't
support path modification - https://github.com/isaacs/node-tar/issues/271
so, will revert this immediately and keep this for future
 2024-07-05 09:26:28 +02:00
Johannes Zellner
340ea3fe9b Fix variable usage bug for noop backup provider 2023-12-18 13:23:40 +01:00
Johannes Zellner
9d9a407c3d Noop provider does not have a rootPath set 2023-10-16 16:36:31 +02:00
Girish Ramakrishnan
aa8c23c8b3 rework backup root 
notes:
* backup root cannot come from backend. for dynamic mounts backend cannot know where it is mounted
* backupConfig is 3 parts - format / mount / password . there is also this rootPath (which should not be in db)
* password should be stored separately in settings at some point
* format has to be passed along everywhere because we allow restore from  same backupConfig but different format. we do this by saving the format in the backups table

fixes #819
 2023-08-15 22:51:45 +05:30
Girish Ramakrishnan
68b4bf1667 backupformat: print the backupFilePath 2023-08-02 09:50:34 +05:30
Girish Ramakrishnan
9a359a27f5 backups: download is now async 2023-07-25 10:33:03 +05:30
Girish Ramakrishnan
c4f4f3e914 logs: use %o to format error 
otherwise, they are printed as multi-line and this messes up tail+date formatting
 2023-04-16 10:49:59 +02:00
Girish Ramakrishnan
962d7030bb replace progress-stream with our implementation 
upstream is mostly unmaintained
 2022-11-06 10:17:14 +01:00