jjkiers/cloudron-box
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
6,665 Commits 33 Branches 583 Tags
27e263e7fb03d69842548b8fcf2d4772e9b6017d
Commit Graph

236 Commits

Author SHA1 Message Date
Girish Ramakrishnan
6adf88a6e5 Make uploads work with very slow upload speeds 
chunk uploads get a timeout of 2mins (derived from http.timeout).
On servers like kimsufi, uploads takes forever (100 MB/sec limit).
Currently, our upload code does not dynamically adapt itself to
changing the concurrency when network is slow.
 2018-03-20 19:37:45 -07:00
Girish Ramakrishnan
7699f6721d Add hack to figure out the position in the queue 
this helps us track the progress a bit in the logs
 2018-03-20 19:37:35 -07:00
Girish Ramakrishnan
565eed015f Add better backup logs 2018-03-20 16:41:45 -07:00
Girish Ramakrishnan
98e19e6df5 fix upload errors causing double callback 2018-02-27 19:16:03 -08:00
Girish Ramakrishnan
f5d426fd69 debug out the progress message 2018-02-22 11:11:36 -08:00
Girish Ramakrishnan
e022dbf8a6 Revert "merge the done callback into the main code" 
This reverts commit c39bec8cc1.

This was committed with extra stuff by mistake
 2018-02-22 10:58:56 -08:00
Girish Ramakrishnan
c39bec8cc1 merge the done callback into the main code 2018-02-21 20:17:58 -08:00
Johannes Zellner
b6384d5025 Remove intrinsicFqdn 2018-02-08 15:07:49 +01:00
Johannes Zellner
b1939e73f4 Remove all occurances of altDomain in the code 
Tests are pending
 2018-02-08 09:00:31 +01:00
Girish Ramakrishnan
56f4cbe44a Fix double callback invocation in cleanupBackup 2018-01-15 20:08:55 -08:00
Girish Ramakrishnan
efc0a3b68d Remove usage of config.appFqdn() 2018-01-10 13:58:05 -08:00
Girish Ramakrishnan
61b56d4679 gcs: keyFilename is not used 
also, some linter fixes
 2017-12-16 21:36:29 +05:30
Aleksandr Bogdanov
5a2de0bcbb Merge remote-tracking branch 'origin/master' into feature/gcs 
# Conflicts:
#	webadmin/src/views/certs.js
#	webadmin/src/views/settings.js
 2017-12-06 22:47:26 +01:00
Girish Ramakrishnan
998f736e6f Add database.exportToFile 2017-11-24 15:29:56 -08:00
Girish Ramakrishnan
0f191324fa Add backups.restore to import from box backup 
Part of #439
 2017-11-22 23:08:59 -08:00
Girish Ramakrishnan
dc1d10837b split read/parse of fsmetadata 2017-11-22 23:07:52 -08:00
Girish Ramakrishnan
136ee363a8 Make backups.download take backupConfig 2017-11-22 10:38:04 -08:00
Johannes Zellner
e43e904622 Refactor all app.location usages to config.appFqdn(app) 2017-11-20 20:01:50 +01:00
Johannes Zellner
143ba831f4 Make appFqdn() multidomain aware 2017-11-20 19:59:40 +01:00
Girish Ramakrishnan
998bc36673 remove manifest arg to backupApp 2017-11-19 17:58:04 -08:00
Girish Ramakrishnan
360fa058ea store format information for restoring 
fixes #483
 2017-11-16 15:01:27 -08:00
Girish Ramakrishnan
92acb2954f Rename restoreConfig to manifest in backup table 
Only the manifest needs to be preserved in the backup table
 2017-11-16 11:25:40 -08:00
Aleksandr Bogdanov
994f771d4d Merge remote-tracking branch 'origin/master' into feature/gcs 2017-11-14 20:16:12 +01:00
Girish Ramakrishnan
a7296a0339 Rename filename to backupId in backup eventlog 2017-11-02 18:17:08 -07:00
Aleksandr Bogdanov
051d04890b Adding Google Cloud Storage support 2017-10-28 15:14:23 +02:00
Girish Ramakrishnan
89fc8efc67 Save as empty array if find output is empty 2017-10-16 10:54:48 -07:00
Girish Ramakrishnan
61e2878b08 save/restore exec bit in files 
this covers the case where user might stash some executable files
that are used by plugins.
 2017-10-12 16:18:11 -07:00
Girish Ramakrishnan
d97034bfb2 Follow backup format for box backups as well 2017-10-12 11:02:52 -07:00
Girish Ramakrishnan
dd68c8f91f Various backup fixes 2017-10-12 11:02:48 -07:00
Girish Ramakrishnan
28ce5f41e3 handle errors in log stream 2017-10-11 12:55:56 -07:00
Girish Ramakrishnan
db8c5a116f Typo 2017-10-11 10:30:03 -07:00
Girish Ramakrishnan
1444bb038f only upload needs to be retried 
copy/delete are already retried in the sdk code
 2017-10-11 00:08:41 -07:00
Girish Ramakrishnan
ac9e421ecf improved backup progress and logging 2017-10-10 22:49:38 -07:00
Girish Ramakrishnan
56d794745b Sprinkle retries in syncer logic 2017-10-10 14:25:03 -07:00
Girish Ramakrishnan
fd3b73bea2 typo in format name 2017-10-10 13:54:54 -07:00
Girish Ramakrishnan
78807782df Various hacks for exoscale-sos 
SOS does not like multipart uploads. They just fail randomly.

As a fix, we try to detect filesystem files and skip multipart uploads
for files < 5GB. For > 5GB, we do multipart upload anyways (mostly fails).

The box backup is switched to flat-file for exoscale for the reason
above.
 2017-10-10 11:03:20 -07:00
Girish Ramakrishnan
ee1a4411f8 Do not crash if prefix is empty string 
('' || undefined) will return undefined ...
 2017-10-05 11:08:01 -07:00
Girish Ramakrishnan
d70ff7cd5b Make copy() return event emitter 
This way the storage logic does not need to rely on progress
 2017-10-04 11:02:50 -07:00
Girish Ramakrishnan
8c7223ceed Fix cleanup logic to use the app backup format 
box backup and app backup can have different format
 2017-10-03 00:56:34 -07:00
Girish Ramakrishnan
7bf70956a1 fix tests 2017-10-02 18:42:13 -07:00
Girish Ramakrishnan
0f543e6703 s3: add progress detail 
this is a bit of a hack and we should add another way to set the progress
(maybe via backups.setProgress or via a progress callback). this is because
some methods like removeDir can be called from backuptask and from box code.
 2017-10-01 18:25:51 -07:00
Girish Ramakrishnan
f9973e765c Add backup cleanup eventlog 2017-10-01 10:35:50 -07:00
Girish Ramakrishnan
e089851ae9 add debugs 2017-09-30 20:36:08 -07:00
Girish Ramakrishnan
c524d68c2f fix crash when cleaning up snapshots 2017-09-30 20:31:41 -07:00
Girish Ramakrishnan
5cccb50a31 fix backup cleanup logic 2017-09-30 18:38:45 -07:00
Girish Ramakrishnan
a93d453963 rename flat-file to rsync 
not a name I like but cannot come up with anything better

https://en.wikipedia.org/wiki/Flat_file_database

the term 'rsync format' seems to be used in a few places
 2017-09-30 14:19:19 -07:00
Girish Ramakrishnan
d5ba73716b add emptydirs test 2017-09-29 15:29:22 -07:00
Girish Ramakrishnan
954224dafb make syncer track directories 2017-09-29 15:29:18 -07:00
Girish Ramakrishnan
4a5cbab194 Do not remove parent directory in fs.remove() 
Do the pruning in the cleanup logic instead
 2017-09-28 20:55:45 -07:00
Girish Ramakrishnan
19999abc50 s3: fix restore 2017-09-28 14:35:49 -07:00