Girish Ramakrishnan
ac9e421ecf
improved backup progress and logging
2017-10-10 22:49:38 -07:00
Girish Ramakrishnan
56d794745b
Sprinkle retries in syncer logic
2017-10-10 14:25:03 -07:00
Girish Ramakrishnan
fd3b73bea2
typo in format name
2017-10-10 13:54:54 -07:00
Girish Ramakrishnan
78807782df
Various hacks for exoscale-sos
...
SOS does not like multipart uploads. They just fail randomly.
As a fix, we try to detect filesystem files and skip multipart uploads
for files < 5GB. For > 5GB, we do multipart upload anyways (mostly fails).
The box backup is switched to flat-file for exoscale for the reason
above.
2017-10-10 11:03:20 -07:00
Girish Ramakrishnan
ee1a4411f8
Do not crash if prefix is empty string
...
('' || undefined) will return undefined ...
2017-10-05 11:08:01 -07:00
Girish Ramakrishnan
d70ff7cd5b
Make copy() return event emitter
...
This way the storage logic does not need to rely on progress
2017-10-04 11:02:50 -07:00
Girish Ramakrishnan
8c7223ceed
Fix cleanup logic to use the app backup format
...
box backup and app backup can have different format
2017-10-03 00:56:34 -07:00
Girish Ramakrishnan
7bf70956a1
fix tests
2017-10-02 18:42:13 -07:00
Girish Ramakrishnan
0f543e6703
s3: add progress detail
...
this is a bit of a hack and we should add another way to set the progress
(maybe via backups.setProgress or via a progress callback). this is because
some methods like removeDir can be called from backuptask and from box code.
2017-10-01 18:25:51 -07:00
Girish Ramakrishnan
f9973e765c
Add backup cleanup eventlog
2017-10-01 10:35:50 -07:00
Girish Ramakrishnan
e089851ae9
add debugs
2017-09-30 20:36:08 -07:00
Girish Ramakrishnan
c524d68c2f
fix crash when cleaning up snapshots
2017-09-30 20:31:41 -07:00
Girish Ramakrishnan
5cccb50a31
fix backup cleanup logic
2017-09-30 18:38:45 -07:00
Girish Ramakrishnan
a93d453963
rename flat-file to rsync
...
not a name I like but cannot come up with anything better
https://en.wikipedia.org/wiki/Flat_file_database
the term 'rsync format' seems to be used in a few places
2017-09-30 14:19:19 -07:00
Girish Ramakrishnan
d5ba73716b
add emptydirs test
2017-09-29 15:29:22 -07:00
Girish Ramakrishnan
954224dafb
make syncer track directories
2017-09-29 15:29:18 -07:00
Girish Ramakrishnan
4a5cbab194
Do not remove parent directory in fs.remove()
...
Do the pruning in the cleanup logic instead
2017-09-28 20:55:45 -07:00
Girish Ramakrishnan
19999abc50
s3: fix restore
2017-09-28 14:35:49 -07:00
Girish Ramakrishnan
565c8445e1
make backup progress work for per-app backups
2017-09-28 11:17:48 -07:00
Girish Ramakrishnan
24dee80aa6
Make box backups always tarball based
...
this makes cloudron easy to restore. in the future, if required,
we can move out the mail data as a separate virtual app backup
2017-09-28 10:22:10 -07:00
Girish Ramakrishnan
f8f6c7d93e
Add progress detail when rotating snapshots
2017-09-28 09:29:46 -07:00
Girish Ramakrishnan
56ee4d8e25
Remove old cache files when backup settings is changed
2017-09-27 21:04:46 -07:00
Girish Ramakrishnan
a34180c27b
Add format to backupsdb
...
Call remove/removeDir based on the format
2017-09-27 18:02:30 -07:00
Girish Ramakrishnan
44a7a2579c
rework backup status
...
* show backup progress even if not initiated by UI
* display backup progress in separate line
2017-09-27 15:07:15 -07:00
Girish Ramakrishnan
39f0e476f2
Start out empty if cache file is missing
2017-09-27 12:09:19 -07:00
Girish Ramakrishnan
8d3fbc5432
Save backup logs and fix backup progress
2017-09-26 21:09:00 -07:00
Girish Ramakrishnan
859311f9e5
Process delete commands before add commands
...
This is required for cases where a dir becomes a file (or vice-versa)
2017-09-26 15:33:54 -07:00
Girish Ramakrishnan
a9e89b57d9
merge caas storage into s3 backend
2017-09-26 12:28:33 -07:00
Girish Ramakrishnan
4e68abe51d
Handle fs errors
2017-09-26 12:10:58 -07:00
Girish Ramakrishnan
d1efb2db56
remove bogus mkdir
2017-09-26 11:34:24 -07:00
Girish Ramakrishnan
0a60365143
Initial version of flat-file uploader
2017-09-26 00:17:11 -07:00
Girish Ramakrishnan
2f6cb3e913
set format in the backup ui
2017-09-26 00:01:36 -07:00
Girish Ramakrishnan
b0f85678d4
Implement downloadDir for flat-file format
2017-09-23 18:07:26 -07:00
Girish Ramakrishnan
e43413e063
implement remove dir in storage backends
2017-09-23 12:34:51 -07:00
Girish Ramakrishnan
e39a5c8872
preserve env in backuptask.js
2017-09-22 11:19:44 -07:00
Girish Ramakrishnan
b61b7f80b5
Add DO spaces
2017-09-21 12:25:39 -07:00
Girish Ramakrishnan
efa850614d
Add a s3-v4-compat provider
2017-09-21 12:13:45 -07:00
Girish Ramakrishnan
97da8717ca
Refactor backup strategy logic into backups.js
2017-09-20 14:09:55 -07:00
Girish Ramakrishnan
cbddb79d15
Resolve the id in rotateAppBackup
2017-09-20 09:38:55 -07:00
Girish Ramakrishnan
26f33a8e9b
Send resolved path to the storage APIs
2017-09-19 21:58:35 -07:00
Girish Ramakrishnan
952b1f6304
Make backuptask call back into backups.js
2017-09-19 20:27:49 -07:00
Girish Ramakrishnan
84649b9471
Bring back backuptask
...
This is required for various small reasons:
* dir iteration with a way to pass messagein back to the upload() easily
* can be killed independently of box code
* allows us to run sync (blocking) commands in the upload logic
2017-09-19 12:32:38 -07:00
Girish Ramakrishnan
c351660a9a
Implement backup rotation
...
Always upload to 'snapshot' dir and then rotate it. This will allow
us to keep pushing incrementally to 'snapshot' and do server side
rotations.
2017-09-18 21:17:34 -07:00
Girish Ramakrishnan
f5d32a9178
copyBackup -> copy
2017-09-18 14:29:15 -07:00
Girish Ramakrishnan
7fc45b3215
Refactor out the backup snapshot logic
2017-09-18 12:43:11 -07:00
Girish Ramakrishnan
be03dd0821
remove unused require
2017-09-18 10:38:26 -07:00
Girish Ramakrishnan
781f543e87
Rename API calls in the storage backend
2017-09-17 18:50:29 -07:00
Girish Ramakrishnan
6525a467a2
Rework backuptask into tar.js
...
This makes it easy to integrate another backup strategy
as the next step
2017-09-17 18:50:26 -07:00
Girish Ramakrishnan
6cddd61a24
Fix style
2017-09-17 18:50:23 -07:00
Girish Ramakrishnan
52095cb8ab
add debugs for timing backup and restore
2017-09-12 15:37:35 -07:00
