Girish Ramakrishnan
1444bb038f
only upload needs to be retried
...
copy/delete are already retried in the sdk code
2017-10-11 00:08:41 -07:00
Girish Ramakrishnan
ac9e421ecf
improved backup progress and logging
2017-10-10 22:49:38 -07:00
Girish Ramakrishnan
78807782df
Various hacks for exoscale-sos
...
SOS does not like multipart uploads. They just fail randomly.
As a fix, we try to detect filesystem files and skip multipart uploads
for files < 5GB. For > 5GB, we do multipart upload anyways (mostly fails).
The box backup is switched to flat-file for exoscale for the reason
above.
2017-10-10 11:03:20 -07:00
Girish Ramakrishnan
9f97f48634
Add note on s3ForcePathStyle
2017-10-09 18:46:14 -07:00
Girish Ramakrishnan
91ec2eaaf5
sos: "/" must separate bucket and key name
2017-10-09 11:50:22 -07:00
Girish Ramakrishnan
f0dd90a1f5
listObjectsV2 does not work on some S3 providers
...
specifically, cloudscale does not support it
2017-10-05 12:07:14 -07:00
Girish Ramakrishnan
ee8ee8e786
KeyCount is not set on some S3 providers
2017-10-05 11:36:54 -07:00
Girish Ramakrishnan
df6e6cb071
Allow s3 backend to accept self-signed certs
...
Fixes #316
2017-10-05 10:14:55 -07:00
Girish Ramakrishnan
ca502a2d55
Display error code
2017-10-04 22:34:44 -07:00
Girish Ramakrishnan
b9efb0b50b
Fix callback invokation
2017-10-04 19:28:40 -07:00
Girish Ramakrishnan
c3d7d867be
Do not set logCallback
2017-10-04 12:32:12 -07:00
Girish Ramakrishnan
96c16cd5d2
remove debug
2017-10-04 11:54:17 -07:00
Girish Ramakrishnan
af182e3df6
caas: cache the creds, otherwise we bombard the server
2017-10-04 11:49:38 -07:00
Girish Ramakrishnan
d70ff7cd5b
Make copy() return event emitter
...
This way the storage logic does not need to rely on progress
2017-10-04 11:02:50 -07:00
Johannes Zellner
38331e71e2
Ensure all S3 CopySource properties are URI encoded
2017-10-04 19:07:08 +02:00
Johannes Zellner
322a9a18d7
Use multipart copy for s3 and files larger than 5GB
2017-10-04 18:56:23 +02:00
Girish Ramakrishnan
b85bc3aa01
s3: Must encode copySource
...
https://github.com/aws/aws-sdk-js/issues/1302
2017-10-03 15:51:05 -07:00
Girish Ramakrishnan
01154d0ae6
s3: better error messages
2017-10-03 14:46:59 -07:00
Girish Ramakrishnan
6494050d66
Make removeDir less noisy
2017-10-03 01:22:37 -07:00
Girish Ramakrishnan
21afc71d89
add tests for storage backends
2017-10-02 23:08:16 -07:00
Girish Ramakrishnan
0f543e6703
s3: add progress detail
...
this is a bit of a hack and we should add another way to set the progress
(maybe via backups.setProgress or via a progress callback). this is because
some methods like removeDir can be called from backuptask and from box code.
2017-10-01 18:25:51 -07:00
Girish Ramakrishnan
5cccb50a31
fix backup cleanup logic
2017-09-30 18:38:45 -07:00
Girish Ramakrishnan
a93d453963
rename flat-file to rsync
...
not a name I like but cannot come up with anything better
https://en.wikipedia.org/wiki/Flat_file_database
the term 'rsync format' seems to be used in a few places
2017-09-30 14:19:19 -07:00
Girish Ramakrishnan
954224dafb
make syncer track directories
2017-09-29 15:29:18 -07:00
Girish Ramakrishnan
4a5cbab194
Do not remove parent directory in fs.remove()
...
Do the pruning in the cleanup logic instead
2017-09-28 20:55:45 -07:00
Girish Ramakrishnan
19999abc50
s3: fix restore
2017-09-28 14:35:49 -07:00
Girish Ramakrishnan
5123b669d7
remove options.concurrency
2017-09-28 12:20:15 -07:00
Girish Ramakrishnan
404a019c56
s3: Check IsTruncated before accessing Contents
2017-09-28 10:36:56 -07:00
Girish Ramakrishnan
bafc6dce98
s3: refactor out directory listing
2017-09-27 21:59:51 -07:00
Girish Ramakrishnan
eeef221b4e
Fix race where pipe finishes before file is created
...
When there are 0 length files, this is easily reproducible.
2017-09-27 19:40:26 -07:00
Girish Ramakrishnan
a34180c27b
Add format to backupsdb
...
Call remove/removeDir based on the format
2017-09-27 18:02:30 -07:00
Girish Ramakrishnan
b3c6b8aa15
do not spawn process just for chown
2017-09-27 15:07:19 -07:00
Girish Ramakrishnan
003dc0dbaf
Add todo
2017-09-27 11:50:49 -07:00
Girish Ramakrishnan
e39329218d
Make tests work
2017-09-27 11:38:43 -07:00
Girish Ramakrishnan
2780de631e
writable streams emit finish
2017-09-26 16:43:51 -07:00
Girish Ramakrishnan
399c756735
use exec so that filenames do not have to be escaped
2017-09-26 15:53:42 -07:00
Girish Ramakrishnan
a9e89b57d9
merge caas storage into s3 backend
2017-09-26 12:28:33 -07:00
Girish Ramakrishnan
0a60365143
Initial version of flat-file uploader
2017-09-26 00:17:11 -07:00
Girish Ramakrishnan
b0f85678d4
Implement downloadDir for flat-file format
2017-09-23 18:07:26 -07:00
Girish Ramakrishnan
e43413e063
implement remove dir in storage backends
2017-09-23 12:34:51 -07:00
Girish Ramakrishnan
97da8717ca
Refactor backup strategy logic into backups.js
2017-09-20 14:09:55 -07:00
Girish Ramakrishnan
26f33a8e9b
Send resolved path to the storage APIs
2017-09-19 21:58:35 -07:00
Girish Ramakrishnan
a3293c4c35
Fix tests
2017-09-19 12:43:13 -07:00
Girish Ramakrishnan
4892473eff
backupIds do not have extension anymore
...
this code existed for legacy reasons
2017-09-19 12:34:09 -07:00
Girish Ramakrishnan
221d5f95e1
ensure backupFolder is always set
2017-09-19 12:34:09 -07:00
Girish Ramakrishnan
84649b9471
Bring back backuptask
...
This is required for various small reasons:
* dir iteration with a way to pass messagein back to the upload() easily
* can be killed independently of box code
* allows us to run sync (blocking) commands in the upload logic
2017-09-19 12:32:38 -07:00
Girish Ramakrishnan
8ced0aa78e
copy: use hardlinks to preserve space
2017-09-18 14:29:48 -07:00
Girish Ramakrishnan
f5d32a9178
copyBackup -> copy
2017-09-18 14:29:15 -07:00
Girish Ramakrishnan
71233ecd95
Fix undefined variable
2017-09-18 11:14:04 -07:00
Girish Ramakrishnan
781f543e87
Rename API calls in the storage backend
2017-09-17 18:50:29 -07:00
