Girish Ramakrishnan
78807782df
Various hacks for exoscale-sos
...
SOS does not like multipart uploads. They just fail randomly.
As a fix, we try to detect filesystem files and skip multipart uploads
for files < 5GB. For > 5GB, we do multipart upload anyways (mostly fails).
The box backup is switched to flat-file for exoscale for the reason
above.
2017-10-10 11:03:20 -07:00
Girish Ramakrishnan
9f97f48634
Add note on s3ForcePathStyle
2017-10-09 18:46:14 -07:00
Girish Ramakrishnan
91ec2eaaf5
sos: "/" must separate bucket and key name
2017-10-09 11:50:22 -07:00
Girish Ramakrishnan
f0dd90a1f5
listObjectsV2 does not work on some S3 providers
...
specifically, cloudscale does not support it
2017-10-05 12:07:14 -07:00
Girish Ramakrishnan
ee8ee8e786
KeyCount is not set on some S3 providers
2017-10-05 11:36:54 -07:00
Girish Ramakrishnan
df6e6cb071
Allow s3 backend to accept self-signed certs
...
Fixes #316
2017-10-05 10:14:55 -07:00
Girish Ramakrishnan
ca502a2d55
Display error code
2017-10-04 22:34:44 -07:00
Girish Ramakrishnan
b9efb0b50b
Fix callback invokation
2017-10-04 19:28:40 -07:00
Girish Ramakrishnan
c3d7d867be
Do not set logCallback
2017-10-04 12:32:12 -07:00
Girish Ramakrishnan
96c16cd5d2
remove debug
2017-10-04 11:54:17 -07:00
Girish Ramakrishnan
af182e3df6
caas: cache the creds, otherwise we bombard the server
2017-10-04 11:49:38 -07:00
Girish Ramakrishnan
d70ff7cd5b
Make copy() return event emitter
...
This way the storage logic does not need to rely on progress
2017-10-04 11:02:50 -07:00
Johannes Zellner
38331e71e2
Ensure all S3 CopySource properties are URI encoded
2017-10-04 19:07:08 +02:00
Johannes Zellner
322a9a18d7
Use multipart copy for s3 and files larger than 5GB
2017-10-04 18:56:23 +02:00
Girish Ramakrishnan
b85bc3aa01
s3: Must encode copySource
...
https://github.com/aws/aws-sdk-js/issues/1302
2017-10-03 15:51:05 -07:00
Girish Ramakrishnan
01154d0ae6
s3: better error messages
2017-10-03 14:46:59 -07:00
Girish Ramakrishnan
6494050d66
Make removeDir less noisy
2017-10-03 01:22:37 -07:00
Girish Ramakrishnan
21afc71d89
add tests for storage backends
2017-10-02 23:08:16 -07:00
Girish Ramakrishnan
0f543e6703
s3: add progress detail
...
this is a bit of a hack and we should add another way to set the progress
(maybe via backups.setProgress or via a progress callback). this is because
some methods like removeDir can be called from backuptask and from box code.
2017-10-01 18:25:51 -07:00
Girish Ramakrishnan
a93d453963
rename flat-file to rsync
...
not a name I like but cannot come up with anything better
https://en.wikipedia.org/wiki/Flat_file_database
the term 'rsync format' seems to be used in a few places
2017-09-30 14:19:19 -07:00
Girish Ramakrishnan
19999abc50
s3: fix restore
2017-09-28 14:35:49 -07:00
Girish Ramakrishnan
5123b669d7
remove options.concurrency
2017-09-28 12:20:15 -07:00
Girish Ramakrishnan
404a019c56
s3: Check IsTruncated before accessing Contents
2017-09-28 10:36:56 -07:00
Girish Ramakrishnan
bafc6dce98
s3: refactor out directory listing
2017-09-27 21:59:51 -07:00
Girish Ramakrishnan
a34180c27b
Add format to backupsdb
...
Call remove/removeDir based on the format
2017-09-27 18:02:30 -07:00
Girish Ramakrishnan
003dc0dbaf
Add todo
2017-09-27 11:50:49 -07:00
Girish Ramakrishnan
e39329218d
Make tests work
2017-09-27 11:38:43 -07:00
Girish Ramakrishnan
a9e89b57d9
merge caas storage into s3 backend
2017-09-26 12:28:33 -07:00
Girish Ramakrishnan
0a60365143
Initial version of flat-file uploader
2017-09-26 00:17:11 -07:00
Girish Ramakrishnan
b0f85678d4
Implement downloadDir for flat-file format
2017-09-23 18:07:26 -07:00
Girish Ramakrishnan
e43413e063
implement remove dir in storage backends
2017-09-23 12:34:51 -07:00
Girish Ramakrishnan
97da8717ca
Refactor backup strategy logic into backups.js
2017-09-20 14:09:55 -07:00
Girish Ramakrishnan
26f33a8e9b
Send resolved path to the storage APIs
2017-09-19 21:58:35 -07:00
Girish Ramakrishnan
4892473eff
backupIds do not have extension anymore
...
this code existed for legacy reasons
2017-09-19 12:34:09 -07:00
Girish Ramakrishnan
f5d32a9178
copyBackup -> copy
2017-09-18 14:29:15 -07:00
Girish Ramakrishnan
781f543e87
Rename API calls in the storage backend
2017-09-17 18:50:29 -07:00
Girish Ramakrishnan
b0ee116004
targz: make sourceDir a string
2017-09-17 18:50:15 -07:00
Girish Ramakrishnan
e78d976c8f
Fix backup mapping (mail dir has moved)
2017-09-13 09:51:20 -07:00
Johannes Zellner
11c2cecc9e
Ensure we only add a leading / when we have a prefix
...
Part of #343
2017-07-19 14:35:35 +02:00
Girish Ramakrishnan
e5c42f2b90
Do a multipart download for slow internet connections
...
Fixes #317
2017-04-28 17:28:40 -07:00
Girish Ramakrishnan
9635f9aa24
Use key to determine if we should encrypt or not
...
When encrypting we use the .enc extension. When not encrypting, we
use the plain .tar.gz extension.
Fixes #315
2017-04-28 14:50:20 -07:00
Girish Ramakrishnan
893f9d87bc
make s3 upload use queueSize of 1
2017-04-28 14:50:08 -07:00
Girish Ramakrishnan
bfda0d4891
drop support for old format backups
2017-04-28 14:45:44 -07:00
Girish Ramakrishnan
ca5723bbc7
more work on tests
...
disable the migrate tests for now
2017-04-24 15:45:23 -07:00
Girish Ramakrishnan
f79e1993cb
use debug() more
2017-04-23 22:10:44 -07:00
Girish Ramakrishnan
aeba8e8fd2
Use retentionSecs to cleanup backups
...
Part of #310
2017-04-23 19:34:00 -07:00
Girish Ramakrishnan
d91e8bb87b
add minio as a separate backend
...
Fixes #308
2017-04-22 13:34:43 -07:00
Johannes Zellner
a7d7935451
Remove backup download code
2017-04-22 22:17:29 +02:00
Girish Ramakrishnan
1cca0aee6e
refactor targz logic into separate file
2017-04-21 15:45:45 -07:00
Girish Ramakrishnan
be73ec4b66
Add oldFormat support to caas and s3 backends
2017-04-21 15:06:54 -07:00
