Girish Ramakrishnan
e6c07fc148
merge the done callback into the main code
2018-02-22 11:06:28 -08:00
Girish Ramakrishnan
1f30a4f3ea
Make s3.deleteObjects return error
2018-02-22 11:05:29 -08:00
Girish Ramakrishnan
0bfdaeb2fb
rename to chunkSize
2018-02-22 11:01:04 -08:00
Girish Ramakrishnan
e022dbf8a6
Revert "merge the done callback into the main code"
...
This reverts commit c39bec8cc1
2018-02-22 10:58:56 -08:00
Girish Ramakrishnan
6075a7a890
typo
2018-02-22 10:34:48 -08:00
Girish Ramakrishnan
28b864c346
sos: Copy in 96M chunks as recommended by exoscale
2018-02-22 10:31:56 -08:00
Girish Ramakrishnan
c39bec8cc1
merge the done callback into the main code
2018-02-21 20:17:58 -08:00
Girish Ramakrishnan
727a25f491
DO Spaces: multipart copy now works
2018-02-20 14:48:03 -08:00
Girish Ramakrishnan
77fe595970
lint
2018-02-04 00:15:41 -08:00
Girish Ramakrishnan
14529d313a
sos: CopySource requires fancier encoding rules
2018-02-04 00:12:28 -08:00
Girish Ramakrishnan
f607010396
sos: remove exoscale specific hacks which are obsolete now
2018-02-02 20:13:51 -08:00
Girish Ramakrishnan
e4b12f0c4e
gcs: make testfile deletion work
...
It seems there is a race where the delete gets triggered even before the
file upload is complete. as a result, the delete succeeds but the file
is left on gcs.
2017-12-16 21:36:33 +05:30
Girish Ramakrishnan
61b56d4679
gcs: keyFilename is not used
...
also, some linter fixes
2017-12-16 21:36:29 +05:30
Girish Ramakrishnan
5c4ae6066d
gcs: lint
2017-12-15 17:34:09 +05:30
Girish Ramakrishnan
a35e048665
gcs: oldFilePath is not defined
2017-12-15 17:31:05 +05:30
Girish Ramakrishnan
48f6c39ae5
gcs: Make requires alphabetical
2017-12-15 17:28:45 +05:30
Aleksandr Bogdanov
5a2de0bcbb
Merge remote-tracking branch 'origin/master' into feature/gcs
...
# Conflicts:
# webadmin/src/views/certs.js
# webadmin/src/views/settings.js
2017-12-06 22:47:26 +01:00
Girish Ramakrishnan
568397ec19
caas: send ids in backupDone instead of filenames
2017-11-29 12:39:10 -08:00
Girish Ramakrishnan
693bc094cc
caas: make fqdn part of dns and s3 credentials
2017-11-28 22:44:40 -08:00
Girish Ramakrishnan
76312495fd
Add debug
2017-11-28 15:00:33 -08:00
Girish Ramakrishnan
f9dda85a38
Fix error code handling
2017-11-22 21:03:34 -08:00
Aleksandr Bogdanov
994f771d4d
Merge remote-tracking branch 'origin/master' into feature/gcs
2017-11-14 20:16:12 +01:00
Aleksandr Bogdanov
2cad93dfd2
Fixing UI to not require credentials be set (GCP use-case has no credentials field)
2017-10-31 12:33:15 +01:00
Aleksandr Bogdanov
9b1f8febf1
Fixing listDir to support batchSize = -1 for non-chunked listings. Also strings extrapolation fix (ES6)
2017-10-31 11:40:00 +01:00
Girish Ramakrishnan
f09c89e33f
Remove confusing batchSize logic from listDir
...
This also fixes a bug in removeDir in DO spaces
thanks to @syn for reporting
2017-10-29 19:04:10 -07:00
Aleksandr Bogdanov
51ca1c7384
Refactoring gcs to match the new storage interface
2017-10-29 11:10:50 +01:00
Aleksandr Bogdanov
051d04890b
Adding Google Cloud Storage support
2017-10-28 15:14:23 +02:00
Girish Ramakrishnan
d127b25f0f
Only set the custom https agent for HTTPS minio
...
Otherwise, we get a Cannot set property ‘agent’ of undefined error
2017-10-26 18:38:45 -07:00
Girish Ramakrishnan
8c81a97a4b
Check that the backup location has perms to create a directory
...
The backup itself runs as root and this works fine. But when rotating
the backup, the copy fails because it is unable to create a directory.
2017-10-26 11:41:34 -07:00
Girish Ramakrishnan
9273a6c726
Add option to disable hardlinks
...
We can probably remove this later based on the use
2017-10-16 15:22:40 -07:00
Girish Ramakrishnan
dd68c8f91f
Various backup fixes
2017-10-12 11:02:48 -07:00
Girish Ramakrishnan
28ce5f41e3
handle errors in log stream
2017-10-11 12:55:56 -07:00
Girish Ramakrishnan
1444bb038f
only upload needs to be retried
...
copy/delete are already retried in the sdk code
2017-10-11 00:08:41 -07:00
Girish Ramakrishnan
ac9e421ecf
improved backup progress and logging
2017-10-10 22:49:38 -07:00
Girish Ramakrishnan
78807782df
Various hacks for exoscale-sos
...
SOS does not like multipart uploads. They just fail randomly.
As a fix, we try to detect filesystem files and skip multipart uploads
for files < 5GB. For > 5GB, we do multipart upload anyways (mostly fails).
The box backup is switched to flat-file for exoscale for the reason
above.
2017-10-10 11:03:20 -07:00
Girish Ramakrishnan
9f97f48634
Add note on s3ForcePathStyle
2017-10-09 18:46:14 -07:00
Girish Ramakrishnan
91ec2eaaf5
sos: "/" must separate bucket and key name
2017-10-09 11:50:22 -07:00
Girish Ramakrishnan
f0dd90a1f5
listObjectsV2 does not work on some S3 providers
...
specifically, cloudscale does not support it
2017-10-05 12:07:14 -07:00
Girish Ramakrishnan
ee8ee8e786
KeyCount is not set on some S3 providers
2017-10-05 11:36:54 -07:00
Girish Ramakrishnan
df6e6cb071
Allow s3 backend to accept self-signed certs
...
Fixes #316
2017-10-05 10:14:55 -07:00
Girish Ramakrishnan
ca502a2d55
Display error code
2017-10-04 22:34:44 -07:00
Girish Ramakrishnan
b9efb0b50b
Fix callback invokation
2017-10-04 19:28:40 -07:00
Girish Ramakrishnan
c3d7d867be
Do not set logCallback
2017-10-04 12:32:12 -07:00
Girish Ramakrishnan
96c16cd5d2
remove debug
2017-10-04 11:54:17 -07:00
Girish Ramakrishnan
af182e3df6
caas: cache the creds, otherwise we bombard the server
2017-10-04 11:49:38 -07:00
Girish Ramakrishnan
d70ff7cd5b
Make copy() return event emitter
...
This way the storage logic does not need to rely on progress
2017-10-04 11:02:50 -07:00
Johannes Zellner
38331e71e2
Ensure all S3 CopySource properties are URI encoded
2017-10-04 19:07:08 +02:00
Johannes Zellner
322a9a18d7
Use multipart copy for s3 and files larger than 5GB
2017-10-04 18:56:23 +02:00
Girish Ramakrishnan
b85bc3aa01
s3: Must encode copySource
...
https://github.com/aws/aws-sdk-js/issues/1302
2017-10-03 15:51:05 -07:00
Girish Ramakrishnan
01154d0ae6
s3: better error messages
2017-10-03 14:46:59 -07:00
