Girish Ramakrishnan
f546d53ca2
gcs: add listDir
...
part of #440
2018-07-28 03:51:32 -07:00
Girish Ramakrishnan
2bcc0eef96
fs: downloadDir is unused
2018-07-27 16:56:30 -07:00
Girish Ramakrishnan
ba8a549235
fs: open the source stream using download()
2018-07-27 15:23:48 -07:00
Girish Ramakrishnan
737541f707
storage: standardize the listDir API
2018-07-27 15:23:26 -07:00
Girish Ramakrishnan
94cb222869
filesystem: Use listDir
...
We can now refactor all the iterating/fs write logic into backups.js
(required for restoring encrypted backups)
2018-07-27 14:09:52 -07:00
Girish Ramakrishnan
df98847535
Remove backupDone from storage interface
...
This is only used by caas
2018-07-27 12:08:19 -07:00
Girish Ramakrishnan
7fd1bb8597
backup: Add externalDisk option to fs backend
...
This merely confirms from the user understands that backups have to
stored on an external disk.
2018-06-07 11:14:13 -07:00
Girish Ramakrishnan
2a18070016
do-spaces: Force retry of 4xx error codes when copying
2018-03-21 15:41:21 -07:00
Girish Ramakrishnan
e0ece06b26
s3: improved copy logging
2018-03-21 14:22:41 -07:00
Girish Ramakrishnan
6adf88a6e5
Make uploads work with very slow upload speeds
...
chunk uploads get a timeout of 2mins (derived from http.timeout).
On servers like kimsufi, uploads takes forever (100 MB/sec limit).
Currently, our upload code does not dynamically adapt itself to
changing the concurrency when network is slow.
2018-03-20 19:37:45 -07:00
Girish Ramakrishnan
ce33681c37
Dump etag info
2018-03-20 18:19:14 -07:00
Girish Ramakrishnan
565eed015f
Add better backup logs
2018-03-20 16:41:45 -07:00
Girish Ramakrishnan
98e19e6df5
fix upload errors causing double callback
2018-02-27 19:16:03 -08:00
Girish Ramakrishnan
b6fb49956f
s3: better debug output when copying
2018-02-22 12:41:18 -08:00
Girish Ramakrishnan
0bba985ff1
storage: Add implementation note
2018-02-22 12:30:55 -08:00
Girish Ramakrishnan
3c8c15db01
s3: use a constant backoff since it takes forever to fail otherwise
2018-02-22 12:30:44 -08:00
Girish Ramakrishnan
c8a6294772
lint
2018-02-22 12:24:16 -08:00
Girish Ramakrishnan
cea83889ec
s3: Fix issue where it takes forever to timeout if the backend is down
2018-02-22 12:19:23 -08:00
Girish Ramakrishnan
2ecb66afd7
s3: cleanup code
2018-02-22 12:16:01 -08:00
Girish Ramakrishnan
e6c07fc148
merge the done callback into the main code
2018-02-22 11:06:28 -08:00
Girish Ramakrishnan
1f30a4f3ea
Make s3.deleteObjects return error
2018-02-22 11:05:29 -08:00
Girish Ramakrishnan
0bfdaeb2fb
rename to chunkSize
2018-02-22 11:01:04 -08:00
Girish Ramakrishnan
e022dbf8a6
Revert "merge the done callback into the main code"
...
This reverts commit c39bec8cc1
2018-02-22 10:58:56 -08:00
Girish Ramakrishnan
6075a7a890
typo
2018-02-22 10:34:48 -08:00
Girish Ramakrishnan
28b864c346
sos: Copy in 96M chunks as recommended by exoscale
2018-02-22 10:31:56 -08:00
Girish Ramakrishnan
c39bec8cc1
merge the done callback into the main code
2018-02-21 20:17:58 -08:00
Girish Ramakrishnan
727a25f491
DO Spaces: multipart copy now works
2018-02-20 14:48:03 -08:00
Girish Ramakrishnan
77fe595970
lint
2018-02-04 00:15:41 -08:00
Girish Ramakrishnan
14529d313a
sos: CopySource requires fancier encoding rules
2018-02-04 00:12:28 -08:00
Girish Ramakrishnan
f607010396
sos: remove exoscale specific hacks which are obsolete now
2018-02-02 20:13:51 -08:00
Girish Ramakrishnan
e4b12f0c4e
gcs: make testfile deletion work
...
It seems there is a race where the delete gets triggered even before the
file upload is complete. as a result, the delete succeeds but the file
is left on gcs.
2017-12-16 21:36:33 +05:30
Girish Ramakrishnan
61b56d4679
gcs: keyFilename is not used
...
also, some linter fixes
2017-12-16 21:36:29 +05:30
Girish Ramakrishnan
5c4ae6066d
gcs: lint
2017-12-15 17:34:09 +05:30
Girish Ramakrishnan
a35e048665
gcs: oldFilePath is not defined
2017-12-15 17:31:05 +05:30
Girish Ramakrishnan
48f6c39ae5
gcs: Make requires alphabetical
2017-12-15 17:28:45 +05:30
Aleksandr Bogdanov
5a2de0bcbb
Merge remote-tracking branch 'origin/master' into feature/gcs
...
# Conflicts:
# webadmin/src/views/certs.js
# webadmin/src/views/settings.js
2017-12-06 22:47:26 +01:00
Girish Ramakrishnan
568397ec19
caas: send ids in backupDone instead of filenames
2017-11-29 12:39:10 -08:00
Girish Ramakrishnan
693bc094cc
caas: make fqdn part of dns and s3 credentials
2017-11-28 22:44:40 -08:00
Girish Ramakrishnan
76312495fd
Add debug
2017-11-28 15:00:33 -08:00
Girish Ramakrishnan
f9dda85a38
Fix error code handling
2017-11-22 21:03:34 -08:00
Aleksandr Bogdanov
994f771d4d
Merge remote-tracking branch 'origin/master' into feature/gcs
2017-11-14 20:16:12 +01:00
Aleksandr Bogdanov
2cad93dfd2
Fixing UI to not require credentials be set (GCP use-case has no credentials field)
2017-10-31 12:33:15 +01:00
Aleksandr Bogdanov
9b1f8febf1
Fixing listDir to support batchSize = -1 for non-chunked listings. Also strings extrapolation fix (ES6)
2017-10-31 11:40:00 +01:00
Girish Ramakrishnan
f09c89e33f
Remove confusing batchSize logic from listDir
...
This also fixes a bug in removeDir in DO spaces
thanks to @syn for reporting
2017-10-29 19:04:10 -07:00
Aleksandr Bogdanov
51ca1c7384
Refactoring gcs to match the new storage interface
2017-10-29 11:10:50 +01:00
Aleksandr Bogdanov
051d04890b
Adding Google Cloud Storage support
2017-10-28 15:14:23 +02:00
Girish Ramakrishnan
d127b25f0f
Only set the custom https agent for HTTPS minio
...
Otherwise, we get a Cannot set property ‘agent’ of undefined error
2017-10-26 18:38:45 -07:00
Girish Ramakrishnan
8c81a97a4b
Check that the backup location has perms to create a directory
...
The backup itself runs as root and this works fine. But when rotating
the backup, the copy fails because it is unable to create a directory.
2017-10-26 11:41:34 -07:00
Girish Ramakrishnan
9273a6c726
Add option to disable hardlinks
...
We can probably remove this later based on the use
2017-10-16 15:22:40 -07:00
Girish Ramakrishnan
dd68c8f91f
Various backup fixes
2017-10-12 11:02:48 -07:00
