Commit Graph

430 Commits

Author SHA1 Message Date
Girish Ramakrishnan
1da2450b10 gcs: use copy concurrency 2020-09-28 22:03:08 -07:00
Girish Ramakrishnan
166c06c628 log the partSize 2020-09-10 00:09:54 -07:00
Girish Ramakrishnan
fc08f9823e s3: copy parts in parallel 2020-09-03 14:31:56 -07:00
Girish Ramakrishnan
e25ccc5e9a Double the timeout for upload now that chunks can have custom sizes 2020-08-20 16:50:58 -07:00
Girish Ramakrishnan
0e00492f54 backups: make part size configurable 2020-08-19 14:39:20 -07:00
Girish Ramakrishnan
df6e3eb1e6 Add deleteConcurrency setting 2020-08-11 09:14:09 -07:00
Girish Ramakrishnan
91288c96b1 s3: set queue size to 3
fixes #691
2020-08-07 00:28:00 -07:00
Johannes Zellner
493f1505f0 Check also for mountpoint on filesystem with external disk 2020-07-02 19:08:27 +02:00
Johannes Zellner
a96da20536 TODO is done for filesystem backend moutnpoint check 2020-06-26 17:57:26 +02:00
Johannes Zellner
60f1b2356a Also make nfs storage provider same as cifs and sshfs 2020-06-22 15:51:05 +02:00
Girish Ramakrishnan
c855d75f35 remove mkdirp use
node 10.12 has { recursive: true }
2020-06-11 08:27:48 -07:00
Johannes Zellner
89494ced41 Check for sshfs and cifs backup backends, if they are mounted 2020-06-08 17:46:52 +02:00
Johannes Zellner
ef764c2393 Merge sshfs.js into filesystem.js 2020-06-08 17:08:26 +02:00
Johannes Zellner
8624e2260d add storage api to make preflight checks
Currently there is only disk space checking but sshfs and cifs need
mount point checking as well
2020-06-08 16:25:05 +02:00
Johannes Zellner
3557fcd129 Add sshfs quirks to shared code in filesytstem.js 2020-06-05 13:45:25 +02:00
Johannes Zellner
330b4a613c Retrieve the backupPath from the storage provider itself 2020-06-05 13:27:18 +02:00
Johannes Zellner
7ba3412aae Add some sshfs config tests 2020-06-05 12:43:09 +02:00
Johannes Zellner
6f60495d4d Initial version of sshfs storage backend 2020-06-05 11:39:51 +02:00
Girish Ramakrishnan
4a0a934a76 start using vhost style for accessing s3 style storage
if bucket name has a '.', accept self-signed

fixes #680
2020-05-27 17:50:37 -07:00
Girish Ramakrishnan
f7c406bec9 s3: bucket name cannot contain _ or capitals or .
we can make it more elaborate, but not sure if it's needed

https://blogs.easydynamics.com/2016/10/24/aws-s3-bucket-name-validation-regex/
2020-05-27 17:01:42 -07:00
Girish Ramakrishnan
f84622efa1 fs: add create/unlink tests 2020-05-26 15:31:41 -07:00
Johannes Zellner
9bdeff0a39 Always use constants.SECRET_PLACEHOLDER 2020-05-14 23:02:02 +02:00
Girish Ramakrishnan
101dc3a93c s3: do not retry when testing config 2020-05-12 22:45:01 -07:00
Girish Ramakrishnan
4d050725b7 storage: done events must be called next tick
It seems that listDir() returns synchronously (!), not sure how.
This results in the done event getting called with an error but
the EE event handlers are not setup yet.
2020-02-11 11:48:49 -08:00
Girish Ramakrishnan
57597bd103 s3: bucket name cannot contain / 2020-02-11 11:19:47 -08:00
Girish Ramakrishnan
905bb92bad s3: ensure BoxError return 2019-12-05 21:50:44 -08:00
Girish Ramakrishnan
85e17b570b Use whilst instead of forever
this gets rid of the Error object
2019-12-04 11:17:44 -08:00
Girish Ramakrishnan
30eccfb54b Use BoxError instead of Error in all places
This moves everything other than the addon code and some 'done' logic
2019-12-04 11:02:54 -08:00
Girish Ramakrishnan
d0a3d00492 Use NOT_IMPLEMENTED error code 2019-12-04 10:22:22 -08:00
Girish Ramakrishnan
37c6a96a3a s3: if etag is not present, flag as error 2019-11-20 12:53:36 -08:00
Girish Ramakrishnan
8c9ce30d29 Move BackupsError to BoxError 2019-10-22 21:03:47 -07:00
Girish Ramakrishnan
29ab3e91b3 gcs: remove concurrency logic
this is more complicated than necessary
2019-10-18 18:54:25 -07:00
Girish Ramakrishnan
4fba216af9 scaleway: try to keep part numbers low 2019-09-30 20:42:37 -07:00
Girish Ramakrishnan
05d3f8a667 gcs: fix crash 2019-05-12 18:05:48 -07:00
Girish Ramakrishnan
800e25a7a7 Fix crash because params was undefined 2019-05-10 13:07:29 -07:00
Girish Ramakrishnan
0835a6d690 caas: remove storage backend 2019-05-08 15:33:23 -07:00
Girish Ramakrishnan
3ded411c15 caas: migrate from fqdn to boxId in backup api 2019-04-26 10:55:50 -07:00
Girish Ramakrishnan
706c970314 caas: fixup routes 2019-04-26 10:19:37 -07:00
Girish Ramakrishnan
d1582a435d readdirp args have changed 2019-04-24 10:47:04 -07:00
Girish Ramakrishnan
1119a62576 fs: emit progress messages for copy and removeDir 2019-04-03 11:54:46 -07:00
Girish Ramakrishnan
9a3603fea3 Add progress messages for multi-part copy 2019-03-26 11:58:32 -07:00
Girish Ramakrishnan
a88893b10a remove/inject backups secret fields
follow same pattern as dns code

fixes #615
2019-02-09 20:44:05 -08:00
Girish Ramakrishnan
ee65089eb7 s3: make copying and uploading significantly faster 2019-01-14 13:47:07 -08:00
Girish Ramakrishnan
fd4057df94 shell.exec -> shell.spawn 2018-11-17 19:26:19 -08:00
Girish Ramakrishnan
8efd496579 storage: add access denied function (unused) 2018-09-10 10:41:08 -07:00
Girish Ramakrishnan
d302dbc634 Fix tests 2018-08-02 14:59:50 -07:00
Girish Ramakrishnan
ef5dc7311f Do not encrypt empty path 2018-07-31 19:41:03 -07:00
Girish Ramakrishnan
a1070b7da3 storage: fix file not found message 2018-07-30 07:47:52 -07:00
Girish Ramakrishnan
99ad3e499d s3: do not rely on entry.s3 2018-07-29 09:00:57 -07:00
Girish Ramakrishnan
0b10e2b332 s3: add listDir
Fixes #440
2018-07-28 09:05:44 -07:00