Commit Graph

3900 Commits

Author SHA1 Message Date
Girish Ramakrishnan f0dd90a1f5 listObjectsV2 does not work on some S3 providers
specifically, cloudscale does not support it
2017-10-05 12:07:14 -07:00
Girish Ramakrishnan ee8ee8e786 KeyCount is not set on some S3 providers 2017-10-05 11:36:54 -07:00
Girish Ramakrishnan ee1a4411f8 Do not crash if prefix is empty string
('' || undefined) will return undefined ...
2017-10-05 11:08:01 -07:00
Girish Ramakrishnan df6e6cb071 Allow s3 backend to accept self-signed certs
Fixes #316
2017-10-05 10:14:55 -07:00
Girish Ramakrishnan ca502a2d55 Display error code 2017-10-04 22:34:44 -07:00
Girish Ramakrishnan b9efb0b50b Fix callback invokation 2017-10-04 19:28:40 -07:00
Johannes Zellner 3fb5034ebd Ensure we setup the correct OAuth redirectURI if altDomain is used 2017-10-05 01:10:25 +02:00
Girish Ramakrishnan afed3f3725 Remove duplicate debug 2017-10-04 15:08:26 -07:00
Girish Ramakrishnan c3d7d867be Do not set logCallback 2017-10-04 12:32:12 -07:00
Girish Ramakrishnan 96c16cd5d2 remove debug 2017-10-04 11:54:17 -07:00
Girish Ramakrishnan af182e3df6 caas: cache the creds, otherwise we bombard the server 2017-10-04 11:49:38 -07:00
Girish Ramakrishnan d70ff7cd5b Make copy() return event emitter
This way the storage logic does not need to rely on progress
2017-10-04 11:02:50 -07:00
Johannes Zellner 38331e71e2 Ensure all S3 CopySource properties are URI encoded 2017-10-04 19:07:08 +02:00
Johannes Zellner 322a9a18d7 Use multipart copy for s3 and files larger than 5GB 2017-10-04 18:56:23 +02:00
Dennis Schwerdel e3f3241966 Added user agent to health checks 2017-10-04 13:05:00 +02:00
Girish Ramakrishnan b85bc3aa01 s3: Must encode copySource
https://github.com/aws/aws-sdk-js/issues/1302
2017-10-03 15:51:05 -07:00
Girish Ramakrishnan 01154d0ae6 s3: better error messages 2017-10-03 14:46:59 -07:00
Girish Ramakrishnan 6494050d66 Make removeDir less noisy 2017-10-03 01:22:37 -07:00
Girish Ramakrishnan 8c7223ceed Fix cleanup logic to use the app backup format
box backup and app backup can have different format
2017-10-03 00:56:34 -07:00
Girish Ramakrishnan 21afc71d89 add tests for storage backends 2017-10-02 23:08:16 -07:00
Girish Ramakrishnan 7bf70956a1 fix tests 2017-10-02 18:42:13 -07:00
Girish Ramakrishnan 9e9b8b095e Provider dhparams.pem to the mail container 2017-10-02 01:51:28 -07:00
Girish Ramakrishnan 0f543e6703 s3: add progress detail
this is a bit of a hack and we should add another way to set the progress
(maybe via backups.setProgress or via a progress callback). this is because
some methods like removeDir can be called from backuptask and from box code.
2017-10-01 18:25:51 -07:00
Girish Ramakrishnan f9973e765c Add backup cleanup eventlog 2017-10-01 10:35:50 -07:00
Girish Ramakrishnan e089851ae9 add debugs 2017-09-30 20:36:08 -07:00
Girish Ramakrishnan c524d68c2f fix crash when cleaning up snapshots 2017-09-30 20:31:41 -07:00
Girish Ramakrishnan 5cccb50a31 fix backup cleanup logic 2017-09-30 18:38:45 -07:00
Girish Ramakrishnan 3d375b687a style: Fix quoting 2017-09-30 18:26:38 -07:00
Girish Ramakrishnan a93d453963 rename flat-file to rsync
not a name I like but cannot come up with anything better

https://en.wikipedia.org/wiki/Flat_file_database

the term 'rsync format' seems to be used in a few places
2017-09-30 14:19:19 -07:00
Girish Ramakrishnan d5ba73716b add emptydirs test 2017-09-29 15:29:22 -07:00
Girish Ramakrishnan 954224dafb make syncer track directories 2017-09-29 15:29:18 -07:00
Johannes Zellner 8b341e2bf8 Only make nginx listen on ipv6 connections if it is supported by the system
Could not decide on the ejs formatting, never nice for me
2017-09-29 19:43:37 +02:00
Johannes Zellner 78fb9401ee Add config.hasIPv6() 2017-09-29 19:43:37 +02:00
Girish Ramakrishnan 4a5cbab194 Do not remove parent directory in fs.remove()
Do the pruning in the cleanup logic instead
2017-09-28 20:55:45 -07:00
Girish Ramakrishnan 19999abc50 s3: fix restore 2017-09-28 14:35:49 -07:00
Girish Ramakrishnan 5123b669d7 remove options.concurrency 2017-09-28 12:20:15 -07:00
Girish Ramakrishnan 565c8445e1 make backup progress work for per-app backups 2017-09-28 11:17:48 -07:00
Girish Ramakrishnan 404a019c56 s3: Check IsTruncated before accessing Contents 2017-09-28 10:36:56 -07:00
Girish Ramakrishnan 24dee80aa6 Make box backups always tarball based
this makes cloudron easy to restore. in the future, if required,
we can move out the mail data as a separate virtual app backup
2017-09-28 10:22:10 -07:00
Girish Ramakrishnan ce6df4bf96 Disable encryption for flat-file for now 2017-09-28 09:47:18 -07:00
Girish Ramakrishnan f8f6c7d93e Add progress detail when rotating snapshots 2017-09-28 09:29:46 -07:00
Girish Ramakrishnan bafc6dce98 s3: refactor out directory listing 2017-09-27 21:59:51 -07:00
Girish Ramakrishnan 56ee4d8e25 Remove old cache files when backup settings is changed 2017-09-27 21:04:46 -07:00
Girish Ramakrishnan eeef221b4e Fix race where pipe finishes before file is created
When there are 0 length files, this is easily reproducible.
2017-09-27 19:40:26 -07:00
Girish Ramakrishnan 4674653982 compare size and inode as well 2017-09-27 19:39:03 -07:00
Girish Ramakrishnan a34180c27b Add format to backupsdb
Call remove/removeDir based on the format
2017-09-27 18:02:30 -07:00
Girish Ramakrishnan aa8ce2c62e Use graphite 0.12.0
this fixes an issue where carbon does not startup properly
if a previous pid file was present
2017-09-27 15:35:55 -07:00
Girish Ramakrishnan b3c6b8aa15 do not spawn process just for chown 2017-09-27 15:07:19 -07:00
Girish Ramakrishnan 44a7a2579c rework backup status
* show backup progress even if not initiated by UI
* display backup progress in separate line
2017-09-27 15:07:15 -07:00
Girish Ramakrishnan 39f0e476f2 Start out empty if cache file is missing 2017-09-27 12:09:19 -07:00