Add changes

Fixes #682

CHANGES

@@ -1806,3 +1806,146 @@
 * restore: carefully replace backup config
 * spam: add default corpus and global db
 
+[5.0.2]
+* Show backup disk usage in graphs
+* Add per-user app passwords
+* Make app not responding page customizable
+* Make footer customizable
+* Add UI to import backups
+* Display timestamps in browser timezone in the UI
+* Mail eventlog and usage
+* Add user roles - owner, admin, user manager and user
+* Setup logrotate configs for collectd since upstream does not set it up
+* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
+* linode: add object storage backend
+* restore: carefully replace backup config
+* spam: per mailbox bayes db and training
+
+[5.0.3]
+* Show backup disk usage in graphs
+* Add per-user app passwords
+* Make app not responding page customizable
+* Make footer customizable
+* Add UI to import backups
+* Display timestamps in browser timezone in the UI
+* Mail eventlog and usage
+* Add user roles - owner, admin, user manager and user
+* Setup logrotate configs for collectd since upstream does not set it up
+* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
+* linode: add object storage backend
+* restore: carefully replace backup config
+* spam: per mailbox bayes db and training
+
+[5.0.4]
+* Fix potential previlige escalation because of ghost file
+* linode: dns backend
+* make branding routes owner only
+* add branding API
+* Add app start/stop/restart events
+* Use the primary email for LE account
+* make mail eventlog more descriptive
+
+[5.0.5]
+* Fix bug where incoming mail from dynamic hostnames was rejected
+* Increase token expiry
+* Fix bug in tag UI where tag removal did not work
+
+[5.0.6]
+* Make mail eventlog only visible to owners
+* Make app password work with sftp
+
+[5.1.0]
+* Add turn addon
+* Fix disk usage display
+* Drop support for TLSv1 and TLSv1.1
+* Make cert validation work for ECC certs
+* Add type filter to mail eventlog
+* mail: Fix listing of mailboxes and aliases in the UI
+* branding: fix login page title
+* Only a Cloudron owner can install/update/exec apps with the docker addon
+* security: reset tokens are only valid for a day
+* mail: fix eventlog db perms
+* Fix various bugs in the disk graphs
+
+[5.1.1]
+* Add turn addon
+* Fix disk usage display
+* Drop support for TLSv1 and TLSv1.1
+* Make cert validation work for ECC certs
+* Add type filter to mail eventlog
+* mail: Fix listing of mailboxes and aliases in the UI
+* branding: fix login page title
+* Only a Cloudron owner can install/update/exec apps with the docker addon
+* security: reset tokens are only valid for a day
+* mail: fix eventlog db perms
+* Fix various bugs in the disk graphs
+* Fix collectd installation
+* graphs: sort disk contents by usage
+* backups: show apps that are not automatically backed up in backup view
+
+[5.1.2]
+* Add turn addon
+* Fix disk usage display
+* Drop support for TLSv1 and TLSv1.1
+* Make cert validation work for ECC certs
+* Add type filter to mail eventlog
+* mail: Fix listing of mailboxes and aliases in the UI
+* branding: fix login page title
+* Only a Cloudron owner can install/update/exec apps with the docker addon
+* security: reset tokens are only valid for a day
+* mail: fix eventlog db perms
+* Fix various bugs in the disk graphs
+* Fix collectd installation
+* graphs: sort disk contents by usage
+* backups: show apps that are not automatically backed up in backup view
+* turn: deny local address peers https://www.rtcsec.com/2020/04/01-slack-webrtc-turn-compromise/
+
+[5.1.3]
+* Fix crash with misconfigured reverse proxy
+* Fix issue where invitation links are not working anymore
+
+[5.1.4]
+* Add support for custom .well-known documents to be served
+* Add ECDHE-RSA-AES128-SHA256 to cipher list
+* Fix GPG signature verification
+
+[5.1.5]
+* Check for .well-known routes upstream as fallback. This broke nextcloud's caldav/carddav
+
+[5.2.0]
+* acme: request ECC certs
+* less-strict DKIM check to allow users to set a stronger DKIM key
+* Add members only flag to mailing list
+* oauth: add backward compat layer for backup and uninstall
+* fix bug in disk usage sorting
+* mail: aliases can be across domains
+* mail: allow an external MX to be set
+* Add UI to download backup config as JSON (and import it)
+* Ensure stopped apps are getting backed up
+* Add OVH Object Storage backend
+* Add per-app redis status and configuration to Services
+* spam: large emails were not scanned
+* mail relay: fix delivery event log
+* manual update check always gets the latest updates
+* graphs: fix issue where large number of apps would crash the box code (query param limit exceeded)
+* backups: fix various security issues in encypted backups (thanks @mehdi)
+* graphs: add app graphs
+* older encrypted backups cannot be used in this version
+* Add backup listing UI
+* stopping an app will stop dependent services
+* Add new wasabi s3 storage region us-east-2
+* mail: Fix bug where SRS translation was done on the main domain instead of mailing list domain
+
+[5.2.1]
+* Fix app disk graphs
+* restart apps on addon container change
+
+[5.2.2]
+* regression: import UI
+* Mbps -> MBps
+* Remove verbose logs
+* Set dmode in tar extract
+* mail: fix crash in audit logs
+* import: fix crash because encryption is unset
+* create redis with the correct label
+

README.md

@@ -48,18 +48,8 @@ the dashboard, database addons, graph container, base image etc. Cloudron also r
 on external services such as the App Store for apps to be installed. As such, don't
 clone this repo and npm install and expect something to work.
 
-## Documentation
+## Support
 
 * [Documentation](https://cloudron.io/documentation/)
-
-## Related repos
-
-The [base image repo](https://git.cloudron.io/cloudron/docker-base-image) is the parent image of all
-the containers in the Cloudron.
-
-## Community
-
-* [Chat](https://chat.cloudron.io)
 * [Forum](https://forum.cloudron.io/)
-* [Support](mailto:support@cloudron.io)
 

baseimage/initializeBaseUbuntuImage.sh

@@ -44,7 +44,6 @@ apt-get -y install \
     linux-generic \
     logrotate \
     mysql-server-5.7 \
-    nginx-full \
     openssh-server \
     pwgen \
     resolvconf \
@@ -54,6 +53,17 @@ apt-get -y install \
     unbound \
     xfsprogs
 
+if [[ "${ubuntu_version}" == "16.04" ]]; then
+    echo "==> installing nginx for xenial for TLSv3 support"
+
+    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
+    # apt install with install deps (as opposed to dpkg -i)
+    apt install -y /tmp/nginx.deb
+    rm /tmp/nginx.deb
+else
+    apt install -y nginx-full
+fi
+
 # on some providers like scaleway the sudo file is changed and we want to keep the old one
 apt-get -o Dpkg::Options::="--force-confold" install -y sudo
 
@@ -111,7 +121,7 @@ for image in ${images}; do
 done
 
 echo "==> Install collectd"
-if ! apt-get install -y collectd collectd-utils; then
+if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
     # FQDNLookup is true in default debian config. The box code has a custom collectd.conf that fixes this
     echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
     sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf

migrations/20200330233442-users-add-resetTokenCreationTime.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE users ADD COLUMN resetTokenCreationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE users DROP COLUMN resetTokenCreationTime', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200331164304-apps-alter-mailboxDomain.js

@@ -0,0 +1,28 @@
+'use strict';
+
+let async = require('async');
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps MODIFY mailboxDomain VARCHAR(128)', [], function (error) { // make it nullable
+        if (error) console.error(error);
+
+        // clear mailboxName/Domain for apps that do not use mail addons
+        db.all('SELECT * FROM apps', function (error, apps) {
+            if (error) return callback(error);
+
+            async.eachSeries(apps, function (app, iteratorDone) {
+                var manifest = JSON.parse(app.manifestJson);
+                if (manifest.addons['sendmail'] || manifest.addons['recvmail']) return iteratorDone();
+
+                db.runSql('UPDATE apps SET mailboxName=?, mailboxDomain=? WHERE id=?', [ null, null, app.id ], iteratorDone);
+            }, callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps MODIFY manifestJson VARCHAR(128) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200417235435-mailboxes-add-membersOnly.js

@@ -0,0 +1,17 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes ADD COLUMN membersOnly BOOLEAN DEFAULT 0', function (error) {
+        if (error) return callback(error);
+
+        callback();
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE users DROP COLUMN membersOnly', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+

migrations/20200420013715-mailboxes-add-aliasDomain.js

@@ -0,0 +1,28 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN aliasDomain VARCHAR(128)'),
+        function setAliasDomain(done) {
+            db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
+                async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
+                    if (!mailbox.aliasTarget) return iteratorDone();
+
+                    db.runSql('UPDATE mailboxes SET aliasDomain=? WHERE name=? AND domain=?', [ mailbox.domain, mailbox.name, mailbox.domain ], iteratorDone);
+                }, done);
+            });
+        },
+        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD CONSTRAINT mailboxes_aliasDomain_constraint FOREIGN KEY(aliasDomain) REFERENCES mail(domain)'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasTarget aliasName VARCHAR(128)')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP FOREIGN KEY mailboxes_aliasDomain_constraint'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP COLUMN aliasDomain'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasName aliasTarget VARCHAR(128)')
+    ], callback);
+};

migrations/20200427044800-apps-add-servicesConfigJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN servicesConfigJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN servicesConfigJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200430035310-apps-add-bindsJson.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN bindsJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200512172301-settings-backup-encryption.js

@@ -0,0 +1,35 @@
+'use strict';
+
+const backups = require('../src/backups.js'),
+    fs = require('fs');
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        if (backupConfig.key) {
+            backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.key);
+            backups.cleanupCacheFilesSync();
+
+            fs.writeFileSync('/home/yellowtent/platformdata/BACKUP_PASSWORD',
+                'This file contains your Cloudron backup password.\nBefore Cloudron v5.2, this was saved in the database.' +
+                'From Cloudron 5.2, this password is not required anymore. We generate strong keys based off this password and use those keys to encrypt the backups.\n' +
+                'This means that the password is only required at decryption/restore time.\n\n' +
+                'This file can be safely removed and only exists for the off-chance that you do not remember your backup password.\n\n' +
+                `Password: ${backupConfig.key}\n`,
+                'utf8');
+
+        } else {
+            backupConfig.encryption = null;
+        }
+
+        delete backupConfig.key;
+
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/20200514043113-backups-rename-version-to-packageVersion.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups CHANGE version packageVersion VARCHAR(128) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups CHANGE packageVersion version VARCHAR(128) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200514045746-backups-add-encryptionVersion.js

@@ -0,0 +1,24 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups ADD COLUMN encryptionVersion INTEGER', function (error) {
+        if (error) return callback(error);
+
+        db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+            if (error || results.length === 0) return callback(error);
+
+            var backupConfig = JSON.parse(results[0].value);
+            if (!backupConfig.encryption) return callback(null);
+
+            // mark old encrypted backups as v1
+            db.runSql('UPDATE backups SET encryptionVersion=1', callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups DROP COLUMN encryptionVersion', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/20200514231149-backups-retentionSecs-to-retentionPolicy.js

@@ -0,0 +1,18 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
+        if (error || results.length === 0) return callback(error);
+
+        var backupConfig = JSON.parse(results[0].value);
+        backupConfig.retentionPolicy = { keepWithinSecs: backupConfig.retentionSecs };
+        delete backupConfig.retentionSecs;
+
+        // mark old encrypted backups as v1
+        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};

migrations/schema.sql

@@ -28,6 +28,9 @@ CREATE TABLE IF NOT EXISTS users(
     twoFactorAuthenticationEnabled BOOLEAN DEFAULT false,
     source VARCHAR(128) DEFAULT "",
     role VARCHAR(32),
+    resetToken VARCHAR(128) DEFAULT "",
+    resetTokenCreationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    active BOOLEAN DEFAULT 1,
 
     PRIMARY KEY(id));
 
@@ -76,13 +79,14 @@ CREATE TABLE IF NOT EXISTS apps(
     reverseProxyConfigJson TEXT, // { robotsTxt, csp }
     enableBackup BOOLEAN DEFAULT 1, // misnomer: controls automatic daily backups
     enableAutomaticUpdate BOOLEAN DEFAULT 1,
-    mailboxName VARCHAR(128), // mailbox of this app. default allocated as '.app'
-    mailboxDomain VARCHAR(128) NOT NULL, // mailbox domain of this apps
+    mailboxName VARCHAR(128), // mailbox of this app
+    mailboxDomain VARCHAR(128), // mailbox domain of this apps
     label VARCHAR(128),     // display name
     tagsJson VARCHAR(2048), // array of tags
     dataDir VARCHAR(256) UNIQUE,
     taskId INTEGER, // current task
     errorJson TEXT,
+    bindsJson TEXT, // bind mounts
 
     FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
     FOREIGN KEY(taskId) REFERENCES tasks(id),
@@ -117,7 +121,8 @@ CREATE TABLE IF NOT EXISTS appEnvVars(
 CREATE TABLE IF NOT EXISTS backups(
     id VARCHAR(128) NOT NULL,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    version VARCHAR(128) NOT NULL, /* app version or box version */
+    packageVersion VARCHAR(128) NOT NULL, /* app version or box version */
+    encryptionVersion INTEGER, /* when null, unencrypted backup */
     type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
     dependsOn TEXT, /* comma separate list of objects this backup depends on */
     state VARCHAR(16) NOT NULL,
@@ -174,12 +179,15 @@ CREATE TABLE IF NOT EXISTS mailboxes(
     name VARCHAR(128) NOT NULL,
     type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
     ownerId VARCHAR(128) NOT NULL, /* user id */
-    aliasTarget VARCHAR(128), /* the target name type is an alias */
+    aliasName VARCHAR(128), /* the target name type is an alias */
+    aliasDomain VARCHAR(128), /* the target domain */
     membersJson TEXT, /* members of a group. fully qualified */
+    membersOnly BOOLEAN DEFAULT false,
     creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     domain VARCHAR(128),
 
     FOREIGN KEY(domain) REFERENCES mail(domain),
+    FOREIGN KEY(aliasDomain) REFERENCES mail(domain),
     UNIQUE (name, domain));
 
 CREATE TABLE IF NOT EXISTS subdomains(

package-lock.json

@@ -338,7 +338,7 @@
     },
     "amdefine": {
       "version": "1.0.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/amdefine/-/amdefine-1.0.1.tgz",
       "integrity": "sha1-SlKCrBZHKek2Gbz9OtFR+BfOkfU=",
       "dev": true
     },
@@ -411,7 +411,7 @@
     },
     "assert-plus": {
       "version": "1.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
       "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
     },
     "assertion-error": {
@@ -483,7 +483,7 @@
     },
     "backoff": {
       "version": "2.5.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/backoff/-/backoff-2.5.0.tgz",
       "integrity": "sha1-9hbtqdPktmuMp/ynn2lXIsX44m8=",
       "requires": {
         "precond": "0.2"
@@ -621,7 +621,7 @@
     },
     "buffer-equal-constant-time": {
       "version": "1.0.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
       "integrity": "sha1-+OcRMvf/5uAaXJaXpMbz5I1cyBk="
     },
     "buffer-fill": {
@@ -636,7 +636,7 @@
     },
     "bunyan": {
       "version": "1.8.12",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/bunyan/-/bunyan-1.8.12.tgz",
       "integrity": "sha1-8VDw9nSKvdcq6uhPBEA74u8RN5c=",
       "requires": {
         "dtrace-provider": "~0.8",
@@ -736,22 +736,22 @@
       }
     },
     "cloudron-manifestformat": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/cloudron-manifestformat/-/cloudron-manifestformat-4.0.0.tgz",
-      "integrity": "sha512-St/Quu8ofQOf0rUAMaIsOL0u0dZ46irweU8rYVMvAXU0CGwSD9KDaeLW5NjGRg3FVjNzladUDVUE/BGD4rwEvA==",
+      "version": "5.1.1",
+      "resolved": "https://registry.npmjs.org/cloudron-manifestformat/-/cloudron-manifestformat-5.1.1.tgz",
+      "integrity": "sha512-1mArahTp9qkYRQsUJfpT/x6est1qW+gKPF+HoFU0hPuOVuBdMkfu6UUmwZDYmQF4FrbQkir46GyQAJADaXBg6g==",
       "requires": {
-        "cron": "^1.7.2",
+        "cron": "^1.8.2",
         "java-packagename-regex": "^1.0.0",
-        "safetydance": "0.7.1",
-        "semver": "^6.3.0",
+        "safetydance": "1.0.0",
+        "semver": "^7.1.3",
         "tv4": "^1.3.0",
-        "validator": "^12.0.0"
+        "validator": "^12.2.0"
       },
       "dependencies": {
-        "safetydance": {
-          "version": "0.7.1",
-          "resolved": "https://registry.npmjs.org/safetydance/-/safetydance-0.7.1.tgz",
-          "integrity": "sha1-FOtNQqHKr8UUVVK2zmnJuIJN0qo="
+        "semver": {
+          "version": "7.1.3",
+          "resolved": "https://registry.npmjs.org/semver/-/semver-7.1.3.tgz",
+          "integrity": "sha512-ekM0zfiA9SCBlsKa2X1hxyxiI4L3B6EbVJkkdgQXnSEEaHlGdvyodMruTiulSRWMMB4NeIuYNMC9rTKTz97GxA=="
         },
         "validator": {
           "version": "12.2.0",
@@ -762,7 +762,7 @@
     },
     "code-point-at": {
       "version": "1.1.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz",
       "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=",
       "dev": true
     },
@@ -807,7 +807,7 @@
     },
     "concat-map": {
       "version": "0.0.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
       "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
     },
     "concat-stream": {
@@ -992,7 +992,7 @@
     },
     "core-util-is": {
       "version": "1.0.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
       "integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac="
     },
     "cron": {
@@ -1034,7 +1034,7 @@
     },
     "dashdash": {
       "version": "1.14.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/dashdash/-/dashdash-1.14.1.tgz",
       "integrity": "sha1-hTz6D3y+L+1d4gMmuN1YEDX24vA=",
       "requires": {
         "assert-plus": "^1.0.0"
@@ -1109,7 +1109,7 @@
     },
     "decamelize": {
       "version": "1.2.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/decamelize/-/decamelize-1.2.0.tgz",
       "integrity": "sha1-9lNNFRSCabIDUue+4m9QH5oZEpA="
     },
     "deep-eql": {
@@ -1376,7 +1376,7 @@
     },
     "ent": {
       "version": "2.2.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
       "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0="
     },
     "error-ex": {
@@ -1480,7 +1480,7 @@
     },
     "expect.js": {
       "version": "0.3.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/expect.js/-/expect.js-0.3.1.tgz",
       "integrity": "sha1-sKWaDS7/VDdUTr8M6qYBWEHQm1s=",
       "dev": true
     },
@@ -2267,7 +2267,7 @@
     },
     "inflight": {
       "version": "1.0.6",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
       "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
       "requires": {
         "once": "^1.3.0",
@@ -2303,7 +2303,7 @@
     },
     "is-arrayish": {
       "version": "0.2.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz",
       "integrity": "sha1-d8mYQFJ6qOyxqLppe4BkWnqSap0=",
       "dev": true
     },
@@ -2385,12 +2385,12 @@
     },
     "isarray": {
       "version": "1.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
       "integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE="
     },
     "isexe": {
       "version": "2.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
       "integrity": "sha1-6PvzdNxVb/iUehDcsFctYz8s+hA="
     },
     "isstream": {
@@ -2526,7 +2526,7 @@
     },
     "ldap-filter": {
       "version": "0.2.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/ldap-filter/-/ldap-filter-0.2.2.tgz",
       "integrity": "sha1-8rhCvguG2jNSeYUFsx68rlkNd9A=",
       "requires": {
         "assert-plus": "0.1.5"
@@ -2534,7 +2534,7 @@
       "dependencies": {
         "assert-plus": {
           "version": "0.1.5",
-          "resolved": false,
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-0.1.5.tgz",
           "integrity": "sha1-7nQAlBMALYTOxyGcasgRgS5yMWA="
         }
       }
@@ -2733,19 +2733,19 @@
     "minimatch": {
       "version": "3.0.4",
       "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz",
-      "integrity": "sha1-UWbihkV/AzBgZL5Ul+jbsMPTIIM=",
+      "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==",
       "requires": {
         "brace-expansion": "^1.1.7"
       }
     },
     "minimist": {
       "version": "0.0.8",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
       "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0="
     },
     "mkdirp": {
       "version": "0.5.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
       "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=",
       "requires": {
         "minimist": "0.0.8"
@@ -2876,9 +2876,9 @@
       }
     },
     "moment": {
-      "version": "2.24.0",
-      "resolved": "https://registry.npmjs.org/moment/-/moment-2.24.0.tgz",
-      "integrity": "sha512-bV7f+6l2QigeBBZSM/6yTNq4P2fNpSWj/0e7jQcy87A8e7o2nAfP/34/2ky5Vw4B9S446EtIhodAzkFCcR4dQg=="
+      "version": "2.25.3",
+      "resolved": "https://registry.npmjs.org/moment/-/moment-2.25.3.tgz",
+      "integrity": "sha512-PuYv0PHxZvzc15Sp8ybUCoQ+xpyPWvjOuK72a5ovzp2LI32rJXOiIfyoFoYvG3s6EwwrdkMyWuRiEHSZRLJNdg=="
     },
     "moment-timezone": {
       "version": "0.5.27",
@@ -2943,7 +2943,7 @@
     },
     "mv": {
       "version": "2.1.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/mv/-/mv-2.1.1.tgz",
       "integrity": "sha1-rmzg1vbV4KT32JN5jQPB6pVZtqI=",
       "optional": true,
       "requires": {
@@ -2954,7 +2954,7 @@
       "dependencies": {
         "glob": {
           "version": "6.0.4",
-          "resolved": false,
+          "resolved": "https://registry.npmjs.org/glob/-/glob-6.0.4.tgz",
           "integrity": "sha1-DwiGD2oVUSey+t1PnOJLGqtuTSI=",
           "optional": true,
           "requires": {
@@ -2967,13 +2967,13 @@
         },
         "ncp": {
           "version": "2.0.0",
-          "resolved": false,
+          "resolved": "https://registry.npmjs.org/ncp/-/ncp-2.0.0.tgz",
           "integrity": "sha1-GVoh1sRuNh0vsSgbo4uR6d9727M=",
           "optional": true
         },
         "rimraf": {
           "version": "2.4.5",
-          "resolved": false,
+          "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-2.4.5.tgz",
           "integrity": "sha1-7nEM5dk6j9uFb7Xqj/Di11k0sto=",
           "optional": true,
           "requires": {
@@ -3261,7 +3261,7 @@
     },
     "nopt": {
       "version": "3.0.6",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/nopt/-/nopt-3.0.6.tgz",
       "integrity": "sha1-xkZdvwirzU2zWTF/eaxopkayj/k=",
       "dev": true,
       "requires": {
@@ -3310,7 +3310,7 @@
     },
     "number-is-nan": {
       "version": "1.0.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/number-is-nan/-/number-is-nan-1.0.1.tgz",
       "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=",
       "dev": true
     },
@@ -3470,7 +3470,7 @@
     },
     "parse-json": {
       "version": "2.2.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/parse-json/-/parse-json-2.2.0.tgz",
       "integrity": "sha1-9ID0BDTvgHQfhGkJn43qGPVaTck=",
       "dev": true,
       "requires": {
@@ -3494,7 +3494,7 @@
     },
     "path-is-absolute": {
       "version": "1.0.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
       "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18="
     },
     "path-key": {
@@ -3584,7 +3584,7 @@
     },
     "precond": {
       "version": "0.2.3",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/precond/-/precond-0.2.3.tgz",
       "integrity": "sha1-qpWRvKokkj8eD0hJ0kD0fvwQdaw="
     },
     "pretty-bytes": {
@@ -3652,7 +3652,7 @@
     },
     "pseudomap": {
       "version": "1.0.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/pseudomap/-/pseudomap-1.0.2.tgz",
       "integrity": "sha1-8FKijacOYYkX7wqKw0wa5aaChrM=",
       "dev": true
     },
@@ -3920,7 +3920,7 @@
     },
     "require-directory": {
       "version": "2.1.1",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
       "integrity": "sha1-jGStX9MNqxyXbiNE/+f3kqam30I="
     },
     "require-main-filename": {
@@ -4194,7 +4194,7 @@
     },
     "set-blocking": {
       "version": "2.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz",
       "integrity": "sha1-BF+XgtARrppoA93TgrJDkrPYkPc="
     },
     "setprototypeof": {
@@ -4254,7 +4254,7 @@
     },
     "signal-exit": {
       "version": "3.0.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.2.tgz",
       "integrity": "sha1-tf3AjxKH6hF4Yo5BXiUTK3NkbG0="
     },
     "smtp-connection": {
@@ -4348,7 +4348,7 @@
     },
     "sprintf-js": {
       "version": "1.0.3",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.0.3.tgz",
       "integrity": "sha1-BOaSb2YolTVPPdAVIDYzuFcpfiw="
     },
     "sqlstring": {
@@ -4515,7 +4515,7 @@
     },
     "stubs": {
       "version": "3.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz",
       "integrity": "sha1-6NK6H6nJBXAwPAMLaQD31fiavls="
     },
     "superagent": {
@@ -4812,7 +4812,7 @@
     },
     "typedarray": {
       "version": "0.0.6",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/typedarray/-/typedarray-0.0.6.tgz",
       "integrity": "sha1-hnrHTjhkGHsdPUfZlqeOxciDB3c="
     },
     "uid-safe": {
@@ -4884,7 +4884,7 @@
     },
     "util-deprecate": {
       "version": "1.0.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
       "integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8="
     },
     "utile": {
@@ -4939,7 +4939,7 @@
     },
     "vasync": {
       "version": "1.6.4",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/vasync/-/vasync-1.6.4.tgz",
       "integrity": "sha1-3+k2Fq0OeugBszKp2Iv8XNyOHR8=",
       "requires": {
         "verror": "1.6.0"
@@ -4947,7 +4947,7 @@
       "dependencies": {
         "verror": {
           "version": "1.6.0",
-          "resolved": false,
+          "resolved": "https://registry.npmjs.org/verror/-/verror-1.6.0.tgz",
           "integrity": "sha1-fROyex+swuLakEBetepuW90lLqU=",
           "requires": {
             "extsprintf": "1.2.0"
@@ -4957,7 +4957,7 @@
     },
     "verror": {
       "version": "1.10.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/verror/-/verror-1.10.0.tgz",
       "integrity": "sha1-OhBcoXBTr1XW4nDB+CiGguGNpAA=",
       "requires": {
         "assert-plus": "^1.0.0",
@@ -4980,7 +4980,7 @@
     },
     "which-module": {
       "version": "2.0.0",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/which-module/-/which-module-2.0.0.tgz",
       "integrity": "sha1-2e8H3Od7mQK4o6j6SzHD4/fm6Ho="
     },
     "wide-align": {
@@ -5067,7 +5067,7 @@
     },
     "wrappy": {
       "version": "1.0.2",
-      "resolved": false,
+      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
       "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
     },
     "write-file-atomic": {

package.json

@@ -20,7 +20,7 @@
     "async": "^2.6.3",
     "aws-sdk": "^2.610.0",
     "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^4.0.0",
+    "cloudron-manifestformat": "^5.1.1",
     "connect": "^3.7.0",
     "connect-lastmile": "^1.2.2",
     "connect-timeout": "^1.9.0",
@@ -39,6 +39,7 @@
     "lodash": "^4.17.15",
     "lodash.chunk": "^4.2.0",
     "mime": "^2.4.4",
+    "moment": "^2.25.3",
     "moment-timezone": "^0.5.27",
     "morgan": "^1.9.1",
     "multiparty": "^4.2.1",

scripts/cloudron-setup

@@ -227,7 +227,10 @@ while true; do
     sleep 10
 done
 
-echo -e "\n\n${GREEN}Visit https://<IP> and accept the self-signed certificate to finish setup.${DONE}\n"
+if ! ip=$(curl --fail --connect-timeout 2 --max-time 2 -q https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
+    ip='<IP>'
+fi
+echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
 
 if [[ "${rebootServer}" == "true" ]]; then
     systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables

scripts/cloudron-support

@@ -13,7 +13,7 @@ HELP_MESSAGE="
 This script collects diagnostic information to help debug server related issues
 
  Options:
-   --admin-login   Login as administrator
+   --owner-login   Login as owner
    --enable-ssh    Enable SSH access for the Cloudron support team
    --help          Show this message
 "
@@ -26,7 +26,7 @@ fi
 
 enableSSH="false"
 
-args=$(getopt -o "" -l "help,enable-ssh,admin-login" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,enable-ssh,admin-login,owner-login" -n "$0" -- "$@")
 eval set -- "${args}"
 
 while true; do
@@ -34,10 +34,15 @@ while true; do
     --help) echo -e "${HELP_MESSAGE}"; exit 0;;
     --enable-ssh) enableSSH="true"; shift;;
     --admin-login)
+        # fall through
+        ;&
+    --owner-login)
         admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' LIMIT 1" 2>/dev/null)
         admin_password=$(pwgen -1s 12)
-        printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > /tmp/cloudron_ghost.json
-        echo "Login as ${admin_username} / ${admin_password} . Remove /tmp/cloudron_ghost.json when done."
+        ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
+        printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
+        chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
+        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
         exit 0
         ;;
     --) break;;

scripts/installer.sh

@@ -11,9 +11,8 @@ if [[ ${EUID} -ne 0 ]]; then
     exit 1
 fi
 
-readonly USER=yellowtent
-readonly BOX_SRC_DIR=/home/${USER}/box
-readonly BASE_DATA_DIR=/home/${USER}
+readonly user=yellowtent
+readonly box_src_dir=/home/${user}/box
 
 readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
 readonly script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -24,6 +23,8 @@ readonly ubuntu_codename=$(lsb_release -cs)
 
 readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")
 
+echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION) <=="
+
 echo "==> installer: updating docker"
 
 if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
@@ -56,6 +57,15 @@ if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
     rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
 fi
 
+readonly nginx_version=$(nginx -v)
+if [[ "${nginx_version}" != *"1.14."* && "${ubuntu_version}" == "16.04" ]]; then
+    echo "==> installer: installing nginx for xenial for TLSv3 support"
+    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.14.0-1~xenial_amd64.deb -o /tmp/nginx.deb
+    # apt install with install deps (as opposed to dpkg -i)
+    apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
+    rm /tmp/nginx.deb
+fi
+
 echo "==> installer: updating node"
 if [[ "$(node --version)" != "v10.18.1" ]]; then
     mkdir -p /usr/local/node-10.18.1
@@ -109,22 +119,22 @@ while [[ ! -f "${CLOUDRON_SYSLOG}" || "$(${CLOUDRON_SYSLOG} --version)" != ${CLO
     sleep 5
 done
 
-if ! id "${USER}" 2>/dev/null; then
-    useradd "${USER}" -m
+if ! id "${user}" 2>/dev/null; then
+    useradd "${user}" -m
 fi
 
 if [[ "${is_update}" == "yes" ]]; then
     echo "==> installer: stop cloudron.target service for update"
-    ${BOX_SRC_DIR}/setup/stop.sh
+    ${box_src_dir}/setup/stop.sh
 fi
 
 # ensure we are not inside the source directory, which we will remove now
 cd /root
 
 echo "==> installer: switching the box code"
-rm -rf "${BOX_SRC_DIR}"
-mv "${box_src_tmp_dir}" "${BOX_SRC_DIR}"
-chown -R "${USER}:${USER}" "${BOX_SRC_DIR}"
+rm -rf "${box_src_dir}"
+mv "${box_src_tmp_dir}" "${box_src_dir}"
+chown -R "${user}:${user}" "${box_src_dir}"
 
 echo "==> installer: calling box setup script"
-"${BOX_SRC_DIR}/setup/start.sh"
+"${box_src_dir}/setup/start.sh"

setup/start.sh

@@ -20,6 +20,11 @@ readonly ubuntu_version=$(lsb_release -rs)
 
 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support
 
+# this needs to match the cloudron/base:2.0.0 gid
+if ! getent group media; then
+    addgroup --gid 500 --system media
+fi
+
 echo "==> Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
 systemctl enable apparmor
@@ -56,6 +61,7 @@ mkdir -p "${BOX_DATA_DIR}/profileicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
+mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents
 
 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
@@ -170,9 +176,11 @@ readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'
 
+# set HOME explicity, because it's not set when the installer calls it. this is done because
+# paths.js uses this env var and some of the migrate code requires box code
 echo "==> Migrating data"
 cd "${BOX_SRC_DIR}"
-if ! BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
+if ! HOME=${HOME_DIR} BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
     echo "DB migration failed"
     exit 1
 fi
@@ -191,6 +199,9 @@ fi
 echo "==> Cleaning up stale redis directories"
 find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +
 
+echo "==> Cleaning up old logs"
+rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
+
 echo "==> Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown -R "${USER}" /etc/cloudron

setup/start/cloudron-firewall.sh

@@ -12,6 +12,11 @@ iptables -t filter -I CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
 # ssh is allowed alternately on port 202
 iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443,587,993,4190 -j ACCEPT
 
+# turn and stun service
+iptables -t filter -A CLOUDRON -p tcp -m multiport --dports 3478,5349 -j ACCEPT
+iptables -t filter -A CLOUDRON -p udp -m multiport --dports 3478,5349 -j ACCEPT
+iptables -t filter -A CLOUDRON -p udp -m multiport --dports 50000:51000 -j ACCEPT
+
 iptables -t filter -A CLOUDRON -p icmp --icmp-type echo-request -j ACCEPT
 iptables -t filter -A CLOUDRON -p icmp --icmp-type echo-reply -j ACCEPT
 iptables -t filter -A CLOUDRON -p udp --sport 53 -j ACCEPT

setup/start/cloudron-motd

@@ -3,10 +3,17 @@
 printf "**********************************************************************\n\n"
 
 if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
+    if [[ -f /tmp/.cloudron-motd-cache ]]; then
+        ip=$(cat /tmp/.cloudron-motd-cache)
+    elif ! ip=$(curl --fail --connect-timeout 2 --max-time 2 -q https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
+        ip='<IP>'
+    fi
+    echo "${ip}" > /tmp/.cloudron-motd-cache
+
     printf "\t\t\tWELCOME TO CLOUDRON\n"
     printf "\t\t\t-------------------\n"
 
-    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://<IP> on your browser and accept the self-signed certificate to finish setup."
+    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
     printf "Cloudron overview - https://cloudron.io/documentation/ \n"
     printf "Cloudron setup - https://cloudron.io/documentation/installation/#setup \n"
 else

setup/start/collectd/du.py

@@ -3,10 +3,12 @@ import collectd,os,subprocess,sys,re,time
 # https://www.programcreek.com/python/example/106897/collectd.register_read
 
 PATHS = [] # { name, dir, exclude }
+# there is a pattern in carbon/storage-schemas.conf which stores values every 12h for a year
 INTERVAL = 60 * 60 * 12 # twice a day. change values in docker-graphite if you change this
 
 def du(pathinfo):
-    cmd = 'timeout 1800 du -Dsb "{}"'.format(pathinfo['dir'])
+    # -B1 makes du print block sizes and not apparent sizes (to match df which also uses block sizes)
+    cmd = 'timeout 1800 du -DsB1 "{}"'.format(pathinfo['dir'])
     if pathinfo['exclude'] != '':
         cmd += ' --exclude "{}"'.format(pathinfo['exclude'])
 
@@ -26,6 +28,7 @@ def parseSize(size):
 
 def dockerSize():
     # use --format '{{json .}}' to dump the string. '{{if eq .Type "Images"}}{{.Size}}{{end}}' still creates newlines
+    # https://godoc.org/github.com/docker/go-units#HumanSize is used. so it's 1000 (KB) and not 1024 (KiB)
     cmd = 'timeout 1800 docker system df --format "{{.Size}}" | head -n1'
     try:
         size = subprocess.check_output(cmd, shell=True).strip().decode('utf-8')

setup/start/custom.yml

@@ -1,40 +0,0 @@
-# add customizations here
-# after making changes run "sudo systemctl restart box"
-
-# appstore:
-#   blacklist:
-#     - io.wekan.cloudronapp
-#     - io.cloudron.openvpn
-#   whitelist:
-#     org.wordpress.cloudronapp: {}
-#     chat.rocket.cloudronapp: {}
-#     com.nextcloud.cloudronapp: {}
-#
-# backups:
-#   configurable: true
-#
-# domains:
-#   dynamicDns: true
-#   changeDashboardDomain: true
-#
-# subscription:
-#   configurable: true
-#
-# support:
-#   email: support@cloudron.io
-#   remoteSupport: true
-#
-#   ticketFormBody: |
-#     Use this form to open support tickets. You can also write directly to [support@cloudron.io](mailto:support@cloudron.io).
-#       * [Knowledge Base & App Docs](https://cloudron.io/documentation/apps/?support_view)
-#       * [Custom App Packaging & API](https://cloudron.io/developer/packaging/?support_view)
-#       * [Forum](https://forum.cloudron.io/)
-#
-#   submitTickets: true
-#
-# alerts:
-#   email: support@cloudron.io
-#   notifyCloudronAdmins: false
-#
-# footer:
-#   body: '&copy; 2020 [Cloudron](https://cloudron.io) [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)'

setup/start/logrotate/platform

@@ -10,6 +10,7 @@
 /home/yellowtent/platformdata/logs/redis-*/*.log
 /home/yellowtent/platformdata/logs/crash/*.log
 /home/yellowtent/platformdata/logs/collectd/*.log
+/home/yellowtent/platformdata/logs/turn/*.log
 /home/yellowtent/platformdata/logs/updater/*.log {
     # only keep one rotated file, we currently do not send that over the api
     rotate 1
@@ -17,6 +18,7 @@
     missingok
     # we never compress so we can simply tail the files
     nocompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 

setup/start/systemd/box.service

@@ -13,7 +13,7 @@ Type=idle
 WorkingDirectory=/home/yellowtent/box
 Restart=always
 ; Systemd does not append logs when logging to files, we spawn a shell first and exec to replace it after setting up the pipes
-ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node --max_old_space_size=150 /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
+ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
 Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box*,connect-lastmile" "BOX_ENV=cloudron" "NODE_ENV=production"
 ; kill apptask processes as well
 KillMode=control-group

src/addons.js

@@ -7,6 +7,9 @@ exports = module.exports = {
     getServiceLogs: getServiceLogs,
     restartService: restartService,
 
+    startAppServices,
+    stopAppServices,
+
     startServices: startServices,
     updateServiceConfig: updateServiceConfig,
 
@@ -20,11 +23,7 @@ exports = module.exports = {
     getMountsSync: getMountsSync,
     getContainerNamesSync: getContainerNamesSync,
 
-    getServiceDetails: getServiceDetails,
-
-    // exported for testing
-    _setupOauth: setupOauth,
-    _teardownOauth: teardownOauth,
+    getContainerDetails: getContainerDetails,
 
     SERVICE_STATUS_STARTING: 'starting',  // container up, waiting for healthcheck
     SERVICE_STATUS_ACTIVE: 'active',
@@ -66,7 +65,14 @@ const RMADDONDIR_CMD = path.join(__dirname, 'scripts/rmaddondir.sh');
 
 // setup can be called multiple times for the same app (configure crash restart) and existing data must not be lost
 // teardown is destructive. app data stored with the addon is lost
-var KNOWN_ADDONS = {
+var ADDONS = {
+    turn: {
+        setup: setupTurn,
+        teardown: teardownTurn,
+        backup: NOOP,
+        restore: NOOP,
+        clear: NOOP
+    },
     email: {
         setup: setupEmail,
         teardown: teardownEmail,
@@ -102,13 +108,6 @@ var KNOWN_ADDONS = {
         restore: restoreMySql,
         clear: clearMySql,
     },
-    oauth: {
-        setup: setupOauth,
-        teardown: teardownOauth,
-        backup: NOOP,
-        restore: setupOauth,
-        clear: NOOP,
-    },
     postgresql: {
         setup: setupPostgreSql,
         teardown: teardownPostgreSql,
@@ -150,10 +149,23 @@ var KNOWN_ADDONS = {
         backup: NOOP,
         restore: NOOP,
         clear: NOOP,
+    },
+    oauth: { // kept for backward compatibility. keep teardown for uninstall to work
+        setup: NOOP,
+        teardown: teardownOauth,
+        backup: NOOP,
+        restore: NOOP,
+        clear: NOOP,
     }
 };
 
-const KNOWN_SERVICES = {
+// services are actual containers that are running. addons are the concepts requested by app
+const SERVICES = {
+    turn: {
+        status: statusTurn,
+        restart: restartContainer.bind(null, 'turn'),
+        defaultMemoryLimit: 256 * 1024 * 1024
+    },
     mail: {
         status: containerStatus.bind(null, 'mail', 'CLOUDRON_MAIL_TOKEN'),
         restart: mail.restartMail,
@@ -201,6 +213,16 @@ const KNOWN_SERVICES = {
     }
 };
 
+const APP_SERVICES = {
+    redis: {
+        status: (instance, done) => containerStatus(`redis-${instance}`, 'CLOUDRON_REDIS_TOKEN', done),
+        start: (instance, done) => docker.startContainer(`redis-${instance}`, done),
+        stop: (instance, done) => docker.stopContainer(`redis-${instance}`, done),
+        restart: (instance, done) => restartContainer(`redis-${instance}`, done),
+        defaultMemoryLimit: 150 * 1024 * 1024
+    }
+};
+
 function debugApp(app /*, args */) {
     assert(typeof app === 'object');
 
@@ -237,6 +259,7 @@ function rebuildService(serviceName, callback) {
 
     // this attempts to recreate the service docker container if they don't exist but platform infra version is unchanged
     // passing an infra version of 'none' will not attempt to purge existing data, not sure if this is good or bad
+    if (serviceName === 'turn') return startTurn({ version: 'none' }, callback);
     if (serviceName === 'mongodb') return startMongodb({ version: 'none' }, callback);
     if (serviceName === 'postgresql') return startPostgresql({ version: 'none' }, callback);
     if (serviceName === 'mysql') return startMysql({ version: 'none' }, callback);
@@ -247,25 +270,22 @@ function rebuildService(serviceName, callback) {
     callback();
 }
 
-function restartContainer(serviceName, callback) {
-    assert.strictEqual(typeof serviceName, 'string');
+function restartContainer(name, callback) {
+    assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    docker.stopContainer(serviceName, function (error) {
+    docker.restartContainer(name, function (error) {
+        if (error && error.reason === BoxError.NOT_FOUND) {
+            callback(null); // callback early since rebuilding takes long
+            return rebuildService(name, function (error) { if (error) console.error(`Unable to rebuild service ${name}`, error); });
+        }
         if (error) return callback(error);
 
-        docker.startContainer(serviceName, function (error) {
-            if (error && error.reason === BoxError.NOT_FOUND) {
-                callback(null); // callback early since rebuilding takes long
-                return rebuildService(serviceName, function (error) { if (error) console.error(`Unable to rebuild service ${serviceName}`, error); });
-            }
-
-            callback(error);
-        });
+        callback(error);
     });
 }
 
-function getServiceDetails(containerName, tokenEnvName, callback) {
+function getContainerDetails(containerName, tokenEnvName, callback) {
     assert.strictEqual(typeof containerName, 'string');
     assert.strictEqual(typeof tokenEnvName, 'string');
     assert.strictEqual(typeof callback, 'function');
@@ -288,20 +308,20 @@ function getServiceDetails(containerName, tokenEnvName, callback) {
     });
 }
 
-function containerStatus(addonName, addonTokenName, callback) {
-    assert.strictEqual(typeof addonName, 'string');
-    assert.strictEqual(typeof addonTokenName, 'string');
+function containerStatus(containerName, tokenEnvName, callback) {
+    assert.strictEqual(typeof containerName, 'string');
+    assert.strictEqual(typeof tokenEnvName, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    getServiceDetails(addonName, addonTokenName, function (error, addonDetails) {
-        if (error && error.reason === BoxError.NOT_FOUND) return callback(null, { status: exports.SERVICE_STATUS_STOPPED });
+    getContainerDetails(containerName, tokenEnvName, function (error, addonDetails) {
+        if (error && (error.reason === BoxError.NOT_FOUND || error.reason === BoxError.INACTIVE)) return callback(null, { status: exports.SERVICE_STATUS_STOPPED });
         if (error) return callback(error);
 
         request.get(`https://${addonDetails.ip}:3000/healthcheck?access_token=${addonDetails.token}`, { json: true, rejectUnauthorized: false }, function (error, response) {
-            if (error) return callback(null, { status: exports.SERVICE_STATUS_STARTING, error: `Error waiting for ${addonName}: ${error.message}` });
-            if (response.statusCode !== 200 || !response.body.status) return callback(null, { status: exports.SERVICE_STATUS_STARTING, error: `Error waiting for ${addonName}. Status code: ${response.statusCode} message: ${response.body.message}` });
+            if (error) return callback(null, { status: exports.SERVICE_STATUS_STARTING, error: `Error waiting for ${containerName}: ${error.message}` });
+            if (response.statusCode !== 200 || !response.body.status) return callback(null, { status: exports.SERVICE_STATUS_STARTING, error: `Error waiting for ${containerName}. Status code: ${response.statusCode} message: ${response.body.message}` });
 
-            docker.memoryUsage(addonName, function (error, result) {
+            docker.memoryUsage(containerName, function (error, result) {
                 if (error) return callback(error);
 
                 var tmp = {
@@ -319,19 +339,59 @@ function containerStatus(addonName, addonTokenName, callback) {
 function getServices(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    let services = Object.keys(KNOWN_SERVICES);
+    let services = Object.keys(SERVICES);
 
-    callback(null, services);
+    appdb.getAll(function (error, apps) {
+        if (error) return callback(error);
+
+        for (let app of apps) {
+            if (app.manifest.addons && app.manifest.addons['redis']) services.push(`redis:${app.id}`);
+        }
+
+        callback(null, services);
+    });
 }
 
-function getService(serviceName, callback) {
-    assert.strictEqual(typeof serviceName, 'string');
+function getServicesConfig(id, callback) {
+    assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!KNOWN_SERVICES[serviceName]) return callback(new BoxError(BoxError.NOT_FOUND));
+    const [name, instance ] = id.split(':');
+    if (!instance) {
+        settings.getPlatformConfig(function (error, platformConfig) {
+            if (error) return callback(error);
+
+            callback(null, SERVICES[name], platformConfig);
+        });
+
+        return;
+    }
+
+    appdb.get(instance, function (error, app) {
+        if (error) return callback(error);
+
+        callback(null, APP_SERVICES[name], app.servicesConfig);
+    });
+}
+
+function getService(id, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const [name, instance ] = id.split(':');
+    let containerStatusFunc;
+
+    if (instance) {
+        if (!APP_SERVICES[name]) return callback(new BoxError(BoxError.NOT_FOUND));
+        containerStatusFunc = APP_SERVICES[name].status.bind(null, instance);
+    } else if (SERVICES[name]) {
+        containerStatusFunc = SERVICES[name].status;
+    } else {
+        return callback(new BoxError(BoxError.NOT_FOUND));
+    }
 
     var tmp = {
-        name: serviceName,
+        name: name,
         status: null,
         memoryUsed: 0,
         memoryPercent: 0,
@@ -343,60 +403,76 @@ function getService(serviceName, callback) {
         }
     };
 
-    settings.getPlatformConfig(function (error, platformConfig) {
+    containerStatusFunc(function (error, result) {
         if (error) return callback(error);
 
-        if (platformConfig[serviceName] && platformConfig[serviceName].memory && platformConfig[serviceName].memorySwap) {
-            tmp.config.memory = platformConfig[serviceName].memory;
-            tmp.config.memorySwap = platformConfig[serviceName].memorySwap;
-        } else if (KNOWN_SERVICES[serviceName].defaultMemoryLimit) {
-            tmp.config.memory = KNOWN_SERVICES[serviceName].defaultMemoryLimit;
-            tmp.config.memorySwap = tmp.config.memory * 2;
-        }
+        tmp.status = result.status;
+        tmp.memoryUsed = result.memoryUsed;
+        tmp.memoryPercent = result.memoryPercent;
+        tmp.error = result.error || null;
 
-        KNOWN_SERVICES[serviceName].status(function (error, result) {
+        getServicesConfig(id, function (error, service, servicesConfig) {
             if (error) return callback(error);
 
-            tmp.status = result.status;
-            tmp.memoryUsed = result.memoryUsed;
-            tmp.memoryPercent = result.memoryPercent;
-            tmp.error = result.error || null;
+            const serviceConfig = servicesConfig[name];
+
+            if (serviceConfig && serviceConfig.memory && serviceConfig.memorySwap) {
+                tmp.config.memory = serviceConfig.memory;
+                tmp.config.memorySwap = serviceConfig.memorySwap;
+            } else if (service.defaultMemoryLimit) {
+                tmp.config.memory = service.defaultMemoryLimit;
+                tmp.config.memorySwap = tmp.config.memory * 2;
+            }
 
             callback(null, tmp);
         });
     });
 }
 
-function configureService(serviceName, data, callback) {
-    assert.strictEqual(typeof serviceName, 'string');
+function configureService(id, data, callback) {
+    assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof data, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!KNOWN_SERVICES[serviceName]) return callback(new BoxError(BoxError.NOT_FOUND));
+    const [name, instance ] = id.split(':');
 
-    settings.getPlatformConfig(function (error, platformConfig) {
+    if (instance) {
+        if (!APP_SERVICES[name]) return callback(new BoxError(BoxError.NOT_FOUND));
+    } else if (!SERVICES[name]) {
+        return callback(new BoxError(BoxError.NOT_FOUND));
+    }
+
+    getServicesConfig(id, function (error, service, servicesConfig) {
         if (error) return callback(error);
 
-        if (!platformConfig[serviceName]) platformConfig[serviceName] = {};
+        if (!servicesConfig[name]) servicesConfig[name] = {};
 
         // if not specified we clear the entry and use defaults
         if (!data.memory || !data.memorySwap) {
-            delete platformConfig[serviceName];
+            delete servicesConfig[name];
         } else {
-            platformConfig[serviceName].memory = data.memory;
-            platformConfig[serviceName].memorySwap = data.memorySwap;
+            servicesConfig[name].memory = data.memory;
+            servicesConfig[name].memorySwap = data.memorySwap;
         }
 
-        settings.setPlatformConfig(platformConfig, function (error) {
-            if (error) return callback(error);
+        if (instance) {
+            appdb.update(instance, { servicesConfig }, function (error) {
+                if (error) return callback(error);
 
-            callback(null);
-        });
+                updateAppServiceConfig(name, instance, servicesConfig, callback);
+            });
+        } else {
+            settings.setPlatformConfig(servicesConfig, function (error) {
+                if (error) return callback(error);
+
+                callback(null);
+            });
+        }
     });
 }
 
-function getServiceLogs(serviceName, options, callback) {
-    assert.strictEqual(typeof serviceName, 'string');
+function getServiceLogs(id, options, callback) {
+    assert.strictEqual(typeof id, 'string');
     assert(options && typeof options === 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -404,9 +480,15 @@ function getServiceLogs(serviceName, options, callback) {
     assert.strictEqual(typeof options.format, 'string');
     assert.strictEqual(typeof options.follow, 'boolean');
 
-    if (!KNOWN_SERVICES[serviceName]) return callback(new BoxError(BoxError.NOT_FOUND));
+    const [name, instance ] = id.split(':');
 
-    debug(`Getting logs for ${serviceName}`);
+    if (instance) {
+        if (!APP_SERVICES[name]) return callback(new BoxError(BoxError.NOT_FOUND));
+    } else if (!SERVICES[name]) {
+        return callback(new BoxError(BoxError.NOT_FOUND));
+    }
+
+    debug(`Getting logs for ${name}`);
 
     var lines = options.lines,
         format = options.format || 'json',
@@ -415,11 +497,11 @@ function getServiceLogs(serviceName, options, callback) {
     let cmd, args = [];
 
     // docker and unbound use journald
-    if (serviceName === 'docker' || serviceName === 'unbound') {
+    if (name === 'docker' || name === 'unbound') {
         cmd = 'journalctl';
 
         args.push('--lines=' + (lines === -1 ? 'all' : lines));
-        args.push(`--unit=${serviceName}`);
+        args.push(`--unit=${name}`);
         args.push('--no-pager');
         args.push('--output=short-iso');
 
@@ -429,7 +511,8 @@ function getServiceLogs(serviceName, options, callback) {
 
         args.push('--lines=' + (lines === -1 ? '+1' : lines));
         if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
-        args.push(path.join(paths.LOG_DIR, serviceName, 'app.log'));
+        const containerName = APP_SERVICES[name] ? `${name}-${instance}` : name;
+        args.push(path.join(paths.LOG_DIR, containerName, 'app.log'));
     }
 
     var cp = spawn(cmd, args);
@@ -448,7 +531,7 @@ function getServiceLogs(serviceName, options, callback) {
         return JSON.stringify({
             realtimeTimestamp: timestamp * 1000,
             message: message,
-            source: serviceName
+            source: name
         }) + '\n';
     });
 
@@ -459,23 +542,67 @@ function getServiceLogs(serviceName, options, callback) {
     callback(null, transformStream);
 }
 
-function restartService(serviceName, callback) {
-    assert.strictEqual(typeof serviceName, 'string');
+function restartService(id, callback) {
+    assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!KNOWN_SERVICES[serviceName]) return callback(new BoxError(BoxError.NOT_FOUND));
+    const [name, instance ] = id.split(':');
 
-    KNOWN_SERVICES[serviceName].restart(callback);
+    if (instance) {
+        if (!APP_SERVICES[name]) return callback(new BoxError(BoxError.NOT_FOUND));
+
+        APP_SERVICES[name].restart(instance, callback);
+    } else if (SERVICES[name]) {
+        SERVICES[name].restart(callback);
+    } else {
+        return callback(new BoxError(BoxError.NOT_FOUND));
+    }
 }
 
-function waitForService(containerName, tokenEnvName, callback) {
+// in the future, we can refcount and lazy start global services
+function startAppServices(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const instance = app.id;
+    async.eachSeries(Object.keys(app.manifest.addons || {}), function (addon, iteratorDone) {
+        if (!(addon in APP_SERVICES)) return iteratorDone();
+
+        APP_SERVICES[addon].start(instance, function (error) { // assume addons name is service name
+            // error ignored because we don't want "start app" to error. use can fix it from Services
+            if (error) debug(`startAppServices: ${addon}:${instance}`, error);
+
+            iteratorDone();
+        });
+    }, callback);
+}
+
+// in the future, we can refcount and stop global services as well
+function stopAppServices(app, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const instance = app.id;
+    async.eachSeries(Object.keys(app.manifest.addons || {}), function (addon, iteratorDone) {
+        if (!(addon in APP_SERVICES)) return iteratorDone();
+
+        APP_SERVICES[addon].stop(instance, function (error) { // assume addons name is service name
+            // error ignored because we don't want "start app" to error. use can fix it from Services
+            if (error) debug(`stopAppServices: ${addon}:${instance}`, error);
+
+            iteratorDone();
+        });
+    }, callback);
+}
+
+function waitForContainer(containerName, tokenEnvName, callback) {
     assert.strictEqual(typeof containerName, 'string');
     assert.strictEqual(typeof tokenEnvName, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     debug(`Waiting for ${containerName}`);
 
-    getServiceDetails(containerName, tokenEnvName, function (error, result) {
+    getContainerDetails(containerName, tokenEnvName, function (error, result) {
         if (error) return callback(error);
 
         async.retry({ times: 10, interval: 15000 }, function (retryCallback) {
@@ -499,11 +626,11 @@ function setupAddons(app, addons, callback) {
     debugApp(app, 'setupAddons: Setting up %j', Object.keys(addons));
 
     async.eachSeries(Object.keys(addons), function iterator(addon, iteratorCallback) {
-        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+        if (!(addon in ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
         debugApp(app, 'Setting up addon %s with options %j', addon, addons[addon]);
 
-        KNOWN_ADDONS[addon].setup(app, addons[addon], iteratorCallback);
+        ADDONS[addon].setup(app, addons[addon], iteratorCallback);
     }, callback);
 }
 
@@ -517,11 +644,11 @@ function teardownAddons(app, addons, callback) {
     debugApp(app, 'teardownAddons: Tearing down %j', Object.keys(addons));
 
     async.eachSeries(Object.keys(addons), function iterator(addon, iteratorCallback) {
-        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+        if (!(addon in ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
         debugApp(app, 'Tearing down addon %s with options %j', addon, addons[addon]);
 
-        KNOWN_ADDONS[addon].teardown(app, addons[addon], iteratorCallback);
+        ADDONS[addon].teardown(app, addons[addon], iteratorCallback);
     }, callback);
 }
 
@@ -537,9 +664,9 @@ function backupAddons(app, addons, callback) {
     debugApp(app, 'backupAddons: Backing up %j', Object.keys(addons));
 
     async.eachSeries(Object.keys(addons), function iterator (addon, iteratorCallback) {
-        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+        if (!(addon in ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
-        KNOWN_ADDONS[addon].backup(app, addons[addon], iteratorCallback);
+        ADDONS[addon].backup(app, addons[addon], iteratorCallback);
     }, callback);
 }
 
@@ -555,9 +682,9 @@ function clearAddons(app, addons, callback) {
     debugApp(app, 'clearAddons: clearing %j', Object.keys(addons));
 
     async.eachSeries(Object.keys(addons), function iterator (addon, iteratorCallback) {
-        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+        if (!(addon in ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
-        KNOWN_ADDONS[addon].clear(app, addons[addon], iteratorCallback);
+        ADDONS[addon].clear(app, addons[addon], iteratorCallback);
     }, callback);
 }
 
@@ -573,9 +700,9 @@ function restoreAddons(app, addons, callback) {
     debugApp(app, 'restoreAddons: restoring %j', Object.keys(addons));
 
     async.eachSeries(Object.keys(addons), function iterator (addon, iteratorCallback) {
-        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+        if (!(addon in ADDONS)) return iteratorCallback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
-        KNOWN_ADDONS[addon].restore(app, addons[addon], iteratorCallback);
+        ADDONS[addon].restore(app, addons[addon], iteratorCallback);
     }, callback);
 }
 
@@ -584,12 +711,12 @@ function importAppDatabase(app, addon, callback) {
     assert.strictEqual(typeof addon, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    if (!(addon in KNOWN_ADDONS)) return callback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
+    if (!(addon in ADDONS)) return callback(new BoxError(BoxError.NOT_FOUND, `No such addon: ${addon}`));
 
     async.series([
-        KNOWN_ADDONS[addon].setup.bind(null, app, app.manifest.addons[addon]),
-        KNOWN_ADDONS[addon].clear.bind(null, app, app.manifest.addons[addon]), // clear in case we crashed in a restore
-        KNOWN_ADDONS[addon].restore.bind(null, app, app.manifest.addons[addon])
+        ADDONS[addon].setup.bind(null, app, app.manifest.addons[addon]),
+        ADDONS[addon].clear.bind(null, app, app.manifest.addons[addon]), // clear in case we crashed in a restore
+        ADDONS[addon].restore.bind(null, app, app.manifest.addons[addon])
     ], callback);
 }
 
@@ -631,7 +758,7 @@ function updateServiceConfig(platformConfig, callback) {
             memory = containerConfig.memory;
             memorySwap = containerConfig.memorySwap;
         } else {
-            memory = KNOWN_SERVICES[serviceName].defaultMemoryLimit;
+            memory = SERVICES[serviceName].defaultMemoryLimit;
             memorySwap = memory * 2;
         }
 
@@ -640,6 +767,28 @@ function updateServiceConfig(platformConfig, callback) {
     }, callback);
 }
 
+function updateAppServiceConfig(name, instance, servicesConfig, callback) {
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof instance, 'string');
+    assert.strictEqual(typeof servicesConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`updateAppServiceConfig: ${name}-${instance} ${JSON.stringify(servicesConfig)}`);
+
+    const serviceConfig = servicesConfig[name];
+    let memory, memorySwap;
+    if (serviceConfig && serviceConfig.memory && serviceConfig.memorySwap) {
+        memory = serviceConfig.memory;
+        memorySwap = serviceConfig.memorySwap;
+    } else {
+        memory = APP_SERVICES[name].defaultMemoryLimit;
+        memorySwap = memory * 2;
+    }
+
+    const args = `update --memory ${memory} --memory-swap ${memorySwap} ${name}-${instance}`.split(' ');
+    shell.spawn(`updateAppServiceConfig${name}`, '/usr/bin/docker', args, { }, callback);
+}
+
 function startServices(existingInfra, callback) {
     assert.strictEqual(typeof existingInfra, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -650,6 +799,7 @@ function startServices(existingInfra, callback) {
     if (existingInfra.version !== infra.version) {
         debug(`startServices: ${existingInfra.version} -> ${infra.version}. starting all services`);
         startFuncs.push(
+            startTurn.bind(null, existingInfra),
             startMysql.bind(null, existingInfra),
             startPostgresql.bind(null, existingInfra),
             startMongodb.bind(null, existingInfra),
@@ -658,6 +808,7 @@ function startServices(existingInfra, callback) {
     } else {
         assert.strictEqual(typeof existingInfra.images, 'object');
 
+        if (!existingInfra.images.turn || infra.images.turn.tag !== existingInfra.images.turn.tag) startFuncs.push(startTurn.bind(null, existingInfra));
         if (infra.images.mysql.tag !== existingInfra.images.mysql.tag) startFuncs.push(startMysql.bind(null, existingInfra));
         if (infra.images.postgresql.tag !== existingInfra.images.postgresql.tag) startFuncs.push(startPostgresql.bind(null, existingInfra));
         if (infra.images.mongodb.tag !== existingInfra.images.mongodb.tag) startFuncs.push(startMongodb.bind(null, existingInfra));
@@ -677,7 +828,7 @@ function getEnvironment(app, callback) {
     appdb.getAddonConfigByAppId(app.id, function (error, result) {
         if (error) return callback(error);
 
-        if (app.manifest.addons['docker']) result.push({ name: 'DOCKER_HOST', value: `tcp://172.18.0.1:${constants.DOCKER_PROXY_PORT}` });
+        if (app.manifest.addons['docker']) result.push({ name: 'CLOUDRON_DOCKER_HOST', value: `tcp://172.18.0.1:${constants.DOCKER_PROXY_PORT}` });
 
         return callback(null, result.map(function (e) { return e.name + '=' + e.value; }));
     });
@@ -740,8 +891,8 @@ function setupLocalStorage(app, options, callback) {
 
     // reomve any existing volume in case it's bound with an old dataDir
     async.series([
-        docker.removeVolume.bind(null, app, `${app.id}-localstorage`),
-        docker.createVolume.bind(null, app, `${app.id}-localstorage`, volumeDataDir)
+        docker.removeVolume.bind(null, `${app.id}-localstorage`),
+        docker.createVolume.bind(null, `${app.id}-localstorage`, volumeDataDir, { fqdn: app.fqdn, appId: app.id })
     ], callback);
 }
 
@@ -752,7 +903,7 @@ function clearLocalStorage(app, options, callback) {
 
     debugApp(app, 'clearLocalStorage');
 
-    docker.clearVolume(app, `${app.id}-localstorage`, { removeDirectory: false }, callback);
+    docker.clearVolume(`${app.id}-localstorage`, { removeDirectory: false }, callback);
 }
 
 function teardownLocalStorage(app, options, callback) {
@@ -763,35 +914,42 @@ function teardownLocalStorage(app, options, callback) {
     debugApp(app, 'teardownLocalStorage');
 
     async.series([
-        docker.clearVolume.bind(null, app, `${app.id}-localstorage`, { removeDirectory: true }),
-        docker.removeVolume.bind(null, app, `${app.id}-localstorage`)
+        docker.clearVolume.bind(null, `${app.id}-localstorage`, { removeDirectory: true }),
+        docker.removeVolume.bind(null, `${app.id}-localstorage`)
     ], callback);
 }
 
-function setupOauth(app, options, callback) {
+function setupTurn(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debugApp(app, 'setupOauth');
+    var turnSecret = safe.fs.readFileSync(paths.ADDON_TURN_SECRET_FILE, 'utf8');
+    if (!turnSecret) console.error('No turn secret set. Will leave emtpy, but this is a problem!');
 
-    if (!app.sso) return callback(null);
+    const env = [
+        { name: 'CLOUDRON_STUN_SERVER', value: settings.adminFqdn() },
+        { name: 'CLOUDRON_STUN_PORT', value: '3478' },
+        { name: 'CLOUDRON_STUN_TLS_PORT', value: '5349' },
+        { name: 'CLOUDRON_TURN_SERVER', value: settings.adminFqdn() },
+        { name: 'CLOUDRON_TURN_PORT', value: '3478' },
+        { name: 'CLOUDRON_TURN_TLS_PORT', value: '5349' },
+        { name: 'CLOUDRON_TURN_SECRET', value: turnSecret }
+    ];
 
-    const env = [];
+    debugApp(app, 'Setting up TURN');
 
-    debugApp(app, 'Setting oauth addon config to %j', env);
-
-    appdb.setAddonConfig(app.id, 'oauth', env, callback);
+    appdb.setAddonConfig(app.id, 'turn', env, callback);
 }
 
-function teardownOauth(app, options, callback) {
+function teardownTurn(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debugApp(app, 'teardownOauth');
+    debugApp(app, 'Tearing down TURN');
 
-    appdb.unsetAddonConfig(app.id, 'oauth', callback);
+    appdb.unsetAddonConfig(app.id, 'turn', callback);
 }
 
 function setupEmail(app, options, callback) {
@@ -993,7 +1151,7 @@ function startMysql(existingInfra, callback) {
         shell.exec('startMysql', cmd, function (error) {
             if (error) return callback(error);
 
-            waitForService('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error) {
+            waitForContainer('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error) {
                 if (error) return callback(error);
                 if (!upgrading) return callback(null);
 
@@ -1022,7 +1180,7 @@ function setupMySql(app, options, callback) {
             password: error ? hat(4 * 48) : existingPassword // see box#362 for password length
         };
 
-        getServiceDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
+        getContainerDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
             if (error) return callback(error);
 
             request.post(`https://${result.ip}:3000/` + (options.multipleDatabases ? 'prefixes' : 'databases') + `?access_token=${result.token}`, { rejectUnauthorized: false, json: data }, function (error, response) {
@@ -1061,7 +1219,7 @@ function clearMySql(app, options, callback) {
 
     const database = mysqlDatabaseName(app.id);
 
-    getServiceDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
+    getContainerDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.post(`https://${result.ip}:3000/` + (options.multipleDatabases ? 'prefixes' : 'databases') + `/${database}/clear?access_token=${result.token}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1081,7 +1239,7 @@ function teardownMySql(app, options, callback) {
     const database = mysqlDatabaseName(app.id);
     const username = database;
 
-    getServiceDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
+    getContainerDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.delete(`https://${result.ip}:3000/` + (options.multipleDatabases ? 'prefixes' : 'databases') + `/${database}?access_token=${result.token}&username=${username}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1129,7 +1287,7 @@ function backupMySql(app, options, callback) {
 
     debugApp(app, 'Backing up mysql');
 
-    getServiceDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
+    getContainerDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         const url = `https://${result.ip}:3000/` + (options.multipleDatabases ? 'prefixes' : 'databases') + `/${database}/backup?access_token=${result.token}`;
@@ -1148,7 +1306,7 @@ function restoreMySql(app, options, callback) {
 
     callback = once(callback); // protect from multiple returns with streams
 
-    getServiceDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
+    getContainerDetails('mysql', 'CLOUDRON_MYSQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         var input = fs.createReadStream(dumpPath('mysql', app.id));
@@ -1209,7 +1367,7 @@ function startPostgresql(existingInfra, callback) {
         shell.exec('startPostgresql', cmd, function (error) {
             if (error) return callback(error);
 
-            waitForService('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error) {
+            waitForContainer('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error) {
                 if (error) return callback(error);
                 if (!upgrading) return callback(null);
 
@@ -1237,7 +1395,7 @@ function setupPostgreSql(app, options, callback) {
             password: error ? hat(4 * 128) : existingPassword
         };
 
-        getServiceDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
+        getContainerDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
             if (error) return callback(error);
 
             request.post(`https://${result.ip}:3000/databases?access_token=${result.token}`, { rejectUnauthorized: false, json: data }, function (error, response) {
@@ -1271,7 +1429,7 @@ function clearPostgreSql(app, options, callback) {
 
     debugApp(app, 'Clearing postgresql');
 
-    getServiceDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
+    getContainerDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.post(`https://${result.ip}:3000/databases/${database}/clear?access_token=${result.token}&username=${username}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1290,7 +1448,7 @@ function teardownPostgreSql(app, options, callback) {
 
     const { database, username } = postgreSqlNames(app.id);
 
-    getServiceDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
+    getContainerDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.delete(`https://${result.ip}:3000/databases/${database}?access_token=${result.token}&username=${username}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1311,7 +1469,7 @@ function backupPostgreSql(app, options, callback) {
 
     const { database } = postgreSqlNames(app.id);
 
-    getServiceDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
+    getContainerDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         const url = `https://${result.ip}:3000/databases/${database}/backup?access_token=${result.token}`;
@@ -1330,7 +1488,7 @@ function restorePostgreSql(app, options, callback) {
 
     callback = once(callback); // protect from multiple returns with streams
 
-    getServiceDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
+    getContainerDetails('postgresql', 'CLOUDRON_POSTGRESQL_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         var input = fs.createReadStream(dumpPath('postgresql', app.id));
@@ -1347,6 +1505,43 @@ function restorePostgreSql(app, options, callback) {
     });
 }
 
+function startTurn(existingInfra, callback) {
+    assert.strictEqual(typeof existingInfra, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    // get and ensure we have a turn secret
+    var turnSecret = safe.fs.readFileSync(paths.ADDON_TURN_SECRET_FILE, 'utf8');
+    if (!turnSecret) {
+        turnSecret = 'a' + crypto.randomBytes(15).toString('hex'); // prefix with a to ensure string starts with a letter
+        safe.fs.writeFileSync(paths.ADDON_TURN_SECRET_FILE, turnSecret, 'utf8');
+    }
+
+    const tag = infra.images.turn.tag;
+    const memoryLimit = 256;
+    const realm = settings.adminFqdn();
+
+    if (existingInfra.version === infra.version && existingInfra.images.turn && infra.images.turn.tag === existingInfra.images.turn.tag) return callback();
+
+    // this exports 3478/tcp, 5349/tls and 50000-51000/udp
+    const cmd = `docker run --restart=always -d --name="turn" \
+                --hostname turn \
+                --net host \
+                --log-driver syslog \
+                --log-opt syslog-address=udp://127.0.0.1:2514 \
+                --log-opt syslog-format=rfc5424 \
+                --log-opt tag=turn \
+                -m ${memoryLimit}m \
+                --memory-swap ${memoryLimit * 2}m \
+                --dns 172.18.0.1 \
+                --dns-search=. \
+                -e CLOUDRON_TURN_SECRET="${turnSecret}" \
+                -e CLOUDRON_REALM="${realm}" \
+                --label isCloudronManaged=true \
+                --read-only -v /tmp -v /run "${tag}"`;
+
+    shell.exec('startTurn', cmd, callback);
+}
+
 function startMongodb(existingInfra, callback) {
     assert.strictEqual(typeof existingInfra, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -1386,7 +1581,7 @@ function startMongodb(existingInfra, callback) {
         shell.exec('startMongodb', cmd, function (error) {
             if (error) return callback(error);
 
-            waitForService('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error) {
+            waitForContainer('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error) {
                 if (error) return callback(error);
                 if (!upgrading) return callback(null);
 
@@ -1413,7 +1608,7 @@ function setupMongoDb(app, options, callback) {
             oplog: !!options.oplog
         };
 
-        getServiceDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
+        getContainerDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
             if (error) return callback(error);
 
             request.post(`https://${result.ip}:3000/databases?access_token=${result.token}`, { rejectUnauthorized: false, json: data }, function (error, response) {
@@ -1449,7 +1644,7 @@ function clearMongodb(app, options, callback) {
 
     debugApp(app, 'Clearing mongodb');
 
-    getServiceDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
+    getContainerDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.post(`https://${result.ip}:3000/databases/${app.id}/clear?access_token=${result.token}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1468,7 +1663,7 @@ function teardownMongoDb(app, options, callback) {
 
     debugApp(app, 'Tearing down mongodb');
 
-    getServiceDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
+    getContainerDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.delete(`https://${result.ip}:3000/databases/${app.id}?access_token=${result.token}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1487,7 +1682,7 @@ function backupMongoDb(app, options, callback) {
 
     debugApp(app, 'Backing up mongodb');
 
-    getServiceDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
+    getContainerDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         const url = `https://${result.ip}:3000/databases/${app.id}/backup?access_token=${result.token}`;
@@ -1504,7 +1699,7 @@ function restoreMongoDb(app, options, callback) {
 
     debugApp(app, 'restoreMongoDb');
 
-    getServiceDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
+    getContainerDetails('mongodb', 'CLOUDRON_MONGODB_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         const readStream = fs.createReadStream(dumpPath('mongodb', app.id));
@@ -1528,10 +1723,10 @@ function startRedis(existingInfra, callback) {
     const tag = infra.images.redis.tag;
     const upgrading = existingInfra.version !== 'none' && requiresUpgrade(existingInfra.images.redis.tag, tag);
 
-    appdb.getAll(function (error, apps) {
+    apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        async.eachSeries(apps, function iterator (app, iteratorCallback) {
+        async.eachSeries(allApps, function iterator (app, iteratorCallback) {
             if (!('redis' in app.manifest.addons)) return iteratorCallback(); // app doesn't use the addon
 
             setupRedis(app, app.manifest.addons.redis, iteratorCallback);
@@ -1560,15 +1755,7 @@ function setupRedis(app, options, callback) {
         const redisServiceToken = hat(4 * 48);
 
         // Compute redis memory limit based on app's memory limit (this is arbitrary)
-        var memoryLimit = app.memoryLimit || app.manifest.memoryLimit || 0;
-
-        if (memoryLimit === -1) { // unrestricted (debug mode)
-            memoryLimit = 0;
-        } else if (memoryLimit === 0 || memoryLimit <= (2 * 1024 * 1024 * 1024)) { // less than 2G (ram+swap)
-            memoryLimit = 150 * 1024 * 1024; // 150m
-        } else {
-            memoryLimit = 600 * 1024 * 1024; // 600m
-        }
+        const memoryLimit = app.servicesConfig['redis'] ? app.servicesConfig['redis'].memoryLimit : APP_SERVICES['redis'].defaultMemoryLimit;
 
         const tag = infra.images.redis.tag;
         const label = app.fqdn;
@@ -1612,7 +1799,7 @@ function setupRedis(app, options, callback) {
                 });
             },
             appdb.setAddonConfig.bind(null, app.id, 'redis', env),
-            waitForService.bind(null, 'redis-' + app.id, 'CLOUDRON_REDIS_TOKEN')
+            waitForContainer.bind(null, 'redis-' + app.id, 'CLOUDRON_REDIS_TOKEN')
         ], function (error) {
             if (error) debug('Error setting up redis: ', error);
             callback(error);
@@ -1627,7 +1814,7 @@ function clearRedis(app, options, callback) {
 
     debugApp(app, 'Clearing redis');
 
-    getServiceDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
+    getContainerDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         request.post(`https://${result.ip}:3000/clear?access_token=${result.token}`, { json: true, rejectUnauthorized: false }, function (error, response) {
@@ -1666,7 +1853,7 @@ function backupRedis(app, options, callback) {
 
     debugApp(app, 'Backing up redis');
 
-    getServiceDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
+    getContainerDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         const url = `https://${result.ip}:3000/backup?access_token=${result.token}`;
@@ -1683,7 +1870,7 @@ function restoreRedis(app, options, callback) {
 
     callback = once(callback); // protect from multiple returns with streams
 
-    getServiceDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
+    getContainerDetails('redis-' + app.id, 'CLOUDRON_REDIS_TOKEN', function (error, result) {
         if (error) return callback(error);
 
         let input;
@@ -1706,6 +1893,27 @@ function restoreRedis(app, options, callback) {
     });
 }
 
+function statusTurn(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    docker.inspect('turn', function (error, container) {
+        if (error && error.reason === BoxError.NOT_FOUND) return callback(null, { status: exports.SERVICE_STATUS_STOPPED });
+        if (error) return callback(error);
+
+        docker.memoryUsage(container.Id, function (error, result) {
+            if (error) return callback(error);
+
+            var tmp = {
+                status: container.State.Running ? exports.SERVICE_STATUS_ACTIVE : exports.SERVICE_STATUS_STOPPED,
+                memoryUsed: result.memory_stats.usage,
+                memoryPercent: parseInt(100 * result.memory_stats.usage / result.memory_stats.limit)
+            };
+
+            callback(null, tmp);
+        });
+    });
+}
+
 function statusDocker(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -1800,3 +2008,13 @@ function statusGraphite(callback) {
         });
     });
 }
+
+function teardownOauth(app, options, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debugApp(app, 'teardownOauth');
+
+    appdb.unsetAddonConfig(app.id, 'oauth', callback);
+}

src/appdb.js

@@ -41,7 +41,7 @@ var assert = require('assert'),
 var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
     'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
     'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson',
+    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson', 'apps.bindsJson',
     'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
     'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
     'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');
@@ -94,6 +94,14 @@ function postProcess(result) {
     result.debugMode = safe.JSON.parse(result.debugModeJson);
     delete result.debugModeJson;
 
+    assert(result.servicesConfigJson === null || typeof result.servicesConfigJson === 'string');
+    result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
+    delete result.servicesConfigJson;
+
+    assert(result.bindsJson === null || typeof result.bindsJson === 'string');
+    result.binds = safe.JSON.parse(result.bindsJson) || {};
+    delete result.bindsJson;
+
     result.alternateDomains = result.alternateDomains || [];
     result.alternateDomains.forEach(function (d) {
         delete d.appId;
@@ -427,7 +435,7 @@ function updateWithConstraints(id, app, constraints, callback) {
 
     var fields = [ ], values = [ ];
     for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig') {
+        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig' || p === 'binds') {
             fields.push(`${p}Json = ?`);
             values.push(JSON.stringify(app[p]));
         } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {

src/apphealthmonitor.js

@@ -73,7 +73,6 @@ function checkAppHealth(app, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     if (app.installationState !== apps.ISTATE_INSTALLED || app.runState !== apps.RSTATE_RUNNING) {
-        debugApp(app, 'skipped. istate:%s rstate:%s', app.installationState, app.runState);
         return callback(null);
     }
 
@@ -103,10 +102,8 @@ function checkAppHealth(app, callback) {
             .timeout(HEALTHCHECK_INTERVAL)
             .end(function (error, res) {
                 if (error && !error.response) {
-                    debugApp(app, 'not alive (network error): %s', error.message);
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
-                    debugApp(app, 'not alive : %s', error || res.status);
                     setHealth(app, apps.HEALTH_UNHEALTHY, callback);
                 } else {
                     setHealth(app, apps.HEALTH_HEALTHY, callback);
@@ -180,18 +177,16 @@ function processDockerEvents(intervalSecs, callback) {
 function processApp(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    apps.getAll(function (error, result) {
+    apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
-        async.each(result, checkAppHealth, function (error) {
+        async.each(allApps, checkAppHealth, function (error) {
             if (error) console.error(error);
 
-            const alive = result
-                .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; })
-                .map(a => a.fqdn)
-                .join(', ');
+            const alive = allApps
+                .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; });
 
-            debug('apps alive: [%s]', alive);
+            debug(`app health: ${alive.length} alive / ${allApps.length - alive.length} dead`);
 
             callback(null);
         });

src/appstore.js

@@ -127,7 +127,7 @@ function registerUser(email, password, callback) {
     const url = settings.apiServerOrigin() + '/api/v1/register_user';
     superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
         if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS));
+        if (result.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
         if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `register status code: ${result.statusCode}`));
 
         callback(null);
@@ -340,7 +340,8 @@ function sendAliveStatus(callback) {
     });
 }
 
-function getBoxUpdate(callback) {
+function getBoxUpdate(options, callback) {
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getCloudronToken(function (error, token) {
@@ -348,7 +349,13 @@ function getBoxUpdate(callback) {
 
         const url = `${settings.apiServerOrigin()}/api/v1/boxupdate`;
 
-        superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION }).timeout(10 * 1000).end(function (error, result) {
+        const query = {
+            accessToken: token,
+            boxVersion: constants.VERSION,
+            automatic: options.automatic
+        };
+
+        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -374,16 +381,24 @@ function getBoxUpdate(callback) {
     });
 }
 
-function getAppUpdate(app, callback) {
+function getAppUpdate(app, options, callback) {
     assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     getCloudronToken(function (error, token) {
         if (error) return callback(error);
 
         const url = `${settings.apiServerOrigin()}/api/v1/appupdate`;
+        const query = {
+            accessToken: token,
+            boxVersion: constants.VERSION,
+            appId: app.appStoreId,
+            appVersion: app.manifest.version,
+            automatic: options.automatic
+        };
 
-        superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, appId: app.appStoreId, appVersion: app.manifest.version }).timeout(10 * 1000).end(function (error, result) {
+        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
             if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
             if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
             if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
@@ -415,7 +430,7 @@ function registerCloudron(data, callback) {
 
     superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
         if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unable to register cloudron: ${error.message}`));
+        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unable to register cloudron: ${result.statusCode} ${error.message}`));
 
         // cloudronId, token, licenseKey
         if (!result.body.cloudronId) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response - no cloudron id'));
@@ -471,7 +486,7 @@ function registerWithLicense(license, domain, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT));
+        if (token) return callback(new BoxError(BoxError.CONFLICT, 'Cloudron is already registered'));
 
         const provider = settings.provider();
         const version = constants.VERSION;
@@ -491,7 +506,7 @@ function registerWithLoginCredentials(options, callback) {
     }
 
     getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT));
+        if (token) return callback(new BoxError(BoxError.CONFLICT, 'Cloudron is already registered'));
 
         maybeSignup(function (error) {
             if (error) return callback(error);

src/apptask.js

@@ -431,12 +431,12 @@ function waitForDnsPropagation(app, callback) {
     sysinfo.getServerIp(function (error, ip) {
         if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Error getting public IP: ${error.message}`));
 
-        domains.waitForDnsRecord(app.location, app.domain, 'A', ip, { interval: 5000, times: 240 }, function (error) {
+        domains.waitForDnsRecord(app.location, app.domain, 'A', ip, { times: 240 }, function (error) {
             if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: app.location, domain: app.domain }));
 
             // now wait for alternateDomains, if any
             async.eachSeries(app.alternateDomains, function (domain, iteratorCallback) {
-                domains.waitForDnsRecord(domain.subdomain, domain.domain, 'A', ip, { interval: 5000, times: 240 }, function (error) {
+                domains.waitForDnsRecord(domain.subdomain, domain.domain, 'A', ip, { times: 240 }, function (error) {
                     if (error) return callback(new BoxError(BoxError.DNS_ERROR, `DNS Record is not synced yet: ${error.message}`, { ip: ip, subdomain: domain.subdomain, domain: domain.domain }));
 
                     iteratorCallback();
@@ -776,9 +776,6 @@ function configure(app, args, progressCallback, callback) {
 
         startApp.bind(null, app),
 
-        progressCallback.bind(null, { percent: 80, message: 'Waiting for DNS propagation' }),
-        exports._waitForDnsPropagation.bind(null, app),
-
         progressCallback.bind(null, { percent: 90, message: 'Configuring reverse proxy' }),
         configureReverseProxy.bind(null, app),
 
@@ -902,7 +899,10 @@ function start(app, args, progressCallback, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     async.series([
-        progressCallback.bind(null, { percent: 20, message: 'Starting container' }),
+        progressCallback.bind(null, { percent: 10, message: 'Starting app services' }),
+        addons.startAppServices.bind(null, app),
+
+        progressCallback.bind(null, { percent: 35, message: 'Starting container' }),
         docker.startContainer.bind(null, app.id),
 
         // stopped apps do not renew certs. currently, we don't do DNS to not overwrite existing user settings
@@ -930,6 +930,9 @@ function stop(app, args, progressCallback, callback) {
         progressCallback.bind(null, { percent: 20, message: 'Stopping container' }),
         docker.stopContainers.bind(null, app.id),
 
+        progressCallback.bind(null, { percent: 50, message: 'Stopping app services' }),
+        addons.stopAppServices.bind(null, app),
+
         progressCallback.bind(null, { percent: 100, message: 'Done' }),
         updateApp.bind(null, app, { installationState: apps.ISTATE_INSTALLED, error: null, health: null })
     ], function seriesDone(error) {

src/backupdb.js

@@ -6,7 +6,7 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var BACKUPS_FIELDS = [ 'id', 'creationTime', 'version', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs' ];
+var BACKUPS_FIELDS = [ 'id', 'creationTime', 'packageVersion', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs', 'encryptionVersion' ];
 
 exports = module.exports = {
     add: add,
@@ -106,7 +106,8 @@ function get(id, callback) {
 function add(id, data, callback) {
     assert(data && typeof data === 'object');
     assert.strictEqual(typeof id, 'string');
-    assert.strictEqual(typeof data.version, 'string');
+    assert(data.encryptionVersion === null || typeof data.encryptionVersion === 'number');
+    assert.strictEqual(typeof data.packageVersion, 'string');
     assert(data.type === exports.BACKUP_TYPE_APP || data.type === exports.BACKUP_TYPE_BOX);
     assert(util.isArray(data.dependsOn));
     assert.strictEqual(typeof data.manifest, 'object');
@@ -116,8 +117,8 @@ function add(id, data, callback) {
     var creationTime = data.creationTime || new Date(); // allow tests to set the time
     var manifestJson = JSON.stringify(data.manifest);
 
-    database.query('INSERT INTO backups (id, version, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
-        [ id, data.version, data.type, creationTime, exports.BACKUP_STATE_NORMAL, data.dependsOn.join(','), manifestJson, data.format ],
+    database.query('INSERT INTO backups (id, encryptionVersion, packageVersion, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)',
+        [ id, data.encryptionVersion, data.packageVersion, data.type, creationTime, exports.BACKUP_STATE_NORMAL, data.dependsOn.join(','), manifestJson, data.format ],
         function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));

src/backups.js

@@ -32,12 +32,13 @@ exports = module.exports = {
 
     configureCollectd: configureCollectd,
 
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8),
+    generateEncryptionKeysSync: generateEncryptionKeysSync,
 
     // for testing
     _getBackupFilePath: getBackupFilePath,
     _restoreFsMetadata: restoreFsMetadata,
-    _saveFsMetadata: saveFsMetadata
+    _saveFsMetadata: saveFsMetadata,
+    _applyBackupRetentionPolicy: applyBackupRetentionPolicy
 };
 
 var addons = require('./addons.js'),
@@ -57,6 +58,7 @@ var addons = require('./addons.js'),
     eventlog = require('./eventlog.js'),
     fs = require('fs'),
     locker = require('./locker.js'),
+    moment = require('moment'),
     mkdirp = require('mkdirp'),
     once = require('once'),
     path = require('path'),
@@ -69,6 +71,7 @@ var addons = require('./addons.js'),
     syncer = require('./syncer.js'),
     tar = require('tar-fs'),
     tasks = require('./tasks.js'),
+    TransformStream = require('stream').Transform,
     util = require('util'),
     zlib = require('zlib');
 
@@ -94,19 +97,30 @@ function api(provider) {
     case 'wasabi': return require('./storage/s3.js');
     case 'scaleway-objectstorage': return require('./storage/s3.js');
     case 'linode-objectstorage': return require('./storage/s3.js');
+    case 'ovh-objectstorage': return require('./storage/s3.js');
     case 'noop': return require('./storage/noop.js');
     default: return null;
     }
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.key === exports.SECRET_PLACEHOLDER) newConfig.key = currentConfig.key;
+    if ('password' in newConfig) {
+        if (newConfig.password === constants.SECRET_PLACEHOLDER) {
+            delete newConfig.password;
+        }
+        newConfig.encryption = currentConfig.encryption || null;
+    } else {
+        newConfig.encryption = null;
+    }
     if (newConfig.provider === currentConfig.provider) api(newConfig.provider).injectPrivateFields(newConfig, currentConfig);
 }
 
 function removePrivateFields(backupConfig) {
     assert.strictEqual(typeof backupConfig, 'object');
-    if (backupConfig.key) backupConfig.key = exports.SECRET_PLACEHOLDER;
+    if (backupConfig.encryption) {
+        delete backupConfig.encryption;
+        backupConfig.password = constants.SECRET_PLACEHOLDER;
+    }
     return api(backupConfig.provider).removePrivateFields(backupConfig);
 }
 
@@ -120,12 +134,25 @@ function testConfig(backupConfig, callback) {
     if (backupConfig.format !== 'tgz' && backupConfig.format !== 'rsync') return callback(new BoxError(BoxError.BAD_FIELD, 'unknown format', { field: 'format' }));
 
     // remember to adjust the cron ensureBackup task interval accordingly
-    if (backupConfig.intervalSecs < 6 * 60 * 60) return callback(new BoxError(BoxError.BAD_FIELD, 'Interval must be atleast 6 hours', { field: 'interval' }));
+    if (backupConfig.intervalSecs < 6 * 60 * 60) return callback(new BoxError(BoxError.BAD_FIELD, 'Interval must be atleast 6 hours', { field: 'intervalSecs' }));
+
+    if ('password' in backupConfig) {
+        if (typeof backupConfig.password !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'password must be a string', { field: 'password' }));
+        if (backupConfig.password.length < 8) return callback(new BoxError(BoxError.BAD_FIELD, 'password must be atleast 8 characters', { field: 'password' }));
+    }
+
+    const policy = backupConfig.retentionPolicy;
+    if (!policy) return callback(new BoxError(BoxError.BAD_FIELD, 'retentionPolicy is required', { field: 'retentionPolicy' }));
+    if ('keepWithinSecs' in policy && typeof policy.keepWithinSecs !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepWithinSecs must be a number', { field: 'retentionPolicy' }));
+    if ('keepDaily' in policy && typeof policy.keepDaily !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepDaily must be a number', { field: 'retentionPolicy' }));
+    if ('keepWeekly' in policy && typeof policy.keepWeekly !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepWeekly must be a number', { field: 'retentionPolicy' }));
+    if ('keepMonthly' in policy && typeof policy.keepMonthly !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepMonthly must be a number', { field: 'retentionPolicy' }));
+    if ('keepYearly' in policy && typeof policy.keepYearly !== 'number') return callback(new BoxError(BoxError.BAD_FIELD, 'keepYearly must be a number', { field: 'retentionPolicy' }));
 
     api(backupConfig.provider).testConfig(backupConfig, callback);
 }
 
-
+// this skips password check since that policy is only at creation time
 function testProviderConfig(backupConfig, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -136,6 +163,18 @@ function testProviderConfig(backupConfig, callback) {
     api(backupConfig.provider).testConfig(backupConfig, callback);
 }
 
+function generateEncryptionKeysSync(password) {
+    assert.strictEqual(typeof password, 'string');
+
+    const aesKeys = crypto.scryptSync(password, Buffer.from('CLOUDRONSCRYPTSALT', 'utf8'), 128);
+    return {
+        dataKey: aesKeys.slice(0, 32).toString('hex'),
+        dataHmacKey: aesKeys.slice(32, 64).toString('hex'),
+        filenameKey: aesKeys.slice(64, 96).toString('hex'),
+        filenameHmacKey: aesKeys.slice(96).toString('hex')
+    };
+}
+
 function getByStatePaged(state, page, perPage, callback) {
     assert.strictEqual(typeof state, 'string');
     assert(typeof page === 'number' && page > 0);
@@ -179,21 +218,23 @@ function getBackupFilePath(backupConfig, backupId, format) {
     assert.strictEqual(typeof format, 'string');
 
     if (format === 'tgz') {
-        const fileType = backupConfig.key ? '.tar.gz.enc' : '.tar.gz';
+        const fileType = backupConfig.encryption ? '.tar.gz.enc' : '.tar.gz';
         return path.join(backupConfig.prefix || backupConfig.backupFolder || '', backupId+fileType);
     } else {
         return path.join(backupConfig.prefix || backupConfig.backupFolder || '', backupId);
     }
 }
 
-function encryptFilePath(filePath, key) {
+function encryptFilePath(filePath, encryption) {
     assert.strictEqual(typeof filePath, 'string');
-    assert.strictEqual(typeof key, 'string');
+    assert.strictEqual(typeof encryption, 'object');
 
     var encryptedParts = filePath.split('/').map(function (part) {
-        const cipher = crypto.createCipher('aes-256-cbc', key);
+        let hmac = crypto.createHmac('sha256', Buffer.from(encryption.filenameHmacKey, 'hex'));
+        const iv = hmac.update(part).digest().slice(0, 16); // iv has to be deterministic, for our sync (copy) logic to work
+        const cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(encryption.filenameKey, 'hex'), iv);
         let crypt = cipher.update(part);
-        crypt = Buffer.concat([ crypt, cipher.final() ]);
+        crypt = Buffer.concat([ iv, crypt, cipher.final() ]);
 
         return crypt.toString('base64')     // ensures path is valid
             .replace(/\//g, '-')            // replace '/' of base64 since it conflicts with path separator
@@ -203,9 +244,9 @@ function encryptFilePath(filePath, key) {
     return encryptedParts.join('/');
 }
 
-function decryptFilePath(filePath, key) {
+function decryptFilePath(filePath, encryption) {
     assert.strictEqual(typeof filePath, 'string');
-    assert.strictEqual(typeof key, 'string');
+    assert.strictEqual(typeof encryption, 'object');
 
     let decryptedParts = [];
     for (let part of filePath.split('/')) {
@@ -213,61 +254,172 @@ function decryptFilePath(filePath, key) {
         part = part.replace(/-/g, '/');                 // replace with '/'
 
         try {
-            let decrypt = crypto.createDecipher('aes-256-cbc', key);
-            let text = decrypt.update(Buffer.from(part, 'base64'));
-            text = Buffer.concat([ text, decrypt.final() ]);
-            decryptedParts.push(text.toString('utf8'));
+            const buffer = Buffer.from(part, 'base64');
+            const iv = buffer.slice(0, 16);
+            let decrypt = crypto.createDecipheriv('aes-256-cbc', Buffer.from(encryption.filenameKey, 'hex'), iv);
+            const plainText = decrypt.update(buffer.slice(16));
+            const plainTextString = Buffer.concat([ plainText, decrypt.final() ]).toString('utf8');
+            const hmac = crypto.createHmac('sha256', Buffer.from(encryption.filenameHmacKey, 'hex'));
+            if (!hmac.update(plainTextString).digest().slice(0, 16).equals(iv)) return { error: new BoxError(BoxError.CRYPTO_ERROR, `mac error decrypting part ${part} of path ${filePath}`) };
+
+            decryptedParts.push(plainTextString);
         } catch (error) {
-            debug(`Error decrypting file ${filePath} part ${part}:`, error);
-            return null;
+            debug(`Error decrypting part ${part} of path ${filePath}:`, error);
+            return { error: new BoxError(BoxError.CRYPTO_ERROR, `Error decrypting part ${part} of path ${filePath}: ${error.message}`) };
         }
     }
 
-    return decryptedParts.join('/');
+    return { result: decryptedParts.join('/') };
 }
 
-function createReadStream(sourceFile, key) {
+class EncryptStream extends TransformStream {
+    constructor(encryption) {
+        super();
+        this._headerPushed = false;
+        this._iv = crypto.randomBytes(16);
+        this._cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(encryption.dataKey, 'hex'), this._iv);
+        this._hmac = crypto.createHmac('sha256', Buffer.from(encryption.dataHmacKey, 'hex'));
+    }
+
+    pushHeaderIfNeeded() {
+        if (!this._headerPushed) {
+            const magic = Buffer.from('CBV2');
+            this.push(magic);
+            this._hmac.update(magic);
+            this.push(this._iv);
+            this._hmac.update(this._iv);
+            this._headerPushed = true;
+        }
+    }
+
+    _transform(chunk, ignoredEncoding, callback) {
+        this.pushHeaderIfNeeded();
+
+        try {
+            const crypt = this._cipher.update(chunk);
+            this._hmac.update(crypt);
+            callback(null, crypt);
+        } catch (error) {
+            callback(error);
+        }
+    }
+
+    _flush(callback) {
+        try {
+            this.pushHeaderIfNeeded(); // for 0-length files
+            const crypt = this._cipher.final();
+            this.push(crypt);
+            this._hmac.update(crypt);
+            callback(null, this._hmac.digest()); // +32 bytes
+        } catch (error) {
+            callback(error);
+        }
+    }
+}
+
+class DecryptStream extends TransformStream {
+    constructor(encryption) {
+        super();
+        this._key = Buffer.from(encryption.dataKey, 'hex');
+        this._header = Buffer.alloc(0);
+        this._decipher = null;
+        this._hmac = crypto.createHmac('sha256', Buffer.from(encryption.dataHmacKey, 'hex'));
+        this._buffer = Buffer.alloc(0);
+    }
+
+    _transform(chunk, ignoredEncoding, callback) {
+        const needed = 20 - this._header.length; // 4 for magic, 16 for iv
+
+        if (this._header.length !== 20) { // not gotten header yet
+            this._header = Buffer.concat([this._header, chunk.slice(0, needed)]);
+            if (this._header.length !== 20) return callback();
+
+            if (!this._header.slice(0, 4).equals(new Buffer.from('CBV2'))) return callback(new BoxError(BoxError.CRYPTO_ERROR, 'Invalid magic in header'));
+
+            const iv = this._header.slice(4);
+            this._decipher = crypto.createDecipheriv('aes-256-cbc', this._key, iv);
+            this._hmac.update(this._header);
+        }
+
+        this._buffer = Buffer.concat([ this._buffer, chunk.slice(needed) ]);
+        if (this._buffer.length < 32) return callback(); // hmac trailer length is 32
+
+        try {
+            const cipherText = this._buffer.slice(0, -32);
+            this._hmac.update(cipherText);
+            const plainText = this._decipher.update(cipherText);
+            this._buffer = this._buffer.slice(-32);
+            callback(null, plainText);
+        } catch (error) {
+            callback(error);
+        }
+    }
+
+    _flush (callback) {
+        if (this._buffer.length !== 32) return callback(new BoxError(BoxError.CRYPTO_ERROR, 'Invalid password or tampered file (not enough data)'));
+
+        try {
+            if (!this._hmac.digest().equals(this._buffer)) return callback(new BoxError(BoxError.CRYPTO_ERROR, 'Invalid password or tampered file (mac mismatch)'));
+
+            const plainText = this._decipher.final();
+            callback(null, plainText);
+        } catch (error) {
+            callback(error);
+        }
+    }
+}
+
+function createReadStream(sourceFile, encryption) {
     assert.strictEqual(typeof sourceFile, 'string');
-    assert(key === null || typeof key === 'string');
+    assert.strictEqual(typeof encryption, 'object');
 
     var stream = fs.createReadStream(sourceFile);
     var ps = progressStream({ time: 10000 }); // display a progress every 10 seconds
 
     stream.on('error', function (error) {
-        debug('createReadStream: read stream error.', error);
-        ps.emit('error', new BoxError(BoxError.FS_ERROR, error.message));
+        debug(`createReadStream: read stream error at ${sourceFile}`, error);
+        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Error reading ${sourceFile}: ${error.message}`));
     });
 
-    if (key !== null) {
-        var encrypt = crypto.createCipher('aes-256-cbc', key);
-        encrypt.on('error', function (error) {
-            debug('createReadStream: encrypt stream error.', error);
-            ps.emit('error', new BoxError(BoxError.CRYPTO_ERROR, error.message));
+    if (encryption) {
+        let encryptStream = new EncryptStream(encryption);
+
+        encryptStream.on('error', function (error) {
+            debug(`createReadStream: encrypt stream error ${sourceFile}`, error);
+            ps.emit('error', new BoxError(BoxError.CRYPTO_ERROR, `Encryption error at ${sourceFile}: ${error.message}`));
         });
-        return stream.pipe(encrypt).pipe(ps);
+
+        return stream.pipe(encryptStream).pipe(ps);
     } else {
         return stream.pipe(ps);
     }
 }
 
-function createWriteStream(destFile, key) {
+function createWriteStream(destFile, encryption) {
     assert.strictEqual(typeof destFile, 'string');
-    assert(key === null || typeof key === 'string');
+    assert.strictEqual(typeof encryption, 'object');
 
     var stream = fs.createWriteStream(destFile);
     var ps = progressStream({ time: 10000 }); // display a progress every 10 seconds
 
     stream.on('error', function (error) {
-        debug('createWriteStream: write stream error.', error);
-        ps.emit('error', new BoxError(BoxError.FS_ERROR, error.message));
+        debug(`createWriteStream: write stream error ${destFile}`, error);
+        ps.emit('error', new BoxError(BoxError.FS_ERROR, `Write error ${destFile}: ${error.message}`));
     });
 
-    if (key !== null) {
-        var decrypt = crypto.createDecipher('aes-256-cbc', key);
+    stream.on('finish', function () {
+        debug('createWriteStream: done.');
+        // we use a separate event because ps is a through2 stream which emits 'finish' event indicating end of inStream and not write
+        ps.emit('done');
+    });
+
+    if (encryption) {
+        let decrypt = new DecryptStream(encryption);
         decrypt.on('error', function (error) {
-            debug('createWriteStream: decrypt stream error.', error);
-            ps.emit('error', new BoxError(BoxError.CRYPTO_ERROR, error.message));
+            debug(`createWriteStream: decrypt stream error ${destFile}`, error);
+            ps.emit('error', new BoxError(BoxError.CRYPTO_ERROR, `Decryption error at ${destFile}: ${error.message}`));
         });
+
         ps.pipe(decrypt).pipe(stream);
     } else {
         ps.pipe(stream);
@@ -276,9 +428,9 @@ function createWriteStream(destFile, key) {
     return ps;
 }
 
-function tarPack(dataLayout, key, callback) {
+function tarPack(dataLayout, encryption, callback) {
     assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
-    assert(key === null || typeof key === 'string');
+    assert.strictEqual(typeof encryption, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     var pack = tar.pack('/', {
@@ -299,7 +451,7 @@ function tarPack(dataLayout, key, callback) {
     });
 
     var gzip = zlib.createGzip({});
-    var ps = progressStream({ time: 10000 }); // emit 'pgoress' every 10 seconds
+    var ps = progressStream({ time: 10000 }); // emit 'progress' every 10 seconds
 
     pack.on('error', function (error) {
         debug('tarPack: tar stream error.', error);
@@ -311,18 +463,19 @@ function tarPack(dataLayout, key, callback) {
         ps.emit('error', new BoxError(BoxError.EXTERNAL_ERROR, error.message));
     });
 
-    if (key !== null) {
-        var encrypt = crypto.createCipher('aes-256-cbc', key);
-        encrypt.on('error', function (error) {
+    if (encryption) {
+        const encryptStream = new EncryptStream(encryption);
+        encryptStream.on('error', function (error) {
             debug('tarPack: encrypt stream error.', error);
             ps.emit('error', new BoxError(BoxError.EXTERNAL_ERROR, error.message));
         });
-        pack.pipe(gzip).pipe(encrypt).pipe(ps);
+
+        pack.pipe(gzip).pipe(encryptStream).pipe(ps);
     } else {
         pack.pipe(gzip).pipe(ps);
     }
 
-    callback(null, ps);
+    return callback(null, ps);
 }
 
 function sync(backupConfig, backupId, dataLayout, progressCallback, callback) {
@@ -338,7 +491,7 @@ function sync(backupConfig, backupId, dataLayout, progressCallback, callback) {
     syncer.sync(dataLayout, function processTask(task, iteratorCallback) {
         debug('sync: processing task: %j', task);
         // the empty task.path is special to signify the directory
-        const destPath = task.path && backupConfig.key ? encryptFilePath(task.path, backupConfig.key) : task.path;
+        const destPath = task.path && backupConfig.encryption ? encryptFilePath(task.path, backupConfig.encryption) : task.path;
         const backupFilePath = path.join(getBackupFilePath(backupConfig, backupId, backupConfig.format), destPath);
 
         if (task.operation === 'removedir') {
@@ -359,15 +512,15 @@ function sync(backupConfig, backupId, dataLayout, progressCallback, callback) {
             if (task.operation === 'add') {
                 progressCallback({ message: `Adding ${task.path}` + (retryCount > 1 ?  ` (Try ${retryCount})` : '') });
                 debug(`Adding ${task.path} position ${task.position} try ${retryCount}`);
-                var stream = createReadStream(dataLayout.toLocalPath('./' + task.path), backupConfig.key || null);
+                var stream = createReadStream(dataLayout.toLocalPath('./' + task.path), backupConfig.encryption);
                 stream.on('error', function (error) {
                     debug(`read stream error for ${task.path}: ${error.message}`);
                     retryCallback();
                 }); // ignore error if file disappears
                 stream.on('progress', function (progress) {
                     const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
-                    if (!transferred && !speed) return progressCallback({ message: `Uploading ${task.path}` }); // 0M@0Mbps looks wrong
-                    progressCallback({ message: `Uploading ${task.path}: ${transferred}M@${speed}Mbps` }); // 0M@0Mbps looks wrong
+                    if (!transferred && !speed) return progressCallback({ message: `Uploading ${task.path}` }); // 0M@0MBps looks wrong
+                    progressCallback({ message: `Uploading ${task.path}: ${transferred}M@${speed}MBps` }); // 0M@0MBps looks wrong
                 });
                 api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
                     debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
@@ -460,13 +613,13 @@ function upload(backupId, format, dataLayoutString, progressCallback, callback)
                 async.retry({ times: 5, interval: 20000 }, function (retryCallback) {
                     retryCallback = once(retryCallback); // protect again upload() erroring much later after tar stream error
 
-                    tarPack(dataLayout, backupConfig.key || null, function (error, tarStream) {
+                    tarPack(dataLayout, backupConfig.encryption, function (error, tarStream) {
                         if (error) return retryCallback(error);
 
                         tarStream.on('progress', function (progress) {
                             const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
-                            if (!transferred && !speed) return progressCallback({ message: 'Uploading backup' }); // 0M@0Mbps looks wrong
-                            progressCallback({ message: `Uploading backup ${transferred}M@${speed}Mbps` });
+                            if (!transferred && !speed) return progressCallback({ message: 'Uploading backup' }); // 0M@0MBps looks wrong
+                            progressCallback({ message: `Uploading backup ${transferred}M@${speed}MBps` });
                         });
                         tarStream.on('error', retryCallback); // already returns BoxError
 
@@ -483,10 +636,10 @@ function upload(backupId, format, dataLayoutString, progressCallback, callback)
     });
 }
 
-function tarExtract(inStream, dataLayout, key, callback) {
+function tarExtract(inStream, dataLayout, encryption, callback) {
     assert.strictEqual(typeof inStream, 'object');
     assert(dataLayout instanceof DataLayout, 'dataLayout must be a DataLayout');
-    assert(key === null || typeof key === 'string');
+    assert.strictEqual(typeof encryption, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     var gunzip = zlib.createGunzip({});
@@ -495,10 +648,14 @@ function tarExtract(inStream, dataLayout, key, callback) {
         map: function (header) {
             header.name = dataLayout.toLocalPath(header.name);
             return header;
-        }
+        },
+        dmode: 500 // ensure directory is writable
     });
 
-    const emitError = once((error) => ps.emit('error', error));
+    const emitError = once((error) => {
+        inStream.destroy();
+        ps.emit('error', error);
+    });
 
     inStream.on('error', function (error) {
         debug('tarExtract: input stream error.', error);
@@ -521,8 +678,8 @@ function tarExtract(inStream, dataLayout, key, callback) {
         ps.emit('done');
     });
 
-    if (key !== null) {
-        var decrypt = crypto.createDecipher('aes-256-cbc', key);
+    if (encryption) {
+        let decrypt = new DecryptStream(encryption);
         decrypt.on('error', function (error) {
             debug('tarExtract: decrypt stream error.', error);
             emitError(new BoxError(BoxError.EXTERNAL_ERROR, `Failed to decrypt: ${error.message}`));
@@ -573,9 +730,10 @@ function downloadDir(backupConfig, backupFilePath, dataLayout, progressCallback,
 
     function downloadFile(entry, done) {
         let relativePath = path.relative(backupFilePath, entry.fullPath);
-        if (backupConfig.key) {
-            relativePath = decryptFilePath(relativePath, backupConfig.key);
-            if (!relativePath) return done(new BoxError(BoxError.BAD_STATE, 'Unable to decrypt file'));
+        if (backupConfig.encryption) {
+            const { error, result } = decryptFilePath(relativePath, backupConfig.encryption);
+            if (error) return done(new BoxError(BoxError.CRYPTO_ERROR, 'Unable to decrypt file'));
+            relativePath = result;
         }
         const destFilePath = dataLayout.toLocalPath('./' + relativePath);
 
@@ -583,31 +741,35 @@ function downloadDir(backupConfig, backupFilePath, dataLayout, progressCallback,
             if (error) return done(new BoxError(BoxError.FS_ERROR, error.message));
 
             async.retry({ times: 5, interval: 20000 }, function (retryCallback) {
-                let destStream = createWriteStream(destFilePath, backupConfig.key || null);
-
-                destStream.on('progress', function (progress) {
-                    const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
-                    if (!transferred && !speed) return progressCallback({ message: `Downloading ${entry.fullPath}` }); // 0M@0Mbps looks wrong
-                    progressCallback({ message: `Downloading ${entry.fullPath}: ${transferred}M@${speed}Mbps` });
-                });
-
-                // protect against multiple errors. must destroy the write stream so that a previous retry does not write
-                let closeAndRetry = once((error) => {
-                    if (error) progressCallback({ message: `Download ${entry.fullPath} to ${destFilePath} errored: ${error.message}` });
-                    else progressCallback({ message: `Download ${entry.fullPath} to ${destFilePath} finished` });
-                    destStream.destroy();
-                    retryCallback(error);
-                });
-
                 api(backupConfig.provider).download(backupConfig, entry.fullPath, function (error, sourceStream) {
-                    if (error) return closeAndRetry(error);
+                    if (error) {
+                        progressCallback({ message: `Download ${entry.fullPath} to ${destFilePath} errored: ${error.message}` });
+                        return retryCallback(error);
+                    }
+
+                    let destStream = createWriteStream(destFilePath, backupConfig.encryption);
+
+                    // protect against multiple errors. must destroy the write stream so that a previous retry does not write
+                    let closeAndRetry = once((error) => {
+                        if (error) progressCallback({ message: `Download ${entry.fullPath} to ${destFilePath} errored: ${error.message}` });
+                        else progressCallback({ message: `Download ${entry.fullPath} to ${destFilePath} finished` });
+                        sourceStream.destroy();
+                        destStream.destroy();
+                        retryCallback(error);
+                    });
+
+                    destStream.on('progress', function (progress) {
+                        const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
+                        if (!transferred && !speed) return progressCallback({ message: `Downloading ${entry.fullPath}` }); // 0M@0MBps looks wrong
+                        progressCallback({ message: `Downloading ${entry.fullPath}: ${transferred}M@${speed}MBps` });
+                    });
+                    destStream.on('error', closeAndRetry);
 
                     sourceStream.on('error', closeAndRetry);
-                    destStream.on('error', closeAndRetry); // already emits BoxError
 
                     progressCallback({ message: `Downloading ${entry.fullPath} to ${destFilePath}` });
 
-                    sourceStream.pipe(destStream, { end: true }).on('finish', closeAndRetry);
+                    sourceStream.pipe(destStream, { end: true }).on('done', closeAndRetry);
                 });
             }, done);
         });
@@ -638,13 +800,13 @@ function download(backupConfig, backupId, format, dataLayout, progressCallback,
             api(backupConfig.provider).download(backupConfig, backupFilePath, function (error, sourceStream) {
                 if (error) return retryCallback(error);
 
-                tarExtract(sourceStream, dataLayout, backupConfig.key || null, function (error, ps) {
+                tarExtract(sourceStream, dataLayout, backupConfig.encryption, function (error, ps) {
                     if (error) return retryCallback(error);
 
                     ps.on('progress', function (progress) {
                         const transferred = Math.round(progress.transferred/1024/1024), speed = Math.round(progress.speed/1024/1024);
-                        if (!transferred && !speed) return progressCallback({ message: 'Downloading backup' }); // 0M@0Mbps looks wrong
-                        progressCallback({ message: `Downloading ${transferred}M@${speed}Mbps` });
+                        if (!transferred && !speed) return progressCallback({ message: 'Downloading backup' }); // 0M@0MBps looks wrong
+                        progressCallback({ message: `Downloading ${transferred}M@${speed}MBps` });
                     });
                     ps.on('error', retryCallback);
                     ps.on('done', retryCallback);
@@ -817,7 +979,15 @@ function rotateBoxBackup(backupConfig, tag, appBackupIds, progressCallback, call
 
     debug(`Rotating box backup to id ${backupId}`);
 
-    backupdb.add(backupId, { version: constants.VERSION, type: backupdb.BACKUP_TYPE_BOX, dependsOn: appBackupIds, manifest: null, format: format }, function (error) {
+    const data = {
+        encryptionVersion: backupConfig.encryption ? 2 : null,
+        packageVersion: constants.VERSION,
+        type: backupdb.BACKUP_TYPE_BOX,
+        dependsOn: appBackupIds,
+        manifest: null,
+        format: format
+    };
+    backupdb.add(backupId, data, function (error) {
         if (error) return callback(error);
 
         var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, 'snapshot/box', format), getBackupFilePath(backupConfig, backupId, format));
@@ -855,9 +1025,10 @@ function backupBoxWithAppBackupIds(appBackupIds, tag, progressCallback, callback
 }
 
 function canBackupApp(app) {
-    // only backup apps that are installed or pending configure or called from apptask. Rest of them are in some
-    // state not good for consistent backup (i.e addons may not have been setup completely)
-    return (app.installationState === apps.ISTATE_INSTALLED && app.health === apps.HEALTH_HEALTHY) ||
+    // only backup apps that are installed or specific pending states
+    // we used to check the health here but that doesn't work for stopped apps. it's better to just fail
+    // and inform the user if the backup fails and the app addons have not been setup yet.
+    return  app.installationState === apps.ISTATE_INSTALLED ||
             app.installationState === apps.ISTATE_PENDING_CONFIGURE ||
             app.installationState === apps.ISTATE_PENDING_BACKUP ||  // called from apptask
             app.installationState === apps.ISTATE_PENDING_UPDATE; // called from apptask
@@ -898,7 +1069,16 @@ function rotateAppBackup(backupConfig, app, tag, options, progressCallback, call
 
     debug(`Rotating app backup of ${app.id} to id ${backupId}`);
 
-    backupdb.add(backupId, { version: manifest.version, type: backupdb.BACKUP_TYPE_APP, dependsOn: [ ], manifest: manifest, format: format }, function (error) {
+    const data = {
+        encryptionVersion: backupConfig.encryption ? 2 : null,
+        packageVersion: manifest.version,
+        type: backupdb.BACKUP_TYPE_APP,
+        dependsOn: [ ],
+        manifest,
+        format: format
+    };
+
+    backupdb.add(backupId, data, function (error) {
         if (error) return callback(error);
 
         var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, `snapshot/app_${app.id}`, format), getBackupFilePath(backupConfig, backupId, format));
@@ -1078,9 +1258,57 @@ function ensureBackup(auditSource, callback) {
     });
 }
 
-function cleanupBackup(backupConfig, backup, callback) {
+// backups must be descending in creationTime
+function applyBackupRetentionPolicy(backups, policy) {
+    assert(Array.isArray(backups));
+    assert.strictEqual(typeof policy, 'object');
+
+    const now = new Date();
+
+    for (const backup of backups) {
+        if (backup.keepReason) continue; // already kept for some other reason
+
+        if ((now - backup.creationTime) < (backup.preserveSecs * 1000)) {
+            backup.keepReason = 'preserveSecs';
+        } else if ((now - backup.creationTime < policy.keepWithinSecs * 1000) || policy.keepWithinSecs < 0) {
+            backup.keepReason = 'keepWithinSecs';
+        }
+    }
+
+    const KEEP_FORMATS = {
+        keepDaily: 'Y-M-D',
+        keepWeekly: 'Y-W',
+        keepMonthly: 'Y-M',
+        keepYearly: 'Y'
+    };
+
+    for (const format of [ 'keepDaily', 'keepWeekly', 'keepMonthly', 'keepYearly' ]) {
+        if (!(format in policy)) continue;
+
+        const n = policy[format]; // we want to keep "n" backups of format
+        if (!n) continue; // disabled rule
+
+        let lastPeriod = null, keptSoFar = 0;
+        for (const backup of backups) {
+            if (backup.keepReason) continue; // already kept for some other reason
+            const period = moment(backup.creationTime).format(KEEP_FORMATS[format]);
+            if (period === lastPeriod) continue; // already kept for this period
+
+            lastPeriod = period;
+            backup.keepReason = format;
+            if (++keptSoFar === n) break;
+        }
+    }
+
+    for (const backup of backups) {
+        if (backup.keepReason) debug(`applyBackupRetentionPolicy: ${backup.id} ${backup.type} ${backup.keepReason}`);
+    }
+}
+
+function cleanupBackup(backupConfig, backup, progressCallback, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof backup, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
     var backupFilePath = getBackupFilePath(backupConfig, backup.id, backup.format);
@@ -1105,55 +1333,60 @@ function cleanupBackup(backupConfig, backup, callback) {
     }
 
     if (backup.format ==='tgz') {
+        progressCallback({ message: `${backup.id}: Removing ${backupFilePath}`});
         api(backupConfig.provider).remove(backupConfig, backupFilePath, done);
     } else {
         var events = api(backupConfig.provider).removeDir(backupConfig, backupFilePath);
-        events.on('progress', function (detail) { debug(`cleanupBackup: ${detail}`); });
+        events.on('progress', (message) => progressCallback({ message: `${backup.id}: ${message}` }));
         events.on('done', done);
     }
 }
 
-function cleanupAppBackups(backupConfig, referencedAppBackups, callback) {
+function cleanupAppBackups(backupConfig, referencedAppBackupIds, progressCallback, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
-    assert(Array.isArray(referencedAppBackups));
+    assert(Array.isArray(referencedAppBackupIds));
+    assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof callback, 'function');
 
-    const now = new Date();
-    let removedAppBackups = [];
+    let removedAppBackupIds = [];
 
     // we clean app backups of any state because the ones to keep are determined by the box cleanup code
     backupdb.getByTypePaged(backupdb.BACKUP_TYPE_APP, 1, 1000, function (error, appBackups) {
         if (error) return callback(error);
 
+        for (const appBackup of appBackups) { // set the reason so that policy filter can skip it
+            if (referencedAppBackupIds.includes(appBackup.id)) appBackup.keepReason = 'reference';
+        }
+
+        applyBackupRetentionPolicy(appBackups, backupConfig.retentionPolicy);
+
         async.eachSeries(appBackups, function iterator(appBackup, iteratorDone) {
-            if (referencedAppBackups.indexOf(appBackup.id) !== -1) return iteratorDone();
-            if ((now - appBackup.creationTime) < (appBackup.preserveSecs * 1000)) return iteratorDone();
-            if ((now - appBackup.creationTime) < (backupConfig.retentionSecs * 1000)) return iteratorDone();
+            if (appBackup.keepReason) return iteratorDone();
 
-            debug('cleanupAppBackups: removing %s', appBackup.id);
+            progressCallback({ message: `Removing app backup ${appBackup.id}`});
 
-            removedAppBackups.push(appBackup.id);
-            cleanupBackup(backupConfig, appBackup, iteratorDone);
+            removedAppBackupIds.push(appBackup.id);
+            cleanupBackup(backupConfig, appBackup, progressCallback, iteratorDone);
         }, function () {
             debug('cleanupAppBackups: done');
 
-            callback(null, removedAppBackups);
+            callback(null, removedAppBackupIds);
         });
     });
 }
 
-function cleanupBoxBackups(backupConfig, auditSource, callback) {
+function cleanupBoxBackups(backupConfig, progressCallback, auditSource, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
+    assert.strictEqual(typeof progressCallback, 'function');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    const now = new Date();
-    let referencedAppBackups = [], removedBoxBackups = [];
+    let referencedAppBackupIds = [], removedBoxBackupIds = [];
 
     backupdb.getByTypePaged(backupdb.BACKUP_TYPE_BOX, 1, 1000, function (error, boxBackups) {
         if (error) return callback(error);
 
-        if (boxBackups.length === 0) return callback(null, { removedBoxBackups, referencedAppBackups });
+        if (boxBackups.length === 0) return callback(null, { removedBoxBackupIds, referencedAppBackupIds });
 
         // search for the first valid backup
         var i;
@@ -1164,28 +1397,28 @@ function cleanupBoxBackups(backupConfig, auditSource, callback) {
         // keep the first valid backup
         if (i !== boxBackups.length) {
             debug('cleanupBoxBackups: preserving box backup %s (%j)', boxBackups[i].id, boxBackups[i].dependsOn);
-            referencedAppBackups = boxBackups[i].dependsOn;
+            referencedAppBackupIds = boxBackups[i].dependsOn;
             boxBackups.splice(i, 1);
         } else {
             debug('cleanupBoxBackups: no box backup to preserve');
         }
 
+        applyBackupRetentionPolicy(boxBackups, backupConfig.retentionPolicy);
+
         async.eachSeries(boxBackups, function iterator(boxBackup, iteratorNext) {
-            // TODO: errored backups should probably be cleaned up before retention time, but we will
-            // have to be careful not to remove any backup currently being created
-            if ((now - boxBackup.creationTime) < (backupConfig.retentionSecs * 1000)) {
-                referencedAppBackups = referencedAppBackups.concat(boxBackup.dependsOn);
+            if (boxBackup.keepReason) {
+                referencedAppBackupIds = referencedAppBackupIds.concat(boxBackup.dependsOn);
                 return iteratorNext();
             }
 
-            debug('cleanupBoxBackups: removing %s', boxBackup.id);
+            progressCallback({ message: `Removing box backup ${boxBackup.id}`});
 
-            removedBoxBackups.push(boxBackup.id);
-            cleanupBackup(backupConfig, boxBackup, iteratorNext);
+            removedBoxBackupIds.push(boxBackup.id);
+            cleanupBackup(backupConfig, boxBackup, progressCallback, iteratorNext);
         }, function () {
             debug('cleanupBoxBackups: done');
 
-            callback(null, { removedBoxBackups, referencedAppBackups });
+            callback(null, { removedBoxBackupIds, referencedAppBackupIds });
         });
     });
 }
@@ -1247,19 +1480,19 @@ function cleanup(auditSource, progressCallback, callback) {
     settings.getBackupConfig(function (error, backupConfig) {
         if (error) return callback(error);
 
-        if (backupConfig.retentionSecs < 0) {
+        if (backupConfig.retentionPolicy.keepWithinSecs < 0) {
             debug('cleanup: keeping all backups');
             return callback(null, {});
         }
 
         progressCallback({ percent: 10, message: 'Cleaning box backups' });
 
-        cleanupBoxBackups(backupConfig, auditSource, function (error, result) {
+        cleanupBoxBackups(backupConfig, progressCallback, auditSource, function (error, { removedBoxBackupIds, referencedAppBackupIds }) {
             if (error) return callback(error);
 
             progressCallback({ percent: 40, message: 'Cleaning app backups' });
 
-            cleanupAppBackups(backupConfig, result.referencedAppBackups, function (error, removedAppBackups) {
+            cleanupAppBackups(backupConfig, referencedAppBackupIds, progressCallback, function (error, removedAppBackupIds) {
                 if (error) return callback(error);
 
                 progressCallback({ percent: 90, message: 'Cleaning snapshots' });
@@ -1267,7 +1500,7 @@ function cleanup(auditSource, progressCallback, callback) {
                 cleanupSnapshots(backupConfig, function (error) {
                     if (error) return callback(error);
 
-                    callback(null, { removedBoxBackups: result.removedBoxBackups, removedAppBackups: removedAppBackups });
+                    callback(null, { removedBoxBackupIds, removedAppBackupIds });
                 });
             });
         });

src/cert/acme2.js

@@ -332,7 +332,7 @@ Acme2.prototype.createKeyAndCsr = function (hostname, callback) {
         // in some old releases, csr file was corrupt. so always regenerate it
         debug('createKeyAndCsr: reuse the key for renewal at %s', privateKeyFile);
     } else {
-        var key = safe.child_process.execSync('openssl genrsa 4096');
+        var key = safe.child_process.execSync('openssl ecparam -genkey -name secp384r1'); // openssl ecparam -list_curves
         if (!key) return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
         if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
 
@@ -452,7 +452,7 @@ Acme2.prototype.prepareDnsChallenge = function (hostname, domain, authorization,
     domains.upsertDnsRecords(challengeSubdomain, domain, 'TXT', [ `"${txtValue}"` ], function (error) {
         if (error) return callback(error);
 
-        domains.waitForDnsRecord(challengeSubdomain, domain, 'TXT', txtValue, { interval: 5000, times: 200 }, function (error) {
+        domains.waitForDnsRecord(challengeSubdomain, domain, 'TXT', txtValue, { times: 200 }, function (error) {
             if (error) return callback(error);
 
             callback(null, challenge);

src/cloudron.js

@@ -21,7 +21,8 @@ exports = module.exports = {
     runSystemChecks: runSystemChecks,
 };
 
-var apps = require('./apps.js'),
+var addons = require('./addons.js'),
+    apps = require('./apps.js'),
     appstore = require('./appstore.js'),
     assert = require('assert'),
     async = require('async'),
@@ -326,6 +327,7 @@ function setDashboardAndMailDomain(domain, auditSource, callback) {
         if (error) return callback(error);
 
         mail.onMailFqdnChanged(NOOP_CALLBACK); // this will update dns and re-configure mail server
+        addons.restartService('turn', NOOP_CALLBACK); // to update the realm variable
 
         callback(null);
     });

src/constants.js

@@ -32,17 +32,16 @@ exports = module.exports = {
 
     NGINX_DEFAULT_CONFIG_FILE_NAME: 'default.conf',
 
-    GHOST_USER_FILE: '/tmp/cloudron_ghost.json',
-
-    DEFAULT_TOKEN_EXPIRATION: 7 * 24 * 60 * 60 * 1000, // 1 week
+    DEFAULT_TOKEN_EXPIRATION: 365 * 24 * 60 * 60 * 1000, // 1 year
 
     DEFAULT_MEMORY_LIMIT: (256 * 1024 * 1024), // see also client.js
 
     DEMO_USERNAME: 'cloudron',
+    DEMO_BLACKLISTED_APPS: [ 'com.github.cloudtorrent' ],
 
     AUTOUPDATE_PATTERN_NEVER: 'never',
 
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8),
+    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8), // also used in dashboard client.js
 
     CLOUDRON: CLOUDRON,
     TEST: TEST,
@@ -51,6 +50,6 @@ exports = module.exports = {
 
     FOOTER: '&copy; 2020  &nbsp;  [Cloudron](https://cloudron.io) &nbsp; &nbsp; &nbsp;  [Forum <i class="fa fa-comments"></i>](https://forum.cloudron.io)',
 
-    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '4.2.0-test'
+    VERSION: process.env.BOX_ENV === 'cloudron' ? fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim() : '5.1.1-test'
 };
 

src/cron.js

@@ -80,14 +80,14 @@ function startJobs(callback) {
     });
 
     gJobs.boxUpdateCheckerJob = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
-        onTick: () => updateChecker.checkBoxUpdates(NOOP_CALLBACK),
+        cronTime: '00 ' + randomMinute + ' 23 * * *', // once an day
+        onTick: () => updateChecker.checkBoxUpdates({ automatic: true }, NOOP_CALLBACK),
         start: true
     });
 
     gJobs.appUpdateChecker = new CronJob({
-        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
-        onTick: () => updateChecker.checkAppUpdates(NOOP_CALLBACK),
+        cronTime: '00 ' + randomMinute + ' 22 * * *', // once an day
+        onTick: () => updateChecker.checkAppUpdates({ automatic: true }, NOOP_CALLBACK),
         start: true
     });
 

src/dns/caas.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/caas'),
     domains = require('../domains.js'),
     settings = require('../settings.js'),
@@ -31,7 +32,7 @@ function getFqdn(location, domain) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
 
     // do not return the 'key'. in caas, this is private
     delete domainObject.fallbackCertificate.key;
@@ -40,7 +41,7 @@ function removePrivateFields(domainObject) {
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/cloudflare.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     async = require('async'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/cloudflare'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -25,12 +26,12 @@ var assert = require('assert'),
 var CLOUDFLARE_ENDPOINT = 'https://api.cloudflare.com/client/v4';
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function translateRequestError(result, callback) {
@@ -39,9 +40,14 @@ function translateRequestError(result, callback) {
 
     if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND, util.format('%s %j', result.statusCode, 'API does not exist')));
     if (result.statusCode === 422) return callback(new BoxError(BoxError.BAD_FIELD, result.body.message));
-    if ((result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) && result.body.errors.length > 0) {
-        let error = result.body.errors[0];
-        let message = `message: ${error.message} statusCode: ${result.statusCode} code:${error.code}`;
+    if (result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) {
+        let message = 'Unknown error';
+        if (typeof result.body.error === 'string') {
+            message = `message: ${result.body.error} statusCode: ${result.statusCode}`;
+        } else if (Array.isArray(result.body.errors) && result.body.errors.length > 0) {
+            let error = result.body.errors[0];
+            message = `message: ${error.message} statusCode: ${result.statusCode} code:${error.code}`;
+        }
         return callback(new BoxError(BoxError.ACCESS_DENIED, message));
     }
 
@@ -284,7 +290,7 @@ function verifyDnsConfig(domainObject, callback) {
     if (dnsConfig.tokenType !== 'GlobalApiKey' && dnsConfig.tokenType !== 'ApiToken')  return callback(new BoxError(BoxError.BAD_FIELD, 'tokenType is required', { field: 'tokenType' }));
 
     if (dnsConfig.tokenType === 'GlobalApiKey') {
-        if ('email' in dnsConfig && typeof dnsConfig.email !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'email must be a non-empty string', { field: 'email' }));
+        if (typeof dnsConfig.email !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'email must be a non-empty string', { field: 'email' }));
     }
 
     const ip = '127.0.0.1';

src/dns/digitalocean.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     async = require('async'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/digitalocean'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -28,12 +29,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function getInternal(dnsConfig, zoneName, name, type, callback) {

src/dns/gandi.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/gandi'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -26,12 +27,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/gcdns.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/gcdns'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -21,12 +22,12 @@ var assert = require('assert'),
     _ = require('underscore');
 
 function removePrivateFields(domainObject) {
-    domainObject.config.credentials.private_key = domains.SECRET_PLACEHOLDER;
+    domainObject.config.credentials.private_key = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.credentials.private_key === domains.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
+    if (newConfig.credentials.private_key === constants.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
 }
 
 function getDnsCredentials(dnsConfig) {

src/dns/godaddy.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/godaddy'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -32,12 +33,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.apiSecret = domains.SECRET_PLACEHOLDER;
+    domainObject.config.apiSecret = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.apiSecret === domains.SECRET_PLACEHOLDER) newConfig.apiSecret = currentConfig.apiSecret;
+    if (newConfig.apiSecret === constants.SECRET_PLACEHOLDER) newConfig.apiSecret = currentConfig.apiSecret;
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/interface.js

@@ -21,13 +21,13 @@ var assert = require('assert'),
     util = require('util');
 
 function removePrivateFields(domainObject) {
-    // in-place removal of tokens and api keys with domains.SECRET_PLACEHOLDER
+    // in-place removal of tokens and api keys with constants.SECRET_PLACEHOLDER
     return domainObject;
 }
 
 // eslint-disable-next-line no-unused-vars
 function injectPrivateFields(newConfig, currentConfig) {
-    // in-place injection of tokens and api keys which came in with domains.SECRET_PLACEHOLDER
+    // in-place injection of tokens and api keys which came in with constants.SECRET_PLACEHOLDER
 }
 
 function upsert(domainObject, location, type, values, callback) {

src/dns/linode.js

@@ -0,0 +1,310 @@
+'use strict';
+
+exports = module.exports = {
+    removePrivateFields: removePrivateFields,
+    injectPrivateFields: injectPrivateFields,
+    upsert: upsert,
+    get: get,
+    del: del,
+    wait: wait,
+    verifyDnsConfig: verifyDnsConfig
+};
+
+let async = require('async'),
+    assert = require('assert'),
+    constants = require('../constants.js'),
+    BoxError = require('../boxerror.js'),
+    debug = require('debug')('box:dns/linode'),
+    dns = require('../native-dns.js'),
+    domains = require('../domains.js'),
+    superagent = require('superagent'),
+    util = require('util'),
+    waitForDns = require('./waitfordns.js');
+
+const LINODE_ENDPOINT = 'https://api.linode.com/v4';
+
+function formatError(response) {
+    return util.format('Linode DNS error [%s] %j', response.statusCode, response.body);
+}
+
+function removePrivateFields(domainObject) {
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
+    return domainObject;
+}
+
+function injectPrivateFields(newConfig, currentConfig) {
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+}
+
+function getZoneId(dnsConfig, zoneName, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    // returns 100 at a time
+    superagent.get(`${LINODE_ENDPOINT}/domains`)
+        .set('Authorization', 'Bearer ' + dnsConfig.token)
+        .timeout(30 * 1000)
+        .retry(5)
+        .end(function (error, result) {
+            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+            if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.ACCESS_DENIED, formatError(result)));
+            if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+            if (!Array.isArray(result.body.data)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response'));
+
+            const zone = result.body.data.find(d => d.domain === zoneName);
+
+            if (!zone || !zone.id) return callback(new BoxError(BoxError.NOT_FOUND, 'Zone not found'));
+
+            debug(`getZoneId: zone id of ${zoneName} is ${zone.id}`);
+
+            callback(null, zone.id);
+        });
+}
+
+function getZoneRecords(dnsConfig, zoneName, name, type, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug(`getInternal: getting dns records of ${zoneName} with ${name} and type ${type}`);
+
+    getZoneId(dnsConfig, zoneName, function (error, zoneId) {
+        if (error) return callback(error);
+
+        let page = 0, more = false;
+        let records = [];
+
+        async.doWhilst(function (iteratorDone) {
+            const url = `${LINODE_ENDPOINT}/domains/${zoneId}/records?page=${++page}`;
+
+            superagent.get(url)
+                .set('Authorization', 'Bearer ' + dnsConfig.token)
+                .timeout(30 * 1000)
+                .retry(5)
+                .end(function (error, result) {
+                    if (error && !error.response) return iteratorDone(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                    if (result.statusCode === 404) return iteratorDone(new BoxError(BoxError.NOT_FOUND, formatError(result)));
+                    if (result.statusCode === 403 || result.statusCode === 401) return iteratorDone(new BoxError(BoxError.ACCESS_DENIED, formatError(result)));
+                    if (result.statusCode !== 200) return iteratorDone(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                    records = records.concat(result.body.data.filter(function (record) {
+                        return (record.type === type && record.name === name);
+                    }));
+
+                    more = result.body.page !== result.body.pages;
+
+                    iteratorDone();
+                });
+        }, function () { return more; }, function (error) {
+            debug('getZoneRecords:', error, JSON.stringify(records));
+
+            if (error) return callback(error);
+
+            callback(null, { zoneId, records });
+        });
+    });
+}
+
+function get(domainObject, location, type, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '';
+
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { records }) {
+        if (error) return callback(error);
+
+        var tmp = records.map(function (record) { return record.target; });
+
+        debug('get: %j', tmp);
+
+        return callback(null, tmp);
+    });
+}
+
+function upsert(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '';
+
+    debug('upsert: %s for zone %s of type %s with values %j', name, zoneName, type, values);
+
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+        if (error) return callback(error);
+
+        let i = 0, recordIds = []; // used to track available records to update instead of create
+
+        async.eachSeries(values, function (value, iteratorCallback) {
+            let data = {
+                type: type,
+                ttl_sec: 300 // lowest
+            };
+
+            if (type === 'MX') {
+                data.priority = parseInt(value.split(' ')[0], 10);
+                data.target = value.split(' ')[1];
+            } else if (type === 'TXT') {
+                data.target = value.replace(/^"(.*)"$/, '$1'); // strip any double quotes
+            } else {
+                data.target = value;
+            }
+
+            if (i >= records.length) {
+                data.name = name; // only set for new records
+
+                superagent.post(`${LINODE_ENDPOINT}/domains/${zoneId}/records`)
+                    .set('Authorization', 'Bearer ' + dnsConfig.token)
+                    .send(data)
+                    .timeout(30 * 1000)
+                    .retry(5)
+                    .end(function (error, result) {
+                        if (error && !error.response) return iteratorCallback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                        if (result.statusCode === 400) return iteratorCallback(new BoxError(BoxError.BAD_FIELD, formatError(result)));
+                        if (result.statusCode === 403 || result.statusCode === 401) return iteratorCallback(new BoxError(BoxError.ACCESS_DENIED, formatError(result)));
+                        if (result.statusCode !== 200) return iteratorCallback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                        recordIds.push(result.body.id);
+
+                        return iteratorCallback(null);
+                    });
+            } else {
+                superagent.put(`${LINODE_ENDPOINT}/domains/${zoneId}/records/${records[i].id}`)
+                    .set('Authorization', 'Bearer ' + dnsConfig.token)
+                    .send(data)
+                    .timeout(30 * 1000)
+                    .retry(5)
+                    .end(function (error, result) {
+                        // increment, as we have consumed the record
+                        ++i;
+
+                        if (error && !error.response) return iteratorCallback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                        if (result.statusCode === 400) return iteratorCallback(new BoxError(BoxError.BAD_FIELD, formatError(result)));
+                        if (result.statusCode === 403 || result.statusCode === 401) return iteratorCallback(new BoxError(BoxError.ACCESS_DENIED, formatError(result)));
+                        if (result.statusCode !== 200) return iteratorCallback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                        recordIds.push(result.body.id);
+
+                        return iteratorCallback(null);
+                    });
+            }
+        }, function (error) {
+            if (error) return callback(error);
+
+            debug('upsert: completed with recordIds:%j', recordIds);
+
+            callback();
+        });
+    });
+}
+
+function del(domainObject, location, type, values, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName,
+        name = domains.getName(domainObject, location, type) || '';
+
+    getZoneRecords(dnsConfig, zoneName, name, type, function (error, { zoneId, records }) {
+        if (error) return callback(error);
+
+        if (records.length === 0) return callback(null);
+
+        var tmp = records.filter(function (record) { return values.some(function (value) { return value === record.target; }); });
+
+        debug('del: %j', tmp);
+
+        if (tmp.length === 0) return callback(null);
+
+        // FIXME we only handle the first one currently
+
+        superagent.del(`${LINODE_ENDPOINT}/domains/${zoneId}/records/${tmp[0].id}`)
+            .set('Authorization', 'Bearer ' + dnsConfig.token)
+            .timeout(30 * 1000)
+            .retry(5)
+            .end(function (error, result) {
+                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+                if (result.statusCode === 404) return callback(null);
+                if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.ACCESS_DENIED, formatError(result)));
+                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, formatError(result)));
+
+                debug('del: done');
+
+                return callback(null);
+            });
+    });
+}
+
+function wait(domainObject, location, type, value, options, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof value, 'string');
+    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
+    assert.strictEqual(typeof callback, 'function');
+
+    const fqdn = domains.fqdn(location, domainObject);
+
+    waitForDns(fqdn, domainObject.zoneName, type, value, options, callback);
+}
+
+function verifyDnsConfig(domainObject, callback) {
+    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const dnsConfig = domainObject.config,
+        zoneName = domainObject.zoneName;
+
+    if (!dnsConfig.token || typeof dnsConfig.token !== 'string') return callback(new BoxError(BoxError.BAD_FIELD, 'token must be a non-empty string', { field: 'token' }));
+
+    const ip = '127.0.0.1';
+
+    var credentials = {
+        token: dnsConfig.token
+    };
+
+    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here
+
+    dns.resolve(zoneName, 'NS', { timeout: 5000 }, function (error, nameservers) {
+        if (error && error.code === 'ENOTFOUND') return callback(new BoxError(BoxError.BAD_FIELD, 'Unable to resolve nameservers for this domain', { field: 'nameservers' }));
+        if (error || !nameservers) return callback(new BoxError(BoxError.BAD_FIELD, error ? error.message : 'Unable to get nameservers', { field: 'nameservers' }));
+
+        if (nameservers.map(function (n) { return n.toLowerCase(); }).indexOf('ns1.linode.com') === -1) {
+            debug('verifyDnsConfig: %j does not contains DO NS', nameservers);
+            return callback(new BoxError(BoxError.BAD_FIELD, 'Domain nameservers are not set to Linode', { field: 'nameservers' }));
+        }
+
+        const location = 'cloudrontestdns';
+
+        upsert(domainObject, location, 'A', [ ip ], function (error) {
+            if (error) return callback(error);
+
+            debug('verifyDnsConfig: Test A record added');
+
+            del(domainObject, location, 'A', [ ip ], function (error) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record removed again');
+
+                callback(null, credentials);
+            });
+        });
+    });
+}

src/dns/namecheap.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/namecheap'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -25,12 +26,12 @@ var assert = require('assert'),
 const ENDPOINT = 'https://api.namecheap.com/xml.response';
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function getQuery(dnsConfig, callback) {

src/dns/namecom.js

@@ -12,6 +12,7 @@ exports = module.exports = {
 
 var assert = require('assert'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/namecom'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -27,12 +28,12 @@ function formatError(response) {
 }
 
 function removePrivateFields(domainObject) {
-    domainObject.config.token = domains.SECRET_PLACEHOLDER;
+    domainObject.config.token = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.token === domains.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
+    if (newConfig.token === constants.SECRET_PLACEHOLDER) newConfig.token = currentConfig.token;
 }
 
 function addRecord(dnsConfig, zoneName, name, type, values, callback) {
@@ -54,6 +55,10 @@ function addRecord(dnsConfig, zoneName, name, type, values, callback) {
     if (type === 'MX') {
         data.priority = parseInt(values[0].split(' ')[0], 10);
         data.answer = values[0].split(' ')[1];
+    } else if (type === 'TXT') {
+        // we have to strip the quoting for some odd reason for name.com! If you change that also change updateRecord
+        let tmp = values[0];
+        data.answer = tmp.indexOf('"') === 0 && tmp.lastIndexOf('"') === tmp.length-1 ? tmp.slice(1, tmp.length-1) : tmp;
     } else {
         data.answer = values[0];
     }
@@ -91,6 +96,10 @@ function updateRecord(dnsConfig, zoneName, recordId, name, type, values, callbac
     if (type === 'MX') {
         data.priority = parseInt(values[0].split(' ')[0], 10);
         data.answer = values[0].split(' ')[1];
+    } else if (type === 'TXT') {
+        // we have to strip the quoting for some odd reason for name.com! If you change that also change addRecord
+        let tmp = values[0];
+        data.answer = tmp.indexOf('"') === 0 && tmp.lastIndexOf('"') === tmp.length-1 ? tmp.slice(1, tmp.length-1) : tmp;
     } else {
         data.answer = values[0];
     }

src/dns/route53.js

@@ -13,6 +13,7 @@ exports = module.exports = {
 var assert = require('assert'),
     AWS = require('aws-sdk'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:dns/route53'),
     dns = require('../native-dns.js'),
     domains = require('../domains.js'),
@@ -21,12 +22,12 @@ var assert = require('assert'),
     _ = require('underscore');
 
 function removePrivateFields(domainObject) {
-    domainObject.config.secretAccessKey = domains.SECRET_PLACEHOLDER;
+    domainObject.config.secretAccessKey = constants.SECRET_PLACEHOLDER;
     return domainObject;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.secretAccessKey === domains.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
+    if (newConfig.secretAccessKey === constants.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
 }
 
 function getDnsCredentials(dnsConfig) {

src/docker.js

@@ -6,8 +6,6 @@ exports = module.exports = {
     injectPrivateFields: injectPrivateFields,
     removePrivateFields: removePrivateFields,
 
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8),
-
     ping: ping,
 
     info: info,
@@ -55,12 +53,6 @@ const CLEARVOLUME_CMD = path.join(__dirname, 'scripts/clearvolume.sh'),
 const DOCKER_SOCKET_PATH = '/var/run/docker.sock';
 const gConnection = new Docker({ socketPath: DOCKER_SOCKET_PATH });
 
-function debugApp(app) {
-    assert(typeof app === 'object');
-
-    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
-}
-
 function testRegistryConfig(auth, callback) {
     assert.strictEqual(typeof auth, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -73,13 +65,13 @@ function testRegistryConfig(auth, callback) {
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.password === exports.SECRET_PLACEHOLDER) newConfig.password = currentConfig.password;
+    if (newConfig.password === constants.SECRET_PLACEHOLDER) newConfig.password = currentConfig.password;
 }
 
 function removePrivateFields(registryConfig) {
     assert.strictEqual(typeof registryConfig, 'object');
 
-    if (registryConfig.password) registryConfig.password = exports.SECRET_PLACEHOLDER;
+    if (registryConfig.password) registryConfig.password = constants.SECRET_PLACEHOLDER;
 
     return registryConfig;
 }
@@ -188,6 +180,19 @@ function downloadImage(manifest, callback) {
     }, callback);
 }
 
+function getBindsSync(app) {
+    assert.strictEqual(typeof app, 'object');
+
+    let binds = [];
+
+    for (let name of Object.keys(app.binds)) {
+        const bind = app.binds[name];
+        binds.push(`${bind.hostPath}:/media/${name}:${bind.readOnly ? 'ro' : 'rw'}`);
+    }
+
+    return binds;
+}
+
 function createSubcontainer(app, name, cmd, options, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof name, 'string');
@@ -277,6 +282,7 @@ function createSubcontainer(app, name, cmd, options, callback) {
             },
             HostConfig: {
                 Mounts: addons.getMountsSync(app, app.manifest.addons),
+                Binds: getBindsSync(app), // ideally, we have to use 'Mounts' but we have to create volumes then
                 LogConfig: {
                     Type: 'syslog',
                     Config: {
@@ -299,7 +305,8 @@ function createSubcontainer(app, name, cmd, options, callback) {
                 NetworkMode: 'cloudron', // user defined bridge network
                 Dns: ['172.18.0.1'], // use internal dns
                 DnsSearch: ['.'], // use internal dns
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ]
+                SecurityOpt: [ 'apparmor=docker-cloudron-app' ],
+                CapDrop: [ 'NET_RAW' ] // https://docs-stage.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
             },
             NetworkingConfig: {
                 EndpointsConfig: {
@@ -313,14 +320,12 @@ function createSubcontainer(app, name, cmd, options, callback) {
         var capabilities = manifest.capabilities || [];
         if (capabilities.includes('net_admin')) {
             containerOptions.HostConfig.CapAdd = [
-                'NET_ADMIN'
+                'NET_ADMIN', 'NET_RAW'
             ];
         }
 
         containerOptions = _.extend(containerOptions, options);
 
-        debugApp(app, 'Creating container for %s', app.manifest.dockerImage);
-
         gConnection.createContainer(containerOptions, function (error, container) {
             if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
 
@@ -338,7 +343,6 @@ function startContainer(containerId, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     var container = gConnection.getContainer(containerId);
-    debug('Starting container %s', containerId);
 
     container.start(function (error) {
         if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
@@ -354,7 +358,6 @@ function restartContainer(containerId, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     var container = gConnection.getContainer(containerId);
-    debug('Restarting container %s', containerId);
 
     container.restart(function (error) {
         if (error && error.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
@@ -375,7 +378,6 @@ function stopContainer(containerId, callback) {
     }
 
     var container = gConnection.getContainer(containerId);
-    debug('Stopping container %s', containerId);
 
     var options = {
         t: 10 // wait for 10 seconds before killing it
@@ -384,13 +386,9 @@ function stopContainer(containerId, callback) {
     container.stop(options, function (error) {
         if (error && (error.statusCode !== 304 && error.statusCode !== 404)) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error stopping container:' + error.message));
 
-        debug('Waiting for container ' + containerId);
-
-        container.wait(function (error, data) {
+        container.wait(function (error/*, data */) {
             if (error && (error.statusCode !== 304 && error.statusCode !== 404)) return callback(new BoxError(BoxError.DOCKER_ERROR, 'Error waiting on container:' + error.message));
 
-            debug('Container %s stopped with status code [%s]', containerId, data ? String(data.StatusCode) : '');
-
             return callback(null);
         });
     });
@@ -400,8 +398,6 @@ function deleteContainer(containerId, callback) {
     assert(!containerId || typeof containerId === 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('deleting container %s', containerId);
-
     if (containerId === null) return callback(null);
 
     var container = gConnection.getContainer(containerId);
@@ -428,8 +424,6 @@ function deleteContainers(appId, options, callback) {
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('deleting containers of %s', appId);
-
     let labels = [ 'appId=' + appId ];
     if (options.managedOnly) labels.push('isCloudronManaged=true');
 
@@ -446,8 +440,6 @@ function stopContainers(appId, callback) {
     assert.strictEqual(typeof appId, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    debug('Stopping containers of %s', appId);
-
     gConnection.listContainers({ all: 1, filters: JSON.stringify({ label: [ 'appId=' + appId ] }) }, function (error, containers) {
         if (error) return callback(new BoxError(BoxError.DOCKER_ERROR, error));
 
@@ -573,10 +565,10 @@ function memoryUsage(containerId, callback) {
     });
 }
 
-function createVolume(app, name, volumeDataDir, callback) {
-    assert.strictEqual(typeof app, 'object');
+function createVolume(name, volumeDataDir, labels, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof volumeDataDir, 'string');
+    assert.strictEqual(typeof labels, 'object');
     assert.strictEqual(typeof callback, 'function');
 
     const volumeOptions = {
@@ -587,10 +579,7 @@ function createVolume(app, name, volumeDataDir, callback) {
             device: volumeDataDir,
             o: 'bind'
         },
-        Labels: {
-            'fqdn': app.fqdn,
-            'appId': app.id
-        },
+        Labels: labels
     };
 
     // requires sudo because the path can be outside appsdata
@@ -605,8 +594,7 @@ function createVolume(app, name, volumeDataDir, callback) {
     });
 }
 
-function clearVolume(app, name, options, callback) {
-    assert.strictEqual(typeof app, 'object');
+function clearVolume(name, options, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof options, 'object');
     assert.strictEqual(typeof callback, 'function');
@@ -626,14 +614,13 @@ function clearVolume(app, name, options, callback) {
 }
 
 // this only removes the volume and not the data
-function removeVolume(app, name, callback) {
-    assert.strictEqual(typeof app, 'object');
+function removeVolume(name, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof callback, 'function');
 
     let volume = gConnection.getVolume(name);
     volume.remove(function (error) {
-        if (error && error.statusCode !== 404) return callback(new BoxError(BoxError.DOCKER_ERROR, `removeVolume: Error removing volume of ${app.id} ${error.message}`));
+        if (error && error.statusCode !== 404) return callback(new BoxError(BoxError.DOCKER_ERROR, `removeVolume: Error removing volume: ${error.message}`));
 
         callback();
     });

src/dockerproxy.js

@@ -23,11 +23,6 @@ var apps = require('./apps.js'),
 var gHttpServer = null;
 
 function authorizeApp(req, res, next) {
-    // TODO add here some authorization
-    // - block apps not using the docker addon
-    // - block calls regarding platform containers
-    // - only allow managing and inspection of containers belonging to the app
-
     // make the tests pass for now
     if (constants.TEST) {
         req.app = { id: 'testappid' };
@@ -64,6 +59,8 @@ function attachDockerRequest(req, res, next) {
         dockerResponse.pipe(res, { end: true });
     });
 
+    req.dockerRequest.on('error', () => {}); // abort() throws
+
     next();
 }
 
@@ -74,22 +71,21 @@ function containersCreate(req, res, next) {
     safe.set(req.body, 'Labels',  _.extend({ }, safe.query(req.body, 'Labels'), { appId: req.app.id, isCloudronManaged: String(false) }));    // overwrite the app id to track containers of an app
     safe.set(req.body, 'HostConfig.LogConfig', { Type: 'syslog', Config: { 'tag': req.app.id, 'syslog-address': 'udp://127.0.0.1:2514', 'syslog-format': 'rfc5424' }});
 
-    const appDataDir = path.join(paths.APPS_DATA_DIR, req.app.id, 'data'),
-        dockerDataDir = path.join(paths.APPS_DATA_DIR, req.app.id, 'docker');
+    const appDataDir = path.join(paths.APPS_DATA_DIR, req.app.id, 'data');
 
-    debug('Original volume binds:', req.body.HostConfig.Binds);
+    debug('Original bind mounts:', req.body.HostConfig.Binds);
 
     let binds = [];
     for (let bind of (req.body.HostConfig.Binds || [])) {
-        if (bind.startsWith(appDataDir)) binds.push(bind); // eclipse will inspect docker to find out the host folders and pass that to child containers
-        else if (bind.startsWith('/app/data')) binds.push(bind.replace(new RegExp('^/app/data'), appDataDir));
-        else binds.push(`${dockerDataDir}/${bind}`);
+        if (!bind.startsWith('/app/data/')) {
+            req.dockerRequest.abort();
+            return next(new HttpError(400, 'Binds must be under /app/data/'));
+        }
+
+        binds.push(bind.replace(new RegExp('^/app/data/'), appDataDir + '/'));
     }
 
-    // cleanup the paths from potential double slashes
-    binds = binds.map(function (bind) { return bind.replace(/\/+/g, '/'); });
-
-    debug('Rewritten volume binds:', binds);
+    debug('Rewritten bind mounts:', binds);
     safe.set(req.body, 'HostConfig.Binds', binds);
 
     let plainBody = JSON.stringify(req.body);
@@ -117,6 +113,9 @@ function start(callback) {
     assert(gHttpServer === null, 'Already started');
 
     let json = middleware.json({ strict: true });
+
+    // we protect container create as the app/admin can otherwise mount random paths (like the ghost file)
+    // protected other paths is done by preventing install/exec access of apps using docker addon
     let router = new express.Router();
     router.post('/:version/containers/create', containersCreate);
 
@@ -137,7 +136,7 @@ function start(callback) {
         .use(middleware.lastMile());
 
     gHttpServer = http.createServer(proxyServer);
-    gHttpServer.listen(constants.DOCKER_PROXY_PORT, '0.0.0.0', callback);
+    gHttpServer.listen(constants.DOCKER_PROXY_PORT, '172.18.0.1', callback);
 
     // Overwrite the default 2min request timeout. This is required for large builds for example
     gHttpServer.setTimeout(60 * 60 * 1000);

src/domaindb.js

@@ -51,16 +51,21 @@ function getAll(callback) {
     });
 }
 
-function add(name, domain, callback) {
+function add(name, data, callback) {
     assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof domain, 'object');
-    assert.strictEqual(typeof domain.zoneName, 'string');
-    assert.strictEqual(typeof domain.provider, 'string');
-    assert.strictEqual(typeof domain.config, 'object');
-    assert.strictEqual(typeof domain.tlsConfig, 'object');
+    assert.strictEqual(typeof data, 'object');
+    assert.strictEqual(typeof data.zoneName, 'string');
+    assert.strictEqual(typeof data.provider, 'string');
+    assert.strictEqual(typeof data.config, 'object');
+    assert.strictEqual(typeof data.tlsConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO domains (domain, zoneName, provider, configJson, tlsConfigJson) VALUES (?, ?, ?, ?, ?)', [ name, domain.zoneName, domain.provider, JSON.stringify(domain.config), JSON.stringify(domain.tlsConfig) ], function (error) {
+    let queries = [
+        { query: 'INSERT INTO domains (domain, zoneName, provider, configJson, tlsConfigJson) VALUES (?, ?, ?, ?, ?)', args: [ name, data.zoneName, data.provider, JSON.stringify(data.config), JSON.stringify(data.tlsConfig) ] },
+        { query: 'INSERT INTO mail (domain, dkimSelector) VALUES (?, ?)', args: [ name, data.dkimSelector || 'cloudron' ] },
+    ];
+
+    database.transaction(queries, function (error) {
         if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error));
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -100,7 +105,12 @@ function del(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('DELETE FROM domains WHERE domain=?', [ domain ], function (error, result) {
+    let queries = [
+        { query: 'DELETE FROM mail WHERE domain = ?', args: [ domain ] },
+        { query: 'DELETE FROM domains WHERE domain = ?', args: [ domain ] },
+    ];
+
+    database.transaction(queries, function (error, results) {
         if (error && error.code === 'ER_ROW_IS_REFERENCED_2') {
             if (error.message.indexOf('apps_mailDomain_constraint') !== -1) return callback(new BoxError(BoxError.CONFLICT, 'Domain is in use by an app or the mailbox of an app. Check the domains of apps and the Email section of each app.'));
             if (error.message.indexOf('subdomains') !== -1) return callback(new BoxError(BoxError.CONFLICT, 'Domain is in use by one or more app(s).'));
@@ -108,8 +118,9 @@ function del(domain, callback) {
 
             return callback(new BoxError(BoxError.CONFLICT, error.message));
         }
+
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Domain not found'));
+        if (results[1].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'Domain not found'));
 
         callback(null);
     });

src/domains.js

@@ -28,9 +28,7 @@ module.exports = exports = {
 
     checkDnsRecords: checkDnsRecords,
 
-    prepareDashboardDomain: prepareDashboardDomain,
-
-    SECRET_PLACEHOLDER: String.fromCharCode(0x25CF).repeat(8)
+    prepareDashboardDomain: prepareDashboardDomain
 };
 
 var assert = require('assert'),
@@ -40,6 +38,7 @@ var assert = require('assert'),
     debug = require('debug')('box:domains'),
     domaindb = require('./domaindb.js'),
     eventlog = require('./eventlog.js'),
+    mail = require('./mail.js'),
     reverseProxy = require('./reverseproxy.js'),
     safe = require('safetydance'),
     settings = require('./settings.js'),
@@ -48,6 +47,8 @@ var assert = require('assert'),
     util = require('util'),
     _ = require('underscore');
 
+const NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
 // choose which subdomain backend we use for test purpose we use route53
 function api(provider) {
     assert.strictEqual(typeof provider, 'string');
@@ -60,6 +61,7 @@ function api(provider) {
     case 'digitalocean': return require('./dns/digitalocean.js');
     case 'gandi': return require('./dns/gandi.js');
     case 'godaddy': return require('./dns/godaddy.js');
+    case 'linode': return require('./dns/linode.js');
     case 'namecom': return require('./dns/namecom.js');
     case 'namecheap': return require('./dns/namecheap.js');
     case 'noop': return require('./dns/noop.js');
@@ -170,7 +172,7 @@ function add(domain, data, auditSource, callback) {
     assert.strictEqual(typeof data.tlsConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
+    let { zoneName, provider, config, fallbackCertificate, tlsConfig, dkimSelector } = data;
 
     if (!tld.isValid(domain)) return callback(new BoxError(BoxError.BAD_FIELD, 'Invalid domain', { field: 'domain' }));
     if (domain.endsWith('.')) return callback(new BoxError(BoxError.BAD_FIELD, 'Invalid domain', { field: 'domain' }));
@@ -193,10 +195,12 @@ function add(domain, data, auditSource, callback) {
     let error = validateTlsConfig(tlsConfig, provider);
     if (error) return callback(error);
 
+    if (!dkimSelector) dkimSelector = 'cloudron-' + settings.adminDomain().replace(/\./g, '');
+
     verifyDnsConfig(config, domain, zoneName, provider, function (error, sanitizedConfig) {
         if (error) return callback(error);
 
-        domaindb.add(domain, { zoneName: zoneName, provider: provider, config: sanitizedConfig, tlsConfig: tlsConfig }, function (error) {
+        domaindb.add(domain, { zoneName, provider, config: sanitizedConfig, tlsConfig, dkimSelector }, function (error) {
             if (error) return callback(error);
 
             reverseProxy.setFallbackCertificate(domain, fallbackCertificate, function (error) {
@@ -204,6 +208,8 @@ function add(domain, data, auditSource, callback) {
 
                 eventlog.add(eventlog.ACTION_DOMAIN_ADD, auditSource, { domain, zoneName, provider });
 
+                mail.onDomainAdded(domain, NOOP_CALLBACK);
+
                 callback();
             });
         });
@@ -313,6 +319,8 @@ function del(domain, auditSource, callback) {
 
         eventlog.add(eventlog.ACTION_DOMAIN_REMOVE, auditSource, { domain });
 
+        mail.onDomainRemoved(domain, NOOP_CALLBACK);
+
         return callback(null);
     });
 }
@@ -436,6 +444,9 @@ function waitForDnsRecord(location, domain, type, value, options, callback) {
     get(domain, function (error, domainObject) {
         if (error) return callback(error);
 
+        // linode DNS takes ~15mins
+        if (!options.interval) options.interval = domainObject.provider === 'linode' ? 20000 : 5000;
+
         api(domainObject.provider).wait(domainObject, location, type, value, options, callback);
     });
 }

src/eventlog.js

@@ -7,7 +7,7 @@ exports = module.exports = {
     getByCreationTime: getByCreationTime,
     cleanup: cleanup,
 
-    // keep in sync with webadmin index.js filter and CLI tool
+    // keep in sync with webadmin index.js filter
     ACTION_ACTIVATE: 'cloudron.activate',
     ACTION_APP_CLONE: 'app.clone',
     ACTION_APP_CONFIGURE: 'app.configure',
@@ -21,6 +21,9 @@ exports = module.exports = {
     ACTION_APP_OOM: 'app.oom',
     ACTION_APP_UP: 'app.up',
     ACTION_APP_DOWN: 'app.down',
+    ACTION_APP_START: 'app.start',
+    ACTION_APP_STOP: 'app.stop',
+    ACTION_APP_RESTART: 'app.restart',
 
     ACTION_BACKUP_FINISH: 'backup.finish',
     ACTION_BACKUP_START: 'backup.start',

src/infra_version.js

@@ -9,18 +9,19 @@ exports = module.exports = {
     'version': '48.17.0',
 
     'baseImages': [
-        { repo: 'cloudron/base', tag: 'cloudron/base:1.0.0@sha256:147a648a068a2e746644746bbfb42eb7a50d682437cead3c67c933c546357617' }
+        { repo: 'cloudron/base', tag: 'cloudron/base:2.0.0@sha256:f9fea80513aa7c92fe2e7bf3978b54c8ac5222f47a9a32a7f8833edf0eb5a4f4' }
     ],
 
     // a major version bump in the db containers will trigger the restore logic that uses the db dumps
     // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
     'images': {
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.1.0@sha256:eee0dfd3829d563f2063084bc0d7c8802c4bdd6e233159c6226a17ff7a9a3503' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:2.0.2@sha256:6dcee0731dfb9b013ed94d56205eee219040ee806c7e251db3b3886eaa4947ff' },
-        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:2.1.0@sha256:6d1bf221cfe6124957e2c58b57c0a47214353496009296acb16adf56df1da9d5' },
-        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.1.0@sha256:f2cda21bd15c21bbf44432df412525369ef831a2d53860b5c5b1675e6f384de2' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.6.1@sha256:6ff6028c050d0ca34f6dd17395f18c820281ef262e77b01b82d5a74d5b8f6842' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.2.0@sha256:fc9ca69d16e6ebdbd98ed53143d4a0d2212eef60cb638dc71219234e6f427a2c' },
-        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:0.1.0@sha256:e177c5bf5f38c84ce1dea35649c22a1b05f96eec67a54a812c5a35e585670f0f' }
+        'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.1.0@sha256:e1dd22aa6eef5beb7339834b200a8bb787ffc2264ce11139857a054108fefb4f' },
+        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.0@sha256:0cc39004b80eb5ee570b9fd4f38859410a49692e93e0f0d8b104b0b524d7030a' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:2.1.1@sha256:6ea1bcf9b655d628230acda01d46cf21883b112111d89583c94138646895c14c' },
+        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:2.2.0@sha256:205486ff0f6bf6854610572df401cf3651bc62baf28fd26e9c5632497f10c2cb' },
+        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.2.0@sha256:cfdcc1a54cf29818cac99eacedc2ecf04e44995be3d06beea11dcaa09d90ed8d' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.9.3@sha256:2c062e8883cadae4b4c22553f9db141fc441d6c627aa85e2a5ad71e3fcbf2b42' },
+        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.3.0@sha256:b7bc1ca4f4d0603a01369a689129aa273a938ce195fe43d00d42f4f2d5212f50' },
+        'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:1.1.0@sha256:0c1fe4dd6121900624dcb383251ecb0084c3810e095064933de671409d8d6d7b' }
     }
 };

src/ldap.js

@@ -154,7 +154,6 @@ function userSearch(req, res, next) {
                     givenName: firstName,
                     username: user.username,
                     samaccountname: user.username,      // to support ActiveDirectory clients
-                    isadmin: users.compareRoles(user.role, users.ROLE_ADMIN) >= 0,
                     memberof: groups
                 }
             };
@@ -392,7 +391,7 @@ function mailAliasSearch(req, res, next) {
                 objectclass: ['nisMailAlias'],
                 objectcategory: 'nisMailAlias',
                 cn: `${alias.name}@${alias.domain}`,
-                rfc822MailMember: `${alias.aliasTarget}@${alias.domain}`
+                rfc822MailMember: `${alias.aliasName}@${alias.aliasDomain}`
             }
         };
 
@@ -418,7 +417,7 @@ function mailingListSearch(req, res, next) {
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
     const name = parts[0], domain = parts[1];
 
-    mail.resolveList(parts[0], parts[1], function (error, resolvedMembers) {
+    mail.resolveList(parts[0], parts[1], function (error, resolvedMembers, list) {
         if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
         if (error) return next(new ldap.OperationsError(error.toString()));
 
@@ -431,6 +430,7 @@ function mailingListSearch(req, res, next) {
                 objectcategory: 'mailGroup',
                 cn: `${name}@${domain}`, // fully qualified
                 mail: `${name}@${domain}`,
+                membersOnly: list.membersOnly, // ldapjs only supports strings and string array. so this is not a bool!
                 mgrpRFC822MailMember: resolvedMembers // fully qualified
             }
         };
@@ -534,13 +534,16 @@ function authenticateSftp(req, res, next) {
     var parts = email.split('@');
     if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
 
-    // actual user bind
-    users.verifyWithUsername(parts[0], req.credentials, users.AP_SFTP, function (error) {
+    apps.getByFqdn(parts[1], function (error, app) {
         if (error) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
 
-        debug('sftp auth: success');
+        users.verifyWithUsername(parts[0], req.credentials, app.id, function (error) {
+            if (error) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
 
-        res.end();
+            debug('sftp auth: success');
+
+            res.end();
+        });
     });
 }
 
@@ -615,10 +618,7 @@ function authenticateMailAddon(req, res, next) {
         // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
         appdb.getAppIdByAddonConfigValue(addonId, namePattern, req.credentials || '', function (error, appId) {
             if (error && error.reason !== BoxError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) { // matched app password
-                eventlog.add(eventlog.ACTION_APP_LOGIN, { authType: 'ldap', mailboxId: email }, { appId: appId, addonId: addonId });
-                return res.end();
-            }
+            if (appId) return res.end();
 
             mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
                 if (error && error.reason === BoxError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));

src/logrotate.ejs

@@ -8,6 +8,7 @@
     maxsize 1M
     missingok
     delaycompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 
@@ -18,6 +19,7 @@
     missingok
     # we never compress so we can simply tail the files
     nocompress
+    # this truncates the original log file and not the rotated one
     copytruncate
 }
 

src/mail.js

@@ -7,10 +7,11 @@ exports = module.exports = {
     getDomains: getDomains,
 
     getDomain: getDomain,
-    addDomain: addDomain,
-    removeDomain: removeDomain,
     clearDomains: clearDomains,
 
+    onDomainAdded: onDomainAdded,
+    onDomainRemoved: onDomainRemoved,
+
     removePrivateFields: removePrivateFields,
 
     setDnsRecords: setDnsRecords,
@@ -37,7 +38,6 @@ exports = module.exports = {
     updateMailboxOwner: updateMailboxOwner,
     removeMailbox: removeMailbox,
 
-    listAliases: listAliases,
     getAliases: getAliases,
     setAliases: setAliases,
 
@@ -101,7 +101,6 @@ function checkOutboundPort25(callback) {
         'smtp.gmail.com',
         'smtp.live.com',
         'smtp.mail.yahoo.com',
-        'smtp.comcast.net',
         'smtp.1und1.de',
     ]);
 
@@ -208,7 +207,8 @@ function checkDkim(mailDomain, callback) {
 
         if (txtRecords.length !== 0) {
             dkim.value = txtRecords[0].join('');
-            dkim.status = (dkim.value === dkim.expected);
+            const actual = txtToDict(dkim.value);
+            dkim.status = actual.p === dkimKey;
         }
 
         callback(null, dkim);
@@ -269,7 +269,7 @@ function checkMx(domain, mailFqdn, callback) {
         if (error) return callback(error, mx);
         if (mxRecords.length === 0) return callback(null, mx);
 
-        mx.status = mxRecords.length == 1 && mxRecords[0].exchange === mailFqdn;
+        mx.status = mxRecords.some(mx => mx.exchange === mailFqdn); // this lets use change priority and/or setup backup MX
         mx.value = mxRecords.map(function (r) { return r.priority + ' ' + r.exchange + '.'; }).join(' ');
 
         if (mx.status) return callback(null, mx); // MX record is "my."
@@ -313,9 +313,8 @@ function checkDmarc(domain, callback) {
 
         if (txtRecords.length !== 0) {
             dmarc.value = txtRecords[0].join('');
-            // allow extra fields in dmarc like rua
-            const actual = txtToDict(dmarc.value), expected = txtToDict(dmarc.expected);
-            dmarc.status = Object.keys(expected).every(k => expected[k] === actual[k]);
+            const actual = txtToDict(dmarc.value);
+            dmarc.status = actual.v === 'DMARC1'; // see box#666
         }
 
         callback(null, dmarc);
@@ -799,6 +798,7 @@ function ensureDkimKeySync(mailDomain) {
         return new BoxError(BoxError.FS_ERROR, safe.error);
     }
 
+    // https://www.unlocktheinbox.com/dkim-key-length-statistics/ and https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-authentication-dkim-easy.html for key size
     if (!safe.child_process.execSync('openssl genrsa -out ' + dkimPrivateKeyFile + ' 1024')) return new BoxError(BoxError.OPENSSL_ERROR, safe.error);
     if (!safe.child_process.execSync('openssl rsa -in ' + dkimPrivateKeyFile + ' -out ' + dkimPublicKeyFile + ' -pubout -outform PEM')) return new BoxError(BoxError.OPENSSL_ERROR, safe.error);
 
@@ -906,37 +906,21 @@ function onMailFqdnChanged(callback) {
     });
 }
 
-function addDomain(domain, callback) {
+function onDomainAdded(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    const dkimSelector = domain === settings.adminDomain() ? 'cloudron' : ('cloudron-' + settings.adminDomain().replace(/\./g, ''));
-
-    maildb.add(domain, { dkimSelector }, function (error) {
-        if (error) return callback(error);
-
-        async.series([
-            upsertDnsRecords.bind(null, domain, settings.mailFqdn()), // do this first to ensure DKIM keys
-            restartMailIfActivated
-        ], NOOP_CALLBACK); // do these asynchronously
-
-        callback();
-    });
+    async.series([
+        upsertDnsRecords.bind(null, domain, settings.mailFqdn()), // do this first to ensure DKIM keys
+        restartMailIfActivated
+    ], callback);
 }
 
-function removeDomain(domain, callback) {
+function onDomainRemoved(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    if (domain === settings.adminDomain()) return callback(new BoxError(BoxError.CONFLICT));
-
-    maildb.del(domain, function (error) {
-        if (error) return callback(error);
-
-        restartMail(NOOP_CALLBACK);
-
-        callback();
-    });
+    restartMail(callback);
 }
 
 function clearDomains(callback) {
@@ -1142,19 +1126,6 @@ function removeMailbox(name, domain, auditSource, callback) {
     });
 }
 
-function listAliases(domain, page, perPage, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof page, 'number');
-    assert.strictEqual(typeof perPage, 'number');
-    assert.strictEqual(typeof callback, 'function');
-
-    mailboxdb.listAliases(domain, page, perPage, function (error, result) {
-        if (error) return callback(error);
-
-        callback(null, result);
-    });
-}
-
 function getAliases(name, domain, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
@@ -1178,12 +1149,15 @@ function setAliases(name, domain, aliases, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     for (var i = 0; i < aliases.length; i++) {
-        aliases[i] = aliases[i].toLowerCase();
+        let name = aliases[i].name.toLowerCase();
+        let domain = aliases[i].domain.toLowerCase();
 
-        var error = validateName(aliases[i]);
+        let error = validateName(name);
         if (error) return callback(error);
-    }
 
+        if (!validator.isEmail(`${name}@${domain}`)) return callback(new BoxError(BoxError.BAD_FIELD, `Invalid email: ${name}@${domain}`));
+        aliases[i] = { name, domain };
+    }
     mailboxdb.setAliasesForName(name, domain, aliases, function (error) {
         if (error) return callback(error);
 
@@ -1214,10 +1188,11 @@ function getList(name, domain, callback) {
     });
 }
 
-function addList(name, domain, members, auditSource, callback) {
+function addList(name, domain, members, membersOnly, auditSource, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof name, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1230,19 +1205,20 @@ function addList(name, domain, members, auditSource, callback) {
         if (!validator.isEmail(members[i])) return callback(new BoxError(BoxError.BAD_FIELD, 'Invalid mail member: ' + members[i]));
     }
 
-    mailboxdb.addList(name, domain, members, function (error) {
+    mailboxdb.addList(name, domain, members, membersOnly, function (error) {
         if (error) return callback(error);
 
-        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain, members });
+        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain, members, membersOnly });
 
         callback();
     });
 }
 
-function updateList(name, domain, members, auditSource, callback) {
+function updateList(name, domain, members, membersOnly, auditSource, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof auditSource, 'object');
     assert.strictEqual(typeof callback, 'function');
 
@@ -1258,10 +1234,10 @@ function updateList(name, domain, members, auditSource, callback) {
     getList(name, domain, function (error, result) {
         if (error) return callback(error);
 
-        mailboxdb.updateList(name, domain, members, function (error) {
+        mailboxdb.updateList(name, domain, members, membersOnly, function (error) {
             if (error) return callback(error);
 
-            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldMembers: result.members, members });
+            eventlog.add(eventlog.ACTION_MAIL_MAILBOX_UPDATE, auditSource, { name, domain, oldMembers: result.members, members, membersOnly });
 
             callback(null);
         });
@@ -1283,6 +1259,7 @@ function removeList(name, domain, auditSource, callback) {
     });
 }
 
+// resolves the members of a list. i.e the lists and aliases
 function resolveList(listName, listDomain, callback) {
     assert.strictEqual(typeof listName, 'string');
     assert.strictEqual(typeof listDomain, 'string');
@@ -1313,18 +1290,21 @@ function resolveList(listName, listDomain, callback) {
                 visited.push(member);
 
                 mailboxdb.get(memberName, memberDomain, function (error, entry) {
-                    if (error && error.reason == BoxError.NOT_FOUND) { result.push(member); return iteratorCallback(); }
+                    if (error && error.reason == BoxError.NOT_FOUND) { result.push(member); return iteratorCallback(); } // let it bounce
                     if (error) return iteratorCallback(error);
 
-                    if (entry.type === mailboxdb.TYPE_MAILBOX) { result.push(member); return iteratorCallback(); }
-                    // no need to resolve alias because we only allow one level and within same domain
-                    if (entry.type === mailboxdb.TYPE_ALIAS) { result.push(`${entry.aliasTarget}@${entry.domain}`); return iteratorCallback(); }
+                    if (entry.type === mailboxdb.TYPE_MAILBOX) { // concrete mailbox
+                        result.push(member);
+                    } else if (entry.type === mailboxdb.TYPE_ALIAS) { // resolve aliases
+                        toResolve = toResolve.concat(`${entry.aliasName}@${entry.aliasTarget}`);
+                    } else { // resolve list members
+                        toResolve = toResolve.concat(entry.members);
+                    }
 
-                    toResolve = toResolve.concat(entry.members);
                     iteratorCallback();
                 });
             }, function (error) {
-                callback(error, result);
+                callback(error, result, list);
             });
         });
     });

src/mailboxdb.js

@@ -8,7 +8,6 @@ exports = module.exports = {
     updateList: updateList,
     del: del,
 
-    listAliases: listAliases,
     listMailboxes: listMailboxes,
     getLists: getLists,
 
@@ -41,12 +40,14 @@ var assert = require('assert'),
     safe = require('safetydance'),
     util = require('util');
 
-var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasTarget', 'creationTime', 'membersJson', 'domain' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasName', 'aliasDomain', 'creationTime', 'membersJson', 'membersOnly', 'domain' ].join(',');
 
 function postProcess(data) {
     data.members = safe.JSON.parse(data.membersJson) || [ ];
     delete data.membersJson;
 
+    data.membersOnly = !!data.membersOnly;
+
     return data;
 }
 
@@ -78,14 +79,15 @@ function updateMailboxOwner(name, domain, ownerId, callback) {
     });
 }
 
-function addList(name, domain, members, callback) {
+function addList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson) VALUES (?, ?, ?, ?, ?)',
-        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members) ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson, membersOnly) VALUES (?, ?, ?, ?, ?, ?)',
+        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members), membersOnly ], function (error) {
             if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mailbox already exists'));
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
@@ -93,14 +95,15 @@ function addList(name, domain, members, callback) {
         });
 }
 
-function updateList(name, domain, members, callback) {
+function updateList(name, domain, members, membersOnly, callback) {
     assert.strictEqual(typeof name, 'string');
     assert.strictEqual(typeof domain, 'string');
     assert(Array.isArray(members));
+    assert.strictEqual(typeof membersOnly, 'boolean');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('UPDATE mailboxes SET membersJson = ? WHERE name = ? AND domain = ?',
-        [ JSON.stringify(members), name, domain ], function (error, result) {
+    database.query('UPDATE mailboxes SET membersJson = ?, membersOnly = ? WHERE name = ? AND domain = ?',
+        [ JSON.stringify(members), membersOnly, name, domain ], function (error, result) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
             if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -123,7 +126,7 @@ function del(name, domain, callback) {
     assert.strictEqual(typeof callback, 'function');
 
     // deletes aliases as well
-    database.query('DELETE FROM mailboxes WHERE (name=? OR aliasTarget = ?) AND domain = ?', [ name, name, domain ], function (error, result) {
+    database.query('DELETE FROM mailboxes WHERE ((name=? AND domain=?) OR (aliasName = ? AND aliasDomain=?))', [ name, domain, name, domain ], function (error, result) {
         if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
         if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mailbox not found'));
 
@@ -285,10 +288,10 @@ function setAliasesForName(name, domain, aliases, callback) {
 
         var queries = [];
         // clear existing aliases
-        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasTarget = ? AND domain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
+        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasName = ? AND aliasDomain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
         aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, type, domain, aliasTarget, ownerId) VALUES (?, ?, ?, ?, ?)',
-                args: [ alias, exports.TYPE_ALIAS, domain, name, results[0].ownerId ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, domain, type, aliasName, aliasDomain, ownerId) VALUES (?, ?, ?, ?, ?, ?)',
+                args: [ alias.name, alias.domain, exports.TYPE_ALIAS, name, domain, results[0].ownerId ] });
         });
 
         database.transaction(queries, function (error) {
@@ -311,27 +314,10 @@ function getAliasesForName(name, domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');
 
-    database.query('SELECT name FROM mailboxes WHERE type = ? AND aliasTarget = ? AND domain = ? ORDER BY name',
+    database.query('SELECT name, domain FROM mailboxes WHERE type = ? AND aliasName = ? AND aliasDomain = ? ORDER BY name',
         [ exports.TYPE_ALIAS, name, domain ], function (error, results) {
             if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
 
-            results = results.map(function (r) { return r.name; });
-            callback(null, results);
-        });
-}
-
-function listAliases(domain, page, perPage, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof page, 'number');
-    assert.strictEqual(typeof perPage, 'number');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query(`SELECT ${MAILBOX_FIELDS} FROM mailboxes WHERE domain = ? AND type = ? ORDER BY name LIMIT ${(page-1)*perPage},${perPage}`,
-        [ domain, exports.TYPE_ALIAS ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-            results.forEach(function (result) { postProcess(result); });
-
             callback(null, results);
         });
 }

src/maildb.js

@@ -1,8 +1,6 @@
 'use strict';
 
 exports = module.exports = {
-    add: add,
-    del: del,
     get: get,
     list: list,
     update: update,
@@ -34,20 +32,6 @@ function postProcess(data) {
     return data;
 }
 
-function add(domain, data, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof data, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('INSERT INTO mail (domain, dkimSelector) VALUES (?, ?)', [ domain, data.dkimSelector || 'cloudron' ], function (error) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, 'mail domain already exists'));
-        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND), 'no such domain');
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-
-        callback(null);
-    });
-}
-
 function clear(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -58,20 +42,6 @@ function clear(callback) {
     });
 }
 
-function del(domain, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    // deletes aliases as well
-    database.query('DELETE FROM mail WHERE domain=?', [ domain ], function (error, result) {
-        if (error && error.code === 'ER_ROW_IS_REFERENCED_2') return callback(new BoxError(BoxError.CONFLICT));
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.affectedRows === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Mail domain not found'));
-
-        callback(null);
-    });
-}
-
 function get(domain, callback) {
     assert.strictEqual(typeof domain, 'string');
     assert.strictEqual(typeof callback, 'function');

src/mailer.js

@@ -210,7 +210,7 @@ function roleChanged(mailTo, user) {
 
     debug('Sending mail for roleChanged');
 
-    mailUserEvent(mailTo, user, `now has the role '${user.role}`);
+    mailUserEvent(mailTo, user, `now has the role '${user.role}'`);
 }
 
 function passwordReset(user) {

src/nginxconfig.ejs

@@ -58,17 +58,16 @@ server {
     ssl_certificate      <%= certFilePath %>;
     ssl_certificate_key  <%= keyFilePath %>;
     ssl_session_timeout  5m;
-    ssl_session_cache shared:SSL:50m;
+    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
+    ssl_session_tickets off;
 
-    # https://bettercrypto.org/static/applied-crypto-hardening.pdf
-    # https://mozilla.github.io/server-side-tls/ssl-config-generator/
-    # https://cipherli.st/
     # https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
-    ssl_prefer_server_ciphers on;
-    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don't use SSLv3 ref: POODLE
+    # https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#25-use-forward-secrecy
+    # ciphers according to https://ssl-config.mozilla.org/#server=nginx&version=1.14.0&config=intermediate&openssl=1.1.1&guideline=5.4
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256;
+    ssl_prefer_server_ciphers off;
 
-    # ciphers according to https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=nginx-1.10.3&openssl=1.0.2g&hsts=yes&profile=modern
-    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
     ssl_dhparam /home/yellowtent/boxdata/dhparams.pem;
     add_header Strict-Transport-Security "max-age=15768000";
 
@@ -136,8 +135,21 @@ server {
         # internal means this is for internal routing and cannot be accessed as URL from browser
         internal;
     }
-    location /appstatus.html {
-        internal;
+
+    location @wellknown-upstream {
+<% if ( endpoint === 'admin' ) { %>
+        proxy_pass http://127.0.0.1:3000;
+<% } else if ( endpoint === 'app' ) { %>
+        proxy_pass http://127.0.0.1:<%= port %>;
+<% } else if ( endpoint === 'redirect' ) { %>
+        return 302 https://<%= redirectTo %>$request_uri;
+<% } %>
+    }
+
+    # user defined .well-known resources
+    location ~ ^/.well-known/(.*)$ {
+        root /home/yellowtent/boxdata/well-known/$host;
+        try_files /$1 @wellknown-upstream;
     }
 
     location / {

src/notifications.js

@@ -141,7 +141,7 @@ function roleChanged(performedBy, eventId, user, callback) {
 
     actionForAllAdmins([ performedBy, user.id ], function (admin, done) {
         mailer.roleChanged(admin.email, user);
-        add(admin.id, eventId, `User '${user.displayName}'s role changed`, `User '${user.username || user.email || user.fallbackEmail}' now has the role '${user.role}'.`, done);
+        add(admin.id, eventId, `User '${user.displayName}'s role changed`, `User '${user.username || user.email || user.fallbackEmail}' now has the role ${user.role}.`, done);
     }, callback);
 }
 

src/paths.js

@@ -22,7 +22,7 @@ exports = module.exports = {
 
     PLATFORM_DATA_DIR: path.join(baseDir(), 'platformdata'),
     APPS_DATA_DIR: path.join(baseDir(), 'appsdata'),
-    BOX_DATA_DIR: path.join(baseDir(), 'boxdata'),
+    BOX_DATA_DIR: path.join(baseDir(), 'boxdata'),  // box data dir is part of box backup
 
     ACME_CHALLENGES_DIR: path.join(baseDir(), 'platformdata/acme'),
     ADDON_CONFIG_DIR: path.join(baseDir(), 'platformdata/addons'),
@@ -46,11 +46,14 @@ exports = module.exports = {
     APP_CERTS_DIR: path.join(baseDir(), 'boxdata/certs'),
     CLOUDRON_AVATAR_FILE: path.join(baseDir(), 'boxdata/avatar.png'),
     UPDATE_CHECKER_FILE: path.join(baseDir(), 'boxdata/updatechecker.json'),
+    ADDON_TURN_SECRET_FILE: path.join(baseDir(), 'boxdata/addon-turn-secret'),
 
     LOG_DIR: path.join(baseDir(), 'platformdata/logs'),
     TASKS_LOG_DIR: path.join(baseDir(), 'platformdata/logs/tasks'),
     CRASH_LOG_DIR: path.join(baseDir(), 'platformdata/logs/crash'),
 
+    GHOST_USER_FILE: path.join(baseDir(), 'platformdata/cloudron_ghost.json'),
+
     // this pattern is for the cloudron logs API route to work
     BACKUP_LOG_FILE: path.join(baseDir(), 'platformdata/logs/backup/app.log'),
     UPDATER_LOG_FILE: path.join(baseDir(), 'platformdata/logs/updater/app.log')

src/platform.js

@@ -133,11 +133,10 @@ function pruneInfraImages(callback) {
 function stopContainers(existingInfra, callback) {
     // always stop addons to restart them on any infra change, regardless of minor or major update
     if (existingInfra.version !== infra.version) {
-        // TODO: only nuke containers with isCloudronManaged=true
         debug('stopping all containers for infra upgrade');
         async.series([
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'network=cloudron\' | xargs --no-run-if-empty docker stop'),
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'network=cloudron\' | xargs --no-run-if-empty docker rm -f')
+            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker stop'),
+            shell.exec.bind(null, 'stopContainers', 'docker ps -qa --filter \'label=isCloudronManaged\' | xargs --no-run-if-empty docker rm -f')
         ], callback);
     } else {
         assert(typeof infra.images, 'object');
@@ -150,8 +149,8 @@ function stopContainers(existingInfra, callback) {
         let filterArg = changedAddons.map(function (c) { return `--filter 'name=${c}'`; }).join(' '); // name=c matches *c*. required for redis-{appid}
         // ignore error if container not found (and fail later) so that this code works across restarts
         async.series([
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'network=cloudron' | xargs --no-run-if-empty docker stop || true`),
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'network=cloudron' | xargs --no-run-if-empty docker rm -f || true`)
+            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker stop || true`),
+            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker rm -f || true`)
         ], callback);
     }
 }
@@ -165,7 +164,19 @@ function startApps(existingInfra, callback) {
         reverseProxy.removeAppConfigs(); // should we change the cert location, nginx will not start
         apps.configureInstalledApps(callback);
     } else {
-        debug('startApps: apps are already uptodate');
-        callback();
+        let changedAddons = [];
+        if (infra.images.mysql.tag !== existingInfra.images.mysql.tag) changedAddons.push('mysql');
+        if (infra.images.postgresql.tag !== existingInfra.images.postgresql.tag) changedAddons.push('postgresql');
+        if (infra.images.mongodb.tag !== existingInfra.images.mongodb.tag) changedAddons.push('mongodb');
+        if (infra.images.redis.tag !== existingInfra.images.redis.tag) changedAddons.push('redis');
+
+        if (changedAddons.length) {
+            // restart apps if docker image changes since the IP changes and any "persistent" connections fail
+            debug(`startApps: changedAddons: ${JSON.stringify(changedAddons)}`);
+            apps.restartAppsUsingAddons(changedAddons, callback);
+        } else {
+            debug('startApps: apps are already uptodate');
+            callback();
+        }
     }
 }

src/provision.js

@@ -121,7 +121,8 @@ function setup(dnsConfig, sysinfoConfig, auditSource, callback) {
                 provider: dnsConfig.provider,
                 config: dnsConfig.config,
                 fallbackCertificate: dnsConfig.fallbackCertificate || null,
-                tlsConfig: dnsConfig.tlsConfig || { provider: 'letsencrypt-prod' }
+                tlsConfig: dnsConfig.tlsConfig || { provider: 'letsencrypt-prod' },
+                dkimSelector: 'cloudron'
             };
 
             domains.add(domain, data, auditSource, function (error) {
@@ -137,7 +138,6 @@ function setup(dnsConfig, sysinfoConfig, auditSource, callback) {
                         settings.setSysinfoConfig.bind(null, sysinfoConfig),
                         domains.prepareDashboardDomain.bind(null, domain, auditSource, (progress) => setProgress('setup', progress.message, NOOP_CALLBACK)),
                         cloudron.setDashboardDomain.bind(null, domain, auditSource),
-                        mail.addDomain.bind(null, domain), // this relies on settings.mailFqdn() and settings.adminDomain()
                         setProgress.bind(null, 'setup', 'Done'),
                         eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
                     ], function (error) {
@@ -206,9 +206,16 @@ function restore(backupConfig, backupId, version, sysinfoConfig, auditSource, ca
         if (error) return done(error);
         if (activated) return done(new BoxError(BoxError.CONFLICT, 'Already activated. Restore with a fresh Cloudron installation.'));
 
-        backups.testConfig(backupConfig, function (error) {
+        backups.testProviderConfig(backupConfig, function (error) {
             if (error) return done(error);
 
+            if ('password' in backupConfig) {
+                backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.password);
+                delete backupConfig.password;
+            } else {
+                backupConfig.encryption = null;
+            }
+
             sysinfo.testConfig(sysinfoConfig, function (error) {
                 if (error) return done(error);
 

src/reverseproxy.js

@@ -79,7 +79,7 @@ function getCertApi(domainObject, callback) {
     // we simply update the account with the latest email we have each time when getting letsencrypt certs
     // https://github.com/ietf-wg-acme/acme/issues/30
     users.getOwner(function (error, owner) {
-        options.email = error ? 'support@cloudron.io' : (owner.fallbackEmail || owner.email); // can error if not activated yet
+        options.email = error ? 'webmaster@cloudron.io' : owner.email; // can error if not activated yet
 
         callback(null, api, options);
     });
@@ -146,19 +146,19 @@ function validateCertificate(location, domainObject, certificate) {
     // -checkhost checks for SAN or CN exclusively. SAN takes precedence and if present, ignores the CN.
     const fqdn = domains.fqdn(location, domainObject);
 
-    var result = safe.child_process.execSync(`openssl x509 -noout -checkhost "${fqdn}"`, { encoding: 'utf8', input: cert });
+    let result = safe.child_process.execSync(`openssl x509 -noout -checkhost "${fqdn}"`, { encoding: 'utf8', input: cert });
     if (result === null) return new BoxError(BoxError.BAD_FIELD, 'Unable to get certificate subject:' + safe.error.message, { field: 'cert' });
 
     if (result.indexOf('does match certificate') === -1) return new BoxError(BoxError.BAD_FIELD, `Certificate is not valid for this domain. Expecting ${fqdn}`, { field: 'cert' });
 
-    // http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#verify
-    var certModulus = safe.child_process.execSync('openssl x509 -noout -modulus', { encoding: 'utf8', input: cert });
-    if (certModulus === null) return new BoxError(BoxError.BAD_FIELD, `Unable to get cert modulus: ${safe.error.message}`, { field: 'cert' });
+    // check if public key in the cert and private key matches. pkey below works for RSA and ECDSA keys
+    const pubKeyFromCert = safe.child_process.execSync('openssl x509 -noout -pubkey', { encoding: 'utf8', input: cert });
+    if (pubKeyFromCert === null) return new BoxError(BoxError.BAD_FIELD, `Unable to get public key from cert: ${safe.error.message}`, { field: 'cert' });
 
-    var keyModulus = safe.child_process.execSync('openssl rsa -noout -modulus', { encoding: 'utf8', input: key });
-    if (keyModulus === null) return new BoxError(BoxError.BAD_FIELD, `Unable to get key modulus: ${safe.error.message}`, { field: 'cert' });
+    const pubKeyFromKey = safe.child_process.execSync('openssl pkey -pubout', { encoding: 'utf8', input: key });
+    if (pubKeyFromKey === null) return new BoxError(BoxError.BAD_FIELD, `Unable to get public key from private key: ${safe.error.message}`, { field: 'cert' });
 
-    if (certModulus !== keyModulus) return new BoxError(BoxError.BAD_FIELD, 'Key does not match the certificate.', { field: 'cert' });
+    if (pubKeyFromCert !== pubKeyFromKey) return new BoxError(BoxError.BAD_FIELD, 'Public key does not match the certificate.', { field: 'cert' });
 
     // check expiration
     result = safe.child_process.execSync('openssl x509 -checkend 0', { encoding: 'utf8', input: cert });

src/routes/apps.js

@@ -4,16 +4,16 @@ exports = module.exports = {
     getApp: getApp,
     getApps: getApps,
     getAppIcon: getAppIcon,
-    installApp: installApp,
-    uninstallApp: uninstallApp,
-    restoreApp: restoreApp,
+    install: install,
+    uninstall: uninstall,
+    restore: restore,
     importApp: importApp,
-    backupApp: backupApp,
-    updateApp: updateApp,
+    backup: backup,
+    update: update,
     getLogs: getLogs,
     getLogStream: getLogStream,
     listBackups: listBackups,
-    repairApp: repairApp,
+    repair: repair,
 
     setAccessRestriction: setAccessRestriction,
     setLabel: setLabel,
@@ -30,17 +30,20 @@ exports = module.exports = {
     setMailbox: setMailbox,
     setLocation: setLocation,
     setDataDir: setDataDir,
+    setBinds: setBinds,
 
-    stopApp: stopApp,
-    startApp: startApp,
-    restartApp: restartApp,
+    stop: stop,
+    start: start,
+    restart: restart,
     exec: exec,
     execWebSocket: execWebSocket,
 
-    cloneApp: cloneApp,
+    clone: clone,
 
     uploadFile: uploadFile,
-    downloadFile: downloadFile
+    downloadFile: downloadFile,
+
+    load: load
 };
 
 var apps = require('../apps.js'),
@@ -51,19 +54,28 @@ var apps = require('../apps.js'),
     HttpError = require('connect-lastmile').HttpError,
     HttpSuccess = require('connect-lastmile').HttpSuccess,
     safe = require('safetydance'),
+    users = require('../users.js'),
     util = require('util'),
     WebSocket = require('ws');
 
-function getApp(req, res, next) {
+function load(req, res, next) {
     assert.strictEqual(typeof req.params.id, 'string');
 
-    apps.get(req.params.id, function (error, app) {
+    apps.get(req.params.id, function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
-        next(new HttpSuccess(200, apps.removeInternalFields(app)));
+        req.resource = result;
+
+        next();
     });
 }
 
+function getApp(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
+
+    next(new HttpSuccess(200, apps.removeInternalFields(req.resource)));
+}
+
 function getApps(req, res, next) {
     assert.strictEqual(typeof req.user, 'object');
 
@@ -77,19 +89,19 @@ function getApps(req, res, next) {
 }
 
 function getAppIcon(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
-    apps.getIconPath(req.params.id, { original: req.query.original }, function (error, iconPath) {
+    apps.getIconPath(req.resource, { original: req.query.original }, function (error, iconPath) {
         if (error) return next(BoxError.toHttpError(error));
 
         res.sendFile(iconPath);
     });
 }
 
-function installApp(req, res, next) {
+function install(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
 
-    var data = req.body;
+    const data = req.body;
 
     // atleast one
     if ('manifest' in data && typeof data.manifest !== 'object') return next(new HttpError(400, 'manifest must be an object'));
@@ -133,22 +145,28 @@ function installApp(req, res, next) {
 
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
 
-    debug('Installing app :%j', data);
-
-    apps.install(data, req.user, auditSource.fromRequest(req), function (error, result) {
+    apps.downloadManifest(data.appStoreId, data.manifest, function (error, appStoreId, manifest) {
         if (error) return next(BoxError.toHttpError(error));
 
-        next(new HttpSuccess(202, { id: result.id, taskId: result.taskId }));
+        if (safe.query(manifest, 'addons.docker') && req.user.role !== users.ROLE_OWNER) return next(new HttpError(403, '"owner" role is required to install app with docker addon'));
+
+        data.appStoreId = appStoreId;
+        data.manifest = manifest;
+        apps.install(data, auditSource.fromRequest(req), function (error, result) {
+            if (error) return next(BoxError.toHttpError(error));
+
+            next(new HttpSuccess(202, { id: result.id, taskId: result.taskId }));
+        });
     });
 }
 
 function setAccessRestriction(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.accessRestriction !== 'object') return next(new HttpError(400, 'accessRestriction must be an object'));
 
-    apps.setAccessRestriction(req.params.id, req.body.accessRestriction, auditSource.fromRequest(req), function (error) {
+    apps.setAccessRestriction(req.resource, req.body.accessRestriction, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -157,11 +175,11 @@ function setAccessRestriction(req, res, next) {
 
 function setLabel(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.label !== 'string') return next(new HttpError(400, 'label must be a string'));
 
-    apps.setLabel(req.params.id, req.body.label, auditSource.fromRequest(req), function (error) {
+    apps.setLabel(req.resource, req.body.label, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -170,12 +188,12 @@ function setLabel(req, res, next) {
 
 function setTags(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (!Array.isArray(req.body.tags)) return next(new HttpError(400, 'tags must be an array'));
     if (req.body.tags.some((t) => typeof t !== 'string')) return next(new HttpError(400, 'tags array must contain strings'));
 
-    apps.setTags(req.params.id, req.body.tags, auditSource.fromRequest(req), function (error) {
+    apps.setTags(req.resource, req.body.tags, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -184,11 +202,11 @@ function setTags(req, res, next) {
 
 function setIcon(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.icon !== null && typeof req.body.icon !== 'string') return next(new HttpError(400, 'icon is null or a base-64 image string'));
 
-    apps.setIcon(req.params.id, req.body.icon, auditSource.fromRequest(req), function (error) {
+    apps.setIcon(req.resource, req.body.icon, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -197,11 +215,11 @@ function setIcon(req, res, next) {
 
 function setMemoryLimit(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.memoryLimit !== 'number') return next(new HttpError(400, 'memoryLimit is not a number'));
 
-    apps.setMemoryLimit(req.params.id, req.body.memoryLimit, auditSource.fromRequest(req), function (error, result) {
+    apps.setMemoryLimit(req.resource, req.body.memoryLimit, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -210,11 +228,11 @@ function setMemoryLimit(req, res, next) {
 
 function setCpuShares(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.cpuShares !== 'number') return next(new HttpError(400, 'cpuShares is not a number'));
 
-    apps.setCpuShares(req.params.id, req.body.cpuShares, auditSource.fromRequest(req), function (error, result) {
+    apps.setCpuShares(req.resource, req.body.cpuShares, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -223,11 +241,11 @@ function setCpuShares(req, res, next) {
 
 function setAutomaticBackup(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.enable !== 'boolean') return next(new HttpError(400, 'enable must be a boolean'));
 
-    apps.setAutomaticBackup(req.params.id, req.body.enable, auditSource.fromRequest(req), function (error) {
+    apps.setAutomaticBackup(req.resource, req.body.enable, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -236,11 +254,11 @@ function setAutomaticBackup(req, res, next) {
 
 function setAutomaticUpdate(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.enable !== 'boolean') return next(new HttpError(400, 'enable must be a boolean'));
 
-    apps.setAutomaticUpdate(req.params.id, req.body.enable, auditSource.fromRequest(req), function (error) {
+    apps.setAutomaticUpdate(req.resource, req.body.enable, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -249,13 +267,13 @@ function setAutomaticUpdate(req, res, next) {
 
 function setReverseProxyConfig(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.robotsTxt !== null && typeof req.body.robotsTxt !== 'string') return next(new HttpError(400, 'robotsTxt is not a string'));
 
     if (req.body.csp !== null && typeof req.body.csp !== 'string') return next(new HttpError(400, 'csp is not a string'));
 
-    apps.setReverseProxyConfig(req.params.id, req.body, auditSource.fromRequest(req), function (error) {
+    apps.setReverseProxyConfig(req.resource, req.body, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -264,14 +282,14 @@ function setReverseProxyConfig(req, res, next) {
 
 function setCertificate(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.key !== null && typeof req.body.cert !== 'string') return next(new HttpError(400, 'cert must be a string'));
     if (req.body.cert !== null && typeof req.body.key !== 'string') return next(new HttpError(400, 'key must be a string'));
     if (req.body.cert && !req.body.key) return next(new HttpError(400, 'key must be provided'));
     if (!req.body.cert && req.body.key) return next(new HttpError(400, 'cert must be provided'));
 
-    apps.setCertificate(req.params.id, req.body, auditSource.fromRequest(req), function (error) {
+    apps.setCertificate(req.resource, req.body, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, {}));
@@ -280,12 +298,12 @@ function setCertificate(req, res, next) {
 
 function setEnvironment(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (!req.body.env || typeof req.body.env !== 'object') return next(new HttpError(400, 'env must be an object'));
     if (Object.keys(req.body.env).some((key) => typeof req.body.env[key] !== 'string')) return next(new HttpError(400, 'env must contain values as strings'));
 
-    apps.setEnvironment(req.params.id, req.body.env, auditSource.fromRequest(req), function (error, result) {
+    apps.setEnvironment(req.resource, req.body.env, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -294,11 +312,11 @@ function setEnvironment(req, res, next) {
 
 function setDebugMode(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.debugMode !== null && typeof req.body.debugMode !== 'object') return next(new HttpError(400, 'debugMode must be an object'));
 
-    apps.setDebugMode(req.params.id, req.body.debugMode, auditSource.fromRequest(req), function (error, result) {
+    apps.setDebugMode(req.resource, req.body.debugMode, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -307,12 +325,12 @@ function setDebugMode(req, res, next) {
 
 function setMailbox(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.mailboxName !== null && typeof req.body.mailboxName !== 'string') return next(new HttpError(400, 'mailboxName must be a string'));
     if (typeof req.body.mailboxDomain !== 'string') return next(new HttpError(400, 'mailboxDomain must be a string'));
 
-    apps.setMailbox(req.params.id, req.body.mailboxName, req.body.mailboxDomain, auditSource.fromRequest(req), function (error, result) {
+    apps.setMailbox(req.resource, req.body.mailboxName, req.body.mailboxDomain, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -321,7 +339,7 @@ function setMailbox(req, res, next) {
 
 function setLocation(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.body.location !== 'string') return next(new HttpError(400, 'location must be string')); // location may be an empty string
     if (!req.body.domain) return next(new HttpError(400, 'domain is required'));
@@ -336,7 +354,7 @@ function setLocation(req, res, next) {
 
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
 
-    apps.setLocation(req.params.id, req.body, auditSource.fromRequest(req), function (error, result) {
+    apps.setLocation(req.resource, req.body, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -345,51 +363,49 @@ function setLocation(req, res, next) {
 
 function setDataDir(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (req.body.dataDir !== null && typeof req.body.dataDir !== 'string') return next(new HttpError(400, 'dataDir must be a string'));
 
-    apps.setDataDir(req.params.id, req.body.dataDir, auditSource.fromRequest(req), function (error, result) {
+    apps.setDataDir(req.resource, req.body.dataDir, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function repairApp(req, res, next) {
+function repair(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('Repair app id:%s', req.params.id);
+    assert.strictEqual(typeof req.resource, 'object');
 
     const data = req.body;
 
     if ('manifest' in data) {
         if (!data.manifest || typeof data.manifest !== 'object') return next(new HttpError(400, 'manifest must be an object'));
+
+        if (safe.query(data.manifest, 'addons.docker') && req.user.role !== users.ROLE_OWNER) return next(new HttpError(403, '"owner" role is required to repair app with docker addon'));
     }
 
     if ('dockerImage' in data) {
         if (!data.dockerImage || typeof data.dockerImage !== 'string') return next(new HttpError(400, 'dockerImage must be a string'));
     }
 
-    apps.repair(req.params.id, data, auditSource.fromRequest(req), function (error, result) {
+    apps.repair(req.resource, data, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function restoreApp(req, res, next) {
+function restore(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var data = req.body;
 
-    debug('Restore app id:%s', req.params.id);
-
     if (!data.backupId || typeof data.backupId !== 'string') return next(new HttpError(400, 'backupId must be non-empty string'));
 
-    apps.restore(req.params.id, data.backupId, auditSource.fromRequest(req), function (error, result) {
+    apps.restore(req.resource, data.backupId, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
@@ -398,12 +414,10 @@ function restoreApp(req, res, next) {
 
 function importApp(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var data = req.body;
 
-    debug('Importing app id:%s', req.params.id);
-
     if ('backupId' in data) { // if not provided, we import in-place
         if (typeof data.backupId !== 'string') return next(new HttpError(400, 'backupId must be string'));
         if (typeof data.backupFormat !== 'string') return next(new HttpError(400, 'backupFormat must be string'));
@@ -414,7 +428,7 @@ function importApp(req, res, next) {
 
         if (req.body.backupConfig) {
             if (typeof backupConfig.provider !== 'string') return next(new HttpError(400, 'provider is required'));
-            if ('key' in backupConfig && typeof backupConfig.key !== 'string') return next(new HttpError(400, 'key must be a string'));
+            if ('password' in backupConfig && typeof backupConfig.password !== 'string') return next(new HttpError(400, 'password must be a string'));
             if ('acceptSelfSignedCerts' in backupConfig && typeof backupConfig.acceptSelfSignedCerts !== 'boolean') return next(new HttpError(400, 'format must be a boolean'));
 
             // testing backup config can take sometime
@@ -422,21 +436,19 @@ function importApp(req, res, next) {
         }
     }
 
-    apps.importApp(req.params.id, data, auditSource.fromRequest(req), function (error, result) {
+    apps.importApp(req.resource, data, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function cloneApp(req, res, next) {
+function clone(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var data = req.body;
 
-    debug('Clone app id:%s', req.params.id);
-
     if (typeof data.backupId !== 'string') return next(new HttpError(400, 'backupId must be a string'));
     if (typeof data.location !== 'string') return next(new HttpError(400, 'location is required'));
     if (typeof data.domain !== 'string') return next(new HttpError(400, 'domain is required'));
@@ -444,76 +456,66 @@ function cloneApp(req, res, next) {
 
     if ('overwriteDns' in req.body && typeof req.body.overwriteDns !== 'boolean') return next(new HttpError(400, 'overwriteDns must be boolean'));
 
-    apps.clone(req.params.id, data, req.user, auditSource.fromRequest(req), function (error, result) {
+    apps.clone(req.resource, data, req.user, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(201, { id: result.id, taskId: result.taskId }));
     });
 }
 
-function backupApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function backup(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
 
-    debug('Backup app id:%s', req.params.id);
-
-    apps.backup(req.params.id, function (error, result) {
+    apps.backup(req.resource, function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function uninstallApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function uninstall(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
 
-    debug('Uninstalling app id:%s', req.params.id);
-
-    apps.uninstall(req.params.id, auditSource.fromRequest(req), function (error, result) {
+    apps.uninstall(req.resource, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function startApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function start(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
 
-    debug('Start app id:%s', req.params.id);
-
-    apps.start(req.params.id, function (error, result) {
+    apps.start(req.resource, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function stopApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function stop(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
 
-    debug('Stop app id:%s', req.params.id);
-
-    apps.stop(req.params.id, function (error, result) {
+    apps.stop(req.resource, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function restartApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function restart(req, res, next) {
+    assert.strictEqual(typeof req.resource, 'object');
 
-    debug('Restart app id:%s', req.params.id);
-
-    apps.restart(req.params.id, function (error, result) {
+    apps.restart(req.resource, auditSource.fromRequest(req), function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(202, { taskId: result.taskId }));
     });
 }
 
-function updateApp(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+function update(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var data = req.body;
 
@@ -525,20 +527,24 @@ function updateApp(req, res, next) {
     if ('skipBackup' in data && typeof data.skipBackup !== 'boolean') return next(new HttpError(400, 'skipBackup must be a boolean'));
     if ('force' in data && typeof data.force !== 'boolean') return next(new HttpError(400, 'force must be a boolean'));
 
-    debug('Update app id:%s to manifest:%j', req.params.id, data.manifest);
-
-    apps.update(req.params.id, req.body, auditSource.fromRequest(req), function (error, result) {
+    apps.downloadManifest(data.appStoreId, data.manifest, function (error, appStoreId, manifest) {
         if (error) return next(BoxError.toHttpError(error));
 
-        next(new HttpSuccess(202, { taskId: result.taskId }));
+        if (safe.query(manifest, 'addons.docker') && req.user.role !== users.ROLE_OWNER) return next(new HttpError(403, '"owner" role is required to update app with docker addon'));
+
+        data.appStoreId = appStoreId;
+        data.manifest = manifest;
+        apps.update(req.resource, data, auditSource.fromRequest(req), function (error, result) {
+            if (error) return next(BoxError.toHttpError(error));
+
+            next(new HttpSuccess(202, { taskId: result.taskId }));
+        });
     });
 }
 
 // this route is for streaming logs
 function getLogStream(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('Getting logstream of app id:%s', req.params.id);
+    assert.strictEqual(typeof req.resource, 'object');
 
     var lines = 'lines' in req.query ? parseInt(req.query.lines, 10) : 10; // we ignore last-event-id
     if (isNaN(lines)) return next(new HttpError(400, 'lines must be a valid number'));
@@ -553,7 +559,7 @@ function getLogStream(req, res, next) {
         format: 'json'
     };
 
-    apps.getLogs(req.params.id, options, function (error, logStream) {
+    apps.getLogs(req.resource, options, function (error, logStream) {
         if (error) return next(BoxError.toHttpError(error));
 
         res.writeHead(200, {
@@ -575,20 +581,18 @@ function getLogStream(req, res, next) {
 }
 
 function getLogs(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var lines = 'lines' in req.query ? parseInt(req.query.lines, 10) : 10;
     if (isNaN(lines)) return next(new HttpError(400, 'lines must be a number'));
 
-    debug('Getting logs of app id:%s', req.params.id);
-
     var options = {
         lines: lines,
         follow: false,
         format: req.query.format || 'json'
     };
 
-    apps.getLogs(req.params.id, options, function (error, logStream) {
+    apps.getLogs(req.resource, options, function (error, logStream) {
         if (error) return next(BoxError.toHttpError(error));
 
         res.writeHead(200, {
@@ -624,9 +628,7 @@ function demuxStream(stream, stdin) {
 }
 
 function exec(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('Execing into app id:%s and cmd:%s', req.params.id, req.query.cmd);
+    assert.strictEqual(typeof req.resource, 'object');
 
     var cmd = null;
     if (req.query.cmd) {
@@ -640,13 +642,16 @@ function exec(req, res, next) {
     var rows = req.query.rows ? parseInt(req.query.rows, 10) : null;
     if (isNaN(rows)) return next(new HttpError(400, 'rows must be a number'));
 
-    var tty = req.query.tty === 'true' ? true : false;
+    var tty = req.query.tty === 'true';
 
-    apps.exec(req.params.id, { cmd: cmd, rows: rows, columns: columns, tty: tty }, function (error, duplexStream) {
+    if (safe.query(req.resource, 'manifest.addons.docker') && req.user.role !== users.ROLE_OWNER) return next(new HttpError(403, '"owner" role is requied to exec app with docker addon'));
+
+    // in a badly configured reverse proxy, we might be here without an upgrade
+    if (req.headers['upgrade'] !== 'tcp') return next(new HttpError(404, 'exec requires TCP upgrade'));
+
+    apps.exec(req.resource, { cmd: cmd, rows: rows, columns: columns, tty: tty }, function (error, duplexStream) {
         if (error) return next(BoxError.toHttpError(error));
 
-        if (req.headers['upgrade'] !== 'tcp') return next(new HttpError(404, 'exec requires TCP upgrade'));
-
         req.clearTimeout();
         res.sendUpgradeHandshake();
 
@@ -664,9 +669,7 @@ function exec(req, res, next) {
 }
 
 function execWebSocket(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('Execing websocket into app id:%s and cmd:%s', req.params.id, req.query.cmd);
+    assert.strictEqual(typeof req.resource, 'object');
 
     var cmd = null;
     if (req.query.cmd) {
@@ -682,10 +685,11 @@ function execWebSocket(req, res, next) {
 
     var tty = req.query.tty === 'true' ? true : false;
 
-    apps.exec(req.params.id, { cmd: cmd, rows: rows, columns: columns, tty: tty }, function (error, duplexStream) {
-        if (error) return next(BoxError.toHttpError(error));
+    // in a badly configured reverse proxy, we might be here without an upgrade
+    if (req.headers['upgrade'] !== 'websocket') return next(new HttpError(404, 'exec requires websocket'));
 
-        debug('Connected to terminal');
+    apps.exec(req.resource, { cmd: cmd, rows: rows, columns: columns, tty: tty }, function (error, duplexStream) {
+        if (error) return next(BoxError.toHttpError(error));
 
         req.clearTimeout();
 
@@ -714,7 +718,7 @@ function execWebSocket(req, res, next) {
 }
 
 function listBackups(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
+    assert.strictEqual(typeof req.resource, 'object');
 
     var page = typeof req.query.page !== 'undefined' ? parseInt(req.query.page) : 1;
     if (!page || page < 0) return next(new HttpError(400, 'page query param has to be a postive number'));
@@ -722,7 +726,7 @@ function listBackups(req, res, next) {
     var perPage = typeof req.query.per_page !== 'undefined'? parseInt(req.query.per_page) : 25;
     if (!perPage || perPage < 0) return next(new HttpError(400, 'per_page query param has to be a postive number'));
 
-    apps.listBackups(page, perPage, req.params.id, function (error, result) {
+    apps.listBackups(req.resource, page, perPage, function (error, result) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(200, { backups: result }));
@@ -730,30 +734,24 @@ function listBackups(req, res, next) {
 }
 
 function uploadFile(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('uploadFile: %s %j -> %s', req.params.id, req.files, req.query.file);
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.query.file !== 'string' || !req.query.file) return next(new HttpError(400, 'file query argument must be provided'));
     if (!req.files.file) return next(new HttpError(400, 'file must be provided as multipart'));
 
-    apps.uploadFile(req.params.id, req.files.file.path, req.query.file, function (error) {
+    apps.uploadFile(req.resource, req.files.file.path, req.query.file, function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
-        debug('uploadFile: done');
-
         next(new HttpSuccess(202, {}));
     });
 }
 
 function downloadFile(req, res, next) {
-    assert.strictEqual(typeof req.params.id, 'string');
-
-    debug('downloadFile: ', req.params.id, req.query.file);
+    assert.strictEqual(typeof req.resource, 'object');
 
     if (typeof req.query.file !== 'string' || !req.query.file) return next(new HttpError(400, 'file query argument must be provided'));
 
-    apps.downloadFile(req.params.id, req.query.file, function (error, stream, info) {
+    apps.downloadFile(req.resource, req.query.file, function (error, stream, info) {
         if (error) return next(BoxError.toHttpError(error));
 
         var headers = {
@@ -767,3 +765,23 @@ function downloadFile(req, res, next) {
         stream.pipe(res);
     });
 }
+
+function setBinds(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+    assert.strictEqual(typeof req.resource, 'object');
+
+    if (!req.body.binds || typeof req.body.binds !== 'object') return next(new HttpError(400, 'binds should be an object'));
+
+    for (let name of Object.keys(req.body.binds)) {
+        if (!req.body.binds[name] || typeof req.body.binds[name] !== 'object') return next(new HttpError(400, 'each bind should be an object'));
+        if (typeof req.body.binds[name].hostPath !== 'string') return next(new HttpError(400, 'hostPath must be a string'));
+        if (typeof req.body.binds[name].readOnly !== 'boolean') return next(new HttpError(400, 'readOnly must be a boolean'));
+    }
+
+    apps.setBinds(req.resource, req.body.binds, auditSource.fromRequest(req), function (error, result) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, { taskId: result.taskId }));
+    });
+}
+

src/routes/branding.js

@@ -0,0 +1,140 @@
+'use strict';
+
+exports = module.exports = {
+    get,
+    set,
+
+    getCloudronAvatar
+};
+
+var assert = require('assert'),
+    BoxError = require('../boxerror.js'),
+    HttpError = require('connect-lastmile').HttpError,
+    HttpSuccess = require('connect-lastmile').HttpSuccess,
+    safe = require('safetydance'),
+    settings = require('../settings.js'),
+    _ = require('underscore');
+
+function getFooter(req, res, next) {
+    settings.getFooter(function (error, footer) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, { footer }));
+    });
+}
+
+function setFooter(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    if (typeof req.body.footer !== 'string') return next(new HttpError(400, 'footer is required'));
+
+    settings.setFooter(req.body.footer, function (error) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, {}));
+    });
+}
+
+function setCloudronName(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    if (typeof req.body.name !== 'string') return next(new HttpError(400, 'name is required'));
+
+    settings.setCloudronName(req.body.name, function (error) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, {}));
+    });
+}
+
+function getCloudronName(req, res, next) {
+    settings.getCloudronName(function (error, name) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, { name: name }));
+    });
+}
+
+function setAppstoreListingConfig(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    const listingConfig = _.pick(req.body, 'whitelist', 'blacklist');
+    if (Object.keys(listingConfig).length === 0) return next(new HttpError(400, 'blacklist or whitelist is required'));
+
+    if ('whitelist' in listingConfig) {
+        if (listingConfig.whitelist !== null && !Array.isArray(listingConfig.whitelist)) return next(new HttpError(400, 'whitelist is null or an array of strings'));
+
+        if (listingConfig.whitelist && !listingConfig.whitelist.every(id => typeof id === 'string')) return next(new HttpError(400, 'whitelist must be array of strings'));
+    }
+
+    if ('blacklist' in listingConfig) {
+        if (!Array.isArray(listingConfig.blacklist)) return next(new HttpError(400, 'blacklist an array of strings'));
+
+        if (!listingConfig.blacklist.every(id => typeof id === 'string')) return next(new HttpError(400, 'blacklist must be array of strings'));
+    }
+
+    settings.setAppstoreListingConfig(listingConfig, function (error) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, {}));
+    });
+}
+
+function getAppstoreListingConfig(req, res, next) {
+    settings.getAppstoreListingConfig(function (error, listingConfig) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(200, listingConfig));
+    });
+}
+
+function setCloudronAvatar(req, res, next) {
+    assert.strictEqual(typeof req.files, 'object');
+
+    if (!req.files.avatar) return next(new HttpError(400, 'avatar must be provided'));
+    var avatar = safe.fs.readFileSync(req.files.avatar.path);
+
+    settings.setCloudronAvatar(avatar, function (error) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        next(new HttpSuccess(202, {}));
+    });
+}
+
+function getCloudronAvatar(req, res, next) {
+    settings.getCloudronAvatar(function (error, avatar) {
+        if (error) return next(BoxError.toHttpError(error));
+
+        // avoid caching the avatar on the client to see avatar changes immediately
+        res.set('Cache-Control', 'no-cache');
+
+        res.set('Content-Type', 'image/png');
+        res.status(200).send(avatar);
+    });
+}
+
+function get(req, res, next) {
+    assert.strictEqual(typeof req.params.setting, 'string');
+
+    switch (req.params.setting) {
+    case settings.APPSTORE_LISTING_CONFIG_KEY: return getAppstoreListingConfig(req, res, next);
+    case settings.CLOUDRON_AVATAR_KEY: return getCloudronAvatar(req, res, next);
+    case settings.CLOUDRON_NAME_KEY: return getCloudronName(req, res, next);
+    case settings.FOOTER_KEY: return getFooter(req, res, next);
+
+    default: return next(new HttpError(404, 'No such setting'));
+    }
+}
+
+function set(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    switch (req.params.setting) {
+    case settings.APPSTORE_LISTING_CONFIG_KEY: return setAppstoreListingConfig(req, res, next);
+    case settings.CLOUDRON_AVATAR_KEY: return setCloudronAvatar(req, res, next);
+    case settings.CLOUDRON_NAME_KEY: return setCloudronName(req, res, next);
+    case settings.FOOTER_KEY: return setFooter(req, res, next);
+
+    default: return next(new HttpError(404, 'No such branding'));
+    }
+}

src/routes/cloudron.js

@@ -29,7 +29,6 @@ let assert = require('assert'),
     BoxError = require('../boxerror.js'),
     cloudron = require('../cloudron.js'),
     constants = require('../constants.js'),
-    debug = require('debug')('box:routes/cloudron'),
     eventlog = require('../eventlog.js'),
     externalLdap = require('../externalldap.js'),
     HttpError = require('connect-lastmile').HttpError,
@@ -103,6 +102,7 @@ function passwordReset(req, res, next) {
     users.getByResetToken(req.body.resetToken, function (error, userObject) {
         if (error) return next(new HttpError(401, 'Invalid resetToken'));
 
+        if (Date.now() - userObject.resetTokenCreationTime > 24 * 60 * 60 * 1000) return next(new HttpError(401, 'Token expired'));
         if (!userObject.username) return next(new HttpError(409, 'No username set'));
 
         // setPassword clears the resetToken
@@ -128,11 +128,11 @@ function setupAccount(req, res, next) {
     if (!req.body.username || typeof req.body.username !== 'string') return next(new HttpError(400, 'username must be a non-empty string'));
     if (!req.body.displayName || typeof req.body.displayName !== 'string') return next(new HttpError(400, 'displayName must be a non-empty string'));
 
-    debug(`setupAccount: for email ${req.body.email} and username ${req.body.username} with token ${req.body.resetToken}`);
-
     users.getByResetToken(req.body.resetToken, function (error, userObject) {
         if (error) return next(new HttpError(401, 'Invalid Reset Token'));
 
+        if (Date.now() - userObject.resetTokenCreationTime > 24 * 60 * 60 * 1000) return next(new HttpError(401, 'Token expired'));
+
         users.update(userObject, { username: req.body.username, displayName: req.body.displayName }, auditSource.fromRequest(req), function (error) {
             if (error && error.reason === BoxError.ALREADY_EXISTS) return next(new HttpError(409, 'Username already used'));
             if (error && error.reason === BoxError.BAD_FIELD) return next(new HttpError(400, error.message));
@@ -218,8 +218,8 @@ function checkForUpdates(req, res, next) {
     req.clearTimeout();
 
     async.series([
-        updateChecker.checkAppUpdates,
-        updateChecker.checkBoxUpdates
+        (done) => updateChecker.checkAppUpdates({ automatic: false }, done),
+        (done) => updateChecker.checkBoxUpdates({ automatic: false }, done),
     ], function () {
         next(new HttpSuccess(200, { update: updateChecker.getUpdateInfo() }));
     });

src/routes/index.js

@@ -6,6 +6,7 @@ exports = module.exports = {
     apps: require('./apps.js'),
     appstore: require('./appstore.js'),
     backups: require('./backups.js'),
+    branding: require('./branding.js'),
     cloudron: require('./cloudron.js'),
     domains: require('./domains.js'),
     eventlog: require('./eventlog.js'),

src/routes/mail.js

@@ -2,8 +2,6 @@
 
 exports = module.exports = {
     getDomain: getDomain,
-    addDomain: addDomain,
-    removeDomain: removeDomain,
 
     setDnsRecords: setDnsRecords,
 
@@ -22,7 +20,6 @@ exports = module.exports = {
     updateMailbox: updateMailbox,
     removeMailbox: removeMailbox,
 
-    listAliases: listAliases,
     getAliases: getAliases,
     setAliases: setAliases,
 
@@ -50,18 +47,6 @@ function getDomain(req, res, next) {
     });
 }
 
-function addDomain(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-
-    if (typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
-
-    mail.addDomain(req.body.domain, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(201, { domain: req.body.domain }));
-    });
-}
-
 function setDnsRecords(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
     assert.strictEqual(typeof req.params.domain, 'string');
@@ -77,16 +62,6 @@ function setDnsRecords(req, res, next) {
     });
 }
 
-function removeDomain(req, res, next) {
-    assert.strictEqual(typeof req.params.domain, 'string');
-
-    mail.removeDomain(req.params.domain, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(204));
-    });
-}
-
 function getStatus(req, res, next) {
     assert.strictEqual(typeof req.params.domain, 'string');
 
@@ -239,22 +214,6 @@ function removeMailbox(req, res, next) {
     });
 }
 
-function listAliases(req, res, next) {
-    assert.strictEqual(typeof req.params.domain, 'string');
-
-    var page = typeof req.query.page !== 'undefined' ? parseInt(req.query.page) : 1;
-    if (!page || page < 0) return next(new HttpError(400, 'page query param has to be a positive number'));
-
-    var perPage = typeof req.query.per_page !== 'undefined'? parseInt(req.query.per_page) : 25;
-    if (!perPage || perPage < 0) return next(new HttpError(400, 'per_page query param has to be a positive number'));
-
-    mail.listAliases(req.params.domain, page, perPage, function (error, result) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(200, { aliases: result }));
-    });
-}
-
 function getAliases(req, res, next) {
     assert.strictEqual(typeof req.params.domain, 'string');
     assert.strictEqual(typeof req.params.name, 'string');
@@ -273,8 +232,10 @@ function setAliases(req, res, next) {
 
     if (!Array.isArray(req.body.aliases)) return next(new HttpError(400, 'aliases must be an array'));
 
-    for (var i = 0; i < req.body.aliases.length; i++) {
-        if (typeof req.body.aliases[i] !== 'string') return next(new HttpError(400, 'alias must be a string'));
+    for (let alias of req.body.aliases) {
+        if (!alias || typeof alias !== 'object') return next(new HttpError(400, 'each alias must have a name and domain'));
+        if (typeof alias.name !== 'string') return next(new HttpError(400, 'name must be a string'));
+        if (typeof alias.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
     }
 
     mail.setAliases(req.params.name, req.params.domain, req.body.aliases, function (error) {
@@ -316,8 +277,9 @@ function addList(req, res, next) {
     for (var i = 0; i < req.body.members.length; i++) {
         if (typeof req.body.members[i] !== 'string') return next(new HttpError(400, 'member must be a string'));
     }
+    if (typeof req.body.membersOnly !== 'boolean') return next(new HttpError(400, 'membersOnly must be a boolean'));
 
-    mail.addList(req.body.name, req.params.domain, req.body.members, auditSource.fromRequest(req), function (error) {
+    mail.addList(req.body.name, req.params.domain, req.body.members, req.body.membersOnly, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(201, {}));
@@ -334,8 +296,9 @@ function updateList(req, res, next) {
     for (var i = 0; i < req.body.members.length; i++) {
         if (typeof req.body.members[i] !== 'string') return next(new HttpError(400, 'member must be a string'));
     }
+    if (typeof req.body.membersOnly !== 'boolean') return next(new HttpError(400, 'membersOnly must be a boolean'));
 
-    mail.updateList(req.params.name, req.params.domain, req.body.members, auditSource.fromRequest(req), function (error) {
+    mail.updateList(req.params.name, req.params.domain, req.body.members, req.body.membersOnly, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));
 
         next(new HttpSuccess(204));

src/routes/mailserver.js

@@ -21,7 +21,7 @@ function proxy(req, res, next) {
     delete req.headers['authorization'];
     delete req.headers['cookies'];
 
-    addons.getServiceDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
+    addons.getContainerDetails('mail', 'CLOUDRON_MAIL_TOKEN', function (error, addonDetails) {
         if (error) return next(BoxError.toHttpError(error));
 
         parsedUrl.query['access_token'] = addonDetails.token;

src/routes/provision.js

@@ -98,11 +98,11 @@ function restore(req, res, next) {
 
     var backupConfig = req.body.backupConfig;
     if (typeof backupConfig.provider !== 'string') return next(new HttpError(400, 'provider is required'));
-    if ('key' in backupConfig && typeof backupConfig.key !== 'string') return next(new HttpError(400, 'key must be a string'));
+    if ('password' in backupConfig && typeof backupConfig.password !== 'string') return next(new HttpError(400, 'password must be a string'));
     if (typeof backupConfig.format !== 'string') return next(new HttpError(400, 'format must be a string'));
     if ('acceptSelfSignedCerts' in backupConfig && typeof backupConfig.acceptSelfSignedCerts !== 'boolean') return next(new HttpError(400, 'format must be a boolean'));
 
-    if (typeof req.body.backupId !== 'string') return next(new HttpError(400, 'backupId must be a string or null'));
+    if (typeof req.body.backupId !== 'string') return next(new HttpError(400, 'backupId must be a string'));
     if (typeof req.body.version !== 'string') return next(new HttpError(400, 'version must be a string'));
 
     if ('sysinfoConfig' in req.body && typeof req.body.sysinfoConfig !== 'object') return next(new HttpError(400, 'sysinfoConfig must be an object'));

src/routes/services.js

@@ -12,7 +12,6 @@ exports = module.exports = {
 var addons = require('../addons.js'),
     assert = require('assert'),
     BoxError = require('../boxerror.js'),
-    debug = require('debug')('box:routes/addons'),
     HttpError = require('connect-lastmile').HttpError,
     HttpSuccess = require('connect-lastmile').HttpSuccess;
 
@@ -60,8 +59,6 @@ function getLogs(req, res, next) {
     var lines = 'lines' in req.query ? parseInt(req.query.lines, 10) : 10; // we ignore last-event-id
     if (isNaN(lines)) return next(new HttpError(400, 'lines must be a number'));
 
-    debug(`Getting logs of service ${req.params.service}`);
-
     var options = {
         lines: lines,
         follow: false,
@@ -85,8 +82,6 @@ function getLogs(req, res, next) {
 function getLogStream(req, res, next) {
     assert.strictEqual(typeof req.params.service, 'string');
 
-    debug(`Getting logstream of service ${req.params.service}`);
-
     var lines = 'lines' in req.query ? parseInt(req.query.lines, 10) : 10; // we ignore last-event-id
     if (isNaN(lines)) return next(new HttpError(400, 'lines must be a valid number'));
 
@@ -124,8 +119,6 @@ function getLogStream(req, res, next) {
 function restart(req, res, next) {
     assert.strictEqual(typeof req.params.service, 'string');
 
-    debug(`Restarting service ${req.params.service}`);
-
     addons.restartService(req.params.service, function (error) {
         if (error) return next(BoxError.toHttpError(error));
 

src/routes/settings.js

@@ -1,10 +1,11 @@
 'use strict';
 
 exports = module.exports = {
-    set: set,
-    get: get,
+    set,
+    get,
 
-    getCloudronAvatar: getCloudronAvatar
+    // owner only settings
+    setBackupConfig
 };
 
 var assert = require('assert'),
@@ -14,7 +15,6 @@ var assert = require('assert'),
     externalLdap = require('../externalldap.js'),
     HttpError = require('connect-lastmile').HttpError,
     HttpSuccess = require('connect-lastmile').HttpSuccess,
-    safe = require('safetydance'),
     settings = require('../settings.js');
 
 function getAppAutoupdatePattern(req, res, next) {
@@ -57,26 +57,6 @@ function setBoxAutoupdatePattern(req, res, next) {
     });
 }
 
-function setCloudronName(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-
-    if (typeof req.body.name !== 'string') return next(new HttpError(400, 'name is required'));
-
-    settings.setCloudronName(req.body.name, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(202, {}));
-    });
-}
-
-function getCloudronName(req, res, next) {
-    settings.getCloudronName(function (error, name) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(200, { name: name }));
-    });
-}
-
 function getTimeZone(req, res, next) {
     settings.getTimeZone(function (error, tz) {
         if (error) return next(BoxError.toHttpError(error));
@@ -97,26 +77,6 @@ function setTimeZone(req, res, next) {
     });
 }
 
-function getFooter(req, res, next) {
-    settings.getFooter(function (error, footer) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(200, { footer }));
-    });
-}
-
-function setFooter(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-
-    if (typeof req.body.footer !== 'string') return next(new HttpError(400, 'footer is required'));
-
-    settings.setFooter(req.body.footer, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(200, {}));
-    });
-}
-
 function getSupportConfig(req, res, next) {
     settings.getSupportConfig(function (error, supportConfig) {
         if (error) return next(BoxError.toHttpError(error));
@@ -125,31 +85,6 @@ function getSupportConfig(req, res, next) {
     });
 }
 
-function setCloudronAvatar(req, res, next) {
-    assert.strictEqual(typeof req.files, 'object');
-
-    if (!req.files.avatar) return next(new HttpError(400, 'avatar must be provided'));
-    var avatar = safe.fs.readFileSync(req.files.avatar.path);
-
-    settings.setCloudronAvatar(avatar, function (error) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        next(new HttpSuccess(202, {}));
-    });
-}
-
-function getCloudronAvatar(req, res, next) {
-    settings.getCloudronAvatar(function (error, avatar) {
-        if (error) return next(BoxError.toHttpError(error));
-
-        // avoid caching the avatar on the client to see avatar changes immediately
-        res.set('Cache-Control', 'no-cache');
-
-        res.set('Content-Type', 'image/png');
-        res.status(200).send(avatar);
-    });
-}
-
 function getBackupConfig(req, res, next) {
     settings.getBackupConfig(function (error, backupConfig) {
         if (error) return next(BoxError.toHttpError(error));
@@ -162,9 +97,8 @@ function setBackupConfig(req, res, next) {
     assert.strictEqual(typeof req.body, 'object');
 
     if (typeof req.body.provider !== 'string') return next(new HttpError(400, 'provider is required'));
-    if (typeof req.body.retentionSecs !== 'number') return next(new HttpError(400, 'retentionSecs is required'));
     if (typeof req.body.intervalSecs !== 'number') return next(new HttpError(400, 'intervalSecs is required'));
-    if ('key' in req.body && typeof req.body.key !== 'string') return next(new HttpError(400, 'key must be a string'));
+    if ('password' in req.body && typeof req.body.password !== 'string') return next(new HttpError(400, 'password must be a string'));
     if ('syncConcurrency' in req.body) {
         if (typeof req.body.syncConcurrency !== 'number') return next(new HttpError(400, 'syncConcurrency must be a positive integer'));
         if (req.body.syncConcurrency < 1) return next(new HttpError(400, 'syncConcurrency must be a positive integer'));
@@ -172,6 +106,8 @@ function setBackupConfig(req, res, next) {
     if (typeof req.body.format !== 'string') return next(new HttpError(400, 'format must be a string'));
     if ('acceptSelfSignedCerts' in req.body && typeof req.body.acceptSelfSignedCerts !== 'boolean') return next(new HttpError(400, 'format must be a boolean'));
 
+    if (!req.body.retentionPolicy || typeof req.body.retentionPolicy !== 'object') return next(new HttpError(400, 'retentionPolicy is required'));
+
     // testing the backup using put/del takes a bit of time at times
     req.clearTimeout();
 
@@ -332,11 +268,6 @@ function get(req, res, next) {
     case settings.APP_AUTOUPDATE_PATTERN_KEY: return getAppAutoupdatePattern(req, res, next);
     case settings.BOX_AUTOUPDATE_PATTERN_KEY: return getBoxAutoupdatePattern(req, res, next);
     case settings.TIME_ZONE_KEY: return getTimeZone(req, res, next);
-    case settings.CLOUDRON_NAME_KEY: return getCloudronName(req, res, next);
-
-    case settings.FOOTER_KEY: return getFooter(req, res, next);
-
-    case settings.CLOUDRON_AVATAR_KEY: return getCloudronAvatar(req, res, next);
 
     case settings.SUPPORT_CONFIG_KEY: return getSupportConfig(req, res, next);
 
@@ -349,7 +280,6 @@ function set(req, res, next) {
 
     switch (req.params.setting) {
     case settings.DYNAMIC_DNS_KEY: return setDynamicDnsConfig(req, res, next);
-    case settings.BACKUP_CONFIG_KEY: return setBackupConfig(req, res, next);
     case settings.PLATFORM_CONFIG_KEY: return setPlatformConfig(req, res, next);
     case settings.EXTERNAL_LDAP_KEY: return setExternalLdapConfig(req, res, next);
     case settings.UNSTABLE_APPS_KEY: return setUnstableAppsConfig(req, res, next);
@@ -359,11 +289,6 @@ function set(req, res, next) {
     case settings.APP_AUTOUPDATE_PATTERN_KEY: return setAppAutoupdatePattern(req, res, next);
     case settings.BOX_AUTOUPDATE_PATTERN_KEY: return setBoxAutoupdatePattern(req, res, next);
     case settings.TIME_ZONE_KEY: return setTimeZone(req, res, next);
-    case settings.CLOUDRON_NAME_KEY: return setCloudronName(req, res, next);
-
-    case settings.FOOTER_KEY: return setFooter(req, res, next);
-
-    case settings.CLOUDRON_AVATAR_KEY: return setCloudronAvatar(req, res, next);
 
     default: return next(new HttpError(404, 'No such setting'));
     }

src/routes/test/backups-test.js

@@ -62,7 +62,7 @@ function setup(done) {
         },
 
         function createSettings(callback) {
-            settings.setBackupConfig({ provider: 'filesystem', backupFolder: '/tmp', format: 'tgz' }, callback);
+            settings.setBackupConfig({ provider: 'filesystem', backupFolder: '/tmp', format: 'tgz', retentionPolicy: {} }, callback);
         }
     ], done);
 }

src/routes/test/branding-test.js

@@ -0,0 +1,238 @@
+'use strict';
+
+/* global it:false */
+/* global describe:false */
+/* global before:false */
+/* global after:false */
+
+var async = require('async'),
+    constants = require('../../constants.js'),
+    database = require('../../database.js'),
+    expect = require('expect.js'),
+    fs = require('fs'),
+    paths = require('../../paths.js'),
+    server = require('../../server.js'),
+    superagent = require('superagent');
+
+var SERVER_URL = 'http://localhost:' + constants.PORT;
+
+var USERNAME = 'superadmin', PASSWORD = 'Foobar?1337', EMAIL ='silly@me.com';
+var token = null;
+
+function setup(done) {
+    async.series([
+        server.start.bind(null),
+        database._clear.bind(null),
+
+        function createAdmin(callback) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
+                .query({ setupToken: 'somesetuptoken' })
+                .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
+                .end(function (error, result) {
+                    expect(result).to.be.ok();
+                    expect(result.statusCode).to.eql(201);
+
+                    // stash token for further use
+                    token = result.body.token;
+
+                    callback();
+                });
+        }
+    ], done);
+}
+
+function cleanup(done) {
+    database._clear(function (error) {
+        expect(!error).to.be.ok();
+
+        server.stop(done);
+    });
+}
+
+describe('Branding API', function () {
+    before(setup);
+    after(cleanup);
+
+    describe('cloudron_name', function () {
+        var name = 'foobar';
+
+        it('get default succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/cloudron_name')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.name).to.be.ok();
+                    done();
+                });
+        });
+
+        it('cannot set without name', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/cloudron_name')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('cannot set empty name', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/cloudron_name')
+                .query({ access_token: token })
+                .send({ name: '' })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('set succeeds', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/cloudron_name')
+                .query({ access_token: token })
+                .send({ name: name })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('get succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/cloudron_name')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.name).to.eql(name);
+                    done();
+                });
+        });
+    });
+
+    describe('cloudron_avatar', function () {
+        it('get default succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/cloudron_avatar')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body).to.be.a(Buffer);
+                    done();
+                });
+        });
+
+        it('cannot set without data', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/cloudron_avatar')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('set succeeds', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/cloudron_avatar')
+                .query({ access_token: token })
+                .attach('avatar', paths.CLOUDRON_DEFAULT_AVATAR_FILE)
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('get succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/cloudron_avatar')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.toString()).to.eql(fs.readFileSync(paths.CLOUDRON_DEFAULT_AVATAR_FILE, 'utf-8'));
+                    done(err);
+                });
+        });
+    });
+
+    describe('appstore listing config', function () {
+        it('get default succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.whitelist).to.eql(null);
+                    expect(res.body.blacklist).to.eql([]);
+
+                    done();
+                });
+        });
+
+        it('cannot set with no bl or wl', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('cannot set bad bl', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .send({ blacklist: [ 1 ] })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('cannot set bad wl', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .send({ whitelist: 4 })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('set bl succeeds', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .send({ blacklist: [ 'id1', 'id2' ] })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('get bl succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.blacklist).to.eql([ 'id1', 'id2' ]);
+                    expect(res.body.whitelist).to.be(undefined);
+
+                    done();
+                });
+        });
+
+        it('set wl succeeds', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .send({ whitelist: [ 'id1', 'id2' ] })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('get wl succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/branding/appstore_listing_config')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(200);
+                    expect(res.body.whitelist).to.eql([ 'id1', 'id2' ]);
+                    expect(res.body.blacklist).to.be(undefined);
+
+                    done();
+                });
+        });
+
+    });
+
+});

src/routes/test/cloudron-test.js

@@ -32,7 +32,7 @@ function setup(done) {
         server.start.bind(server),
         database._clear,
         settings._setApiServerOrigin.bind(null, 'http://localhost:6060'),
-        settings.setBackupConfig.bind(null, { provider: 'filesystem', backupFolder: '/tmp', format: 'tgz' })
+        settings.setBackupConfig.bind(null, { provider: 'filesystem', backupFolder: '/tmp', format: 'tgz', retentionPolicy: {} })
     ], done);
 }
 

src/routes/test/mail-test.js

@@ -63,8 +63,6 @@ function setup(done) {
                     .end(function (error, result) {
                         if (!result || result.statusCode !== 200) return retryCallback(new Error('Bad result'));
 
-                        console.dir(result.body);
-
                         if (!result.body.setup.active && result.body.setup.errorMessage === '' && result.body.adminFqdn) return retryCallback();
 
                         retryCallback(new Error('Not done yet: ' + JSON.stringify(result.body)));
@@ -124,46 +122,6 @@ describe('Mail API', function () {
     after(cleanup);
 
     describe('crud', function () {
-        it('cannot add non-existing domain', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: 'doesnotexist.com' })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(404);
-                    done();
-                });
-        });
-
-        it('domain must be a string', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: ['doesnotexist.com'] })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('can add domain', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
-        it('cannot add domain twice', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(409);
-                    done();
-                });
-        });
-
         it('cannot get non-existing domain', function (done) {
             superagent.get(SERVER_URL + '/api/v1/mail/doesnotexist.com')
                 .query({ access_token: token })
@@ -188,33 +146,6 @@ describe('Mail API', function () {
                     done();
                 });
         });
-
-        it('cannot delete non-existing domain', function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/doesnotexist.com')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(404);
-                    done();
-                });
-        });
-
-        it('cannot delete admin mail domain', function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + ADMIN_DOMAIN.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(409);
-                    done();
-                });
-        });
-
-        it('can delete admin mail domain', function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
-        });
     });
 
     describe('status', function () {
@@ -243,20 +174,13 @@ describe('Mail API', function () {
             mxDomain = DOMAIN_0.domain;
             dmarcDomain = '_dmarc.' + DOMAIN_0.domain;
 
-            superagent.post(SERVER_URL + '/api/v1/mail')
+            superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/enable')
                 .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
+                .send({ enabled: true })
                 .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
+                    expect(res.statusCode).to.equal(202);
 
-                    superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/enable')
-                        .query({ access_token: token })
-                        .send({ enabled: true })
-                        .end(function (err, res) {
-                            expect(res.statusCode).to.equal(202);
-
-                            done();
-                        });
+                    done();
                 });
         });
 
@@ -265,12 +189,7 @@ describe('Mail API', function () {
 
             dns.resolve = resolve;
 
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
+            done();
         });
 
         it('does not fail when dns errors', function (done) {
@@ -379,7 +298,7 @@ describe('Mail API', function () {
         it('succeeds with all different spf, dkim, dmarc, mx, ptr records', function (done) {
             clearDnsAnswerQueue();
 
-            dnsAnswerQueue[mxDomain].MX = [ { priority: '20', exchange: settings.mailFqdn() }, { priority: '30', exchange: settings.mailFqdn() } ];
+            dnsAnswerQueue[mxDomain].MX = [ { priority: '20', exchange: settings.mailFqdn() }, { priority: '10', exchange: 'some.other.server' } ];
             dnsAnswerQueue[dmarcDomain].TXT = [['v=DMARC2; p=reject; pct=100']];
             dnsAnswerQueue[dkimDomain].TXT = [['v=DKIM2; t=s; p=' + mail._readDkimPublicKeySync(DOMAIN_0.domain)]];
             dnsAnswerQueue[spfDomain].TXT = [['v=spf1 a:random.com ~all']];
@@ -396,7 +315,7 @@ describe('Mail API', function () {
 
                     expect(res.body.dns.dkim).to.be.an('object');
                     expect(res.body.dns.dkim.expected).to.eql('v=DKIM1; t=s; p=' + mail._readDkimPublicKeySync(DOMAIN_0.domain));
-                    expect(res.body.dns.dkim.status).to.eql(false);
+                    expect(res.body.dns.dkim.status).to.eql(true); // as long as p= matches we are good
                     expect(res.body.dns.dkim.value).to.eql('v=DKIM2; t=s; p=' + mail._readDkimPublicKeySync(DOMAIN_0.domain));
 
                     expect(res.body.dns.dmarc).to.be.an('object');
@@ -405,9 +324,9 @@ describe('Mail API', function () {
                     expect(res.body.dns.dmarc.value).to.eql('v=DMARC2; p=reject; pct=100');
 
                     expect(res.body.dns.mx).to.be.an('object');
-                    expect(res.body.dns.mx.status).to.eql(false);
+                    expect(res.body.dns.mx.status).to.eql(true);
                     expect(res.body.dns.mx.expected).to.eql('10 ' + settings.mailFqdn() + '.');
-                    expect(res.body.dns.mx.value).to.eql('20 ' + settings.mailFqdn() + '. 30 ' + settings.mailFqdn() + '.');
+                    expect(res.body.dns.mx.value).to.eql('20 ' + settings.mailFqdn() + '. 10 some.other.server.');
 
                     expect(res.body.dns.ptr).to.be.an('object');
                     expect(res.body.dns.ptr.expected).to.eql(settings.mailFqdn());
@@ -503,25 +422,6 @@ describe('Mail API', function () {
     });
 
     describe('mail from validation', function () {
-        before(function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
-        after(function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
-        });
-
         it('get mail from validation succeeds', function (done) {
             superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
                 .query({ access_token: token })
@@ -554,25 +454,6 @@ describe('Mail API', function () {
     });
 
     describe('catch_all', function () {
-        before(function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
-        after(function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
-        });
-
         it('get catch_all succeeds', function (done) {
             superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
                 .query({ access_token: token })
@@ -624,25 +505,6 @@ describe('Mail API', function () {
     });
 
     describe('mail relay', function () {
-        before(function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
-        after(function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
-        });
-
         it('get mail relay succeeds', function (done) {
             superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
                 .query({ access_token: token })
@@ -701,25 +563,6 @@ describe('Mail API', function () {
     });
 
     describe('mailboxes', function () {
-        before(function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
-        after(function (done) {
-            superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(204);
-                    done();
-                });
-        });
-
         it('add succeeds', function (done) {
             superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes')
                 .send({ name: MAILBOX_NAME, userId: userId })
@@ -757,7 +600,8 @@ describe('Mail API', function () {
                     expect(res.body.mailbox).to.be.an('object');
                     expect(res.body.mailbox.name).to.equal(MAILBOX_NAME);
                     expect(res.body.mailbox.ownerId).to.equal(userId);
-                    expect(res.body.mailbox.aliasTarget).to.equal(null);
+                    expect(res.body.mailbox.aliasName).to.equal(null);
+                    expect(res.body.mailbox.aliasDomain).to.equal(null);
                     expect(res.body.mailbox.domain).to.equal(DOMAIN_0.domain);
                     done();
                 });
@@ -772,7 +616,8 @@ describe('Mail API', function () {
                     expect(res.body.mailboxes[0]).to.be.an('object');
                     expect(res.body.mailboxes[0].name).to.equal(MAILBOX_NAME);
                     expect(res.body.mailboxes[0].ownerId).to.equal(userId);
-                    expect(res.body.mailboxes[0].aliasTarget).to.equal(null);
+                    expect(res.body.mailboxes[0].aliasName).to.equal(null);
+                    expect(res.body.mailboxes[0].aliasDomain).to.equal(null);
                     expect(res.body.mailboxes[0].domain).to.equal(DOMAIN_0.domain);
                     done();
                 });
@@ -803,69 +648,14 @@ describe('Mail API', function () {
     });
 
     describe('aliases', function () {
-        before(function (done) {
-            superagent.post(SERVER_URL + '/api/v1/mail')
-                .query({ access_token: token })
-                .send({ domain: DOMAIN_0.domain })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(201);
-                    done();
-                });
-        });
-
         after(function (done) {
             mail.removeMailboxes(DOMAIN_0.domain, function (error) {
                 if (error) return done(error);
-
-                superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                    .query({ access_token: token })
-                    .end(function (err, res) {
-                        expect(res.statusCode).to.equal(204);
-                        done();
-                    });
+                done();
             });
         });
 
-        it('set fails if aliases is missing', function (done) {
-            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + MAILBOX_NAME)
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('set fails if user does not exist', function (done) {
-            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + 'someuserdoesnotexist')
-                .send({ aliases: ['hello', 'there'] })
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(404);
-                    done();
-                });
-        });
-
-        it('set fails if aliases is the wrong type', function (done) {
-            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + MAILBOX_NAME)
-                .send({ aliases: 'hello, there' })
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('set fails if user is not enabled', function (done) {
-            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + MAILBOX_NAME)
-                .send({ aliases: ['hello', 'there'] })
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(404);
-                    done();
-                });
-        });
-
-        it('now add the mailbox', function (done) {
+        it('add the mailbox', function (done) {
             superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes')
                 .send({ name: MAILBOX_NAME, userId: userId })
                 .query({ access_token: token })
@@ -875,9 +665,38 @@ describe('Mail API', function () {
                 });
         });
 
+        it('set fails if aliases is missing', function (done) {
+            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/' + MAILBOX_NAME + '/aliases')
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('set fails if user does not exist', function (done) {
+            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/randomuser/aliases')
+                .send({ aliases: [{ name: 'hello', domain: DOMAIN_0.domain}, {name: 'there', domain: DOMAIN_0.domain}] })
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(404);
+                    done();
+                });
+        });
+
+        it('set fails if aliases is the wrong type', function (done) {
+            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/' + MAILBOX_NAME + '/aliases')
+                .send({ aliases: 'hello, there' })
+                .query({ access_token: token })
+                .end(function (err, res) {
+                    expect(res.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
         it('set succeeds', function (done) {
-            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + MAILBOX_NAME)
-                .send({ aliases: ['hello', 'there'] })
+            superagent.put(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/' + MAILBOX_NAME + '/aliases')
+                .send({ aliases: [{ name: 'hello', domain: DOMAIN_0.domain}, {name: 'there', domain: DOMAIN_0.domain}] })
                 .query({ access_token: token })
                 .end(function (err, res) {
                     expect(res.statusCode).to.equal(202);
@@ -886,35 +705,17 @@ describe('Mail API', function () {
         });
 
         it('get succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + MAILBOX_NAME)
+            superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/' + MAILBOX_NAME + '/aliases')
                 .query({ access_token: token })
                 .end(function (err, res) {
                     expect(res.statusCode).to.equal(200);
-                    expect(res.body.aliases).to.eql(['hello', 'there']);
-                    done();
-                });
-        });
-
-        it('listing succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(200);
-                    expect(res.body.aliases.length).to.eql(2);
-                    expect(res.body.aliases[0].name).to.equal('hello');
-                    expect(res.body.aliases[0].ownerId).to.equal(userId);
-                    expect(res.body.aliases[0].aliasTarget).to.equal(MAILBOX_NAME);
-                    expect(res.body.aliases[0].domain).to.equal(DOMAIN_0.domain);
-                    expect(res.body.aliases[1].name).to.equal('there');
-                    expect(res.body.aliases[1].ownerId).to.equal(userId);
-                    expect(res.body.aliases[1].aliasTarget).to.equal(MAILBOX_NAME);
-                    expect(res.body.aliases[1].domain).to.equal(DOMAIN_0.domain);
+                    expect(res.body.aliases).to.eql([{ name: 'hello', domain: DOMAIN_0.domain}, {name: 'there', domain: DOMAIN_0.domain}]);
                     done();
                 });
         });
 
         it('get fails if mailbox does not exist', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/aliases/' + 'someuserdoesnotexist')
+            superagent.get(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/mailboxes/somerandomuser/aliases')
                 .query({ access_token: token })
                 .end(function (err, res) {
                     expect(res.statusCode).to.equal(404);
@@ -924,30 +725,11 @@ describe('Mail API', function () {
     });
 
     describe('mailinglists', function () {
-        before(function (done) {
-            async.series([
-                function (done) {
-                    superagent.post(SERVER_URL + '/api/v1/mail')
-                        .query({ access_token: token })
-                        .send({ domain: DOMAIN_0.domain })
-                        .end(function (err, res) {
-                            expect(res.statusCode).to.equal(201);
-                            done();
-                        });
-                }
-            ], done);
-        });
-
         after(function (done) {
             mail.removeMailboxes(DOMAIN_0.domain, function (error) {
                 if (error) return done(error);
 
-                superagent.del(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain)
-                    .query({ access_token: token })
-                    .end(function (err, res) {
-                        expect(res.statusCode).to.equal(204);
-                        done();
-                    });
+                done();
             });
         });
 
@@ -982,7 +764,7 @@ describe('Mail API', function () {
 
         it('add succeeds', function (done) {
             superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/lists')
-                .send({ name: LIST_NAME, members: [ `admin2@${DOMAIN_0.domain}`, `${USERNAME}@${DOMAIN_0.domain}`] })
+                .send({ name: LIST_NAME, members: [ `admin2@${DOMAIN_0.domain}`, `${USERNAME}@${DOMAIN_0.domain}`], membersOnly: false })
                 .query({ access_token: token })
                 .end(function (err, res) {
                     expect(res.statusCode).to.equal(201);
@@ -992,7 +774,7 @@ describe('Mail API', function () {
 
         it('add twice fails', function (done) {
             superagent.post(SERVER_URL + '/api/v1/mail/' + DOMAIN_0.domain + '/lists')
-                .send({ name: LIST_NAME, members: [ `admin2@${DOMAIN_0.domain}`, `${USERNAME}@${DOMAIN_0.domain}`] })
+                .send({ name: LIST_NAME, members: [ `admin2@${DOMAIN_0.domain}`, `${USERNAME}@${DOMAIN_0.domain}`], membersOnly: false })
                 .query({ access_token: token })
                 .end(function (err, res) {
                     expect(res.statusCode).to.equal(409);
@@ -1017,9 +799,10 @@ describe('Mail API', function () {
                     expect(res.body.list).to.be.an('object');
                     expect(res.body.list.name).to.equal(LIST_NAME);
                     expect(res.body.list.ownerId).to.equal('admin');
-                    expect(res.body.list.aliasTarget).to.equal(null);
+                    expect(res.body.list.aliasName).to.equal(null);
                     expect(res.body.list.domain).to.equal(DOMAIN_0.domain);
                     expect(res.body.list.members).to.eql([ `admin2@${DOMAIN_0.domain}`, `superadmin@${DOMAIN_0.domain}` ]);
+                    expect(res.body.list.membersOnly).to.be(false);
                     done();
                 });
         });
@@ -1033,9 +816,10 @@ describe('Mail API', function () {
                     expect(res.body.lists.length).to.equal(1);
                     expect(res.body.lists[0].name).to.equal(LIST_NAME);
                     expect(res.body.lists[0].ownerId).to.equal('admin');
-                    expect(res.body.lists[0].aliasTarget).to.equal(null);
+                    expect(res.body.lists[0].aliasName).to.equal(null);
                     expect(res.body.lists[0].domain).to.equal(DOMAIN_0.domain);
                     expect(res.body.lists[0].members).to.eql([ `admin2@${DOMAIN_0.domain}`, `superadmin@${DOMAIN_0.domain}` ]);
+                    expect(res.body.lists[0].membersOnly).to.be(false);
                     done();
                 });
         });

src/routes/test/settings-test.js

@@ -9,8 +9,6 @@ var async = require('async'),
     constants = require('../../constants.js'),
     database = require('../../database.js'),
     expect = require('expect.js'),
-    fs = require('fs'),
-    paths = require('../../paths.js'),
     server = require('../../server.js'),
     superagent = require('superagent');
 
@@ -195,100 +193,6 @@ describe('Settings API', function () {
         });
     });
 
-    describe('cloudron_name', function () {
-        var name = 'foobar';
-
-        it('get default succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/settings/cloudron_name')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(200);
-                    expect(res.body.name).to.be.ok();
-                    done();
-                });
-        });
-
-        it('cannot set without name', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/settings/cloudron_name')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('cannot set empty name', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/settings/cloudron_name')
-                .query({ access_token: token })
-                .send({ name: '' })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('set succeeds', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/settings/cloudron_name')
-                .query({ access_token: token })
-                .send({ name: name })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(202);
-                    done();
-                });
-        });
-
-        it('get succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/settings/cloudron_name')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(200);
-                    expect(res.body.name).to.eql(name);
-                    done();
-                });
-        });
-    });
-
-    describe('cloudron_avatar', function () {
-        it('get default succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/settings/cloudron_avatar')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(200);
-                    expect(res.body).to.be.a(Buffer);
-                    done();
-                });
-        });
-
-        it('cannot set without data', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/settings/cloudron_avatar')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(400);
-                    done();
-                });
-        });
-
-        it('set succeeds', function (done) {
-            superagent.post(SERVER_URL + '/api/v1/settings/cloudron_avatar')
-                .query({ access_token: token })
-                .attach('avatar', paths.CLOUDRON_DEFAULT_AVATAR_FILE)
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(202);
-                    done();
-                });
-        });
-
-        it('get succeeds', function (done) {
-            superagent.get(SERVER_URL + '/api/v1/settings/cloudron_avatar')
-                .query({ access_token: token })
-                .end(function (err, res) {
-                    expect(res.statusCode).to.equal(200);
-                    expect(res.body.toString()).to.eql(fs.readFileSync(paths.CLOUDRON_DEFAULT_AVATAR_FILE, 'utf-8'));
-                    done(err);
-                });
-        });
-    });
-
     describe('time_zone', function () {
         it('succeeds', function (done) {
             superagent.get(SERVER_URL + '/api/v1/settings/time_zone')

src/routes/test/users-test.js

@@ -47,7 +47,6 @@ function setup(done) {
         server.start,
         database._clear,
         domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
-        mail.addDomain.bind(null, DOMAIN_0.domain)
     ], function (error) {
         expect(error).to.not.be.ok();
 
@@ -116,7 +115,7 @@ describe('Users API', function () {
                 });
         });
 
-        it('create admin', function (done) {
+        it('create owner', function (done) {
             superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
                 .query({ setupToken: 'somesetuptoken' })
                 .send({ username: USERNAME_0, password: PASSWORD, email: EMAIL_0 })

src/routes/users.js

@@ -71,6 +71,8 @@ function update(req, res, next) {
     if ('role' in req.body) {
         if (typeof req.body.role !== 'string') return next(new HttpError(400, 'role must be a string'));
         if (req.user.id === req.resource.id) return next(new HttpError(409, 'Cannot set role flag on self'));
+
+        if (users.compareRoles(req.user.role, req.body.role) < 0) return next(new HttpError(403, `role '${req.body.role}' is required but you are only '${req.user.role}'`));
     }
 
     if ('active' in req.body) {
@@ -78,7 +80,7 @@ function update(req, res, next) {
         if (req.user.id === req.resource.id) return next(new HttpError(409, 'Cannot set active flag on self'));
     }
 
-    if (users.compareRoles(req.user.role, req.body.role) < 0) return next(new HttpError(403, `role '${req.body.role}' is required but you are only '${req.user.role}'`));
+    if (users.compareRoles(req.user.role, req.resource.role) < 0) return next(new HttpError(403, `role '${req.resource.role}' is required but you are only '${req.user.role}'`));
 
     users.update(req.resource, req.body, auditSource.fromRequest(req), function (error) {
         if (error) return next(BoxError.toHttpError(error));

src/scheduler.js

@@ -23,8 +23,6 @@ function sync(callback) {
 
     callback = callback || NOOP_CALLBACK;
 
-    debug('sync: synchronizing global state with installed app state');
-
     apps.getAll(function (error, allApps) {
         if (error) return callback(error);
 
@@ -49,8 +47,6 @@ function sync(callback) {
                     return iteratorDone(); // nothing changed
                 }
 
-                debug(`sync: app ${app.fqdn} changed`);
-
                 stopJobs(app.id, appState, function (error) {
                     if (error) debug(`sync: error stopping jobs of ${app.fqdn} : ${error.message}`);
 
@@ -67,8 +63,6 @@ function sync(callback) {
                     iteratorDone();
                 });
             });
-
-            debug('sync: done');
         });
     });
 }
@@ -92,8 +86,6 @@ function stopJobs(appId, appState, callback) {
     assert.strictEqual(typeof appState, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    debug(`stopJobs: stopping jobs of ${appId}`);
-
     if (!appState) return callback();
 
     async.eachSeries(Object.keys(appState.schedulerConfig), function (taskName, iteratorDone) {
@@ -109,8 +101,6 @@ function createCronJobs(app, schedulerConfig) {
     assert.strictEqual(typeof app, 'object');
     assert(schedulerConfig && typeof schedulerConfig === 'object');
 
-    debug(`createCronJobs: creating cron jobs for app ${app.fqdn}`);
-
     var jobs = { };
 
     Object.keys(schedulerConfig).forEach(function (taskName) {
@@ -120,8 +110,6 @@ function createCronJobs(app, schedulerConfig) {
 
         var cronTime = (constants.TEST ? '*/5 ' : `${randomSecond} `) + task.schedule; // time ticks faster in tests
 
-        debug(`createCronJobs: ${app.fqdn} task ${taskName} scheduled at ${cronTime} with cmd ${task.command}`);
-
         var cronJob = new CronJob({
             cronTime: cronTime, // at this point, the pattern has been validated
             onTick: runTask.bind(null, app.id, taskName), //  put the app id in closure, so we don't use the outdated app object by mistake
@@ -143,8 +131,6 @@ function runTask(appId, taskName, callback) {
 
     callback = callback || NOOP_CALLBACK;
 
-    debug(`runTask: running task ${taskName} of ${appId}`);
-
     apps.get(appId, function (error, app) {
         if (error) return callback(error);
 
@@ -158,20 +144,13 @@ function runTask(appId, taskName, callback) {
         docker.inspectByName(containerName, function (err, data) {
             if (!err && data && data.State.Running === true) {
                 const jobStartTime = new Date(data.State.StartedAt); // iso 8601
-                if (new Date() - jobStartTime < JOB_MAX_TIME) {
-                    debug(`runTask: skipped task ${taskName} of app ${app.fqdn} since it was started at ${jobStartTime}`);
-                    return callback();
-                }
+                if (new Date() - jobStartTime < JOB_MAX_TIME) return callback();
             }
 
-            debug(`runTask: removing any old task ${taskName} of app ${app.fqdn}`);
-
             killContainer(containerName, function (error) {
                 if (error) return callback(error);
                 const cmd = gState[appId].schedulerConfig[taskName].command;
 
-                debug(`runTask: starting task ${taskName} of app ${app.fqdn} with cmd ${cmd}`);
-
                 // NOTE: if you change container name here, fix addons.js to return correct container names
                 docker.createSubcontainer(app, containerName, [ '/bin/sh', '-c', cmd ], { } /* options */, function (error, container) {
                     if (error) return callback(error);

src/server.js

@@ -30,7 +30,7 @@ function initializeExpressSync() {
     var QUERY_LIMIT = '1mb', // max size for json and urlencoded queries (see also client_max_body_size in nginx)
         FIELD_LIMIT = 2 * 1024 * 1024; // max fields that can appear in multipart
 
-    var REQUEST_TIMEOUT = 10000; // timeout for all requests (see also setTimeout on the httpServer)
+    var REQUEST_TIMEOUT = 20000; // timeout for all requests (see also setTimeout on the httpServer)
 
     var json = middleware.json({ strict: true, limit: QUERY_LIMIT }), // application/json
         urlencoded = middleware.urlencoded({ extended: false, limit: QUERY_LIMIT }); // application/x-www-form-urlencoded
@@ -79,6 +79,7 @@ function initializeExpressSync() {
     // to keep routes code short
     const password = routes.accesscontrol.passwordAuth;
     const token = routes.accesscontrol.tokenAuth;
+    const authorizeOwner = routes.accesscontrol.authorize(users.ROLE_OWNER);
     const authorizeAdmin = routes.accesscontrol.authorize(users.ROLE_ADMIN);
     const authorizeUserManager = routes.accesscontrol.authorize(users.ROLE_USER_MANAGER);
 
@@ -88,7 +89,7 @@ function initializeExpressSync() {
     router.post('/api/v1/cloudron/activate', routes.provision.activate);
     router.get ('/api/v1/cloudron/status', routes.provision.getStatus);
 
-    router.get ('/api/v1/cloudron/avatar', routes.settings.getCloudronAvatar); // this is a public alias for /api/v1/settings/cloudron_avatar
+    router.get ('/api/v1/cloudron/avatar', routes.branding.getCloudronAvatar); // this is a public alias for /api/v1/branding/cloudron_avatar
 
     // login/logout routes
     router.post('/api/v1/cloudron/login', password, routes.cloudron.login);
@@ -190,58 +191,69 @@ function initializeExpressSync() {
 
     // app routes
     router.get ('/api/v1/apps',          token, routes.apps.getApps);
-    router.get ('/api/v1/apps/:id',      token, authorizeAdmin, routes.apps.getApp);
-    router.get ('/api/v1/apps/:id/icon', token, routes.apps.getAppIcon);
+    router.get ('/api/v1/apps/:id',      token, authorizeAdmin, routes.apps.load, routes.apps.getApp);
+    router.get ('/api/v1/apps/:id/icon', token, routes.apps.load, routes.apps.getAppIcon);
 
-    router.post('/api/v1/apps/install',       token, authorizeAdmin, routes.apps.installApp);
-    router.post('/api/v1/apps/:id/uninstall', token, authorizeAdmin, routes.apps.uninstallApp);
+    router.post('/api/v1/apps/install',       token, authorizeAdmin, routes.apps.install);
+    router.post('/api/v1/apps/:id/uninstall', token, authorizeAdmin, routes.apps.load, routes.apps.uninstall);
 
-    router.post('/api/v1/apps/:id/configure/access_restriction', token, authorizeAdmin, routes.apps.setAccessRestriction);
-    router.post('/api/v1/apps/:id/configure/label', token, authorizeAdmin, routes.apps.setLabel);
-    router.post('/api/v1/apps/:id/configure/tags', token, authorizeAdmin, routes.apps.setTags);
-    router.post('/api/v1/apps/:id/configure/icon', token, authorizeAdmin, routes.apps.setIcon);
-    router.post('/api/v1/apps/:id/configure/memory_limit', token, authorizeAdmin, routes.apps.setMemoryLimit);
-    router.post('/api/v1/apps/:id/configure/cpu_shares', token, authorizeAdmin, routes.apps.setCpuShares);
-    router.post('/api/v1/apps/:id/configure/automatic_backup', token, authorizeAdmin, routes.apps.setAutomaticBackup);
-    router.post('/api/v1/apps/:id/configure/automatic_update', token, authorizeAdmin, routes.apps.setAutomaticUpdate);
-    router.post('/api/v1/apps/:id/configure/reverse_proxy', token, authorizeAdmin, routes.apps.setReverseProxyConfig);
-    router.post('/api/v1/apps/:id/configure/cert', token, authorizeAdmin, routes.apps.setCertificate);
-    router.post('/api/v1/apps/:id/configure/debug_mode', token, authorizeAdmin, routes.apps.setDebugMode);
-    router.post('/api/v1/apps/:id/configure/mailbox', token, authorizeAdmin, routes.apps.setMailbox);
-    router.post('/api/v1/apps/:id/configure/env', token, authorizeAdmin, routes.apps.setEnvironment);
-    router.post('/api/v1/apps/:id/configure/data_dir', token, authorizeAdmin, routes.apps.setDataDir);
-    router.post('/api/v1/apps/:id/configure/location', token, authorizeAdmin, routes.apps.setLocation);
+    router.post('/api/v1/apps/:id/configure/access_restriction', token, authorizeAdmin, routes.apps.load, routes.apps.setAccessRestriction);
+    router.post('/api/v1/apps/:id/configure/label', token, authorizeAdmin, routes.apps.load, routes.apps.setLabel);
+    router.post('/api/v1/apps/:id/configure/tags', token, authorizeAdmin, routes.apps.load, routes.apps.setTags);
+    router.post('/api/v1/apps/:id/configure/icon', token, authorizeAdmin, routes.apps.load, routes.apps.setIcon);
+    router.post('/api/v1/apps/:id/configure/memory_limit', token, authorizeAdmin, routes.apps.load, routes.apps.setMemoryLimit);
+    router.post('/api/v1/apps/:id/configure/cpu_shares', token, authorizeAdmin, routes.apps.load, routes.apps.setCpuShares);
+    router.post('/api/v1/apps/:id/configure/automatic_backup', token, authorizeAdmin, routes.apps.load, routes.apps.setAutomaticBackup);
+    router.post('/api/v1/apps/:id/configure/automatic_update', token, authorizeAdmin, routes.apps.load, routes.apps.setAutomaticUpdate);
+    router.post('/api/v1/apps/:id/configure/reverse_proxy', token, authorizeAdmin, routes.apps.load, routes.apps.setReverseProxyConfig);
+    router.post('/api/v1/apps/:id/configure/cert', token, authorizeAdmin, routes.apps.load, routes.apps.setCertificate);
+    router.post('/api/v1/apps/:id/configure/debug_mode', token, authorizeAdmin, routes.apps.load, routes.apps.setDebugMode);
+    router.post('/api/v1/apps/:id/configure/mailbox', token, authorizeAdmin, routes.apps.load, routes.apps.setMailbox);
+    router.post('/api/v1/apps/:id/configure/env', token, authorizeAdmin, routes.apps.load, routes.apps.setEnvironment);
+    router.post('/api/v1/apps/:id/configure/data_dir', token, authorizeAdmin, routes.apps.load, routes.apps.setDataDir);
+    router.post('/api/v1/apps/:id/configure/location', token, authorizeAdmin, routes.apps.load, routes.apps.setLocation);
+    router.post('/api/v1/apps/:id/configure/binds', token, authorizeAdmin, routes.apps.load, routes.apps.setBinds);
 
-    router.post('/api/v1/apps/:id/repair',    token, authorizeAdmin, routes.apps.repairApp);
-    router.post('/api/v1/apps/:id/update',    token, authorizeAdmin, routes.apps.updateApp);
-    router.post('/api/v1/apps/:id/restore',   token, authorizeAdmin, routes.apps.restoreApp);
-    router.post('/api/v1/apps/:id/import',    token, authorizeAdmin, routes.apps.importApp);
-    router.post('/api/v1/apps/:id/backup',    token, authorizeAdmin, routes.apps.backupApp);
-    router.get ('/api/v1/apps/:id/backups',   token, authorizeAdmin, routes.apps.listBackups);
-    router.post('/api/v1/apps/:id/stop',      token, authorizeAdmin, routes.apps.stopApp);
-    router.post('/api/v1/apps/:id/start',     token, authorizeAdmin, routes.apps.startApp);
-    router.post('/api/v1/apps/:id/restart',   token, authorizeAdmin, routes.apps.restartApp);
-    router.get ('/api/v1/apps/:id/logstream', token, authorizeAdmin, routes.apps.getLogStream);
-    router.get ('/api/v1/apps/:id/logs',      token, authorizeAdmin, routes.apps.getLogs);
-    router.get ('/api/v1/apps/:id/exec',      token, authorizeAdmin, routes.apps.exec);
+    router.post('/api/v1/apps/:id/repair',    token, authorizeAdmin, routes.apps.load, routes.apps.repair);
+    router.post('/api/v1/apps/:id/update',    token, authorizeAdmin, routes.apps.load, routes.apps.update);
+    router.post('/api/v1/apps/:id/restore',   token, authorizeAdmin, routes.apps.load, routes.apps.restore);
+    router.post('/api/v1/apps/:id/import',    token, authorizeAdmin, routes.apps.load, routes.apps.importApp);
+    router.post('/api/v1/apps/:id/backup',    token, authorizeAdmin, routes.apps.load, routes.apps.backup);
+    router.get ('/api/v1/apps/:id/backups',   token, authorizeAdmin, routes.apps.load, routes.apps.listBackups);
+    router.post('/api/v1/apps/:id/start',     token, authorizeAdmin, routes.apps.load, routes.apps.start);
+    router.post('/api/v1/apps/:id/stop',      token, authorizeAdmin, routes.apps.load, routes.apps.stop);
+    router.post('/api/v1/apps/:id/restart',   token, authorizeAdmin, routes.apps.load, routes.apps.restart);
+    router.get ('/api/v1/apps/:id/logstream', token, authorizeAdmin, routes.apps.load, routes.apps.getLogStream);
+    router.get ('/api/v1/apps/:id/logs',      token, authorizeAdmin, routes.apps.load, routes.apps.getLogs);
+    router.get ('/api/v1/apps/:id/exec',      token, authorizeAdmin, routes.apps.load, routes.apps.exec);
     // websocket cannot do bearer authentication
-    router.get ('/api/v1/apps/:id/execws',    routes.accesscontrol.websocketAuth.bind(null, users.ROLE_ADMIN), routes.apps.execWebSocket);
-    router.post('/api/v1/apps/:id/clone',     token, authorizeAdmin, routes.apps.cloneApp);
-    router.get ('/api/v1/apps/:id/download',  token, authorizeAdmin, routes.apps.downloadFile);
-    router.post('/api/v1/apps/:id/upload',    token, authorizeAdmin, multipart, routes.apps.uploadFile);
+    router.get ('/api/v1/apps/:id/execws',    routes.accesscontrol.websocketAuth.bind(null, users.ROLE_ADMIN), routes.apps.load, routes.apps.execWebSocket);
+    router.post('/api/v1/apps/:id/clone',     token, authorizeAdmin, routes.apps.load, routes.apps.clone);
+    router.get ('/api/v1/apps/:id/download',  token, authorizeAdmin, routes.apps.load, routes.apps.downloadFile);
+    router.post('/api/v1/apps/:id/upload',    token, authorizeAdmin, multipart, routes.apps.load, routes.apps.uploadFile);
+
+    router.get ('/api/v1/branding/:setting', token, authorizeOwner, routes.branding.get);
+    router.post('/api/v1/branding/:setting', token, authorizeOwner,  (req, res, next) => {
+        return req.params.setting === 'cloudron_avatar' ? multipart(req, res, next) : next();
+    }, routes.branding.set);
 
     // settings routes (these are for the settings tab - avatar & name have public routes for normal users. see above)
     router.get ('/api/v1/settings/:setting', token, authorizeAdmin, routes.settings.get);
+    router.post('/api/v1/settings/backup_config', token, authorizeOwner, routes.settings.setBackupConfig);
     router.post('/api/v1/settings/:setting', token, authorizeAdmin, (req, res, next) => {
         return req.params.setting === 'cloudron_avatar' ? multipart(req, res, next) : next();
     }, routes.settings.set);
 
     // email routes
-    router.get('/api/v1/mailserver/:pathname', token, authorizeAdmin, routes.mailserver.proxy);
+    router.get('/api/v1/mailserver/:pathname', token, (req, res, next) => {
+        // some routes are more special than others
+        if (req.params.pathname === 'eventlog' || req.params.pathname === 'clear_eventlog') {
+            return authorizeOwner(req, res, next);
+        }
+        authorizeAdmin(req, res, next);
+    }, routes.mailserver.proxy);
 
     router.get ('/api/v1/mail/:domain',       token, authorizeAdmin, routes.mail.getDomain);
-    router.post('/api/v1/mail',               token, authorizeAdmin, routes.mail.addDomain);
-    router.del ('/api/v1/mail/:domain',       token, authorizeAdmin, routes.mail.removeDomain);
     router.get ('/api/v1/mail/:domain/status',       token, authorizeAdmin, routes.mail.getStatus);
     router.post('/api/v1/mail/:domain/mail_from_validation', token, authorizeAdmin, routes.mail.setMailFromValidation);
     router.post('/api/v1/mail/:domain/catch_all',  token, authorizeAdmin, routes.mail.setCatchAllAddress);
@@ -254,9 +266,9 @@ function initializeExpressSync() {
     router.post('/api/v1/mail/:domain/mailboxes',  token, authorizeAdmin, routes.mail.addMailbox);
     router.post('/api/v1/mail/:domain/mailboxes/:name',  token, authorizeAdmin, routes.mail.updateMailbox);
     router.del ('/api/v1/mail/:domain/mailboxes/:name',  token, authorizeAdmin, routes.mail.removeMailbox);
-    router.get ('/api/v1/mail/:domain/aliases', token, authorizeAdmin, routes.mail.listAliases);
-    router.get ('/api/v1/mail/:domain/aliases/:name', token, authorizeAdmin, routes.mail.getAliases);
-    router.put ('/api/v1/mail/:domain/aliases/:name', token, authorizeAdmin, routes.mail.setAliases);
+    router.get ('/api/v1/mail/:domain/mailboxes/:name/aliases', token, authorizeAdmin, routes.mail.getAliases);
+    router.put ('/api/v1/mail/:domain/mailboxes/:name/aliases', token, authorizeAdmin, routes.mail.setAliases);
+
     router.get ('/api/v1/mail/:domain/lists', token, authorizeAdmin, routes.mail.getLists);
     router.post('/api/v1/mail/:domain/lists', token, authorizeAdmin, routes.mail.addList);
     router.get ('/api/v1/mail/:domain/lists/:name', token, authorizeAdmin, routes.mail.getList);

src/settings.js

@@ -51,6 +51,8 @@ exports = module.exports = {
     setFooter: setFooter,
 
     getAppstoreListingConfig: getAppstoreListingConfig,
+    setAppstoreListingConfig: setAppstoreListingConfig,
+
     getSupportConfig: getSupportConfig,
 
     provider: provider,
@@ -140,10 +142,10 @@ let gDefaults = (function () {
     result[exports.CLOUDRON_TOKEN_KEY] = '';
     result[exports.BACKUP_CONFIG_KEY] = {
         provider: 'filesystem',
-        key: '',
         backupFolder: '/var/backups',
         format: 'tgz',
-        retentionSecs: 2 * 24 * 60 * 60, // 2 days
+        encryption: null,
+        retentionPolicy: { keepWithinSecs: 2 * 24 * 60 * 60 }, // 2 days
         intervalSecs: 24 * 60 * 60 // ~1 day
     };
     result[exports.PLATFORM_CONFIG_KEY] = {};
@@ -384,7 +386,7 @@ function getBackupConfig(callback) {
         if (error && error.reason === BoxError.NOT_FOUND) return callback(null, gDefaults[exports.BACKUP_CONFIG_KEY]);
         if (error) return callback(error);
 
-        callback(null, JSON.parse(value)); // provider, token, key, region, prefix, bucket
+        callback(null, JSON.parse(value)); // provider, token, password, region, prefix, bucket
     });
 }
 
@@ -392,14 +394,19 @@ function setBackupConfig(backupConfig, callback) {
     assert.strictEqual(typeof backupConfig, 'object');
     assert.strictEqual(typeof callback, 'function');
 
-    getBackupConfig(function (error, curentConfig) {
+    getBackupConfig(function (error, currentConfig) {
         if (error) return callback(error);
 
-        backups.injectPrivateFields(backupConfig, curentConfig);
+        backups.injectPrivateFields(backupConfig, currentConfig);
 
         backups.testConfig(backupConfig, function (error) {
             if (error) return callback(error);
 
+            if ('password' in backupConfig) { // user set password
+                backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.password);
+                delete backupConfig.password;
+            }
+
             backups.cleanupCacheFilesSync();
 
             settingsdb.set(exports.BACKUP_CONFIG_KEY, JSON.stringify(backupConfig), function (error) {
@@ -421,7 +428,7 @@ function setBackupCredentials(credentials, callback) {
         if (error) return callback(error);
 
         // preserve these fields
-        const extra = _.pick(currentConfig, 'retentionSecs', 'intervalSecs', 'copyConcurrency', 'syncConcurrency');
+        const extra = _.pick(currentConfig, 'retentionPolicy', 'intervalSecs', 'copyConcurrency', 'syncConcurrency');
 
         const backupConfig = _.extend({}, credentials, extra);
 
@@ -577,6 +584,20 @@ function getAppstoreListingConfig(callback) {
     });
 }
 
+function setAppstoreListingConfig(listingConfig, callback) {
+    assert.strictEqual(typeof listingConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    settingsdb.set(exports.APPSTORE_LISTING_CONFIG_KEY, JSON.stringify(listingConfig), function (error) {
+        if (error) return callback(error);
+
+        notifyChange(exports.APPSTORE_LISTING_CONFIG_KEY, listingConfig);
+
+        callback(null);
+    });
+}
+
+
 function getSupportConfig(callback) {
     assert.strictEqual(typeof callback, 'function');
 

src/storage/gcs.js

@@ -21,8 +21,8 @@ exports = module.exports = {
 
 var assert = require('assert'),
     async = require('async'),
-    backups = require('../backups.js'),
     BoxError = require('../boxerror.js'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:storage/gcs'),
     EventEmitter = require('events'),
     GCS = require('@google-cloud/storage').Storage,
@@ -254,10 +254,10 @@ function testConfig(apiConfig, callback) {
 }
 
 function removePrivateFields(apiConfig) {
-    apiConfig.credentials.private_key = backups.SECRET_PLACEHOLDER;
+    apiConfig.credentials.private_key = constants.SECRET_PLACEHOLDER;
     return apiConfig;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.credentials.private_key === backups.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
+    if (newConfig.credentials.private_key === constants.SECRET_PLACEHOLDER && currentConfig.credentials) newConfig.credentials.private_key = currentConfig.credentials.private_key;
 }

src/storage/interface.js

@@ -32,13 +32,13 @@ var assert = require('assert'),
     EventEmitter = require('events');
 
 function removePrivateFields(apiConfig) {
-    // in-place removal of tokens and api keys with domains.SECRET_PLACEHOLDER
+    // in-place removal of tokens and api keys with constants.SECRET_PLACEHOLDER
     return apiConfig;
 }
 
 // eslint-disable-next-line no-unused-vars
 function injectPrivateFields(newConfig, currentConfig) {
-    // in-place injection of tokens and api keys which came in with domains.SECRET_PLACEHOLDER
+    // in-place injection of tokens and api keys which came in with constants.SECRET_PLACEHOLDER
 }
 
 function upload(apiConfig, backupFilePath, sourceStream, callback) {

src/storage/s3.js

@@ -22,15 +22,16 @@ exports = module.exports = {
 var assert = require('assert'),
     async = require('async'),
     AWS = require('aws-sdk'),
-    backups = require('../backups.js'),
     BoxError = require('../boxerror.js'),
     chunk = require('lodash.chunk'),
+    constants = require('../constants.js'),
     debug = require('debug')('box:storage/s3'),
     EventEmitter = require('events'),
     https = require('https'),
     PassThrough = require('stream').PassThrough,
     path = require('path'),
-    S3BlockReadStream = require('s3-block-read-stream');
+    S3BlockReadStream = require('s3-block-read-stream'),
+    _ = require('underscore');
 
 // test only
 var originalAWS;
@@ -429,7 +430,7 @@ function testConfig(apiConfig, callback) {
             Body: 'testcontent'
         };
 
-        var s3 = new AWS.S3(credentials);
+        var s3 = new AWS.S3(_.omit(credentials, 'retryDelayOptions', 'maxRetries'));
         s3.putObject(params, function (error) {
             if (error) return callback(new BoxError(BoxError.EXTERNAL_ERROR, error.message || error.code)); // DO sets 'code'
 
@@ -448,10 +449,10 @@ function testConfig(apiConfig, callback) {
 }
 
 function removePrivateFields(apiConfig) {
-    apiConfig.secretAccessKey = backups.SECRET_PLACEHOLDER;
+    apiConfig.secretAccessKey = constants.SECRET_PLACEHOLDER;
     return apiConfig;
 }
 
 function injectPrivateFields(newConfig, currentConfig) {
-    if (newConfig.secretAccessKey === backups.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
+    if (newConfig.secretAccessKey === constants.SECRET_PLACEHOLDER) newConfig.secretAccessKey = currentConfig.secretAccessKey;
 }

src/system.js

@@ -40,7 +40,7 @@ function getDisks(callback) {
             const ext4Disks = values[0].filter((r) => r.type === 'ext4').sort((a, b) => a.mountpoint.localeCompare(b.mountpoint));
 
             const disks = {
-                disks: ext4Disks, // root disk is first
+                disks: ext4Disks, // root disk is first. { filesystem, type, size, used, avialable, capacity, mountpoint }
                 boxDataDisk: values[1].filesystem,
                 mailDataDisk: values[1].filesystem,
                 platformDataDisk: values[2].filesystem,

src/test/apps-test.js

@@ -15,6 +15,7 @@ var appdb = require('../appdb.js'),
     groupdb = require('../groupdb.js'),
     groups = require('../groups.js'),
     hat = require('../hat.js'),
+    settings = require('../settings.js'),
     userdb = require('../userdb.js');
 
 let AUDIT_SOURCE = { ip: '1.2.3.4' };
@@ -173,6 +174,7 @@ describe('Apps', function () {
         async.series([
             database.initialize,
             database._clear,
+            settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             domains.add.bind(null, DOMAIN_1.domain, DOMAIN_1, AUDIT_SOURCE),
             userdb.add.bind(null, ADMIN_0.id, ADMIN_0),
@@ -208,6 +210,13 @@ describe('Apps', function () {
             expect(apps._validatePortBindings({ port: 1567 }, { tcpPorts: { port3: null } })).to.be.an(Error);
         });
 
+        it('does not allow reserved ports', function () {
+            expect(apps._validatePortBindings({ port: 443 }, { tcpPorts: { port: 5000 } })).to.be.an(Error);
+            expect(apps._validatePortBindings({ port: 50000 }, { tcpPorts: { port: 5000 } })).to.be.an(Error);
+            expect(apps._validatePortBindings({ port: 51000 }, { tcpPorts: { port: 5000 } })).to.be.an(Error);
+            expect(apps._validatePortBindings({ port: 50100 }, { tcpPorts: { port: 5000 } })).to.be.an(Error);
+        });
+
         it('allows valid bindings', function () {
             expect(apps._validatePortBindings({ port: 1024 }, { tcpPorts: { port: 5000 } })).to.be(null);
 

src/test/apptask-test.js

@@ -18,6 +18,7 @@ var addons = require('../addons.js'),
     net = require('net'),
     nock = require('nock'),
     paths = require('../paths.js'),
+    settings = require('../settings.js'),
     userdb = require('../userdb.js'),
     _ = require('underscore');
 
@@ -119,6 +120,7 @@ describe('apptask', function () {
         async.series([
             database.initialize,
             database._clear,
+            settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             userdb.add.bind(null, ADMIN.id, ADMIN),
             appdb.add.bind(null, APP.id, APP.appStoreId, APP.manifest, APP.location, APP.domain, APP.portBindings, APP)
@@ -184,27 +186,6 @@ describe('apptask', function () {
         });
     });
 
-    it('allocate OAuth credentials', function (done) {
-        addons._setupOauth(APP, {}, function (error) {
-            expect(error).to.be(null);
-            done();
-        });
-    });
-
-    it('remove OAuth credentials', function (done) {
-        addons._teardownOauth(APP, {}, function (error) {
-            expect(error).to.be(null);
-            done();
-        });
-    });
-
-    it('remove OAuth credentials twice succeeds', function (done) {
-        addons._teardownOauth(APP, {}, function (error) {
-            expect(!error).to.be.ok();
-            done();
-        });
-    });
-
     it('barfs on empty manifest', function (done) {
         var badApp = _.extend({ }, APP);
         badApp.manifest = { };

src/test/backups-test.js

@@ -17,6 +17,7 @@ var async = require('async'),
     fs = require('fs'),
     os = require('os'),
     mkdirp = require('mkdirp'),
+    moment = require('moment'),
     path = require('path'),
     rimraf = require('rimraf'),
     settings = require('../settings.js'),
@@ -68,6 +69,69 @@ function cleanupBackups(callback) {
     });
 }
 
+describe('retention policy', function () {
+    it('always keeps if reason is set', function () {
+        let backup = { keepReason: 'somereason' };
+        backups._applyBackupRetentionPolicy([backup], { keepWithinSecs: 1 });
+        expect(backup.keepReason).to.be('somereason');
+    });
+
+    it('always keeps forever policy', function () {
+        let backup = { creationTime: new Date() };
+        backups._applyBackupRetentionPolicy([backup], { keepWithinSecs: -1 });
+        expect(backup.keepReason).to.be('keepWithinSecs');
+    });
+
+    it('preserveSecs takes precedence', function () {
+        let backup = { creationTime: new Date(), preserveSecs: 3000 };
+        backups._applyBackupRetentionPolicy([backup], { keepWithinSecs: 1 });
+        expect(backup.keepReason).to.be('preserveSecs');
+    });
+
+    it('1 daily', function () {
+        let b = [
+            { id: '1', creationTime: moment().toDate() },
+            { id: '2', creationTime: moment().subtract(3, 'h').toDate() },
+            { id: '3', creationTime: moment().subtract(20, 'h').toDate() },
+            { id: '4', creationTime: moment().subtract(5, 'd').toDate() }
+        ];
+        backups._applyBackupRetentionPolicy(b, { keepDaily: 1 });
+        expect(b[0].keepReason).to.be('keepDaily');
+        expect(b[1].keepReason).to.be(undefined);
+        expect(b[2].keepReason).to.be(undefined);
+        expect(b[3].keepReason).to.be(undefined);
+    });
+
+    it('2 daily, 1 weekly', function () {
+        let b = [
+            { id: '1', creationTime: moment().toDate() },
+            { id: '2', creationTime: moment().subtract(3, 'h').toDate() },
+            { id: '3', creationTime: moment().subtract(26, 'h').toDate() },
+            { id: '4', creationTime: moment().subtract(5, 'd').toDate() }
+        ];
+        backups._applyBackupRetentionPolicy(b, { keepDaily: 2, keepWeekly: 1 });
+        expect(b[0].keepReason).to.be('keepDaily'); // today
+        expect(b[1].keepReason).to.be('keepWeekly');
+        expect(b[2].keepReason).to.be('keepDaily'); // yesterday
+        expect(b[3].keepReason).to.be(undefined);
+    });
+
+    it('2 daily, 6 monthly, 1 yearly', function () {
+        let b = [
+            { id: '1', creationTime: moment().toDate() },
+            { id: '2', creationTime: moment().subtract(3, 'h').toDate() },
+            { id: '3', creationTime: moment().subtract(26, 'h').toDate() },
+            { id: '4', creationTime: moment().subtract(5, 'd').toDate() }
+        ];
+        backups._applyBackupRetentionPolicy(b, { keepDaily: 2, keepMonthly: 6, keepYearly: 1 });
+        expect(b[0].keepReason).to.be('keepDaily'); // today
+        expect(b[1].keepReason).to.be('keepMonthly');
+        expect(b[2].keepReason).to.be('keepDaily'); // yesterday
+        expect(b[3].keepReason).to.be('keepYearly');
+    });
+
+});
+
 describe('backups', function () {
     before(function (done) {
         const BACKUP_DIR = path.join(os.tmpdir(), 'cloudron-backup-test');
@@ -78,9 +142,9 @@ describe('backups', function () {
             database._clear,
             settings.setBackupConfig.bind(null, {
                 provider: 'filesystem',
-                key: 'enckey',
+                password: 'supersecret',
                 backupFolder: BACKUP_DIR,
-                retentionSecs: 1,
+                retentionPolicy: { keepWithinSecs: 1 },
                 format: 'tgz'
             })
         ], done);
@@ -95,7 +159,8 @@ describe('backups', function () {
     describe('cleanup', function () {
         var BACKUP_0 = {
             id: 'backup-box-0',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_BOX,
             dependsOn: [ 'backup-app-00', 'backup-app-01' ],
             manifest: null,
@@ -104,7 +169,8 @@ describe('backups', function () {
 
         var BACKUP_0_APP_0 = {
             id: 'backup-app-00',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_APP,
             dependsOn: [],
             manifest: null,
@@ -113,7 +179,8 @@ describe('backups', function () {
 
         var BACKUP_0_APP_1 = {
             id: 'backup-app-01',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_APP,
             dependsOn: [],
             manifest: null,
@@ -122,7 +189,8 @@ describe('backups', function () {
 
         var BACKUP_1 = {
             id: 'backup-box-1',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_BOX,
             dependsOn: [ 'backup-app-10', 'backup-app-11' ],
             manifest: null,
@@ -131,7 +199,8 @@ describe('backups', function () {
 
         var BACKUP_1_APP_0 = {
             id: 'backup-app-10',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_APP,
             dependsOn: [],
             manifest: null,
@@ -140,7 +209,8 @@ describe('backups', function () {
 
         var BACKUP_1_APP_1 = {
             id: 'backup-app-11',
-            version: '1.0.0',
+            encryptionVersion: null,
+            packageVersion: '1.0.0',
             type: backupdb.BACKUP_TYPE_APP,
             dependsOn: [],
             manifest: null,
@@ -271,25 +341,26 @@ describe('backups', function () {
     describe('filesystem', function () {
         var backupInfo1;
 
-        var gBackupConfig = {
+        var backupConfig = {
             provider: 'filesystem',
             backupFolder: path.join(os.tmpdir(), 'backups-test-filesystem'),
-            format: 'tgz'
+            format: 'tgz',
+            retentionPolicy: { keepWithinSecs: 10000 }
         };
 
         before(function (done) {
-            rimraf.sync(gBackupConfig.backupFolder);
+            rimraf.sync(backupConfig.backupFolder);
 
             done();
         });
 
         after(function (done) {
-            rimraf.sync(gBackupConfig.backupFolder);
+            rimraf.sync(backupConfig.backupFolder);
             done();
         });
 
         it('fails to set backup config for non-existing folder', function (done) {
-            settings.setBackupConfig(gBackupConfig, function (error) {
+            settings.setBackupConfig(backupConfig, function (error) {
                 expect(error).to.be.a(BoxError);
                 expect(error.reason).to.equal(BoxError.BAD_FIELD);
 
@@ -298,9 +369,9 @@ describe('backups', function () {
         });
 
         it('succeeds to set backup config', function (done) {
-            mkdirp.sync(gBackupConfig.backupFolder);
+            mkdirp.sync(backupConfig.backupFolder);
 
-            settings.setBackupConfig(gBackupConfig, function (error) {
+            settings.setBackupConfig(backupConfig, function (error) {
                 expect(error).to.be(null);
 
                 done();
@@ -313,8 +384,8 @@ describe('backups', function () {
 
             createBackup(function (error, result) {
                 expect(error).to.be(null);
-                expect(fs.statSync(path.join(gBackupConfig.backupFolder, 'snapshot/box.tar.gz')).nlink).to.be(2); // hard linked to a rotated backup
-                expect(fs.statSync(path.join(gBackupConfig.backupFolder, `${result.id}.tar.gz`)).nlink).to.be(2);
+                expect(fs.statSync(path.join(backupConfig.backupFolder, 'snapshot/box.tar.gz')).nlink).to.be(2); // hard linked to a rotated backup
+                expect(fs.statSync(path.join(backupConfig.backupFolder, `${result.id}.tar.gz`)).nlink).to.be(2);
 
                 backupInfo1 = result;
 
@@ -328,9 +399,9 @@ describe('backups', function () {
 
             createBackup(function (error, result) {
                 expect(error).to.be(null);
-                expect(fs.statSync(path.join(gBackupConfig.backupFolder, 'snapshot/box.tar.gz')).nlink).to.be(2); // hard linked to a rotated backup
-                expect(fs.statSync(path.join(gBackupConfig.backupFolder, `${result.id}.tar.gz`)).nlink).to.be(2); // hard linked to new backup
-                expect(fs.statSync(path.join(gBackupConfig.backupFolder, `${backupInfo1.id}.tar.gz`)).nlink).to.be(1); // not hard linked anymore
+                expect(fs.statSync(path.join(backupConfig.backupFolder, 'snapshot/box.tar.gz')).nlink).to.be(2); // hard linked to a rotated backup
+                expect(fs.statSync(path.join(backupConfig.backupFolder, `${result.id}.tar.gz`)).nlink).to.be(2); // hard linked to new backup
+                expect(fs.statSync(path.join(backupConfig.backupFolder, `${backupInfo1.id}.tar.gz`)).nlink).to.be(1); // not hard linked anymore
 
                 done();
             });

src/test/database-test.js

@@ -40,7 +40,8 @@ var USER_0 = {
     twoFactorAuthenticationSecret: '',
     role: 'user',
     active: true,
-    source: ''
+    source: '',
+    resetTokenCreationTime: Date.now()
 };
 
 var USER_1 = {
@@ -58,7 +59,8 @@ var USER_1 = {
     twoFactorAuthenticationSecret: '',
     role: 'user',
     active: true,
-    source: ''
+    source: '',
+    resetTokenCreationTime: Date.now()
 };
 
 var USER_2 = {
@@ -76,7 +78,8 @@ var USER_2 = {
     twoFactorAuthenticationSecret: '',
     role: 'user',
     active: true,
-    source: ''
+    source: '',
+    resetTokenCreationTime: Date.now()
 };
 
 const DOMAIN_0 = {
@@ -415,7 +418,9 @@ describe('database', function () {
             dataDir: null,
             tags: [],
             label: null,
-            taskId: null
+            taskId: null,
+            binds: {},
+            servicesConfig: {}
         };
 
         it('cannot delete referenced domain', function (done) {
@@ -886,7 +891,9 @@ describe('database', function () {
             dataDir: null,
             tags: [],
             label: null,
-            taskId: null
+            taskId: null,
+            binds: {},
+            servicesConfig: {}
         };
 
         var APP_1 = {
@@ -917,7 +924,9 @@ describe('database', function () {
             dataDir: null,
             tags: [],
             label: null,
-            taskId: null
+            taskId: null,
+            binds: {},
+            servicesConfig: {}
         };
 
         before(function (done) {
@@ -980,7 +989,7 @@ describe('database', function () {
             appdb.get(APP_0.id, function (error, result) {
                 expect(error).to.be(null);
                 expect(result).to.be.an('object');
-                expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime'])).to.be.eql(APP_0);
+                expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime', 'resetTokenCreationTime'])).to.be.eql(APP_0);
                 done();
             });
         });
@@ -1020,7 +1029,7 @@ describe('database', function () {
                 appdb.get(APP_0.id, function (error, result) {
                     expect(error).to.be(null);
                     expect(result).to.be.an('object');
-                    expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime'])).to.be.eql(APP_0);
+                    expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime','resetTokenCreationTime'])).to.be.eql(APP_0);
                     done();
                 });
             });
@@ -1030,7 +1039,7 @@ describe('database', function () {
             appdb.getByHttpPort(APP_0.httpPort, function (error, result) {
                 expect(error).to.be(null);
                 expect(result).to.be.an('object');
-                expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime'])).to.be.eql(APP_0);
+                expect(_.omit(result, ['creationTime', 'updateTime', 'ts', 'healthTime','resetTokenCreationTime'])).to.be.eql(APP_0);
                 done();
             });
         });
@@ -1055,8 +1064,8 @@ describe('database', function () {
                 expect(error).to.be(null);
                 expect(result).to.be.an(Array);
                 expect(result.length).to.be(2);
-                expect(_.omit(result[0], ['creationTime', 'updateTime','ts', 'healthTime'])).to.be.eql(APP_0);
-                expect(_.omit(result[1], ['creationTime', 'updateTime','ts', 'healthTime'])).to.be.eql(APP_1);
+                expect(_.omit(result[0], ['creationTime', 'updateTime','ts', 'healthTime', 'resetTokenCreationTime'])).to.be.eql(APP_0);
+                expect(_.omit(result[1], ['creationTime', 'updateTime','ts', 'healthTime', 'resetTokenCreationTime'])).to.be.eql(APP_1);
                 done();
             });
         });
@@ -1315,7 +1324,8 @@ describe('database', function () {
         it('add succeeds', function (done) {
             var backup = {
                 id: 'backup-box',
-                version: '1.0.0',
+                encryptionVersion: 2,
+                packageVersion: '1.0.0',
                 type: backupdb.BACKUP_TYPE_BOX,
                 dependsOn: [ 'dep1' ],
                 manifest: null,
@@ -1331,7 +1341,8 @@ describe('database', function () {
         it('get succeeds', function (done) {
             backupdb.get('backup-box', function (error, result) {
                 expect(error).to.be(null);
-                expect(result.version).to.be('1.0.0');
+                expect(result.encryptionVersion).to.be(2);
+                expect(result.packageVersion).to.be('1.0.0');
                 expect(result.type).to.be(backupdb.BACKUP_TYPE_BOX);
                 expect(result.creationTime).to.be.a(Date);
                 expect(result.dependsOn).to.eql(['dep1']);
@@ -1356,7 +1367,8 @@ describe('database', function () {
                 expect(results.length).to.be(1);
 
                 expect(results[0].id).to.be('backup-box');
-                expect(results[0].version).to.be('1.0.0');
+                expect(results[0].encryptionVersion).to.be(2);
+                expect(results[0].packageVersion).to.be('1.0.0');
                 expect(results[0].dependsOn).to.eql(['dep1']);
                 expect(results[0].manifest).to.eql(null);
 
@@ -1381,7 +1393,8 @@ describe('database', function () {
         it('add app succeeds', function (done) {
             var backup = {
                 id: 'app_appid_123',
-                version: '1.0.0',
+                encryptionVersion: null,
+                packageVersion: '1.0.0',
                 type: backupdb.BACKUP_TYPE_APP,
                 dependsOn: [ ],
                 manifest: { foo: 'bar' },
@@ -1397,7 +1410,8 @@ describe('database', function () {
         it('get succeeds', function (done) {
             backupdb.get('app_appid_123', function (error, result) {
                 expect(error).to.be(null);
-                expect(result.version).to.be('1.0.0');
+                expect(result.encryptionVersion).to.be(null);
+                expect(result.packageVersion).to.be('1.0.0');
                 expect(result.type).to.be(backupdb.BACKUP_TYPE_APP);
                 expect(result.creationTime).to.be.a(Date);
                 expect(result.dependsOn).to.eql([]);
@@ -1413,7 +1427,8 @@ describe('database', function () {
                 expect(results.length).to.be(1);
 
                 expect(results[0].id).to.be('app_appid_123');
-                expect(results[0].version).to.be('1.0.0');
+                expect(results[0].encryptionVersion).to.be(null);
+                expect(results[0].packageVersion).to.be('1.0.0');
                 expect(results[0].dependsOn).to.eql([]);
                 expect(results[0].manifest).to.eql({ foo: 'bar' });
 
@@ -1774,7 +1789,6 @@ describe('database', function () {
         before(function (done) {
             async.series([
                 domaindb.add.bind(null, DOMAIN_0.domain, { zoneName: DOMAIN_0.zoneName, provider: DOMAIN_0.provider, config: DOMAIN_0.config, tlsConfig: DOMAIN_0.tlsConfig }),
-                maildb.add.bind(null, DOMAIN_0.domain, {})
             ], done);
         });
 
@@ -1827,7 +1841,7 @@ describe('database', function () {
         });
 
         it('can set alias', function (done) {
-            mailboxdb.setAliasesForName('support', DOMAIN_0.domain, [ 'support2', 'help' ], function (error) {
+            mailboxdb.setAliasesForName('support', DOMAIN_0.domain, [ { name: 'support2', domain: DOMAIN_0.domain }, { name: 'help', domain: DOMAIN_0.domain } ], function (error) {
                 expect(error).to.be(null);
                 done();
             });
@@ -1837,8 +1851,10 @@ describe('database', function () {
             mailboxdb.getAliasesForName('support', DOMAIN_0.domain, function (error, results) {
                 expect(error).to.be(null);
                 expect(results.length).to.be(2);
-                expect(results[0]).to.be('help');
-                expect(results[1]).to.be('support2');
+                expect(results[0].name).to.be('help');
+                expect(results[0].domain).to.be(DOMAIN_0.domain);
+                expect(results[1].name).to.be('support2');
+                expect(results[1].domain).to.be(DOMAIN_0.domain);
                 done();
             });
         });
@@ -1847,18 +1863,8 @@ describe('database', function () {
             mailboxdb.getAlias('support2', DOMAIN_0.domain, function (error, result) {
                 expect(error).to.be(null);
                 expect(result.name).to.be('support2');
-                expect(result.aliasTarget).to.be('support');
-                done();
-            });
-        });
-
-        it('can list aliases', function (done) {
-            mailboxdb.listAliases(DOMAIN_0.domain, 1, 10, function (error, results) {
-                expect(error).to.be(null);
-                expect(results.length).to.be(2);
-                expect(results[0].name).to.be('help');
-                expect(results[0].aliasTarget).to.be('support');
-                expect(results[1].name).to.be('support2');
+                expect(result.aliasName).to.be('support');
+                expect(result.aliasDomain).to.be(DOMAIN_0.domain);
                 done();
             });
         });
@@ -1948,23 +1954,6 @@ describe('database', function () {
             database._clear(done);
         });
 
-        it('cannot add non-existing domain', function (done) {
-            maildb.add(MAIL_DOMAIN_0.domain + 'nope', {}, function (error) {
-                expect(error).to.be.ok();
-                expect(error.reason).to.be(BoxError.NOT_FOUND);
-
-                done();
-            });
-        });
-
-        it('can add domain', function (done) {
-            maildb.add(MAIL_DOMAIN_0.domain, {}, function (error) {
-                expect(error).to.equal(null);
-
-                done();
-            });
-        });
-
         it('can get all domains', function (done) {
             maildb.list(function (error, result) {
                 expect(error).to.equal(null);

src/test/dockerproxy-test.js

@@ -11,7 +11,7 @@ var constants = require('../constants.js'),
     exec = require('child_process').exec,
     expect = require('expect.js');
 
-const DOCKER = `docker -H tcp://localhost:${constants.DOCKER_PROXY_PORT} `;
+const DOCKER = `docker -H tcp://172.18.0.1:${constants.DOCKER_PROXY_PORT} `;
 
 describe('Dockerproxy', function () {
     var containerId;

src/test/ldap-test.js

@@ -19,6 +19,7 @@ var appdb = require('../appdb.js'),
     maildb = require('../maildb.js'),
     mailboxdb = require('../mailboxdb.js'),
     ldap = require('ldapjs'),
+    settings = require('../settings.js'),
     users = require('../users.js');
 
 const DOMAIN_0 = {
@@ -87,8 +88,8 @@ function setup(done) {
         database.initialize.bind(null),
         database._clear.bind(null),
         ldapServer.start.bind(null),
+        settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
         domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
-        maildb.add.bind(null, DOMAIN_0.domain, {}),
         function (callback) {
             users.createOwner(USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE, function (error, result) {
                 if (error) return callback(error);
@@ -107,7 +108,7 @@ function setup(done) {
             });
         },
         (done) => mailboxdb.addMailbox(USER_0.username.toLowerCase(), DOMAIN_0.domain, USER_0.id, done),
-        (done) => mailboxdb.setAliasesForName(USER_0.username.toLowerCase(), DOMAIN_0.domain, [ USER_0_ALIAS.toLocaleLowerCase() ], done),
+        (done) => mailboxdb.setAliasesForName(USER_0.username.toLowerCase(), DOMAIN_0.domain, [ { name: USER_0_ALIAS.toLocaleLowerCase(), domain: DOMAIN_0.domain} ], done),
         appdb.update.bind(null, APP_0.id, { containerId: APP_0.containerId }),
         appdb.setAddonConfig.bind(null, APP_0.id, 'sendmail', [{ name: 'MAIL_SMTP_PASSWORD', value : 'sendmailpassword' }]),
         appdb.setAddonConfig.bind(null, APP_0.id, 'recvmail', [{ name: 'MAIL_IMAP_PASSWORD', value : 'recvmailpassword' }]),
@@ -789,7 +790,7 @@ describe('Ldap', function () {
 
     describe('search mailing list', function () {
         before(function (done) {
-            mailboxdb.addList('devs', DOMAIN_0.domain, [ USER_0.username.toLowerCase() + '@' + DOMAIN_0.domain , USER_1.username.toLowerCase() + '@external.com' ], done);
+            mailboxdb.addList('devs', DOMAIN_0.domain, [ USER_0.username.toLowerCase() + '@' + DOMAIN_0.domain , USER_1.username.toLowerCase() + '@external.com' ], false /* membersOnly */, done);
         });
 
         it('get specific list', function (done) {

src/test/mail-test.js

@@ -34,7 +34,6 @@ function setup(done) {
         database._clear,
         domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
         settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-        mail.addDomain.bind(null, DOMAIN_0.domain)
     ], done);
 }
 

src/test/reverseproxy-test.js

@@ -67,6 +67,17 @@ describe('Certificates', function () {
         var validCert2 = '-----BEGIN CERTIFICATE-----\nMIIBwjCCAWwCCQCZjm6jL50XfTANBgkqhkiG9w0BAQsFADBoMQswCQYDVQQGEwJE\nRTEPMA0GA1UECAwGQmVybGluMQ8wDQYDVQQHDAZCZXJsaW4xEDAOBgNVBAoMB05l\nYnVsb24xDDAKBgNVBAsMA0NUTzEXMBUGA1UEAwwOYmF6LmZvb2Jhci5jb20wHhcN\nMTYxMTA4MDgyMDE1WhcNMjAxMTA3MDgyMDE1WjBoMQswCQYDVQQGEwJERTEPMA0G\nA1UECAwGQmVybGluMQ8wDQYDVQQHDAZCZXJsaW4xEDAOBgNVBAoMB05lYnVsb24x\nDDAKBgNVBAsMA0NUTzEXMBUGA1UEAwwOYmF6LmZvb2Jhci5jb20wXDANBgkqhkiG\n9w0BAQEFAANLADBIAkEAtKoyTPrf2DjKbnW7Xr1HbRvV+quHTcGmUq5anDI7G4w/\nabqDXGYyakHHlPyZxYp7FWQxCm83rHUuDT1LiLIBZQIDAQABMA0GCSqGSIb3DQEB\nCwUAA0EAVaD2Q6bF9hcUUBev5NyjaMdDYURuWfjuwWUkb8W50O2ed3O+MATKrDdS\nyVaBy8W02KJ4Y1ym4je/MF8nilPurA==\n-----END CERTIFICATE-----';
         var validKey2 = '-----BEGIN RSA PRIVATE KEY-----\nMIIBPQIBAAJBALSqMkz639g4ym51u169R20b1fqrh03BplKuWpwyOxuMP2m6g1xm\nMmpBx5T8mcWKexVkMQpvN6x1Lg09S4iyAWUCAwEAAQJBAJXu7YHPbjfuoalcUZzF\nbuKRCFtZQRf5z0Os6QvZ8A3iR0SzYJzx+c2ibp7WdifMXp3XaKm4tHSOfumrjUIq\nt10CIQDrs9Xo7bq0zuNjUV5IshNfaiYKZRfQciRVW2O8xBP9VwIhAMQ5CCEDZy+u\nsaF9RtmB0bjbe6XonBlAzoflfH/MAwWjAiEA50hL+ohr0MfCMM7DKaozgEj0kvan\n645VQLywnaX5x3kCIQDCwjinS9FnKmV0e/uOd6PJb0/S5IXLKt/TUpu33K5DMQIh\nAM9peu3B5t9pO59MmeUGZwI+bEJfEb+h03WTptBxS3pO\n-----END RSA PRIVATE KEY-----';
 
+        /*
+            Generate these with:
+              openssl ecparam -genkey -name prime256v1 -out server.key
+              openssl req -new -sha256 -key server.key -out server.csr
+              openssl req -x509 -sha256 -days 1460 -key server.key -in server.csr -out server.crt
+        */
+
+        // *.foobar.com
+        var validCert4 = '-----BEGIN CERTIFICATE-----\nMIICDDCCAbOgAwIBAgIUduLaSQC6kh9LxVdua1EUBCgQOHYwCgYIKoZIzj0EAwIw\nXDELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGElu\ndGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEVMBMGA1UEAwwMKi5mb29iYXIuY29tMB4X\nDTIwMDMyNTA0MTYxMloXDTI0MDMyNDA0MTYxMlowXDELMAkGA1UEBhMCQVUxEzAR\nBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5\nIEx0ZDEVMBMGA1UEAwwMKi5mb29iYXIuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0D\nAQcDQgAEmBum8MbyGXKuLP+NEOmR15XlemPEHR4b68A+B0Zjh/cuLQncAIwfmLT7\nutUOh3CivEKvZYkQIdd71xhCbVtbkqNTMFEwHQYDVR0OBBYEFCxEvAFsSFyAITNw\niBttbdsyEwO4MB8GA1UdIwQYMBaAFCxEvAFsSFyAITNwiBttbdsyEwO4MA8GA1Ud\nEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDRwAwRAIgd+rxp8xTXy7wsV45hiu1HQ2p\nwrEEPFmfPinVHwhDCiECIAEnIr5bEYUzSjujiHg7C2q3zh41XJhZWQie3VHLY/Kt\n-----END CERTIFICATE-----\n';
+        var validKey4 = '-----BEGIN EC PARAMETERS-----\nBggqhkjOPQMBBw==\n-----END EC PARAMETERS-----\n-----BEGIN EC PRIVATE KEY-----\nMHcCAQEEIAXuQG4YDaQuwOCvWOZjkOvw/Y5V8Oum+rWnliMTsA5woAoGCCqGSM49\nAwEHoUQDQgAEmBum8MbyGXKuLP+NEOmR15XlemPEHR4b68A+B0Zjh/cuLQncAIwf\nmLT7utUOh3CivEKvZYkQIdd71xhCbVtbkg==\n-----END EC PRIVATE KEY-----\n';
+
         // cp /etc/ssl/openssl.cnf /tmp/openssl.cnf
         // echo -e "[SAN]\nsubjectAltName=DNS:amazing.com,DNS:*.amazing.com\n" >> /tmp/openssl.cnf
         // openssl req -x509 -newkey rsa:2048 -keyout amazing.key -out amazing.crt -days 3650 -subj /CN=*.amazing.com -nodes -extensions SAN -config /tmp/openssl.cnf
@@ -123,6 +134,10 @@ describe('Certificates', function () {
             expect(reverseProxy.validateCertificate('', amazingDomain, { cert: validCert3, key: validKey3 })).to.be(null);
             expect(reverseProxy.validateCertificate('subdomain', amazingDomain, { cert: validCert3, key: validKey3 })).to.be(null);
         });
+
+        it('allows valid cert with matching domain (subdomain) - ecdsa', function () {
+            expect(reverseProxy.validateCertificate('baz', foobarDomain, { cert: validCert4, key: validKey4 })).to.be(null);
+        });
     });
 
     describe('generateFallbackCertificiate - non-hyphenated', function () {
@@ -181,7 +196,7 @@ describe('Certificates', function () {
             reverseProxy._getCertApi(DOMAIN_0, function (error, api, options) {
                 expect(error).to.be(null);
                 expect(api._name).to.be('caas');
-                expect(options).to.eql({ email: 'support@cloudron.io', 'performHttpAuthorization': false, 'prod': false, 'wildcard': false });
+                expect(options).to.eql({ email: 'webmaster@cloudron.io', 'performHttpAuthorization': false, 'prod': false, 'wildcard': false });
                 done();
             });
         });

src/test/updatechecker-test.js

@@ -13,7 +13,6 @@ var appdb = require('../appdb.js'),
     database = require('../database.js'),
     domains = require('../domains.js'),
     expect = require('expect.js'),
-    mail = require('../mail.js'),
     mailer = require('../mailer.js'),
     nock = require('nock'),
     paths = require('../paths.js'),
@@ -82,7 +81,6 @@ describe('updatechecker - box - manual (email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             settings.setBoxAutoupdatePattern.bind(null, constants.AUTOUPDATE_PATTERN_NEVER),
             settingsdb.set.bind(null, settings.CLOUDRON_TOKEN_KEY, 'atoken'),
@@ -96,10 +94,10 @@ describe('updatechecker - box - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/boxupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken' })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', automatic: false })
             .reply(204, { } );
 
-        updatechecker.checkBoxUpdates(function (error) {
+        updatechecker.checkBoxUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().box).to.be(null);
             expect(scope.isDone()).to.be.ok();
@@ -113,10 +111,10 @@ describe('updatechecker - box - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/boxupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken' })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', automatic: false })
             .reply(200, { version: UPDATE_VERSION, changelog: [''], sourceTarballUrl: 'box.tar.gz', sourceTarballSigUrl: 'box.tar.gz.sig', boxVersionsUrl: 'box.versions', boxVersionsSigUrl: 'box.versions.sig' } );
 
-        updatechecker.checkBoxUpdates(function (error) {
+        updatechecker.checkBoxUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().box.version).to.be(UPDATE_VERSION);
             expect(updatechecker.getUpdateInfo().box.sourceTarballUrl).to.be('box.tar.gz');
@@ -131,10 +129,10 @@ describe('updatechecker - box - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/boxupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken' })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', automatic: false })
             .reply(404, { version: '2.0.0-pre.0', changelog: [''], sourceTarballUrl: 'box-pre.tar.gz' } );
 
-        updatechecker.checkBoxUpdates(function (error) {
+        updatechecker.checkBoxUpdates({ automatic: false }, function (error) {
             expect(error).to.be.ok();
             expect(updatechecker.getUpdateInfo().box).to.be(null);
             expect(scope.isDone()).to.be.ok();
@@ -154,7 +152,6 @@ describe('updatechecker - box - automatic (no email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             settingsdb.set.bind(null, settings.CLOUDRON_TOKEN_KEY, 'atoken'),
         ], done);
@@ -167,10 +164,10 @@ describe('updatechecker - box - automatic (no email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/boxupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken' })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', automatic: false })
             .reply(200, { version: UPDATE_VERSION, changelog: [''], sourceTarballUrl: 'box.tar.gz', sourceTarballSigUrl: 'box.tar.gz.sig', boxVersionsUrl: 'box.versions', boxVersionsSigUrl: 'box.versions.sig' } );
 
-        updatechecker.checkBoxUpdates(function (error) {
+        updatechecker.checkBoxUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().box.version).to.be(UPDATE_VERSION);
             expect(scope.isDone()).to.be.ok();
@@ -190,7 +187,6 @@ describe('updatechecker - box - automatic free (email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             settingsdb.set.bind(null, settings.CLOUDRON_TOKEN_KEY, 'atoken'),
         ], done);
@@ -203,10 +199,10 @@ describe('updatechecker - box - automatic free (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/boxupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken' })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', automatic: false })
             .reply(200, { version: UPDATE_VERSION, changelog: [''], sourceTarballUrl: 'box.tar.gz', sourceTarballSigUrl: 'box.tar.gz.sig', boxVersionsUrl: 'box.versions', boxVersionsSigUrl: 'box.versions.sig' } );
 
-        updatechecker.checkBoxUpdates(function (error) {
+        updatechecker.checkBoxUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().box.version).to.be(UPDATE_VERSION);
             expect(scope.isDone()).to.be.ok();
@@ -254,7 +250,6 @@ describe('updatechecker - app - manual (email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             appdb.add.bind(null, APP_0.id, APP_0.appStoreId, APP_0.manifest, APP_0.location, APP_0.domain, apps._translatePortBindings(APP_0.portBindings, APP_0.manifest), APP_0),
             settings.setAppAutoupdatePattern.bind(null, constants.AUTOUPDATE_PATTERN_NEVER),
@@ -269,10 +264,10 @@ describe('updatechecker - app - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/appupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version, automatic: false })
             .reply(204, { } );
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({});
             expect(scope.isDone()).to.be.ok();
@@ -286,10 +281,10 @@ describe('updatechecker - app - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/appupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version, automatic: false })
             .reply(500, { update: { manifest: { version: '1.0.0', changelog: '* some changes' } } } );
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({});
             expect(scope.isDone()).to.be.ok();
@@ -303,10 +298,10 @@ describe('updatechecker - app - manual (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/appupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version, automatic: false })
             .reply(200, { manifest: { version: '2.0.0', changelog: '* some changes' } } );
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({ 'appid-0': { manifest: { version: '2.0.0', changelog: '* some changes' } } });
             expect(scope.isDone()).to.be.ok();
@@ -318,7 +313,7 @@ describe('updatechecker - app - manual (email)', function () {
     it('does not offer old version', function (done) {
         nock.cleanAll();
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({ });
             checkMails(0, done);
@@ -364,7 +359,6 @@ describe('updatechecker - app - automatic (no email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             appdb.add.bind(null, APP_0.id, APP_0.appStoreId, APP_0.manifest, APP_0.location, APP_0.domain, apps._translatePortBindings(APP_0.portBindings, APP_0.manifest), APP_0),
             settings.setAppAutoupdatePattern.bind(null, '00 00 1,3,5,23 * * *'),
@@ -379,10 +373,10 @@ describe('updatechecker - app - automatic (no email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/appupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version, automatic: false })
             .reply(200, { manifest: { version: '2.0.0', changelog: 'c' } } );
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({ 'appid-0': { manifest: { version: '2.0.0', changelog: 'c' } } });
             expect(scope.isDone()).to.be.ok();
@@ -430,7 +424,6 @@ describe('updatechecker - app - automatic free (email)', function () {
             cron.startJobs,
             domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
             settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-            mail.addDomain.bind(null, DOMAIN_0.domain),
             users.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
             appdb.add.bind(null, APP_0.id, APP_0.appStoreId, APP_0.manifest, APP_0.location, APP_0.domain, apps._translatePortBindings(APP_0.portBindings, APP_0.manifest), APP_0),
             settings.setAppAutoupdatePattern.bind(null, '00 00 1,3,5,23 * * *'),
@@ -445,10 +438,10 @@ describe('updatechecker - app - automatic free (email)', function () {
 
         var scope = nock('http://localhost:4444')
             .get('/api/v1/appupdate')
-            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version })
+            .query({ boxVersion: constants.VERSION, accessToken: 'atoken', appId: APP_0.appStoreId, appVersion: APP_0.manifest.version, automatic: false })
             .reply(200, { manifest: { version: '2.0.0', changelog: 'c' } } );
 
-        updatechecker.checkAppUpdates(function (error) {
+        updatechecker.checkAppUpdates({ automatic: false }, function (error) {
             expect(!error).to.be.ok();
             expect(updatechecker.getUpdateInfo().apps).to.eql({ 'appid-0': { manifest: { version: '2.0.0', changelog: 'c' } } });
             expect(scope.isDone()).to.be.ok();

src/test/users-test.js

@@ -18,6 +18,7 @@ var async = require('async'),
     mailboxdb = require('../mailboxdb.js'),
     maildb = require('../maildb.js'),
     mailer = require('../mailer.js'),
+    paths = require('../paths.js'),
     settings = require('../settings.js'),
     userdb = require('../userdb.js'),
     users = require('../users.js'),
@@ -73,7 +74,6 @@ function setup(done) {
         database._clear,
         domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
         settings.setAdmin.bind(null, DOMAIN_0.domain, 'my.' + DOMAIN_0.domain),
-        mail.addDomain.bind(null, DOMAIN_0.domain),
     ], done);
 }
 
@@ -283,10 +283,10 @@ describe('User', function () {
         it('fails for ghost with wrong password', function (done) {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verify(userObject.id, 'foobar', users.AP_WEBADMIN, function (error) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.be.a(BoxError);
                 expect(error.reason).to.equal(BoxError.INVALID_CREDENTIALS);
@@ -297,10 +297,10 @@ describe('User', function () {
         it('succeeds for ghost', function (done) {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verify(userObject.id, 'testpassword', users.AP_WEBADMIN, function (error, result) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.equal(null);
                 expect(result.id).to.equal(userObject.id);
@@ -315,10 +315,10 @@ describe('User', function () {
         it('succeeds for normal user password when ghost file exists', function (done) {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verify(userObject.id, PASSWORD, users.AP_WEBADMIN, function (error, result) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.not.be.ok();
                 expect(result).to.be.ok();
@@ -385,10 +385,10 @@ describe('User', function () {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
 
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verifyWithUsername(USERNAME, 'foobar', users.AP_WEBADMIN, function (error) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.be.a(BoxError);
                 expect(error.reason).to.equal(BoxError.INVALID_CREDENTIALS);
@@ -400,10 +400,10 @@ describe('User', function () {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
 
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verifyWithUsername(USERNAME, 'testpassword', users.AP_WEBADMIN, function (error, result) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.equal(null);
                 expect(result.id).to.equal(userObject.id);
@@ -472,10 +472,10 @@ describe('User', function () {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
 
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verifyWithEmail(EMAIL, 'foobar', users.AP_WEBADMIN, function (error) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.be.a(BoxError);
                 expect(error.reason).to.equal(BoxError.INVALID_CREDENTIALS);
@@ -487,10 +487,10 @@ describe('User', function () {
             var ghost = { };
             ghost[userObject.username] = 'testpassword';
 
-            fs.writeFileSync(constants.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
+            fs.writeFileSync(paths.GHOST_USER_FILE, JSON.stringify(ghost), 'utf8');
 
             users.verifyWithEmail(EMAIL, 'testpassword', users.AP_WEBADMIN, function (error, result) {
-                fs.unlinkSync(constants.GHOST_USER_FILE);
+                fs.unlinkSync(paths.GHOST_USER_FILE);
 
                 expect(error).to.equal(null);
                 expect(result.id).to.equal(userObject.id);