Use async unlink

(cherry picked from commit 8dd3c55ecf)
typoe
2018-09-28 17:06:05 -07:00 · 2018-09-28 17:02:17 -07:00 · 2018-09-28 14:36:05 -07:00 · 2018-09-28 14:33:26 -07:00 · 2018-09-28 13:48:34 -07:00 · 2018-09-27 20:22:13 -07:00
146 changed files with 5398 additions and 7428 deletions
@@ -1,29 +0,0 @@
-{
-    "env": {
-        "es6": true,
-        "node": true
-    },
-    "extends": "eslint:recommended",
-    "parserOptions": {
-        "ecmaVersion": 2017
-    },
-    "rules": {
-        "indent": [
-            "error",
-            4
-        ],
-        "linebreak-style": [
-            "error",
-            "unix"
-        ],
-        "quotes": [
-            "error",
-            "single"
-        ],
-        "semi": [
-            "error",
-            "always"
-        ],
-        "no-console": "off"
-    }
-}
@@ -6,3 +6,4 @@ installer/src/certs/server.key
 # vim swap files
 *.swp

+
@@ -0,0 +1,9 @@
+{
+    "node": true,
+    "browser": true,
+    "unused": true,
+    "multistr": true,
+    "globalstrict": true,
+    "predef": [ "angular", "$" ],
+    "esnext": true
+}
@@ -1376,18 +1376,14 @@
 [3.1.3]
 * Prevent dashboard domain from being deleted
 * Add alternateDomains to app install route
-* cloudflare: Fix crash when access denied
+
+[3.1.4]
+* Fix issue where support tab was redirecting

 [3.1.4]
 * Fix issue where support tab was redirecting

 [3.2.0]
-* Add DO Spaces SFO2 region
-* Wildcard DNS now validates the config
-* Add ACMEv2 support
-* Add Wildcard Let's Encrypt provider
-
-[3.2.1]
 * Add acme2 support. This provides DNS based validation removing inbound port 80 requirement
 * Add support for wildcard certificates
 * Allow mailbox name to be reset to the buit-in '.app' name
@@ -1397,111 +1393,3 @@
 * Add SFO2 region for DigitalOcean Spaces
 * Show the title in port bindings instead of the long description

-[3.2.2]
-* Update Haraka to 2.8.20
-* (mail) Fix issue where LDAP connections where not cleaned up
-
-[3.3.0]
-* Use new addons with REST APIs
-* Ubuntu 18.04 LTS support
-* Custom env vars can be set per application
-* Add a button to renew certs
-* Add better support for private builds
-* cloudflare: Fix crash when using bad email
-* cloudflare: HTTP proxying works now
-* add new exoscale-sos regions
-* Add UI to toggle dynamic DNS
-* Add support for hyphenated subdomains
-
-[3.3.1]
-* Use new addons with REST APIs
-* Ubuntu 18.04 LTS support
-* Custom env vars can be set per application
-* Add a button to renew certs
-* Add better support for private builds
-* cloudflare: Fix crash when using bad email
-* cloudflare: HTTP proxying works now
-* add new exoscale-sos regions
-* Add UI to toggle dynamic DNS
-* Add support for hyphenated subdomains
-
-[3.3.2]
-* Use new addons with REST APIs
-* Ubuntu 18.04 LTS support
-* Custom env vars can be set per application
-* Add a button to renew certs
-* Add better support for private builds
-* cloudflare: Fix crash when using bad email
-* cloudflare: HTTP proxying works now
-* add new exoscale-sos regions
-* Add UI to toggle dynamic DNS
-* Add support for hyphenated subdomains
-* Add domain, mail events to eventlog
-
-[3.3.3]
-* Use new addons with REST APIs
-* Ubuntu 18.04 LTS support
-* Custom env vars can be set per application
-* Add a button to renew certs
-* Add better support for private builds
-* cloudflare: Fix crash when using bad email
-* cloudflare: HTTP proxying works now
-* add new exoscale-sos regions
-* Add UI to toggle dynamic DNS
-* Add support for hyphenated subdomains
-* Add domain, mail events to eventlog
-
-[3.3.4]
-* Use new addons with REST APIs
-* Ubuntu 18.04 LTS support
-* Custom env vars can be set per application
-* Add a button to renew certs
-* Add better support for private builds
-* cloudflare: Fix crash when using bad email
-* cloudflare: HTTP proxying works now
-* add new exoscale-sos regions
-* Add UI to toggle dynamic DNS
-* Add support for hyphenated subdomains
-* Add domain, mail events to eventlog
-
-[3.4.0]
-* Improve error page
-* Add system view to manage addons and view their status
-* Fix iconset regression for account and Cloudron name edits
-* Add server reboot button and warn if reboot is required for security updates
-* Backup and update tasks are now cancelable
-* Move graphite away from port 3000 (reserved by ESXi)
-* Flexible mailbox management
-* Automatic updates can be toggled per app
-
-[3.4.1]
-* Improve error page
-* Add system view to manage addons and view their status
-* Fix iconset regression for account and Cloudron name edits
-* Add server reboot button and warn if reboot is required for security updates
-* Backup and update tasks are now cancelable
-* Move graphite away from port 3000 (reserved by ESXi)
-* Flexible mailbox management
-* Automatic updates can be toggled per app
-
-[3.4.2]
-* Improve error page
-* Add system view to manage addons and view their status
-* Fix iconset regression for account and Cloudron name edits
-* Add server reboot button and warn if reboot is required for security updates
-* Backup and update tasks are now cancelable
-* Move graphite away from port 3000 (reserved by ESXi)
-* Flexible mailbox management
-* Automatic updates can be toggled per app
-
-[3.4.3]
-* Improve error page
-* Add system view to manage addons and view their status
-* Fix iconset regression for account and Cloudron name edits
-* Add server reboot button and warn if reboot is required for security updates
-* Backup and update tasks are now cancelable
-* Move graphite away from port 3000 (reserved by ESXi)
-* Flexible mailbox management
-* Automatic updates can be toggled per app
-* Fix issue where OOM mails are sent out without a rate limit
-
@@ -1 +0,0 @@
-# release version. do not edit manually
@@ -26,7 +26,6 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password password pas
 debconf-set-selections <<< 'mysql-server mysql-server/root_password_again password password'

 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
-# resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
 apt-get -y install \
    acl \
    awscli \
@@ -35,14 +34,12 @@ apt-get -y install \
    curl \
    dmsetup \
    iptables \
-    libpython2.7 \
    logrotate \
    mysql-server-5.7 \
    nginx-full \
    openssh-server \
    pwgen \
-    resolvconf \
-    sudo \
+    rcconf \
    swaks \
    unattended-upgrades \
    unbound \
@@ -90,12 +87,11 @@ if [ ! -f "${arg_infraversionpath}/infra_version.js" ]; then
    exit 1
 fi

-images=$(node -e "var i = require('${arg_infraversionpath}/infra_version.js'); console.log(i.baseImages.map(function (x) { return x.tag; }).join(' '), Object.keys(i.images).map(function (x) { return i.images[x].tag; }).join(' '));")
+images=$(node -e "var i = require('${arg_infraversionpath}/infra_version.js'); console.log(i.baseImages.join(' '), Object.keys(i.images).map(function (x) { return i.images[x].tag; }).join(' '));")

 echo -e "\tPulling docker images: ${images}"
 for image in ${images}; do
    docker pull "${image}"
-    docker pull "${image%@sha256:*}" # this will tag the image for readability
 done

 echo "==> Install collectd"
@@ -105,11 +101,6 @@ if ! apt-get install -y collectd collectd-utils; then
    sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
 fi

-echo "==> Configuring host"
-sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
-timedatectl set-ntp 1
-timedatectl set-timezone UTC
-
 # Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
 systemctl stop bind9 || true
 systemctl disable bind9 || true
@@ -122,7 +113,3 @@ systemctl disable dnsmasq || true
 systemctl stop postfix || true
 systemctl disable postfix || true

-# on ubuntu 18.04, this is the default. this requires resolvconf for DNS to work further after the disable
-systemctl stop systemd-resolved || true
-systemctl disable systemd-resolved || true
-
@@ -2,18 +2,15 @@

 'use strict';

-// prefix all output with a timestamp
-// debug() already prefixes and uses process.stderr NOT console.*
-['log', 'info', 'warn', 'debug', 'error'].forEach(function (log) {
-    var orig = console[log];
-    console[log] = function () {
-        orig.apply(console, [new Date().toISOString()].concat(Array.prototype.slice.call(arguments)));
-    };
-});
-
 require('supererror')({ splatchError: true });

-let async = require('async'),
+// remove timestamp from debug() based output
+require('debug').formatArgs = function formatArgs(args) {
+    args[0] = this.namespace + ' ' + args[0];
+};
+
+var appHealthMonitor = require('./src/apphealthmonitor.js'),
+    async = require('async'),
    config = require('./src/config.js'),
    ldap = require('./src/ldap.js'),
    dockerProxy = require('./src/dockerproxy.js'),
@@ -39,7 +36,8 @@ console.log();
 async.series([
    server.start,
    ldap.start,
-    dockerProxy.start
+    dockerProxy.start,
+    appHealthMonitor.start,
 ], function (error) {
    if (error) {
        console.error('Error starting server', error);
@@ -51,8 +49,6 @@ async.series([
 var NOOP_CALLBACK = function () { };

 process.on('SIGINT', function () {
-    console.log('Received SIGINT. Shutting down.');
-
    server.stop(NOOP_CALLBACK);
    ldap.stop(NOOP_CALLBACK);
    dockerProxy.stop(NOOP_CALLBACK);
@@ -60,8 +56,6 @@ process.on('SIGINT', function () {
 });

 process.on('SIGTERM', function () {
-    console.log('Received SIGTERM. Shutting down.');
-
    server.stop(NOOP_CALLBACK);
    ldap.stop(NOOP_CALLBACK);
    dockerProxy.stop(NOOP_CALLBACK);
@@ -1,7 +1,7 @@
 'use strict';

 exports.up = function(db, callback) {
-    var cmd = "CREATE TABLE userGroups(" +
+    var cmd = "CREATE TABLE groups(" +
                "id VARCHAR(128) NOT NULL UNIQUE," +
                "name VARCHAR(128) NOT NULL UNIQUE," +
                "PRIMARY KEY(id))";
@@ -13,7 +13,7 @@ exports.up = function(db, callback) {
 };

 exports.down = function(db, callback) {
-    db.runSql('DROP TABLE userGroups', function (error) {
+    db.runSql('DROP TABLE groups', function (error) {
        if (error) console.error(error);
        callback(error);
    });
@@ -4,7 +4,7 @@ exports.up = function(db, callback) {
 	var cmd = "CREATE TABLE IF NOT EXISTS groupMembers(" +
    			"groupId VARCHAR(128) NOT NULL," +
    			"userId VARCHAR(128) NOT NULL," +
-    			"FOREIGN KEY(groupId) REFERENCES userGroups(id)," +
+    			"FOREIGN KEY(groupId) REFERENCES groups(id)," +
    			"FOREIGN KEY(userId) REFERENCES users(id));";

    db.runSql(cmd, function (error) {
@@ -7,7 +7,7 @@ var ADMIN_GROUP_ID = 'admin'; // see constants.js
 exports.up = function(db, callback) {
 	async.series([
 		db.runSql.bind(db, 'START TRANSACTION;'),
-		db.runSql.bind(db, 'INSERT INTO userGroups (id, name) VALUES (?, ?)', [ ADMIN_GROUP_ID, 'admin' ]),
+		db.runSql.bind(db, 'INSERT INTO groups (id, name) VALUES (?, ?)', [ ADMIN_GROUP_ID, 'admin' ]),
 		function migrateAdminFlag(done) {
 			db.all('SELECT * FROM users WHERE admin=1', function (error, results) {
 				if (error) return done(error);
@@ -10,7 +10,7 @@ exports.up = function(db, callback) {
        function addGroupMailboxes(done) {
            console.log('Importing group mailboxes');

-            db.all('SELECT id, name FROM userGroups', function (error, results) {
+            db.all('SELECT id, name FROM groups', function (error, results) {
                if (error) return done(error);

                async.eachSeries(results, function (g, next) {
@@ -16,7 +16,7 @@ exports.up = function(db, callback) {
        db.runSql.bind(db, 'ALTER TABLE clients CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
        db.runSql.bind(db, 'ALTER TABLE eventlog CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
        db.runSql.bind(db, 'ALTER TABLE groupMembers CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
-        db.runSql.bind(db, 'ALTER TABLE userGroups CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE groups CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
        db.runSql.bind(db, 'ALTER TABLE mailboxes CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
        db.runSql.bind(db, 'ALTER TABLE migrations CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
        db.runSql.bind(db, 'ALTER TABLE settings CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
@@ -29,7 +29,7 @@ exports.up = function(db, callback) {

                // this will be finally created once we have a domain when we create the owner in user.js
                const ADMIN_GROUP_ID = 'admin'; // see constants.js
-                db.runSql('DELETE FROM userGroups WHERE id = ?', [ ADMIN_GROUP_ID ], function (error) {
+                db.runSql('DELETE FROM groups WHERE id = ?', [ ADMIN_GROUP_ID ], function (error) {
                    if (error) return done(error);

                    db.runSql('DELETE FROM mailboxes WHERE ownerId = ?', [ ADMIN_GROUP_ID ], done);
@@ -19,8 +19,8 @@ exports.up = function(db, callback) {
        },
        function getGroups(done) {
            db.all('SELECT id, name, GROUP_CONCAT(groupMembers.userId) AS userIds ' +
-                ' FROM userGroups LEFT OUTER JOIN groupMembers ON userGroups.id = groupMembers.groupId ' +
-                ' GROUP BY userGroups.id', [ ], function (error, results) {
+                ' FROM groups LEFT OUTER JOIN groupMembers ON groups.id = groupMembers.groupId ' +
+                ' GROUP BY groups.id', [ ], function (error, results) {
                if (error) return done(error);

                results.forEach(function (result) {
@@ -18,7 +18,7 @@ exports.up = function(db, callback) {

                async.series([
                    db.runSql.bind(db, 'DELETE FROM groupMembers WHERE groupId=?', [ 'admin' ]),
-                    db.runSql.bind(db, 'DELETE FROM userGroups WHERE id=?', [ 'admin' ])
+                    db.runSql.bind(db, 'DELETE FROM groups WHERE id=?', [ 'admin' ])
                ], callback);
            });
        });
@@ -1,21 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    var cmd = 'CREATE TABLE IF NOT EXISTS appEnvVars(' +
-                'appId VARCHAR(128) NOT NULL,' +
-                'name TEXT NOT NULL,' +
-                'value TEXT NOT NULL,' +
-                'FOREIGN KEY(appId) REFERENCES apps(id)) CHARACTER SET utf8 COLLATE utf8_bin';
-
-    db.runSql(cmd, function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('DROP TABLE appEnvVars', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,27 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    var cmd = 'CREATE TABLE tasks(' +
-            'id int NOT NULL AUTO_INCREMENT,' +
-            'type VARCHAR(32) NOT NULL,' +
-            'argsJson TEXT,' +
-            'percent INTEGER DEFAULT 0,' +
-            'message TEXT,' +
-            'errorMessage TEXT,' +
-            'result TEXT,' +
-            'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
-            'ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,' +
-            'PRIMARY KEY (id))';
-
-    db.runSql(cmd, function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('DROP TABLE tasks', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('SELECT 1 FROM groups LIMIT 1', function (error) {
-        if (error) return callback(); // groups table does not exist
-
-        db.runSql('RENAME TABLE groups TO userGroups', function (error) {
-            if (error) console.error(error);
-            callback(error);
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    // this is a one way renaming since the previous migration steps have been already updated to match the new name
-    callback();
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps MODIFY creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP'),
-        db.runSql.bind(db, 'ALTER TABLE apps MODIFY updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP'),
-        db.runSql.bind(db, 'ALTER TABLE eventlog MODIFY creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP'),
-        db.runSql.bind(db, 'ALTER TABLE backups MODIFY creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP'),
-        db.runSql.bind(db, 'ALTER TABLE mailboxes MODIFY creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP'),
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,28 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps ADD COLUMN mailboxName VARCHAR(128)'),
-        db.runSql.bind(db, 'START TRANSACTION;'),
-
-        function migrateMailboxNames(done) {
-            db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
-                if (error) return done(error);
-
-                async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
-                    if (mailbox.ownerType !== 'app') return iteratorDone();
-
-                    db.runSql('UPDATE apps SET mailboxName = ? WHERE id = ?', [ mailbox.name, mailbox.ownerId ], iteratorDone);
-                }, done);
-            });
-        },
-
-        db.runSql.bind(db, 'COMMIT')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,28 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'START TRANSACTION;'),
-
-        function migrateMailboxNames(done) {
-            db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
-                if (error) return done(error);
-
-                async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
-                    if (mailbox.ownerType !== 'app') return iteratorDone();
-
-                    db.runSql('DELETE FROM mailboxes WHERE name = ?', [ mailbox.name ], iteratorDone);
-                }, done);
-            });
-        },
-
-        db.runSql.bind(db, 'COMMIT'),
-        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP COLUMN ownerType')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,16 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps ADD COLUMN enableAutomaticUpdate BOOLEAN DEFAULT 1', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps DROP COLUMN enableAutomaticUpdate', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -29,7 +29,7 @@ CREATE TABLE IF NOT EXISTS users(

    PRIMARY KEY(id));

-CREATE TABLE IF NOT EXISTS userGroups(
+CREATE TABLE IF NOT EXISTS groups(
    id VARCHAR(128) NOT NULL UNIQUE,
    name VARCHAR(254) NOT NULL UNIQUE,
    PRIMARY KEY(id));
@@ -37,7 +37,7 @@ CREATE TABLE IF NOT EXISTS userGroups(
 CREATE TABLE IF NOT EXISTS groupMembers(
    groupId VARCHAR(128) NOT NULL,
    userId VARCHAR(128) NOT NULL,
-    FOREIGN KEY(groupId) REFERENCES userGroups(id),
+    FOREIGN KEY(groupId) REFERENCES groups(id),
    FOREIGN KEY(userId) REFERENCES users(id));

 CREATE TABLE IF NOT EXISTS tokens(
@@ -71,8 +71,8 @@ CREATE TABLE IF NOT EXISTS apps(
    location VARCHAR(128) NOT NULL,
    domain VARCHAR(128) NOT NULL,
    accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
-    updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
+    creationTime TIMESTAMP(2) NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
+    updateTime TIMESTAMP(2) NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, // when this db record was updated (useful for UI caching)
    memoryLimit BIGINT DEFAULT 0,
    xFrameOptions VARCHAR(512),
@@ -80,8 +80,6 @@ CREATE TABLE IF NOT EXISTS apps(
    debugModeJson TEXT, // options for development mode
    robotsTxt TEXT,
    enableBackup BOOLEAN DEFAULT 1, // misnomer: controls automatic daily backups
-    enableAutomaticUpdate BOOLEAN DEFAULT 1,
-    mailboxName VARCHAR(128), // mailbox of this app

    // the following fields do not belong here, they can be removed when we use a queue for apptask
    restoreConfigJson VARCHAR(256), // used to pass backupId to restore from to apptask
@@ -120,15 +118,9 @@ CREATE TABLE IF NOT EXISTS appAddonConfigs(
    value VARCHAR(512) NOT NULL,
    FOREIGN KEY(appId) REFERENCES apps(id));

-CREATE TABLE IF NOT EXISTS appEnvVars(
-    appId VARCHAR(128) NOT NULL,
-    name TEXT NOT NULL,
-    value TEXT NOT NULL,
-    FOREIGN KEY(appId) REFERENCES apps(id));
-
 CREATE TABLE IF NOT EXISTS backups(
    id VARCHAR(128) NOT NULL,
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    creationTime TIMESTAMP,
    version VARCHAR(128) NOT NULL, /* app version or box version */
    type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
    dependsOn TEXT, /* comma separate list of objects this backup depends on */
@@ -143,7 +135,7 @@ CREATE TABLE IF NOT EXISTS eventlog(
    action VARCHAR(128) NOT NULL,
    source TEXT, /* { userId, username, ip }. userId can be null for cron,sysadmin */
    data TEXT, /* free flowing json based on action */
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    createdAt TIMESTAMP(2) NOT NULL,

    PRIMARY KEY (id));

@@ -175,17 +167,15 @@ CREATE TABLE IF NOT EXISTS mail(
 /* Future fields:
   * accessRestriction - to determine who can access it. So this has foreign keys
   * quota - per mailbox quota
-
-   NOTE: this table exists only real mailboxes. And has unique constraint to handle
-   conflict with aliases and mailbox names
 */
 CREATE TABLE IF NOT EXISTS mailboxes(
    name VARCHAR(128) NOT NULL,
    type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
-    ownerId VARCHAR(128) NOT NULL, /* user id */
+    ownerId VARCHAR(128) NOT NULL, /* app id or user id or group id */
+    ownerType VARCHAR(16) NOT NULL, /* 'app' or 'user' or 'group' */
    aliasTarget VARCHAR(128), /* the target name type is an alias */
    membersJson TEXT, /* members of a group */
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    creationTime TIMESTAMP,
    domain VARCHAR(128),

    FOREIGN KEY(domain) REFERENCES mail(domain),
@@ -199,18 +189,6 @@ CREATE TABLE IF NOT EXISTS subdomains(

    FOREIGN KEY(domain) REFERENCES domains(domain),
    FOREIGN KEY(appId) REFERENCES apps(id),
-    UNIQUE (subdomain, domain));
-
-CREATE TABLE IF NOT EXISTS tasks(
-    id int NOT NULL AUTO_INCREMENT,
-    type VARCHAR(32) NOT NULL,
-    percent INTEGER DEFAULT 0,
-    message TEXT,
-    errorMessage TEXT,
-    result TEXT,
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-    PRIMARY KEY (id));
+    UNIQUE (subdomain, domain))

    CHARACTER SET utf8 COLLATE utf8_bin;
-
@@ -20,14 +20,14 @@
    "async": "^2.6.1",
    "aws-sdk": "^2.253.1",
    "body-parser": "^1.18.3",
-    "cloudron-manifestformat": "^2.14.2",
+    "cloudron-manifestformat": "^2.13.1",
    "connect": "^3.6.6",
    "connect-ensure-login": "^0.1.1",
    "connect-lastmile": "^1.0.2",
    "connect-timeout": "^1.9.0",
    "cookie-parser": "^1.3.5",
    "cookie-session": "^1.3.2",
-    "cron": "^1.5.1",
+    "cron": "^1.3.0",
    "csurf": "^1.6.6",
    "db-migrate": "^0.11.1",
    "db-migrate-mysql": "^1.1.10",
@@ -92,7 +92,7 @@
  "scripts": {
    "migrate_local": "DATABASE_URL=mysql://root:@localhost/box node_modules/.bin/db-migrate up",
    "migrate_test": "BOX_ENV=test DATABASE_URL=mysql://root:@localhost/boxtest node_modules/.bin/db-migrate up",
-    "test": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- --no-timeouts --exit -R spec ./src/test ./src/routes/test",
+    "test": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- --exit -R spec ./src/test ./src/routes/test",
    "postmerge": "/bin/true",
    "precommit": "/bin/true",
    "prepush": "npm test",
@@ -0,0 +1,122 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
+
+function get_status() {
+    key="$1"
+    if status=$($curl -q -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
+        currentValue=$(echo "${status}" | python3 -c 'import sys, json; print(json.dumps(json.load(sys.stdin)[sys.argv[1]]))' "${key}")
+        echo "${currentValue}"
+        return 0
+    fi
+
+    return 1
+}
+
+function wait_for_status() {
+    key="$1"
+    expectedValue="$2"
+
+    echo "wait_for_status: $key to be $expectedValue"
+    while true; do
+        if currentValue=$(get_status "${key}"); then
+            echo "wait_for_status: $key is current: $currentValue expecting: $expectedValue"
+            if [[ "${currentValue}" == $expectedValue ]]; then
+                break
+            fi
+        fi
+        sleep 3
+    done
+}
+
+domain=""
+domainProvider=""
+domainConfigJson="{}"
+domainTlsProvider="letsencrypt-prod"
+adminUsername="superadmin"
+adminPassword="Secret123#"
+adminEmail="admin@server.local"
+appstoreUserId=""
+appstoreToken=""
+backupDir="/var/backups"
+
+args=$(getopt -o "" -l "domain:,domain-provider:,domain-tls-provider:,admin-username:,admin-password:,admin-email:,appstore-user:,appstore-token:,backup-dir:" -n "$0" -- "$@")
+eval set -- "${args}"
+
+while true; do
+    case "$1" in
+    --domain) domain="$2"; shift 2;;
+    --domain-provider) domainProvider="$2"; shift 2;;
+    --domain-tls-provider) domainTlsProvider="$2"; shift 2;;
+    --admin-username) adminUsername="$2"; shift 2;;
+    --admin-password) adminPassword="$2"; shift 2;;
+    --admin-email) adminEmail="$2"; shift 2;;
+    --appstore-user) appstoreUser="$2"; shift 2;;
+    --appstore-token) appstoreToken="$2"; shift 2;;
+    --backup-dir) backupDir="$2"; shift 2;;
+    --) break;;
+    *) echo "Unknown option $1"; exit 1;;
+    esac
+done
+
+echo "=> Waiting for cloudron to be ready"
+wait_for_status "version" '*'
+
+if [[ $(get_status "webadminStatus") != *'"tls": true'* ]]; then
+    echo "=> Domain setup"
+    dnsSetupData=$(printf '{ "domain": "%s", "adminFqdn": "%s", "provider": "%s", "config": %s, "tlsConfig": { "provider": "%s" } }' "${domain}" "my.${domain}" "${domainProvider}" "$domainConfigJson" "${domainTlsProvider}")
+
+    if ! $curl -X POST -H "Content-Type: application/json" -d "${dnsSetupData}" http://localhost:3000/api/v1/cloudron/dns_setup; then
+        echo "DNS Setup Failed"
+        exit 1
+    fi
+
+    wait_for_status "webadminStatus" '*"tls": true*'
+else
+    echo "=> Skipping Domain setup"
+fi
+
+activationData=$(printf '{"username": "%s", "password":"%s", "email": "%s" }' "${adminUsername}" "${adminPassword}" "${adminEmail}")
+if [[ $(get_status "activated") == "false" ]]; then
+    echo "=> Activating"
+
+    if ! activationResult=$($curl -X POST -H "Content-Type: application/json" -d "${activationData}" http://localhost:3000/api/v1/cloudron/activate); then
+        echo "Failed to activate with ${activationData}: ${activationResult}"
+        exit 1
+    fi
+
+    wait_for_status "activated" "true"
+else
+    echo "=> Skipping Activation"
+fi
+
+echo "=> Getting token"
+if ! activationResult=$($curl -X POST -H "Content-Type: application/json" -d "${activationData}" http://localhost:3000/api/v1/developer/login); then
+    echo "Failed to login with ${activationData}: ${activationResult}"
+    exit 1
+fi
+
+accessToken=$(echo "${activationResult}" | python3 -c 'import sys, json; print(json.load(sys.stdin)[sys.argv[1]])' "accessToken")
+
+echo "=> Setting up App Store account with accessToken ${accessToken}"
+appstoreData=$(printf '{"userId":"%s", "token":"%s" }' "${appstoreUser}" "${appstoreToken}")
+
+if ! appstoreResult=$($curl -X POST -H "Content-Type: application/json" -d "${appstoreData}" "http://localhost:3000/api/v1/settings/appstore_config?access_token=${accessToken}"); then
+    echo "Failed to setup Appstore account with ${appstoreData}: ${appstoreResult}"
+    exit 1
+fi
+
+echo "=> Setting up Backup Directory with accessToken ${accessToken}"
+backupData=$(printf '{"provider":"filesystem", "key":"", "backupFolder":"%s", "retentionSecs": 864000, "format": "tgz", "externalDisk": true}' "${backupDir}")
+
+chown -R yellowtent:yellowtent "${backupDir}"
+
+if ! backupResult=$($curl -X POST -H "Content-Type: application/json" -d "${backupData}" "http://localhost:3000/api/v1/settings/backup_config?access_token=${accessToken}"); then
+    echo "Failed to setup backup configuration with ${backupDir}: ${backupResult}"
+    exit 1
+fi
+
+echo "=> Done!"
+
@@ -1,106 +0,0 @@
-#!/bin/bash
-
-set -eu -o pipefail
-
-readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400 --http1.1"
-
-ip=""
-dns_config=""
-tls_cert_file=""
-tls_key_file=""
-license_file=""
-backup_config=""
-
-args=$(getopt -o "" -l "ip:,backup-config:,license:,dns-config:,tls-cert:,tls-key:" -n "$0" -- "$@")
-eval set -- "${args}"
-
-while true; do
-    case "$1" in
-    --ip) ip="$2"; shift 2;;
-    --dns-config) dns_config="$2"; shift 2;;
-    --tls-cert) tls_cert_file="$2"; shift 2;;
-    --tls-key) tls_key_file="$2"; shift 2;;
-    --license) license_file="$2"; shift 2;;
-    --backup-config) backup_config="$2"; shift 2;;
-    --) break;;
-    *) echo "Unknown option $1"; exit 1;;
-    esac
-done
-
-# validate arguments in the absence of data
-if [[ -z "${ip}" ]]; then
-    echo "--ip is required"
-    exit 1
-fi
-
-if [[ -z "${dns_config}" ]]; then
-    echo "--dns-config is required"
-    exit 1
-fi
-
-if [[ ! -f "${license_file}" ]]; then
-    echo "--license must be a valid license file"
-    exit 1
-fi
-
-function get_status() {
-    key="$1"
-    if status=$($curl -q -f -k "https://${ip}/api/v1/cloudron/status" 2>/dev/null); then
-        currentValue=$(echo "${status}" | python3 -c 'import sys, json; print(json.dumps(json.load(sys.stdin)[sys.argv[1]]))' "${key}")
-        echo "${currentValue}"
-        return 0
-    fi
-
-    return 1
-}
-
-function wait_for_status() {
-    key="$1"
-    expectedValue="$2"
-
-    echo "wait_for_status: $key to be $expectedValue"
-    while true; do
-        if currentValue=$(get_status "${key}"); then
-            echo "wait_for_status: $key is current: $currentValue expecting: $expectedValue"
-            if [[ "${currentValue}" == $expectedValue ]]; then
-                break
-            fi
-        fi
-        sleep 3
-    done
-}
-
-echo "=> Waiting for cloudron to be ready"
-wait_for_status "version" '*'
-
-domain=$(echo "${dns_config}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["domain"])')
-
-echo "Provisioning Cloudron ${domain}"
-if [[ -n "${tls_cert_file}" && -n "${tls_key_file}" ]]; then
-    tls_cert=$(cat "${tls_cert_file}" | awk '{printf "%s\\n", $0}')
-    tls_key=$(cat "${tls_key_file}" | awk '{printf "%s\\n", $0}')
-    fallback_cert=$(printf '{ "cert": "%s", "key": "%s", "provider": "fallback", "restricted": true }' "${tls_cert}" "${tls_key}")
-else
-    fallback_cert=None
-fi
-
-tls_config='{ "provider": "fallback" }'
-dns_config=$(echo "${dns_config}" | python3 -c "import json,sys;obj=json.load(sys.stdin);obj.update(tlsConfig=${tls_config});obj.update(fallbackCertficate=${fallback_cert});print(json.dumps(obj))")
-
-license=$(cat "${license_file}")
-
-if [[ -z "${backup_config:-}" ]]; then
-    backup_config='{ "provider": "filesystem", "backupFolder": "/var/backups", "format": "tgz" }'
-fi
-
-setupData=$(printf '{ "dnsConfig": %s, "autoconf": { "appstoreConfig": %s, "backupConfig": %s } }' "${dns_config}" "${license}" "${backup_config}")
-
-if ! setupResult=$($curl -kq -X POST -H "Content-Type: application/json" -d "${setupData}" https://${ip}/api/v1/cloudron/setup); then
-    echo "Failed to setup with ${setupData} ${setupResult}"
-    exit 1
-fi
-
-wait_for_status "webadminStatus" '*"tls": true*'
-
-echo "Cloudron is ready at https://my-${domain}"
-
@@ -4,6 +4,7 @@ set -eu -o pipefail

 # change this to a hash when we make a upgrade release
 readonly LOG_FILE="/var/log/cloudron-setup.log"
+readonly DATA_FILE="/root/cloudron-install-data.json"
 readonly MINIMUM_DISK_SIZE_GB="18" # this is the size of "/" and required to fit in docker images 18 is a safe bet for different reporting on 20GB min
 readonly MINIMUM_MEMORY="974"      # this is mostly reported for 1GB main memory (DO 992, EC2 990, Linode 989, Serverdiscounter.com 974)

@@ -47,11 +48,14 @@ edition=""
 requestedVersion=""
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
+prerelease="false"
 sourceTarballUrl=""
 rebootServer="true"
 baseDataDir=""

-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,edition:,skip-reboot" -n "$0" -- "$@")
+echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"
+
+args=$(getopt -o "" -l "help,skip-baseimage-init,data-dir:,provider:,version:,env:,prerelease,edition:,skip-reboot" -n "$0" -- "$@")
 eval set -- "${args}"

 while true; do
@@ -64,13 +68,17 @@ while true; do
        if [[ "$2" == "dev" ]]; then
            apiServerOrigin="https://api.dev.cloudron.io"
            webServerOrigin="https://dev.cloudron.io"
+            prerelease="true"
        elif [[ "$2" == "staging" ]]; then
            apiServerOrigin="https://api.staging.cloudron.io"
            webServerOrigin="https://staging.cloudron.io"
+            prerelease="true"
        fi
        shift 2;;
    --skip-baseimage-init) initBaseImage="false"; shift;;
    --skip-reboot) rebootServer="false"; shift;;
+    --prerelease) prerelease="true"; shift;;
+    --data-dir) baseDataDir=$(realpath "$2"); shift 2;;
    --) break;;
    *) echo "Unknown option $1"; exit 1;;
    esac
@@ -83,18 +91,14 @@ if [[ ${EUID} -ne 0 ]]; then
 fi

 # Only --help works with mismatched ubuntu
-ubuntu_version=$(lsb_release -rs)
-if [[ "${ubuntu_version}" != "16.04" && "${ubuntu_version}" != "18.04" ]]; then
-    echo "Cloudron requires Ubuntu 16.04 or 18.04" > /dev/stderr
+if [[ $(lsb_release -rs) != "16.04" ]]; then
+    echo "Cloudron requires Ubuntu 16.04" > /dev/stderr
    exit 1
 fi

-# Can only write after we have confirmed script has root access
-echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"
-
 # validate arguments in the absence of data
 if [[ -z "${provider}" ]]; then
-    echo "--provider is required (azure, digitalocean, ec2, exoscale, gce, hetzner, lightsail, linode, netcup, ovh, rosehosting, scaleway, vultr or generic)"
+    echo "--provider is required (azure, digitalocean, ec2, exoscale, gce, hetzner, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic)"
    exit 1
 elif [[  \
            "${provider}" != "ami" && \
@@ -110,14 +114,13 @@ elif [[  \
            "${provider}" != "hetzner" && \
            "${provider}" != "lightsail" && \
            "${provider}" != "linode" && \
-            "${provider}" != "netcup" && \
            "${provider}" != "ovh" && \
            "${provider}" != "rosehosting" && \
            "${provider}" != "scaleway" && \
            "${provider}" != "vultr" && \
            "${provider}" != "generic" \
        ]]; then
-    echo "--provider must be one of: azure, cloudscale.ch, digitalocean, ec2, exoscale, galaxygate, gce, hetzner, lightsail, linode, netcup, ovh, rosehosting, scaleway, vultr or generic"
+    echo "--provider must be one of: azure, cloudscale.ch, digitalocean, ec2, exoscale, galaxygate, gce, hetzner, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic"
    exit 1
 fi

@@ -138,12 +141,6 @@ echo " Join us at https://forum.cloudron.io for any questions."
 echo ""

 if [[ "${initBaseImage}" == "true" ]]; then
-    echo "=> Ensure required apt sources"
-    if ! add-apt-repository universe &>> "${LOG_FILE}"; then
-        echo "Could not add required apt sources (for nginx-full). See ${LOG_FILE}"
-        exit 1
-    fi
-
    echo "=> Updating apt and installing script dependencies"
    if ! apt-get update &>> "${LOG_FILE}"; then
        echo "Could not update package repositories. See ${LOG_FILE}"
@@ -157,7 +154,7 @@ if [[ "${initBaseImage}" == "true" ]]; then
 fi

 echo "=> Checking version"
-if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?boxVersion=${requestedVersion}"); then
+if ! releaseJson=$($curl -s "${apiServerOrigin}/api/v1/releases?prerelease=${prerelease}&boxVersion=${requestedVersion}"); then
    echo "Failed to get release information"
    exit 1
 fi
@@ -173,6 +170,19 @@ if ! sourceTarballUrl=$(echo "${releaseJson}" | python3 -c 'import json,sys;obj=
    exit 1
 fi

+# Build data
+# from 1.9, we use autoprovision.json
+data=$(cat <<EOF
+{
+    "provider": "${provider}",
+    "edition": "${edition}",
+    "apiServerOrigin": "${apiServerOrigin}",
+    "webServerOrigin": "${webServerOrigin}",
+    "version": "${version}"
+}
+EOF
+)
+
 echo "=> Downloading version ${version} ..."
 box_src_tmp_dir=$(mktemp -dt box-src-XXXXXX)

@@ -190,44 +200,13 @@ if [[ "${initBaseImage}" == "true" ]]; then
    echo ""
 fi

-# NOTE: this install script only supports 3.x and above
 echo "=> Installing version ${version} (this takes some time) ..."
-if [[ "${version}" =~ 3\.[0-2]+\.[0-9]+ ]]; then
-    readonly DATA_FILE="/root/cloudron-install-data.json"
-    data=$(cat <<EOF
-{
-    "provider": "${provider}",
-    "edition": "${edition}",
-    "apiServerOrigin": "${apiServerOrigin}",
-    "webServerOrigin": "${webServerOrigin}",
-    "version": "${version}"
-}
-EOF
-)
-    echo "${data}" > "${DATA_FILE}"
-
-    if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" --data-file "${DATA_FILE}" &>> "${LOG_FILE}"; then
-        echo "Failed to install cloudron. See ${LOG_FILE} for details"
-        exit 1
-    fi
-
-    rm "${DATA_FILE}"
-else
-    mkdir -p /etc/cloudron
-    cat > "/etc/cloudron/cloudron.conf" <<CONF_END
-{
-    "apiServerOrigin": "${apiServerOrigin}",
-    "webServerOrigin": "${webServerOrigin}",
-    "provider": "${provider}",
-    "edition": "${edition}"
-}
-CONF_END
-
-    if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
-        echo "Failed to install cloudron. See ${LOG_FILE} for details"
-        exit 1
-    fi
+echo "${data}" > "${DATA_FILE}"
+if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" --data-file "${DATA_FILE}" --data-dir "${baseDataDir}" &>> "${LOG_FILE}"; then
+    echo "Failed to install cloudron. See ${LOG_FILE} for details"
+    exit 1
 fi
+rm "${DATA_FILE}"

 echo -n "=> Waiting for cloudron to be ready (this takes some time) ..."
 while true; do
@@ -238,15 +217,10 @@ while true; do
    sleep 10
 done

-echo -e "\n\n${GREEN}Visit https://<IP> and accept the self-signed certificate to finish setup.${DONE}\n"
+echo -e "\n\n${GREEN}Visit https://<IP> and accept the self-signed certificate to finish setup.${DONE}"

 if [[ "${rebootServer}" == "true" ]]; then
-    systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables
-
-    read -p "This server has to rebooted to apply all the settings. Reboot now ? [Y/n] " yn
-    yn=${yn:-y}
-    case $yn in
-        [Yy]* ) systemctl reboot;;
-        * ) exit;;
-    esac
+    echo -e "\n${RED}Rebooting this server now to let changes take effect.${DONE}\n"
+    systemctl stop mysql # sometimes mysql ends up having corrupt privilege tables
+    systemctl reboot
 fi
@@ -1,115 +0,0 @@
-#!/bin/bash
-
-# This script collects diagnostic information to help debug server related issues
-# It also enables SSH access for the cloudron support team
-
-PASTEBIN="https://paste.cloudron.io"
-OUT="/tmp/cloudron-support.log"
-LINE="\n========================================================\n"
-CLOUDRON_SUPPORT_PUBLIC_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQVilclYAIu+ioDp/sgzzFz6YU0hPcRYY7ze/LiF/lC7uQqK062O54BFXTvQ3ehtFZCx3bNckjlT2e6gB8Qq07OM66De4/S/g+HJW4TReY2ppSPMVNag0TNGxDzVH8pPHOysAm33LqT2b6L/wEXwC6zWFXhOhHjcMqXvi8Ejaj20H1HVVcf/j8qs5Thkp9nAaFTgQTPu8pgwD8wDeYX1hc9d0PYGesTADvo6HF4hLEoEnefLw7PaStEbzk2fD3j7/g5r5HcgQQXBe74xYZ/1gWOX2pFNuRYOBSEIrNfJEjFJsqk3NR1+ZoMGK7j+AZBR4k0xbrmncQLcQzl6MMDzkp support@cloudron.io"
-HELP_MESSAGE="
-This script collects diagnostic information to help debug server related issues
-
- Options:
-   --enable-ssh    Enable SSH access for the Cloudron support team
-   --help          Show this message
-"
-
-# We require root
-if [[ ${EUID} -ne 0 ]]; then
-    echo "This script should be run as root." > /dev/stderr
-    exit 1
-fi
-
-enableSSH="false"
-
-args=$(getopt -o "" -l "help,enable-ssh" -n "$0" -- "$@")
-eval set -- "${args}"
-
-while true; do
-    case "$1" in
-    --help) echo -e "${HELP_MESSAGE}"; exit 0;;
-    --enable-ssh) enableSSH="true"; shift;;
-    --) break;;
-    *) echo "Unknown option $1"; exit 1;;
-    esac
-done
-
-# check if at least 10mb root partition space is available
-if [[ "`df --output="avail" / | sed -n 2p`" -lt "10240" ]]; then
-    echo "No more space left on /"
-    echo "This is likely the root case of the issue. Free up some space and also check other partitions below:"
-    echo ""
-    df -h
-    echo ""
-    echo "To recover from a full disk, follow the guide at https://cloudron.io/documentation/server/#recovery-after-disk-full"
-    exit 1
-fi
-
-# check for at least 5mb free /tmp space for the log file
-if [[ "`df --output="avail" /tmp | sed -n 2p`" -lt "5120" ]]; then
-    echo "Not enough space left on /tmp"
-    echo "Free up some space first by deleting files from /tmp"
-    exit 1
-fi
-
-echo -n "Generating Cloudron Support stats..."
-
-# clear file
-rm -rf $OUT
-
-ssh_port=$(cat /etc/ssh/sshd_config | grep "Port " | sed -e "s/.*Port //")
-if [[ $SUDO_USER == "" ]]; then
-    ssh_user="root"
-    ssh_folder="/root/.ssh/"
-    authorized_key_file="${ssh_folder}/authorized_keys"
-else
-    ssh_user="$SUDO_USER"
-    ssh_folder="/home/$SUDO_USER/.ssh/"
-    authorized_key_file="${ssh_folder}/authorized_keys"
-fi
-
-echo -e $LINE"SSH"$LINE >> $OUT
-echo "Username: ${ssh_user}" >> $OUT
-echo "Port:     ${ssh_port}" >> $OUT
-
-echo -e $LINE"cloudron.conf"$LINE >> $OUT
-cat /etc/cloudron/cloudron.conf &>> $OUT
-
-echo -e $LINE"Docker container"$LINE >> $OUT
-if ! timeout --kill-after 10s 15s docker ps -a &>> $OUT 2>&1; then
-    echo -e "Docker is not responding" >> $OUT
-fi
-
-echo -e $LINE"Filesystem stats"$LINE >> $OUT
-df -h &>> $OUT
-
-echo -e $LINE"System daemon status"$LINE >> $OUT
-systemctl status --lines=100 cloudron.target box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
-
-echo -e $LINE"Box logs"$LINE >> $OUT
-tail -n 100 /home/yellowtent/platformdata/logs/box.log &>> $OUT
-
-echo -e $LINE"Firewall chains"$LINE >> $OUT
-iptables -L &>> $OUT
-
-echo "Done"
-
-echo -n "Uploading information..."
-# for some reason not using $(cat $OUT) will not contain newlines!?
-paste_key=$(curl -X POST ${PASTEBIN}/documents --silent -d "$(cat $OUT)" | python3 -c "import sys, json; print(json.load(sys.stdin)['key'])")
-echo "Done"
-
-if [[ "${enableSSH}" == "true" ]]; then
-    echo -n "Enabling ssh access for the Cloudron support team..."
-    mkdir -p "${ssh_folder}"
-    echo "${CLOUDRON_SUPPORT_PUBLIC_KEY}" >> ${authorized_key_file}
-    chown -R ${ssh_user} "${ssh_folder}"
-    chmod 600 "${authorized_key_file}"
-    echo "Done"
-fi
-
-echo ""
-echo "Please email the following link to support@cloudron.io"
-echo ""
-echo "${PASTEBIN}/${paste_key}"
@@ -7,28 +7,21 @@ set -eu
 [[ $(uname -s) == "Darwin" ]] && GNU_GETOPT="/usr/local/opt/gnu-getopt/bin/getopt" || GNU_GETOPT="getopt"
 readonly GNU_GETOPT

-args=$(${GNU_GETOPT} -o "" -l "output:,version:" -n "$0" -- "$@")
+args=$(${GNU_GETOPT} -o "" -l "output:" -n "$0" -- "$@")
 eval set -- "${args}"

 readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"

 bundle_file=""
-version=""

 while true; do
    case "$1" in
    --output) bundle_file="$2"; shift 2;;
-    --version) version="$2"; shift 2;;
    --) break;;
    *) echo "Unknown option $1"; exit 1;;
    esac
 done

-if [[ -z "${version}" ]]; then
-    echo "--version is required"
-    exit 1
-fi
-
 readonly TMPDIR=${TMPDIR:-/tmp} # why is this not set on mint?

 if ! $(cd "${SOURCE_DIR}" && git diff --exit-code >/dev/null); then
@@ -48,16 +41,19 @@ fi

 box_version=$(cd "${SOURCE_DIR}" && git rev-parse "HEAD")
 branch=$(git rev-parse --abbrev-ref HEAD)
-dashboard_version=$(cd "${SOURCE_DIR}/../dashboard" && git fetch && git rev-parse "${branch}")
+if [[ "${branch}" == "master" ]]; then
+    dashboard_version=$(cd "${SOURCE_DIR}/../dashboard" && git rev-parse "${branch}")
+else
+    dashboard_version=$(cd "${SOURCE_DIR}/../dashboard" && git fetch && git rev-parse "${branch}")
+fi
 bundle_dir=$(mktemp -d -t box 2>/dev/null || mktemp -d box-XXXXXXXXXX --tmpdir=$TMPDIR)
-[[ -z "$bundle_file" ]] && bundle_file="${TMPDIR}/box-${box_version:0:10}-${dashboard_version:0:10}-${version}.tar.gz"
+[[ -z "$bundle_file" ]] && bundle_file="${TMPDIR}/box-${box_version:0:10}-${dashboard_version:0:10}.tar.gz"

 chmod "o+rx,g+rx" "${bundle_dir}" # otherwise extracted tarball director won't be readable by others/group
 echo "==> Checking out code box version [${box_version}] and dashboard version [${dashboard_version}] into ${bundle_dir}"
 (cd "${SOURCE_DIR}" && git archive --format=tar ${box_version} | (cd "${bundle_dir}" && tar xf -))
 (cd "${SOURCE_DIR}/../dashboard" && git archive --format=tar ${dashboard_version} | (mkdir -p "${bundle_dir}/dashboard.build" && cd "${bundle_dir}/dashboard.build" && tar xf -))
 (cp "${SOURCE_DIR}/../dashboard/LICENSE" "${bundle_dir}")
-echo "${version}" > "${bundle_dir}/VERSION"

 echo "==> Installing modules for dashboard asset generation"
 (cd "${bundle_dir}/dashboard.build" && npm install --production)
@@ -1,9 +1,5 @@
 #!/bin/bash

-# This script is run before the box code is switched. This means that we can
-# put network related/curl downloads here. If the script fails, the old code
-# will continue to run
-
 set -eu -o pipefail

 if [[ ${EUID} -ne 0 ]]; then
@@ -14,12 +10,29 @@ fi
 readonly USER=yellowtent
 readonly BOX_SRC_DIR=/home/${USER}/box
 readonly BASE_DATA_DIR=/home/${USER}
+readonly CLOUDRON_CONF=/home/yellowtent/configs/cloudron.conf

 readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
 readonly script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 readonly box_src_tmp_dir="$(realpath ${script_dir}/..)"

-readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")
+readonly is_update=$([[ -f "${CLOUDRON_CONF}" ]] && echo "yes" || echo "no")
+
+arg_data=""
+arg_data_dir=""
+
+args=$(getopt -o "" -l "data:,data-file:,data-dir:" -n "$0" -- "$@")
+eval set -- "${args}"
+
+while true; do
+    case "$1" in
+    --data) arg_data="$2"; shift 2;;
+    --data-file) arg_data=$(cat $2); shift 2;;
+    --data-dir) arg_data_dir="$2"; shift 2;;
+    --) break;;
+    *) echo "Unknown option $1"; exit 1;;
+    esac
+done

 echo "==> installer: updating docker"
 if [[ $(docker version --format {{.Client.Version}}) != "18.03.1-ce" ]]; then
@@ -81,21 +94,6 @@ if [[ ${try} -eq 10 ]]; then
    exit 4
 fi

-echo "==> installer: downloading new addon images"
-images=$(node -e "var i = require('${box_src_tmp_dir}/src/infra_version.js'); console.log(i.baseImages.map(function (x) { return x.tag; }).join(' '), Object.keys(i.images).map(function (x) { return i.images[x].tag; }).join(' '));")
-
-echo -e "\tPulling docker images: ${images}"
-for image in ${images}; do
-    if ! docker pull "${image}"; then           # this pulls the image using the sha256
-        echo "==> installer: Could not pull ${image}"
-        exit 5
-    fi
-    if ! docker pull "${image%@sha256:*}"; then  # this will tag the image for readability
-        echo "==> installer: Could not pull ${image%@sha256:*}"
-        exit 6
-   fi
-done
-
 echo "==> installer: update cloudron-syslog"
 CLOUDRON_SYSLOG_DIR=/usr/local/cloudron-syslog
 CLOUDRON_SYSLOG="${CLOUDRON_SYSLOG_DIR}/bin/cloudron-syslog"
@@ -117,6 +115,15 @@ if [[ "${is_update}" == "yes" ]]; then
    ${BOX_SRC_DIR}/setup/stop.sh
 fi

+# setup links to data directory
+if [[ -n "${arg_data_dir}" ]]; then
+    echo "==> installer: setting up links to data directory"
+    mkdir "${arg_data_dir}/appsdata"
+    ln -s "${arg_data_dir}/appsdata" "${BASE_DATA_DIR}/appsdata"
+    mkdir "${arg_data_dir}/platformdata"
+    ln -s "${arg_data_dir}/platformdata" "${BASE_DATA_DIR}/platformdata"
+fi
+
 # ensure we are not inside the source directory, which we will remove now
 cd /root

@@ -126,4 +133,4 @@ mv "${box_src_tmp_dir}" "${BOX_SRC_DIR}"
 chown -R "${USER}:${USER}" "${BOX_SRC_DIR}"

 echo "==> installer: calling box setup script"
-"${BOX_SRC_DIR}/setup/start.sh"
+"${BOX_SRC_DIR}/setup/start.sh" --data "${arg_data}"
@@ -0,0 +1,76 @@
+#!/bin/bash
+
+source_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+json="${source_dir}/../node_modules/.bin/json"
+
+arg_api_server_origin=""
+arg_fqdn=""         # remove after 1.10
+arg_admin_domain=""
+arg_admin_location=""
+arg_admin_fqdn=""
+arg_retire_reason=""
+arg_retire_info=""
+arg_version=""
+arg_web_server_origin=""
+arg_provider=""
+arg_is_demo="false"
+arg_edition=""
+
+args=$(getopt -o "" -l "data:,retire-reason:,retire-info:" -n "$0" -- "$@")
+eval set -- "${args}"
+
+while true; do
+    case "$1" in
+    --retire-reason)
+        arg_retire_reason="$2"
+        shift 2
+        ;;
+    --retire-info)
+        arg_retire_info="$2"
+        shift 2
+        ;;
+    --data)
+        # these params must be valid in all cases
+        arg_fqdn=$(echo "$2" | $json fqdn)
+        arg_admin_fqdn=$(echo "$2" | $json adminFqdn)
+
+        arg_admin_location=$(echo "$2" | $json adminLocation)
+        [[ "${arg_admin_location}" == "" ]] && arg_admin_location="my"
+
+        arg_admin_domain=$(echo "$2" | $json adminDomain)
+        [[ "${arg_admin_domain}" == "" ]] && arg_admin_domain="${arg_fqdn}"
+
+        # only update/restore have this valid (but not migrate)
+        arg_api_server_origin=$(echo "$2" | $json apiServerOrigin)
+        [[ "${arg_api_server_origin}" == "" ]] && arg_api_server_origin="https://api.cloudron.io"
+        arg_web_server_origin=$(echo "$2" | $json webServerOrigin)
+        [[ "${arg_web_server_origin}" == "" ]] && arg_web_server_origin="https://cloudron.io"
+
+        # TODO check if and where this is used
+        arg_version=$(echo "$2" | $json version)
+
+        # read possibly empty parameters here
+        arg_is_demo=$(echo "$2" | $json isDemo)
+        [[ "${arg_is_demo}" == "" ]] && arg_is_demo="false"
+
+        arg_provider=$(echo "$2" | $json provider)
+        [[ "${arg_provider}" == "" ]] && arg_provider="generic"
+
+        arg_edition=$(echo "$2" | $json edition)
+        [[ "${arg_edition}" == "" ]] && arg_edition=""
+
+        shift 2
+        ;;
+    --) break;;
+    *) echo "Unknown option $1"; exit 1;;
+    esac
+done
+
+echo "Parsed arguments:"
+echo "api server: ${arg_api_server_origin}"
+echo "admin fqdn: ${arg_admin_fqdn}"
+echo "fqdn: ${arg_fqdn}"
+echo "version: ${arg_version}"
+echo "web server: ${arg_web_server_origin}"
+echo "provider: ${arg_provider}"
+echo "edition: ${arg_edition}"
@@ -2,9 +2,6 @@

 set -eu -o pipefail

-# This script is run after the box code is switched. This means that this script
-# should pretty much always succeed. No network logic/download code here.
-
 echo "==> Cloudron Start"

 readonly USER="yellowtent"
@@ -15,8 +12,17 @@ readonly APPS_DATA_DIR="${HOME_DIR}/appsdata" # app data
 readonly BOX_DATA_DIR="${HOME_DIR}/boxdata" # box data
 readonly CONFIG_DIR="${HOME_DIR}/configs"

+readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
+
 readonly script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly json="$(realpath ${script_dir}/../node_modules/.bin/json)"
+
+source "${script_dir}/argparser.sh" "$@" # this injects the arg_* variables used below
+
+echo "==> Configuring host"
+sed -e 's/^#NTP=/NTP=0.ubuntu.pool.ntp.org 1.ubuntu.pool.ntp.org 2.ubuntu.pool.ntp.org 3.ubuntu.pool.ntp.org/' -i /etc/systemd/timesyncd.conf
+timedatectl set-ntp 1
+timedatectl set-timezone UTC
+hostnamectl set-hostname "${arg_admin_fqdn}"

 echo "==> Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
@@ -42,6 +48,19 @@ if [[ ! -f /etc/systemd/system/docker.service.d/cloudron.conf ]] || ! diff -q /e
 fi
 docker network create --subnet=172.18.0.0/16 cloudron || true

+# caas has ssh on port 202 and we disable password login
+if [[ "${arg_provider}" == "caas" ]]; then
+    # https://stackoverflow.com/questions/4348166/using-with-sed on why ? must be escaped
+    sed -e 's/^#\?PermitRootLogin .*/PermitRootLogin without-password/g' \
+        -e 's/^#\?PermitEmptyPasswords .*/PermitEmptyPasswords no/g' \
+        -e 's/^#\?PasswordAuthentication .*/PasswordAuthentication no/g' \
+        -e 's/^#\?Port .*/Port 202/g' \
+        -i /etc/ssh/sshd_config
+
+    # required so we can connect to this machine since port 22 is blocked by iptables by now
+    systemctl reload sshd
+fi
+
 mkdir -p "${BOX_DATA_DIR}"
 mkdir -p "${APPS_DATA_DIR}"

@@ -52,13 +71,12 @@ mkdir -p "${PLATFORM_DATA_DIR}/graphite"
 mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
-mkdir -p "${PLATFORM_DATA_DIR}/redis"
 mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
 mkdir -p "${PLATFORM_DATA_DIR}/backup"
-mkdir -p "${PLATFORM_DATA_DIR}/logs/backup" "${PLATFORM_DATA_DIR}/logs/updater" "${PLATFORM_DATA_DIR}/logs/tasks"
+mkdir -p "${PLATFORM_DATA_DIR}/logs/backup"
 mkdir -p "${PLATFORM_DATA_DIR}/update"

 mkdir -p "${BOX_DATA_DIR}/appicons"
@@ -91,11 +109,6 @@ setfacl -n -m u:${USER}:r /var/log/journal/*/system.journal
 echo "==> Creating config directory"
 mkdir -p "${CONFIG_DIR}"

-# remove old cloudron.conf. Can be removed after 3.4
-rm -f "${CONFIG_DIR}/cloudron.conf"
-$json -f /etc/cloudron/cloudron.conf -I -e "delete this.version" # remove the version field
-chown -R "${USER}" /etc/cloudron
-
 echo "==> Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
@@ -139,8 +152,8 @@ echo "==> Configuring logrotate"
 if ! grep -q "^include ${PLATFORM_DATA_DIR}/logrotate.d" /etc/logrotate.conf; then
    echo -e "\ninclude ${PLATFORM_DATA_DIR}/logrotate.d\n" >> /etc/logrotate.conf
 fi
-cp "${script_dir}/start/box-logrotate" "${script_dir}/start/app-logrotate" "${PLATFORM_DATA_DIR}/logrotate.d/"
-chown root:root "${PLATFORM_DATA_DIR}/logrotate.d/box-logrotate" "${PLATFORM_DATA_DIR}/logrotate.d/app-logrotate"
+cp "${script_dir}/start/app-logrotate" "${PLATFORM_DATA_DIR}/logrotate.d/app-logrotate"
+chown root:root "${PLATFORM_DATA_DIR}/logrotate.d/app-logrotate"

 echo "==> Adding motd message for admins"
 cp "${script_dir}/start/cloudron-motd" /etc/update-motd.d/92-cloudron
@@ -158,8 +171,14 @@ if ! grep -q "^Restart=" /etc/systemd/system/multi-user.target.wants/nginx.servi
    echo -e "\n[Service]\nRestart=always\n" >> /etc/systemd/system/multi-user.target.wants/nginx.service
    systemctl daemon-reload
 fi
+# remove this migration after 1.10
+[[ -f /etc/nginx/cert/host.cert ]] && cp /etc/nginx/cert/host.cert "/etc/nginx/cert/${arg_admin_domain}.host.cert"
+[[ -f /etc/nginx/cert/host.key ]] && cp /etc/nginx/cert/host.key "/etc/nginx/cert/${arg_admin_domain}.host.key"
 systemctl start nginx

+# bookkeep the version as part of data
+echo "{ \"version\": \"${arg_version}\", \"apiServerOrigin\": \"${arg_api_server_origin}\" }" > "${BOX_DATA_DIR}/version"
+
 # restart mysql to make sure it has latest config
 if [[ ! -f /etc/mysql/mysql.cnf ]] || ! diff -q "${script_dir}/start/mysql.cnf" /etc/mysql/mysql.cnf >/dev/null; then
    # wait for all running mysql jobs
@@ -189,6 +208,28 @@ cd "${BOX_SRC_DIR}"
 BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up
 EOF

+echo "==> Creating cloudron.conf"
+cat > "${CONFIG_DIR}/cloudron.conf" <<CONF_END
+{
+    "version": "${arg_version}",
+    "apiServerOrigin": "${arg_api_server_origin}",
+    "webServerOrigin": "${arg_web_server_origin}",
+    "adminDomain": "${arg_admin_domain}",
+    "adminFqdn": "${arg_admin_fqdn}",
+    "adminLocation": "${arg_admin_location}",
+    "provider": "${arg_provider}",
+    "isDemo": ${arg_is_demo},
+    "edition": "${arg_edition}"
+}
+CONF_END
+
+echo "==> Creating config.json for dashboard"
+cat > "${BOX_SRC_DIR}/dashboard/dist/config.json" <<CONF_END
+{
+    "webServerOrigin": "${arg_web_server_origin}"
+}
+CONF_END
+
 if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
    echo "==> Generating dhparams (takes forever)"
    openssl dhparam -out "${BOX_DATA_DIR}/dhparams.pem" 2048
@@ -199,11 +240,9 @@ fi

 echo "==> Changing ownership"
 chown "${USER}:${USER}" -R "${CONFIG_DIR}"
-# be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/nginx" "${PLATFORM_DATA_DIR}/collectd" "${PLATFORM_DATA_DIR}/addons" "${PLATFORM_DATA_DIR}/acme" "${PLATFORM_DATA_DIR}/backup" "${PLATFORM_DATA_DIR}/logs" "${PLATFORM_DATA_DIR}/update"
 chown "${USER}:${USER}" "${PLATFORM_DATA_DIR}/INFRA_VERSION" 2>/dev/null || true
 chown "${USER}:${USER}" "${PLATFORM_DATA_DIR}"
-chown "${USER}:${USER}" "${APPS_DATA_DIR}"

 # logrotate files have to be owned by root, this is here to fixup existing installations where we were resetting the owner to yellowtent
 chown root:root -R "${PLATFORM_DATA_DIR}/logrotate.d"
@@ -1,9 +0,0 @@
-# logrotate config for box logs
-
-/home/yellowtent/platformdata/logs/box.log {
-    rotate 10
-    size 10M
-    # we never compress so we can simply tail the files
-    nocompress
-    copytruncate
-}
@@ -162,7 +162,7 @@ server {
        # graphite paths (uncomment block below and visit /graphite/index.html)
        # remember to comment out the CSP policy as well to access the graphite dashboard
        # location ~ ^/(graphite|content|metrics|dashboard|render|browser|composer)/ {
-        #     proxy_pass   http://127.0.0.1:8417;
+        #     proxy_pass   http://127.0.0.1:8000;
        #     client_max_body_size 1m;
        # }

@@ -1,11 +1,11 @@
 # sudo logging breaks journalctl output with very long urls (systemd bug)
 Defaults !syslog

-Defaults!/home/yellowtent/box/src/scripts/rmvolume.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmvolume.sh
+Defaults!/home/yellowtent/box/src/scripts/createappdir.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/createappdir.sh

-Defaults!/home/yellowtent/box/src/scripts/rmaddondir.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmaddondir.sh
+Defaults!/home/yellowtent/box/src/scripts/rmappdir.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmappdir.sh

 Defaults!/home/yellowtent/box/src/scripts/reloadnginx.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reloadnginx.sh
@@ -31,15 +31,9 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/authorized_keys
 Defaults!/home/yellowtent/box/src/scripts/configurelogrotate.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/configurelogrotate.sh

-Defaults!/home/yellowtent/box/src/scripts/backupupload.js env_keep="HOME BOX_ENV"
-Defaults!/home/yellowtent/box/src/scripts/backupupload.js closefrom_override
-yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/backupupload.js
+Defaults!/home/yellowtent/box/src/backuptask.js env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/backuptask.js

 Defaults!/home/yellowtent/box/src/scripts/restart.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restart.sh

-Defaults!/home/yellowtent/box/src/scripts/restartdocker.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartdocker.sh
-
-Defaults!/home/yellowtent/box/src/scripts/restartunbound.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartunbound.sh
@@ -12,8 +12,7 @@ Wants=cloudron-resize-fs.service
 Type=idle
 WorkingDirectory=/home/yellowtent/box
 Restart=always
-; Systemd does not append logs when logging to files, we spawn a shell first and exec to replace it after setting up the pipes
-ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node --max_old_space_size=150 /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
+ExecStart=/usr/bin/node --max_old_space_size=150 /home/yellowtent/box/box.js
 Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box*,connect-lastmile" "BOX_ENV=cloudron" "NODE_ENV=production"
 ; kill apptask processes as well
 KillMode=control-group
@@ -18,7 +18,6 @@ exports = module.exports = {
    getAddonConfigByName: getAddonConfigByName,
    unsetAddonConfig: unsetAddonConfig,
    unsetAddonConfigByAppId: unsetAddonConfigByAppId,
-    getAppIdByAddonConfigValue: getAppIdByAddonConfigValue,

    setHealth: setHealth,
    setInstallationCommand: setInstallationCommand,
@@ -62,6 +61,7 @@ var assert = require('assert'),
    async = require('async'),
    database = require('./database.js'),
    DatabaseError = require('./databaseerror'),
+    mailboxdb = require('./mailboxdb.js'),
    safe = require('safetydance'),
    util = require('util');

@@ -69,7 +69,7 @@ var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationSta
    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
    'apps.accessRestrictionJson', 'apps.restoreConfigJson', 'apps.oldConfigJson', 'apps.updateConfigJson', 'apps.memoryLimit',
    'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson', 'apps.robotsTxt', 'apps.enableBackup',
-    'apps.creationTime', 'apps.updateTime', 'apps.ownerId', 'apps.mailboxName', 'apps.enableAutomaticUpdate', 'apps.ts' ].join(',');
+    'apps.creationTime', 'apps.updateTime', 'apps.ownerId', 'apps.ts' ].join(',');

 var PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');

@@ -106,7 +106,7 @@ function postProcess(result) {
    delete result.environmentVariables;
    delete result.portTypes;

-    for (let i = 0; i < environmentVariables.length; i++) {
+    for (var i = 0; i < environmentVariables.length; i++) {
        result.portBindings[environmentVariables[i]] = { hostPort: parseInt(hostPorts[i], 10), type: portTypes[i] };
    }

@@ -120,7 +120,6 @@ function postProcess(result) {

    result.sso = !!result.sso; // make it bool
    result.enableBackup = !!result.enableBackup; // make it bool
-    result.enableAutomaticUpdate = !!result.enableAutomaticUpdate; // make it bool

    assert(result.debugModeJson === null || typeof result.debugModeJson === 'string');
    result.debugMode = safe.JSON.parse(result.debugModeJson);
@@ -131,14 +130,6 @@ function postProcess(result) {
        delete d.appId;
        delete d.type;
    });
-
-    let envNames = JSON.parse(result.envNames), envValues = JSON.parse(result.envValues);
-    delete result.envNames;
-    delete result.envValues;
-    result.env = {};
-    for (let i = 0; i < envNames.length; i++) { // NOTE: envNames is [ null ] when env of an app is empty
-        if (envNames[i]) result.env[envNames[i]] = envValues[i];
-    }
 }

 function get(id, callback) {
@@ -146,11 +137,9 @@ function get(id, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes, '
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes'
        + ' FROM apps'
        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -164,7 +153,7 @@ function get(id, callback) {
            postProcess(result[0]);

            callback(null, result[0]);
-        });
+        })
    });
 }

@@ -173,11 +162,9 @@ function getByHttpPort(httpPort, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-    + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-    + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+    + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes'
        + ' FROM apps'
        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -199,11 +186,9 @@ function getByContainerId(containerId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes'
        + ' FROM apps'
        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -224,11 +209,9 @@ function getAll(callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + APPS_FIELDS_PREFIXED + ','
-        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,'
-        + 'JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues'
+        + 'GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes'
        + ' FROM apps'
        + '  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId'
-        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -276,15 +259,13 @@ function add(id, appStoreId, manifest, location, domain, ownerId, portBindings,
    var sso = 'sso' in data ? data.sso : null;
    var robotsTxt = 'robotsTxt' in data ? data.robotsTxt : null;
    var debugModeJson = data.debugMode ? JSON.stringify(data.debugMode) : null;
-    var env = data.env || {};
-    const mailboxName = data.mailboxName || null;

    var queries = [];

    queries.push({
-        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions, restoreConfigJson, sso, debugModeJson, robotsTxt, ownerId, mailboxName) ' +
-            ' VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        args: [ id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions, restoreConfigJson, sso, debugModeJson, robotsTxt, ownerId, mailboxName ]
+        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions, restoreConfigJson, sso, debugModeJson, robotsTxt, ownerId) ' +
+            ' VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
+        args: [ id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions, restoreConfigJson, sso, debugModeJson, robotsTxt, ownerId ]
    });

    queries.push({
@@ -299,12 +280,13 @@ function add(id, appStoreId, manifest, location, domain, ownerId, portBindings,
        });
    });

-    Object.keys(env).forEach(function (name) {
+    // only allocate a mailbox if mailboxName is set
+    if (data.mailboxName) {
        queries.push({
-            query: 'INSERT INTO appEnvVars (appId, name, value) VALUES (?, ?, ?)',
-            args: [ id, name, env[name] ]
+            query: 'INSERT INTO mailboxes (name, type, domain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?)',
+            args: [ data.mailboxName, mailboxdb.TYPE_MAILBOX, domain, id, mailboxdb.OWNER_TYPE_APP ]
        });
-    });
+    }

    if (data.alternateDomains) {
        data.alternateDomains.forEach(function (d) {
@@ -370,8 +352,8 @@ function del(id, callback) {

    var queries = [
        { query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ] },
+        { query: 'DELETE FROM mailboxes WHERE ownerId=?', args: [ id ] },
        { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
-        { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
        { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
    ];

@@ -390,7 +372,6 @@ function clear(callback) {
        database.query.bind(null, 'DELETE FROM subdomains'),
        database.query.bind(null, 'DELETE FROM appPortBindings'),
        database.query.bind(null, 'DELETE FROM appAddonConfigs'),
-        database.query.bind(null, 'DELETE FROM appEnvVars'),
        database.query.bind(null, 'DELETE FROM apps')
    ], function (error) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -410,7 +391,6 @@ function updateWithConstraints(id, app, constraints, callback) {
    assert(!('portBindings' in app) || typeof app.portBindings === 'object');
    assert(!('accessRestriction' in app) || typeof app.accessRestriction === 'object' || app.accessRestriction === '');
    assert(!('alternateDomains' in app) || Array.isArray(app.alternateDomains));
-    assert(!('env' in app) || typeof app.env === 'object');

    var queries = [ ];

@@ -424,19 +404,12 @@ function updateWithConstraints(id, app, constraints, callback) {
        });
    }

-    if ('env' in app) {
-        queries.push({ query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] });
-
-        Object.keys(app.env).forEach(function (name) {
-            queries.push({
-                query: 'INSERT INTO appEnvVars (appId, name, value) VALUES (?, ?, ?)',
-                args: [ id, name, app.env[name] ]
-            });
-        });
+    if ('location' in app) {
+        queries.push({ query: 'UPDATE subdomains SET subdomain = ? WHERE appId = ? AND type = ?', args: [ app.location, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
    }

-    if ('location' in app && 'domain' in app) { // must be updated together as they are unique together
-        queries.push({ query: 'UPDATE subdomains SET subdomain = ?, domain = ? WHERE appId = ? AND type = ?', args: [ app.location, app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
+    if ('domain' in app) {
+        queries.push({ query: 'UPDATE subdomains SET domain = ? WHERE appId = ? AND type = ?', args: [ app.domain, id, exports.SUBDOMAIN_TYPE_PRIMARY ]});
    }

    if ('alternateDomains' in app) {
@@ -451,7 +424,7 @@ function updateWithConstraints(id, app, constraints, callback) {
        if (p === 'manifest' || p === 'oldConfig' || p === 'updateConfig' || p === 'restoreConfig' || p === 'accessRestriction' || p === 'debugMode') {
            fields.push(`${p}Json = ?`);
            values.push(JSON.stringify(app[p]));
-        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
+        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains') {
            fields.push(p + ' = ?');
            values.push(app[p]);
        }
@@ -610,20 +583,6 @@ function getAddonConfigByAppId(appId, callback) {
    });
 }

-function getAppIdByAddonConfigValue(addonId, name, value, callback) {
-    assert.strictEqual(typeof addonId, 'string');
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof value, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    database.query('SELECT appId FROM appAddonConfigs WHERE addonId = ? AND name = ? AND value = ?', [ addonId, name, value ], function (error, results) {
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
-        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
-
-        callback(null, results[0].appId);
-    });
-}
-
 function getAddonConfigByName(appId, addonId, name, callback) {
    assert.strictEqual(typeof appId, 'string');
    assert.strictEqual(typeof addonId, 'string');
@@ -658,7 +617,7 @@ function transferOwnership(oldOwnerId, newOwnerId, callback) {
    assert.strictEqual(typeof newOwnerId, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('UPDATE apps SET ownerId=? WHERE ownerId=?', [ newOwnerId, oldOwnerId ], function (error) {
+    database.query('UPDATE apps SET ownerId=? WHERE ownerId=?', [ newOwnerId, oldOwnerId ], function (error, results) {
        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new DatabaseError(DatabaseError.NOT_FOUND, 'No such user'));
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

@@ -12,15 +12,15 @@ var appdb = require('./appdb.js'),
    util = require('util');

 exports = module.exports = {
-    run: run
+    start: start,
+    stop: stop
 };

-const HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
-const UNHEALTHY_THRESHOLD = 10 * 60 * 1000; // 10 minutes
-let gHealthInfo = { }; // { time, emailSent }
-
-const OOM_MAIL_LIMIT = 60 * 60 * 1000; // 60 minutes
-let gLastOomMailTime = Date.now() - (5 * 60 * 1000); // pretend we sent email 5 minutes ago
+var HEALTHCHECK_INTERVAL = 10 * 1000; // every 10 seconds. this needs to be small since the UI makes only healthy apps clickable
+var UNHEALTHY_THRESHOLD = 10 * 60 * 1000; // 10 minutes
+var gHealthInfo = { }; // { time, emailSent }
+var gRunTimeout = null;
+var gDockerEventStream = null;

 function debugApp(app) {
    assert(typeof app === 'object');
@@ -88,9 +88,6 @@ function checkAppHealth(app, callback) {
            return setHealth(app, appdb.HEALTH_DEAD, callback);
        }

-        // non-appstore apps may not have healthCheckPath
-        if (!manifest.healthCheckPath) return setHealth(app, appdb.HEALTH_HEALTHY, callback);
-
        // poll through docker network instead of nginx to bypass any potential oauth proxy
        var healthCheckUrl = 'http://127.0.0.1:' + app.httpPort + manifest.healthCheckPath;
        superagent
@@ -113,58 +110,7 @@ function checkAppHealth(app, callback) {
    });
 }

-/*
-    OOM can be tested using stress tool like so:
-        docker run -ti -m 100M cloudron/base:0.10.0 /bin/bash
-        apt-get update && apt-get install stress
-        stress --vm 1 --vm-bytes 200M --vm-hang 0
-*/
-function processDockerEvents(intervalSecs, callback) {
-    assert.strictEqual(typeof intervalSecs, 'number');
-    assert.strictEqual(typeof callback, 'function');
-
-    const since = ((new Date().getTime() / 1000) - intervalSecs).toFixed(0);
-    const until = ((new Date().getTime() / 1000) - 1).toFixed(0);
-
-    docker.getEvents({ since: since, until: until, filters: JSON.stringify({ event: [ 'oom' ] }) }, function (error, stream) {
-        if (error) return callback(error);
-
-        stream.setEncoding('utf8');
-        stream.on('data', function (data) {
-            var ev = JSON.parse(data);
-            appdb.getByContainerId(ev.id, function (error, app) { // this can error for addons
-                var program = error || !app.appStoreId ? ev.id : app.appStoreId;
-                var context = JSON.stringify(ev);
-                var now = Date.now();
-                if (app) context = context + '\n\n' + JSON.stringify(app, null, 4) + '\n';
-
-                const notifyUser = (!app || !app.debugMode) && (now - gLastOomMailTime > OOM_MAIL_LIMIT);
-
-                debug('OOM Context: %s. notifyUser: %s. lastOomTime: %s (now: %s)', context, notifyUser, gLastOomMailTime, now);
-
-                // do not send mails for dev apps
-                if (notifyUser) {
-                    mailer.oomEvent(program, context); // app can be null if it's an addon crash
-                    gLastOomMailTime = now;
-                }
-            });
-        });
-
-        stream.on('error', function (error) {
-            debug('Error reading docker events', error);
-            callback();
-        });
-
-        stream.on('end', callback);
-
-        // safety hatch if 'until' doesn't work (there are cases where docker is working with a different time)
-        setTimeout(stream.destroy.bind(stream), 3000); // https://github.com/apocas/dockerode/issues/179
-    });
-}
-
-function processApp(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
+function processApps(callback) {
    apps.getAll(function (error, result) {
        if (error) return callback(error);

@@ -182,16 +128,79 @@ function processApp(callback) {
    });
 }

-function run(intervalSecs, callback) {
-    assert.strictEqual(typeof intervalSecs, 'number');
-    assert.strictEqual(typeof callback, 'function');
+function run() {
+    processApps(function (error) {
+        if (error) console.error(error);

-    async.series([
-        processApp, // this is first because docker.getEvents seems to get 'stuck' sometimes
-        processDockerEvents.bind(null, intervalSecs)
-    ], function (error) {
-        if (error) debug(error);
-
-        callback();
+        gRunTimeout = setTimeout(run, HEALTHCHECK_INTERVAL);
    });
 }
+
+/*
+    OOM can be tested using stress tool like so:
+        docker run -ti -m 100M cloudron/base:0.10.0 /bin/bash
+        apt-get update && apt-get install stress
+        stress --vm 1 --vm-bytes 200M --vm-hang 0
+*/
+function processDockerEvents() {
+    // note that for some reason, the callback is called only on the first event
+    debug('Listening for docker events');
+    const OOM_MAIL_LIMIT = 60 * 60 * 1000; // 60 minutes
+    var lastOomMailTime = new Date(new Date() - OOM_MAIL_LIMIT);
+
+    docker.getEvents({ filters: JSON.stringify({ event: [ 'oom' ] }) }, function (error, stream) {
+        if (error) return console.error(error);
+
+        gDockerEventStream = stream;
+
+        stream.setEncoding('utf8');
+        stream.on('data', function (data) {
+            var ev = JSON.parse(data);
+            appdb.getByContainerId(ev.id, function (error, app) { // this can error for addons
+                var program = error || !app.appStoreId ? ev.id : app.appStoreId;
+                var context = JSON.stringify(ev);
+                var now = new Date();
+                if (app) context = context + '\n\n' + JSON.stringify(app, null, 4) + '\n';
+
+                debug('OOM Context: %s', context);
+
+                // do not send mails for dev apps
+                if ((!app || !app.debugMode) && (now - lastOomMailTime > OOM_MAIL_LIMIT)) {
+                    mailer.oomEvent(program, context); // app can be null if it's an addon crash
+                    lastOomMailTime = now;
+                }
+            });
+        });
+
+        stream.on('error', function (error) {
+            console.error('Error reading docker events', error);
+            gDockerEventStream = null; // TODO: reconnect?
+        });
+
+        stream.on('end', function () {
+            console.error('Docker event stream ended');
+            gDockerEventStream = null; // TODO: reconnect?
+        });
+    });
+}
+
+function start(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('Starting apphealthmonitor');
+
+    processDockerEvents();
+
+    run();
+
+    callback();
+}
+
+function stop(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    clearTimeout(gRunTimeout);
+    if (gDockerEventStream) gDockerEventStream.end();
+
+    callback();
+}
@@ -72,6 +72,7 @@ var appdb = require('./appdb.js'),
    eventlog = require('./eventlog.js'),
    fs = require('fs'),
    mail = require('./mail.js'),
+    mailboxdb = require('./mailboxdb.js'),
    manifestFormat = require('cloudron-manifestformat'),
    os = require('os'),
    path = require('path'),
@@ -153,8 +154,7 @@ function validatePortBindings(portBindings, manifest) {
        config.get('ldapPort'), /* ldap server (lo) */
        3306, /* mysql (lo) */
        4190, /* managesieve */
-        8000, /* ESXi monitoring */
-        8417, /* graphite (lo) */
+        8000, /* graphite (lo) */
    ];

    if (!portBindings) return null;
@@ -291,16 +291,6 @@ function validateBackupFormat(format) {
    return new AppsError(AppsError.BAD_FIELD, 'Invalid backup format');
 }

-function validateEnv(env) {
-    for (let key in env) {
-        if (key.length > 512) return new AppsError(AppsError.BAD_FIELD, 'Max env var key length is 512');
-        // http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap08.html
-        if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(key)) return new AppsError(AppsError.BAD_FIELD, `"${key}" is not a valid environment variable`);
-    }
-
-    return null;
-}
-
 function getDuplicateErrorDetails(location, portBindings, error) {
    assert.strictEqual(typeof location, 'string');
    assert.strictEqual(typeof portBindings, 'object');
@@ -335,8 +325,7 @@ function getAppConfig(app) {
        xFrameOptions: app.xFrameOptions || 'SAMEORIGIN',
        robotsTxt: app.robotsTxt,
        sso: app.sso,
-        alternateDomains: app.alternateDomains || [],
-        env: app.env
+        alternateDomains: app.alternateDomains || []
    };
 }

@@ -346,7 +335,7 @@ function removeInternalFields(app) {
        'location', 'domain', 'fqdn', 'mailboxName',
        'accessRestriction', 'manifest', 'portBindings', 'iconUrl', 'memoryLimit', 'xFrameOptions',
        'sso', 'debugMode', 'robotsTxt', 'enableBackup', 'creationTime', 'updateTime', 'ts',
-        'alternateDomains', 'ownerId', 'env', 'enableAutomaticUpdate');
+        'alternateDomains', 'ownerId');
 }

 function removeRestrictedFields(app) {
@@ -399,7 +388,13 @@ function get(appId, callback) {
            app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
            app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });

-            callback(null, app);
+            mailboxdb.getByOwnerId(app.id, function (error, mailboxes) {
+                if (error && error.reason !== DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+
+                if (!error) app.mailboxName = mailboxes[0].name;
+
+                callback(null, app);
+            });
        });
    });
 }
@@ -433,7 +428,13 @@ function getByIpAddress(ip, callback) {
                    app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
                    app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });

-                    callback(null, app);
+                    mailboxdb.getByOwnerId(app.id, function (error, mailboxes) {
+                        if (error && error.reason !== DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+
+                        if (!error) app.mailboxName = mailboxes[0].name;
+
+                        callback(null, app);
+                    });
                });
            });
        });
@@ -459,7 +460,13 @@ function getAll(callback) {
                app.fqdn = domains.fqdn(app.location, domainObjectMap[app.domain]);
                app.alternateDomains.forEach(function (ad) { ad.fqdn = domains.fqdn(ad.subdomain, domainObjectMap[ad.domain]); });

-                iteratorDone(null, app);
+                mailboxdb.getByOwnerId(app.id, function (error, mailboxes) {
+                    if (error && error.reason !== DatabaseError.NOT_FOUND) return iteratorDone(new AppsError(AppsError.INTERNAL_ERROR, error));
+
+                    if (!error) app.mailboxName = mailboxes[0].name;
+
+                    iteratorDone(null, app);
+                });
            }, function (error) {
                if (error) return callback(error);

@@ -525,13 +532,10 @@ function install(data, user, auditSource, callback) {
        debugMode = data.debugMode || null,
        robotsTxt = data.robotsTxt || null,
        enableBackup = 'enableBackup' in data ? data.enableBackup : true,
-        enableAutomaticUpdate = 'enableAutomaticUpdate' in data ? data.enableAutomaticUpdate : true,
        backupId = data.backupId || null,
        backupFormat = data.backupFormat || 'tgz',
        ownerId = data.ownerId,
-        alternateDomains = data.alternateDomains || [],
-        env = data.env || {},
-        mailboxName = data.mailboxName || '';
+        alternateDomains = data.alternateDomains || [];

    assert(data.appStoreId || data.manifest); // atleast one of them is required

@@ -569,16 +573,6 @@ function install(data, user, auditSource, callback) {
        // if sso was unspecified, enable it by default if possible
        if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];

-        error = validateEnv(env);
-        if (error) return callback(error);
-
-        if (mailboxName) {
-            error = mail.validateName(mailboxName);
-            if (error) return callback(error);
-        } else {
-            mailboxName = mailboxNameForLocation(location, manifest);
-        }
-
        var appId = uuid.v4();

        if (icon) {
@@ -600,7 +594,8 @@ function install(data, user, auditSource, callback) {
            if (error) return callback(new AppsError(AppsError.BAD_FIELD, 'Bad location: ' + error.message));

            if (cert && key) {
-                error = reverseProxy.validateCertificate(location, domainObject, { cert, key });
+                let fqdn = domains.fqdn(location, domain, domainObject);
+                error = reverseProxy.validateCertificate(fqdn, cert, key);
                if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
            }

@@ -612,13 +607,11 @@ function install(data, user, auditSource, callback) {
                xFrameOptions: xFrameOptions,
                sso: sso,
                debugMode: debugMode,
-                mailboxName: mailboxName,
+                mailboxName: mailboxNameForLocation(location, manifest),
                restoreConfig: backupId ? { backupId: backupId, backupFormat: backupFormat } : null,
                enableBackup: enableBackup,
-                enableAutomaticUpdate: enableAutomaticUpdate,
                robotsTxt: robotsTxt,
-                alternateDomains: alternateDomains,
-                env: env
+                alternateDomains: alternateDomains
            };

            appdb.add(appId, appStoreId, manifest, location, domain, ownerId, translatePortBindings(portBindings, manifest), data, function (error) {
@@ -626,11 +619,11 @@ function install(data, user, auditSource, callback) {
                if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, error.message));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                appstore.purchase(appId, { appstoreId: appStoreId, manifestId: manifest.id }, function (appstoreError) {
+                appstore.purchase(appId, appStoreId, function (appstoreError) {
                    // if purchase failed, rollback the appdb record
                    if (appstoreError) {
                        appdb.del(appId, function (error) {
-                            if (error) debug('install: Failed to rollback app installation.', error);
+                            if (error) console.error('Failed to rollback app installation.', error);

                            if (appstoreError.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, appstoreError.message));
                            if (appstoreError && appstoreError.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, appstoreError.message));
@@ -644,8 +637,9 @@ function install(data, user, auditSource, callback) {

                    // save cert to boxdata/certs
                    if (cert && key) {
-                        let error = reverseProxy.setAppCertificateSync(location, domainObject, { cert, key });
-                        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error setting cert: ' + error.message));
+                        let fqdn = domains.fqdn(location, domainObject);
+                        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, fqdn + '.user.cert'), cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
+                        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, fqdn + '.user.key'), key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
                    }

                    taskmanager.restartAppTask(appId);
@@ -674,14 +668,12 @@ function configure(appId, data, user, auditSource, callback) {
    get(appId, function (error, app) {
        if (error) return callback(error);

-        let domain, location, portBindings, values = { };
-        if ('location' in data && 'domain' in data) {
-            location = values.location = data.location.toLowerCase();
-            domain = values.domain = data.domain.toLowerCase();
-        } else {
-            location = app.location;
-            domain = app.domain;
-        }
+        let domain, location, portBindings, values = { }, mailboxName;
+        if ('location' in data) location = values.location = data.location.toLowerCase();
+        else location = app.location;
+
+        if ('domain' in data) domain = values.domain = data.domain.toLowerCase();
+        else domain = app.domain;

        if ('accessRestriction' in data) {
            values.accessRestriction = data.accessRestriction;
@@ -723,15 +715,15 @@ function configure(appId, data, user, auditSource, callback) {
        }

        if ('mailboxName' in data) {
-            if (data.mailboxName) {
+            if (data.mailboxName === '') { // special case to reset back to .app
+                mailboxName = mailboxNameForLocation(location, app.manifest);
+            } else {
                error = mail.validateName(data.mailboxName);
                if (error) return callback(error);
-                values.mailboxName = data.mailboxName;
-            } else {
-                values.mailboxName = mailboxNameForLocation(location, app.manifest);
+                mailboxName = data.mailboxName;
            }
        } else { // keep existing name or follow the new location
-            values.mailboxName = app.mailboxName.endsWith('.app') ? mailboxNameForLocation(location, app.manifest) : app.mailboxName;
+            mailboxName = app.mailboxName.endsWith('.app') ? mailboxNameForLocation(location, app.manifest) : app.mailboxName;
        }

        if ('alternateDomains' in data) {
@@ -740,12 +732,6 @@ function configure(appId, data, user, auditSource, callback) {
            values.alternateDomains.forEach(function (ad) { ad.subdomain = addSpacesSuffix(ad.subdomain, user); }); // TODO: validate these
        }

-        if ('env' in data) {
-            values.env = data.env;
-            error = validateEnv(data.env);
-            if (error) return callback(error);
-        }
-
        domains.get(domain, function (error, domainObject) {
            if (error && error.reason === DomainsError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such domain'));
            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Could not get domain info:' + error.message));
@@ -757,36 +743,49 @@ function configure(appId, data, user, auditSource, callback) {

            // save cert to boxdata/certs. TODO: move this to apptask when we have a real task queue
            if ('cert' in data && 'key' in data) {
-                if (data.cert && data.key) {
-                    error = reverseProxy.validateCertificate(location, domainObject, { cert: data.cert, key: data.key });
-                    if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
-                }
+                let fqdn = domains.fqdn(location, domainObject);

-                error = reverseProxy.setAppCertificateSync(location, domainObject, { cert: data.cert, key: data.key });
-                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error setting cert: ' + error.message));
+                if (data.cert && data.key) {
+                    error = reverseProxy.validateCertificate(fqdn, data.cert, data.key);
+                    if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
+
+                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.cert`), data.cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
+                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.key`), data.key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
+                } else { // remove existing cert/key
+                    if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.cert`))) debug('Error removing cert: ' + safe.error.message);
+                    if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.key`))) debug('Error removing key: ' + safe.error.message);
+                }
            }

            if ('enableBackup' in data) values.enableBackup = data.enableBackup;
-            if ('enableAutomaticUpdate' in data) values.enableAutomaticUpdate = data.enableAutomaticUpdate;

            values.oldConfig = getAppConfig(app);

            debug('Will configure app with id:%s values:%j', appId, values);

-            appdb.setInstallationCommand(appId, appdb.ISTATE_PENDING_CONFIGURE, values, function (error) {
-                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+            // make the mailbox name follow the apps new location, if the user did not set it explicitly
+            mailboxdb.updateName(app.mailboxName /* old */, values.oldConfig.domain, mailboxName, domain, function (error) {
+                if (mailboxName.endsWith('.app')) error = null; // ignore internal mailbox conflict errors since we want to show location conflict errors in the UI
+
+                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new AppsError(AppsError.ALREADY_EXISTS, 'This mailbox is already taken'));
                if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                taskmanager.restartAppTask(appId);
-
-                // fetch fresh app object for eventlog
-                get(appId, function (error, result) {
+                appdb.setInstallationCommand(appId, appdb.ISTATE_PENDING_CONFIGURE, values, function (error) {
+                    if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+                    if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE));
                    if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                    eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId: appId, app: result });
+                    taskmanager.restartAppTask(appId);

-                    callback(null);
+                    // fetch fresh app object for eventlog
+                    get(appId, function (error, result) {
+                        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+
+                        eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId: appId, app: result });
+
+                        callback(null);
+                    });
                });
            });
        });
@@ -870,7 +869,7 @@ function getLogs(appId, options, callback) {

    debug('Getting logs for %s', appId);

-    get(appId, function (error, app) {
+    get(appId, function (error /*, app */) {
        if (error) return callback(error);

        var lines = options.lines || 100,
@@ -884,7 +883,6 @@ function getLogs(appId, options, callback) {
        if (follow) args.push('--follow', '--retry', '--quiet'); // same as -F. to make it work if file doesn't exist, --quiet to not output file headers, which are no logs
        args.push(path.join(paths.LOG_DIR, appId, 'apptask.log'));
        args.push(path.join(paths.LOG_DIR, appId, 'app.log'));
-        if (app.manifest.addons && app.manifest.addons.redis) args.push(path.join(paths.LOG_DIR, `redis-${appId}/app.log`));

        var cp = spawn('/usr/bin/tail', args);

@@ -973,8 +971,7 @@ function clone(appId, data, user, auditSource, callback) {
        domain = data.domain.toLowerCase(),
        portBindings = data.portBindings || null,
        backupId = data.backupId,
-        ownerId = data.ownerId,
-        mailboxName = data.mailboxName || '';
+        ownerId = data.ownerId;

    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof location, 'string');
@@ -992,22 +989,13 @@ function clone(appId, data, user, auditSource, callback) {

            if (!backupInfo.manifest) callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Could not get restore config'));

-            const manifest = backupInfo.manifest;
-
            // re-validate because this new box version may not accept old configs
-            error = checkManifestConstraints(manifest);
+            error = checkManifestConstraints(backupInfo.manifest);
            if (error) return callback(error);

-            error = validatePortBindings(portBindings, manifest);
+            error = validatePortBindings(portBindings, backupInfo.manifest);
            if (error) return callback(error);

-            if (mailboxName) {
-                error = mail.validateName(mailboxName);
-                if (error) return callback(error);
-            } else {
-                mailboxName = mailboxNameForLocation(location, manifest);
-            }
-
            domains.get(domain, function (error, domainObject) {
                if (error && error.reason === DomainsError.NOT_FOUND) return callback(new AppsError(AppsError.EXTERNAL_ERROR, 'No such domain'));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Could not get domain info:' + error.message));
@@ -1016,7 +1004,7 @@ function clone(appId, data, user, auditSource, callback) {
                error = domains.validateHostname(location, domainObject);
                if (error) return callback(new AppsError(AppsError.BAD_FIELD, 'Bad location: ' + error.message));

-                var newAppId = uuid.v4();
+                var newAppId = uuid.v4(), manifest = backupInfo.manifest;

                var data = {
                    installationState: appdb.ISTATE_PENDING_CLONE,
@@ -1025,21 +1013,20 @@ function clone(appId, data, user, auditSource, callback) {
                    xFrameOptions: app.xFrameOptions,
                    restoreConfig: { backupId: backupId, backupFormat: backupInfo.format },
                    sso: !!app.sso,
-                    mailboxName: mailboxName,
+                    mailboxName: (location ? location : manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app',
                    enableBackup: app.enableBackup,
-                    robotsTxt: app.robotsTxt,
-                    env: app.env
+                    robotsTxt: app.robotsTxt
                };

                appdb.add(newAppId, app.appStoreId, manifest, location, domain, ownerId, translatePortBindings(portBindings, manifest), data, function (error) {
                    if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
                    if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                    appstore.purchase(newAppId, { appstoreId: app.appStoreId, manifestId: manifest.id }, function (appstoreError) {
+                    appstore.purchase(newAppId, app.appStoreId, function (appstoreError) {
                        // if purchase failed, rollback the appdb record
                        if (appstoreError) {
                            appdb.del(newAppId, function (error) {
-                                if (error) debug('install: Failed to rollback app installation.', error);
+                                if (error) console.error('Failed to rollback app installation.', error);

                                if (appstoreError.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, appstoreError.message));
                                if (appstoreError && appstoreError.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, appstoreError.message));
@@ -1078,7 +1065,7 @@ function uninstall(appId, auditSource, callback) {
    get(appId, function (error, app) {
        if (error) return callback(error);

-        appstore.unpurchase(appId, { appstoreId: app.appStoreId, manifestId: app.manifest.id }, function (error) {
+        appstore.unpurchase(appId, app.appStoreId, function (error) {
            if (error && error.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
            if (error && error.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, error.message));
            if (error && error.reason === AppstoreError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
@@ -1176,8 +1163,6 @@ function exec(appId, options, callback) {
        };

        container.exec(execOptions, function (error, exec) {
-            if (error && error.statusCode === 409) return callback(new AppsError(AppsError.BAD_STATE, error.message)); // container restarting/not running
-
            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
            var startOptions = {
                Detach: false,
@@ -1216,7 +1201,6 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
    assert.strictEqual(typeof callback, 'function');

    function canAutoupdateApp(app, newManifest) {
-        if (!app.enableAutomaticUpdate) return new Error('Automatic update disabled');
        if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(newManifest.version))) return new Error('Major version change'); // major changes are blocking

        const newTcpPorts = newManifest.tcpPorts || { };
@@ -19,7 +19,8 @@ exports = module.exports = {
    AppstoreError: AppstoreError
 };

-var apps = require('./apps.js'),
+var appdb = require('./appdb.js'),
+    apps = require('./apps.js'),
    assert = require('assert'),
    async = require('async'),
    config = require('./config.js'),
@@ -95,16 +96,18 @@ function isFreePlan(subscription) {
 }

 // See app.js install it will create a db record first but remove it again if appstore purchase fails
-function purchase(appId, data, callback) {
+function purchase(appId, appstoreId, callback) {
    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof data, 'object');
-    assert(data.appstoreId || data.manifestId);
+    assert.strictEqual(typeof appstoreId, 'string');
    assert.strictEqual(typeof callback, 'function');

+    if (appstoreId === '') return callback(null);
+
    getAppstoreConfig(function (error, appstoreConfig) {
        if (error) return callback(error);

        var url = config.apiServerOrigin() + '/api/v1/users/' + appstoreConfig.userId + '/cloudrons/' + appstoreConfig.cloudronId + '/apps/' + appId;
+        var data = { appstoreId: appstoreId };

        superagent.post(url).send(data).query({ accessToken: appstoreConfig.token }).timeout(30 * 1000).end(function (error, result) {
            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
@@ -118,12 +121,13 @@ function purchase(appId, data, callback) {
    });
 }

-function unpurchase(appId, data, callback) {
+function unpurchase(appId, appstoreId, callback) {
    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof data, 'object');
-    assert(data.appstoreId || data.manifestId);
+    assert.strictEqual(typeof appstoreId, 'string');
    assert.strictEqual(typeof callback, 'function');

+    if (appstoreId === '') return callback(null);
+
    getAppstoreConfig(function (error, appstoreConfig) {
        if (error) return callback(error);

@@ -135,7 +139,7 @@ function unpurchase(appId, data, callback) {
            if (result.statusCode === 404) return callback(null);   // was never purchased
            if (result.statusCode !== 201 && result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));

-            superagent.del(url).send(data).query({ accessToken: appstoreConfig.token }).timeout(30 * 1000).end(function (error, result) {
+            superagent.del(url).query({ accessToken: appstoreConfig.token }).timeout(30 * 1000).end(function (error, result) {
                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
                if (result.statusCode === 403 || result.statusCode === 401) return callback(new AppstoreError(AppstoreError.BILLING_REQUIRED));
                if (result.statusCode !== 204) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));
@@ -10,8 +10,8 @@ exports = module.exports = {
    _reserveHttpPort: reserveHttpPort,
    _configureReverseProxy: configureReverseProxy,
    _unconfigureReverseProxy: unconfigureReverseProxy,
-    _createAppDir: createAppDir,
-    _deleteAppDir: deleteAppDir,
+    _createVolume: createVolume,
+    _deleteVolume: deleteVolume,
    _verifyManifest: verifyManifest,
    _registerSubdomain: registerSubdomain,
    _unregisterSubdomain: unregisterSubdomain,
@@ -36,7 +36,6 @@ var addons = require('./addons.js'),
    ejs = require('ejs'),
    fs = require('fs'),
    manifestFormat = require('cloudron-manifestformat'),
-    mkdirp = require('mkdirp'),
    net = require('net'),
    os = require('os'),
    path = require('path'),
@@ -53,9 +52,9 @@ var addons = require('./addons.js'),
 var COLLECTD_CONFIG_EJS = fs.readFileSync(__dirname + '/collectd.config.ejs', { encoding: 'utf8' }),
    CONFIGURE_COLLECTD_CMD = path.join(__dirname, 'scripts/configurecollectd.sh'),
    LOGROTATE_CONFIG_EJS = fs.readFileSync(__dirname + '/logrotate.ejs', { encoding: 'utf8' }),
-    CONFIGURE_LOGROTATE_CMD = path.join(__dirname, 'scripts/configurelogrotate.sh');
-
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+    CONFIGURE_LOGROTATE_CMD = path.join(__dirname, 'scripts/configurelogrotate.sh'),
+    RMAPPDIR_CMD = path.join(__dirname, 'scripts/rmappdir.sh'),
+    CREATEAPPDIR_CMD = path.join(__dirname, 'scripts/createappdir.sh');

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');
@@ -162,47 +161,19 @@ function deleteContainers(app, callback) {
    });
 }

-function createAppDir(app, callback) {
+function createVolume(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    mkdirp(path.join(paths.APPS_DATA_DIR, app.id), callback);
+    shell.sudo('createVolume', [ CREATEAPPDIR_CMD, app.id ], callback);
 }

-function deleteAppDir(app, options, callback) {
+function deleteVolume(app, options, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof options, 'object');
    assert.strictEqual(typeof callback, 'function');

-    const appDataDir = path.join(paths.APPS_DATA_DIR, app.id);
-
-    // resolve any symlinked data dir
-    const stat = safe.fs.lstatSync(appDataDir);
-    if (!stat) return callback(null);
-
-    const resolvedAppDataDir = stat.isSymbolicLink() ? safe.fs.readlinkSync(appDataDir) : appDataDir;
-
-    if (safe.fs.existsSync(resolvedAppDataDir)) {
-        const entries = safe.fs.readdirSync(resolvedAppDataDir);
-        if (!entries) return callback(`Error listing ${resolvedAppDataDir}: ${safe.error.message}`);
-
-        // remove only files. directories inside app dir are currently volumes managed by the addons
-        entries.forEach(function (entry) {
-            let stat = safe.fs.statSync(path.join(resolvedAppDataDir, entry));
-            if (stat && !stat.isDirectory()) safe.fs.unlinkSync(path.join(resolvedAppDataDir, entry));
-        });
-    }
-
-    // if this fails, it's probably because the localstorage/redis addons have not cleaned up properly
-    if (options.removeDirectory) {
-        if (stat.isSymbolicLink()) {
-            if (!safe.fs.unlinkSync(appDataDir)) return callback(safe.error.code === 'ENOENT' ? null : safe.error);
-        } else {
-            if (!safe.fs.rmdirSync(appDataDir)) return callback(safe.error.code === 'ENOENT' ? null : safe.error);
-        }
-    }
-
-    callback(null);
+    shell.sudo('deleteVolume', [ RMAPPDIR_CMD, app.id, !!options.removeDirectory ], callback);
 }

 function addCollectdProfile(app, callback) {
@@ -212,7 +183,7 @@ function addCollectdProfile(app, callback) {
    var collectdConf = ejs.render(COLLECTD_CONFIG_EJS, { appId: app.id, containerId: app.containerId });
    fs.writeFile(path.join(paths.COLLECTD_APPCONFIG_DIR, app.id + '.conf'), collectdConf, function (error) {
        if (error) return callback(error);
-        shell.sudo('addCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'add', app.id ], {}, callback);
+        shell.sudo('addCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'add', app.id ], callback);
    });
 }

@@ -222,7 +193,7 @@ function removeCollectdProfile(app, callback) {

    fs.unlink(path.join(paths.COLLECTD_APPCONFIG_DIR, app.id + '.conf'), function (error) {
        if (error && error.code !== 'ENOENT') debugApp(app, 'Error removing collectd profile', error);
-        shell.sudo('removeCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'remove', app.id ], {}, callback);
+        shell.sudo('removeCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'remove', app.id ], callback);
    });
 }

@@ -241,7 +212,7 @@ function addLogrotateConfig(app, callback) {
        var tmpFilePath = path.join(os.tmpdir(), app.id + '.logrotate');
        fs.writeFile(tmpFilePath, logrotateConf, function (error) {
            if (error) return callback(error);
-            shell.sudo('addLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'add', app.id, tmpFilePath ], {}, callback);
+            shell.sudo('addLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'add', app.id, tmpFilePath ], callback);
        });
    });
 }
@@ -250,7 +221,7 @@ function removeLogrotateConfig(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    shell.sudo('removeLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'remove', app.id ], {}, callback);
+    shell.sudo('removeLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'remove', app.id ], callback);
 }

 function verifyManifest(manifest, callback) {
@@ -312,10 +283,7 @@ function registerSubdomain(app, overwrite, callback) {
                if (values.length !== 0 && !overwrite) return retryCallback(null, new Error('DNS Record already exists'));

                domains.upsertDnsRecords(app.location, app.domain, 'A', [ ip ], function (error) {
-                    if (error && (error.reason === DomainsError.STILL_BUSY || error.reason === DomainsError.EXTERNAL_ERROR)) {
-                        debug('Upsert error. Will retry.', error.message);
-                        return retryCallback(error); // try again
-                    }
+                    if (error && (error.reason === DomainsError.STILL_BUSY || error.reason === DomainsError.EXTERNAL_ERROR)) return retryCallback(error); // try again

                    retryCallback(null, error);
                });
@@ -372,10 +340,8 @@ function registerAlternateDomains(app, overwrite, callback) {
                    if (values.length !== 0 && !overwrite) return retryCallback(null, new Error('DNS Record already exists'));

                    domains.upsertDnsRecords(domain.subdomain, domain.domain, 'A', [ ip ], function (error) {
-                        if (error && (error.reason === DomainsError.STILL_BUSY || error.reason === DomainsError.EXTERNAL_ERROR)) {
-                            debug('Upsert error. Will retry.', error.message);
-                            return retryCallback(error); // try again
-                        }
+                        if (error && (error.reason === DomainsError.STILL_BUSY || error.reason === DomainsError.EXTERNAL_ERROR)) return retryCallback(error); // try again
+
                        retryCallback(null, error);
                    });
                });
@@ -392,14 +358,9 @@ function unregisterAlternateDomains(app, all, callback) {
    assert.strictEqual(typeof all, 'boolean');
    assert.strictEqual(typeof callback, 'function');

-    let obsoleteDomains = [];
-    if (all) {
-        obsoleteDomains = app.alternateDomains;
-    } else if (app.oldConfig) { // oldConfig can be null during an infra update
-        obsoleteDomains = app.oldConfig.alternateDomains.filter(function (o) {
-            return !app.alternateDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; });
-        });
-    }
+    var obsoleteDomains;
+    if (all) obsoleteDomains = app.alternateDomains;
+    else obsoleteDomains = app.oldConfig.alternateDomains.filter(function (o) { return !app.alternateDomains.some(function (n) { return n.subdomain === o.subdomain && n.domain === o.domain; }); });

    if (obsoleteDomains.length === 0) return callback();

@@ -438,10 +399,8 @@ function cleanupLogs(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    // note that redis container logs are cleaned up by the addon
    rimraf(path.join(paths.LOG_DIR, app.id), function (error) {
        if (error) debugApp(app, 'cannot cleanup logs: %s', error);
-
        callback(null);
    });
 }
@@ -500,11 +459,13 @@ function install(app, callback) {
        deleteMainContainer.bind(null, app),
        function teardownAddons(next) {
            // when restoring, app does not require these addons anymore. remove carefully to preserve the db passwords
-            var addonsToRemove = !isRestoring ? app.manifest.addons : _.omit(app.oldConfig.manifest.addons, Object.keys(app.manifest.addons));
+            var addonsToRemove = !isRestoring
+                ? app.manifest.addons
+                : _.omit(app.oldConfig.manifest.addons, Object.keys(app.manifest.addons));

            addons.teardownAddons(app, addonsToRemove, next);
        },
-        deleteAppDir.bind(null, app, { removeDirectory: false }), // do not remove any symlinked volume
+        deleteVolume.bind(null, app, { removeDirectory: false }), // do not remove any symlinked volume

        // for restore case
        function deleteImageIfChanged(done) {
@@ -528,7 +489,7 @@ function install(app, callback) {
        docker.downloadImage.bind(null, app.manifest),

        updateApp.bind(null, app, { installationProgress: '50, Creating volume' }),
-        createAppDir.bind(null, app),
+        createVolume.bind(null, app),

        function restoreFromBackup(next) {
            if (!restoreConfig) {
@@ -539,9 +500,7 @@ function install(app, callback) {
            } else {
                async.series([
                    updateApp.bind(null, app, { installationProgress: '60, Download backup and restoring addons' }),
-                    addons.setupAddons.bind(null, app, app.manifest.addons),
-                    addons.clearAddons.bind(null, app, app.manifest.addons),
-                    backups.restoreApp.bind(null, app, app.manifest.addons, restoreConfig, (progress) => updateApp(app, { installationProgress: progress.message }, NOOP_CALLBACK))
+                    backups.restoreApp.bind(null, app, app.manifest.addons, restoreConfig),
                ], next);
            }
        },
@@ -583,7 +542,7 @@ function backup(app, callback) {

    async.series([
        updateApp.bind(null, app, { installationProgress: '10, Backing up' }),
-        backups.backupApp.bind(null, app, (progress) => updateApp(app, { installationProgress: progress.message }, NOOP_CALLBACK)),
+        backups.backupApp.bind(null, app),

        // done!
        function (callback) {
@@ -637,7 +596,7 @@ function configure(app, callback) {
        docker.downloadImage.bind(null, app.manifest),

        updateApp.bind(null, app, { installationProgress: '45, Ensuring volume' }),
-        createAppDir.bind(null, app),
+        createVolume.bind(null, app),

        // re-setup addons since they rely on the app's fqdn (e.g oauth)
        updateApp.bind(null, app, { installationProgress: '50, Setting up addons' }),
@@ -696,7 +655,7 @@ function update(app, callback) {

            async.series([
                updateApp.bind(null, app, { installationProgress: '15, Backing up app' }),
-                backups.backupApp.bind(null, app, (progress) => updateApp(app, { installationProgress: progress.message }, NOOP_CALLBACK))
+                backups.backupApp.bind(null, app)
            ], function (error) {
                if (error) error.backupError = true;
                next(error);
@@ -806,7 +765,7 @@ function uninstall(app, callback) {
        addons.teardownAddons.bind(null, app, app.manifest.addons),

        updateApp.bind(null, app, { installationProgress: '40, Deleting volume' }),
-        deleteAppDir.bind(null, app, { removeDirectory: true }),
+        deleteVolume.bind(null, app, { removeDirectory: true }),

        updateApp.bind(null, app, { installationProgress: '50, Deleting image' }),
        docker.deleteImage.bind(null, app.manifest),
@@ -10,9 +10,9 @@ exports = module.exports = {

    get: get,

-    startBackupTask: startBackupTask,
    ensureBackup: ensureBackup,

+    backup: backup,
    restore: restore,

    backupApp: backupApp,
@@ -53,6 +53,7 @@ var addons = require('./addons.js'),
    once = require('once'),
    path = require('path'),
    paths = require('./paths.js'),
+    progress = require('./progress.js'),
    progressStream = require('progress-stream'),
    safe = require('safetydance'),
    shell = require('./shell.js'),
@@ -60,12 +61,12 @@ var addons = require('./addons.js'),
    superagent = require('superagent'),
    syncer = require('./syncer.js'),
    tar = require('tar-fs'),
-    tasks = require('./tasks.js'),
    util = require('util'),
    zlib = require('zlib');

-const NOOP_CALLBACK = function (error) { if (error) debug(error); };
-const BACKUP_UPLOAD_CMD = path.join(__dirname, 'scripts/backupupload.js');
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+var BACKUPTASK_CMD = path.join(__dirname, 'backuptask.js');

 function debugApp(app) {
    assert(typeof app === 'object');
@@ -180,6 +181,11 @@ function getBackupFilePath(backupConfig, backupId, format) {
    }
 }

+function log(detail) {
+    safe.fs.appendFileSync(paths.BACKUP_LOG_FILE, detail + '\n', 'utf8');
+    progress.setDetail(progress.BACKUP, detail);
+}
+
 function encryptFilePath(filePath, key) {
    assert.strictEqual(typeof filePath, 'string');
    assert.strictEqual(typeof key, 'string');
@@ -232,6 +238,10 @@ function createReadStream(sourceFile, key) {
        ps.emit('error', new BackupsError(BackupsError.EXTERNAL_ERROR, error.message));
    });

+    ps.on('progress', function(progress) {
+        debug('createReadStream: %s@%s (%s)', Math.round(progress.transferred/1024/1024) + 'M', Math.round(progress.speed/1024/1024) + 'Mbps', sourceFile);
+    });
+
    if (key !== null) {
        var encrypt = crypto.createCipher('aes-256-cbc', key);
        encrypt.on('error', function (error) {
@@ -277,7 +287,7 @@ function createTarPackStream(sourceDir, key) {
    });

    var gzip = zlib.createGzip({});
-    var ps = progressStream({ time: 10000 }); // emit 'pgoress' every 10 seconds
+    var ps = progressStream({ time: 10000 }); // display a progress every 10 seconds

    pack.on('error', function (error) {
        debug('createTarPackStream: tar stream error.', error);
@@ -289,6 +299,10 @@ function createTarPackStream(sourceDir, key) {
        ps.emit('error', new BackupsError(BackupsError.EXTERNAL_ERROR, error.message));
    });

+    ps.on('progress', function(progress) {
+        debug('createTarPackStream: %s@%s', Math.round(progress.transferred/1024/1024) + 'M', Math.round(progress.speed/1024/1024) + 'Mbps');
+    });
+
    if (key !== null) {
        var encrypt = crypto.createCipher('aes-256-cbc', key);
        encrypt.on('error', function (error) {
@@ -301,13 +315,17 @@ function createTarPackStream(sourceDir, key) {
    }
 }

-function sync(backupConfig, backupId, dataDir, progressCallback, callback) {
+function sync(backupConfig, backupId, dataDir, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof dataDir, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

+    function setBackupProgress(message) {
+        debug('%s', message);
+        safe.fs.writeFileSync(paths.BACKUP_RESULT_FILE, message);
+    }
+
    syncer.sync(dataDir, function processTask(task, iteratorCallback) {
        debug('sync: processing task: %j', task);
        // the empty task.path is special to signify the directory
@@ -315,12 +333,12 @@ function sync(backupConfig, backupId, dataDir, progressCallback, callback) {
        const backupFilePath = path.join(getBackupFilePath(backupConfig, backupId, backupConfig.format), destPath);

        if (task.operation === 'removedir') {
-            debug(`Removing directory ${backupFilePath}`);
+            setBackupProgress(`Removing directory ${backupFilePath}`);
            return api(backupConfig.provider).removeDir(backupConfig, backupFilePath)
-                .on('progress', (message) => progressCallback({ message }))
+                .on('progress', setBackupProgress)
                .on('done', iteratorCallback);
        } else if (task.operation === 'remove') {
-            debug(`Removing ${backupFilePath}`);
+            setBackupProgress(`Removing ${backupFilePath}`);
            return api(backupConfig.provider).remove(backupConfig, backupFilePath, iteratorCallback);
        }

@@ -329,19 +347,16 @@ function sync(backupConfig, backupId, dataDir, progressCallback, callback) {
            retryCallback = once(retryCallback); // protect again upload() erroring much later after read stream error

            ++retryCount;
-            progressCallback({ message: `${task.operation} ${task.path} try ${retryCount}` });
+            debug(`${task.operation} ${task.path} try ${retryCount}`);
            if (task.operation === 'add') {
-                debug(`Adding ${task.path} position ${task.position} try ${retryCount}`);
+                setBackupProgress(`Adding ${task.path} position ${task.position} try ${retryCount}`);
                var stream = createReadStream(path.join(dataDir, task.path), backupConfig.key || null);
                stream.on('error', function (error) {
-                    debug(`read stream error for ${task.path}: ${error.message}`);
+                    setBackupProgress(`read stream error for ${task.path}: ${error.message}`);
                    retryCallback();
                }); // ignore error if file disappears
-                stream.on('progress', function(progress) {
-                    progressCallback({ message: `Uploading ${task.path}: ${Math.round(progress.transferred/1024/1024)}M@${Math.round(progress.speed/1024/1024)}` });
-                });
                api(backupConfig.provider).upload(backupConfig, backupFilePath, stream, function (error) {
-                    debug(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
+                    setBackupProgress(error ? `Error uploading ${task.path} try ${retryCount}: ${error.message}` : `Uploaded ${task.path}`);
                    retryCallback(error);
                });
            }
@@ -373,15 +388,14 @@ function saveFsMetadata(appDataDir, callback) {
    callback();
 }

-// this function is called via backupupload (since it needs root to traverse app's directory)
-function upload(backupId, format, dataDir, progressCallback, callback) {
+// this function is called via backuptask (since it needs root to traverse app's directory)
+function upload(backupId, format, dataDir, callback) {
    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof format, 'string');
    assert.strictEqual(typeof dataDir, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

-    debug(`upload: id ${backupId} format ${format} dataDir ${dataDir}`);
+    debug('upload: id %s format %s dataDir %s', backupId, format, dataDir);

    settings.getBackupConfig(function (error, backupConfig) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));
@@ -391,9 +405,6 @@ function upload(backupId, format, dataDir, progressCallback, callback) {
                retryCallback = once(retryCallback); // protect again upload() erroring much later after tar stream error

                var tarStream = createTarPackStream(dataDir, backupConfig.key || null);
-                tarStream.on('progress', function(progress) {
-                    progressCallback({ message: `Uploading ${Math.round(progress.transferred/1024/1024)}M@${Math.round(progress.speed/1024/1024)}Mbps` });
-                });
                tarStream.on('error', retryCallback); // already returns BackupsError

                api(backupConfig.provider).upload(backupConfig, getBackupFilePath(backupConfig, backupId, format), tarStream, retryCallback);
@@ -401,7 +412,7 @@ function upload(backupId, format, dataDir, progressCallback, callback) {
        } else {
            async.series([
                saveFsMetadata.bind(null, dataDir),
-                sync.bind(null, backupConfig, backupId, dataDir, progressCallback)
+                sync.bind(null, backupConfig, backupId, dataDir)
            ], callback);
        }
    });
@@ -424,6 +435,10 @@ function tarExtract(inStream, destination, key, callback) {
        callback(new BackupsError(BackupsError.EXTERNAL_ERROR, error.message));
    });

+    ps.on('progress', function(progress) {
+        debug('tarExtract: %s@%s', Math.round(progress.transferred/1024/1024) + 'M', Math.round(progress.speed/1024/1024) + 'Mbps');
+    });
+
    gunzip.on('error', function (error) {
        debug('tarExtract: gunzip stream error.', error);
        callback(new BackupsError(BackupsError.EXTERNAL_ERROR, error.message));
@@ -449,15 +464,13 @@ function tarExtract(inStream, destination, key, callback) {
    } else {
        inStream.pipe(ps).pipe(gunzip).pipe(extract);
    }
-
-    return ps;
 }

 function restoreFsMetadata(appDataDir, callback) {
    assert.strictEqual(typeof appDataDir, 'string');
    assert.strictEqual(typeof callback, 'function');

-    debug(`Recreating empty directories in ${appDataDir}`);
+    log('Recreating empty directories');

    var metadataJson = safe.fs.readFileSync(path.join(appDataDir, 'fsmetadata.json'), 'utf8');
    if (metadataJson === null) return callback(new BackupsError(BackupsError.EXTERNAL_ERROR, 'Error loading fsmetadata.txt:' + safe.error.message));
@@ -479,11 +492,10 @@ function restoreFsMetadata(appDataDir, callback) {
    });
 }

-function downloadDir(backupConfig, backupFilePath, destDir, progressCallback, callback) {
+function downloadDir(backupConfig, backupFilePath, destDir, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof backupFilePath, 'string');
    assert.strictEqual(typeof destDir, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    debug(`downloadDir: ${backupFilePath} to ${destDir}`);
@@ -507,7 +519,7 @@ function downloadDir(backupConfig, backupFilePath, destDir, progressCallback, ca
                let destStream = createWriteStream(destFilePath, backupConfig.key || null);
                destStream.on('error', callback);

-                progressCallback({ message: `Downloading ${entry.fullPath} to ${destFilePath}` });
+                debug(`downloadDir: Copying ${entry.fullPath} to ${destFilePath}`);

                sourceStream.pipe(destStream, { end: true }).on('finish', callback);
            });
@@ -519,27 +531,25 @@ function downloadDir(backupConfig, backupFilePath, destDir, progressCallback, ca
    }, callback);
 }

-function download(backupConfig, backupId, format, dataDir, progressCallback, callback) {
+function download(backupConfig, backupId, format, dataDir, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof format, 'string');
    assert.strictEqual(typeof dataDir, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

-    debug(`download - Downloading ${backupId} of format ${format} to ${dataDir}`);
+    safe.fs.unlinkSync(paths.BACKUP_LOG_FILE); // start fresh log file
+
+    log(`Downloading ${backupId} of format ${format} to ${dataDir}`);

    if (format === 'tgz') {
        api(backupConfig.provider).download(backupConfig, getBackupFilePath(backupConfig, backupId, format), function (error, sourceStream) {
            if (error) return callback(error);

-            let ps = tarExtract(sourceStream, dataDir, backupConfig.key || null, callback);
-            ps.on('progress', function (progress) {
-                progressCallback({ message: `Downloading ${Math.round(progress.transferred/1024/1024)}M@${Math.round(progress.speed/1024/1024)}Mbps` });
-            });
+            tarExtract(sourceStream, dataDir, backupConfig.key || null, callback);
        });
    } else {
-        downloadDir(backupConfig, getBackupFilePath(backupConfig, backupId, format), dataDir, progressCallback, function (error) {
+        downloadDir(backupConfig, getBackupFilePath(backupConfig, backupId, format), dataDir, function (error) {
            if (error) return callback(error);

            restoreFsMetadata(dataDir, callback);
@@ -547,13 +557,12 @@ function download(backupConfig, backupId, format, dataDir, progressCallback, cal
    }
 }

-function restore(backupConfig, backupId, progressCallback, callback) {
+function restore(backupConfig, backupId, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

-    download(backupConfig, backupId, backupConfig.format, paths.BOX_DATA_DIR, progressCallback, function (error) {
+    download(backupConfig, backupId, backupConfig.format, paths.BOX_DATA_DIR, function (error) {
        if (error) return callback(error);

        debug('restore: download completed, importing database');
@@ -568,11 +577,10 @@ function restore(backupConfig, backupId, progressCallback, callback) {
    });
 }

-function restoreApp(app, addonsToRestore, restoreConfig, progressCallback, callback) {
+function restoreApp(app, addonsToRestore, restoreConfig, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof addonsToRestore, 'object');
    assert.strictEqual(typeof restoreConfig, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    var appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
@@ -583,7 +591,7 @@ function restoreApp(app, addonsToRestore, restoreConfig, progressCallback, callb
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

        async.series([
-            download.bind(null, backupConfig, restoreConfig.backupId, restoreConfig.backupFormat, appDataDir, progressCallback),
+            download.bind(null, backupConfig, restoreConfig.backupId, restoreConfig.backupFormat, appDataDir),
            addons.restoreAddons.bind(null, app, addonsToRestore)
        ], function (error) {
            debug('restoreApp: time: %s', (new Date() - startTime)/1000);
@@ -593,27 +601,44 @@ function restoreApp(app, addonsToRestore, restoreConfig, progressCallback, callb
    });
 }

-function runBackupUpload(backupId, format, dataDir, progressCallback, callback) {
+function runBackupTask(backupId, format, dataDir, callback) {
    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof format, 'string');
    assert.strictEqual(typeof dataDir, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

-    let result = '';
+    var killTimerId = null, progressTimerId = null;
+
+    var logStream = fs.createWriteStream(paths.BACKUP_LOG_FILE, { flags: 'a' });
+    var cp = shell.sudo(`backup-${backupId}`, [ BACKUPTASK_CMD, backupId, format, dataDir ], { env: process.env, logStream: logStream }, function (error) {
+        clearTimeout(killTimerId);
+        clearInterval(progressTimerId);
+
+        cp = null;

-    shell.sudo(`backup-${backupId}`, [ BACKUP_UPLOAD_CMD, backupId, format, dataDir ], { preserveEnv: true, ipc: true }, function (error) {
        if (error && (error.code === null /* signal */ || (error.code !== 0 && error.code !== 50))) { // backuptask crashed
            return callback(new BackupsError(BackupsError.INTERNAL_ERROR, 'Backuptask crashed'));
        } else if (error && error.code === 50) { // exited with error
+            var result = safe.fs.readFileSync(paths.BACKUP_RESULT_FILE, 'utf8') || safe.error.message;
            return callback(new BackupsError(BackupsError.EXTERNAL_ERROR, result));
        }

        callback();
-    }).on('message', function (message) {
-        if (!message.result) return progressCallback(message);
-        debug(`runBackupUpload: result - ${message}`);
-        result = message.result;
+    });
+
+    progressTimerId = setInterval(function () {
+        var result = safe.fs.readFileSync(paths.BACKUP_RESULT_FILE, 'utf8');
+        if (result) progress.setDetail(progress.BACKUP, result);
+    }, 1000); // every second
+
+    killTimerId = setTimeout(function () {
+        debug('runBackupTask: backup task taking too long. killing');
+        cp.kill();
+    }, 4 * 60 * 60 * 1000); // 4 hours
+
+    logStream.on('error', function (error) {
+        debug('runBackupTask: error in logging stream', error);
+        cp.kill();
    });
 }

@@ -639,11 +664,10 @@ function setSnapshotInfo(id, info, callback) {
    callback();
 }

-function snapshotBox(progressCallback, callback) {
-    assert.strictEqual(typeof progressCallback, 'function');
+function snapshotBox(callback) {
    assert.strictEqual(typeof callback, 'function');

-    progressCallback({ message: 'Snapshotting box' });
+    log('Snapshotting box');

    database.exportToFile(`${paths.BOX_DATA_DIR}/box.mysqldump`, function (error) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));
@@ -652,17 +676,16 @@ function snapshotBox(progressCallback, callback) {
    });
 }

-function uploadBoxSnapshot(backupConfig, progressCallback, callback) {
+function uploadBoxSnapshot(backupConfig, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    var startTime = new Date();

-    snapshotBox(progressCallback, function (error) {
+    snapshotBox(function (error) {
        if (error) return callback(error);

-        runBackupUpload('snapshot/box', backupConfig.format, paths.BOX_DATA_DIR, progressCallback, function (error) {
+        runBackupTask('snapshot/box', backupConfig.format, paths.BOX_DATA_DIR, function (error) {
            if (error) return callback(error);

            debug('uploadBoxSnapshot: time: %s secs', (new Date() - startTime)/1000);
@@ -700,11 +723,10 @@ function backupDone(apiConfig, backupId, appBackupIds, callback) {
    });
 }

-function rotateBoxBackup(backupConfig, timestamp, appBackupIds, progressCallback, callback) {
+function rotateBoxBackup(backupConfig, timestamp, appBackupIds, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof timestamp, 'string');
    assert(Array.isArray(appBackupIds));
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    var snapshotInfo = getSnapshotInfo('box');
@@ -714,13 +736,13 @@ function rotateBoxBackup(backupConfig, timestamp, appBackupIds, progressCallback
    var backupId = util.format('%s/box_%s_v%s', timestamp, snapshotTime, config.version());
    const format = backupConfig.format;

-    debug(`Rotating box backup to id ${backupId}`);
+    log(`Rotating box backup to id ${backupId}`);

    backupdb.add({ id: backupId, version: config.version(), type: backupdb.BACKUP_TYPE_BOX, dependsOn: appBackupIds, manifest: null, format: format }, function (error) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

        var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, 'snapshot/box', format), getBackupFilePath(backupConfig, backupId, format));
-        copy.on('progress', (message) => progressCallback({ message }));
+        copy.on('progress', log);
        copy.on('done', function (copyBackupError) {
            const state = copyBackupError ? backupdb.BACKUP_STATE_ERROR : backupdb.BACKUP_STATE_NORMAL;

@@ -728,7 +750,7 @@ function rotateBoxBackup(backupConfig, timestamp, appBackupIds, progressCallback
                if (copyBackupError) return callback(copyBackupError);
                if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

-                debug(`Rotated box backup successfully as id ${backupId}`);
+                log(`Rotated box backup successfully as id ${backupId}`);

                backupDone(backupConfig, backupId, appBackupIds, function (error) {
                    if (error) return callback(error);
@@ -740,19 +762,18 @@ function rotateBoxBackup(backupConfig, timestamp, appBackupIds, progressCallback
    });
 }

-function backupBoxWithAppBackupIds(appBackupIds, timestamp, progressCallback, callback) {
+function backupBoxWithAppBackupIds(appBackupIds, timestamp, callback) {
    assert(Array.isArray(appBackupIds));
    assert.strictEqual(typeof timestamp, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    settings.getBackupConfig(function (error, backupConfig) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

-        uploadBoxSnapshot(backupConfig, progressCallback, function (error) {
+        uploadBoxSnapshot(backupConfig, function (error) {
            if (error) return callback(error);

-            rotateBoxBackup(backupConfig, timestamp, appBackupIds, progressCallback, callback);
+            rotateBoxBackup(backupConfig, timestamp, appBackupIds, callback);
        });
    });
 }
@@ -766,12 +787,11 @@ function canBackupApp(app) {
            app.installationState === appdb.ISTATE_PENDING_UPDATE; // called from apptask
 }

-function snapshotApp(app, progressCallback, callback) {
+function snapshotApp(app, callback) {
    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

-    progressCallback({ message: `Snapshotting app ${app.id}` });
+    log(`Snapshotting app ${app.id}`);

    if (!safe.fs.writeFileSync(path.join(paths.APPS_DATA_DIR, app.id + '/config.json'), JSON.stringify(apps.getAppConfig(app)))) {
        return callback(new BackupsError(BackupsError.EXTERNAL_ERROR, 'Error creating config.json: ' + safe.error.message));
@@ -784,11 +804,10 @@ function snapshotApp(app, progressCallback, callback) {
    });
 }

-function rotateAppBackup(backupConfig, app, timestamp, progressCallback, callback) {
+function rotateAppBackup(backupConfig, app, timestamp, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof timestamp, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    var snapshotInfo = getSnapshotInfo(app.id);
@@ -799,13 +818,13 @@ function rotateAppBackup(backupConfig, app, timestamp, progressCallback, callbac
    var backupId = util.format('%s/app_%s_%s_v%s', timestamp, app.id, snapshotTime, manifest.version);
    const format = backupConfig.format;

-    debug(`Rotating app backup of ${app.id} to id ${backupId}`);
+    log(`Rotating app backup of ${app.id} to id ${backupId}`);

    backupdb.add({ id: backupId, version: manifest.version, type: backupdb.BACKUP_TYPE_APP, dependsOn: [ ], manifest: manifest, format: format }, function (error) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

        var copy = api(backupConfig.provider).copy(backupConfig, getBackupFilePath(backupConfig, `snapshot/app_${app.id}`, format), getBackupFilePath(backupConfig, backupId, format));
-        copy.on('progress', (message) => progressCallback({ message }));
+        copy.on('progress', log);
        copy.on('done', function (copyBackupError) {
            const state = copyBackupError ? backupdb.BACKUP_STATE_ERROR : backupdb.BACKUP_STATE_NORMAL;

@@ -813,7 +832,7 @@ function rotateAppBackup(backupConfig, app, timestamp, progressCallback, callbac
                if (copyBackupError) return callback(copyBackupError);
                if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

-                debug(`Rotated app backup of ${app.id} successfully to id ${backupId}`);
+                log(`Rotated app backup of ${app.id} successfully to id ${backupId}`);

                callback(null, backupId);
            });
@@ -821,22 +840,21 @@ function rotateAppBackup(backupConfig, app, timestamp, progressCallback, callbac
    });
 }

-function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
+function uploadAppSnapshot(backupConfig, app, callback) {
    assert.strictEqual(typeof backupConfig, 'object');
    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    if (!canBackupApp(app)) return callback(); // nothing to do

    var startTime = new Date();

-    snapshotApp(app, progressCallback, function (error) {
+    snapshotApp(app, function (error) {
        if (error) return callback(error);

        var backupId = util.format('snapshot/app_%s', app.id);
        var appDataDir = safe.fs.realpathSync(path.join(paths.APPS_DATA_DIR, app.id));
-        runBackupUpload(backupId, backupConfig.format, appDataDir, progressCallback, function (error) {
+        runBackupTask(backupId, backupConfig.format, appDataDir, function (error) {
            if (error) return callback(error);

            debugApp(app, 'uploadAppSnapshot: %s done time: %s secs', backupId, (new Date() - startTime)/1000);
@@ -846,10 +864,9 @@ function uploadAppSnapshot(backupConfig, app, progressCallback, callback) {
    });
 }

-function backupAppWithTimestamp(app, timestamp, progressCallback, callback) {
+function backupAppWithTimestamp(app, timestamp, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof timestamp, 'string');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    if (!canBackupApp(app)) return callback(); // nothing to do
@@ -857,88 +874,110 @@ function backupAppWithTimestamp(app, timestamp, progressCallback, callback) {
    settings.getBackupConfig(function (error, backupConfig) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

-        uploadAppSnapshot(backupConfig, app, progressCallback, function (error) {
+        uploadAppSnapshot(backupConfig, app, function (error) {
            if (error) return callback(error);

-            rotateAppBackup(backupConfig, app, timestamp, progressCallback, callback);
+            rotateAppBackup(backupConfig, app, timestamp, callback);
        });
    });
 }

-function backupApp(app, progressCallback, callback) {
+function backupApp(app, callback) {
    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

    const timestamp = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
+    safe.fs.unlinkSync(paths.BACKUP_LOG_FILE); // start fresh log file

-    debug(`backupApp - Backing up ${app.fqdn} with timestamp ${timestamp}`);
+    progress.set(progress.BACKUP, 10,  'Backing up ' + app.fqdn);

-    backupAppWithTimestamp(app, timestamp, progressCallback, callback);
+    backupAppWithTimestamp(app, timestamp, function (error) {
+        progress.set(progress.BACKUP, 100, error ? error.message : '');
+
+        callback(error);
+    });
 }

-// this function expects you to have a lock. Unlike other progressCallback this also has a progress field
-function backupBoxAndApps(progressCallback, callback) {
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
+// this function expects you to have a lock
+function backupBoxAndApps(auditSource, callback) {
+    assert.strictEqual(typeof auditSource, 'object');
+
+    callback = callback || NOOP_CALLBACK;

    var timestamp = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
+    safe.fs.unlinkSync(paths.BACKUP_LOG_FILE); // start fresh log file
+
+    eventlog.add(eventlog.ACTION_BACKUP_START, auditSource, { });

    apps.getAll(function (error, allApps) {
        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));

-        let percent = 1;
-        let step = 100/(allApps.length+2);
+        var processed = 1;
+        var step = 100/(allApps.length+2);

        async.mapSeries(allApps, function iterator(app, iteratorCallback) {
-            progressCallback({ percent: percent, message: `Backing up ${app.fqdn}` });
-            percent += step;
+            progress.set(progress.BACKUP, step * processed,  'Backing up ' + app.fqdn);
+
+            ++processed;

            if (!app.enableBackup) {
-                debug(`Skipped backup ${app.fqdn}`);
+                progress.set(progress.BACKUP, step * processed, 'Skipped backup ' + app.fqdn);
                return iteratorCallback(null, null); // nothing to backup
            }

-            backupAppWithTimestamp(app, timestamp, (progress) => progressCallback({ percent: percent, message: progress.message }), function (error, backupId) {
+            backupAppWithTimestamp(app, timestamp, function (error, backupId) {
                if (error && error.reason !== BackupsError.BAD_STATE) {
                    debugApp(app, 'Unable to backup', error);
                    return iteratorCallback(error);
                }

-                debugApp(app, 'Backed up');
+                progress.set(progress.BACKUP, step * processed, 'Backed up ' + app.fqdn);

                iteratorCallback(null, backupId || null); // clear backupId if is in BAD_STATE and never backed up
            });
        }, function appsBackedUp(error, backupIds) {
-            if (error) return callback(error);
+            if (error) {
+                progress.set(progress.BACKUP, 100, error.message);
+                return callback(error);
+            }

            backupIds = backupIds.filter(function (id) { return id !== null; }); // remove apps in bad state that were never backed up

-            progressCallback({ percent: percent, message: 'Backing up system data' });
-            percent += step;
+            progress.set(progress.BACKUP, step * processed, 'Backing up system data');

-            backupBoxWithAppBackupIds(backupIds, timestamp, (progress) => progressCallback({ percent: percent, message: progress.message }), callback);
+            backupBoxWithAppBackupIds(backupIds, timestamp, function (error, backupId) {
+                progress.set(progress.BACKUP, 100, error ? error.message : '');
+
+                eventlog.add(eventlog.ACTION_BACKUP_FINISH, auditSource, { errorMessage: error ? error.message : null, backupId: backupId, timestamp: timestamp });
+
+                callback(error, backupId);
+            });
        });
    });
 }

-function startBackupTask(auditSource, callback) {
-    let error = locker.lock(locker.OP_FULL_BACKUP);
-    if (error) return callback(error);
+function backup(auditSource, callback) {
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    var error = locker.lock(locker.OP_FULL_BACKUP);
+    if (error) return callback(new BackupsError(BackupsError.BAD_STATE, error.message));
+
+    var startTime = new Date();
+    progress.set(progress.BACKUP, 0, 'Starting'); // ensure tools can 'wait' on progress
+
+    backupBoxAndApps(auditSource, function (error) { // start the backup operation in the background
+        if (error) {
+            debug('backup failed.', error);
+            mailer.backupFailed(error);
+        }

-    let task = tasks.startTask(tasks.TASK_BACKUP, [], auditSource);
-    task.on('error', (error) => callback(new BackupsError(BackupsError.INTERNAL_ERROR, error)));
-    task.on('start', (taskId) => {
-        eventlog.add(eventlog.ACTION_BACKUP_START, auditSource, { taskId });
-        callback(null, taskId);
-    });
-    task.on('finish', (error, result) => {
        locker.unlock(locker.OP_FULL_BACKUP);

-        if (error) mailer.backupFailed(error);
-
-        eventlog.add(eventlog.ACTION_BACKUP_FINISH, auditSource, { errorMessage: error ? error.message : null, backupId: result });
+        debug('backup took %s seconds', (new Date() - startTime)/1000);
    });
+
+    callback(null);
 }

 function ensureBackup(auditSource, callback) {
@@ -960,7 +999,7 @@ function ensureBackup(auditSource, callback) {
                return callback(null);
            }

-            startBackupTask(auditSource, callback);
+            backup(auditSource, callback);
        });
    });
 }
@@ -9,10 +9,17 @@ if (process.argv[2] === '--check') return console.log('OK');

 require('supererror')({ splatchError: true });

+// remove timestamp from debug() based output
+require('debug').formatArgs = function formatArgs(args) {
+    args[0] = this.namespace + ' ' + args[0];
+};
+
 var assert = require('assert'),
-    backups = require('../backups.js'),
-    database = require('../database.js'),
-    debug = require('debug')('box:backupupload');
+    backups = require('./backups.js'),
+    database = require('./database.js'),
+    debug = require('debug')('box:backuptask'),
+    paths = require('./paths.js'),
+    safe = require('safetydance');

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');
@@ -31,21 +38,17 @@ process.on('SIGTERM', function () {
    process.exit(0);
 });

-// this can happen when the backup task is terminated (not box code)
-process.on('disconnect', function () {
-    debug('parent process died');
-    process.exit(0);
-});
-
 initialize(function (error) {
    if (error) throw error;

-    backups.upload(backupId, format, dataDir, (progress) => process.send(progress), function resultHandler(error) {
+    safe.fs.writeFileSync(paths.BACKUP_RESULT_FILE, '');
+
+    backups.upload(backupId, format, dataDir, function resultHandler(error) {
        if (error) debug('upload completed with error', error);

        debug('upload completed');

-        process.send({ result: error ? error.message : '' });
+        safe.fs.writeFileSync(paths.BACKUP_RESULT_FILE, error ? error.message : '');

        // https://nodejs.org/api/process.html are exit codes used by node. apps.js uses the value below
        // to check apptask crashes
@@ -4,18 +4,29 @@ exports = module.exports = {
    verifySetupToken: verifySetupToken,
    setupDone: setupDone,

+    changePlan: changePlan,
+    upgrade: upgrade,
    sendHeartbeat: sendHeartbeat,
+    getBoxAndUserDetails: getBoxAndUserDetails,
    setPtrRecord: setPtrRecord,

    CaasError: CaasError
 };

 var assert = require('assert'),
+    backups = require('./backups.js'),
    config = require('./config.js'),
    debug = require('debug')('box:caas'),
+    locker = require('./locker.js'),
+    path = require('path'),
+    progress = require('./progress.js'),
    settings = require('./settings.js'),
+    shell = require('./shell.js'),
    superagent = require('superagent'),
-    util = require('util');
+    util = require('util'),
+    _ = require('underscore');
+
+const RETIRE_CMD = path.join(__dirname, 'scripts/retire.sh');

 function CaasError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
@@ -42,6 +53,20 @@ CaasError.INVALID_TOKEN = 'Invalid Token';
 CaasError.INTERNAL_ERROR = 'Internal Error';
 CaasError.EXTERNAL_ERROR = 'External Error';

+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+function retire(reason, info, callback) {
+    assert(reason === 'migrate' || reason === 'upgrade');
+    info = info || { };
+    callback = callback || NOOP_CALLBACK;
+
+    var data = {
+        apiServerOrigin: config.apiServerOrigin(),
+        adminFqdn: config.adminFqdn()
+    };
+    shell.sudo('retire', [ RETIRE_CMD, reason, JSON.stringify(info), JSON.stringify(data) ], callback);
+}
+
 function getCaasConfig(callback) {
    assert.strictEqual(typeof callback, 'function');

@@ -92,6 +117,96 @@ function setupDone(setupToken, callback) {
            });
    });
 }
+function doMigrate(options, caasConfig, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof caasConfig, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    var error = locker.lock(locker.OP_MIGRATE);
+    if (error) return callback(new CaasError(CaasError.BAD_STATE, error.message));
+
+    function unlock(error) {
+        debug('Failed to migrate', error);
+        locker.unlock(locker.OP_MIGRATE);
+        progress.set(progress.MIGRATE, -1, 'Backup failed: ' + error.message);
+    }
+
+    progress.set(progress.MIGRATE, 10, 'Backing up for migration');
+
+    // initiate the migration in the background
+    backups.backupBoxAndApps({ userId: null, username: 'migrator' }, function (error) {
+        if (error) return unlock(error);
+
+        debug('migrate: domain: %s size %s region %s', options.domain, options.size, options.region);
+
+        superagent
+            .post(config.apiServerOrigin() + '/api/v1/boxes/' + caasConfig.boxId + '/migrate')
+            .query({ token: caasConfig.token })
+            .send(options)
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return unlock(error); // network error
+                if (result.statusCode === 409) return unlock(new CaasError(CaasError.BAD_STATE));
+                if (result.statusCode === 404) return unlock(new CaasError(CaasError.NOT_FOUND));
+                if (result.statusCode !== 202) return unlock(new CaasError(CaasError.EXTERNAL_ERROR, util.format('%s %j', result.status, result.body)));
+
+                progress.set(progress.MIGRATE, 10, 'Migrating');
+
+                retire('migrate', _.pick(options, 'domain', 'size', 'region'));
+            });
+    });
+
+    callback(null);
+}
+
+function changePlan(options, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (config.isDemo()) return callback(new CaasError(CaasError.BAD_FIELD, 'Not allowed in demo mode'));
+
+    getCaasConfig(function (error, result) {
+        if (error) return callback(error);
+
+        doMigrate(options, result, callback);
+    });
+}
+
+// this function expects a lock
+function upgrade(boxUpdateInfo, callback) {
+    assert(boxUpdateInfo !== null && typeof boxUpdateInfo === 'object');
+
+    function upgradeError(e) {
+        progress.set(progress.UPDATE, -1, e.message);
+        callback(e);
+    }
+
+    progress.set(progress.UPDATE, 5, 'Backing up for upgrade');
+
+    backups.backupBoxAndApps({ userId: null, username: 'upgrader' }, function (error) {
+        if (error) return upgradeError(error);
+
+        getCaasConfig(function (error, result) {
+            if (error) return upgradeError(error);
+
+            superagent.post(config.apiServerOrigin() + '/api/v1/boxes/' + result.boxId + '/upgrade')
+                .query({ token: result.token })
+                .send({ version: boxUpdateInfo.version })
+                .timeout(30 * 1000)
+                .end(function (error, result) {
+                    if (error && !error.response) return upgradeError(new Error('Network error making upgrade request: ' + error));
+                    if (result.statusCode !== 202) return upgradeError(new Error(util.format('Server not ready to upgrade. statusCode: %s body: %j', result.status, result.body)));
+
+                    progress.set(progress.UPDATE, 10, 'Updating base system');
+
+                    // no need to unlock since this is the last thing we ever do on this box
+                    callback();
+
+                    retire('upgrade');
+                });
+        });
+    });
+}

 function sendHeartbeat() {
    assert(config.provider() === 'caas', 'Heartbeat is only sent for managed cloudrons');
@@ -108,6 +223,27 @@ function sendHeartbeat() {
    });
 }

+function getBoxAndUserDetails(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    if (config.provider() !== 'caas') return callback(null, {});
+
+    getCaasConfig(function (error, caasConfig) {
+        if (error) return callback(error);
+
+        superagent
+            .get(config.apiServerOrigin() + '/api/v1/boxes/' + caasConfig.boxId)
+            .query({ token: caasConfig.token })
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return callback(new CaasError(CaasError.EXTERNAL_ERROR, 'Cannot reach appstore'));
+                if (result.statusCode !== 200) return callback(new CaasError(CaasError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+
+                return callback(null, result.body);
+            });
+    });
+}
+
 function setPtrRecord(domain, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');
@@ -0,0 +1,481 @@
+'use strict';
+
+var assert = require('assert'),
+    async = require('async'),
+    crypto = require('crypto'),
+    debug = require('debug')('box:cert/acme1'),
+    execSync = require('safetydance').child_process.execSync,
+    fs = require('fs'),
+    parseLinks = require('parse-links'),
+    path = require('path'),
+    paths = require('../paths.js'),
+    safe = require('safetydance'),
+    superagent = require('superagent'),
+    util = require('util'),
+    _ = require('underscore');
+
+var CA_PROD = 'https://acme-v01.api.letsencrypt.org',
+    CA_STAGING = 'https://acme-staging.api.letsencrypt.org',
+    LE_AGREEMENT = 'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf';
+
+exports = module.exports = {
+    getCertificate: getCertificate,
+
+    // testing
+    _name: 'acme'
+};
+
+function Acme1Error(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(Acme1Error, Error);
+Acme1Error.INTERNAL_ERROR = 'Internal Error';
+Acme1Error.EXTERNAL_ERROR = 'External Error';
+Acme1Error.ALREADY_EXISTS = 'Already Exists';
+Acme1Error.NOT_COMPLETED = 'Not Completed';
+Acme1Error.FORBIDDEN = 'Forbidden';
+
+// http://jose.readthedocs.org/en/latest/
+// https://www.ietf.org/proceedings/92/slides/slides-92-acme-1.pdf
+// https://community.letsencrypt.org/t/list-of-client-implementations/2103
+
+function Acme1(options) {
+    assert.strictEqual(typeof options, 'object');
+
+    this.caOrigin = options.prod ? CA_PROD : CA_STAGING;
+    this.accountKeyPem = null; // Buffer
+    this.email = options.email;
+}
+
+Acme1.prototype.getNonce = function (callback) {
+    superagent.get(this.caOrigin + '/directory').timeout(30 * 1000).end(function (error, response) {
+        if (error && !error.response) return callback(error);
+        if (response.statusCode !== 200) return callback(new Error('Invalid response code when fetching nonce : ' + response.statusCode));
+
+        return callback(null, response.headers['Replay-Nonce'.toLowerCase()]);
+    });
+};
+
+// urlsafe base64 encoding (jose)
+function urlBase64Encode(string) {
+    return string.replace(/\+/g, '-').replace(/\//g, '_').replace(/=/g, '');
+}
+
+function b64(str) {
+    var buf = util.isBuffer(str) ? str : new Buffer(str);
+   return urlBase64Encode(buf.toString('base64'));
+}
+
+function getModulus(pem) {
+    assert(util.isBuffer(pem));
+
+    var stdout = execSync('openssl rsa -modulus -noout', { input: pem, encoding: 'utf8' });
+    if (!stdout) return null;
+    var match = stdout.match(/Modulus=([0-9a-fA-F]+)$/m);
+    if (!match) return null;
+    return Buffer.from(match[1], 'hex');
+}
+
+Acme1.prototype.sendSignedRequest = function (url, payload, callback) {
+    assert.strictEqual(typeof url, 'string');
+    assert.strictEqual(typeof payload, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    assert(util.isBuffer(this.accountKeyPem));
+
+    var that = this;
+    var header = {
+        alg: 'RS256',
+        jwk: {
+            e: b64(Buffer.from([0x01, 0x00, 0x01])), // exponent - 65537
+            kty: 'RSA',
+            n: b64(getModulus(this.accountKeyPem))
+        }
+    };
+
+    var payload64 = b64(payload);
+
+    this.getNonce(function (error, nonce) {
+        if (error) return callback(error);
+
+        debug('sendSignedRequest: using nonce %s for url %s', nonce, url);
+
+        var protected64 = b64(JSON.stringify(_.extend({ }, header, { nonce: nonce })));
+
+        var signer = crypto.createSign('RSA-SHA256');
+        signer.update(protected64 + '.' + payload64, 'utf8');
+        var signature64 = urlBase64Encode(signer.sign(that.accountKeyPem, 'base64'));
+
+        var data = {
+            header: header,
+            protected: protected64,
+            payload: payload64,
+            signature: signature64
+        };
+
+        superagent.post(url).set('Content-Type', 'application/x-www-form-urlencoded').send(JSON.stringify(data)).timeout(30 * 1000).end(function (error, res) {
+            if (error && !error.response) return callback(error); // network errors
+
+            callback(null, res);
+        });
+    });
+};
+
+Acme1.prototype.updateContact = function (registrationUri, callback) {
+    assert.strictEqual(typeof registrationUri, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('updateContact: %s %s', registrationUri, this.email);
+
+    // https://github.com/ietf-wg-acme/acme/issues/30
+    var payload = {
+        resource: 'reg',
+        contact: [ 'mailto:' + this.email ],
+        agreement: LE_AGREEMENT
+    };
+
+    var that = this;
+    this.sendSignedRequest(registrationUri, JSON.stringify(payload), function (error, result) {
+        if (error) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when registering user: ' + error.message));
+        if (result.statusCode !== 202) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 202, got %s %s', result.statusCode, result.text)));
+
+        debug('updateContact: contact of user updated to %s', that.email);
+
+        callback();
+    });
+};
+
+Acme1.prototype.registerUser = function (callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    var payload = {
+        resource: 'new-reg',
+        contact: [ 'mailto:' + this.email ],
+        agreement: LE_AGREEMENT
+    };
+
+    debug('registerUser: %s', this.email);
+
+    var that = this;
+    this.sendSignedRequest(this.caOrigin + '/acme/new-reg', JSON.stringify(payload), function (error, result) {
+        if (error) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when registering user: ' + error.message));
+        if (result.statusCode === 409) return that.updateContact(result.headers.location, callback); // already exists
+        if (result.statusCode !== 201) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+
+        debug('registerUser: registered user %s', that.email);
+
+        callback(null);
+    });
+};
+
+Acme1.prototype.registerDomain = function (domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var payload = {
+        resource: 'new-authz',
+        identifier: {
+            type: 'dns',
+            value: domain
+        }
+    };
+
+    debug('registerDomain: %s', domain);
+
+    this.sendSignedRequest(this.caOrigin + '/acme/new-authz', JSON.stringify(payload), function (error, result) {
+        if (error) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when registering domain: ' + error.message));
+        if (result.statusCode === 403) return callback(new Acme1Error(Acme1Error.FORBIDDEN, result.body.detail));
+        if (result.statusCode !== 201) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+
+        debug('registerDomain: registered %s', domain);
+
+        callback(null, result.body);
+    });
+};
+
+Acme1.prototype.prepareHttpChallenge = function (challenge, callback) {
+    assert.strictEqual(typeof challenge, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('prepareHttpChallenge: preparing for challenge %j', challenge);
+
+    var token = challenge.token;
+
+    assert(util.isBuffer(this.accountKeyPem));
+
+    var jwk = {
+        e: b64(Buffer.from([0x01, 0x00, 0x01])), // Exponent - 65537
+        kty: 'RSA',
+        n: b64(getModulus(this.accountKeyPem))
+    };
+
+    var shasum = crypto.createHash('sha256');
+    shasum.update(JSON.stringify(jwk));
+    var thumbprint = urlBase64Encode(shasum.digest('base64'));
+    var keyAuthorization = token + '.' + thumbprint;
+
+    debug('prepareHttpChallenge: writing %s to %s', keyAuthorization, path.join(paths.ACME_CHALLENGES_DIR, token));
+
+    fs.writeFile(path.join(paths.ACME_CHALLENGES_DIR, token), token + '.' + thumbprint, function (error) {
+        if (error) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, error));
+
+        callback();
+    });
+};
+
+Acme1.prototype.notifyChallengeReady = function (challenge, callback) {
+    assert.strictEqual(typeof challenge, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('notifyChallengeReady: %s was met', challenge.uri);
+
+    var keyAuthorization = fs.readFileSync(path.join(paths.ACME_CHALLENGES_DIR, challenge.token), 'utf8');
+
+    var payload = {
+        resource: 'challenge',
+        keyAuthorization: keyAuthorization
+    };
+
+    this.sendSignedRequest(challenge.uri, JSON.stringify(payload), function (error, result) {
+        if (error) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when notifying challenge: ' + error.message));
+        if (result.statusCode !== 202) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 202, got %s %s', result.statusCode, result.text)));
+
+        callback();
+    });
+};
+
+Acme1.prototype.waitForChallenge = function (challenge, callback) {
+    assert.strictEqual(typeof challenge, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('waitingForChallenge: %j', challenge);
+
+    async.retry({ times: 10, interval: 5000 }, function (retryCallback) {
+        debug('waitingForChallenge: getting status');
+
+        superagent.get(challenge.uri).timeout(30 * 1000).end(function (error, result) {
+            if (error && !error.response) {
+                debug('waitForChallenge: network error getting uri %s', challenge.uri);
+                return retryCallback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, error.message)); // network error
+            }
+            if (result.statusCode !== 202) {
+                debug('waitForChallenge: invalid response code getting uri %s', result.statusCode);
+                return retryCallback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Bad response code:' + result.statusCode));
+            }
+
+            debug('waitForChallenge: status is "%s %j', result.body.status, result.body);
+
+            if (result.body.status === 'pending') return retryCallback(new Acme1Error(Acme1Error.NOT_COMPLETED));
+            else if (result.body.status === 'valid') return retryCallback();
+            else return retryCallback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Unexpected status: ' + result.body.status));
+        });
+    }, function retryFinished(error) {
+        // async.retry will pass 'undefined' as second arg making it unusable with async.waterfall()
+        callback(error);
+    });
+};
+
+// https://community.letsencrypt.org/t/public-beta-rate-limits/4772 for rate limits
+Acme1.prototype.signCertificate = function (domain, csrDer, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert(util.isBuffer(csrDer));
+    assert.strictEqual(typeof callback, 'function');
+
+    var outdir = paths.APP_CERTS_DIR;
+
+    var payload = {
+        resource: 'new-cert',
+        csr: b64(csrDer)
+    };
+
+    debug('signCertificate: sending new-cert request');
+
+    this.sendSignedRequest(this.caOrigin + '/acme/new-cert', JSON.stringify(payload), function (error, result) {
+        if (error) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when signing certificate: ' + error.message));
+        // 429 means we reached the cert limit for this domain
+        if (result.statusCode !== 201) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 201, got %s %s', result.statusCode, result.text)));
+
+        var certUrl = result.headers.location;
+
+        if (!certUrl) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Missing location in downloadCertificate'));
+
+        safe.fs.writeFileSync(path.join(outdir, domain + '.url'), certUrl, 'utf8'); // maybe use for renewal
+
+        return callback(null, result.headers.location);
+    });
+};
+
+Acme1.prototype.createKeyAndCsr = function (domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var outdir = paths.APP_CERTS_DIR;
+    var csrFile = path.join(outdir, domain + '.csr');
+    var privateKeyFile = path.join(outdir, domain + '.key');
+
+    if (safe.fs.existsSync(privateKeyFile)) {
+        // in some old releases, csr file was corrupt. so always regenerate it
+        debug('createKeyAndCsr: reuse the key for renewal at %s', privateKeyFile);
+    } else {
+        var key = execSync('openssl genrsa 4096');
+        if (!key) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+        if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+
+        debug('createKeyAndCsr: key file saved at %s', privateKeyFile);
+    }
+
+    var csrDer = execSync(util.format('openssl req -new -key %s -outform DER -subj /CN=%s', privateKeyFile, domain));
+    if (!csrDer) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+    if (!safe.fs.writeFileSync(csrFile, csrDer)) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error)); // bookkeeping
+
+    debug('createKeyAndCsr: csr file (DER) saved at %s', csrFile);
+
+    callback(null, csrDer);
+};
+
+// TODO: download the chain in a loop following 'up' header
+Acme1.prototype.downloadChain = function (linkHeader, callback) {
+    if (!linkHeader) return new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Empty link header when downloading certificate chain');
+
+    debug('downloadChain: linkHeader %s', linkHeader);
+
+    var linkInfo = parseLinks(linkHeader);
+    if (!linkInfo || !linkInfo.up) return new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Failed to parse link header when downloading certificate chain');
+
+    var intermediateCertUrl = linkInfo.up.startsWith('https://') ? linkInfo.up : (this.caOrigin + linkInfo.up);
+
+    debug('downloadChain: downloading from %s', intermediateCertUrl);
+
+    superagent.get(intermediateCertUrl).buffer().parse(function (res, done) {
+        var data = [ ];
+        res.on('data', function(chunk) { data.push(chunk); });
+        res.on('end', function () { res.text = Buffer.concat(data); done(); });
+    }).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when downloading certificate'));
+        if (result.statusCode !== 200) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+
+        var chainDer = result.text;
+        var chainPem = execSync('openssl x509 -inform DER -outform PEM', { input: chainDer }); // this is really just base64 encoding with header
+        if (!chainPem) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+
+        callback(null, chainPem);
+    });
+};
+
+Acme1.prototype.downloadCertificate = function (domain, certUrl, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof certUrl, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var outdir = paths.APP_CERTS_DIR;
+    var that = this;
+
+    superagent.get(certUrl).buffer().parse(function (res, done) {
+        var data = [ ];
+        res.on('data', function(chunk) { data.push(chunk); });
+        res.on('end', function () { res.text = Buffer.concat(data); done(); });
+    }).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'Network error when downloading certificate'));
+        if (result.statusCode === 202) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, 'Retry not implemented yet'));
+        if (result.statusCode !== 200) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+
+        var certificateDer = result.text;
+
+        safe.fs.writeFileSync(path.join(outdir, domain + '.der'), certificateDer);
+        debug('downloadCertificate: cert der file for %s saved', domain);
+
+        var certificatePem = execSync('openssl x509 -inform DER -outform PEM', { input: certificateDer }); // this is really just base64 encoding with header
+        if (!certificatePem) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+
+        that.downloadChain(result.header['link'], function (error, chainPem) {
+            if (error) return callback(error);
+
+            var certificateFile = path.join(outdir, domain + '.cert');
+            var fullChainPem = Buffer.concat([certificatePem, chainPem]);
+            if (!safe.fs.writeFileSync(certificateFile, fullChainPem)) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+
+            debug('downloadCertificate: cert file for %s saved at %s', domain, certificateFile);
+
+            callback();
+        });
+    });
+};
+
+Acme1.prototype.acmeFlow = function (domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!fs.existsSync(paths.ACME_ACCOUNT_KEY_FILE)) {
+        debug('getCertificate: generating acme account key on first run');
+        this.accountKeyPem = safe.child_process.execSync('openssl genrsa 4096');
+        if (!this.accountKeyPem) return callback(new Acme1Error(Acme1Error.INTERNAL_ERROR, safe.error));
+
+        safe.fs.writeFileSync(paths.ACME_ACCOUNT_KEY_FILE, this.accountKeyPem);
+    } else {
+        debug('getCertificate: using existing acme account key');
+        this.accountKeyPem = fs.readFileSync(paths.ACME_ACCOUNT_KEY_FILE);
+    }
+
+    var that = this;
+    this.registerUser(function (error) {
+        if (error) return callback(error);
+
+        that.registerDomain(domain, function (error, result) {
+            if (error) return callback(error);
+
+            debug('acmeFlow: challenges: %j', result);
+
+            var httpChallenges = result.challenges.filter(function(x) { return x.type === 'http-01'; });
+            if (httpChallenges.length === 0) return callback(new Acme1Error(Acme1Error.EXTERNAL_ERROR, 'no http challenges'));
+            var challenge = httpChallenges[0];
+
+            async.waterfall([
+                that.prepareHttpChallenge.bind(that, challenge),
+                that.notifyChallengeReady.bind(that, challenge),
+                that.waitForChallenge.bind(that, challenge),
+                that.createKeyAndCsr.bind(that, domain),
+                that.signCertificate.bind(that, domain),
+                that.downloadCertificate.bind(that, domain)
+            ], callback);
+        });
+    });
+};
+
+Acme1.prototype.getCertificate = function (hostname, domain, callback) {
+    assert.strictEqual(typeof hostname, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('getCertificate: start acme flow for %s from %s', hostname, this.caOrigin);
+    this.acmeFlow(hostname, function (error) {
+        if (error) return callback(error);
+
+        var outdir = paths.APP_CERTS_DIR;
+        callback(null, path.join(outdir, hostname + '.cert'), path.join(outdir, hostname + '.key'));
+    });
+};
+
+function getCertificate(hostname, domain, options, callback) {
+    assert.strictEqual(typeof hostname, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    var acme = new Acme1(options || { });
+    acme.getCertificate(hostname, domain, callback);
+}
@@ -5,6 +5,7 @@ var assert = require('assert'),
    crypto = require('crypto'),
    debug = require('debug')('box:cert/acme2'),
    domains = require('../domains.js'),
+    execSync = require('safetydance').child_process.execSync,
    fs = require('fs'),
    path = require('path'),
    paths = require('../paths.js'),
@@ -20,8 +21,7 @@ exports = module.exports = {
    getCertificate: getCertificate,

    // testing
-    _name: 'acme',
-    _getChallengeSubdomain: getChallengeSubdomain
+    _name: 'acme'
 };

 function Acme2Error(reason, errorOrMessage) {
@@ -87,7 +87,7 @@ function b64(str) {
 function getModulus(pem) {
    assert(util.isBuffer(pem));

-    var stdout = safe.child_process.execSync('openssl rsa -modulus -noout', { input: pem, encoding: 'utf8' });
+    var stdout = execSync('openssl rsa -modulus -noout', { input: pem, encoding: 'utf8' });
    if (!stdout) return null;
    var match = stdout.match(/Modulus=([0-9a-fA-F]+)$/m);
    if (!match) return null;
@@ -350,14 +350,14 @@ Acme2.prototype.createKeyAndCsr = function (hostname, callback) {
        // in some old releases, csr file was corrupt. so always regenerate it
        debug('createKeyAndCsr: reuse the key for renewal at %s', privateKeyFile);
    } else {
-        var key = safe.child_process.execSync('openssl genrsa 4096');
+        var key = execSync('openssl genrsa 4096');
        if (!key) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));
        if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));

        debug('createKeyAndCsr: key file saved at %s', privateKeyFile);
    }

-    var csrDer = safe.child_process.execSync(`openssl req -new -key ${privateKeyFile} -outform DER -subj /CN=${hostname}`);
+    var csrDer = execSync(`openssl req -new -key ${privateKeyFile} -outform DER -subj /CN=${hostname}`);
    if (!csrDer) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));
    if (!safe.fs.writeFileSync(csrFile, csrDer)) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error)); // bookkeeping

@@ -435,14 +435,11 @@ function getChallengeSubdomain(hostname, domain) {
    if (hostname === domain) {
        challengeSubdomain = '_acme-challenge';
    } else if (hostname.includes('*')) { // wildcard
-        let subdomain = hostname.slice(0, -domain.length - 1);
-        challengeSubdomain = subdomain ? subdomain.replace('*', '_acme-challenge') : '_acme-challenge';
+        challengeSubdomain = hostname.replace('*', '_acme-challenge').slice(0, -domain.length - 1);
    } else {
        challengeSubdomain = '_acme-challenge.' + hostname.slice(0, -domain.length - 1);
    }

-    debug(`getChallengeSubdomain: challenge subdomain for hostname ${hostname} at domain ${domain} is ${challengeSubdomain}`);
-
    return challengeSubdomain;
 }

@@ -469,7 +466,7 @@ Acme2.prototype.prepareDnsChallenge = function (hostname, domain, authorization,
    domains.upsertDnsRecords(challengeSubdomain, domain, 'TXT', [ `"${txtValue}"` ], function (error) {
        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message));

-        domains.waitForDnsRecord(challengeSubdomain, domain, 'TXT', txtValue, { interval: 5000, times: 200 }, function (error) {
+        domains.waitForDnsRecord(`${challengeSubdomain}`, domain, 'TXT', txtValue, { interval: 5000, times: 200 }, function (error) {
            if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message));

            callback(null, challenge);
@@ -8,44 +8,32 @@ exports = module.exports = {
    getConfig: getConfig,
    getDisks: getDisks,
    getLogs: getLogs,
-    getStatus: getStatus,

    reboot: reboot,
-    isRebootRequired: isRebootRequired,

    onActivated: onActivated,

-    setDashboardDomain: setDashboardDomain,
-    renewCerts: renewCerts,
-
-    checkDiskSpace: checkDiskSpace,
-
-    configureWebadmin: configureWebadmin,
-    getWebadminStatus: getWebadminStatus
+    checkDiskSpace: checkDiskSpace
 };

 var assert = require('assert'),
    async = require('async'),
-    clients = require('./clients.js'),
    config = require('./config.js'),
    cron = require('./cron.js'),
    debug = require('debug')('box:cloudron'),
-    domains = require('./domains.js'),
-    DomainsError = require('./domains.js').DomainsError,
    df = require('@sindresorhus/df'),
-    fs = require('fs'),
    mailer = require('./mailer.js'),
    os = require('os'),
    path = require('path'),
    paths = require('./paths.js'),
    platform = require('./platform.js'),
+    progress = require('./progress.js'),
    reverseProxy = require('./reverseproxy.js'),
+    safe = require('safetydance'),
    settings = require('./settings.js'),
    shell = require('./shell.js'),
    spawn = require('child_process').spawn,
    split = require('split'),
-    sysinfo = require('./sysinfo.js'),
-    tasks = require('./tasks.js'),
    users = require('./users.js'),
    util = require('util');

@@ -53,16 +41,6 @@ var REBOOT_CMD = path.join(__dirname, 'scripts/reboot.sh');

 var NOOP_CALLBACK = function (error) { if (error) debug(error); };

-let gWebadminStatus = {
-    dns: false,
-    tls: false,
-    configuring: false,
-    restore: {
-        active: false,
-        error: null
-    }
-};
-
 function CloudronError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
@@ -87,13 +65,18 @@ CloudronError.INTERNAL_ERROR = 'Internal Error';
 CloudronError.EXTERNAL_ERROR = 'External Error';
 CloudronError.BAD_STATE = 'Bad state';
 CloudronError.ALREADY_UPTODATE = 'No Update Available';
+CloudronError.NOT_FOUND = 'Not found';
+CloudronError.SELF_UPGRADE_NOT_SUPPORTED = 'Self upgrade not supported';

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');

-    cron.startPreActivationJobs(callback);
-
-    runStartupTasks();
+    async.series([
+        settings.initialize,
+        reverseProxy.configureDefaultServer,
+        cron.startPreActivationJobs,
+        onActivated
+    ], callback);
 }

 function uninitialize(callback) {
@@ -101,36 +84,25 @@ function uninitialize(callback) {

    async.series([
        cron.stopJobs,
-        platform.stop
+        platform.stop,
+        settings.uninitialize
    ], callback);
 }

 function onActivated(callback) {
-    assert.strictEqual(typeof callback, 'function');
+    callback = callback || NOOP_CALLBACK;

    // Starting the platform after a user is available means:
    // 1. mail bounces can now be sent to the cloudron owner
    // 2. the restore code path can run without sudo (since mail/ is non-root)
-    async.series([
-        platform.start,
-        cron.startPostActivationJobs
-    ], callback);
-}
+    users.count(function (error, count) {
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
+        if (!count) return callback(); // not activated

-// each of these tasks can fail. we will add some routes to fix/re-run them
-function runStartupTasks() {
-    // configure nginx to be reachable by IP
-    reverseProxy.configureDefaultServer(NOOP_CALLBACK);
-
-    // always generate webadmin config since we have no versioning mechanism for the ejs
-    configureWebadmin(NOOP_CALLBACK);
-
-    // check activation state and start the platform
-    users.isActivated(function (error, activated) {
-        if (error) return debug(error);
-        if (!activated) return debug('initialize: not activated yet'); // not activated
-
-        onActivated(NOOP_CALLBACK);
+        async.series([
+            platform.start,
+            cron.startPostActivationJobs
+        ], callback);
    });
 }

@@ -174,6 +146,7 @@ function getConfig(callback) {
            adminFqdn: config.adminFqdn(),
            mailFqdn: config.mailFqdn(),
            version: config.version(),
+            progress: progress.getAll(),
            isDemo: config.isDemo(),
            edition: config.edition(),
            memory: os.totalmem(),
@@ -184,14 +157,7 @@ function getConfig(callback) {
 }

 function reboot(callback) {
-    shell.sudo('reboot', [ REBOOT_CMD ], {}, callback);
-}
-
-function isRebootRequired(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    // https://serverfault.com/questions/92932/how-does-ubuntu-keep-track-of-the-system-restart-required-flag-in-motd
-    callback(null, fs.existsSync('/var/run/reboot-required'));
+    shell.sudo('reboot', [ REBOOT_CMD ], callback);
 }

 function checkDiskSpace(callback) {
@@ -253,28 +219,49 @@ function getLogs(unit, options, callback) {

    debug('Getting logs for %s as %s', unit, format);

-    let args = [ '--lines=' + lines ];
-    if (follow) args.push('--follow');
+    var cp, transformStream;
+    if (unit === 'box') {
+        let args = [ '--no-pager', `--lines=${lines}` ];
+        if (format === 'short') args.push('--output=short', '-a'); else args.push('--output=json');
+        if (follow) args.push('--follow');
+        args.push('--unit=box');
+        args.push('--unit=cloudron-updater');
+        cp = spawn('/bin/journalctl', args);

-    // need to handle box.log without subdir
-    if (unit === 'box') args.push(path.join(paths.LOG_DIR, 'box.log'));
-    else args.push(path.join(paths.LOG_DIR, unit, 'app.log'));
+        transformStream = split(function mapper(line) {
+            if (format !== 'json') return line + '\n';

-    var cp = spawn('/usr/bin/tail', args);
+            var obj = safe.JSON.parse(line);
+            if (!obj) return undefined;

-    var transformStream = split(function mapper(line) {
-        if (format !== 'json') return line + '\n';
+            return JSON.stringify({
+                realtimeTimestamp: obj.__REALTIME_TIMESTAMP,
+                monotonicTimestamp: obj.__MONOTONIC_TIMESTAMP,
+                message: obj.MESSAGE,
+                source: obj.SYSLOG_IDENTIFIER || ''
+            }) + '\n';
+        });
+    } else { // mail, mongodb, mysql, postgresql, backup
+        let args = [ '--lines=' + lines ];
+        if (follow) args.push('--follow');
+        args.push(path.join(paths.LOG_DIR, unit, 'app.log'));

-        var data = line.split(' '); // logs are <ISOtimestamp> <msg>
-        var timestamp = (new Date(data[0])).getTime();
-        if (isNaN(timestamp)) timestamp = 0;
+        cp = spawn('/usr/bin/tail', args);

-        return JSON.stringify({
-            realtimeTimestamp: timestamp * 1000,
-            message: line.slice(data[0].length+1),
-            source: unit
-        }) + '\n';
-    });
+        transformStream = split(function mapper(line) {
+            if (format !== 'json') return line + '\n';
+
+            var data = line.split(' '); // logs are <ISOtimestamp> <msg>
+            var timestamp = (new Date(data[0])).getTime();
+            if (isNaN(timestamp)) timestamp = 0;
+
+            return JSON.stringify({
+                realtimeTimestamp: timestamp * 1000,
+                message: line.slice(data[0].length+1),
+                source: unit
+            }) + '\n';
+        });
+    }

    transformStream.close = cp.kill.bind(cp, 'SIGKILL'); // closing stream kills the child process

@@ -282,108 +269,3 @@ function getLogs(unit, options, callback) {

    return callback(null, transformStream);
 }
-
-function configureWebadmin(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('configureWebadmin: adminDomain:%s status:%j', config.adminDomain(), gWebadminStatus);
-
-    if (process.env.BOX_ENV === 'test' || !config.adminDomain() || gWebadminStatus.configuring) return callback();
-
-    gWebadminStatus.configuring = true; // re-entracy guard
-
-    function configureReverseProxy(error) {
-        debug('configureReverseProxy: error %j', error || null);
-
-        reverseProxy.configureAdmin({ userId: null, username: 'setup' }, function (error) {
-            debug('configureWebadmin: done error: %j', error || {});
-            gWebadminStatus.configuring = false;
-
-            if (error) return callback(error);
-
-            gWebadminStatus.tls = true;
-
-            callback();
-        });
-    }
-
-    // update the DNS. configure nginx regardless of whether it succeeded so that
-    // box is accessible even if dns creds are invalid
-    sysinfo.getPublicIp(function (error, ip) {
-        if (error) return configureReverseProxy(error);
-
-        domains.upsertDnsRecords(config.adminLocation(), config.adminDomain(), 'A', [ ip ], function (error) {
-            debug('addWebadminDnsRecord: updated records with error:', error);
-            if (error) return configureReverseProxy(error);
-
-            domains.waitForDnsRecord(config.adminLocation(), config.adminDomain(), 'A', ip, { interval: 30000, times: 50000 }, function (error) {
-                if (error) return configureReverseProxy(error);
-
-                gWebadminStatus.dns = true;
-
-                configureReverseProxy();
-            });
-        });
-    });
-}
-
-function getWebadminStatus() {
-    return gWebadminStatus;
-}
-
-function getStatus(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    users.isActivated(function (error, activated) {
-        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-        settings.getCloudronName(function (error, cloudronName) {
-            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-            callback(null, {
-                version: config.version(),
-                apiServerOrigin: config.apiServerOrigin(), // used by CaaS tool
-                provider: config.provider(),
-                cloudronName: cloudronName,
-                adminFqdn: config.adminDomain() ? config.adminFqdn() : null,
-                activated: activated,
-                edition: config.edition(),
-                webadminStatus: gWebadminStatus // only valid when !activated
-            });
-        });
-    });
-}
-
-function setDashboardDomain(domain, callback) {
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug(`setDashboardDomain: ${domain}`);
-
-    domains.get(domain, function (error, result) {
-        if (error && error.reason === DomainsError.NOT_FOUND) return callback(new CloudronError(CloudronError.BAD_FIELD, 'No such domain'));
-        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-        config.setAdminDomain(result.domain);
-        config.setAdminLocation('my');
-        config.setAdminFqdn('my' + (result.config.hyphenatedSubdomains ? '-' : '.') + result.domain);
-
-        clients.addDefaultClients(config.adminOrigin(), function (error) {
-            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-            callback(null);
-
-            configureWebadmin(NOOP_CALLBACK); // ## trigger as task
-        });
-    });
-}
-
-function renewCerts(options, auditSource, callback) {
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    let task = tasks.startTask(tasks.TASK_RENEW_CERTS, [ options, auditSource ]);
-    task.on('error', (error) => callback(new CloudronError(CloudronError.INTERNAL_ERROR, error)));
-    task.on('start', (taskId) => callback(null, taskId));
-}
@@ -22,6 +22,7 @@ exports = module.exports = {
    setAdminFqdn: setAdminFqdn,
    setAdminLocation: setAdminLocation,
    version: version,
+    setVersion: setVersion,
    database: database,
    edition: edition,

@@ -36,11 +37,13 @@ exports = module.exports = {
    hasIPv6: hasIPv6,
    dkimSelector: dkimSelector,

-    isManaged: isManaged,
    isDemo: isDemo,

    // feature flags based on editions (these have a separate license from standard edition)
    isSpacesEnabled: isSpacesEnabled,
+    allowHyphenatedSubdomains: allowHyphenatedSubdomains,
+    allowOperatorActions: allowOperatorActions,
+    isAdminDomainLocked: isAdminDomainLocked,

    // for testing resets to defaults
    _reset: _reset
@@ -56,20 +59,24 @@ var assert = require('assert'),
 // assert on unknown environment can't proceed
 assert(exports.CLOUDRON || exports.TEST, 'Unknown environment. This should not happen!');

+var homeDir = process.env.HOME || process.env.HOMEPATH || process.env.USERPROFILE;
+
 var data = { };

 function baseDir() {
-    const homeDir = process.env.HOME || process.env.HOMEPATH || process.env.USERPROFILE;
    if (exports.CLOUDRON) return homeDir;
    if (exports.TEST) return path.join(homeDir, '.cloudron_test');
-    // cannot reach
 }

-const cloudronConfigFileName = exports.CLOUDRON ? '/etc/cloudron/cloudron.conf' : path.join(baseDir(), 'cloudron.conf');
+var cloudronConfigFileName = path.join(baseDir(), 'configs/cloudron.conf');
+
+// only tests can run without a config file on disk, they use the defaults with runtime overrides
+if (exports.CLOUDRON) assert(fs.existsSync(cloudronConfigFileName), 'No cloudron.conf found, cannot proceed');

 function saveSync() {
    // only save values we want to have in the cloudron.conf, see start.sh
    var conf = {
+        version: data.version,
        apiServerOrigin: data.apiServerOrigin,
        webServerOrigin: data.webServerOrigin,
        adminDomain: data.adminDomain,
@@ -97,6 +104,7 @@ function initConfig() {
    data.adminDomain = '';
    data.adminLocation = 'my';
    data.port = 3000;
+    data.version = null;
    data.apiServerOrigin = null;
    data.webServerOrigin = null;
    data.provider = 'generic';
@@ -117,6 +125,7 @@ function initConfig() {

    // overrides for local testings
    if (exports.TEST) {
+        data.version = '1.1.1-test';
        data.port = 5454;
        data.apiServerOrigin = 'http://localhost:6060'; // hock doesn't support https
        data.database.password = '';
@@ -205,8 +214,11 @@ function sysadminOrigin() {
 }

 function version() {
-    if (exports.TEST) return '3.0.0-test';
-    return fs.readFileSync(path.join(__dirname, '../VERSION'), 'utf8').trim();
+    return get('version');
+}
+
+function setVersion(version) {
+    set('version', version);
 }

 function database() {
@@ -221,12 +233,21 @@ function isSpacesEnabled() {
    return get('edition') === 'education';
 }

-function provider() {
-    return get('provider');
+function allowHyphenatedSubdomains() {
+    // we should move caas also to hostingprovider edition at some point
+    return get('edition') === 'hostingprovider' || get('provider') === 'caas';
 }

-function isManaged() {
-    return edition() === 'hostingprovider';
+function allowOperatorActions() {
+    return get('edition') !== 'hostingprovider';
+}
+
+function isAdminDomainLocked() {
+    return get('edition') === 'hostingprovider';
+}
+
+function provider() {
+    return get('provider');
 }

 function hasIPv6() {
@@ -20,7 +20,6 @@ exports = module.exports = {
    ADMIN_NAME: 'Settings',

    NGINX_ADMIN_CONFIG_FILE_NAME: 'admin.conf',
-    NGINX_DEFAULT_CONFIG_FILE_NAME: 'default.conf',

    GHOST_USER_FILE: '/tmp/cloudron_ghost.json',

@@ -7,8 +7,7 @@ exports = module.exports = {
    stopJobs: stopJobs
 };

-var appHealthMonitor = require('./apphealthmonitor.js'),
-    apps = require('./apps.js'),
+var apps = require('./apps.js'),
    appstore = require('./appstore.js'),
    assert = require('assert'),
    backups = require('./backups.js'),
@@ -22,6 +21,7 @@ var appHealthMonitor = require('./apphealthmonitor.js'),
    dyndns = require('./dyndns.js'),
    eventlog = require('./eventlog.js'),
    janitor = require('./janitor.js'),
+    reverseProxy = require('./reverseproxy.js'),
    scheduler = require('./scheduler.js'),
    settings = require('./settings.js'),
    updater = require('./updater.js'),
@@ -42,12 +42,11 @@ var gJobs = {
    cleanupTokens: null,
    digestEmail: null,
    dockerVolumeCleaner: null,
-    dynamicDns: null,
-    schedulerSync: null,
-    appHealthMonitor: null
+    dynamicDNS: null,
+    schedulerSync: null
 };

-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+var NOOP_CALLBACK = function (error) { if (error) console.error(error); };
 var AUDIT_SOURCE = { userId: null, username: 'cron' };

 // cron format
@@ -85,10 +84,10 @@ function startPostActivationJobs(callback) {
        start: true
    });

-    settings.on(settings.TIME_ZONE_KEY, recreateJobs);
-    settings.on(settings.APP_AUTOUPDATE_PATTERN_KEY, appAutoupdatePatternChanged);
-    settings.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, boxAutoupdatePatternChanged);
-    settings.on(settings.DYNAMIC_DNS_KEY, dynamicDnsChanged);
+    settings.events.on(settings.TIME_ZONE_KEY, recreateJobs);
+    settings.events.on(settings.APP_AUTOUPDATE_PATTERN_KEY, appAutoupdatePatternChanged);
+    settings.events.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, boxAutoupdatePatternChanged);
+    settings.events.on(settings.DYNAMIC_DNS_KEY, dynamicDnsChanged);

    settings.getAll(function (error, allSettings) {
        if (error) return callback(error);
@@ -185,7 +184,7 @@ function recreateJobs(tz) {
    if (gJobs.certificateRenew) gJobs.certificateRenew.stop();
    gJobs.certificateRenew = new CronJob({
        cronTime: '00 00 */12 * * *', // every 12 hours
-        onTick: cloudron.renewCerts.bind(null, {}, AUDIT_SOURCE, NOOP_CALLBACK),
+        onTick: reverseProxy.renewAll.bind(null, AUDIT_SOURCE, NOOP_CALLBACK),
        start: true,
        timeZone: tz
    });
@@ -197,14 +196,6 @@ function recreateJobs(tz) {
        start: true,
        timeZone: tz
    });
-
-    if (gJobs.appHealthMonitor) gJobs.appHealthMonitor.stop();
-    gJobs.appHealthMonitor = new CronJob({
-        cronTime: '*/10 * * * * *', // every 10 seconds
-        onTick: appHealthMonitor.run.bind(null, 10, NOOP_CALLBACK),
-        start: true,
-        timeZone: tz
-    });
 }

 function boxAutoupdatePatternChanged(pattern) {
@@ -266,25 +257,25 @@ function dynamicDnsChanged(enabled) {
    debug('Dynamic DNS setting changed to %s', enabled);

    if (enabled) {
-        gJobs.dynamicDns = new CronJob({
+        gJobs.dynamicDNS = new CronJob({
            cronTime: '00 */10 * * * *',
            onTick: dyndns.sync,
            start: true,
            timeZone: gJobs.boxUpdateCheckerJob.cronTime.zone // hack
        });
    } else {
-        if (gJobs.dynamicDns) gJobs.dynamicDns.stop();
-        gJobs.dynamicDns = null;
+        if (gJobs.dynamicDNS) gJobs.dynamicDNS.stop();
+        gJobs.dynamicDNS = null;
    }
 }

 function stopJobs(callback) {
    assert.strictEqual(typeof callback, 'function');

-    settings.removeListener(settings.TIME_ZONE_KEY, recreateJobs);
-    settings.removeListener(settings.APP_AUTOUPDATE_PATTERN_KEY, appAutoupdatePatternChanged);
-    settings.removeListener(settings.BOX_AUTOUPDATE_PATTERN_KEY, boxAutoupdatePatternChanged);
-    settings.removeListener(settings.DYNAMIC_DNS_KEY, dynamicDnsChanged);
+    settings.events.removeListener(settings.TIME_ZONE_KEY, recreateJobs);
+    settings.events.removeListener(settings.APP_AUTOUPDATE_PATTERN_KEY, appAutoupdatePatternChanged);
+    settings.events.removeListener(settings.BOX_AUTOUPDATE_PATTERN_KEY, boxAutoupdatePatternChanged);
+    settings.events.removeListener(settings.DYNAMIC_DNS_KEY, dynamicDnsChanged);

    for (var job in gJobs) {
        if (!gJobs[job]) continue;
@@ -29,6 +29,13 @@ function translateRequestError(result, callback) {
    if ((result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) && result.body.errors.length > 0) {
        let error = result.body.errors[0];
        let message = `message: ${error.message} statusCode: ${result.statusCode} code:${error.code}`;
+        if (error.code === 6003) {
+            if (error.error_chain[0] && error.error_chain[0].code === 6103) message = 'Invalid API Key';
+            else message = 'Invalid credentials';
+        } else if (error.error_chain[0] && error.error_chain[0].message) {
+            message = message + ' ' + error.error_chain[0].message;
+        }
+
        return callback(new DomainsError(DomainsError.ACCESS_DENIED, message));
    }

@@ -1,13 +1,7 @@
 'use strict';

 exports = module.exports = {
-    DockerError: DockerError,
-
    connection: connectionInstance(),
-    setRegistryConfig: setRegistryConfig,
-
-    ping: ping,
-
    downloadImage: downloadImage,
    createContainer: createContainer,
    startContainer: startContainer,
@@ -22,26 +16,21 @@ exports = module.exports = {
    getContainerIdByIp: getContainerIdByIp,
    inspect: inspect,
    inspectByName: inspect,
-    memoryUsage: memoryUsage,
-    execContainer: execContainer,
-    createVolume: createVolume,
-    removeVolume: removeVolume,
-    clearVolume: clearVolume
+    execContainer: execContainer
 };

-// timeout is optional
-function connectionInstance(timeout) {
+function connectionInstance() {
    var Docker = require('dockerode');
    var docker;

    if (process.env.BOX_ENV === 'test') {
        // test code runs a docker proxy on this port
-        docker = new Docker({ host: 'http://localhost', port: 5687, timeout: timeout });
+        docker = new Docker({ host: 'http://localhost', port: 5687 });

        // proxy code uses this to route to the real docker
        docker.options = { socketPath: '/var/run/docker.sock' };
    } else {
-        docker = new Docker({ socketPath: '/var/run/docker.sock', timeout: timeout });
+        docker = new Docker({ socketPath: '/var/run/docker.sock' });
    }

    return docker;
@@ -54,83 +43,25 @@ var addons = require('./addons.js'),
    config = require('./config.js'),
    constants = require('./constants.js'),
    debug = require('debug')('box:docker.js'),
-    mkdirp = require('mkdirp'),
    once = require('once'),
-    path = require('path'),
-    paths = require('./paths.js'),
    safe = require('safetydance'),
    shell = require('./shell.js'),
    spawn = child_process.spawn,
    util = require('util'),
    _ = require('underscore');

-const RMVOLUME_CMD = path.join(__dirname, 'scripts/rmvolume.sh');
-
-function DockerError(reason, errorOrMessage) {
-    assert.strictEqual(typeof reason, 'string');
-    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
-
-    Error.call(this);
-    Error.captureStackTrace(this, this.constructor);
-
-    this.name = this.constructor.name;
-    this.reason = reason;
-    if (typeof errorOrMessage === 'undefined') {
-        this.message = reason;
-    } else if (typeof errorOrMessage === 'string') {
-        this.message = errorOrMessage;
-    } else {
-        this.message = 'Internal error';
-        this.nestedError = errorOrMessage;
-    }
-}
-util.inherits(DockerError, Error);
-DockerError.INTERNAL_ERROR = 'Internal Error';
-DockerError.NOT_FOUND = 'Not found';
-DockerError.BAD_FIELD = 'Bad field';
-
 function debugApp(app, args) {
    assert(typeof app === 'object');

    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
 }

-function setRegistryConfig(auth, callback) {
-    assert.strictEqual(typeof auth, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    const isLogin = !!auth.password;
-
-    // currently, auth info is not stashed in the db but maybe it should for restore to work?
-    const cmd = isLogin ? `docker login ${auth.serveraddress} --username ${auth.username} --password ${auth.password}` : `docker logout ${auth.serveraddress}`;
-
-    child_process.exec(cmd, { }, function (error, stdout, stderr) {
-        if (error) return callback(new DockerError(DockerError.BAD_FIELD, stderr));
-
-        callback();
-    });
-}
-
-function ping(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    // do not let the request linger
-    var docker = connectionInstance(1000);
-
-    docker.ping(function (error, result) {
-        if (error) return callback(new DockerError(DockerError.INTERNAL_ERROR, error));
-        if (result !== 'OK') return callback(new DockerError(DockerError.INTERNAL_ERROR, 'Unable to ping the docker daemon'));
-
-        callback(null);
-    });
-}
-
 function pullImage(manifest, callback) {
    var docker = exports.connection;

    // Use docker CLI here to support downloading of private repos. for dockerode, we have to use
    // https://github.com/apocas/dockerode#pull-from-private-repos
-    shell.spawn('pullImage', '/usr/bin/docker', [ 'pull', manifest.dockerImage ], {}, function (error) {
+    shell.exec('pullImage', '/usr/bin/docker', [ 'pull', manifest.dockerImage ], { }, function (error) {
        if (error) {
            debug(`pullImage: Error pulling image ${manifest.dockerImage} of ${manifest.id}: ${error.message}`);
            return callback(new Error('Failed to pull image'));
@@ -182,10 +113,8 @@ function createSubcontainer(app, name, cmd, options, callback) {
    var manifest = app.manifest;
    var exposedPorts = {}, dockerPortBindings = { };
    var domain = app.fqdn;
-    // TODO: these should all have the CLOUDRON_ prefix
    var stdEnv = [
        'CLOUDRON=1',
-        'CLOUDRON_PROXY_IP=172.18.0.1',
        'WEBADMIN_ORIGIN=' + config.adminOrigin(),
        'API_ORIGIN=' + config.adminOrigin(),
        'APP_ORIGIN=https://' + domain,
@@ -211,9 +140,6 @@ function createSubcontainer(app, name, cmd, options, callback) {
        dockerPortBindings[`${containerPort}/${portType}`] = [ { HostIp: '0.0.0.0', HostPort: hostPort + '' } ];
    }

-    let appEnv = [];
-    Object.keys(app.env).forEach(function (name) { appEnv.push(`${name}=${app.env[name]}`); });
-
    // first check db record, then manifest
    var memoryLimit = app.memoryLimit || manifest.memoryLimit || 0;

@@ -227,6 +153,9 @@ function createSubcontainer(app, name, cmd, options, callback) {
    // if required, we can make this a manifest and runtime argument later
    if (!isAppContainer) memoryLimit *= 2;

+    // apparmor is disabled on few servers
+    var enableSecurityOpt = config.CLOUDRON && safe(function () { return child_process.spawnSync('aa-enabled').status === 0; }, false);
+
    addons.getEnvironment(app, function (error, addonEnv) {
        if (error) return callback(new Error('Error getting addon environment : ' + error));

@@ -238,10 +167,9 @@ function createSubcontainer(app, name, cmd, options, callback) {
        var containerOptions = {
            name: name, // used for filtering logs
            Tty: isAppContainer,
-            Hostname: app.id, // set to something 'constant' so app containers can use this to communicate (across app updates)
            Image: app.manifest.dockerImage,
            Cmd: (isAppContainer && app.debugMode && app.debugMode.cmd) ? app.debugMode.cmd : cmd,
-            Env: stdEnv.concat(addonEnv).concat(portEnv).concat(appEnv),
+            Env: stdEnv.concat(addonEnv).concat(portEnv),
            ExposedPorts: isAppContainer ? exposedPorts : { },
            Volumes: { // see also ReadonlyRootfs
                '/tmp': {},
@@ -250,11 +178,10 @@ function createSubcontainer(app, name, cmd, options, callback) {
            Labels: {
                'fqdn': app.fqdn,
                'appId': app.id,
-                'isSubcontainer': String(!isAppContainer),
-                'isCloudronManaged': String(true)
+                'isSubcontainer': String(!isAppContainer)
            },
            HostConfig: {
-                Mounts: addons.getMountsSync(app, app.manifest.addons),
+                Binds: addons.getBindsSync(app, app.manifest.addons),
                LogConfig: {
                    Type: 'syslog',
                    Config: {
@@ -277,7 +204,7 @@ function createSubcontainer(app, name, cmd, options, callback) {
                NetworkMode: 'cloudron',
                Dns: ['172.18.0.1'], // use internal dns
                DnsSearch: ['.'], // use internal dns
-                SecurityOpt: [ 'apparmor=docker-cloudron-app' ]
+                SecurityOpt: enableSecurityOpt ? [ 'apparmor=docker-cloudron-app' ] : null // profile available only on cloudron
            }
        };

@@ -425,8 +352,7 @@ function deleteImage(manifest, callback) {
    // just removes the tag). we used to remove the image by id. this is not required anymore because aliases are
    // not created anymore after https://github.com/docker/docker/pull/10571
    docker.getImage(dockerImage).remove(removeOptions, function (error) {
-        if (error && error.statusCode === 400) return callback(null); // invalid image format. this can happen if user installed with a bad --docker-image
-        if (error && error.statusCode === 404) return callback(null); // not found
+        if (error && error.statusCode === 404) return callback(null);
        if (error && error.statusCode === 409) return callback(null); // another container using the image

        if (error) debug('Error removing image %s : %j', dockerImage, error);
@@ -465,23 +391,7 @@ function inspect(containerId, callback) {
    var container = exports.connection.getContainer(containerId);

    container.inspect(function (error, result) {
-        if (error && error.statusCode === 404) return callback(new DockerError(DockerError.NOT_FOUND));
-        if (error) return callback(new DockerError(DockerError.INTERNAL_ERROR, error));
-
-        callback(null, result);
-    });
-}
-
-function memoryUsage(containerId, callback) {
-    assert.strictEqual(typeof containerId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    var container = exports.connection.getContainer(containerId);
-
-    container.stats({ stream: false }, function (error, result) {
-        if (error && error.statusCode === 404) return callback(new DockerError(DockerError.NOT_FOUND));
-        if (error) return callback(new DockerError(DockerError.INTERNAL_ERROR, error));
-
+        if (error) return callback(error);
        callback(null, result);
    });
 }
@@ -516,66 +426,3 @@ function execContainer(containerId, cmd, options, callback) {

    if (options.stdin) options.stdin.pipe(cp.stdin).on('error', callback);
 }
-
-function createVolume(app, name, subdir, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof subdir, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    let docker = exports.connection;
-
-    const volumeDataDir = path.join(paths.APPS_DATA_DIR, app.id, subdir);
-
-    const volumeOptions = {
-        Name: name,
-        Driver: 'local',
-        DriverOpts: { // https://github.com/moby/moby/issues/19990#issuecomment-248955005
-            type: 'none',
-            device: volumeDataDir,
-            o: 'bind'
-        },
-        Labels: {
-            'fqdn': app.fqdn,
-            'appId': app.id
-        },
-    };
-
-    mkdirp(volumeDataDir, function (error) {
-        if (error) return callback(new Error(`Error creating app data dir: ${error.message}`));
-
-        docker.createVolume(volumeOptions, function (error) {
-            if (error) return callback(error);
-
-            callback();
-        });
-    });
-}
-
-function clearVolume(app, name, subdir, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof subdir, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    shell.sudo('removeVolume', [ RMVOLUME_CMD, app.id, subdir ], {}, callback);
-}
-
-function removeVolume(app, name, subdir, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof subdir, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    let docker = exports.connection;
-
-    let volume = docker.getVolume(name);
-    volume.remove(function (error) {
-        if (error && error.statusCode !== 404) {
-            debug(`removeVolume: Error removing volume of ${app.id} ${error}`);
-            callback(error);
-        }
-
-        shell.sudo('removeVolume', [ RMVOLUME_CMD, app.id, subdir ], {}, callback);
-    });
-}
@@ -8,7 +8,8 @@ exports = module.exports = {
    getAll: getAll,
    update: update,
    del: del,
-    clear: clear
+
+    _clear: clear
 };

 var assert = require('assert'),
@@ -6,10 +6,10 @@ module.exports = exports = {
    getAll: getAll,
    update: update,
    del: del,
-    clear: clear,
    isLocked: isLocked,

    fqdn: fqdn,
+    setAdmin: setAdmin,

    getDnsRecords: getDnsRecords,
    upsertDnsRecords: upsertDnsRecords,
@@ -24,29 +24,29 @@ module.exports = exports = {

    makeWildcard: makeWildcard,

-    parentDomain: parentDomain,
-
-    DomainsError: DomainsError,
-
-    // exported for testing
-    _getName: getName
+    DomainsError: DomainsError
 };

 var assert = require('assert'),
+    caas = require('./caas.js'),
    config = require('./config.js'),
    constants = require('./constants.js'),
    DatabaseError = require('./databaseerror.js'),
    debug = require('debug')('box:domains'),
    domaindb = require('./domaindb.js'),
-    eventlog = require('./eventlog.js'),
+    path = require('path'),
    reverseProxy = require('./reverseproxy.js'),
    ReverseProxyError = reverseProxy.ReverseProxyError,
    safe = require('safetydance'),
+    shell = require('./shell.js'),
    sysinfo = require('./sysinfo.js'),
    tld = require('tldjs'),
    util = require('util'),
    _ = require('underscore');

+var RESTART_CMD = path.join(__dirname, 'scripts/restart.sh');
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
 function DomainsError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
@@ -97,11 +97,6 @@ function api(provider) {
    }
 }

-function parentDomain(domain) {
-    assert.strictEqual(typeof domain, 'string');
-    return domain.replace(/^\S+?\./, ''); // +? means non-greedy
-}
-
 function verifyDnsConfig(dnsConfig, domain, zoneName, provider, ip, callback) {
    assert(dnsConfig && typeof dnsConfig === 'object'); // the dns config to test with
    assert.strictEqual(typeof domain, 'string');
@@ -192,17 +187,15 @@ function validateTlsConfig(tlsConfig, dnsProvider) {
    return null;
 }

-function add(domain, data, auditSource, callback) {
+function add(domain, zoneName, provider, dnsConfig, fallbackCertificate, tlsConfig, callback) {
    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof data.zoneName, 'string');
-    assert.strictEqual(typeof data.provider, 'string');
-    assert.strictEqual(typeof data.config, 'object');
-    assert.strictEqual(typeof data.fallbackCertificate, 'object');
-    assert.strictEqual(typeof data.tlsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof fallbackCertificate, 'object');
+    assert.strictEqual(typeof tlsConfig, 'object');
    assert.strictEqual(typeof callback, 'function');

-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
-
    if (!tld.isValid(domain)) return callback(new DomainsError(DomainsError.BAD_FIELD, 'Invalid domain'));
    if (domain.endsWith('.')) return callback(new DomainsError(DomainsError.BAD_FIELD, 'Invalid domain'));

@@ -214,31 +207,28 @@ function add(domain, data, auditSource, callback) {
    }

    if (fallbackCertificate) {
-        let error = reverseProxy.validateCertificate('test', { domain, config }, fallbackCertificate);
+        let error = reverseProxy.validateCertificate(`test.${domain}`, fallbackCertificate.cert, fallbackCertificate.key);
        if (error) return callback(new DomainsError(DomainsError.BAD_FIELD, error.message));
-    } else {
-        fallbackCertificate = reverseProxy.generateFallbackCertificateSync({ domain, config });
-        if (fallbackCertificate.error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, fallbackCertificate.error));
    }

    let error = validateTlsConfig(tlsConfig, provider);
    if (error) return callback(error);

+    if (dnsConfig.hyphenatedSubdomains && !config.allowHyphenatedSubdomains()) return callback(new DomainsError(DomainsError.BAD_FIELD, 'Not allowed in this edition'));
+
    sysinfo.getPublicIp(function (error, ip) {
        if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, 'Error getting IP:' + error.message));

-        verifyDnsConfig(config, domain, zoneName, provider, ip, function (error, sanitizedConfig) {
+        verifyDnsConfig(dnsConfig, domain, zoneName, provider, ip, function (error, result) {
            if (error) return callback(error);

-            domaindb.add(domain, { zoneName: zoneName, provider: provider, config: sanitizedConfig, tlsConfig: tlsConfig }, function (error) {
+            domaindb.add(domain, { zoneName: zoneName, provider: provider, config: result, tlsConfig: tlsConfig }, function (error) {
                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new DomainsError(DomainsError.ALREADY_EXISTS));
                if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

                reverseProxy.setFallbackCertificate(domain, fallbackCertificate, function (error) {
                    if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

-                    eventlog.add(eventlog.ACTION_DOMAIN_ADD, auditSource, { domain, zoneName, provider });
-
                    callback();
                });
            });
@@ -247,7 +237,7 @@ function add(domain, data, auditSource, callback) {
 }

 function isLocked(domain) {
-    return domain === config.adminDomain() && config.edition() === 'hostingprovider';
+    return domain === config.adminDomain() && config.isAdminDomainLocked();
 }

 function get(domain, callback) {
@@ -288,43 +278,42 @@ function getAll(callback) {
    });
 }

-function update(domain, data, auditSource, callback) {
+function update(domain, zoneName, provider, dnsConfig, fallbackCertificate, tlsConfig, callback) {
    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof data.zoneName, 'string');
-    assert.strictEqual(typeof data.provider, 'string');
-    assert.strictEqual(typeof data.config, 'object');
-    assert.strictEqual(typeof data.fallbackCertificate, 'object');
-    assert.strictEqual(typeof data.tlsConfig, 'object');
-    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof fallbackCertificate, 'object');
+    assert.strictEqual(typeof tlsConfig, 'object');
    assert.strictEqual(typeof callback, 'function');

-    let { zoneName, provider, config, fallbackCertificate, tlsConfig } = data;
-
-    domaindb.get(domain, function (error, domainObject) {
+    domaindb.get(domain, function (error, result) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainsError(DomainsError.NOT_FOUND));
        if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

        if (zoneName) {
            if (!tld.isValid(zoneName)) return callback(new DomainsError(DomainsError.BAD_FIELD, 'Invalid zoneName'));
        } else {
-            zoneName = domainObject.zoneName;
+            zoneName = result.zoneName;
        }

        if (fallbackCertificate) {
-            let error = reverseProxy.validateCertificate('test', domainObject, fallbackCertificate);
+            let error = reverseProxy.validateCertificate(`test.${domain}`, fallbackCertificate.cert, fallbackCertificate.key);
            if (error) return callback(new DomainsError(DomainsError.BAD_FIELD, error.message));
        }

        error = validateTlsConfig(tlsConfig, provider);
        if (error) return callback(error);

+        if (dnsConfig.hyphenatedSubdomains && !config.allowHyphenatedSubdomains()) return callback(new DomainsError(DomainsError.BAD_FIELD, 'Not allowed in this edition'));
+
        sysinfo.getPublicIp(function (error, ip) {
            if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, 'Error getting IP:' + error.message));

-            verifyDnsConfig(config, domain, zoneName, provider, ip, function (error, sanitizedConfig) {
+            verifyDnsConfig(dnsConfig, domain, zoneName, provider, ip, function (error, result) {
                if (error) return callback(error);

-                domaindb.update(domain, { zoneName: zoneName, provider: provider, config: sanitizedConfig, tlsConfig: tlsConfig }, function (error) {
+                domaindb.update(domain, { zoneName: zoneName, provider: provider, config: result, tlsConfig: tlsConfig }, function (error) {
                    if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainsError(DomainsError.NOT_FOUND));
                    if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

@@ -333,8 +322,6 @@ function update(domain, data, auditSource, callback) {
                    reverseProxy.setFallbackCertificate(domain, fallbackCertificate, function (error) {
                        if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

-                        eventlog.add(eventlog.ACTION_DOMAIN_UPDATE, auditSource, { domain, zoneName, provider });
-
                        callback();
                    });
                });
@@ -343,9 +330,8 @@ function update(domain, data, auditSource, callback) {
    });
 }

-function del(domain, auditSource, callback) {
+function del(domain, callback) {
    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    if (domain === config.adminDomain()) return callback(new DomainsError(DomainsError.IN_USE));
@@ -355,43 +341,25 @@ function del(domain, auditSource, callback) {
        if (error && error.reason === DatabaseError.IN_USE) return callback(new DomainsError(DomainsError.IN_USE));
        if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));

-        eventlog.add(eventlog.ACTION_DOMAIN_REMOVE, auditSource, { domain });
-
-        return callback(null);
-    });
-}
-
-function clear(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    domaindb.clear(function (error) {
-        if (error) return callback(new DomainsError(DomainsError.INTERNAL_ERROR, error));
-
        return callback(null);
    });
 }

 // returns the 'name' that needs to be inserted into zone
 function getName(domain, subdomain, type) {
-    // hack for supporting special caas domains. if we want to remove this, we have to fix the appstore domain API first
+    // support special caas domains
    if (domain.provider === 'caas') return subdomain;

-    const part = domain.domain.slice(0, -domain.zoneName.length - 1);
+    if (domain.domain === domain.zoneName) return subdomain;

-    if (subdomain === '') return part;
+    var part = domain.domain.slice(0, -domain.zoneName.length - 1);

-    if (!domain.config.hyphenatedSubdomains) return part ? `${subdomain}.${part}` : subdomain;
-
-    // hyphenatedSubdomains
-    if (type !== 'TXT') return `${subdomain}-${part}`;
-
-    if (subdomain.startsWith('_acme-challenge.')) {
-        return `${subdomain}-${part}`;
-    } else if (subdomain === '_acme-challenge') {
-        const up = part.replace(/^[^.]*\.?/, ''); // this gets the domain one level up
-        return up ? `${subdomain}.${up}` : subdomain;
-    } else {
+    if (subdomain === '') {
+        return part;
+    } else if (type === 'TXT') {
        return `${subdomain}.${part}`;
+    } else {
+        return subdomain + (domain.config.hyphenatedSubdomains ? '-' : '.') + part;
    }
 }

@@ -470,6 +438,31 @@ function waitForDnsRecord(subdomain, domain, type, value, options, callback) {
    });
 }

+function setAdmin(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('setAdmin domain:%s', domain);
+
+    get(domain, function (error, result) {
+        if (error) return callback(error);
+
+        var setPtrRecord = config.provider() === 'caas' ? caas.setPtrRecord : function (d, next) { next(); };
+
+        setPtrRecord(domain, function (error) {
+            if (error) return callback(new DomainsError(DomainsError.EXTERNAL_ERROR, 'Error setting PTR record:' + error.message));
+
+            config.setAdminDomain(result.domain);
+            config.setAdminLocation('my');
+            config.setAdminFqdn('my' + (result.config.hyphenatedSubdomains ? '-' : '.') + result.domain);
+
+            callback();
+
+            shell.sudo('restart', [ RESTART_CMD ], NOOP_CALLBACK);
+        });
+    });
+}
+
 // removes all fields that are strictly private and should never be returned by API calls
 function removePrivateFields(domain) {
    var result = _.pick(domain, 'domain', 'zoneName', 'provider', 'config', 'tlsConfig', 'fallbackCertificate', 'locked');
@@ -18,30 +18,12 @@ exports = module.exports = {
    ACTION_APP_UNINSTALL: 'app.uninstall',
    ACTION_APP_UPDATE: 'app.update',
    ACTION_APP_LOGIN: 'app.login',
-
    ACTION_BACKUP_FINISH: 'backup.finish',
    ACTION_BACKUP_START: 'backup.start',
    ACTION_BACKUP_CLEANUP: 'backup.cleanup',
-
-    ACTION_CERTIFICATE_NEW: 'certificate.new',
    ACTION_CERTIFICATE_RENEWAL: 'certificate.renew',
-
-    ACTION_DOMAIN_ADD: 'domain.add',
-    ACTION_DOMAIN_UPDATE: 'domain.update',
-    ACTION_DOMAIN_REMOVE: 'domain.remove',
-
-    ACTION_MAIL_ENABLED: 'mail.enabled',
-    ACTION_MAIL_DISABLED: 'mail.disabled',
-    ACTION_MAIL_MAILBOX_ADD: 'mail.box.add',
-    ACTION_MAIL_MAILBOX_REMOVE: 'mail.box.remove',
-    ACTION_MAIL_LIST_ADD: 'mail.list.add',
-    ACTION_MAIL_LIST_REMOVE: 'mail.list.remove',
-
-    ACTION_PROVISION: 'cloudron.provision',
-    ACTION_RESTORE: 'cloudron.restore', // unused
    ACTION_START: 'cloudron.start',
    ACTION_UPDATE: 'cloudron.update',
-
    ACTION_USER_ADD: 'user.add',
    ACTION_USER_LOGIN: 'user.login',
    ACTION_USER_REMOVE: 'user.remove',
@@ -1,40 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    startGraphite: startGraphite
-};
-
-var assert = require('assert'),
-    infra = require('./infra_version.js'),
-    paths = require('./paths.js'),
-    shell = require('./shell.js');
-
-function startGraphite(existingInfra, callback) {
-    assert.strictEqual(typeof existingInfra, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    const tag = infra.images.graphite.tag;
-    const dataDir = paths.PLATFORM_DATA_DIR;
-
-    if (existingInfra.version === infra.version && infra.images.graphite.tag === existingInfra.images.graphite.tag) return callback();
-
-    const cmd = `docker run --restart=always -d --name="graphite" \
-                --net cloudron \
-                --net-alias graphite \
-                --log-driver syslog \
-                --log-opt syslog-address=udp://127.0.0.1:2514 \
-                --log-opt syslog-format=rfc5424 \
-                --log-opt tag=graphite \
-                -m 75m \
-                --memory-swap 150m \
-                --dns 172.18.0.1 \
-                --dns-search=. \
-                -p 127.0.0.1:2003:2003 \
-                -p 127.0.0.1:2004:2004 \
-                -p 127.0.0.1:8417:8000 \
-                -v "${dataDir}/graphite:/var/lib/graphite" \
-                --label isCloudronManaged=true \
-                --read-only -v /tmp -v /run "${tag}"`;
-
-    shell.exec('startGraphite', cmd, callback);
-}
@@ -34,7 +34,7 @@ function get(groupId, callback) {
    assert.strictEqual(typeof groupId, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups WHERE id = ? ORDER BY name', [ groupId ], function (error, result) {
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM groups WHERE id = ? ORDER BY name', [ groupId ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -47,9 +47,9 @@ function getWithMembers(groupId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + GROUPS_FIELDS + ',GROUP_CONCAT(groupMembers.userId) AS userIds ' +
-                    ' FROM userGroups LEFT OUTER JOIN groupMembers ON userGroups.id = groupMembers.groupId ' +
-                    ' WHERE userGroups.id = ? ' +
-                    ' GROUP BY userGroups.id', [ groupId ], function (error, results) {
+                    ' FROM groups LEFT OUTER JOIN groupMembers ON groups.id = groupMembers.groupId ' +
+                    ' WHERE groups.id = ? ' +
+                    ' GROUP BY groups.id', [ groupId ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -63,7 +63,7 @@ function getWithMembers(groupId, callback) {
 function getAll(callback) {
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + GROUPS_FIELDS + ' FROM userGroups', function (error, results) {
+    database.query('SELECT ' + GROUPS_FIELDS + ' FROM groups', function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
@@ -72,8 +72,8 @@ function getAll(callback) {

 function getAllWithMembers(callback) {
    database.query('SELECT ' + GROUPS_FIELDS + ',GROUP_CONCAT(groupMembers.userId) AS userIds ' +
-                    ' FROM userGroups LEFT OUTER JOIN groupMembers ON userGroups.id = groupMembers.groupId ' +
-                    ' GROUP BY userGroups.id', function (error, results) {
+                    ' FROM groups LEFT OUTER JOIN groupMembers ON groups.id = groupMembers.groupId ' +
+                    ' GROUP BY groups.id', function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -88,7 +88,7 @@ function add(id, name, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('INSERT INTO userGroups (id, name) VALUES (?, ?)', [ id, name ], function (error, result) {
+    database.query('INSERT INTO groups (id, name) VALUES (?, ?)', [ id, name ], function (error, result) {
        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, error));
        if (error || result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

@@ -112,8 +112,8 @@ function update(id, data, callback) {
    }
    args.push(id);

-    database.query('UPDATE userGroups SET ' + fields.join(', ') + ' WHERE id = ?', args, function (error, result) {
-        if (error && error.code === 'ER_DUP_ENTRY' && error.sqlMessage.indexOf('userGroups_name') !== -1) return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'name already exists'));
+    database.query('UPDATE groups SET ' + fields.join(', ') + ' WHERE id = ?', args, function (error, result) {
+        if (error && error.code === 'ER_DUP_ENTRY' && error.sqlMessage.indexOf('groups_name') !== -1) return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'name already exists'));
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -128,7 +128,7 @@ function del(id, callback) {
    // also cleanup the groupMembers table
    var queries = [];
    queries.push({ query: 'DELETE FROM groupMembers WHERE groupId = ?', args: [ id ] });
-    queries.push({ query: 'DELETE FROM userGroups WHERE id = ?', args: [ id ] });
+    queries.push({ query: 'DELETE FROM groups WHERE id = ?', args: [ id ] });

    database.transaction(queries, function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
@@ -141,7 +141,7 @@ function del(id, callback) {
 function count(callback) {
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT COUNT(*) AS total FROM userGroups', function (error, result) {
+    database.query('SELECT COUNT(*) AS total FROM groups', function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        return callback(null, result[0].total);
@@ -152,7 +152,7 @@ function clear(callback) {
    database.query('DELETE FROM groupMembers', function (error) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

-        database.query('DELETE FROM userGroups', function (error) {
+        database.query('DELETE FROM groups', function (error) {
            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            callback(error);
@@ -266,7 +266,7 @@ function getGroups(userId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + GROUPS_FIELDS + ' ' +
-        ' FROM userGroups INNER JOIN groupMembers ON userGroups.id = groupMembers.groupId AND groupMembers.userId = ?', [ userId ], function (error, results) {
+        ' FROM groups INNER JOIN groupMembers ON groups.id = groupMembers.groupId AND groupMembers.userId = ?', [ userId ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
@@ -5,21 +5,20 @@
 // Do not require anything here!

 exports = module.exports = {
-    // a version change recreates all containers with latest docker config
-    'version': '48.12.1',
+    // a major version makes all apps restore from backup. #451 must be fixed before we do this.
+    // a minor version makes all apps re-configure themselves
+    'version': '48.11.0',

-    'baseImages': [
-        { repo: 'cloudron/base', tag: 'cloudron/base:1.0.0@sha256:147a648a068a2e746644746bbfb42eb7a50d682437cead3c67c933c546357617' }
-    ],
+    'baseImages': [ 'cloudron/base:0.10.0' ],

-    // a major version bump in the db containers will trigger the restore logic that uses the db dumps
-    // docker inspect --format='{{index .RepoDigests 0}}' $IMAGE to get the sha256
+    // Note that if any of the databases include an upgrade, bump the infra version above
+    // This is because we upgrade using dumps instead of mysql_upgrade, pg_upgrade etc
    'images': {
-        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.0.1@sha256:5a13360da4a2085c7d474bea6b1090c5eb24732d4f73459942af7612d4993d7f' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:2.0.2@sha256:6dcee0731dfb9b013ed94d56205eee219040ee806c7e251db3b3886eaa4947ff' },
-        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:2.0.2@sha256:95e006390ddce7db637e1672eb6f3c257d3c2652747424f529b1dee3cbe6728c' },
-        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.0.0@sha256:8a88dd334b62b578530a014ca1a2425a54cb9df1e475f5d3a36806e5cfa22121' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.0.1@sha256:deee3739011670d45abd8997a8a0b8d3c4cd577a93f235417614dea58338e0f9' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.0.2@sha256:454f035d60b768153d4f31210380271b5ba1c09367c9d95c7fa37f9e39d2f59c' }
+        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:1.1.0' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:1.1.0' },
+        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:1.1.0' },
+        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:1.0.0' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:1.4.0' },
+        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:1.0.0' }
    }
 };
@@ -271,6 +271,7 @@ function mailboxSearch(req, res, next) {
                    cn: `${mailbox.name}@${mailbox.domain}`,
                    uid: `${mailbox.name}@${mailbox.domain}`,
                    mail: `${mailbox.name}@${mailbox.domain}`,
+                    ownerType: mailbox.ownerType,
                    displayname: 'Max Mustermann',
                    givenName: 'Max',
                    username: 'mmustermann',
@@ -296,6 +297,9 @@ function mailboxSearch(req, res, next) {

            var results = [];

+            // only send user mailboxes
+            result = result.filter(function (m) { return m.ownerType === mailboxdb.OWNER_TYPE_USER; });
+
            // send mailbox objects
            result.forEach(function (mailbox) {
                var dn = ldap.parseDN(`cn=${mailbox.name}@${domain},domain=${domain},ou=mailboxes,dc=cloudron`);
@@ -307,7 +311,8 @@ function mailboxSearch(req, res, next) {
                        objectcategory: 'mailbox',
                        cn: `${mailbox.name}@${domain}`,
                        uid: `${mailbox.name}@${domain}`,
-                        mail: `${mailbox.name}@${domain}`
+                        mail: `${mailbox.name}@${domain}`,
+                        ownerType: mailbox.ownerType
                    }
                };

@@ -450,8 +455,8 @@ function authorizeUserForApp(req, res, next) {
    });
 }

-function authenticateUserMailbox(req, res, next) {
-    debug('user mailbox auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
+function authenticateMailbox(req, res, next) {
+    debug('mailbox auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);

    if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));

@@ -459,61 +464,30 @@ function authenticateUserMailbox(req, res, next) {
    var parts = email.split('@');
    if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));

-    mail.getDomain(parts[1], function (error, domain) {
-        if (error && error.reason === MailError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+    mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
        if (error) return next(new ldap.OperationsError(error.message));

-        if (!domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-
-        mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
-            if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+        mail.getDomain(parts[1], function (error, domain) {
+            if (error && error.reason === MailError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
            if (error) return next(new ldap.OperationsError(error.message));

-            users.verify(mailbox.ownerId, req.credentials || '', function (error, result) {
-                if (error && error.reason === UsersError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-                if (error && error.reason === UsersError.WRONG_PASSWORD) return next(new ldap.InvalidCredentialsError(req.dn.toString()));
-                if (error) return next(new ldap.OperationsError(error.message));
+            if (mailbox.ownerType === mailboxdb.OWNER_TYPE_APP) {
+                var addonId = req.dn.rdns[1].attrs.ou.value.toLowerCase(); // 'sendmail' or 'recvmail'
+                var name;
+                if (addonId === 'sendmail') name = 'MAIL_SMTP_PASSWORD';
+                else if (addonId === 'recvmail') name = 'MAIL_IMAP_PASSWORD';
+                else return next(new ldap.OperationsError('Invalid DN'));

-                eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
-                res.end();
-            });
-        });
-    });
-}
+                appdb.getAddonConfigByName(mailbox.ownerId, addonId, name, function (error, value) {
+                    if (error) return next(new ldap.OperationsError(error.message));
+                    if (req.credentials !== value) return next(new ldap.InvalidCredentialsError(req.dn.toString()));

-function authenticateMailAddon(req, res, next) {
-    debug('mail addon auth: %s (from %s)', req.dn.toString(), req.connection.ldap.id);
-
-    if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-
-    var email = req.dn.rdns[0].attrs.cn.value.toLowerCase();
-    var parts = email.split('@');
-    if (parts.length !== 2) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-
-    const addonId = req.dn.rdns[1].attrs.ou.value.toLowerCase(); // 'sendmail' or 'recvmail'
-
-    mail.getDomain(parts[1], function (error, domain) {
-        if (error && error.reason === MailError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-        if (error) return next(new ldap.OperationsError(error.message));
-
-        if (addonId === 'recvmail' && !domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-
-        let name;
-        if (addonId === 'sendmail') name = 'MAIL_SMTP_PASSWORD';
-        else if (addonId === 'recvmail') name = 'MAIL_IMAP_PASSWORD';
-        else return next(new ldap.OperationsError('Invalid DN'));
-
-        // note: with sendmail addon, apps can send mail without a mailbox (unlike users)
-        appdb.getAppIdByAddonConfigValue(addonId, name, req.credentials || '', function (error, appId) {
-            if (error && error.reason !== DatabaseError.NOT_FOUND) return next(new ldap.OperationsError(error.message));
-            if (appId) { // matched app password
-                eventlog.add(eventlog.ACTION_APP_LOGIN, { authType: 'ldap', mailboxId: email }, { appId: appId, addonId: addonId });
-                return res.end();
-            }
-
-            mailboxdb.getMailbox(parts[0], parts[1], function (error, mailbox) {
-                if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-                if (error) return next(new ldap.OperationsError(error.message));
+                    eventlog.add(eventlog.ACTION_APP_LOGIN, { authType: 'ldap', mailboxId: name }, { appId: mailbox.ownerId, addonId: addonId });
+                    return res.end();
+                });
+            } else if (mailbox.ownerType === mailboxdb.OWNER_TYPE_USER) {
+                if (!domain.enabled) return next(new ldap.NoSuchObjectError(req.dn.toString()));

                users.verify(mailbox.ownerId, req.credentials || '', function (error, result) {
                    if (error && error.reason === UsersError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
@@ -523,7 +497,9 @@ function authenticateMailAddon(req, res, next) {
                    eventlog.add(eventlog.ACTION_USER_LOGIN, { authType: 'ldap', mailboxId: email }, { userId: result.id, user: users.removePrivateFields(result) });
                    res.end();
                });
-            });
+            } else {
+                return next(new ldap.OperationsError('Unknown ownerType for mailbox'));
+            }
        });
    });
 }
@@ -547,13 +523,13 @@ function start(callback) {
    gServer.bind('ou=users,dc=cloudron', authenticateApp, authenticateUser, authorizeUserForApp);

    // http://www.ietf.org/proceedings/43/I-D/draft-srivastava-ldap-mail-00.txt
-    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch); // haraka, dovecot
-    gServer.bind('ou=mailboxes,dc=cloudron', authenticateUserMailbox); // apps like sogo can use domain=${domain} to authenticate a mailbox
-    gServer.search('ou=mailaliases,dc=cloudron', mailAliasSearch); // haraka
-    gServer.search('ou=mailinglists,dc=cloudron', mailingListSearch); // haraka
+    gServer.search('ou=mailboxes,dc=cloudron', mailboxSearch);
+    gServer.search('ou=mailaliases,dc=cloudron', mailAliasSearch);
+    gServer.search('ou=mailinglists,dc=cloudron', mailingListSearch);

-    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailAddon); // dovecot
-    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailAddon); // haraka
+    gServer.bind('ou=mailboxes,dc=cloudron', authenticateMailbox);
+    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailbox);
+    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailbox);

    gServer.compare('cn=users,ou=groups,dc=cloudron', authenticateApp, groupUsersCompare);
    gServer.compare('cn=admins,ou=groups,dc=cloudron', authenticateApp, groupAdminsCompare);
@@ -18,6 +18,7 @@ Locker.prototype.OP_BOX_UPDATE = 'box_update';
 Locker.prototype.OP_PLATFORM_START = 'platform_start';
 Locker.prototype.OP_FULL_BACKUP = 'full_backup';
 Locker.prototype.OP_APPTASK = 'apptask';
+Locker.prototype.OP_MIGRATE = 'migrate';

 Locker.prototype.lock = function (operation) {
    assert.strictEqual(typeof operation, 'string');
@@ -22,8 +22,6 @@ function collectLogs(unitName, callback) {
    assert.strictEqual(typeof callback, 'function');

    var logs = safe.child_process.execSync('sudo ' + COLLECT_LOGS_CMD + ' ' + unitName, { encoding: 'utf8' });
-    if (!logs) return callback(safe.error);
-
    logs = logs + '\n\n=====================================\n\n';

    callback(null, logs);
@@ -8,7 +8,6 @@ exports = module.exports = {
    getDomain: getDomain,
    addDomain: addDomain,
    removeDomain: removeDomain,
-    clearDomains: clearDomains,

    setDnsRecords: setDnsRecords,

@@ -23,11 +22,11 @@ exports = module.exports = {

    sendTestMail: sendTestMail,

-    listMailboxes: listMailboxes,
+    getMailboxes: getMailboxes,
    removeMailboxes: removeMailboxes,
    getMailbox: getMailbox,
    addMailbox: addMailbox,
-    updateMailboxOwner: updateMailboxOwner,
+    updateMailbox: updateMailbox,
    removeMailbox: removeMailbox,

    listAliases: listAliases,
@@ -52,14 +51,13 @@ var assert = require('assert'),
    debug = require('debug')('box:mail'),
    dns = require('./native-dns.js'),
    domains = require('./domains.js'),
-    eventlog = require('./eventlog.js'),
-    hat = require('./hat.js'),
    infra = require('./infra_version.js'),
    mailboxdb = require('./mailboxdb.js'),
    maildb = require('./maildb.js'),
    mailer = require('./mailer.js'),
    net = require('net'),
    nodemailer = require('nodemailer'),
+    os = require('os'),
    path = require('path'),
    paths = require('./paths.js'),
    reverseProxy = require('./reverseproxy.js'),
@@ -110,6 +108,9 @@ function validateName(name) {
    // also need to consider valid LDAP characters here (e.g '+' is reserved)
    if (/[^a-zA-Z0-9.-]/.test(name)) return new MailError(MailError.BAD_FIELD, 'mailbox name can only contain alphanumerals and dot');

+    // app emails are sent using the .app suffix
+    if (name.indexOf('.app') !== -1) return new MailError(MailError.BAD_FIELD, 'mailbox name pattern is reserved for apps');
+
    return null;
 }

@@ -552,8 +553,7 @@ function restartMail(callback) {
    if (process.env.BOX_ENV === 'test' && !process.env.TEST_CREATE_INFRA) return callback();

    const tag = infra.images.mail.tag;
-    const memoryLimit = 4 * 256;
-    const cloudronToken = hat(8 * 128);
+    const memoryLimit = Math.max((1 + Math.round(os.totalmem()/(1024*1024*1024)/4)) * 128, 256);

    // admin and mail share the same certificate
    reverseProxy.getCertificate({ fqdn: config.adminFqdn(), domain: config.adminDomain() }, function (error, bundle) {
@@ -566,35 +566,33 @@ function restartMail(callback) {
        if (!safe.child_process.execSync(`cp ${bundle.certFilePath} ${mailCertFilePath}`)) return callback(new Error('Could not create cert file:' + safe.error.message));
        if (!safe.child_process.execSync(`cp ${bundle.keyFilePath} ${mailKeyFilePath}`)) return callback(new Error('Could not create key file:' + safe.error.message));

-        shell.exec('startMail', 'docker rm -f mail || true', function (error) {
+        shell.execSync('startMail', 'docker rm -f mail || true');
+
+        createMailConfig(function (error, allowInbound) {
            if (error) return callback(error);

-            createMailConfig(function (error, allowInbound) {
-                if (error) return callback(error);
+            var ports = allowInbound ? '-p 587:2525 -p 993:9993 -p 4190:4190 -p 25:2525' : '';

-                var ports = allowInbound ? '-p 587:2525 -p 993:9993 -p 4190:4190 -p 25:2525' : '';
+            const cmd = `docker run --restart=always -d --name="mail" \
+                        --net cloudron \
+                        --net-alias mail \
+                        --log-driver syslog \
+                        --log-opt syslog-address=udp://127.0.0.1:2514 \
+                        --log-opt syslog-format=rfc5424 \
+                        --log-opt tag=mail \
+                        -m ${memoryLimit}m \
+                        --memory-swap ${memoryLimit * 2}m \
+                        --dns 172.18.0.1 \
+                        --dns-search=. \
+                        -v "${paths.MAIL_DATA_DIR}:/app/data" \
+                        -v "${paths.PLATFORM_DATA_DIR}/addons/mail:/etc/mail" \
+                        ${ports} \
+                        -p 127.0.0.1:2020:2020 \
+                        --read-only -v /run -v /tmp ${tag}`;

-                const cmd = `docker run --restart=always -d --name="mail" \
-                            --net cloudron \
-                            --net-alias mail \
-                            --log-driver syslog \
-                            --log-opt syslog-address=udp://127.0.0.1:2514 \
-                            --log-opt syslog-format=rfc5424 \
-                            --log-opt tag=mail \
-                            -m ${memoryLimit}m \
-                            --memory-swap ${memoryLimit * 2}m \
-                            --dns 172.18.0.1 \
-                            --dns-search=. \
-                            -e CLOUDRON_MAIL_TOKEN="${cloudronToken}" \
-                            -v "${paths.MAIL_DATA_DIR}:/app/data" \
-                            -v "${paths.PLATFORM_DATA_DIR}/addons/mail:/etc/mail" \
-                            ${ports} \
-                            -p 127.0.0.1:2020:2020 \
-                            --label isCloudronManaged=true \
-                            --read-only -v /run -v /tmp ${tag}`;
+            shell.execSync('startMail', cmd);

-                shell.exec('startMail', cmd, callback);
-            });
+            callback();
        });
    });
 }
@@ -602,9 +600,9 @@ function restartMail(callback) {
 function restartMailIfActivated(callback) {
    assert.strictEqual(typeof callback, 'function');

-    users.isActivated(function (error, activated) {
+    users.count(function (error, count) {
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));
-        if (!activated) {
+        if (!count) {
            debug('restartMailIfActivated: skipping restart of mail container since Cloudron is not activated yet');
            return callback(); // not provisioned yet, do not restart container after dns setup
        }
@@ -628,7 +626,7 @@ function getDomain(domain, callback) {
 function getDomains(callback) {
    assert.strictEqual(typeof callback, 'function');

-    maildb.list(function (error, results) {
+    maildb.getAll(function (error, results) {
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

        return callback(null, results);
@@ -745,24 +743,24 @@ function setDnsRecords(domain, callback) {
            records.push({ subdomain: '', domain: domain, type: 'MX', values: [ '10 ' + config.mailFqdn() + '.' ] });
        }

-        debug('setDnsRecords: %j', records);
+        debug('addDnsRecords: %j', records);

        txtRecordsWithSpf(domain, function (error, txtRecords) {
            if (error) return callback(error);

            if (txtRecords) records.push({ subdomain: '', domain: domain, type: 'TXT', values: txtRecords });

-            debug('setDnsRecords: will update %j', records);
+            debug('addDnsRecords: will update %j', records);

            async.mapSeries(records, function (record, iteratorCallback) {
                domains.upsertDnsRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
            }, function (error, changeIds) {
                if (error) {
-                    debug(`setDnsRecords: failed to update: ${error}`);
+                    debug(`addDnsRecords: failed to update: ${error}`);
                    return callback(new MailError(MailError.EXTERNAL_ERROR, error.message));
                }

-                debug('setDnsRecords: records %j added with changeIds %j', records, changeIds);
+                debug('addDnsRecords: records %j added with changeIds %j', records, changeIds);

                callback(null);
            });
@@ -805,16 +803,6 @@ function removeDomain(domain, callback) {
    });
 }

-function clearDomains(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    maildb.clear(function (error) {
-        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));
-
-        callback();
-    });
-}
-
 function setMailFromValidation(domain, enabled, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof enabled, 'boolean');
@@ -864,10 +852,9 @@ function setMailRelay(domain, relay, callback) {
    });
 }

-function setMailEnabled(domain, enabled, auditSource, callback) {
+function setMailEnabled(domain, enabled, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof enabled, 'boolean');
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    maildb.update(domain, { enabled: enabled }, function (error) {
@@ -876,8 +863,6 @@ function setMailEnabled(domain, enabled, auditSource, callback) {

        restartMail(NOOP_CALLBACK);

-        eventlog.add(enabled ? eventlog.ACTION_MAIL_ENABLED : eventlog.ACTION_MAIL_DISABLED, auditSource, { domain });
-
        callback(null);
    });
 }
@@ -896,7 +881,7 @@ function sendTestMail(domain, to, callback) {
    });
 }

-function listMailboxes(domain, callback) {
+function getMailboxes(domain, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

@@ -931,11 +916,10 @@ function getMailbox(name, domain, callback) {
    });
 }

-function addMailbox(name, domain, userId, auditSource, callback) {
+function addMailbox(name, domain, userId, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof userId, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    name = name.toLowerCase();
@@ -943,17 +927,15 @@ function addMailbox(name, domain, userId, auditSource, callback) {
    var error = validateName(name);
    if (error) return callback(error);

-    mailboxdb.addMailbox(name, domain, userId, function (error) {
+    mailboxdb.addMailbox(name, domain, userId, mailboxdb.OWNER_TYPE_USER, function (error) {
        if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new MailError(MailError.ALREADY_EXISTS, `mailbox ${name} already exists`));
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_ADD, auditSource, { name, domain, userId });
-
        callback(null);
    });
 }

-function updateMailboxOwner(name, domain, userId, callback) {
+function updateMailbox(name, domain, userId, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof userId, 'string');
@@ -961,7 +943,10 @@ function updateMailboxOwner(name, domain, userId, callback) {

    name = name.toLowerCase();

-    mailboxdb.updateMailboxOwner(name, domain, userId, function (error) {
+    var error = validateName(name);
+    if (error) return callback(error);
+
+    mailboxdb.updateMailbox(name, domain, userId, mailboxdb.OWNER_TYPE_USER, function (error) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new MailError(MailError.NOT_FOUND, 'no such mailbox'));
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

@@ -969,18 +954,15 @@ function updateMailboxOwner(name, domain, userId, callback) {
    });
 }

-function removeMailbox(name, domain, auditSource, callback) {
+function removeMailbox(name, domain, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    mailboxdb.del(name, domain, function (error) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new MailError(MailError.NOT_FOUND, 'no such mailbox'));
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

-        eventlog.add(eventlog.ACTION_MAIL_MAILBOX_REMOVE, auditSource, { name, domain });
-
        callback(null);
    });
 }
@@ -1065,11 +1047,10 @@ function getList(domain, listName, callback) {
    });
 }

-function addList(name, domain, members, auditSource, callback) {
+function addList(name, domain, members, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof name, 'string');
    assert(Array.isArray(members));
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    name = name.toLowerCase();
@@ -1088,8 +1069,6 @@ function addList(name, domain, members, auditSource, callback) {
        if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new MailError(MailError.ALREADY_EXISTS, 'list already exits'));
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

-        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain });
-
        callback();
    });
 }
@@ -1120,18 +1099,15 @@ function updateList(name, domain, members, callback) {
    });
 }

-function removeList(name, domain, auditSource, callback) {
-    assert.strictEqual(typeof name, 'string');
+function removeList(domain, listName, callback) {
    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof listName, 'string');
    assert.strictEqual(typeof callback, 'function');

-    mailboxdb.del(name, domain, function (error) {
+    mailboxdb.del(listName, domain, function (error) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new MailError(MailError.NOT_FOUND, 'no such list'));
        if (error) return callback(new MailError(MailError.INTERNAL_ERROR, error));

-        eventlog.add(eventlog.ACTION_MAIL_LIST_ADD, auditSource, { name, domain });
-
        callback();
    });
 }
@@ -4,7 +4,7 @@ exports = module.exports = {
    addMailbox: addMailbox,
    addGroup: addGroup,

-    updateMailboxOwner: updateMailboxOwner,
+    updateMailbox: updateMailbox,
    updateList: updateList,
    del: del,

@@ -29,7 +29,11 @@ exports = module.exports = {

    TYPE_MAILBOX: 'mailbox',
    TYPE_LIST: 'list',
-    TYPE_ALIAS: 'alias'
+    TYPE_ALIAS: 'alias',
+
+    OWNER_TYPE_USER: 'user',
+    OWNER_TYPE_APP: 'app',
+    OWNER_TYPE_GROUP: 'group' // obsolete
 };

 var assert = require('assert'),
@@ -38,7 +42,7 @@ var assert = require('assert'),
    safe = require('safetydance'),
    util = require('util');

-var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'aliasTarget', 'creationTime', 'membersJson', 'domain' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'type', 'ownerId', 'ownerType', 'aliasTarget', 'creationTime', 'membersJson', 'domain' ].join(',');

 function postProcess(data) {
    data.members = safe.JSON.parse(data.membersJson) || [ ];
@@ -47,13 +51,14 @@ function postProcess(data) {
    return data;
 }

-function addMailbox(name, domain, ownerId, callback) {
+function addMailbox(name, domain, ownerId, ownerType, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId) VALUES (?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType) VALUES (?, ?, ?, ?, ?)', [ name, exports.TYPE_MAILBOX, domain, ownerId, ownerType ], function (error) {
        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'mailbox already exists'));
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

@@ -61,13 +66,14 @@ function addMailbox(name, domain, ownerId, callback) {
    });
 }

-function updateMailboxOwner(name, domain, ownerId, callback) {
+function updateMailbox(name, domain, ownerId, ownerType, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof ownerType, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('UPDATE mailboxes SET ownerId = ? WHERE name = ? AND domain = ?', [ ownerId, name, domain ], function (error, result) {
+    database.query('UPDATE mailboxes SET ownerId = ? WHERE name = ? AND domain = ? AND ownerType = ?', [ ownerId, name, domain, ownerType ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (result.affectedRows === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -81,8 +87,8 @@ function addGroup(name, domain, members, callback) {
    assert(Array.isArray(members));
    assert.strictEqual(typeof callback, 'function');

-    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, membersJson) VALUES (?, ?, ?, ?, ?)',
-        [ name, exports.TYPE_LIST, domain, 'admin', JSON.stringify(members) ], function (error) {
+    database.query('INSERT INTO mailboxes (name, type, domain, ownerId, ownerType, membersJson) VALUES (?, ?, ?, ?, ?, ?)',
+        [ name, exports.TYPE_LIST, domain, 'admin', exports.OWNER_TYPE_GROUP, JSON.stringify(members) ], function (error) {
            if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'mailbox already exists'));
            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

@@ -253,8 +259,8 @@ function setAliasesForName(name, domain, aliases, callback) {
        // clear existing aliases
        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasTarget = ? AND domain = ? AND type = ?', args: [ name, domain, exports.TYPE_ALIAS ] });
        aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, type, domain, aliasTarget, ownerId) VALUES (?, ?, ?, ?, ?)',
-                args: [ alias, exports.TYPE_ALIAS, domain, name, results[0].ownerId ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, type, domain, aliasTarget, ownerId, ownerType) VALUES (?, ?, ?, ?, ?, ?)',
+                args: [ alias, exports.TYPE_ALIAS, domain, name, results[0].ownerId, results[0].ownerType ] });
        });

        database.transaction(queries, function (error) {
@@ -4,10 +4,10 @@ exports = module.exports = {
    add: add,
    del: del,
    get: get,
-    list: list,
+    getAll: getAll,
    update: update,

-    clear: clear,
+    _clear: clear,

    TYPE_USER: 'user',
    TYPE_APP: 'app',
@@ -49,8 +49,7 @@ function add(domain, callback) {
 function clear(callback) {
    assert.strictEqual(typeof callback, 'function');

-    // using TRUNCATE makes it fail foreign key check
-    database.query('DELETE FROM mail', [], function (error) {
+    database.query('TRUNCATE TABLE mail', [], function (error) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        callback(null);
    });
@@ -82,7 +81,7 @@ function get(domain, callback) {
    });
 }

-function list(callback) {
+function getAll(callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + MAILDB_FIELDS + ' FROM mail ORDER BY domain', function (error, results) {
@@ -7,6 +7,7 @@ var config = require('./config.js'),
 exports = module.exports = {
    CLOUDRON_DEFAULT_AVATAR_FILE: path.join(__dirname + '/../assets/avatar.png'),
    INFRA_VERSION_FILE: path.join(config.baseDir(), 'platformdata/INFRA_VERSION'),
+    BACKUP_RESULT_FILE: path.join(config.baseDir(), 'platformdata/backup/result.txt'),

    OLD_DATA_DIR: path.join(config.baseDir(), 'data'),
    PLATFORM_DATA_DIR: path.join(config.baseDir(), 'platformdata'),
@@ -32,10 +33,9 @@ exports = module.exports = {
    CLOUDRON_AVATAR_FILE: path.join(config.baseDir(), 'boxdata/avatar.png'),
    UPDATE_CHECKER_FILE: path.join(config.baseDir(), 'boxdata/updatechecker.json'),

-    LOG_DIR: path.join(config.baseDir(), 'platformdata/logs'),
-    TASKS_LOG_DIR: path.join(config.baseDir(), 'platformdata/logs/tasks'),
+    AUTO_PROVISION_FILE: path.join(config.baseDir(), 'configs/autoprovision.json'),

+    LOG_DIR: path.join(config.baseDir(), 'platformdata/logs'),
    // this pattern is for the cloudron logs API route to work
    BACKUP_LOG_FILE: path.join(config.baseDir(), 'platformdata/logs/backup/app.log'),
-    UPDATER_LOG_FILE: path.join(config.baseDir(), 'platformdata/logs/updater/app.log')
 };
@@ -4,31 +4,32 @@ exports = module.exports = {
    start: start,
    stop: stop,

-    handleCertChanged: handleCertChanged,
-
-    // exported for testing
-    _isReady: false
+    handleCertChanged: handleCertChanged
 };

-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
+var apps = require('./apps.js'),
    assert = require('assert'),
    async = require('async'),
    config = require('./config.js'),
    debug = require('debug')('box:platform'),
    fs = require('fs'),
-    graphs = require('./graphs.js'),
+    hat = require('./hat.js'),
    infra = require('./infra_version.js'),
    locker = require('./locker.js'),
    mail = require('./mail.js'),
+    os = require('os'),
    paths = require('./paths.js'),
    reverseProxy = require('./reverseproxy.js'),
    safe = require('safetydance'),
+    semver = require('semver'),
    settings = require('./settings.js'),
    shell = require('./shell.js'),
    taskmanager = require('./taskmanager.js'),
+    util = require('util'),
    _ = require('underscore');

+var gPlatformReadyTimer = null;
+
 var NOOP_CALLBACK = function (error) { if (error) debug(error); };

 function start(callback) {
@@ -44,11 +45,13 @@ function start(callback) {
        if (!existingInfra) existingInfra = { version: 'corrupt' };
    }

+    settings.events.on(settings.PLATFORM_CONFIG_KEY, updateAddons);
+
    // short-circuit for the restart case
    if (_.isEqual(infra, existingInfra)) {
        debug('platform is uptodate at version %s', infra.version);

-        onPlatformReady();
+        emitPlatformReady();

        return callback();
    }
@@ -60,121 +63,261 @@ function start(callback) {

    async.series([
        stopContainers.bind(null, existingInfra),
-        // mark app state before we start addons. this gives the db import logic a chance to mark an app as errored
+        startAddons.bind(null, existingInfra),
+        removeOldImages,
        startApps.bind(null, existingInfra),
-        graphs.startGraphite.bind(null, existingInfra),
-        addons.startServices.bind(null, existingInfra),
        fs.writeFile.bind(fs, paths.INFRA_VERSION_FILE, JSON.stringify(infra, null, 4))
    ], function (error) {
        if (error) return callback(error);

        locker.unlock(locker.OP_PLATFORM_START);

-        onPlatformReady();
+        emitPlatformReady();

        callback();
    });
 }

 function stop(callback) {
+    clearTimeout(gPlatformReadyTimer);
+    gPlatformReadyTimer = null;
+    exports.events = null;
    taskmanager.pauseTasks(callback);
 }

-function onPlatformReady() {
-    debug('onPlatformReady: platform is ready');
-    exports._isReady = true;
-    taskmanager.resumeTasks();
+function updateAddons(platformConfig, callback) {
+    callback = callback || NOOP_CALLBACK;

-    applyPlatformConfig(NOOP_CALLBACK);
-    pruneInfraImages(NOOP_CALLBACK);
-}
+    // TODO: this should possibly also rollback memory to default
+    async.eachSeries([ 'mysql', 'postgresql', 'mail', 'mongodb' ], function iterator(containerName, iteratorCallback) {
+        const containerConfig = platformConfig[containerName];
+        if (!containerConfig) return iteratorCallback();

-function applyPlatformConfig(callback) {
-    // scale back db containers, if possible. this is retried because updating memory constraints can fail
-    // with failed to write to memory.memsw.limit_in_bytes: write /sys/fs/cgroup/memory/docker/xx/memory.memsw.limit_in_bytes: device or resource busy
+        if (!containerConfig.memory || !containerConfig.memorySwap) return iteratorCallback();

-    async.retry({ times: 10, interval: 5 * 60 * 1000 }, function (retryCallback) {
-        settings.getPlatformConfig(function (error, platformConfig) {
-            if (error) return retryCallback(error);
-
-            addons.updateServiceConfig(platformConfig, function (error) {
-                if (error) debug('Error updating services. Will rety in 5 minutes', platformConfig, error);
-
-                retryCallback(error);
-            });
-        });
+        const args = `update --memory ${containerConfig.memory} --memory-swap ${containerConfig.memorySwap} ${containerName}`.split(' ');
+        shell.exec(`update${containerName}`, '/usr/bin/docker', args, { }, iteratorCallback);
    }, callback);
 }

-function pruneInfraImages(callback) {
-    debug('pruneInfraImages: checking existing images');
+function emitPlatformReady() {
+    // give some time for the platform to "settle". For example, mysql might still be initing the
+    // database dir and we cannot call service scripts until that's done.
+    // TODO: make this smarter to not wait for 15secs for the crash-restart case
+    gPlatformReadyTimer = setTimeout(function () {
+        debug('emitting platform ready');
+        gPlatformReadyTimer = null;
+        taskmanager.resumeTasks();
+    }, 15000);
+}

-    // cannot blindly remove all unused images since redis image may not be used
-    const images = infra.baseImages.concat(Object.keys(infra.images).map(function (addon) { return infra.images[addon]; }));
+function removeOldImages(callback) {
+    debug('removing old addon images');

-    async.eachSeries(images, function (image, iteratorCallback) {
-        let output = safe.child_process.execSync(`docker images --digests ${image.repo} --format "{{.ID}} {{.Repository}}:{{.Tag}}@{{.Digest}}"`, { encoding: 'utf8' });
-        if (output === null) return iteratorCallback(safe.error);
+    for (var imageName in infra.images) {
+        if (imageName === 'redis') continue; // see #223
+        var image = infra.images[imageName];
+        debug('cleaning up images of %j', image);
+        var cmd = 'docker images "%s" | tail -n +2 | awk \'{ print $1 ":" $2 }\' | grep -v "%s" | xargs --no-run-if-empty docker rmi';
+        shell.execSync('removeOldImagesSync', util.format(cmd, image.repo, image.tag));
+    }

-        let lines = output.trim().split('\n');
-        for (let line of lines) {
-            if (!line) continue;
-            let parts = line.split(' '); // [ ID, Repo:Tag@Digest ]
-            if (image.tag === parts[1]) continue; // keep
-            debug(`pruneInfraImages: removing unused image of ${image.repo}: ${line}`);
-
-            shell.exec('pruneInfraImages', `docker rmi ${parts[0]}`, iteratorCallback);
-        }
-    }, callback);
+    callback();
 }

 function stopContainers(existingInfra, callback) {
+    // TODO: be nice and stop addons cleanly (example, shutdown commands)
+
    // always stop addons to restart them on any infra change, regardless of minor or major update
    if (existingInfra.version !== infra.version) {
-        // TODO: only nuke containers with isCloudronManaged=true
        debug('stopping all containers for infra upgrade');
-        async.series([
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa | xargs --no-run-if-empty docker stop'),
-            shell.exec.bind(null, 'stopContainers', 'docker ps -qa | xargs --no-run-if-empty docker rm -f')
-        ], callback);
+        shell.execSync('stopContainers', 'docker ps -qa | xargs --no-run-if-empty docker rm -f');
    } else {
        assert(typeof infra.images, 'object');
        var changedAddons = [ ];
        for (var imageName in infra.images) {
+            if (imageName === 'redis') continue; // see #223
            if (infra.images[imageName].tag !== existingInfra.images[imageName].tag) changedAddons.push(imageName);
        }

-        debug('stopContainer: stopping addons for incremental infra update: %j', changedAddons);
-        let filterArg = changedAddons.map(function (c) { return `--filter 'name=${c}'`; }).join(' '); // name=c matches *c*. required for redis-{appid}
+        debug('stopping addons for incremental infra update: %j', changedAddons);
        // ignore error if container not found (and fail later) so that this code works across restarts
-        async.series([
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} | xargs --no-run-if-empty docker stop || true`),
-            shell.exec.bind(null, 'stopContainers', `docker ps -qa ${filterArg} | xargs --no-run-if-empty docker rm -f || true`)
-        ], callback);
+        shell.execSync('stopContainers', 'docker rm -f ' + changedAddons.join(' ') + ' || true');
    }
-}
-
-function startApps(existingInfra, callback) {
-    if (existingInfra.version === 'none') { // cloudron is being restored from backup
-        debug('startApps: restoring installed apps');
-        apps.restoreInstalledApps(callback);
-    } else if (existingInfra.version !== infra.version) {
-        debug('startApps: reconfiguring installed apps');
-        reverseProxy.removeAppConfigs(); // should we change the cert location, nginx will not start
-        apps.configureInstalledApps(callback);
-    } else {
-        debug('startApps: apps are already uptodate');
-        callback();
-    }
-}
-
-function handleCertChanged(cn, callback) {
-    assert.strictEqual(typeof cn, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('handleCertChanged', cn);
-
-    if (cn === '*.' + config.adminDomain() || cn === config.adminFqdn()) return mail.startMail(callback);

    callback();
 }
+
+function startGraphite(callback) {
+    const tag = infra.images.graphite.tag;
+    const dataDir = paths.PLATFORM_DATA_DIR;
+
+    const cmd = `docker run --restart=always -d --name="graphite" \
+                --net cloudron \
+                --net-alias graphite \
+                --log-driver syslog \
+                --log-opt syslog-address=udp://127.0.0.1:2514 \
+                --log-opt syslog-format=rfc5424 \
+                --log-opt tag=graphite \
+                -m 75m \
+                --memory-swap 150m \
+                --dns 172.18.0.1 \
+                --dns-search=. \
+                -p 127.0.0.1:2003:2003 \
+                -p 127.0.0.1:2004:2004 \
+                -p 127.0.0.1:8000:8000 \
+                -v "${dataDir}/graphite:/app/data" \
+                --read-only -v /tmp -v /run "${tag}"`;
+
+    shell.execSync('startGraphite', cmd);
+
+    callback();
+}
+
+function startMysql(callback) {
+    const tag = infra.images.mysql.tag;
+    const dataDir = paths.PLATFORM_DATA_DIR;
+    const rootPassword = hat(8 * 128);
+    const memoryLimit = (1 + Math.round(os.totalmem()/(1024*1024*1024)/4)) * 256;
+
+    if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/mysql_vars.sh',
+        'MYSQL_ROOT_PASSWORD=' + rootPassword +'\nMYSQL_ROOT_HOST=172.18.0.1', 'utf8')) {
+        return callback(new Error('Could not create mysql var file:' + safe.error.message));
+    }
+
+    const cmd = `docker run --restart=always -d --name="mysql" \
+                --net cloudron \
+                --net-alias mysql \
+                --log-driver syslog \
+                --log-opt syslog-address=udp://127.0.0.1:2514 \
+                --log-opt syslog-format=rfc5424 \
+                --log-opt tag=mysql \
+                -m ${memoryLimit}m \
+                --memory-swap ${memoryLimit * 2}m \
+                --dns 172.18.0.1 \
+                --dns-search=. \
+                -v "${dataDir}/mysql:/var/lib/mysql" \
+                -v "${dataDir}/addons/mysql_vars.sh:/etc/mysql/mysql_vars.sh:ro" \
+                --read-only -v /tmp -v /run "${tag}"`;
+
+    shell.execSync('startMysql', cmd);
+
+    setTimeout(callback, 5000);
+}
+
+function startPostgresql(callback) {
+    const tag = infra.images.postgresql.tag;
+    const dataDir = paths.PLATFORM_DATA_DIR;
+    const rootPassword = hat(8 * 128);
+    const memoryLimit = (1 + Math.round(os.totalmem()/(1024*1024*1024)/4)) * 256;
+
+    if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/postgresql_vars.sh', 'POSTGRESQL_ROOT_PASSWORD=' + rootPassword, 'utf8')) {
+        return callback(new Error('Could not create postgresql var file:' + safe.error.message));
+    }
+
+    const cmd = `docker run --restart=always -d --name="postgresql" \
+                --net cloudron \
+                --net-alias postgresql \
+                --log-driver syslog \
+                --log-opt syslog-address=udp://127.0.0.1:2514 \
+                --log-opt syslog-format=rfc5424 \
+                --log-opt tag=postgresql \
+                -m ${memoryLimit}m \
+                --memory-swap ${memoryLimit * 2}m \
+                --dns 172.18.0.1 \
+                --dns-search=. \
+                -v "${dataDir}/postgresql:/var/lib/postgresql" \
+                -v "${dataDir}/addons/postgresql_vars.sh:/etc/postgresql/postgresql_vars.sh:ro" \
+                --read-only -v /tmp -v /run "${tag}"`;
+
+    shell.execSync('startPostgresql', cmd);
+
+    setTimeout(callback, 5000);
+}
+
+function startMongodb(callback) {
+    const tag = infra.images.mongodb.tag;
+    const dataDir = paths.PLATFORM_DATA_DIR;
+    const rootPassword = hat(8 * 128);
+    const memoryLimit = (1 + Math.round(os.totalmem()/(1024*1024*1024)/4)) * 200;
+
+    if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/mongodb_vars.sh', 'MONGODB_ROOT_PASSWORD=' + rootPassword, 'utf8')) {
+        return callback(new Error('Could not create mongodb var file:' + safe.error.message));
+    }
+
+    const cmd = `docker run --restart=always -d --name="mongodb" \
+                --net cloudron \
+                --net-alias mongodb \
+                --log-driver syslog \
+                --log-opt syslog-address=udp://127.0.0.1:2514 \
+                --log-opt syslog-format=rfc5424 \
+                --log-opt tag=mongodb \
+                -m ${memoryLimit}m \
+                --memory-swap ${memoryLimit * 2}m \
+                --dns 172.18.0.1 \
+                --dns-search=. \
+                -v "${dataDir}/mongodb:/var/lib/mongodb" \
+                -v "${dataDir}/addons/mongodb_vars.sh:/etc/mongodb_vars.sh:ro" \
+                --read-only -v /tmp -v /run "${tag}"`;
+
+    shell.execSync('startMongodb', cmd);
+
+    setTimeout(callback, 5000);
+}
+
+function startAddons(existingInfra, callback) {
+    var startFuncs = [ ];
+
+    // always start addons on any infra change, regardless of minor or major update
+    if (existingInfra.version !== infra.version) {
+        debug('startAddons: no existing infra or infra upgrade. starting all addons');
+        startFuncs.push(startGraphite, startMysql, startPostgresql, startMongodb, mail.startMail);
+    } else {
+        assert.strictEqual(typeof existingInfra.images, 'object');
+
+        if (infra.images.graphite.tag !== existingInfra.images.graphite.tag) startFuncs.push(startGraphite);
+        if (infra.images.mysql.tag !== existingInfra.images.mysql.tag) startFuncs.push(startMysql);
+        if (infra.images.postgresql.tag !== existingInfra.images.postgresql.tag) startFuncs.push(startPostgresql);
+        if (infra.images.mongodb.tag !== existingInfra.images.mongodb.tag) startFuncs.push(startMongodb);
+        if (infra.images.mail.tag !== existingInfra.images.mail.tag) startFuncs.push(mail.startMail);
+
+        debug('startAddons: existing infra. incremental addon create %j', startFuncs.map(function (f) { return f.name; }));
+    }
+
+    async.series(startFuncs, function (error) {
+        if (error) return callback(error);
+
+        settings.getPlatformConfig(function (error, platformConfig) {
+            if (error) return callback(error);
+
+            updateAddons(platformConfig, callback);
+        });
+    });
+}
+
+function startApps(existingInfra, callback) {
+    // Infra version change strategy:
+    // * no existing version - restore apps
+    // * major versions - restore apps
+    // * minor versions - reconfigure apps
+
+    if (existingInfra.version === infra.version) {
+        debug('startApp: apps are already uptodate');
+        callback();
+    } else if (existingInfra.version === 'none' || !semver.valid(existingInfra.version) || semver.major(existingInfra.version) !== semver.major(infra.version)) {
+        debug('startApps: restoring installed apps');
+        apps.restoreInstalledApps(callback);
+    } else {
+        debug('startApps: reconfiguring installed apps');
+        reverseProxy.removeAppConfigs(); // should we change the cert location, nginx will not start
+        apps.configureInstalledApps(callback);
+    }
+}
+
+function handleCertChanged(cn) {
+    assert.strictEqual(typeof cn, 'string');
+
+    if (cn === '*.' + config.adminDomain() || cn === config.adminFqdn()) {
+        mail.startMail(NOOP_CALLBACK);
+    }
+}
@@ -0,0 +1,59 @@
+'use strict';
+
+exports = module.exports = {
+    set: set,
+    setDetail: setDetail,
+    clear: clear,
+    getAll: getAll,
+
+    UPDATE: 'update',
+    BACKUP: 'backup',
+    MIGRATE: 'migrate'
+};
+
+var assert = require('assert'),
+    debug = require('debug')('box:progress');
+
+// if progress.update or progress.backup are object, they will contain 'percent' and 'message' properties
+// otherwise no such operation is currently ongoing
+var progress = {
+    update: null,
+    backup: null,
+    migrate: null
+};
+
+// We use -1 for percentage to indicate errors
+function set(tag, percent, message) {
+    assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof percent, 'number');
+    assert.strictEqual(typeof message, 'string');
+
+    progress[tag] = {
+        percent: percent,
+        message: message,
+        detail: ''
+    };
+
+    debug('%s: %s %s', tag, percent, message);
+}
+
+function setDetail(tag, detail) {
+    assert.strictEqual(typeof tag, 'string');
+    assert.strictEqual(typeof detail, 'string');
+
+    if (!progress[tag]) return debug('[%s] %s', tag, detail);
+
+    progress[tag].detail = detail;
+}
+
+function clear(tag) {
+    assert.strictEqual(typeof tag, 'string');
+
+    progress[tag] = null;
+
+    debug('clearing %s', tag);
+}
+
+function getAll() {
+    return progress;
+}
@@ -1,268 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    setup: setup,
-    restore: restore,
-    activate: activate,
-
-    ProvisionError: ProvisionError
-};
-
-var assert = require('assert'),
-    async = require('async'),
-    backups = require('./backups.js'),
-    BackupsError = require('./backups.js').BackupsError,
-    config = require('./config.js'),
-    constants = require('./constants.js'),
-    clients = require('./clients.js'),
-    cloudron = require('./cloudron.js'),
-    debug = require('debug')('box:provision'),
-    domains = require('./domains.js'),
-    DomainsError = domains.DomainsError,
-    eventlog = require('./eventlog.js'),
-    mail = require('./mail.js'),
-    path = require('path'),
-    safe = require('safetydance'),
-    semver = require('semver'),
-    settingsdb = require('./settingsdb.js'),
-    settings = require('./settings.js'),
-    shell = require('./shell.js'),
-    superagent = require('superagent'),
-    users = require('./users.js'),
-    UsersError = users.UsersError,
-    tld = require('tldjs'),
-    util = require('util');
-
-var RESTART_CMD = path.join(__dirname, 'scripts/restart.sh');
-
-var NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
-function ProvisionError(reason, errorOrMessage) {
-    assert.strictEqual(typeof reason, 'string');
-    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
-
-    Error.call(this);
-    Error.captureStackTrace(this, this.constructor);
-
-    this.name = this.constructor.name;
-    this.reason = reason;
-    if (typeof errorOrMessage === 'undefined') {
-        this.message = reason;
-    } else if (typeof errorOrMessage === 'string') {
-        this.message = errorOrMessage;
-    } else {
-        this.message = 'Internal error';
-        this.nestedError = errorOrMessage;
-    }
-}
-util.inherits(ProvisionError, Error);
-ProvisionError.BAD_FIELD = 'Field error';
-ProvisionError.BAD_STATE = 'Bad State';
-ProvisionError.ALREADY_SETUP = 'Already Setup';
-ProvisionError.INTERNAL_ERROR = 'Internal Error';
-ProvisionError.EXTERNAL_ERROR = 'External Error';
-ProvisionError.ALREADY_PROVISIONED = 'Already Provisioned';
-
-function autoprovision(autoconf, callback) {
-    assert.strictEqual(typeof autoconf, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    async.eachSeries(Object.keys(autoconf), function (key, iteratorDone) {
-        debug(`autoprovision: ${key}`);
-
-        switch (key) {
-        case 'appstoreConfig':
-            if (config.provider() === 'caas') { // skip registration
-                settingsdb.set(settings.APPSTORE_CONFIG_KEY, JSON.stringify(autoconf[key]), iteratorDone);
-            } else { // register cloudron
-                settings.setAppstoreConfig(autoconf[key], iteratorDone);
-            }
-            break;
-        case 'caasConfig':
-            settingsdb.set(settings.CAAS_CONFIG_KEY, JSON.stringify(autoconf[key]), iteratorDone);
-            break;
-        case 'backupConfig':
-            settings.setBackupConfig(autoconf[key], iteratorDone);
-            break;
-        default:
-            debug(`autoprovision: ${key} ignored`);
-            return iteratorDone();
-        }
-    }, function (error) {
-        if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-        callback(null);
-    });
-}
-
-function unprovision(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('unprovision');
-
-    config.setAdminDomain('');
-    config.setAdminFqdn('');
-    config.setAdminLocation('my');
-
-    // TODO: also cancel any existing configureWebadmin task
-    async.series([
-        mail.clearDomains,
-        domains.clear
-    ], callback);
-}
-
-function setup(dnsConfig, autoconf, auditSource, callback) {
-    assert.strictEqual(typeof dnsConfig, 'object');
-    assert.strictEqual(typeof autoconf, 'object');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    users.isActivated(function (error, activated) {
-        if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-        if (activated) return callback(new ProvisionError(ProvisionError.ALREADY_SETUP));
-
-        unprovision(function (error) {
-            if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-            let webadminStatus = cloudron.getWebadminStatus();
-
-            if (webadminStatus.configuring || webadminStatus.restore.active) return callback(new ProvisionError(ProvisionError.BAD_STATE, 'Already restoring or configuring'));
-
-            const domain = dnsConfig.domain.toLowerCase();
-            const zoneName = dnsConfig.zoneName ? dnsConfig.zoneName : (tld.getDomain(domain) || domain);
-
-            const adminFqdn = 'my' + (dnsConfig.config.hyphenatedSubdomains ? '-' : '.') + domain;
-
-            debug(`provision: Setting up Cloudron with domain ${domain} and zone ${zoneName} using admin fqdn ${adminFqdn}`);
-
-            let data = {
-                zoneName: zoneName,
-                provider: dnsConfig.provider,
-                config: dnsConfig.config,
-                fallbackCertificate: dnsConfig.fallbackCertificate || null,
-                tlsConfig: dnsConfig.tlsConfig || { provider: 'letsencrypt-prod' }
-            };
-
-            domains.add(domain, data, auditSource, function (error) {
-                if (error && error.reason === DomainsError.BAD_FIELD) return callback(new ProvisionError(ProvisionError.BAD_FIELD, error.message));
-                if (error && error.reason === DomainsError.ALREADY_EXISTS) return callback(new ProvisionError(ProvisionError.BAD_FIELD, error.message));
-                if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-                async.series([
-                    mail.addDomain.bind(null, domain),
-                    cloudron.setDashboardDomain.bind(null, domain), // triggers task to setup my. dns/cert/reverseproxy
-                    autoprovision.bind(null, autoconf),
-                    eventlog.add.bind(null, eventlog.ACTION_PROVISION, auditSource, { })
-                ], callback);
-            });
-        });
-    });
-}
-
-function setTimeZone(ip, callback) {
-    assert.strictEqual(typeof ip, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('setTimeZone ip:%s', ip);
-
-    superagent.get('https://geolocation.cloudron.io/json').query({ ip: ip }).timeout(10 * 1000).end(function (error, result) {
-        if ((error && !error.response) || result.statusCode !== 200) {
-            debug('Failed to get geo location: %s', error.message);
-            return callback(null);
-        }
-
-        var timezone = safe.query(result.body, 'location.time_zone');
-
-        if (!timezone || typeof timezone !== 'string') {
-            debug('No timezone in geoip response : %j', result.body);
-            return callback(null);
-        }
-
-        debug('Setting timezone to ', timezone);
-
-        settings.setTimeZone(timezone, callback);
-    });
-}
-
-function activate(username, password, email, displayName, ip, auditSource, callback) {
-    assert.strictEqual(typeof username, 'string');
-    assert.strictEqual(typeof password, 'string');
-    assert.strictEqual(typeof email, 'string');
-    assert.strictEqual(typeof displayName, 'string');
-    assert.strictEqual(typeof ip, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('activating user:%s email:%s', username, email);
-
-    setTimeZone(ip, function () { }); // TODO: get this from user. note that timezone is detected based on the browser location and not the cloudron region
-
-    users.createOwner(username, password, email, displayName, auditSource, function (error, userObject) {
-        if (error && error.reason === UsersError.ALREADY_EXISTS) return callback(new ProvisionError(ProvisionError.ALREADY_PROVISIONED, 'Already activated'));
-        if (error && error.reason === UsersError.BAD_FIELD) return callback(new ProvisionError(ProvisionError.BAD_FIELD, error.message));
-        if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-        clients.addTokenByUserId('cid-webadmin', userObject.id, Date.now() + constants.DEFAULT_TOKEN_EXPIRATION, {}, function (error, result) {
-            if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-            eventlog.add(eventlog.ACTION_ACTIVATE, auditSource, { });
-
-            callback(null, {
-                userId: userObject.id,
-                token: result.accessToken,
-                expires: result.expires
-            });
-
-            setImmediate(cloudron.onActivated.bind(null, NOOP_CALLBACK)); // hack for now to not block the above http response
-        });
-    });
-}
-
-function restore(backupConfig, backupId, version, autoconf, auditSource, callback) {
-    assert.strictEqual(typeof backupConfig, 'object');
-    assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof version, 'string');
-    assert.strictEqual(typeof autoconf, 'object');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    if (!semver.valid(version)) return callback(new ProvisionError(ProvisionError.BAD_STATE, 'version is not a valid semver'));
-    if (semver.major(config.version()) !== semver.major(version) || semver.minor(config.version()) !== semver.minor(version)) return callback(new ProvisionError(ProvisionError.BAD_STATE, `Run cloudron-setup with --version ${version} to restore from this backup`));
-
-    let webadminStatus = cloudron.getWebadminStatus();
-
-    if (webadminStatus.configuring || webadminStatus.restore.active) return callback(new ProvisionError(ProvisionError.BAD_STATE, 'Already restoring or configuring'));
-
-    users.isActivated(function (error, activated) {
-        if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-        if (activated) return callback(new ProvisionError(ProvisionError.ALREADY_PROVISIONED, 'Already activated'));
-
-        backups.testConfig(backupConfig, function (error) {
-            if (error && error.reason === BackupsError.BAD_FIELD) return callback(new ProvisionError(ProvisionError.BAD_FIELD, error.message));
-            if (error && error.reason === BackupsError.EXTERNAL_ERROR) return callback(new ProvisionError(ProvisionError.EXTERNAL_ERROR, error.message));
-            if (error) return callback(new ProvisionError(ProvisionError.INTERNAL_ERROR, error));
-
-            debug(`restore: restoring from ${backupId} from provider ${backupConfig.provider} with format ${backupConfig.format}`);
-
-            webadminStatus.restore.active = true;
-            webadminStatus.restore.error = null;
-
-            callback(null); // do no block
-
-            async.series([
-                backups.restore.bind(null, backupConfig, backupId, (progress) => debug(`restore: ${progress}`)),
-                eventlog.add.bind(null, eventlog.ACTION_RESTORE, auditSource, { backupId }),
-                autoprovision.bind(null, autoconf),
-                // currently, our suggested restore flow is after a dnsSetup. The dnSetup creates DKIM keys and updates the DNS
-                // for this reason, we have to re-setup DNS after a restore so it has DKIm from the backup
-                // Once we have a 100% IP based restore, we can skip this
-                mail.setDnsRecords.bind(null, config.adminDomain()),
-                shell.sudo.bind(null, 'restart', [ RESTART_CMD ], {})
-            ], function (error) {
-                debug('restore:', error);
-                if (error) webadminStatus.restore.error = error.message;
-                webadminStatus.restore.active = false;
-            });
-        });
-    });
-}
@@ -6,15 +6,11 @@ exports = module.exports = {
    setFallbackCertificate: setFallbackCertificate,
    getFallbackCertificate: getFallbackCertificate,

-    generateFallbackCertificateSync: generateFallbackCertificateSync,
-    setAppCertificateSync: setAppCertificateSync,
-
    validateCertificate: validateCertificate,

    getCertificate: getCertificate,

    renewAll: renewAll,
-    renewCerts: renewCerts,

    configureDefaultServer: configureDefaultServer,

@@ -37,7 +33,7 @@ var acme2 = require('./cert/acme2.js'),
    config = require('./config.js'),
    constants = require('./constants.js'),
    crypto = require('crypto'),
-    debug = require('debug')('box:reverseproxy'),
+    debug = require('debug')('box:certificates'),
    domains = require('./domains.js'),
    ejs = require('ejs'),
    eventlog = require('./eventlog.js'),
@@ -55,7 +51,8 @@ var acme2 = require('./cert/acme2.js'),
    util = require('util');

 var NGINX_APPCONFIG_EJS = fs.readFileSync(__dirname + '/../setup/start/nginx/appconfig.ejs', { encoding: 'utf8' }),
-    RELOAD_NGINX_CMD = path.join(__dirname, 'scripts/reloadnginx.sh');
+    RELOAD_NGINX_CMD = path.join(__dirname, 'scripts/reloadnginx.sh'),
+    NOOP_CALLBACK = function (error) { if (error) debug(error); };

 function ReverseProxyError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
@@ -80,29 +77,33 @@ ReverseProxyError.INTERNAL_ERROR = 'Internal Error';
 ReverseProxyError.INVALID_CERT = 'Invalid certificate';
 ReverseProxyError.NOT_FOUND = 'Not Found';

-function getCertApi(domainObject, callback) {
-    assert.strictEqual(typeof domainObject, 'object');
+function getCertApi(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    if (domainObject.tlsConfig.provider === 'fallback') return callback(null, fallback, { fallback: true });
+    domains.get(domain, function (error, result) {
+        if (error) return callback(error);

-    var api = domainObject.tlsConfig.provider === 'caas' ? caas : acme2;
+        if (result.tlsConfig.provider === 'fallback') return callback(null, fallback, {});

-    var options = { prod: false, performHttpAuthorization: false, wildcard: false, email: '' };
-    if (domainObject.tlsConfig.provider !== 'caas') { // matches 'le-prod' or 'letsencrypt-prod'
-        options.prod = domainObject.tlsConfig.provider.match(/.*-prod/) !== null;
-        options.performHttpAuthorization = domainObject.provider.match(/noop|manual|wildcard/) !== null;
-        options.wildcard = !!domainObject.tlsConfig.wildcard;
-    }
+        var api = result.tlsConfig.provider === 'caas' ? caas : acme2;

-    // registering user with an email requires A or MX record (https://github.com/letsencrypt/boulder/issues/1197)
-    // we cannot use admin@fqdn because the user might not have set it up.
-    // we simply update the account with the latest email we have each time when getting letsencrypt certs
-    // https://github.com/ietf-wg-acme/acme/issues/30
-    users.getOwner(function (error, owner) {
-        options.email = error ? 'support@cloudron.io' : (owner.fallbackEmail || owner.email); // can error if not activated yet
+        var options = { };
+        if (result.tlsConfig.provider !== 'caas') { // matches 'le-prod' or 'letsencrypt-prod'
+            options.prod = result.tlsConfig.provider.match(/.*-prod/) !== null;
+            options.performHttpAuthorization = result.provider.match(/noop|manual|wildcard/) !== null;
+            options.wildcard = !!result.tlsConfig.wildcard;
+        }

-        callback(null, api, options);
+        // registering user with an email requires A or MX record (https://github.com/letsencrypt/boulder/issues/1197)
+        // we cannot use admin@fqdn because the user might not have set it up.
+        // we simply update the account with the latest email we have each time when getting letsencrypt certs
+        // https://github.com/ietf-wg-acme/acme/issues/30
+        users.getOwner(function (error, owner) {
+            options.email = error ? 'support@cloudron.io' : (owner.fallbackEmail || owner.email); // can error if not activated yet
+
+            callback(null, api, options);
+        });
    });
 }

@@ -114,71 +115,31 @@ function isExpiringSync(certFilePath, hours) {

    var result = safe.child_process.spawnSync('/usr/bin/openssl', [ 'x509', '-checkend', String(60 * 60 * hours), '-in', certFilePath ]);

-    if (!result) return 3; // some error
-
    debug('isExpiringSync: %s %s %s', certFilePath, result.stdout.toString('utf8').trim(), result.status);

    return result.status === 1; // 1 - expired 0 - not expired
 }

-// checks if the certificate matches the options provided by user (like wildcard, le-staging etc)
-function providerMatchesSync(domainObject, certFilePath, apiOptions) {
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof certFilePath, 'string');
-    assert.strictEqual(typeof apiOptions, 'object');
-
-    if (!fs.existsSync(certFilePath)) return false; // not found
-
-    if (apiOptions.fallback) return certFilePath.includes('.host.cert');
-
-    const subjectAndIssuer = safe.child_process.execSync(`/usr/bin/openssl x509 -noout -subject -issuer -in "${certFilePath}"`, { encoding: 'utf8' });
-    if (!subjectAndIssuer) return false; // something bad happenned
-
-    const subject = subjectAndIssuer.match(/^subject=(.*)$/m)[1];
-    const domain = subject.substr(subject.indexOf('=') + 1).trim(); // subject can be /CN=, CN=, CN = and other forms
-    const issuer = subjectAndIssuer.match(/^issuer=(.*)$/m)[1];
-    const isWildcardCert = domain.includes('*');
-    const isLetsEncryptProd = issuer.includes('Let\'s Encrypt Authority');
-
-    const issuerMismatch = (apiOptions.prod && !isLetsEncryptProd) || (!apiOptions.prod && isLetsEncryptProd);
-    // bare domain is not part of wildcard SAN
-    const wildcardMismatch = (domain !== domainObject.domain) && (apiOptions.wildcard && !isWildcardCert) || (!apiOptions.wildcard && isWildcardCert);
-
-    const mismatch = issuerMismatch || wildcardMismatch;
-
-    debug(`providerMatchesSync: ${certFilePath} subject=${subject} domain=${domain} issuer=${issuer} wildcard=${isWildcardCert}/${apiOptions.wildcard} prod=${isLetsEncryptProd}/${apiOptions.prod} match=${!mismatch}`);
-
-    return !mismatch;
-}
-
 // note: https://tools.ietf.org/html/rfc4346#section-7.4.2 (certificate_list) requires that the
 // servers certificate appears first (and not the intermediate cert)
-function validateCertificate(location, domainObject, certificate) {
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof domainObject, 'object');
-    assert(certificate && typeof certificate, 'object');
-
-    const cert = certificate.cert, key = certificate.key;
+function validateCertificate(domain, cert, key) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof cert, 'string');
+    assert.strictEqual(typeof key, 'string');

    // check for empty cert and key strings
    if (!cert && key) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, 'missing cert');
    if (cert && !key) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, 'missing key');

    // -checkhost checks for SAN or CN exclusively. SAN takes precedence and if present, ignores the CN.
-    const fqdn = domains.fqdn(location, domainObject);
+    var result = safe.child_process.execSync(`openssl x509 -noout -checkhost "${domain}"`, { encoding: 'utf8', input: cert });
+    if (!result) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, 'Unable to get certificate subject.');

-    var result = safe.child_process.execSync(`openssl x509 -noout -checkhost "${fqdn}"`, { encoding: 'utf8', input: cert });
-    if (result === null) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, 'Unable to get certificate subject:' + safe.error.message);
-
-    if (result.indexOf('does match certificate') === -1) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, `Certificate is not valid for this domain. Expecting ${fqdn}`);
+    if (result.indexOf('does match certificate') === -1) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, `Certificate is not valid for this domain. Expecting ${domain}`);

    // http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#verify
    var certModulus = safe.child_process.execSync('openssl x509 -noout -modulus', { encoding: 'utf8', input: cert });
-    if (certModulus === null) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, `Unable to get cert modulus: ${safe.error.message}`);
-
    var keyModulus = safe.child_process.execSync('openssl rsa -noout -modulus', { encoding: 'utf8', input: key });
-    if (keyModulus === null) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, `Unable to get key modulus: ${safe.error.message}`);
-
    if (certModulus !== keyModulus) return new ReverseProxyError(ReverseProxyError.INVALID_CERT, 'Key does not match the certificate.');

    // check expiration
@@ -191,65 +152,38 @@ function validateCertificate(location, domainObject, certificate) {
 function reload(callback) {
    if (process.env.BOX_ENV === 'test') return callback();

-    shell.sudo('reload', [ RELOAD_NGINX_CMD ], {}, callback);
-}
-
-function generateFallbackCertificateSync(domainObject) {
-    assert.strictEqual(typeof domainObject, 'object');
-
-    const domain = domainObject.domain;
-    const certFilePath = path.join(os.tmpdir(), `${domain}-${crypto.randomBytes(4).readUInt32LE(0)}.cert`);
-    const keyFilePath = path.join(os.tmpdir(), `${domain}-${crypto.randomBytes(4).readUInt32LE(0)}.key`);
-
-    let opensslConf = safe.fs.readFileSync('/etc/ssl/openssl.cnf', 'utf8');
-    // SAN must contain all the domains since CN check is based on implementation if SAN is found. -checkhost also checks only SAN if present!
-    let opensslConfWithSan;
-    let cn = domainObject.config.hyphenatedSubdomains ? domains.parentDomain(domain) : domain;
-
-    debug(`generateFallbackCertificateSync: domain=${domainObject.domain} cn=${cn} hyphenated=${domainObject.config.hyphenatedSubdomains}`);
-
-    opensslConfWithSan = `${opensslConf}\n[SAN]\nsubjectAltName=DNS:${domain},DNS:*.${cn}\n`;
-    let configFile = path.join(os.tmpdir(), 'openssl-' + crypto.randomBytes(4).readUInt32LE(0) + '.conf');
-    safe.fs.writeFileSync(configFile, opensslConfWithSan, 'utf8');
-    let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=*.${cn} -extensions SAN -config ${configFile} -nodes`);
-    if (!safe.child_process.execSync(certCommand)) return { error: new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message) };
-    safe.fs.unlinkSync(configFile);
-
-    const cert = safe.fs.readFileSync(certFilePath, 'utf8');
-    if (!cert) return { error: safe.error };
-    safe.fs.unlinkSync(certFilePath);
-
-    const key = safe.fs.readFileSync(keyFilePath, 'utf8');
-    if (!key) return { error: safe.error };
-    safe.fs.unlinkSync(keyFilePath);
-
-    return { cert: cert, key: key, error: null };
+    shell.sudo('reload', [ RELOAD_NGINX_CMD ], callback);
 }

 function setFallbackCertificate(domain, fallback, callback) {
    assert.strictEqual(typeof domain, 'string');
-    assert(fallback && typeof fallback === 'object');
    assert.strictEqual(typeof fallback, 'object');
    assert.strictEqual(typeof callback, 'function');

-    if (fallback.restricted) { // restricted certs are not backed up
-        debug(`setFallbackCertificate: setting restricted certs for domain ${domain}`);
-        if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, `${domain}.host.cert`), fallback.cert)) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message));
-        if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, `${domain}.host.key`), fallback.key)) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message));
-    } else {
-        debug(`setFallbackCertificate: setting certs for domain ${domain}`);
+    const certFilePath = path.join(paths.APP_CERTS_DIR, `${domain}.host.cert`);
+    const keyFilePath = path.join(paths.APP_CERTS_DIR, `${domain}.host.key`);
+
+    if (fallback) {
+        // backup the cert
        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${domain}.host.cert`), fallback.cert)) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message));
        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${domain}.host.key`), fallback.key)) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message));
+    } else if (!fs.existsSync(certFilePath) || !fs.existsSync(keyFilePath)) { // generate it
+        let opensslConf = safe.fs.readFileSync('/etc/ssl/openssl.cnf', 'utf8');
+        // SAN must contain all the domains since CN check is based on implementation if SAN is found. -checkhost also checks only SAN if present!
+        let opensslConfWithSan = `${opensslConf}\n[SAN]\nsubjectAltName=DNS:${domain},DNS:*.${domain}\n`;
+        let configFile = path.join(os.tmpdir(), 'openssl-' + crypto.randomBytes(4).readUInt32LE(0) + '.conf');
+        safe.fs.writeFileSync(configFile, opensslConfWithSan, 'utf8');
+        let certCommand = util.format(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=*.${domain} -extensions SAN -config ${configFile} -nodes`);
+        if (!safe.child_process.execSync(certCommand)) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, safe.error.message));
+        safe.fs.unlinkSync(configFile);
    }

-    platform.handleCertChanged('*.' + domain, function (error) {
+    platform.handleCertChanged('*.' + domain);
+
+    reload(function (error) {
        if (error) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, error));

-        reload(function (error) {
-            if (error) return callback(new ReverseProxyError(ReverseProxyError.INTERNAL_ERROR, error));
-
-            return callback(null);
-        });
+        return callback(null);
    });
 }

@@ -270,26 +204,9 @@ function getFallbackCertificate(domain, callback) {
    callback(null, { certFilePath, keyFilePath, type: 'fallback' });
 }

-function setAppCertificateSync(location, domainObject, certificate) {
-    assert.strictEqual(typeof location, 'string');
-    assert.strictEqual(typeof domainObject, 'object');
-    assert.strictEqual(typeof certificate, 'object');
-
-    let fqdn = domains.fqdn(location, domainObject);
-    if (certificate.cert && certificate.key) {
-        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.cert`), certificate.cert)) return safe.error;
-        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.key`), certificate.key)) return safe.error;
-    } else { // remove existing cert/key
-        if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.cert`))) debug('Error removing cert: ' + safe.error.message);
-        if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, `${fqdn}.user.key`))) debug('Error removing key: ' + safe.error.message);
-    }
-
-    return null;
-}
-
-function getCertificateByHostname(hostname, domainObject, callback) {
+function getCertificateByHostname(hostname, domain, callback) {
    assert.strictEqual(typeof hostname, 'string');
-    assert.strictEqual(typeof domainObject, 'object');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

    let certFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.user.cert`);
@@ -297,34 +214,34 @@ function getCertificateByHostname(hostname, domainObject, callback) {

    if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });

-    if (hostname !== domainObject.domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
-        let certName = domains.makeWildcard(hostname).replace('*.', '_.');
-        certFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.cert`);
-        keyFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.key`);
+    domains.get(domain, function (error, domainObject) {
+        if (error) return callback(error);

-        if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
-    } else {
-        certFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.cert`);
-        keyFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.key`);
+        if (hostname !== domain && domainObject.tlsConfig.wildcard) { // bare domain is not part of wildcard SAN
+            let certName = domains.makeWildcard(hostname).replace('*.', '_.');
+            certFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.cert`);
+            keyFilePath = path.join(paths.APP_CERTS_DIR, `${certName}.key`);

-        if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
-    }
+            if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
+        } else {
+            certFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.cert`);
+            keyFilePath = path.join(paths.APP_CERTS_DIR, `${hostname}.key`);

-    callback(null);
+            if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, { certFilePath, keyFilePath });
+        }
+
+        callback(null);
+    });
 }

 function getCertificate(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    domains.get(app.domain, function (error, domainObject) {
-        if (error) return callback(error);
+    getCertificateByHostname(app.fqdn, app.domain, function (error, result) {
+        if (error || result) return callback(error, result);

-        getCertificateByHostname(app.fqdn, domainObject, function (error, result) {
-            if (error || result) return callback(error, result);
-
-            return getFallbackCertificate(app.domain, callback);
-        });
+        return getFallbackCertificate(app.domain, callback);
    });
 }

@@ -334,40 +251,36 @@ function ensureCertificate(vhost, domain, auditSource, callback) {
    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

-    domains.get(domain, function (error, domainObject) {
-        if (error) return callback(error);
+    getCertificateByHostname(vhost, domain, function (error, result) {
+        if (result) {
+            debug(`ensureCertificate: ${vhost}. certificate already exists at ${result.keyFilePath}`);

-        getCertApi(domainObject, function (error, api, apiOptions) {
+            if (result.certFilePath.endsWith('.user.cert')) return callback(null, result); // user certs cannot be renewed
+            if (!isExpiringSync(result.certFilePath, 24 * 30)) return callback(null, result);
+            debug(`ensureCertificate: ${vhost} cert require renewal`);
+        } else {
+            debug(`ensureCertificate: ${vhost} cert does not exist`);
+        }
+
+        getCertApi(domain, function (error, api, apiOptions) {
            if (error) return callback(error);

-            getCertificateByHostname(vhost, domainObject, function (error, currentBundle) {
-                if (currentBundle) {
-                    debug(`ensureCertificate: ${vhost} certificate already exists at ${currentBundle.keyFilePath}`);
+            debug('ensureCertificate: getting certificate for %s with options %j', vhost, apiOptions);

-                    if (currentBundle.certFilePath.endsWith('.user.cert')) return callback(null, currentBundle); // user certs cannot be renewed
-                    if (!isExpiringSync(currentBundle.certFilePath, 24 * 30) && providerMatchesSync(domainObject, currentBundle.certFilePath, apiOptions)) return callback(null, currentBundle);
-                    debug(`ensureCertificate: ${vhost} cert require renewal`);
-                } else {
-                    debug(`ensureCertificate: ${vhost} cert does not exist`);
+            api.getCertificate(vhost, domain, apiOptions, function (error, certFilePath, keyFilePath) {
+                var errorMessage = error ? error.message : '';
+
+                if (error) {
+                    debug('ensureCertificate: could not get certificate. using fallback certs', error);
+                    mailer.certificateRenewalError(vhost, errorMessage);
                }

-                debug('ensureCertificate: getting certificate for %s with options %j', vhost, apiOptions);
+                eventlog.add(eventlog.ACTION_CERTIFICATE_RENEWAL, auditSource, { domain: vhost, errorMessage: errorMessage });

-                api.getCertificate(vhost, domain, apiOptions, function (error, certFilePath, keyFilePath) {
-                    var errorMessage = error ? error.message : '';
+                // if no cert was returned use fallback. the fallback/caas provider will not provide any for example
+                if (!certFilePath || !keyFilePath) return getFallbackCertificate(domain, callback);

-                    if (error) {
-                        debug('ensureCertificate: could not get certificate. using fallback certs', error);
-                        mailer.certificateRenewalError(vhost, errorMessage);
-                    }
-
-                    eventlog.add(currentBundle ? eventlog.ACTION_CERTIFICATE_RENEWAL : eventlog.ACTION_CERTIFICATE_NEW, auditSource, { domain: vhost, errorMessage: errorMessage });
-
-                    // if no cert was returned use fallback. the fallback/caas provider will not provide any for example
-                    if (!certFilePath || !keyFilePath) return getFallbackCertificate(domain, callback);
-
-                    callback(null, { certFilePath, keyFilePath, type: 'new-le' });
-                });
+                callback(null, { certFilePath, keyFilePath, type: 'new-le' });
            });
        });
    });
@@ -507,19 +420,19 @@ function unconfigureApp(app, callback) {
    });
 }

-function renewCerts(options, auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof options, 'object');
+function renewAll(auditSource, callback) {
    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
    assert.strictEqual(typeof callback, 'function');

+    debug('renewAll: Checking certificates for renewal');
+
    apps.getAll(function (error, allApps) {
        if (error) return callback(error);

        var appDomains = [];

        // add webadmin domain
-        appDomains.push({ domain: config.adminDomain(), fqdn: config.adminFqdn(), type: 'webadmin', nginxConfigFilename: path.join(paths.NGINX_APPCONFIG_DIR, constants.NGINX_ADMIN_CONFIG_FILE_NAME) });
+        appDomains.push({ domain: config.adminDomain(), fqdn: config.adminFqdn(), type: 'webadmin', nginxConfigFilename: constants.NGINX_ADMIN_CONFIG_FILE_NAME });

        // add app main
        allApps.forEach(function (app) {
@@ -531,58 +444,41 @@ function renewCerts(options, auditSource, progressCallback, callback) {
            });
        });

-        if (options.domain) appDomains = appDomains.filter(function (appDomain) { return appDomain.domain === options.domain; });
-
-        let progress = 1;
        async.eachSeries(appDomains, function (appDomain, iteratorCallback) {
-            progressCallback({ percent: progress, message: `Renewing certs of ${appDomain.fqdn}` });
-            progress += Math.round(100/appDomains.length);
-
            ensureCertificate(appDomain.fqdn, appDomain.domain, auditSource, function (error, bundle) {
                if (error) return iteratorCallback(error); // this can happen if cloudron is not setup yet

-                // hack to check if the app's cert changed or not. this doesn't handle prod/staging le change since they use same file name
+                // hack to check if the app's cert changed or not
                let currentNginxConfig = safe.fs.readFileSync(appDomain.nginxConfigFilename, 'utf8') || '';
                if (currentNginxConfig.includes(bundle.certFilePath)) return iteratorCallback();

-                debug(`renewCerts: creating new nginx config since ${appDomain.nginxConfigFilename} does not have ${bundle.certFilePath}`);
-
                // reconfigure since the cert changed
                var configureFunc;
                if (appDomain.type === 'webadmin') configureFunc = writeAdminConfig.bind(null, bundle, constants.NGINX_ADMIN_CONFIG_FILE_NAME, config.adminFqdn());
                else if (appDomain.type === 'main') configureFunc = writeAppConfig.bind(null, appDomain.app, bundle);
                else if (appDomain.type === 'alternate') configureFunc = writeAppRedirectConfig.bind(null, appDomain.app, appDomain.fqdn, bundle);
-                else return iteratorCallback(new Error(`Unknown domain type for ${appDomain.fqdn}. This should never happen`));
+                else return callback(new Error(`Unknown domain type for ${appDomain.fqdn}. This should never happen`));

                configureFunc(function (ignoredError) {
                    if (ignoredError) debug('renewAll: error reconfiguring app', ignoredError);

-                    platform.handleCertChanged(appDomain.fqdn, iteratorCallback);
+                    platform.handleCertChanged(appDomain.fqdn);
+
+                    iteratorCallback(); // move to next domain
                });
            });
-        }, callback);
+        });
    });
 }

-function renewAll(auditSource, callback) {
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('renewAll: Checking certificates for renewal');
-
-    renewCerts({}, auditSource, callback);
-}
-
 function removeAppConfigs() {
-    for (let appConfigFile of fs.readdirSync(paths.NGINX_APPCONFIG_DIR)) {
-        if (appConfigFile !== constants.NGINX_DEFAULT_CONFIG_FILE_NAME && appConfigFile !== constants.NGINX_ADMIN_CONFIG_FILE_NAME) {
-            fs.unlinkSync(path.join(paths.NGINX_APPCONFIG_DIR, appConfigFile));
-        }
+    for (var appConfigFile of fs.readdirSync(paths.NGINX_APPCONFIG_DIR)) {
+        fs.unlinkSync(path.join(paths.NGINX_APPCONFIG_DIR, appConfigFile));
    }
 }

 function configureDefaultServer(callback) {
-    assert.strictEqual(typeof callback, 'function');
+    callback = callback || NOOP_CALLBACK;

    var certFilePath = path.join(paths.NGINX_CERT_DIR,  'default.cert');
    var keyFilePath = path.join(paths.NGINX_CERT_DIR, 'default.key');
@@ -591,13 +487,11 @@ function configureDefaultServer(callback) {
        debug('configureDefaultServer: create new cert');

        var cn = 'cloudron-' + (new Date()).toISOString(); // randomize date a bit to keep firefox happy
-        if (!safe.child_process.execSync(`openssl req -x509 -newkey rsa:2048 -keyout ${keyFilePath} -out ${certFilePath} -days 3650 -subj /CN=${cn} -nodes`)) {
-            debug(`configureDefaultServer: could not generate certificate: ${safe.error.message}`);
-            return callback(safe.error);
-        }
+        var certCommand = util.format('openssl req -x509 -newkey rsa:2048 -keyout %s -out %s -days 3650 -subj /CN=%s -nodes', keyFilePath, certFilePath, cn);
+        safe.child_process.execSync(certCommand);
    }

-    writeAdminConfig({ certFilePath, keyFilePath }, constants.NGINX_DEFAULT_CONFIG_FILE_NAME, '', function (error) {
+    writeAdminConfig({ certFilePath, keyFilePath }, 'default.conf', '', function (error) {
        if (error) return callback(error);

        debug('configureDefaultServer: done');
@@ -4,8 +4,6 @@ exports = module.exports = {
    initialize: initialize,
    uninitialize: uninitialize,

-    isUnmanaged: isUnmanaged,
-
    scope: scope,
    websocketAuth: websocketAuth
 };
@@ -17,7 +15,6 @@ var accesscontrol = require('../accesscontrol.js'),
    clients = require('../clients.js'),
    ClientPasswordStrategy = require('passport-oauth2-client-password').Strategy,
    ClientsError = clients.ClientsError,
-    config = require('../config.js'),
    HttpError = require('connect-lastmile').HttpError,
    LocalStrategy = require('passport-local').Strategy,
    passport = require('passport'),
@@ -141,9 +138,3 @@ function websocketAuth(requiredScopes, req, res, next) {
        next();
    });
 }
-
-function isUnmanaged(req, res, next) {
-    if (!config.isManaged()) return next();
-
-    next(new HttpError(401, 'Managed instance does not permit this operation'));
-}
@@ -134,7 +134,6 @@ function installApp(req, res, next) {

    if ('sso' in data && typeof data.sso !== 'boolean') return next(new HttpError(400, 'sso must be a boolean'));
    if ('enableBackup' in data && typeof data.enableBackup !== 'boolean') return next(new HttpError(400, 'enableBackup must be a boolean'));
-    if ('enableAutomaticUpdate' in data && typeof data.enableAutomaticUpdate !== 'boolean') return next(new HttpError(400, 'enableAutomaticUpdate must be a boolean'));

    if (('debugMode' in data) && typeof data.debugMode !== 'object') return next(new HttpError(400, 'debugMode must be an object'));

@@ -145,11 +144,6 @@ function installApp(req, res, next) {
        if (data.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
    }

-    if ('env' in data) {
-        if (!data.env || typeof data.env !== 'object') return next(new HttpError(400, 'env must be an object'));
-        if (Object.keys(data.env).some(function (key) { return typeof data.env[key] !== 'string'; })) return next(new HttpError(400, 'env must contain values as strings'));
-    }
-
    debug('Installing app :%j', data);

    apps.install(data, req.user, auditSource(req), function (error, app) {
@@ -175,10 +169,6 @@ function configureApp(req, res, next) {

    if ('location' in data && typeof data.location !== 'string') return next(new HttpError(400, 'location must be string'));
    if ('domain' in data && typeof data.domain !== 'string') return next(new HttpError(400, 'domain must be string'));
-    // domain, location must both be provided since they are unique together
-    if ('location' in data && !('domain' in data)) return next(new HttpError(400, 'domain must be provided'));
-    if (!('location' in data) && 'domain' in data) return next(new HttpError(400, 'location must be provided'));
-
    if ('portBindings' in data && typeof data.portBindings !== 'object') return next(new HttpError(400, 'portBindings must be an object'));
    if ('accessRestriction' in data && typeof data.accessRestriction !== 'object') return next(new HttpError(400, 'accessRestriction must be an object'));

@@ -192,7 +182,6 @@ function configureApp(req, res, next) {
    if (data.xFrameOptions && typeof data.xFrameOptions !== 'string') return next(new HttpError(400, 'xFrameOptions must be a string'));

    if ('enableBackup' in data && typeof data.enableBackup !== 'boolean') return next(new HttpError(400, 'enableBackup must be a boolean'));
-    if ('enableAutomaticUpdate' in data && typeof data.enableAutomaticUpdate !== 'boolean') return next(new HttpError(400, 'enableAutomaticUpdate must be a boolean'));

    if (('debugMode' in data) && typeof data.debugMode !== 'object') return next(new HttpError(400, 'debugMode must be an object'));

@@ -205,11 +194,6 @@ function configureApp(req, res, next) {
        if (data.alternateDomains.some(function (d) { return (typeof d.domain !== 'string' || typeof d.subdomain !== 'string'); })) return next(new HttpError(400, 'alternateDomains array must contain objects with domain and subdomain strings'));
    }

-    if ('env' in data) {
-        if (!data.env || typeof data.env !== 'object') return next(new HttpError(400, 'env must be an object'));
-        if (Object.keys(data.env).some(function (key) { return typeof data.env[key] !== 'string'; })) return next(new HttpError(400, 'env must contain values as strings'));
-    }
-
    debug('Configuring app id:%s data:%j', req.params.id, data);

    apps.configure(req.params.id, data, req.user, auditSource(req), function (error) {
@@ -519,7 +503,7 @@ function execWebSocket(req, res, next) {
        if (error && error.reason === AppsError.BAD_STATE) return next(new HttpError(409, error.message));
        if (error) return next(new HttpError(500, error));

-        debug('Connected to terminal');
+        console.log('Connected to terminal');

        req.clearTimeout();

@@ -527,7 +511,7 @@ function execWebSocket(req, res, next) {
            duplexStream.on('end', function () { ws.close(); });
            duplexStream.on('close', function () { ws.close(); });
            duplexStream.on('error', function (error) {
-                debug('duplexStream error:', error);
+                console.error('duplexStream error:', error);
            });
            duplexStream.on('data', function (data) {
                if (ws.readyState !== WebSocket.OPEN) return;
@@ -535,7 +519,7 @@ function execWebSocket(req, res, next) {
            });

            ws.on('error', function (error) {
-                debug('websocket error:', error);
+                console.error('websocket error:', error);
            });
            ws.on('message', function (msg) {
                duplexStream.write(msg);
@@ -1,8 +1,8 @@
 'use strict';

 exports = module.exports = {
-    list: list,
-    startBackup: startBackup
+    get: get,
+    create: create
 };

 var backupdb = require('../backupdb.js'),
@@ -16,7 +16,7 @@ function auditSource(req) {
    return { ip: ip, username: req.user ? req.user.username : null, userId: req.user ? req.user.id : null };
 }

-function list(req, res, next) {
+function get(req, res, next) {
    var page = typeof req.query.page !== 'undefined' ? parseInt(req.query.page) : 1;
    if (!page || page < 0) return next(new HttpError(400, 'page query param has to be a postive number'));

@@ -31,11 +31,13 @@ function list(req, res, next) {
    });
 }

-function startBackup(req, res, next) {
-    backups.startBackupTask(auditSource(req), function (error, taskId) {
+function create(req, res, next) {
+    // note that cloudron.backup only waits for backup initiation and not for backup to complete
+    // backup progress can be checked up ny polling the progress api call
+    backups.backup(auditSource(req), function (error) {
        if (error && error.reason === BackupsError.BAD_STATE) return next(new HttpError(409, error.message));
        if (error) return next(new HttpError(500, error));

-        next(new HttpSuccess(202, { taskId }));
+        next(new HttpSuccess(202, {}));
    });
 }
@@ -0,0 +1,60 @@
+'use strict';
+
+exports = module.exports = {
+    getConfig: getConfig,
+    changePlan: changePlan
+};
+
+var caas = require('../caas.js'),
+    CaasError = require('../caas.js').CaasError,
+    config = require('../config.js'),
+    debug = require('debug')('box:routes/cloudron'),
+    HttpError = require('connect-lastmile').HttpError,
+    HttpSuccess = require('connect-lastmile').HttpSuccess,
+    _ = require('underscore');
+
+function getConfig(req, res, next) {
+    if (config.provider() !== 'caas') return next(new HttpError(422, 'Cannot use this API with this provider'));
+
+    caas.getBoxAndUserDetails(function (error, result) {
+        if (error) return next(new HttpError(500, error));
+
+        // the result is { box: { region, size, plan }, user: { billing, currency } }
+        next(new HttpSuccess(200, {
+            region: result.box.region,
+            size: result.box.size,
+            billing: !!result.user.billing,
+            plan: result.box.plan,
+            currency: result.user.currency
+        }));
+    });
+}
+
+function changePlan(req, res, next) {
+    if (config.provider() !== 'caas') return next(new HttpError(422, 'Cannot use this API with this provider'));
+
+    if ('size' in req.body && typeof req.body.size !== 'string') return next(new HttpError(400, 'size must be string'));
+    if ('region' in req.body && typeof req.body.region !== 'string') return next(new HttpError(400, 'region must be string'));
+
+    if ('domain' in req.body) {
+        if (typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be string'));
+        if (typeof req.body.provider !== 'string') return next(new HttpError(400, 'provider must be string'));
+    }
+
+    if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be string'));
+
+    debug('Migration requested domain:%s size:%s region:%s', req.body.domain, req.body.size, req.body.region);
+
+    var options = _.pick(req.body, 'domain', 'size', 'region');
+    if (Object.keys(options).length === 0) return next(new HttpError(400, 'no migrate option provided'));
+
+    if (options.domain) options.domain = options.domain.toLowerCase();
+
+    caas.changePlan(req.body, function (error) { // pass req.body because 'domain' can have arbitrary options
+        if (error && error.reason === CaasError.BAD_STATE) return next(new HttpError(409, error.message));
+        if (error && error.reason === CaasError.BAD_FIELD) return next(new HttpError(400, error.message));
+        if (error) return next(new HttpError(500, error));
+
+        next(new HttpSuccess(202, {}));
+    });
+}
@@ -2,7 +2,7 @@

 exports = module.exports = {
    reboot: reboot,
-    isRebootRequired: isRebootRequired,
+    getProgress: getProgress,
    getConfig: getConfig,
    getDisks: getDisks,
    getUpdateInfo: getUpdateInfo,
@@ -10,10 +10,7 @@ exports = module.exports = {
    feedback: feedback,
    checkForUpdates: checkForUpdates,
    getLogs: getLogs,
-    getLogStream: getLogStream,
-    getStatus: getStatus,
-    setDashboardDomain: setDashboardDomain,
-    renewCerts: renewCerts
+    getLogStream: getLogStream
 };

 var appstore = require('../appstore.js'),
@@ -24,6 +21,7 @@ var appstore = require('../appstore.js'),
    CloudronError = cloudron.CloudronError,
    HttpError = require('connect-lastmile').HttpError,
    HttpSuccess = require('connect-lastmile').HttpSuccess,
+    progress = require('../progress.js'),
    updater = require('../updater.js'),
    updateChecker = require('../updatechecker.js'),
    UpdaterError = require('../updater.js').UpdaterError,
@@ -34,19 +32,15 @@ function auditSource(req) {
    return { ip: ip, username: req.user ? req.user.username : null, userId: req.user ? req.user.id : null };
 }

-function reboot(req, res, next) {
-    // Finish the request, to let the appstore know we triggered the reboot
-    next(new HttpSuccess(202, {}));
-
-    cloudron.reboot(function () {});
+function getProgress(req, res, next) {
+    return next(new HttpSuccess(200, progress.getAll()));
 }

-function isRebootRequired(req, res, next) {
-    cloudron.isRebootRequired(function (error, result) {
-        if (error) return next(new HttpError(500, error));
+function reboot(req, res, next) {
+    // Finish the request, to let the appstore know we triggered the restore it
+    next(new HttpSuccess(202, {}));

-        next(new HttpSuccess(200, { rebootRequired: result }));
-    });
+    cloudron.reboot(function () { });
 }

 function getConfig(req, res, next) {
@@ -66,12 +60,13 @@ function getDisks(req, res, next) {

 function update(req, res, next) {
    // this only initiates the update, progress can be checked via the progress route
-    updater.updateToLatest(auditSource(req), function (error, taskId) {
+    updater.updateToLatest(auditSource(req), function (error) {
        if (error && error.reason === UpdaterError.ALREADY_UPTODATE) return next(new HttpError(422, error.message));
        if (error && error.reason === UpdaterError.BAD_STATE) return next(new HttpError(409, error.message));
+        if (error && error.reason === UpdaterError.SELF_UPGRADE_NOT_SUPPORTED) return next(new HttpError(412, error.message));
        if (error) return next(new HttpError(500, error));

-        next(new HttpSuccess(202, { taskId }));
+        next(new HttpSuccess(202, {}));
    });
 }

@@ -80,9 +75,6 @@ function getUpdateInfo(req, res, next) {
 }

 function checkForUpdates(req, res, next) {
-    // it can take a while sometimes to get all the app updates one by one
-    req.clearTimeout();
-
    async.series([
        updateChecker.checkAppUpdates,
        updateChecker.checkBoxUpdates
@@ -174,31 +166,3 @@ function getLogStream(req, res, next) {
        logStream.on('error', res.end.bind(res, null));
    });
 }
-
-function setDashboardDomain(req, res, next) {
-    if (!req.body.domain || typeof req.body.domain !== 'string') return next(new HttpError(400, 'domain must be a string'));
-
-    cloudron.setDashboardDomain(req.body.domain, function (error) {
-        if (error && error.reason === CloudronError.BAD_FIELD) return next(new HttpError(404, error.message));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(204, {}));
-    });
-}
-
-function getStatus(req, res, next) {
-    cloudron.getStatus(function (error, status) {
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, status));
-    });
-}
-
-function renewCerts(req, res, next) {
-    cloudron.renewCerts({ domain: req.body.domain || null }, auditSource(req), function (error, taskId) {
-        if (error && error.reason === CloudronError.NOT_FOUND) return next(new HttpError(404, error.message));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(202, { taskId }));
-    });
-}
@@ -16,12 +16,6 @@ var assert = require('assert'),
    HttpError = require('connect-lastmile').HttpError,
    HttpSuccess = require('connect-lastmile').HttpSuccess;

-function auditSource(req) {
-    var ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress || null;
-    return { ip: ip, username: req.user ? req.user.username : null, userId: req.user ? req.user.id : null };
-}
-
-// this code exists for the hosting provider edition
 function verifyDomainLock(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');

@@ -42,12 +36,8 @@ function add(req, res, next) {

    if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
    if ('fallbackCertificate' in req.body && typeof req.body.fallbackCertificate !== 'object') return next(new HttpError(400, 'fallbackCertificate must be a object with cert and key strings'));
-    if (req.body.fallbackCertificate) {
-        let fallbackCertificate = req.body.fallbackCertificate;
-        if (!fallbackCertificate.cert || typeof fallbackCertificate.cert !== 'string') return next(new HttpError(400, 'fallbackCertificate.cert must be a string'));
-        if (!fallbackCertificate.key || typeof fallbackCertificate.key !== 'string') return next(new HttpError(400, 'fallbackCertificate.key must be a string'));
-        if ('restricted' in fallbackCertificate && typeof fallbackCertificate.restricted !== 'boolean') return next(new HttpError(400, 'fallbackCertificate.restricted must be a boolean'));
-    }
+    if (req.body.fallbackCertificate && (!req.body.cert || typeof req.body.cert !== 'string')) return next(new HttpError(400, 'fallbackCertificate.cert must be a string'));
+    if (req.body.fallbackCertificate && (!req.body.key || typeof req.body.key !== 'string')) return next(new HttpError(400, 'fallbackCertificate.key must be a string'));

    if ('tlsConfig' in req.body) {
        if (!req.body.tlsConfig || typeof req.body.tlsConfig !== 'object') return next(new HttpError(400, 'tlsConfig must be a object with a provider string property'));
@@ -57,15 +47,7 @@ function add(req, res, next) {
    // some DNS providers like DigitalOcean take a really long time to verify credentials (https://github.com/expressjs/timeout/issues/26)
    req.clearTimeout();

-    let data = {
-        zoneName: req.body.zoneName || '',
-        provider: req.body.provider,
-        config: req.body.config,
-        fallbackCertificate: req.body.fallbackCertificate || null,
-        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' }
-    };
-
-    domains.add(req.body.domain, data, auditSource(req), function (error) {
+    domains.add(req.body.domain, req.body.zoneName || '', req.body.provider, req.body.config, req.body.fallbackCertificate || null, req.body.tlsConfig || { provider: 'letsencrypt-prod' }, function (error) {
        if (error && error.reason === DomainsError.ALREADY_EXISTS) return next(new HttpError(409, error.message));
        if (error && error.reason === DomainsError.BAD_FIELD) return next(new HttpError(400, error.message));
        if (error && error.reason === DomainsError.INVALID_PROVIDER) return next(new HttpError(400, error.message));
@@ -108,12 +90,8 @@ function update(req, res, next) {

    if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
    if ('fallbackCertificate' in req.body && typeof req.body.fallbackCertificate !== 'object') return next(new HttpError(400, 'fallbackCertificate must be a object with cert and key strings'));
-    if (req.body.fallbackCertificate) {
-        let fallbackCertificate = req.body.fallbackCertificate;
-        if (!fallbackCertificate.cert || typeof fallbackCertificate.cert !== 'string') return next(new HttpError(400, 'fallbackCertificate.cert must be a string'));
-        if (!fallbackCertificate.key || typeof fallbackCertificate.key !== 'string') return next(new HttpError(400, 'fallbackCertificate.key must be a string'));
-        if ('restricted' in fallbackCertificate && typeof fallbackCertificate.restricted !== 'boolean') return next(new HttpError(400, 'fallbackCertificate.restricted must be a boolean'));
-    }
+    if (req.body.fallbackCertificate && (!req.body.fallbackCertificate.cert || typeof req.body.fallbackCertificate.cert !== 'string')) return next(new HttpError(400, 'fallbackCertificate.cert must be a string'));
+    if (req.body.fallbackCertificate && (!req.body.fallbackCertificate.key || typeof req.body.fallbackCertificate.key !== 'string')) return next(new HttpError(400, 'fallbackCertificate.key must be a string'));

    if ('tlsConfig' in req.body) {
        if (!req.body.tlsConfig || typeof req.body.tlsConfig !== 'object') return next(new HttpError(400, 'tlsConfig must be a object with a provider string property'));
@@ -123,15 +101,7 @@ function update(req, res, next) {
    // some DNS providers like DigitalOcean take a really long time to verify credentials (https://github.com/expressjs/timeout/issues/26)
    req.clearTimeout();

-    let data = {
-        zoneName: req.body.zoneName || '',
-        provider: req.body.provider,
-        config: req.body.config,
-        fallbackCertificate: req.body.fallbackCertificate || null,
-        tlsConfig: req.body.tlsConfig || { provider: 'letsencrypt-prod' }
-    };
-
-    domains.update(req.params.domain, data, auditSource(req), function (error) {
+    domains.update(req.params.domain, req.body.zoneName || '', req.body.provider, req.body.config, req.body.fallbackCertificate || null, req.body.tlsConfig || { provider: 'letsencrypt-prod' }, function (error) {
        if (error && error.reason === DomainsError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === DomainsError.BAD_FIELD) return next(new HttpError(400, error.message));
        if (error && error.reason === DomainsError.INVALID_PROVIDER) return next(new HttpError(400, error.message));
@@ -144,7 +114,7 @@ function update(req, res, next) {
 function del(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');

-    domains.del(req.params.domain, auditSource(req), function (error) {
+    domains.del(req.params.domain, function (error) {
        if (error && error.reason === DomainsError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === DomainsError.IN_USE) return next(new HttpError(409, 'Domain is still in use. Remove all apps and mailboxes using this domain'));
        if (error) return next(new HttpError(500, error));
@@ -7,7 +7,7 @@ exports = module.exports = {
 var middleware = require('../middleware/index.js'),
    url = require('url');

-var graphiteProxy = middleware.proxy(url.parse('http://127.0.0.1:8417'));
+var graphiteProxy = middleware.proxy(url.parse('http://127.0.0.1:8000'));

 function getGraphs(req, res, next) {
    var parsedUrl = url.parse(req.url, true /* parseQueryString */);
@@ -4,6 +4,7 @@ exports = module.exports = {
    accesscontrol: require('./accesscontrol.js'),
    apps: require('./apps.js'),
    backups: require('./backups.js'),
+    caas: require('./caas.js'),
    clients: require('./clients.js'),
    cloudron: require('./cloudron.js'),
    developer: require('./developer.js'),
@@ -14,11 +15,9 @@ exports = module.exports = {
    oauth2: require('./oauth2.js'),
    mail: require('./mail.js'),
    profile: require('./profile.js'),
-    provision: require('./provision.js'),
-    services: require('./services.js'),
-    settings: require('./settings.js'),
+    setup: require('./setup.js'),
    sysadmin: require('./sysadmin.js'),
+    settings: require('./settings.js'),
    ssh: require('./ssh.js'),
-    tasks: require('./tasks.js'),
    users: require('./users.js')
 };
@@ -17,7 +17,7 @@ exports = module.exports = {

    sendTestMail: sendTestMail,

-    listMailboxes: listMailboxes,
+    getMailboxes: getMailboxes,
    getMailbox: getMailbox,
    addMailbox: addMailbox,
    updateMailbox: updateMailbox,
@@ -44,11 +44,6 @@ var assert = require('assert'),

 var mailProxy = middleware.proxy(url.parse('http://127.0.0.1:2020'));

-function auditSource(req) {
-    var ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress || null;
-    return { ip: ip, username: req.user ? req.user.username : null, userId: req.user ? req.user.id : null };
-}
-
 function getDomain(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');

@@ -91,10 +86,6 @@ function setDnsRecords(req, res, next) {
    assert.strictEqual(typeof req.body, 'object');
    assert.strictEqual(typeof req.params.domain, 'string');

-    // can take a setup all the DNS entries. this is mostly because some backends try to list DNS entries (DO)
-    // for upsert and this takes a lot of time
-    req.clearTimeout();
-
    mail.setDnsRecords(req.params.domain, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === MailError.EXTERNAL_ERROR) return next(new HttpError(424, error.message));
@@ -190,7 +181,7 @@ function setMailEnabled(req, res, next) {

    if (typeof req.body.enabled !== 'boolean') return next(new HttpError(400, 'enabled is required'));

-    mail.setMailEnabled(req.params.domain, !!req.body.enabled, auditSource(req), function (error) {
+    mail.setMailEnabled(req.params.domain, !!req.body.enabled, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === MailError.BAD_FIELD) return next(new HttpError(400, error.message));
        if (error && error.reason === MailError.BILLING_REQUIRED) return next(new HttpError(402, error.message));
@@ -214,10 +205,10 @@ function sendTestMail(req, res, next) {
    });
 }

-function listMailboxes(req, res, next) {
+function getMailboxes(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');

-    mail.listMailboxes(req.params.domain, function (error, result) {
+    mail.getMailboxes(req.params.domain, function (error, result) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error) return next(new HttpError(500, error));

@@ -243,7 +234,7 @@ function addMailbox(req, res, next) {
    if (typeof req.body.name !== 'string') return next(new HttpError(400, 'name must be a string'));
    if (typeof req.body.userId !== 'string') return next(new HttpError(400, 'userId must be a string'));

-    mail.addMailbox(req.body.name, req.params.domain, req.body.userId, auditSource(req), function (error) {
+    mail.addMailbox(req.body.name, req.params.domain, req.body.userId, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === MailError.ALREADY_EXISTS) return next(new HttpError(409, error.message));
        if (error && error.reason === MailError.BAD_FIELD) return next(new HttpError(400, error.message));
@@ -259,7 +250,7 @@ function updateMailbox(req, res, next) {

    if (typeof req.body.userId !== 'string') return next(new HttpError(400, 'userId must be a string'));

-    mail.updateMailboxOwner(req.params.name, req.params.domain, req.body.userId, function (error) {
+    mail.updateMailbox(req.params.name, req.params.domain, req.body.userId, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === MailError.BAD_FIELD) return next(new HttpError(400, error.message));
        if (error) return next(new HttpError(500, error));
@@ -272,7 +263,7 @@ function removeMailbox(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');
    assert.strictEqual(typeof req.params.name, 'string');

-    mail.removeMailbox(req.params.name, req.params.domain, auditSource(req), function (error) {
+    mail.removeMailbox(req.params.name, req.params.domain, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error) return next(new HttpError(500, error));

@@ -358,7 +349,7 @@ function addList(req, res, next) {
        if (typeof req.body.members[i] !== 'string') return next(new HttpError(400, 'member must be a string'));
    }

-    mail.addList(req.body.name, req.params.domain, req.body.members, auditSource(req), function (error) {
+    mail.addList(req.body.name, req.params.domain, req.body.members, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error && error.reason === MailError.ALREADY_EXISTS) return next(new HttpError(409, 'list already exists'));
        if (error && error.reason === MailError.BAD_FIELD) return next(new HttpError(400, error.message));
@@ -391,7 +382,7 @@ function removeList(req, res, next) {
    assert.strictEqual(typeof req.params.domain, 'string');
    assert.strictEqual(typeof req.params.name, 'string');

-    mail.removeList(req.params.name, req.params.domain, auditSource(req), function (error) {
+    mail.removeList(req.params.domain, req.params.name, function (error) {
        if (error && error.reason === MailError.NOT_FOUND) return next(new HttpError(404, error.message));
        if (error) return next(new HttpError(500, error));

@@ -1,134 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    getAll: getAll,
-    get: get,
-    configure: configure,
-    getLogs: getLogs,
-    getLogStream: getLogStream,
-    restart: restart
-};
-
-var addons = require('../addons.js'),
-    AddonsError = addons.AddonsError,
-    assert = require('assert'),
-    debug = require('debug')('box:routes/addons'),
-    HttpError = require('connect-lastmile').HttpError,
-    HttpSuccess = require('connect-lastmile').HttpSuccess;
-
-function getAll(req, res, next) {
-    addons.getServices(function (error, result) {
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, { services: result }));
-    });
-}
-
-function get(req, res, next) {
-    assert.strictEqual(typeof req.params.service, 'string');
-
-    addons.getService(req.params.service, function (error, result) {
-        if (error && error.reason === AddonsError.NOT_FOUND) return next(new HttpError(404, 'No such service'));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, { service: result }));
-    });
-}
-
-function configure(req, res, next) {
-    assert.strictEqual(typeof req.params.service, 'string');
-
-    if (typeof req.body.memory !== 'number') return next(new HttpError(400, 'memory must be a number'));
-
-    const data = {
-        memory: req.body.memory,
-        memorySwap: req.body.memory * 2
-    };
-
-    addons.configureService(req.params.service, data, function (error) {
-        if (error && error.reason === AddonsError.NOT_FOUND) return next(new HttpError(404, 'No such service'));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(202, {}));
-    });
-}
-
-function getLogs(req, res, next) {
-    assert.strictEqual(typeof req.params.service, 'string');
-
-    var lines = req.query.lines ? parseInt(req.query.lines, 10) : 100;
-    if (isNaN(lines)) return next(new HttpError(400, 'lines must be a number'));
-
-    debug(`Getting logs of service ${req.params.service}`);
-
-    var options = {
-        lines: lines,
-        follow: false,
-        format: req.query.format
-    };
-
-    addons.getServiceLogs(req.params.service, options, function (error, logStream) {
-        if (error && error.reason === AddonsError.NOT_FOUND) return next(new HttpError(404, 'No such service'));
-        if (error) return next(new HttpError(500, error));
-
-        res.writeHead(200, {
-            'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
-            'Cache-Control': 'no-cache',
-            'X-Accel-Buffering': 'no' // disable nginx buffering
-        });
-        logStream.pipe(res);
-    });
-}
-
-// this route is for streaming logs
-function getLogStream(req, res, next) {
-    assert.strictEqual(typeof req.params.service, 'string');
-
-    debug(`Getting logstream of service ${req.params.service}`);
-
-    var lines = req.query.lines ? parseInt(req.query.lines, 10) : -10; // we ignore last-event-id
-    if (isNaN(lines)) return next(new HttpError(400, 'lines must be a valid number'));
-
-    function sse(id, data) { return 'id: ' + id + '\ndata: ' + data + '\n\n'; }
-
-    if (req.headers.accept !== 'text/event-stream') return next(new HttpError(400, 'This API call requires EventStream'));
-
-    var options = {
-        lines: lines,
-        follow: true
-    };
-
-    addons.getServiceLogs(req.params.service, options, function (error, logStream) {
-        if (error && error.reason === AddonsError.NOT_FOUND) return next(new HttpError(404, 'No such service'));
-        if (error) return next(new HttpError(500, error));
-
-        res.writeHead(200, {
-            'Content-Type': 'text/event-stream',
-            'Cache-Control': 'no-cache',
-            'Connection': 'keep-alive',
-            'X-Accel-Buffering': 'no', // disable nginx buffering
-            'Access-Control-Allow-Origin': '*'
-        });
-        res.write('retry: 3000\n');
-        res.on('close', logStream.close);
-        logStream.on('data', function (data) {
-            var obj = JSON.parse(data);
-            res.write(sse(obj.monotonicTimestamp, JSON.stringify(obj))); // send timestamp as id
-        });
-        logStream.on('end', res.end.bind(res));
-        logStream.on('error', res.end.bind(res, null));
-    });
-}
-
-function restart(req, res, next) {
-    assert.strictEqual(typeof req.params.service, 'string');
-
-    debug(`Restarting service ${req.params.service}`);
-
-    addons.restartService(req.params.service, function (error) {
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(202, {}));
-    });
-}
@@ -23,17 +23,10 @@ exports = module.exports = {
    setAppstoreConfig: setAppstoreConfig,

    getPlatformConfig: getPlatformConfig,
-    setPlatformConfig: setPlatformConfig,
-
-    getDynamicDnsConfig: getDynamicDnsConfig,
-    setDynamicDnsConfig: setDynamicDnsConfig,
-
-    setRegistryConfig: setRegistryConfig
+    setPlatformConfig: setPlatformConfig
 };

 var assert = require('assert'),
-    docker = require('../docker.js'),
-    DockerError = docker.DockerError,
    HttpError = require('connect-lastmile').HttpError,
    HttpSuccess = require('connect-lastmile').HttpSuccess,
    safe = require('safetydance'),
@@ -211,28 +204,6 @@ function setPlatformConfig(req, res, next) {
    });
 }

-function getDynamicDnsConfig(req, res, next) {
-    settings.getDynamicDnsConfig(function (error, enabled) {
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, { enabled: enabled }));
-    });
-}
-
-function setDynamicDnsConfig(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-
-    if (typeof req.body.enabled !== 'boolean') return next(new HttpError(400, 'enabled boolean is required'));
-
-    settings.setDynamicDnsConfig(req.body.enabled, function (error) {
-        if (error && error.reason === SettingsError.BAD_FIELD) return next(new HttpError(400, error.message));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, {}));
-    });
-}
-
-
 function getAppstoreConfig(req, res, next) {
    settings.getAppstoreConfig(function (error, result) {
        if (error) return next(new HttpError(500, error));
@@ -264,18 +235,3 @@ function setAppstoreConfig(req, res, next) {
        });
    });
 }
-
-function setRegistryConfig(req, res, next) {
-    assert.strictEqual(typeof req.body, 'object');
-
-    if (typeof req.body.serveraddress !== 'string') return next(new HttpError(400, 'serveraddress is required'));
-    if ('username' in req.body && typeof req.body.username !== 'string') return next(new HttpError(400, 'username is required'));
-    if ('password' in req.body && typeof req.body.password !== 'string') return next(new HttpError(400, 'password is required'));
-
-    docker.setRegistryConfig(req.body, function (error) {
-        if (error && error.reason === DockerError.BAD_FIELD) return next(new HttpError(400, error.message));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200));
-    });
-}
@@ -3,9 +3,10 @@
 exports = module.exports = {
    providerTokenAuth: providerTokenAuth,
    setupTokenAuth: setupTokenAuth,
-    setup: setup,
+    dnsSetup: dnsSetup,
    activate: activate,
-    restore: restore
+    restore: restore,
+    getStatus: getStatus,
 };

 var assert = require('assert'),
@@ -15,8 +16,8 @@ var assert = require('assert'),
    debug = require('debug')('box:routes/setup'),
    HttpError = require('connect-lastmile').HttpError,
    HttpSuccess = require('connect-lastmile').HttpSuccess,
-    provision = require('../provision.js'),
-    ProvisionError = require('../provision.js').ProvisionError,
+    setup = require('../setup.js'),
+    SetupError = require('../setup.js').SetupError,
    superagent = require('superagent');

 function auditSource(req) {
@@ -61,38 +62,37 @@ function setupTokenAuth(req, res, next) {
    });
 }

-function setup(req, res, next) {
+function dnsSetup(req, res, next) {
    assert.strictEqual(typeof req.body, 'object');

-    if (!req.body.dnsConfig || typeof req.body.dnsConfig !== 'object') return next(new HttpError(400, 'dnsConfig is required'));
+    if (typeof req.body.provider !== 'string' || !req.body.provider) return next(new HttpError(400, 'provider is required'));
+    if (typeof req.body.domain !== 'string' || !req.body.domain) return next(new HttpError(400, 'domain is required'));
+    if (typeof req.body.adminFqdn !== 'string' || !req.body.adminFqdn) return next(new HttpError(400, 'adminFqdn is required'));

-    const dnsConfig = req.body.dnsConfig;
+    if ('zoneName' in req.body && typeof req.body.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
+    if (!req.body.config || typeof req.body.config !== 'object') return next(new HttpError(400, 'config must be an object'));

-    if (typeof dnsConfig.provider !== 'string' || !dnsConfig.provider) return next(new HttpError(400, 'provider is required'));
-    if (typeof dnsConfig.domain !== 'string' || !dnsConfig.domain) return next(new HttpError(400, 'domain is required'));
+    if ('tlsConfig' in req.body && typeof req.body.tlsConfig !== 'object') return next(new HttpError(400, 'tlsConfig must be an object'));
+    if (req.body.tlsConfig && (!req.body.tlsConfig.provider || typeof req.body.tlsConfig.provider !== 'string')) return next(new HttpError(400, 'tlsConfig.provider must be a string'));

-    if ('zoneName' in dnsConfig && typeof dnsConfig.zoneName !== 'string') return next(new HttpError(400, 'zoneName must be a string'));
-    if (!dnsConfig.config || typeof dnsConfig.config !== 'object') return next(new HttpError(400, 'config must be an object'));
-
-    if ('tlsConfig' in dnsConfig && typeof dnsConfig.tlsConfig !== 'object') return next(new HttpError(400, 'tlsConfig must be an object'));
-    if (dnsConfig.tlsConfig && (!dnsConfig.tlsConfig.provider || typeof dnsConfig.tlsConfig.provider !== 'string')) return next(new HttpError(400, 'tlsConfig.provider must be a string'));
-
-    // TODO: validate subfields of these objects
-    if (req.body.autoconf && typeof req.body.autoconf !== 'object') return next(new HttpError(400, 'autoconf must be an object'));
-
-    // it can take sometime to setup DNS, register cloudron
-    req.clearTimeout();
-
-    provision.setup(dnsConfig, req.body.autoconf || {}, auditSource(req), function (error) {
-        if (error && error.reason === ProvisionError.ALREADY_SETUP) return next(new HttpError(409, error.message));
-        if (error && error.reason === ProvisionError.BAD_FIELD) return next(new HttpError(400, error.message));
-        if (error && error.reason === ProvisionError.BAD_STATE) return next(new HttpError(409, error.message));
+    setup.dnsSetup(req.body.adminFqdn.toLowerCase(), req.body.domain.toLowerCase(), req.body.zoneName || '', req.body.provider, req.body.config, req.body.tlsConfig || { provider: 'letsencrypt-prod' }, function (error) {
+        if (error && error.reason === SetupError.ALREADY_SETUP) return next(new HttpError(409, error.message));
+        if (error && error.reason === SetupError.BAD_FIELD) return next(new HttpError(400, error.message));
+        if (error && error.reason === SetupError.BAD_STATE) return next(new HttpError(409, error.message));
        if (error) return next(new HttpError(500, error));

        next(new HttpSuccess(200));
    });
 }

+function getStatus(req, res, next) {
+    setup.getStatus(function (error, status) {
+        if (error) return next(new HttpError(500, error));
+
+        next(new HttpSuccess(200, status));
+    });
+}
+
 function activate(req, res, next) {
    assert.strictEqual(typeof req.body, 'object');

@@ -109,9 +109,9 @@ function activate(req, res, next) {
    var ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress;
    debug('activate: username:%s ip:%s', username, ip);

-    provision.activate(username, password, email, displayName, ip, auditSource(req), function (error, info) {
-        if (error && error.reason === ProvisionError.ALREADY_PROVISIONED) return next(new HttpError(409, 'Already setup'));
-        if (error && error.reason === ProvisionError.BAD_FIELD) return next(new HttpError(400, error.message));
+    setup.activate(username, password, email, displayName, ip, auditSource(req), function (error, info) {
+        if (error && error.reason === SetupError.ALREADY_PROVISIONED) return next(new HttpError(409, 'Already setup'));
+        if (error && error.reason === SetupError.BAD_FIELD) return next(new HttpError(400, error.message));
        if (error) return next(new HttpError(500, error));

        // only in caas case do we have to notify the api server about activation
@@ -143,14 +143,11 @@ function restore(req, res, next) {
    if (typeof req.body.backupId !== 'string') return next(new HttpError(400, 'backupId must be a string or null'));
    if (typeof req.body.version !== 'string') return next(new HttpError(400, 'version must be a string'));

-    // TODO: validate subfields of these objects
-    if (req.body.autoconf && typeof req.body.autoconf !== 'object') return next(new HttpError(400, 'autoconf must be an object'));
-
-    provision.restore(backupConfig, req.body.backupId, req.body.version, req.body.autoconf || {}, auditSource(req), function (error) {
-        if (error && error.reason === ProvisionError.ALREADY_SETUP) return next(new HttpError(409, error.message));
-        if (error && error.reason === ProvisionError.BAD_FIELD) return next(new HttpError(400, error.message));
-        if (error && error.reason === ProvisionError.BAD_STATE) return next(new HttpError(409, error.message));
-        if (error && error.reason === ProvisionError.EXTERNAL_ERROR) return next(new HttpError(424, error.message));
+    setup.restore(backupConfig, req.body.backupId, req.body.version, function (error) {
+        if (error && error.reason === SetupError.ALREADY_SETUP) return next(new HttpError(409, error.message));
+        if (error && error.reason === SetupError.BAD_FIELD) return next(new HttpError(400, error.message));
+        if (error && error.reason === SetupError.BAD_STATE) return next(new HttpError(409, error.message));
+        if (error && error.reason === SetupError.EXTERNAL_ERROR) return next(new HttpError(424, error.message));
        if (error) return next(new HttpError(500, error));

        next(new HttpSuccess(200));
@@ -3,15 +3,10 @@
 exports = module.exports = {
    backup: backup,
    update: update,
-    retire: retire,
-
-    importAppDatabase: importAppDatabase
+    retire: retire
 };

-var apps = require('../apps.js'),
-    AppsError = apps.AppsError,
-    addons = require('../addons.js'),
-    backups = require('../backups.js'),
+var backups = require('../backups.js'),
    BackupsError = require('../backups.js').BackupsError,
    cloudron = require('../cloudron.js'),
    debug = require('debug')('box:routes/sysadmin'),
@@ -26,11 +21,11 @@ function backup(req, res, next) {
    // note that cloudron.backup only waits for backup initiation and not for backup to complete
    // backup progress can be checked up ny polling the progress api call
    var auditSource = { userId: null, username: 'sysadmin' };
-    backups.startBackupTask(auditSource, function (error, taskId) {
+    backups.backup(auditSource, function (error) {
        if (error && error.reason === BackupsError.BAD_STATE) return next(new HttpError(409, error.message));
        if (error) return next(new HttpError(500, error));

-        next(new HttpSuccess(202, { taskId }));
+        next(new HttpSuccess(202, {}));
    });
 }

@@ -39,12 +34,13 @@ function update(req, res, next) {

    // this only initiates the update, progress can be checked via the progress route
    var auditSource = { userId: null, username: 'sysadmin' };
-    updater.updateToLatest(auditSource, function (error, taskId) {
+    updater.updateToLatest(auditSource, function (error) {
        if (error && error.reason === UpdaterError.ALREADY_UPTODATE) return next(new HttpError(422, error.message));
        if (error && error.reason === UpdaterError.BAD_STATE) return next(new HttpError(409, error.message));
+        if (error && error.reason === UpdaterError.SELF_UPGRADE_NOT_SUPPORTED) return next(new HttpError(412, error.message));
        if (error) return next(new HttpError(500, error));

-        next(new HttpSuccess(202, { taskId }));
+        next(new HttpSuccess(202, {}));
    });
 }

@@ -52,21 +48,8 @@ function retire(req, res, next) {
    debug('triggering retire');

    cloudron.retire('migrate', { }, function (error) {
-        if (error) debug('Retire failed.', error);
+        if (error) console.error('Retire failed.', error);
    });

    next(new HttpSuccess(202, {}));
 }
-
-function importAppDatabase(req, res, next) {
-    apps.get(req.params.id, function (error, app) {
-        if (error && error.reason === AppsError.NOT_FOUND) return next(new HttpError(404, 'No such app'));
-        if (error) return next(new HttpError(500, error));
-
-        addons.importAppDatabase(app, req.query.addon || '', function (error) {
-            if (error) return next(new HttpError(500, error));
-
-            next(new HttpSuccess(202, {}));
-        });
-    });
-}
@@ -1,121 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    get: get,
-    stopTask: stopTask,
-    list: list,
-
-    getLogs: getLogs,
-    getLogStream: getLogStream
-};
-
-let assert = require('assert'),
-    HttpError = require('connect-lastmile').HttpError,
-    HttpSuccess = require('connect-lastmile').HttpSuccess,
-    TaskError = require('../tasks.js').TaskError,
-    tasks = require('../tasks.js');
-
-function stopTask(req, res, next) {
-    assert.strictEqual(typeof req.params.taskId, 'string');
-
-    tasks.stopTask(req.params.taskId, function (error) {
-        if (error && error.reason === TaskError.NOT_FOUND) return next(new HttpError(404, 'No such task'));
-        if (error && error.reason === TaskError.BAD_STATE) return next(new HttpError(409, error.message));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(204, {}));
-    });
-}
-
-function get(req, res, next) {
-    assert.strictEqual(typeof req.params.taskId, 'string');
-
-    tasks.get(req.params.taskId, function (error, task) {
-        if (error && error.reason === TaskError.NOT_FOUND) return next(new HttpError(404, 'No such task'));
-        if (error) return next(new HttpError(500, error));
-
-        next(new HttpSuccess(200, tasks.removePrivateFields(task)));
-    });
-}
-
-function list(req, res, next) {
-    var page = typeof req.query.page !== 'undefined' ? parseInt(req.query.page) : 1;
-    if (!page || page < 0) return next(new HttpError(400, 'page query param has to be a postive number'));
-
-    var perPage = typeof req.query.per_page !== 'undefined'? parseInt(req.query.per_page) : 25;
-    if (!perPage || perPage < 0) return next(new HttpError(400, 'per_page query param has to be a postive number'));
-
-    if (req.query.type && typeof req.query.type !== 'string') return next(new HttpError(400, 'type must be a string'));
-
-    tasks.listByTypePaged(req.query.type || null, page, perPage, function (error, result) {
-        if (error) return next(new HttpError(500, error));
-
-        result = result.map(tasks.removePrivateFields);
-
-        next(new HttpSuccess(200, { tasks: result }));
-    });
-}
-
-function getLogs(req, res, next) {
-    assert.strictEqual(typeof req.params.taskId, 'string');
-
-    var lines = req.query.lines ? parseInt(req.query.lines, 10) : 100;
-    if (isNaN(lines)) return next(new HttpError(400, 'lines must be a number'));
-
-    var options = {
-        lines: lines,
-        follow: false,
-        format: req.query.format
-    };
-
-    tasks.getLogs(req.params.taskId, options, function (error, logStream) {
-        if (error && error.reason === TaskError.NOT_FOUND) return next(new HttpError(404, 'No such task'));
-        if (error) return next(new HttpError(500, error));
-
-        res.writeHead(200, {
-            'Content-Type': 'application/x-logs',
-            'Content-Disposition': 'attachment; filename="log.txt"',
-            'Cache-Control': 'no-cache',
-            'X-Accel-Buffering': 'no' // disable nginx buffering
-        });
-        logStream.pipe(res);
-    });
-}
-
-// this route is for streaming logs
-function getLogStream(req, res, next) {
-    assert.strictEqual(typeof req.params.taskId, 'string');
-
-    var lines = req.query.lines ? parseInt(req.query.lines, 10) : -10; // we ignore last-event-id
-    if (isNaN(lines)) return next(new HttpError(400, 'lines must be a valid number'));
-
-    function sse(id, data) { return 'id: ' + id + '\ndata: ' + data + '\n\n'; }
-
-    if (req.headers.accept !== 'text/event-stream') return next(new HttpError(400, 'This API call requires EventStream'));
-
-    var options = {
-        lines: lines,
-        follow: true
-    };
-
-    tasks.getLogs(req.params.taskId, options, function (error, logStream) {
-        if (error && error.reason === TaskError.NOT_FOUND) return next(new HttpError(404, 'No such task'));
-        if (error) return next(new HttpError(500, error));
-
-        res.writeHead(200, {
-            'Content-Type': 'text/event-stream',
-            'Cache-Control': 'no-cache',
-            'Connection': 'keep-alive',
-            'X-Accel-Buffering': 'no', // disable nginx buffering
-            'Access-Control-Allow-Origin': '*'
-        });
-        res.write('retry: 3000\n');
-        res.on('close', logStream.close);
-        logStream.on('data', function (data) {
-            var obj = JSON.parse(data);
-            res.write(sse(obj.monotonicTimestamp, JSON.stringify(obj))); // send timestamp as id
-        });
-        logStream.on('end', res.end.bind(res));
-        logStream.on('error', res.end.bind(res, null));
-    });
-}
@@ -15,6 +15,7 @@ var accesscontrol = require('../../accesscontrol.js'),
    clients = require('../../clients.js'),
    config = require('../../config.js'),
    constants = require('../../constants.js'),
+    apphealthmonitor = require('../../apphealthmonitor.js'),
    database = require('../../database.js'),
    docker = require('../../docker.js').connection,
    expect = require('expect.js'),
@@ -27,7 +28,6 @@ var accesscontrol = require('../../accesscontrol.js'),
    nock = require('nock'),
    path = require('path'),
    paths = require('../../paths.js'),
-    platform = require('../../platform.js'),
    safe = require('safetydance'),
    server = require('../../server.js'),
    settings = require('../../settings.js'),
@@ -234,16 +234,13 @@ function startBox(done) {
                }
                return false;
            }, callback);
-        },
-
-        function (callback) {
-            async.retry({ times: 50, interval: 10000 }, function (retryCallback) {
-                if (platform._isReady) return retryCallback();
-                console.log('Platform not ready yet, retry in 10 secs');
-                retryCallback('Platform not ready yet');
-            }, callback);
        }
-    ], done);
+    ], function (error) {
+        if (error) return done(error);
+
+        console.log('This test can take ~40 seconds to start as it waits for infra to be ready');
+        setTimeout(done, 40000);
+    });
 }

 function stopBox(done) {
@@ -254,6 +251,7 @@ function stopBox(done) {

    // db is not cleaned up here since it's too late to call it after server.stop. if called before server.stop taskmanager apptasks are unhappy :/
    async.series([
+        apphealthmonitor.stop,
        taskmanager.stopPendingTasks,
        taskmanager.waitForPendingTasks,
        appdb._clear,
@@ -264,6 +262,8 @@ function stopBox(done) {
 }

 describe('App API', function () {
+    this.timeout(100000);
+
    before(startBox);
    after(stopBox);

@@ -447,7 +447,7 @@ describe('App API', function () {
    it('app install succeeds with purchase', function (done) {
        var fake1 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons') >= 0; }, { 'domain': DOMAIN_0.domain }).reply(201, { cloudron: { id: CLOUDRON_ID } });
        var fake2 = nock(config.apiServerOrigin()).get('/api/v1/apps/' + APP_STORE_ID).reply(200, { manifest: APP_MANIFEST });
-        var fake3 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID, 'manifestId': APP_MANIFEST.id }).reply(201, { });
+        var fake3 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID }).reply(201, { });

        settings.setAppstoreConfig({ userId: user_1_id, token: USER_1_APPSTORE_TOKEN }, function (error) {
            if (error) return done(error);
@@ -577,7 +577,7 @@ describe('App API', function () {

    it('app install succeeds again', function (done) {
        var fake1 = nock(config.apiServerOrigin()).get('/api/v1/apps/' + APP_STORE_ID).reply(200, { manifest: APP_MANIFEST });
-        var fake2 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID, 'manifestId': APP_MANIFEST.id }).reply(201, { });
+        var fake2 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID }).reply(201, { });

        superagent.post(SERVER_URL + '/api/v1/apps/install')
            .query({ access_token: token })
@@ -592,7 +592,7 @@ describe('App API', function () {
            });
    });

-    it('app install fails with developer token', function (done) {
+    it('app install succeeds without password but developer token', function (done) {
        superagent.post(SERVER_URL + '/api/v1/developer/login')
            .send({ username: USERNAME, password: PASSWORD })
            .end(function (error, result) {
@@ -608,15 +608,30 @@ describe('App API', function () {
                    .query({ access_token: token })
                    .send({ manifest: APP_MANIFEST, location: APP_LOCATION+APP_LOCATION, domain: DOMAIN_0.domain, portBindings: null, accessRestriction: null })
                    .end(function (err, res) {
-                        expect(res.statusCode).to.equal(424); // appstore purchase external error
+                        expect(res.statusCode).to.equal(202);
+                        expect(res.body.id).to.be.a('string');
+                        APP_ID = res.body.id;
                        done();
                    });
            });
    });
+
+    it('can uninstall app without password but developer token', function (done) {
+        superagent.post(SERVER_URL + '/api/v1/apps/' + APP_ID + '/uninstall')
+            .query({ access_token: token })
+            .end(function (err, res) {
+                expect(res.statusCode).to.equal(202);
+                done();
+            });
+    });
 });

 describe('App installation', function () {
+    this.timeout(100000);
+
    var apiHockInstance = hock.createHock({ throwOnUnmatched: false });
+    var apiHockServer;
+
    var validCert1, validKey1;

    before(function (done) {
@@ -628,6 +643,7 @@ describe('App installation', function () {

        async.series([
            startBox,
+            apphealthmonitor.start,

            function (callback) {
                apiHockInstance
@@ -657,7 +673,7 @@ describe('App installation', function () {

    it('can install test app', function (done) {
        var fake1 = nock(config.apiServerOrigin()).get('/api/v1/apps/' + APP_STORE_ID).reply(200, { manifest: APP_MANIFEST });
-        var fake2 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID, 'manifestId': APP_MANIFEST.id }).reply(201, { });
+        var fake2 = nock(config.apiServerOrigin()).post(function (uri) { return uri.indexOf('/api/v1/users/' + user_1_id + '/cloudrons/' + CLOUDRON_ID + '/apps/') >= 0; }, { 'appstoreId': APP_STORE_ID }).reply(201, { });

        var count = 0;
        function checkInstallStatus() {
@@ -729,13 +745,7 @@ describe('App installation', function () {

    it('installation - volume created', function (done) {
        expect(fs.existsSync(paths.APPS_DATA_DIR + '/' + APP_ID));
-        let volume = docker.getVolume(APP_ID + '-localstorage');
-        volume.inspect(function (error, volume) {
-            expect(error).to.be(null);
-            expect(volume.Labels.appId).to.eql(APP_ID);
-            expect(volume.Options.device).to.eql(paths.APPS_DATA_DIR + '/' + APP_ID + '/data');
-            done();
-        });
+        done();
    });

    it('installation - http is up and running', function (done) {
@@ -771,7 +781,13 @@ describe('App installation', function () {
    it('installation - running container has volume mounted', function (done) {
        docker.getContainer(appEntry.containerId).inspect(function (error, data) {
            expect(error).to.not.be.ok();
-            expect(data.Mounts.filter(function (mount) { return mount.Destination === '/app/data'; })[0].Type).to.eql('volume');
+
+            // support newer docker versions
+            if (data.Volumes) {
+                expect(data.Volumes['/app/data']).to.eql(paths.APPS_DATA_DIR + '/' + APP_ID + '/data');
+            } else {
+                expect(data.Mounts.filter(function (mount) { return mount.Destination === '/app/data'; })[0].Source).to.eql(paths.APPS_DATA_DIR + '/' + APP_ID + '/data');
+            }

            done();
        });
@@ -814,6 +830,7 @@ describe('App installation', function () {
    });

    it('installation - app can check addons', function (done) {
+        this.timeout(120000);
        console.log('This test can take a while as it waits for scheduler addon to tick 3');
        checkAddons(appEntry, done);
    });
@@ -942,6 +959,7 @@ describe('App installation', function () {
    });

    it('installation - app can check addons', function (done) {
+        this.timeout(120000);
        console.log('This test can take a while as it waits for scheduler addon to tick 2');
        checkAddons(appEntry, done);
    });
@@ -1068,6 +1086,7 @@ describe('App installation', function () {
    });

    it('installation - app can check addons', function (done) {
+        this.timeout(120000);
        console.log('This test can take a while as it waits for scheduler addon to tick 4');
        checkAddons(appEntry, done);
    });
@@ -29,8 +29,6 @@ const DOMAIN_0 = {
    tlsConfig: { provider: 'fallback' }
 };

-let AUDIT_SOURCE = { ip: '1.2.3.4' };
-
 var token = null, ownerId = null;

 function setup(done) {
@@ -40,7 +38,7 @@ function setup(done) {
    async.series([
        server.start,
        database._clear,
-        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
+        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0.zoneName, DOMAIN_0.provider, DOMAIN_0.config, DOMAIN_0.fallbackCertificate, DOMAIN_0.tlsConfig),

        function createAdmin(callback) {
            superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
@@ -14,6 +14,7 @@ var appdb = require('../../appdb.js'),
    expect = require('expect.js'),
    locker = require('../../locker.js'),
    nock = require('nock'),
+    os = require('os'),
    superagent = require('superagent'),
    server = require('../../server.js'),
    settings = require('../../settings.js'),
@@ -34,8 +35,6 @@ const DOMAIN_0 = {
    tlsConfig: { provider: 'fallback' }
 };

-let AUDIT_SOURCE = { ip: '1.2.3.4' };
-
 var token = null, ownerId = null;
 var gSudoOriginal = null;
 function injectShellMock() {
@@ -51,6 +50,7 @@ function setup(done) {
    nock.cleanAll();
    config._reset();
    config.set('provider', 'caas');
+    config.setVersion('1.2.3');

    async.series([
        server.start.bind(server),
@@ -58,7 +58,7 @@ function setup(done) {
        database._clear,

        settingsdb.set.bind(null, settings.CAAS_CONFIG_KEY, JSON.stringify({ boxId: 'BOX_ID', token: 'ACCESS_TOKEN2' })),
-        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
+        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0.zoneName, DOMAIN_0.provider, DOMAIN_0.config, DOMAIN_0.fallbackCertificate, DOMAIN_0.tlsConfig),

        function createAdmin(callback) {
            var scope1 = nock(config.apiServerOrigin()).get('/api/v1/boxes/BOX_ID/setup/verify?setupToken=somesetuptoken').reply(200, {});
@@ -145,6 +145,29 @@ describe('Caas', function () {
        });
    });

+    describe('get config', function () {
+        before(setup);
+        after(cleanup);
+
+        it('succeeds (admin)', function (done) {
+            var scope = nock(config.apiServerOrigin())
+                .get('/api/v1/boxes/BOX_ID?token=ACCESS_TOKEN2')
+                .reply(200, { box: { region: 'sfo', size: '1gb' }, user: { }});
+
+            superagent.get(SERVER_URL + '/api/v1/caas/config')
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(200);
+                    expect(result.body.size).to.eql('1gb');
+                    expect(result.body.region).to.eql('sfo');
+
+                    expect(scope.isDone()).to.be.ok();
+
+                    done();
+                });
+        });
+    });
+
    describe('Backups API', function () {
        var scope1 = nock(config.apiServerOrigin()).post('/api/v1/boxes/BOX_ID/awscredentials?token=BACKUP_TOKEN')
            .reply(201, { credentials: { AccessKeyId: 'accessKeyId', SecretAccessKey: 'secretAccessKey' } }, { 'Content-Type': 'application/json' });
@@ -168,5 +191,169 @@ describe('Caas', function () {
                });
        });
    });
+
+    xdescribe('migrate', function () {
+        before(function (done) {
+            async.series([
+                setup,
+
+                function (callback) {
+                    var scope1 = nock(config.apiServerOrigin()).get('/api/v1/boxes/BOX_ID/setup/verify?setupToken=somesetuptoken').reply(200, {});
+                    var scope2 = nock(config.apiServerOrigin()).post('/api/v1/boxes/BOX_ID/setup/done?setupToken=somesetuptoken').reply(201, {});
+
+                    superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
+                        .query({ setupToken: 'somesetuptoken' })
+                        .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
+                        .end(function (error, result) {
+                            expect(result).to.be.ok();
+                            expect(scope1.isDone()).to.be.ok();
+                            expect(scope2.isDone()).to.be.ok();
+
+                            // stash token for further use
+                            token = result.body.token;
+
+                            callback();
+                        });
+                }
+            ], done);
+        });
+
+        after(function (done) {
+            locker.unlock(locker._operation); // migrate never unlocks
+            cleanup(done);
+        });
+
+        it('fails without token', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', region: 'sfo'})
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(401);
+                    done();
+                });
+        });
+
+        it('fails without password', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', region: 'sfo'})
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('succeeds without size', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ region: 'sfo', password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('fails with wrong size type', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 4, region: 'sfo', password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('succeeds without region', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(202);
+                    done();
+                });
+        });
+
+        it('fails with wrong region type', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', region: 4, password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(400);
+                    done();
+                });
+        });
+
+        it('fails when in wrong state', function (done) {
+            var scope2 = nock(config.apiServerOrigin())
+                .post('/api/v1/boxes/BOX_ID/awscredentials?token=BACKUP_TOKEN')
+                .reply(201, { credentials: { AccessKeyId: 'accessKeyId', SecretAccessKey: 'secretAccessKey', SessionToken: 'sessionToken' } });
+
+            var scope3 = nock(config.apiServerOrigin())
+                .post('/api/v1/boxes/BOX_ID/backupDone?token=APPSTORE_TOKEN', function (body) {
+                    return body.boxVersion && body.restoreKey && !body.appId && !body.appVersion && body.appBackupIds.length === 0;
+                })
+                .reply(200, { id: 'someid' });
+
+            var scope1 = nock(config.apiServerOrigin())
+                .post('/api/v1/boxes/BOX_ID/migrate?token=APPSTORE_TOKEN', function (body) {
+                    return body.size && body.region && body.restoreKey;
+                }).reply(409, {});
+
+            injectShellMock();
+
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', region: 'sfo', password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(202);
+
+                    function checkAppstoreServerCalled() {
+                        if (scope1.isDone() && scope2.isDone() && scope3.isDone()) {
+                            restoreShellMock();
+                            return done();
+                        }
+
+                        setTimeout(checkAppstoreServerCalled, 100);
+                    }
+
+                    checkAppstoreServerCalled();
+                });
+        });
+
+        it('succeeds', function (done) {
+            var scope1 = nock(config.apiServerOrigin()).post('/api/v1/boxes/BOX_ID/migrate?token=APPSTORE_TOKEN', function (body) {
+                return body.size && body.region && body.restoreKey;
+            }).reply(202, {});
+
+            var scope2 = nock(config.apiServerOrigin())
+                .post('/api/v1/boxes/BOX_ID/backupDone?token=APPSTORE_TOKEN', function (body) {
+                    return body.boxVersion && body.restoreKey && !body.appId && !body.appVersion && body.appBackupIds.length === 0;
+                })
+                .reply(200, { id: 'someid' });
+
+            var scope3 = nock(config.apiServerOrigin())
+                .post('/api/v1/boxes/BOX_ID/awscredentials?token=BACKUP_TOKEN')
+                .reply(201, { credentials: { AccessKeyId: 'accessKeyId', SecretAccessKey: 'secretAccessKey', SessionToken: 'sessionToken' } });
+
+            injectShellMock();
+
+            superagent.post(SERVER_URL + '/api/v1/cloudron/migrate')
+                .send({ size: 'small', region: 'sfo', password: PASSWORD })
+                .query({ access_token: token })
+                .end(function (error, result) {
+                    expect(result.statusCode).to.equal(202);
+
+                    function checkAppstoreServerCalled() {
+                        if (scope1.isDone() && scope2.isDone() && scope3.isDone()) {
+                            restoreShellMock();
+                            return done();
+                        }
+
+                        setTimeout(checkAppstoreServerCalled, 100);
+                    }
+
+                    checkAppstoreServerCalled();
+                });
+        });
+    });
 });

@@ -483,6 +483,8 @@ describe('Clients', function () {
    });

    describe('delete tokens by client', function () {
+        this.timeout(5000);
+
        before(setup2);
        after(cleanup);

@@ -191,6 +191,7 @@ describe('Cloudron', function () {
                    expect(result.body.apiServerOrigin).to.eql('http://localhost:6060');
                    expect(result.body.webServerOrigin).to.eql(null);
                    expect(result.body.adminFqdn).to.eql(config.adminFqdn());
+                    expect(result.body.progress).to.be.an('object');
                    expect(result.body.version).to.eql(config.version());
                    expect(result.body.memory).to.eql(os.totalmem());
                    expect(result.body.cloudronName).to.be.a('string');
@@ -37,6 +37,8 @@ function cleanup(done) {
 }

 describe('Developer API', function () {
+    this.timeout(20000);
+
    describe('login', function () {
        before(function (done) {
            async.series([
@@ -43,6 +43,8 @@ var DOMAIN_1 = {
 };

 describe('Domains API', function () {
+    this.timeout(10000);
+
    before(function (done) {
        config._reset();
        config.setFqdn(DOMAIN);
@@ -78,6 +78,8 @@ function cleanup(done) {
 }

 describe('Eventlog API', function () {
+    this.timeout(10000);
+
    before(setup);
    after(cleanup);

@@ -10,6 +10,7 @@ var async = require('async'),
    database = require('../../database.js'),
    expect = require('expect.js'),
    mail = require('../../mail.js'),
+    domains = require('../../domains.js'),
    maildb = require('../../maildb.js'),
    server = require('../../server.js'),
    superagent = require('superagent'),
@@ -47,8 +48,8 @@ function setup(done) {
        database._clear.bind(null),

        function dnsSetup(callback) {
-            superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-                .send({ dnsConfig: { provider: ADMIN_DOMAIN.provider, domain: ADMIN_DOMAIN.domain, config: ADMIN_DOMAIN.config } })
+            superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+                .send({ provider: ADMIN_DOMAIN.provider, domain: ADMIN_DOMAIN.domain, adminFqdn: 'my.' + ADMIN_DOMAIN.domain, config: ADMIN_DOMAIN.config })
                .end(function (error, result) {
                    expect(result).to.be.ok();
                    expect(result.statusCode).to.eql(200);
@@ -105,6 +106,8 @@ function cleanup(done) {
 }

 describe('Mail API', function () {
+    this.timeout(10000);
+
    before(setup);
    after(cleanup);

@@ -229,6 +232,8 @@ describe('Mail API', function () {
        var dnsAnswerQueue = [];
        var dkimDomain, spfDomain, mxDomain, dmarcDomain;

+        this.timeout(10000);
+
        before(function (done) {
            var dns = require('../../native-dns.js');

@@ -761,6 +766,7 @@ describe('Mail API', function () {
                    expect(res.body.mailbox).to.be.an('object');
                    expect(res.body.mailbox.name).to.equal(MAILBOX_NAME);
                    expect(res.body.mailbox.ownerId).to.equal(userId);
+                    expect(res.body.mailbox.ownerType).to.equal('user');
                    expect(res.body.mailbox.aliasTarget).to.equal(null);
                    expect(res.body.mailbox.domain).to.equal(DOMAIN_0.domain);
                    done();
@@ -776,6 +782,7 @@ describe('Mail API', function () {
                    expect(res.body.mailboxes[0]).to.be.an('object');
                    expect(res.body.mailboxes[0].name).to.equal(MAILBOX_NAME);
                    expect(res.body.mailboxes[0].ownerId).to.equal(userId);
+                    expect(res.body.mailboxes[0].ownerType).to.equal('user');
                    expect(res.body.mailboxes[0].aliasTarget).to.equal(null);
                    expect(res.body.mailboxes[0].domain).to.equal(DOMAIN_0.domain);
                    done();
@@ -908,10 +915,12 @@ describe('Mail API', function () {
                    expect(res.body.aliases.length).to.eql(2);
                    expect(res.body.aliases[0].name).to.equal('hello');
                    expect(res.body.aliases[0].ownerId).to.equal(userId);
+                    expect(res.body.aliases[0].ownerType).to.equal('user');
                    expect(res.body.aliases[0].aliasTarget).to.equal(MAILBOX_NAME);
                    expect(res.body.aliases[0].domain).to.equal(DOMAIN_0.domain);
                    expect(res.body.aliases[1].name).to.equal('there');
                    expect(res.body.aliases[1].ownerId).to.equal(userId);
+                    expect(res.body.aliases[1].ownerType).to.equal('user');
                    expect(res.body.aliases[1].aliasTarget).to.equal(MAILBOX_NAME);
                    expect(res.body.aliases[1].domain).to.equal(DOMAIN_0.domain);
                    done();
@@ -1023,6 +1032,7 @@ describe('Mail API', function () {
                    expect(res.body.list).to.be.an('object');
                    expect(res.body.list.name).to.equal(LIST_NAME);
                    expect(res.body.list.ownerId).to.equal('admin');
+                    expect(res.body.list.ownerType).to.equal('group');
                    expect(res.body.list.aliasTarget).to.equal(null);
                    expect(res.body.list.domain).to.equal(DOMAIN_0.domain);
                    expect(res.body.list.members).to.eql([ 'admin2', 'superadmin' ]);
@@ -1039,6 +1049,7 @@ describe('Mail API', function () {
                    expect(res.body.lists.length).to.equal(1);
                    expect(res.body.lists[0].name).to.equal(LIST_NAME);
                    expect(res.body.lists[0].ownerId).to.equal('admin');
+                    expect(res.body.lists[0].ownerType).to.equal('group');
                    expect(res.body.lists[0].aliasTarget).to.equal(null);
                    expect(res.body.lists[0].domain).to.equal(DOMAIN_0.domain);
                    expect(res.body.lists[0].members).to.eql([ 'admin2', 'superadmin' ]);
@@ -30,8 +30,6 @@ var accesscontrol = require('../../accesscontrol.js'),

 var SERVER_URL = 'http://localhost:' + config.get('port');

-let AUDIT_SOURCE = { ip: '1.2.3.4' };
-
 describe('OAuth2', function () {

    describe('flow', function () {
@@ -210,7 +208,7 @@ describe('OAuth2', function () {
            async.series([
                server.start,
                database._clear,
-                domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
+                domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0.zoneName, DOMAIN_0.provider, DOMAIN_0.config, DOMAIN_0.fallbackCertificate, DOMAIN_0.tlsConfig),
                clientdb.add.bind(null, CLIENT_0.id, CLIENT_0.appId, CLIENT_0.type, CLIENT_0.clientSecret, CLIENT_0.redirectURI, CLIENT_0.scope),
                clientdb.add.bind(null, CLIENT_1.id, CLIENT_1.appId, CLIENT_1.type, CLIENT_1.clientSecret, CLIENT_1.redirectURI, CLIENT_1.scope),
                clientdb.add.bind(null, CLIENT_2.id, CLIENT_2.appId, CLIENT_2.type, CLIENT_2.clientSecret, CLIENT_2.redirectURI, CLIENT_2.scope),
@@ -24,6 +24,8 @@ const EMAIL_0_NEW_FALLBACK = 'stupIDfallback@me.com';
 const DISPLAY_NAME_0_NEW = 'New Name';

 describe('Profile API', function () {
+    this.timeout(5000);
+
    var user_0 = null;
    var token_0;

@@ -20,6 +20,7 @@ var token = null;

 function setup(done) {
    config._reset();
+    config.setVersion('1.2.3');

    async.series([
        server.start,
@@ -39,223 +40,234 @@ describe('REST API', function () {
    after(cleanup);

    it('dns setup fails without provider', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { domain: DOMAIN, config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ domain: DOMAIN, adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with invalid provider', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'foobar', domain: DOMAIN, config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'foobar', domain: DOMAIN, adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with missing domain', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with invalid domain', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: '.foo', config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: '.foo', adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
+    });
+
+    it('dns setup fails with missing adminFqdn', function (done) {
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);
+
+            done();
+        });
+    });
+
+    it('dns setup fails with invalid adminFqdn', function (done) {
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my', config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);
+
+            done();
+        });
    });

    it('dns setup fails with invalid config', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, config: 'not an object' } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my' + DOMAIN, config: 'not an object' })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with invalid zoneName', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, config: {}, zoneName: 1337 } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my' + DOMAIN, config: {}, zoneName: 1337 })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with invalid tlsConfig', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, config: {}, tlsConfig: 'foobar' } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my' + DOMAIN, config: {}, tlsConfig: 'foobar' })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup fails with invalid tlsConfig provider', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, config: {}, tlsConfig: { provider: 1337 } } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my' + DOMAIN, config: {}, tlsConfig: { provider: 1337 } })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('dns setup succeeds', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, adminFqdn: 'my.' + DOMAIN, config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(200);
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(200);

-                done();
-            });
+            done();
+        });
    });

-    it('dns setup twice succeeds', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, DOMAIN, config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(200);
+    it('dns setup twice fails', function (done) {
+        superagent.post(SERVER_URL + '/api/v1/cloudron/dns_setup')
+        .send({ provider: 'noop', domain: DOMAIN, adminFqdn: 'my.' + DOMAIN, config: {} })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(409);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails without username', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ password: PASSWORD, email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ password: PASSWORD, email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails with invalid username', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: '?this.is-not!valid', password: PASSWORD, email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: '?this.is-not!valid', password: PASSWORD, email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails without email', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, password: PASSWORD })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, password: PASSWORD })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails with invalid email', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, password: PASSWORD, email: 'notanemail' })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, password: PASSWORD, email: 'notanemail' })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails without password', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation fails with invalid password', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, password: 'short', email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(400);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, password: 'short', email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(400);

-                done();
-            });
+            done();
+        });
    });

    it('activation succeeds', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(201);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(201);

-                // stash token for further use
-                token = result.body.token;
+            // stash token for further use
+            token = result.body.token;

-                done();
-            });
+            done();
+        });
    });

    it('activating twice fails', function (done) {
        superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
-            .query({ setupToken: 'somesetuptoken' })
-            .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(409);
+        .query({ setupToken: 'somesetuptoken' })
+        .send({ username: USERNAME, password: PASSWORD, email: EMAIL })
+        .end(function (error, result) {
+            expect(result).to.be.ok();
+            expect(result.statusCode).to.eql(409);

-                done();
-            });
-    });
-
-    it('dns setup after activation fails', function (done) {
-        superagent.post(SERVER_URL + '/api/v1/cloudron/setup')
-            .send({ dnsConfig: { provider: 'noop', domain: DOMAIN, DOMAIN, config: {} } })
-            .end(function (error, result) {
-                expect(result).to.be.ok();
-                expect(result.statusCode).to.eql(409);
-
-                done();
-            });
+            done();
+        });
    });

    it('does not crash with invalid JSON', function (done) {
@@ -82,7 +82,7 @@ describe('Settings API', function () {

        it('can set app_autoupdate_pattern', function (done) {
            var eventPattern = null;
-            settings.on(settings.APP_AUTOUPDATE_PATTERN_KEY, function (pattern) {
+            settings.events.on(settings.APP_AUTOUPDATE_PATTERN_KEY, function (pattern) {
                eventPattern = pattern;
            });

@@ -98,7 +98,7 @@ describe('Settings API', function () {

        it('can set app_autoupdate_pattern to never', function (done) {
            var eventPattern = null;
-            settings.on(settings.APP_AUTOUPDATE_PATTERN_KEY, function (pattern) {
+            settings.events.on(settings.APP_AUTOUPDATE_PATTERN_KEY, function (pattern) {
                eventPattern = pattern;
            });

@@ -145,7 +145,7 @@ describe('Settings API', function () {

        it('can set box_autoupdate_pattern', function (done) {
            var eventPattern = null;
-            settings.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, function (pattern) {
+            settings.events.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, function (pattern) {
                eventPattern = pattern;
            });

@@ -161,7 +161,7 @@ describe('Settings API', function () {

        it('can set box_autoupdate_pattern to never', function (done) {
            var eventPattern = null;
-            settings.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, function (pattern) {
+            settings.events.on(settings.BOX_AUTOUPDATE_PATTERN_KEY, function (pattern) {
                eventPattern = pattern;
            });

@@ -61,6 +61,8 @@ function cleanup(done) {
 }

 describe('SSH API', function () {
+    this.timeout(10000);
+
    before(setup);
    after(cleanup);

@@ -29,16 +29,15 @@ const DOMAIN_0 = {
    tlsConfig: { provider: 'fallback' }
 };

-let AUDIT_SOURCE = { ip: '1.2.3.4' };
-
 function setup(done) {
    config._reset();
    config.setFqdn(DOMAIN_0.domain);
+    config.setVersion('1.2.3');

    async.series([
        server.start,
        database._clear,
-        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0, AUDIT_SOURCE),
+        domains.add.bind(null, DOMAIN_0.domain, DOMAIN_0.zoneName, DOMAIN_0.provider, DOMAIN_0.config, DOMAIN_0.fallbackCertificate, DOMAIN_0.tlsConfig),

        function createAdmin(callback) {
            superagent.post(SERVER_URL + '/api/v1/cloudron/activate')
@@ -65,27 +64,29 @@ function cleanup(done) {
 }

 describe('Internal API', function () {
+    this.timeout(5000);
+
    before(setup);
    after(cleanup);

    describe('backup', function () {
        it('succeeds', function (done) {
            superagent.post(config.sysadminOrigin() + '/api/v1/backup')
-                .end(function (error, result) {
-                    expect(result.statusCode).to.equal(202);
+              .end(function (error, result) {
+                expect(result.statusCode).to.equal(202);

-                    function checkBackupStartEvent() {
-                        eventlog.getAllPaged([ eventlog.ACTION_BACKUP_START ], '', 1, 100, function (error, result) {
-                            expect(error).to.equal(null);
+                function checkBackupStartEvent() {
+                    eventlog.getAllPaged([ eventlog.ACTION_BACKUP_START ], '', 1, 100, function (error, result) {
+                        expect(error).to.equal(null);

-                            if (result.length === 0) return setTimeout(checkBackupStartEvent, 1000);
+                        if (result.length === 0) return setTimeout(checkBackupStartEvent, 1000);

-                            done();
-                        });
-                    }
+                        done();
+                    });
+                }

-                    checkBackupStartEvent();
-                });
+                checkBackupStartEvent();
+            });
        });
    });
-});
+});
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Girish Ramakrishnan	7547f6f6a6	Use async unlink (cherry picked from commit `8dd3c55ecf`)	2018-09-28 17:06:05 -07:00
Girish Ramakrishnan	f9ae208eaa	typoe (cherry picked from commit `1ee902a541`)	2018-09-28 17:02:17 -07:00
Girish Ramakrishnan	f9cd81d262	godaddy: empty the text record for del (cherry picked from commit `55cbe46c7c`)	2018-09-28 14:36:05 -07:00
Girish Ramakrishnan	db2fda0b18	acme2: Display any errors when cleaning up challenge (cherry picked from commit `5a8a4e7907`)	2018-09-28 14:33:26 -07:00
Girish Ramakrishnan	32744d4fdf	acme2: fix challenge subdomain calculation in cleanup (cherry picked from commit `3b5be641f0`)	2018-09-28 13:48:34 -07:00
Girish Ramakrishnan	e79857b9cf	TXT values must be quoted (cherry picked from commit `a34fe120fb`)	2018-09-27 20:22:13 -07:00
Girish Ramakrishnan	ffb02a3ba8	Use the local branch in hotfix	2018-09-26 22:26:46 -07:00
Girish Ramakrishnan	47f404cbca	3.2 changes	2018-09-26 16:16:52 -07:00
Girish Ramakrishnan	a9c1af50f7	start most cron jobs only after activation Importing the db might take some time. If a cron runs in the middle, it crashes. TypeError: Cannot read property 'domain' of undefined at Object.fqdn (/home/yellowtent/box/src/domains.js:126:111) at /home/yellowtent/box/src/apps.js:460:36 at /home/yellowtent/box/node_modules/async/dist/async.js:3110:16 at replenish (/home/yellowtent/box/node_modules/async/dist/async.js:1011:17) at /home/yellowtent/box/node_modules/async/dist/async.js:1016:9 at eachLimit$1 (/home/yellowtent/box/node_modules/async/dist/async.js:3196:24) at Object.<anonymous> (/home/yellowtent/box/node_modules/async/dist/async.js:1046:16) at /home/yellowtent/box/src/apps.js:458:19 at /home/yellowtent/box/src/appdb.js:232:13 at Query.args.(anonymous function) [as _callback] (/home/yellowtent/box/src/database.js	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	2967ad0ec8	more debugs and comments	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	1fd2c3381d	Fix perms issue when restoring Fixes #536	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	9d2f729089	Fix case where mailbox already exists	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	2aecf4be08	reset mailboxname to .app when empty fixes #587	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	cde43c8a47	3.1.4 changes	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	60fff645c4	Use alternateDomain fqdn for ensuring certificate this makes it work for hyphenated domains as well	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	299bcf5574	list domains only once	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	29745c277f	Attach fqdn to all the alternateDomains	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	7a43b742c0	allow hyphenated subdomains in caas	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	f85a7f4d5c	waitForDnsRecord: use subdomain as argument this allows to hyphenate the subdomain correctly in all places the original issue was that altDomain in caas was not working because waitForDnsRecord was not hyphenating.	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	291bee7f6a	register alt domains in install route	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	78365acc43	Fix new account return value https://tools.ietf.org/html/draft-ietf-acme-acme-07#section-7.3	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	976fe1c67d	acme2: register new account returns 201	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	e5ecfeb43f	calculate subdomain correctly for non-wildcard domains	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	6c9d093da0	Fix double callback	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	0725d40484	select app's cert based on domain's wildcard flag this also removes the confusing type field in the bundle. we instead check the current nginx config to see what cert is in use.	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	27b655c62e	rework args to ensureCertificate	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	2a7a0f04e4	Allow wildcard only with programmable DNS backend	2018-09-26 16:10:43 -07:00
Girish Ramakrishnan	b2f7eac629	make ensureCertificate check any wildcard cert	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	40d95a45a8	acme2: implement wildcard certs	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	ef89b09817	Move type validation to routes logic	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	cfa29fa82e	refactor to validateTlsConfig	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	7a07e0220f	consolidate hyphenatedSubdomains handling	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	d0eba4d3e3	acme2: wait for dns	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	5eccf7dc3b	Enhance waitForDns to support TXT records	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	29bd5fdbc7	acme2: dns authorization	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	7c28fa1175	pass domain arg to getCertificate API	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	323a498f04	rename func	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	2ea0d57d95	lint	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	55093af2ae	Fix callback use	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	8f4c1055c3	acme2 implementation	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	0bb81bf0ee	Fix tests	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	9a6b095563	acme -> acme1	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	b4f756dceb	storage: add access denied function (unused)	2018-09-26 16:06:13 -07:00
Girish Ramakrishnan	3554f39fdd	cloudflare: Add the chain message	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	bcd3a30579	cloudflare: priority is now an integer	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	2e756a9b25	Return 424 for external errors	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	bd4c5883f5	dns: implement wildcard dns validation	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	004f00a97b	Make wildcard a separate provider this is required because the config object is not returned for locked domains and the UI display for the provider field is then wrong.	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	44ed650538	typo	2018-09-26 16:05:54 -07:00
Girish Ramakrishnan	a3ae73d48f	3.1.4 changes	2018-09-12 20:29:55 -07:00