Add 1.9.3 changes

1.9.2 changes
The DNS provider property moved to the root dns config object
2018-01-18 12:51:18 +01:00 · 2018-01-18 12:50:47 +01:00 · 2018-01-18 12:50:20 +01:00 · 2018-01-17 19:38:05 +01:00 · 2018-01-16 22:02:09 +01:00 · 2018-01-16 19:09:33 +01:00
231 changed files with 29885 additions and 12076 deletions
@@ -948,3 +948,236 @@
 * Add a custom graphite plugin to collect disk usage statistics
 * Rotate logs of all apps automatically

+[1.6.0]
+* Allow apps to have 'network' capability (thanks @mehdi)
+* Fix crash in collectd disk usage collection script
+* Fix layout issues in update and oauth views
+* Use maxsize rule instead of size in lograte configs
+* Make it possible to skip backups per-app
+* Hide restore button for noop backend
+* Add popups and warnings for noop backend
+* Add webterminal to shell into apps from the admin UI
+* Update Haraka for a few crash fixes
+
+[1.6.1]
+* Patch release for 1.6.0 to fix regressions
+* Allow apps to have 'network' capability (thanks @mehdi)
+* Fix crash in collectd disk usage collection script
+* Fix layout issues in update and oauth views
+* Use maxsize rule instead of size in lograte configs
+* Make it possible to skip backups per-app
+* Hide restore button for noop backend
+* Add popups and warnings for noop backend
+* Add webterminal to shell into apps from the admin UI
+* Update Haraka for a few crash fixes
+
+[1.6.2]
+* Allow apps to have 'network' capability (thanks @mehdi)
+* Fix crash in collectd disk usage collection script
+* Fix layout issues in update and oauth views
+* Use maxsize rule instead of size in lograte configs
+* Make it possible to skip backups per-app
+* Hide restore button for noop backend
+* Add popups and warnings for noop backend
+* Add webterminal to shell into apps from the admin UI
+* Update Haraka for a few crash fixes
+
+[1.6.3]
+* Fixes selection issue while clicking on empty flexbox space
+* Indicate directories can be downloaded in the web terminal
+* Do not show app update indicator for normal users
+* Display email notice when using Cloudflare DNS
+* Set MX records correctly when using Cloudflare DNS
+* Fix bug where webterminal can incorrectly appear in main view
+* Do not crash if DNS credentials are invalid
+
+[1.6.4]
+* More descriptive Postmark email relay form
+* Fix file upload in chrome
+* Support Ctrl/Cmd+v webterminal pasting
+* Ensure unbound always starts up
+* Add option to run app in repair mode
+
+[1.6.5]
+* DigitalOcean DNS: Add pagination
+* Cloudflare DNS: Optimize listing of DNS entries
+* Update node to 6.11.3
+* App volumes can now be symlinked individually to external storage
+* Periodically check if IP is blacklisted and notify admins
+* Do not ask password when re-configuring app (since it is non-destructive)
+* Move mail data inside boxdata directory. This makes the no-op backend more useful
+* Remove collectd stats when app is uninstalled
+
+[1.7.0]
+* Add rsync format for backups. This feature allows incremental backups
+* Add Google DNS backend (thanks @syn)
+* Add DigitalOcean spaces backup storage backend
+* Add Cloudscale and Exoscale as supported VPS providers
+* Display backup progress and status in the web interface
+* Preliminary IPv6 support
+* Add IP RBL status to web interface
+* Add auto-update pattern `Every wednesday night`
+* Update Haraka to 2.8.15. This fixes the issue where emails were bounced with the message 'Send MAIL FROM first'
+* Do not overwrite existing subdomain when app's location is changed
+* Add button to send test email
+* Fix crash in carbon which made graphs disappear on some Cloudrons
+
+[1.7.1]
+* Add rsync format for backups. This feature allows incremental backups
+* Add Google DNS backend (thanks @syn)
+* Add DigitalOcean spaces backup storage backend
+* Add Cloudscale and Exoscale as supported VPS providers
+* Display backup progress and status in the web interface
+* Preliminary IPv6 support
+* Add IP RBL status to web interface
+* Add auto-update pattern `Every wednesday night`
+* Update Haraka to 2.8.15. This fixes the issue where emails were bounced with the message 'Send MAIL FROM first'
+* Do not overwrite existing subdomain when app's location is changed
+* Add button to send test email
+* Fix crash in carbon which made graphs disappear on some Cloudrons
+
+[1.7.2]
+* Add rsync format for backups. This feature allows incremental backups
+* Add Google DNS backend (thanks @syn)
+* Add Cloudscale and Exoscale as supported VPS providers
+* Display backup progress and status in the web interface
+* Preliminary IPv6 support
+* Add IP RBL status to web interface
+* Add auto-update pattern `Every wednesday night`
+* Update Haraka to 2.8.15. This fixes the issue where emails were bounced with the message 'Send MAIL FROM first'
+* Do not overwrite existing subdomain when app's location is changed
+* Add button to send test email
+* Fix crash in carbon which made graphs disappear on some Cloudrons
+* Fix issue where OAuth SSO did not work when alternate domain was used
+
+[1.7.3]
+* Add rsync format for backups. This feature allows incremental backups
+* Add Google DNS backend (thanks @syn)
+* Add Cloudscale and Exoscale as supported VPS providers
+* Display backup progress and status in the web interface
+* Preliminary IPv6 support
+* Add IP RBL status to web interface
+* Add auto-update pattern `Every wednesday night`
+* Update Haraka to 2.8.15. This fixes the issue where emails were bounced with the message 'Send MAIL FROM first'
+* Do not overwrite existing subdomain when app's location is changed
+* Add button to send test email
+* Fix crash in carbon which made graphs disappear on some Cloudrons
+* Fix issue where OAuth SSO did not work when alternate domain was used
+
+[1.7.4]
+* Add rsync format for backups. This feature allows incremental backups
+* Add Google DNS backend (thanks @syn)
+* Add DigitalOcean spaces backup storage backend
+* Add Cloudscale and Exoscale as supported VPS providers
+* Display backup progress and status in the web interface
+* Preliminary IPv6 support
+* Add IP RBL status to web interface
+* Add auto-update pattern `Every wednesday night`
+* Update Haraka to 2.8.15. This fixes the issue where emails were bounced with the message 'Send MAIL FROM first'
+* Do not overwrite existing subdomain when app's location is changed
+* Add button to send test email
+* Fix crash in carbon which made graphs disappear on some Cloudrons
+* Fix issue where OAuth SSO did not work when alternate domain was used
+* Changelog is now rendered in markdown format
+
+[1.7.5]
+* Expose a TLS relay port from mail container for Go applications
+
+[1.7.6]
+* Port bindings cannot be configured in update route anymore
+* Implement LDAP group compare
+* Pre-releases are now offered by appstore and not handled in box code anymore
+* LDAP pagination support. This will fix the warnings in NextCloud and Rocket.Chat
+* Check if directories can be created in the backup directory
+* Do not set the HTTPS agent when using HTTP with minio backup backend
+* Fix regression where a new domain config could not be set in the UI
+* New mail container release that fixes email sending with SOGo
+* Show 404 page for unknown domains
+
+[1.7.7]
+* Allow setting app memory till memory limit
+* Make the dkim selector dynamic
+* Fix issue where app update dialog did not close
+* Fix LE cert renewal failures
+* Send user and cert info in digest emails
+* Send oom, app failures and other important mails to cloudron owner's alt mail
+
+[1.8.0]
+* Fix group email bounce when a group has users that have not signed up yet
+* Do not restrict app memory limit to 4GB
+* Fix display of the latest backup in the weekly digest
+* Add UI to select users for access restriction
+* Update docker to 17.09
+* Update node to 6.11.5
+* Display package version of installed apps in the info dialog
+
+[1.8.1]
+* Update node modules
+* Allow a restore operation if app is already restoring
+* Remove pre-install bundle support since it was hardly used
+* Make the test email mail address configurable
+* Allow admins to access all apps
+* Send feedback via appstore API (instead of email)
+* Show documentation URL in the app info dialog
+* Update Lets Encrypt agrement URL (https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf)
+
+[1.8.2]
+* Update node modules
+* Allow a restore operation if app is already restoring
+* Remove pre-install bundle support since it was hardly used
+* Make the test email mail address configurable
+* Allow admins to access all apps
+* Send feedback via appstore API (instead of email)
+* Show documentation URL in the app info dialog
+* Update Lets Encrypt agrement URL (https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf)
+
+[1.8.3]
+* Ensure domain database record exists
+
+[1.8.4]
+* Fix issue where internal email was not delivered when email relay is enabled
+* Fix display of DNS records when email relay is enabled
+
+[1.8.5]
+* Fix issues where unused addons were not cleaned on an app update causing uninstall to fail
+* Change UI text from 'Waiting' to 'Pending'
+
+[1.9.0]
+* Prepare Cloudron for supporting multiple domains
+* Add Cloudron restore UI
+* Do not put app in errored state if backup fails
+* Display backup progress in CaaS
+* Add Google Cloud Storage backend for backups
+* Update node to 8.9.3 LTS
+* Set max email recepient limit (in outgoing emails) to 500
+
+[1.9.1]
+* Prepare Cloudron for supporting multiple domains
+* Add Cloudron restore UI
+* Do not put app in errored state if backup fails
+* Display backup progress in CaaS
+* Add Google Cloud Storage backend for backups
+* Update node to 8.9.3 LTS
+* Set max email recepient limit (in outgoing emails) to 500
+* Put terminal and app logs viewer to separate window
+
+[1.9.2]
+* Prepare Cloudron for supporting multiple domains
+* Add Cloudron restore UI
+* Do not put app in errored state if backup fails
+* Display backup progress in CaaS
+* Add Google Cloud Storage backend for backups
+* Update node to 8.9.3 LTS
+* Set max email recepient limit (in outgoing emails) to 500
+* Put terminal and app logs viewer to separate window
+
+[1.9.3]
+* Prepare Cloudron for supporting multiple domains
+* Add Cloudron restore UI
+* Do not put app in errored state if backup fails
+* Display backup progress in CaaS
+* Add Google Cloud Storage backend for backups
+* Update node to 8.9.3 LTS
+* Set max email recepient limit (in outgoing emails) to 500
+* Put terminal and app logs viewer to separate window
+
@@ -630,7 +630,7 @@ state the exclusion of warranty; and each file should have at least
 the "copyright" line and a pointer to where the full notice is found.

    box
-    Copyright (C) 2016 Cloudron UG
+    Copyright (C) 2016,2017 Cloudron UG

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published
@@ -9,10 +9,6 @@ a complex task.
 We are building the ultimate platform for self-hosting web apps. The Cloudron allows
 anyone to effortlessly host web applications on their server on their own terms.

-Support us on
-[![Flattr Cloudron](https://button.flattr.com/flattr-badge-large.png)](https://flattr.com/submit/auto?user_id=cloudron&url=https://cloudron.io&title=Cloudron&tags=opensource&category=software)
-or [pay us a coffee](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=8982CKNM46D8U)
-
 ## Features

 * Single click install for apps. Check out the [App Store](https://cloudron.io/appstore.html).
@@ -33,9 +29,9 @@ or [pay us a coffee](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_
 * Trivially migrate to another server keeping your apps and data (for example, switch your
  infrastructure provider or move to a bigger server).

-* Comprehensive [REST API](https://cloudron.io/references/api.html).
+* Comprehensive [REST API](https://cloudron.io/documentation/developer/api/).

-* [CLI](https://git.cloudron.io/cloudron/cloudron-cli) to configure apps.
+* [CLI](https://cloudron.io/documentation/cli/) to configure apps.

 * Alerts, audit logs, graphs, dns management ... and much more

@@ -49,33 +45,18 @@ You can install the Cloudron platform on your own server or get a managed server
 from cloudron.io. In either case, the Cloudron platform will keep your server and
 apps up-to-date and secure.

-* [Selfhosting](https://cloudron.io/references/selfhosting.html) - [Pricing](https://cloudron.io/pricing.html)
+* [Selfhosting](https://cloudron.io/documentation/installation/) - [Pricing](https://cloudron.io/pricing.html)
 * [Managed Hosting](https://cloudron.io/managed.html)

-The wiki has instructions on how you can install and update the Cloudron and the
-apps from source.
-
 ## Documentation

-* [User manual](https://cloudron.io/references/usermanual.html)
-* [Developer docs](https://cloudron.io/documentation.html)
-* [Architecture](https://cloudron.io/references/architecture.html)
+* [Documentation](https://cloudron.io/documentation/)

 ## Related repos

 The [base image repo](https://git.cloudron.io/cloudron/docker-base-image) is the parent image of all
 the containers in the Cloudron.

-The [graphite repo](https://git.cloudron.io/cloudron/docker-graphite) contains the graphite code
-that collects metrics for graphs.
-
-The addons are located in separate repositories
-* [Redis](https://git.cloudron.io/cloudron/redis-addon)
-* [Postgresql](https://git.cloudron.io/cloudron/postgresql-addon)
-* [MySQL](https://git.cloudron.io/cloudron/mysql-addon)
-* [Mongodb](https://git.cloudron.io/cloudron/mongodb-addon)
-* [Mail](https://git.cloudron.io/cloudron/mail-addon)
-
 ## Community

 * [Chat](https://chat.cloudron.io/)
@@ -39,17 +39,18 @@ apt-get -y install \
    rcconf \
    swaks \
    unattended-upgrades \
-    unbound
+    unbound \
+    xfsprogs

 # this ensures that unattended upgades are enabled, if it was disabled during ubuntu install time (see #346)
 # debconf-set-selection of unattended-upgrades/enable_auto_updates + dpkg-reconfigure does not work
 cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades

 echo "==> Installing node.js"
-mkdir -p /usr/local/node-6.11.2
-curl -sL https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-6.11.2
-ln -sf /usr/local/node-6.11.2/bin/node /usr/bin/node
-ln -sf /usr/local/node-6.11.2/bin/npm /usr/bin/npm
+mkdir -p /usr/local/node-8.9.3
+curl -sL https://nodejs.org/dist/v8.9.3/node-v8.9.3-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-8.9.3
+ln -sf /usr/local/node-8.9.3/bin/node /usr/bin/node
+ln -sf /usr/local/node-8.9.3/bin/npm /usr/bin/npm
 apt-get install -y python   # Install python which is required for npm rebuild
 [[ "$(python --version 2>&1)" == "Python 2.7."* ]] || die "Expecting python version to be 2.7.x"

@@ -60,7 +61,7 @@ echo "==> Installing Docker"
 mkdir -p /etc/systemd/system/docker.service.d
 echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2" > /etc/systemd/system/docker.service.d/cloudron.conf

-curl -sL https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/docker-ce_17.03.1~ce-0~ubuntu-xenial_amd64.deb -o /tmp/docker.deb
+curl -sL https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/docker-ce_17.09.0~ce-0~ubuntu_amd64.deb -o /tmp/docker.deb
 # apt install with install deps (as opposed to dpkg -i)
 apt install -y /tmp/docker.deb
 rm /tmp/docker.deb
@@ -6,9 +6,9 @@ var argv = require('yargs').argv,
    autoprefixer = require('gulp-autoprefixer'),
    concat = require('gulp-concat'),
    cssnano = require('gulp-cssnano'),
-    del = require('del'),
    ejs = require('gulp-ejs'),
    gulp = require('gulp'),
+    rimraf = require('rimraf'),
    sass = require('gulp-sass'),
    serve = require('gulp-serve'),
    sourcemaps = require('gulp-sourcemaps'),
@@ -50,7 +50,7 @@ if (argv.help || argv.h) {
    process.exit(1);
 }

-gulp.task('js', ['js-index', 'js-setup', 'js-setupdns', 'js-update'], function () {});
+gulp.task('js', ['js-index', 'js-logs', 'js-terminal', 'js-setup', 'js-setupdns', 'js-restore', 'js-update'], function () {});

 var oauth = {
    clientId: argv.clientId || 'cid-webadmin',
@@ -82,7 +82,7 @@ gulp.task('js-index', function () {
        'webadmin/src/js/main.js',
        'webadmin/src/views/*.js'
        ])
-        .pipe(ejs({ oauth: oauth }, { ext: '.js' }))
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
        .pipe(sourcemaps.init())
        .pipe(concat('index.js', { newLine: ';' }))
        .pipe(uglifyer)
@@ -90,6 +90,38 @@ gulp.task('js-index', function () {
        .pipe(gulp.dest('webadmin/dist/js'));
 });

+gulp.task('js-logs', function () {
+    // needs special treatment for error handling
+    var uglifyer = uglify();
+    uglifyer.on('error', function (error) {
+        console.error(error);
+    });
+
+    gulp.src(['webadmin/src/js/logs.js', 'webadmin/src/js/client.js'])
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
+        .pipe(sourcemaps.init())
+        .pipe(concat('logs.js', { newLine: ';' }))
+        .pipe(uglifyer)
+        .pipe(sourcemaps.write())
+        .pipe(gulp.dest('webadmin/dist/js'));
+});
+
+gulp.task('js-terminal', function () {
+    // needs special treatment for error handling
+    var uglifyer = uglify();
+    uglifyer.on('error', function (error) {
+        console.error(error);
+    });
+
+    gulp.src(['webadmin/src/js/terminal.js', 'webadmin/src/js/client.js'])
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
+        .pipe(sourcemaps.init())
+        .pipe(concat('terminal.js', { newLine: ';' }))
+        .pipe(uglifyer)
+        .pipe(sourcemaps.write())
+        .pipe(gulp.dest('webadmin/dist/js'));
+});
+
 gulp.task('js-setup', function () {
    // needs special treatment for error handling
    var uglifyer = uglify();
@@ -98,7 +130,7 @@ gulp.task('js-setup', function () {
    });

    gulp.src(['webadmin/src/js/setup.js', 'webadmin/src/js/client.js'])
-        .pipe(ejs({ oauth: oauth }, { ext: '.js' }))
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
        .pipe(sourcemaps.init())
        .pipe(concat('setup.js', { newLine: ';' }))
        .pipe(uglifyer)
@@ -114,7 +146,7 @@ gulp.task('js-setupdns', function () {
    });

    gulp.src(['webadmin/src/js/setupdns.js', 'webadmin/src/js/client.js'])
-        .pipe(ejs({ oauth: oauth }, { ext: '.js' }))
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
        .pipe(sourcemaps.init())
        .pipe(concat('setupdns.js', { newLine: ';' }))
        .pipe(uglifyer)
@@ -122,6 +154,23 @@ gulp.task('js-setupdns', function () {
        .pipe(gulp.dest('webadmin/dist/js'));
 });

+gulp.task('js-restore', function () {
+    // needs special treatment for error handling
+    var uglifyer = uglify();
+    uglifyer.on('error', function (error) {
+        console.error(error);
+    });
+
+    gulp.src(['webadmin/src/js/restore.js', 'webadmin/src/js/client.js'])
+        .pipe(ejs({ oauth: oauth }, {}, { ext: '.js' }))
+        .pipe(sourcemaps.init())
+        .pipe(concat('restore.js', { newLine: ';' }))
+        .pipe(uglifyer)
+        .pipe(sourcemaps.write())
+        .pipe(gulp.dest('webadmin/dist/js'));
+});
+
+
 gulp.task('js-update', function () {
    // needs special treatment for error handling
    var uglifyer = uglify();
@@ -143,7 +192,7 @@ gulp.task('js-update', function () {
 // --------------

 gulp.task('html', ['html-views', 'html-update', 'html-templates'], function () {
-    return gulp.src('webadmin/src/*.html').pipe(ejs({ apiOriginHostname: oauth.apiOriginHostname }, { ext: '.html' })).pipe(gulp.dest('webadmin/dist'));
+    return gulp.src('webadmin/src/*.html').pipe(ejs({ apiOriginHostname: oauth.apiOriginHostname }, {}, { ext: '.html' })).pipe(gulp.dest('webadmin/dist'));
 });

 gulp.task('html-update', function () {
@@ -191,12 +240,16 @@ gulp.task('watch', ['default'], function () {
    gulp.watch(['webadmin/src/js/update.js'], ['js-update']);
    gulp.watch(['webadmin/src/js/setup.js', 'webadmin/src/js/client.js'], ['js-setup']);
    gulp.watch(['webadmin/src/js/setupdns.js', 'webadmin/src/js/client.js'], ['js-setupdns']);
+    gulp.watch(['webadmin/src/js/restore.js', 'webadmin/src/js/client.js'], ['js-restore']);
+    gulp.watch(['webadmin/src/js/logs.js', 'webadmin/src/js/client.js'], ['js-logs']);
+    gulp.watch(['webadmin/src/js/terminal.js', 'webadmin/src/js/client.js'], ['js-terminal']);
    gulp.watch(['webadmin/src/js/index.js', 'webadmin/src/js/client.js', 'webadmin/src/js/appstore.js', 'webadmin/src/js/main.js', 'webadmin/src/views/*.js'], ['js-index']);
    gulp.watch(['webadmin/src/3rdparty/**/*'], ['3rdparty']);
 });

 gulp.task('clean', function () {
-    del.sync(['webadmin/dist', 'setup/splash/website']);
+    rimraf.sync('webadmin/dist');
+    rimraf.sync('setup/splash/website');
 });

 gulp.task('default', ['clean', 'html', 'js', '3rdparty', 'images', 'css'], function () {});
@@ -1,32 +0,0 @@
-#!/usr/bin/env node
-
-'use strict';
-
-var tar = require('tar-fs'),
-    fs = require('fs'),
-    path = require('path'),
-    zlib = require('zlib');
-
-if (process.argv.length < 4) {
-    console.error('Usage: tarjs <cwd> <dir>');
-    process.exit(1);
-}
-
-var dir = process.argv[3];
-var cwd = process.argv[2];
-
-console.error('Packing directory "'+ dir +'" from within "' + cwd + '" and stream to stdout');
-
-process.chdir(cwd);
-
-var stat = fs.statSync(dir);
-if (!stat.isDirectory()) throw(dir + ' is not a directory');
-
-var gzipStream = zlib.createGzip({});
-
-tar.pack(path.resolve(dir), {
-    ignore: function (name) {
-        if (name === '.') return true;
-        return false;
-    }
-}).pipe(gzipStream).pipe(process.stdout);
@@ -2,7 +2,7 @@

 var async = require('async');

-var ADMIN_GROUP_ID = 'admin'; // see groups.js
+var ADMIN_GROUP_ID = 'admin'; // see constants.js

 exports.up = function(db, callback) {
 	async.series([
@@ -0,0 +1,16 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN enableBackup BOOLEAN DEFAULT 1', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN enableBackup', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE settings MODIFY value TEXT', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE settings MODIFY value VARCHAR(512)', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,25 @@
+'use strict';
+
+// ensure backupFolder and format are not empty
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM settings WHERE name=?', [ 'backup_config' ], function (error, result) {
+        if (error || result.length === 0) return callback(error);
+
+        var value = JSON.parse(result[0].value);
+        value.format = 'tgz'; // set the format
+
+        if (value.provider === 'filesystem' && !value.backupFolder) {
+            value.backupFolder = '/var/backups'; // set the backupFolder
+        }
+
+        db.runSql('UPDATE settings SET value = ? WHERE name = ?', [ JSON.stringify(value), 'backup_config' ], function (error) {
+            if (error) console.error('Error setting ownerid ' + JSON.stringify(u) + error);
+            callback();
+        });
+
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE backups ADD COLUMN format VARCHAR(16) DEFAULT "tgz"', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE backups DROP COLUMN format', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN newConfigJson TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN newConfigJson', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,40 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE backups ADD COLUMN manifestJson TEXT'),
+
+        db.runSql.bind(db, 'START TRANSACTION;'),
+
+        // fill all the backups with restoreConfigs from current apps
+        function addManifests(callback) {
+            console.log('Importing manifests');
+
+            db.all('SELECT * FROM backups WHERE type="app"', function (error, backups) {
+                if (error) return callback(error);
+
+                async.eachSeries(backups, function (backup, next) {
+                    var m = backup.restoreConfigJson ? JSON.parse(backup.restoreConfigJson) : null;
+                    if (m) m = JSON.stringify(m.manifest);
+
+                    db.runSql('UPDATE backups SET manifestJson=? WHERE id=?', [ m, backup.id ], next);
+                }, callback);
+            });
+        },
+
+        db.runSql.bind(db, 'COMMIT'),
+
+        // remove the restoreConfig
+        db.runSql.bind(db, 'ALTER TABLE backups DROP COLUMN restoreConfigJson')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE backups DROP COLUMN manifestJson'),
+        db.runSql.bind(db, 'ALTER TABLE backups ADD COLUMN restoreConfigJson TEXT'),
+    ], callback);
+};
+
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE newConfigJson updateConfigJson TEXT', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE updateConfigJson newConfigJson TEXT', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE lastBackupId restoreConfigJson TEXT', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE restoreConfigJson lastBackupId TEXT', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,31 @@
+'use strict';
+
+// WARNING!!
+// At this point the default db collation is utf8mb4_unicode_ci however we already have foreign key constraits
+// already with tables on utf8_bin charset, so we cannot convert all tables here to utf8mb4 collation without
+// a reimport from a sql dump, as foreign keys across different collations are not supported
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'ALTER TABLE appPortBindings CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE apps CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE authcodes CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE backups CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE clients CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE eventlog CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE groupMembers CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE groups CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE mailboxes CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE migrations CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE settings CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE tokens CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin'),
+        db.runSql.bind(db, 'ALTER TABLE users CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    // nothing to be done here
+    callback();
+};
@@ -0,0 +1,70 @@
+'use strict';
+
+var async = require('async'),
+    safe = require('safetydance');
+
+exports.up = function(db, callback) {
+    // first check precondtion of domain entry in settings
+    db.all('SELECT * FROM settings WHERE name = ?', [ 'domain' ], function (error, result) {
+        if (error) return callback(error);
+
+        var domain = {};
+        if (result[0]) domain = safe.JSON.parse(result[0].value) || {};
+
+        async.series([
+            db.runSql.bind(db, 'START TRANSACTION;'),
+            function addAppsDomainColumn(done) {
+                db.runSql('ALTER TABLE apps ADD COLUMN domain VARCHAR(128)', [], done);
+            },
+            function setAppDomain(done) {
+                if (!domain.fqdn) return done(); // skip for new cloudrons without a domain
+                db.runSql('UPDATE apps SET domain = ?', [ domain.fqdn ], done);
+            },
+            function addAppsLocationDomainUniqueConstraint(done) {
+                db.runSql('ALTER TABLE apps ADD UNIQUE location_domain_unique_index (location, domain)', [], done);
+            },
+            function removePresetupAdminGroupIfNew(done) {
+                // do not delete on update, will update the record in setMailboxesDomain()
+                if (domain.fqdn) return done();
+
+                // this will be finally created once we have a domain when we create the owner in user.js
+                const ADMIN_GROUP_ID = 'admin'; // see constants.js
+                db.runSql('DELETE FROM groups WHERE id = ?', [ ADMIN_GROUP_ID ], function (error) {
+                    if (error) return done(error);
+
+                    db.runSql('DELETE FROM mailboxes WHERE ownerId = ?', [ ADMIN_GROUP_ID ], done);
+                });
+            },
+            function addMailboxesDomainColumn(done) {
+                db.runSql('ALTER TABLE mailboxes ADD COLUMN domain VARCHAR(128)', [], done);
+            },
+            function setMailboxesDomain(done) {
+                if (!domain.fqdn) return done(); // skip for new cloudrons without a domain
+                db.runSql('UPDATE mailboxes SET domain = ?', [ domain.fqdn ], done);
+            },
+            function dropAppsLocationUniqueConstraint(done) {
+                db.runSql('ALTER TABLE apps DROP INDEX location', [], done);
+            },
+            db.runSql.bind(db, 'COMMIT')
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    async.series([
+        db.runSql.bind(db, 'START TRANSACTION;'),
+        function dropMailboxesDomainColumn(done) {
+            db.runSql('ALTER TABLE mailboxes DROP COLUMN domain', [], done);
+        },
+        function dropLocationDomainUniqueConstraint(done) {
+            db.runSql('ALTER TABLE apps DROP INDEX location_domain_unique_index', [], done);
+        },
+        function dropAppsDomainColumn(done) {
+            db.runSql('ALTER TABLE apps DROP COLUMN domain', [], done);
+        },
+        function addAppsLocationUniqueConstraint(done) {
+            db.runSql('ALTER TABLE apps ADD UNIQUE location (location)', [], done);
+        },
+        db.runSql.bind(db, 'COMMIT')
+    ], callback);
+};
@@ -0,0 +1,61 @@
+'use strict';
+
+var async = require('async'),
+    safe = require('safetydance'),
+    tld = require('tldjs');
+
+exports.up = function(db, callback) {
+    var fqdn, zoneName, configJson;
+
+    async.series([
+        function gatherDomain(done) {
+            db.all('SELECT * FROM settings WHERE name = ?', [ 'domain' ], function (error, result) {
+                if (error) return done(error);
+
+                var domain = {};
+                if (result[0]) domain = safe.JSON.parse(result[0].value) || {};
+
+                fqdn = domain.fqdn || ''; // will be null pre-setup
+                zoneName = domain.zoneName || tld.getDomain(fqdn) || fqdn;
+
+                done();
+            });
+        },
+        function gatherDNSConfig(done) {
+            db.all('SELECT * FROM settings WHERE name = ?', [ 'dns_config' ], function (error, result) {
+                if (error) return done(error);
+
+                configJson = (result[0] && result[0].value) ? result[0].value : JSON.stringify({ provider: 'manual'});
+
+                // caas dns config needs an fqdn
+                var config = JSON.parse(configJson);
+                if (config.provider === 'caas') config.fqdn = fqdn;
+                configJson = JSON.stringify(config);
+
+                done();
+            });
+        },
+        db.runSql.bind(db, 'START TRANSACTION;'),
+        function createDomainsTable(done) {
+            var cmd = `
+                CREATE TABLE domains(
+                domain VARCHAR(128) NOT NULL UNIQUE,
+                zoneName VARCHAR(128) NOT NULL,
+                configJson TEXT,
+                PRIMARY KEY (domain)) CHARACTER SET utf8 COLLATE utf8_bin
+            `;
+
+            db.runSql(cmd, [], done);
+        },
+        function addInitialDomain(done) {
+            if (!fqdn) return done();
+
+            db.runSql('INSERT INTO domains (domain, zoneName, configJson) VALUES (?, ?, ?)', [ fqdn, zoneName, configJson ], done);
+        },
+        db.runSql.bind(db, 'COMMIT')
+    ], callback);
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DROP TABLE domains', callback);
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD CONSTRAINT apps_domain_constraint FOREIGN KEY(domain) REFERENCES domains(domain)', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP FOREIGN KEY apps_domain_constraint', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes ADD CONSTRAINT mailboxes_domain_constraint FOREIGN KEY(domain) REFERENCES domains(domain)', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP FOREIGN KEY mailboxes_domain_constraint', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP PRIMARY KEY', function (error) {
+      if (error) console.error(error);
+      callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes ADD PRIMARY KEY(name)', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes ADD UNIQUE mailboxes_name_domain_unique_index (name, domain)', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE mailboxes DROP INDEX mailboxes_name_domain_unique_index', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN updateTime', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE createdAt creationTime TIMESTAMP(2) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps CHANGE creationTime createdAt TIMESTAMP(2) NOT NULL', [], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,28 @@
+'use strict';
+
+var async = require('async');
+
+// NOTE: This migration is incorrect because 'caas' domain is not guaranteed to be present in all Caas cloudrons
+exports.up = function(db, callback) {
+    db.all('SELECT * FROM domains', function (error, domains) {
+        if (error) return callback(error);
+
+        var caasDomains = domains.filter(function (d) { return JSON.parse(d.configJson).provider === 'caas'; });
+        if (caasDomains.length === 0) return callback();
+        var caasDomain = caasDomains[0].domain;
+
+        db.all('SELECT * FROM settings WHERE name=?', [ 'backup_config' ], function (error, settings) {
+            if (error) return callback(error);
+
+            var setting = settings[0];
+            var config = JSON.parse(setting.value);
+            config.fqdn = caasDomain;
+
+            db.runSql('UPDATE settings SET value=? WHERE name=?', [ JSON.stringify(config), setting.name ], callback);
+        });
+    });
+};
+
+exports.down = function(db, callback) {
+    callback();
+};
@@ -0,0 +1,23 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    var backupConfig = {
+        "provider": "filesystem",
+        "backupFolder": "/var/backups",
+        "format": "tgz",
+        "retentionSecs": 172800
+    };
+
+    db.runSql('INSERT settings (name, value) VALUES(?, ?)', [ 'backup_config', JSON.stringify(backupConfig) ], function (error) {
+        if (!error || error.code === 'ER_DUP_ENTRY') return callback(); // dup entry is OK for existing cloudrons
+
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('DELETE FROM settings WHERE name=?', ['backup_config'], function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -0,0 +1,33 @@
+'use strict';
+
+var async = require('async');
+
+exports.up = function(db, callback) {
+    // first check precondtion of domain entry in settings
+    db.all('SELECT * FROM domains', [ ], function (error, domains) {
+        if (error) return callback(error);
+
+        async.series([
+            db.runSql.bind(db, 'START TRANSACTION;'),
+            db.runSql.bind(db, 'ALTER TABLE domains ADD COLUMN provider VARCHAR(16) DEFAULT ""'),
+            function setProvider(done) {
+                async.eachSeries(domains, function (domain, iteratorCallback) {
+                    var config = JSON.parse(domain.configJson);
+                    var provider = config.provider;
+                    delete config.provider;
+
+                    db.runSql('UPDATE domains SET provider = ?, configJson = ? WHERE domain = ?', [ provider, JSON.stringify(config), domain.domain ], iteratorCallback);
+                }, done);
+            },
+            db.runSql.bind(db, 'ALTER TABLE domains MODIFY provider VARCHAR(16) NOT NULL'),
+            db.runSql.bind(db, 'COMMIT')
+        ], callback);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE domains DROP COLUMN provider', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
@@ -9,6 +9,10 @@
 #### BLOB - stored offline from table row (use for binary data)
 #### https://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html

+# The code uses zero dates. Make sure sql_mode does NOT have NO_ZERO_DATE
+# http://johnemb.blogspot.com/2014/09/adding-or-removing-individual-sql-modes.html
+# SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'NO_ZERO_DATE',''));
+
 CREATE TABLE IF NOT EXISTS users(
    id VARCHAR(128) NOT NULL UNIQUE,
    username VARCHAR(254) UNIQUE,
@@ -59,21 +63,26 @@ CREATE TABLE IF NOT EXISTS apps(
    containerId VARCHAR(128),
    manifestJson TEXT,
    httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
-    location VARCHAR(128) NOT NULL UNIQUE,
-    dnsRecordId VARCHAR(512), // tracks any id that we got back to track dns updates (unused)
+    location VARCHAR(128) NOT NULL,
+    domain VARCHAR(128) NOT NULL,
+    dnsRecordId VARCHAR(512), // tracks any id that we got back to track dns updates
    accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
    createdAt TIMESTAMP(2) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    updatedAt TIMESTAMP(2) NOT NULL DEFAULT CURRENT_TIMESTAMP,
    memoryLimit BIGINT DEFAULT 0,
    altDomain VARCHAR(256),
    xFrameOptions VARCHAR(512),
    sso BOOLEAN DEFAULT 1, // whether user chose to enable SSO
    debugModeJson TEXT, // options for development mode
    robotsTxt TEXT,
+    enableBackup BOOLEAN DEFAULT 1, // misnomer: controls automatic daily backups

    // the following fields do not belong here, they can be removed when we use a queue for apptask
-    lastBackupId VARCHAR(128), // used to pass backupId to restore from to apptask
-    oldConfigJson TEXT, // used to pass old config for apptask
+    restoreConfigJson VARCHAR(256), // used to pass backupId to restore from to apptask
+    oldConfigJson TEXT, // used to pass old config for apptask (configure, restore)
+    updateConfigJson TEXT, // used to pass new config for apptask (update)

+    FOREIGN KEY(domain) REFERENCES domains(domain),
    PRIMARY KEY(id));

 CREATE TABLE IF NOT EXISTS appPortBindings(
@@ -92,7 +101,7 @@ CREATE TABLE IF NOT EXISTS authcodes(

 CREATE TABLE IF NOT EXISTS settings(
    name VARCHAR(128) NOT NULL UNIQUE,
-    value VARCHAR(512),
+    value TEXT,
    PRIMARY KEY(name));

 CREATE TABLE IF NOT EXISTS appAddonConfigs(
@@ -109,7 +118,8 @@ CREATE TABLE IF NOT EXISTS backups(
    type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
    dependsOn TEXT, /* comma separate list of objects this backup depends on */
    state VARCHAR(16) NOT NULL,
-    restoreConfigJson TEXT, /* JSON including the manifest of the backed up app */
+    manifestJson TEXT, /* to validate if the app can be installed in this version of box */
+    format VARCHAR(16) DEFAULT "tgz",

    PRIMARY KEY (id));

@@ -132,5 +142,18 @@ CREATE TABLE IF NOT EXISTS mailboxes(
    ownerType VARCHAR(16) NOT NULL, /* 'app' or 'user' or 'group' */
    aliasTarget VARCHAR(128), /* the target name type is an alias */
    creationTime TIMESTAMP,
+    domain VARCHAR(128),

+    FOREIGN KEY(domain) REFERENCES domains(domain),
    PRIMARY KEY (name));
+
+CREATE TABLE IF NOT EXISTS domains(
+    domain VARCHAR(128) NOT NULL UNIQUE, /* if this needs to be larger, InnoDB has a limit of 767 bytes for PRIMARY KEY values! */
+    zoneName VARCHAR(128) NOT NULL, /* this mostly contains the domain itself again */
+    provider VARCHAR(16) NOT NULL,
+    configJson TEXT, /* JSON containing the dns backend provider config */
+
+    PRIMARY KEY (domain))
+
+    /* the default db collation is utf8mb4_unicode_ci but for the app table domain constraint we have to use the old one */
+    CHARACTER SET utf8 COLLATE utf8_bin;
@@ -1,105 +1,108 @@
 {
-  "name": "Cloudron",
+  "name": "cloudron",
  "description": "Main code for a cloudron",
-  "version": "0.0.1",
-  "private": "true",
+  "version": "1.0.0",
+  "private": true,
  "author": {
    "name": "Cloudron authors"
  },
  "repository": {
-    "type": "git"
+    "type": "git",
+    "url": "https://git.cloudron.io/cloudron/box.git"
+  },
+  "engines": {
+    "node": ">=4.0.0 <=4.1.1"
  },
-  "engines": [
-    "node >=4.0.0 <=4.1.1"
-  ],
  "dependencies": {
+    "@google-cloud/dns": "^0.7.0",
+    "@google-cloud/storage": "^1.2.1",
    "@sindresorhus/df": "^2.1.0",
-    "async": "^2.1.4",
-    "aws-sdk": "^2.41.0",
-    "body-parser": "^1.13.1",
-    "cloudron-manifestformat": "^2.8.0",
+    "async": "^2.6.0",
+    "aws-sdk": "^2.151.0",
+    "body-parser": "^1.18.2",
+    "cloudron-manifestformat": "^2.10.0",
    "connect-ensure-login": "^0.1.1",
-    "connect-lastmile": "^0.1.0",
-    "connect-timeout": "^1.5.0",
+    "connect-lastmile": "^1.0.2",
+    "connect-timeout": "^1.9.0",
    "cookie-parser": "^1.3.5",
-    "cookie-session": "^1.1.0",
-    "cron": "^1.0.9",
+    "cookie-session": "^1.3.2",
+    "cron": "^1.3.0",
    "csurf": "^1.6.6",
-    "db-migrate": "^0.10.0-beta.20",
+    "db-migrate": "^0.10.0-beta.24",
    "db-migrate-mysql": "^1.1.10",
-    "debug": "^2.2.0",
-    "dockerode": "^2.4.3",
-    "ejs": "^2.2.4",
-    "ejs-cli": "^1.2.0",
-    "express": "^4.12.4",
-    "express-session": "^1.11.3",
-    "gulp-sass": "^3.0.0",
+    "debug": "^3.1.0",
+    "dockerode": "^2.5.3",
+    "ejs": "^2.5.7",
+    "ejs-cli": "^2.0.0",
+    "express": "^4.16.2",
+    "express-session": "^1.15.6",
    "hat": "0.0.3",
-    "hock": "https://registry.npmjs.org/hock/-/hock-1.3.2.tgz",
    "json": "^9.0.3",
    "ldapjs": "^1.0.0",
-    "mime": "^1.3.4",
-    "moment-timezone": "^0.5.5",
-    "morgan": "^1.7.0",
+    "lodash.chunk": "^4.2.0",
+    "mime": "^2.0.3",
+    "moment-timezone": "^0.5.14",
+    "morgan": "^1.9.0",
    "multiparty": "^4.1.2",
-    "mysql": "^2.7.0",
-    "node-uuid": "^1.4.3",
-    "nodemailer": "^4.0.1",
+    "mysql": "^2.15.0",
+    "nodemailer": "^4.4.0",
    "nodemailer-smtp-transport": "^2.7.4",
-    "oauth2orize": "^1.0.1",
+    "oauth2orize": "^1.11.0",
    "once": "^1.3.2",
    "parse-links": "^0.1.0",
-    "passport": "^0.2.2",
-    "passport-http": "^0.2.2",
+    "passport": "^0.4.0",
+    "passport-http": "^0.3.0",
    "passport-http-bearer": "^1.0.1",
    "passport-local": "^1.0.0",
    "passport-oauth2-client-password": "^0.1.2",
-    "password-generator": "^2.0.2",
+    "password-generator": "^2.2.0",
    "progress-stream": "^2.0.0",
-    "proxy-middleware": "^0.13.0",
+    "proxy-middleware": "^0.15.0",
+    "recursive-readdir": "^2.2.1",
+    "request": "^2.83.0",
    "s3-block-read-stream": "^0.2.0",
-    "safetydance": "^0.2.0",
-    "semver": "^4.3.6",
-    "showdown": "^1.6.0",
+    "safetydance": "^0.7.1",
+    "semver": "^5.4.1",
+    "showdown": "^1.8.2",
    "split": "^1.0.0",
-    "superagent": "^1.8.3",
+    "superagent": "^3.8.1",
    "supererror": "^0.7.1",
-    "tar-fs": "https://registry.npmjs.org/tar-fs/-/tar-fs-1.15.2.tgz",
-    "tldjs": "^1.6.2",
+    "tar-fs": "^1.16.0",
+    "tar-stream": "^1.5.5",
+    "tldjs": "^2.2.0",
    "underscore": "^1.7.0",
+    "uuid": "^3.1.0",
    "valid-url": "^1.0.9",
-    "validator": "^4.9.0"
+    "validator": "^9.1.1",
+    "ws": "^3.3.1"
  },
  "devDependencies": {
    "bootstrap-sass": "^3.3.3",
-    "deep-extend": "^0.4.1",
-    "del": "^1.1.1",
    "expect.js": "*",
-    "gulp": "^3.8.11",
-    "gulp-autoprefixer": "^2.3.0",
+    "gulp": "^3.9.1",
+    "gulp-autoprefixer": "^4.0.0",
    "gulp-concat": "^2.4.3",
    "gulp-cssnano": "^2.1.0",
-    "gulp-ejs": "^1.0.0",
-    "gulp-sass": "^3.0.0",
+    "gulp-ejs": "^3.1.0",
+    "gulp-sass": "^3.1.0",
    "gulp-serve": "^1.0.0",
-    "gulp-sourcemaps": "^1.5.2",
-    "gulp-uglify": "^1.1.0",
-    "hock": "~1.2.0",
+    "gulp-sourcemaps": "^2.6.1",
+    "gulp-uglify": "^3.0.0",
+    "hock": "^1.3.2",
    "istanbul": "*",
-    "js2xmlparser": "^1.0.0",
+    "js2xmlparser": "^3.0.0",
    "mocha": "*",
-    "mock-aws-s3": "^2.4.0",
-    "nock": "^9.0.2",
-    "node-sass": "^3.0.0-alpha.0",
+    "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
+    "nock": "^9.0.14",
+    "node-sass": "^4.6.1",
    "readdirp": "https://registry.npmjs.org/readdirp/-/readdirp-2.1.0.tgz",
-    "request": "^2.65.0",
-    "yargs": "^3.15.0"
+    "yargs": "^10.0.3"
  },
  "scripts": {
    "migrate_local": "DATABASE_URL=mysql://root:@localhost/box node_modules/.bin/db-migrate up",
    "migrate_test": "BOX_ENV=test DATABASE_URL=mysql://root:@localhost/boxtest node_modules/.bin/db-migrate up",
-    "test": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- -R spec ./src/test ./src/routes/test/[^a]*",
-    "test_all": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- -R spec ./src/test ./src/routes/test",
+    "test": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- --exit -R spec ./src/test ./src/routes/test/[^a]*",
+    "test_all": "npm run migrate_test && src/test/setupTest && BOX_ENV=test ./node_modules/istanbul/lib/cli.js test $1 ./node_modules/mocha/bin/_mocha -- --exit -R spec ./src/test ./src/routes/test",
    "postmerge": "/bin/true",
    "precommit": "/bin/true",
    "prepush": "npm test",
@@ -45,11 +45,9 @@ fi
 initBaseImage="true"
 # provisioning data
 domain=""
+adminLocation="my"
 zoneName=""
 provider=""
-encryptionKey=""
-restoreUrl=""
-dnsProvider="manual"
 tlsProvider="le-prod"
 requestedVersion=""
 apiServerOrigin="https://api.cloudron.io"
@@ -60,31 +58,30 @@ sourceTarballUrl=""
 rebootServer="true"
 baseDataDir=""

-# TODO this is still there for the restore case, see other occasions below
-versionsUrl="https://s3.amazonaws.com/prod-cloudron-releases/versions.json"
+# these are here for pre-1.9 compat
+encryptionKey=""
+restoreUrl=""

-args=$(getopt -o "" -l "domain:,help,skip-baseimage-init,data:,data-dir:,provider:,encryption-key:,restore-url:,tls-provider:,version:,dns-provider:,env:,prerelease,skip-reboot,source-url:" -n "$0" -- "$@")
+args=$(getopt -o "" -l "domain:,help,skip-baseimage-init,data:,data-dir:,provider:,encryption-key:,restore-url:,tls-provider:,version:,dns-provider:,env:,admin-location:,prerelease,skip-reboot,source-url:" -n "$0" -- "$@")
 eval set -- "${args}"

 while true; do
    case "$1" in
    --domain) domain="$2"; shift 2;;
-    --help) echo "See https://cloudron.io/references/selfhosting.html on how to install Cloudron"; exit 0;;
+    --admin-location) adminLocation="$2"; shift 2;;
+    --help) echo "See https://cloudron.io/documentation/installation/ on how to install Cloudron"; exit 0;;
    --provider) provider="$2"; shift 2;;
    --encryption-key) encryptionKey="$2"; shift 2;;
    --restore-url) restoreUrl="$2"; shift 2;;
    --tls-provider) tlsProvider="$2"; shift 2;;
-    --dns-provider) dnsProvider="$2"; shift 2;;
    --version) requestedVersion="$2"; shift 2;;
    --env)
        if [[ "$2" == "dev" ]]; then
-            versionsUrl="https://s3.amazonaws.com/dev-cloudron-releases/versions.json"
            apiServerOrigin="https://api.dev.cloudron.io"
            webServerOrigin="https://dev.cloudron.io"
            tlsProvider="le-staging"
            prerelease="true"
        elif [[ "$2" == "staging" ]]; then
-            versionsUrl="https://s3.amazonaws.com/staging-cloudron-releases/versions.json"
            apiServerOrigin="https://api.staging.cloudron.io"
            webServerOrigin="https://staging.cloudron.io"
            tlsProvider="le-staging"
@@ -105,13 +102,15 @@ done
 # validate arguments in the absence of data
 if [[ -z "${dataJson}" ]]; then
    if [[ -z "${provider}" ]]; then
-        echo "--provider is required (azure, digitalocean, ec2, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic)"
+        echo "--provider is required (azure, cloudscale.ch, digitalocean, ec2, exoscale, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic)"
        exit 1
    elif [[  \
                "${provider}" != "ami" && \
                "${provider}" != "azure" && \
+                "${provider}" != "cloudscale.ch" && \
                "${provider}" != "digitalocean" && \
                "${provider}" != "ec2" && \
+                "${provider}" != "exoscale" && \
                "${provider}" != "gce" && \
                "${provider}" != "lightsail" && \
                "${provider}" != "linode" && \
@@ -121,7 +120,7 @@ if [[ -z "${dataJson}" ]]; then
                "${provider}" != "vultr" && \
                "${provider}" != "generic" \
            ]]; then
-        echo "--provider must be one of: azure, digitalocean, ec2, gce, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic"
+        echo "--provider must be one of: azure, cloudscale.ch, digitalocean, ec2, exoscale, gce, lightsail, linode, ovh, rosehosting, scaleway, vultr or generic"
        exit 1
    fi

@@ -130,14 +129,6 @@ if [[ -z "${dataJson}" ]]; then
        exit 1
    fi

-    if [[ -z "${dnsProvider}" ]]; then
-        echo "--dns-provider is required (noop, manual)"
-        exit 1
-    elif [[ "${dnsProvider}" != "noop" && "${dnsProvider}" != "manual" ]]; then
-        echo "--dns-provider must be one of : manual, noop"
-        exit 1
-    fi
-
    if [[ -n "${baseDataDir}" && ! -d "${baseDataDir}" ]]; then
        echo "${baseDataDir} does not exist"
        exit 1
@@ -188,41 +179,39 @@ if [[ "${sourceTarballUrl}" == "" ]]; then
 fi

 # Build data
-# TODO versionsUrl is still there for the cloudron restore case
+# tlsConfig, dnsConfig, backupConfig are here for backward compat with < 1.9
+# from 1.9, we use autoprovision.json
 if [[ -z "${dataJson}" ]]; then
    if [[ -z "${restoreUrl}" ]]; then
        data=$(cat <<EOF
    {
-        "boxVersionsUrl": "${versionsUrl}",
        "fqdn": "${domain}",
+        "adminLocation": "${adminLocation}",
+        "adminFqdn": "${adminLocation}.${domain}",
        "zoneName": "${zoneName}",
        "provider": "${provider}",
        "apiServerOrigin": "${apiServerOrigin}",
        "webServerOrigin": "${webServerOrigin}",
+        "version": "${version}",
        "tlsConfig": {
            "provider": "${tlsProvider}"
        },
-        "dnsConfig": {
-            "provider": "${dnsProvider}"
-        },
        "backupConfig" : {
            "provider": "filesystem",
            "backupFolder": "/var/backups",
            "key": "${encryptionKey}",
+            "format": "tgz",
            "retentionSecs": 172800
-        },
-        "updateConfig": {
-            "prerelease": ${prerelease}
-        },
-        "version": "${version}"
+        }
    }
 EOF
    )
    else
        data=$(cat <<EOF
    {
-        "boxVersionsUrl": "${versionsUrl}",
        "fqdn": "${domain}",
+        "adminLocation": "${adminLocation}",
+        "adminFqdn": "${adminLocation}.${domain}",
        "zoneName": "${zoneName}",
        "provider": "${provider}",
        "apiServerOrigin": "${apiServerOrigin}",
@@ -259,17 +248,9 @@ fi

 echo "=> Installing version ${version} (this takes some time) ..."
 echo "${data}" > "${DATA_FILE}"
-# poor mans semver
-if [[ ${version} == "0.10"* ]]; then
-    if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" --data-file "${DATA_FILE}" &>> "${LOG_FILE}"; then
-        echo "Failed to install cloudron. See ${LOG_FILE} for details"
-        exit 1
-    fi
-else
-    if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" --data-file "${DATA_FILE}" --data-dir "${baseDataDir}" &>> "${LOG_FILE}"; then
-        echo "Failed to install cloudron. See ${LOG_FILE} for details"
-        exit 1
-    fi
+if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" --data-file "${DATA_FILE}" --data-dir "${baseDataDir}" &>> "${LOG_FILE}"; then
+    echo "Failed to install cloudron. See ${LOG_FILE} for details"
+    exit 1
 fi
 rm "${DATA_FILE}"

@@ -283,6 +264,17 @@ while true; do
    sleep 10
 done

+autoprovision_data=$(cat <<EOF
+{
+    "tlsConfig": {
+        "provider": "${tlsProvider}"
+    }
+}
+EOF
+)
+
+echo "${autoprovision_data}" > /home/yellowtent/configs/autoprovision.json
+
 if [[ -n "${domain}" ]]; then
    echo -e "\n\nVisit https://my.${domain} to finish setup once the server has rebooted.\n"
 else
@@ -31,8 +31,8 @@ if ! $(cd "${SOURCE_DIR}" && git diff --exit-code >/dev/null); then
    exit 1
 fi

-if [[ "$(node --version)" != "v6.11.2" ]]; then
-    echo "This script requires node 6.11.2"
+if [[ "$(node --version)" != "v8.9.3" ]]; then
+    echo "This script requires node 8.9.3"
    exit 1
 fi

@@ -44,7 +44,7 @@ chmod "o+rx,g+rx" "${bundle_dir}" # otherwise extracted tarball director won't b
 echo "Checking out code [${version}] into ${bundle_dir}"
 (cd "${SOURCE_DIR}" && git archive --format=tar ${version} | (cd "${bundle_dir}" && tar xf -))

-if diff "${TMPDIR}/boxtarball.cache/npm-shrinkwrap.json.all" "${bundle_dir}/npm-shrinkwrap.json" >/dev/null 2>&1; then
+if diff "${TMPDIR}/boxtarball.cache/package-lock.json.all" "${bundle_dir}/package-lock.json" >/dev/null 2>&1; then
    echo "Reusing dev modules from cache"
    cp -r "${TMPDIR}/boxtarball.cache/node_modules-all/." "${bundle_dir}/node_modules"
 else
@@ -54,7 +54,7 @@ else
    echo "Caching dev dependencies"
    mkdir -p "${TMPDIR}/boxtarball.cache/node_modules-all"
    rsync -a --delete "${bundle_dir}/node_modules/" "${TMPDIR}/boxtarball.cache/node_modules-all/"
-    cp "${bundle_dir}/npm-shrinkwrap.json" "${TMPDIR}/boxtarball.cache/npm-shrinkwrap.json.all"
+    cp "${bundle_dir}/package-lock.json" "${TMPDIR}/boxtarball.cache/package-lock.json.all"
 fi

 echo "Building webadmin assets"
@@ -65,7 +65,7 @@ rm -rf "${bundle_dir}/node_modules/"
 rm -rf "${bundle_dir}/webadmin/src"
 rm -rf "${bundle_dir}/gulpfile.js"

-if diff "${TMPDIR}/boxtarball.cache/npm-shrinkwrap.json.prod" "${bundle_dir}/npm-shrinkwrap.json" >/dev/null 2>&1; then
+if diff "${TMPDIR}/boxtarball.cache/package-lock.json.prod" "${bundle_dir}/package-lock.json" >/dev/null 2>&1; then
    echo "Reusing prod modules from cache"
    cp -r "${TMPDIR}/boxtarball.cache/node_modules-prod/." "${bundle_dir}/node_modules"
 else
@@ -75,7 +75,7 @@ else
    echo "Caching prod dependencies"
    mkdir -p "${TMPDIR}/boxtarball.cache/node_modules-prod"
    rsync -a --delete "${bundle_dir}/node_modules/" "${TMPDIR}/boxtarball.cache/node_modules-prod/"
-    cp "${bundle_dir}/npm-shrinkwrap.json" "${TMPDIR}/boxtarball.cache/npm-shrinkwrap.json.prod"
+    cp "${bundle_dir}/package-lock.json" "${TMPDIR}/boxtarball.cache/package-lock.json.prod"
 fi

 echo "Create final tarball"
@@ -34,13 +34,41 @@ while true; do
    esac
 done

+echo "==> installer: updating docker"
+if [[ $(docker version --format {{.Client.Version}}) != "17.09.0-ce" ]]; then
+    $curl -sL https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/docker-ce_17.09.0~ce-0~ubuntu_amd64.deb -o /tmp/docker.deb
+
+    # https://download.docker.com/linux/ubuntu/dists/xenial/stable/binary-amd64/Packages
+    if [[ $(sha256sum /tmp/docker.deb | cut -d' ' -f1) != "d33f6eb134f0ab0876148bd96de95ea47d583d7f2cddfdc6757979453f9bd9bf" ]]; then
+        echo "docker binary download is corrupt"
+        exit 5
+    fi
+
+    echo "Waiting for all dpkg tasks to finish..."
+    while fuser /var/lib/dpkg/lock; do
+        sleep 1
+    done
+
+    while ! dpkg --force-confold --configure -a; do
+        echo "Failed to fix packages. Retry"
+        sleep 1
+    done
+
+    while ! apt install -y /tmp/docker.deb; do
+        echo "Failed to install docker. Retry"
+        sleep 1
+    done
+
+    rm /tmp/docker.deb
+fi
+
 echo "==> installer: updating node"
-if [[ "$(node --version)" != "v6.11.2" ]]; then
-    mkdir -p /usr/local/node-6.11.2
-    $curl -sL https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-6.11.2
-    ln -sf /usr/local/node-6.11.2/bin/node /usr/bin/node
-    ln -sf /usr/local/node-6.11.2/bin/npm /usr/bin/npm
-    rm -rf /usr/local/node-6.11.1
+if [[ "$(node --version)" != "v8.9.3" ]]; then
+    mkdir -p /usr/local/node-8.9.3
+    $curl -sL https://nodejs.org/dist/v8.9.3/node-v8.9.3-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-8.9.3
+    ln -sf /usr/local/node-8.9.3/bin/node /usr/bin/node
+    ln -sf /usr/local/node-8.9.3/bin/npm /usr/bin/npm
+    rm -rf /usr/local/node-6.11.5
 fi

 for try in `seq 1 10`; do
@@ -81,9 +109,6 @@ fi
 # ensure we are not inside the source directory, which we will remove now
 cd /root

-echo "==> installer: updating packages"
-# add logic to update apt packages here
-
 echo "==> installer: switching the box code"
 rm -rf "${BOX_SRC_DIR}"
 mv "${box_src_tmp_dir}" "${BOX_SRC_DIR}"
@@ -6,23 +6,16 @@ json="${source_dir}/../node_modules/.bin/json"
 # IMPORTANT: Fix cloudron.js:doUpdate if you add/remove any arg. keep these sorted for readability
 arg_api_server_origin=""
 arg_fqdn=""
+arg_admin_location=""
+arg_admin_fqdn=""
 arg_zone_name=""
-arg_is_custom_domain="false"
-arg_restore_key=""
-arg_restore_url=""
+arg_is_custom_domain="false" # can be removed after 1.9
 arg_retire_reason=""
 arg_retire_info=""
-arg_tls_config=""
-arg_tls_cert=""
-arg_tls_key=""
 arg_token=""
 arg_version=""
 arg_web_server_origin=""
-arg_backup_config=""
-arg_dns_config=""
-arg_update_config=""
 arg_provider=""
-arg_app_bundle=""
 arg_is_demo="false"

 args=$(getopt -o "" -l "data:,retire-reason:,retire-info:" -n "$0" -- "$@")
@@ -41,54 +34,35 @@ while true; do
    --data)
        # these params must be valid in all cases
        arg_fqdn=$(echo "$2" | $json fqdn)
+        arg_admin_fqdn=$(echo "$2" | $json adminFqdn)
        arg_zone_name=$(echo "$2" | $json zoneName)
+        [[ "${arg_zone_name}" == "" ]] && arg_zone_name="${arg_fqdn}"

+        # can be removed after 1.9
        arg_is_custom_domain=$(echo "$2" | $json isCustomDomain)
        [[ "${arg_is_custom_domain}" == "" ]] && arg_is_custom_domain="true"

+        arg_admin_location=$(echo "$2" | $json adminLocation)
+        [[ "${arg_admin_location}" == "" ]] && arg_admin_location="my"
+
        # only update/restore have this valid (but not migrate)
        arg_api_server_origin=$(echo "$2" | $json apiServerOrigin)
        [[ "${arg_api_server_origin}" == "" ]] && arg_api_server_origin="https://api.cloudron.io"
        arg_web_server_origin=$(echo "$2" | $json webServerOrigin)
        [[ "${arg_web_server_origin}" == "" ]] && arg_web_server_origin="https://cloudron.io"

-        # TODO check if an where this is used
+        # TODO check if and where this is used
        arg_version=$(echo "$2" | $json version)

        # read possibly empty parameters here
-        arg_app_bundle=$(echo "$2" | $json appBundle)
-        [[ "${arg_app_bundle}" == "" ]] && arg_app_bundle="[]"
-
        arg_is_demo=$(echo "$2" | $json isDemo)
        [[ "${arg_is_demo}" == "" ]] && arg_is_demo="false"

-        arg_tls_cert=$(echo "$2" | $json tlsCert)
-        [[ "${arg_tls_cert}" == "null" ]] && arg_tls_cert=""
-        arg_tls_key=$(echo "$2" | $json tlsKey)
-        [[ "${arg_tls_key}" == "null" ]] && arg_tls_key=""
        arg_token=$(echo "$2" | $json token)

        arg_provider=$(echo "$2" | $json provider)
        [[ "${arg_provider}" == "" ]] && arg_provider="generic"

-        arg_tls_config=$(echo "$2" | $json tlsConfig)
-        [[ "${arg_tls_config}" == "null" ]] && arg_tls_config=""
-
-        arg_restore_url=$(echo "$2" | $json restore.url)
-        [[ "${arg_restore_url}" == "null" ]] && arg_restore_url=""
-
-        arg_restore_key=$(echo "$2" | $json restore.key)
-        [[ "${arg_restore_key}" == "null" ]] && arg_restore_key=""
-
-        arg_backup_config=$(echo "$2" | $json backupConfig)
-        [[ "${arg_backup_config}" == "null" ]] && arg_backup_config=""
-
-        arg_dns_config=$(echo "$2" | $json dnsConfig)
-        [[ "${arg_dns_config}" == "null" ]] && arg_dns_config=""
-
-        arg_update_config=$(echo "$2" | $json updateConfig)
-        [[ "${arg_update_config}" == "null" ]] && arg_update_config=""
-
        shift 2
        ;;
    --) break;;
@@ -100,13 +74,8 @@ echo "Parsed arguments:"
 echo "api server: ${arg_api_server_origin}"
 echo "fqdn: ${arg_fqdn}"
 echo "custom domain: ${arg_is_custom_domain}"
-echo "restore url: ${arg_restore_url}"
-echo "tls cert: ${arg_tls_cert}"
 # do not dump these as they might become available via logs API
-#echo "restore key: ${arg_restore_key}"
-#echo "tls key: ${arg_tls_key}"
 #echo "token: ${arg_token}"
-echo "tlsConfig: ${arg_tls_config}"
 echo "version: ${arg_version}"
 echo "web server: ${arg_web_server_origin}"
 echo "provider: ${arg_provider}"
@@ -7,7 +7,6 @@ readonly SETUP_WEBSITE_DIR="/home/yellowtent/setup/website"
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 readonly box_src_dir="$(realpath ${script_dir}/..)"
 readonly PLATFORM_DATA_DIR="/home/yellowtent/platformdata"
-readonly ADMIN_LOCATION="my" # keep this in sync with constants.js

 echo "Setting up nginx update page"

@@ -19,8 +18,7 @@ fi
 source "${script_dir}/argparser.sh" "$@" # this injects the arg_* variables used below

 # keep this is sync with config.js appFqdn()
-admin_fqdn=$([[ "${arg_is_custom_domain}" == "true" ]] && echo "${ADMIN_LOCATION}.${arg_fqdn}" ||  echo "${ADMIN_LOCATION}-${arg_fqdn}")
-admin_origin="https://${admin_fqdn}"
+admin_origin="https://${arg_admin_fqdn}"

 # copy the website
 rm -rf "${SETUP_WEBSITE_DIR}" && mkdir -p "${SETUP_WEBSITE_DIR}"
@@ -34,11 +32,11 @@ if [[ "${arg_retire_reason}" != "" || "${existing_infra}" != "${current_infra}"
    echo "Showing progress bar on all subdomains in retired mode or infra update. retire: ${arg_retire_reason} existing: ${existing_infra} current: ${current_infra}"
    rm -f ${PLATFORM_DATA_DIR}/nginx/applications/*
    ${box_src_dir}/node_modules/.bin/ejs-cli -f "${script_dir}/start/nginx/appconfig.ejs" \
-        -O "{ \"vhost\": \"~^(.+)\$\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
+        -O "{ \"vhost\": \"~^(.+)\$\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null, \"hasIPv6\": false }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
 else
    echo "Show progress bar only on admin domain for normal update"
    ${box_src_dir}/node_modules/.bin/ejs-cli -f "${script_dir}/start/nginx/appconfig.ejs" \
-        -O "{ \"vhost\": \"${admin_fqdn}\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
+        -O "{ \"vhost\": \"${arg_admin_fqdn}\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null, \"hasIPv6\": false }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
 fi

 if [[ "${arg_retire_reason}" == "migrate" ]]; then
@@ -7,7 +7,6 @@ echo "==> Cloudron Start"
 readonly USER="yellowtent"
 readonly HOME_DIR="/home/${USER}"
 readonly BOX_SRC_DIR="${HOME_DIR}/box"
-readonly OLD_DATA_DIR="${HOME_DIR}/data";
 readonly PLATFORM_DATA_DIR="${HOME_DIR}/platformdata" # platform data
 readonly APPS_DATA_DIR="${HOME_DIR}/appsdata" # app data
 readonly BOX_DATA_DIR="${HOME_DIR}/boxdata" # box data
@@ -73,54 +72,29 @@ fi

 mkdir -p "${BOX_DATA_DIR}"
 mkdir -p "${APPS_DATA_DIR}"
-mkdir -p "${PLATFORM_DATA_DIR}"

 # keep these in sync with paths.js
 echo "==> Ensuring directories"
-if [[ ! -d "${PLATFORM_DATA_DIR}/mail" ]]; then
-    if [[ -d "${OLD_DATA_DIR}/mail" ]]; then
-        echo "==> Migrate old mail data"
-        # Migrate mail data to new format
-        docker stop mail || true # otherwise the move below might fail if mail container writes in the middle
-        mkdir -p "${PLATFORM_DATA_DIR}/mail"
-        # we can't move the whole folder as it is a btrfs subvolume mount
-        mv -f "${OLD_DATA_DIR}/mail/"* "${PLATFORM_DATA_DIR}/mail/" # this used to be mail container's run directory
-    else
-        echo "==> Create new mail data dir"
-        mkdir -p "${PLATFORM_DATA_DIR}/mail"
-    fi
-fi

 mkdir -p "${PLATFORM_DATA_DIR}/graphite"
-mkdir -p "${PLATFORM_DATA_DIR}/mail/dkim"
 mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
-mkdir -p "${PLATFORM_DATA_DIR}/snapshots"
 mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
+mkdir -p "${PLATFORM_DATA_DIR}/backup"

 mkdir -p "${BOX_DATA_DIR}/appicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
+mkdir -p "${BOX_DATA_DIR}/mail/dkim"

 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
 chmod 777 /var/backups

-echo "==> Check for old btrfs volumes"
-if mountpoint -q "${OLD_DATA_DIR}"; then
-    echo "==> Cleanup btrfs volumes"
-    # First stop all container to be able to unmount
-    docker ps -q | xargs docker stop
-    umount "${OLD_DATA_DIR}"
-    rm -rf "/root/user_data.img"
-else
-    echo "==> No btrfs volumes found";
-fi
-
 echo "==> Configuring journald"
 sed -e "s/^#SystemMaxUse=.*$/SystemMaxUse=100M/" \
    -e "s/^#ForwardToSyslog=.*$/ForwardToSyslog=no/" \
@@ -146,7 +120,10 @@ echo "==> Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
 # We listen on 0.0.0.0 because there is no way control ordering of docker (which creates the 172.18.0.0/16) and unbound
-echo -e "server:\n\tinterface: 0.0.0.0\n\taccess-control: 127.0.0.1 allow\n\taccess-control: 172.18.0.1/16 allow\n\tcache-max-negative-ttl: 30\n\tcache-max-ttl: 300" > /etc/unbound/unbound.conf.d/cloudron-network.conf
+# If IP6 is not enabled, dns queries seem to fail on some hosts
+echo -e "server:\n\tinterface: 0.0.0.0\n\tdo-ip6: yes\n\taccess-control: 127.0.0.1 allow\n\taccess-control: 172.18.0.1/16 allow\n\tcache-max-negative-ttl: 30\n\tcache-max-ttl: 300\n\t#logfile: /var/log/unbound.log\n\t#verbosity: 10" > /etc/unbound/unbound.conf.d/cloudron-network.conf
+# update the root anchor after a out-of-disk-space situation (see #269)
+unbound-anchor -a /var/lib/unbound/root.key

 echo "==> Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
@@ -209,7 +186,11 @@ if [[ ! -f /etc/mysql/mysql.cnf ]] || ! diff -q "${script_dir}/start/mysql.cnf"
        echo "Waiting for mysql jobs..."
        sleep 1
    done
-    systemctl restart mysql
+    while true; do
+        if systemctl restart mysql; then break; fi
+        echo "Restarting MySql again after sometime since this fails randomly"
+        sleep 1
+    done
 else
    systemctl start mysql
 fi
@@ -218,38 +199,20 @@ readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'

-if [[ -n "${arg_restore_url}" ]]; then
-    set_progress "30" "Downloading restore data"
-
-    decrypt=""
-    if [[ "${arg_restore_url}" == *.tar.gz.enc || -n "${arg_restore_key}" ]]; then
-        echo "==> Downloading encrypted backup: ${arg_restore_url} and key: ${arg_restore_key}"
-        decrypt=(openssl aes-256-cbc -d -nosalt -pass "pass:${arg_restore_key}")
-    else
-        echo "==> Downloading backup: ${arg_restore_url}"
-        decrypt=(cat -)
-    fi
-
-    while true; do
-        if $curl -L "${arg_restore_url}" | "${decrypt[@]}" \
-        | tar -zxf - --overwrite --transform="s,^box/\?,boxdata/," --transform="s,^mail/\?,platformdata/mail/," --show-transformed-names -C "${HOME_DIR}"; then break; fi
-        echo "Failed to download data, trying again"
-    done
-
-    set_progress "35" "Setting up MySQL"
-    if [[ -f "${BOX_DATA_DIR}/box.mysqldump" ]]; then
-        echo "==> Importing existing database into MySQL"
-        mysql -u root -p${mysql_root_password} box < "${BOX_DATA_DIR}/box.mysqldump"
-    fi
-fi
-
 set_progress "40" "Migrating data"
 sudo -u "${USER}" -H bash <<EOF
 set -eu
 cd "${BOX_SRC_DIR}"
-BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@localhost/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up
+BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up
 EOF

+if [[ -z "${arg_admin_fqdn:-}" ]]; then
+    # can be removed after 1.9
+    admin_fqdn=$([[ "${arg_is_custom_domain}" == "true" ]] && echo "${arg_admin_location}.${arg_fqdn}" ||  echo "${arg_admin_location}-${arg_fqdn}")
+else
+    admin_fqdn="${arg_admin_fqdn}"
+fi
+
 echo "==> Creating cloudron.conf"
 cat > "${CONFIG_DIR}/cloudron.conf" <<CONF_END
 {
@@ -258,25 +221,13 @@ cat > "${CONFIG_DIR}/cloudron.conf" <<CONF_END
    "apiServerOrigin": "${arg_api_server_origin}",
    "webServerOrigin": "${arg_web_server_origin}",
    "fqdn": "${arg_fqdn}",
+    "adminFqdn": "${admin_fqdn}",
+    "adminLocation": "${arg_admin_location}",
    "zoneName": "${arg_zone_name}",
-    "isCustomDomain": ${arg_is_custom_domain},
    "provider": "${arg_provider}",
-    "isDemo": ${arg_is_demo},
-    "database": {
-        "hostname": "localhost",
-        "username": "root",
-        "password": "${mysql_root_password}",
-        "port": 3306,
-        "name": "box"
-    },
-    "appBundle": ${arg_app_bundle}
+    "isDemo": ${arg_is_demo}
 }
 CONF_END
-# pass these out-of-band because they have new lines which interfere with json
-if [[ -n "${arg_tls_cert}" && -n "${arg_tls_key}" ]]; then
-    echo "${arg_tls_cert}" > "${CONFIG_DIR}/host.cert"
-    echo "${arg_tls_key}" > "${CONFIG_DIR}/host.key"
-fi

 echo "==> Creating config.json for webadmin"
 cat > "${BOX_SRC_DIR}/webadmin/dist/config.json" <<CONF_END
@@ -285,39 +236,25 @@ cat > "${BOX_SRC_DIR}/webadmin/dist/config.json" <<CONF_END
 }
 CONF_END

+if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
+    echo "==> Generating dhparams (takes forever)"
+    openssl dhparam -out "${BOX_DATA_DIR}/dhparams.pem" 2048
+    cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
+else
+    cp "${BOX_DATA_DIR}/dhparams.pem" "${PLATFORM_DATA_DIR}/addons/mail/dhparams.pem"
+fi
+
 echo "==> Changing ownership"
 chown "${USER}:${USER}" -R "${CONFIG_DIR}"
-chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/nginx" "${PLATFORM_DATA_DIR}/collectd" "${PLATFORM_DATA_DIR}/logrotate.d" "${PLATFORM_DATA_DIR}/addons" "${PLATFORM_DATA_DIR}/acme"
-chown "${USER}:${USER}" -R "${BOX_DATA_DIR}"
-chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys
+chown "${USER}:${USER}" -R "${PLATFORM_DATA_DIR}/nginx" "${PLATFORM_DATA_DIR}/collectd" "${PLATFORM_DATA_DIR}/logrotate.d" "${PLATFORM_DATA_DIR}/addons" "${PLATFORM_DATA_DIR}/acme" "${PLATFORM_DATA_DIR}/backup"
 chown "${USER}:${USER}" "${PLATFORM_DATA_DIR}/INFRA_VERSION" 2>/dev/null || true
 chown "${USER}:${USER}" "${PLATFORM_DATA_DIR}"

-echo "==> Adding automated configs"
-if [[ ! -z "${arg_backup_config}" ]]; then
-    mysql -u root -p${mysql_root_password} \
-        -e "REPLACE INTO settings (name, value) VALUES (\"backup_config\", '$arg_backup_config')" box
-fi
-
-if [[ ! -z "${arg_dns_config}" ]]; then
-    mysql -u root -p${mysql_root_password} \
-        -e "REPLACE INTO settings (name, value) VALUES (\"dns_config\", '$arg_dns_config')" box
-fi
-
-if [[ ! -z "${arg_update_config}" ]]; then
-    mysql -u root -p${mysql_root_password} \
-        -e "REPLACE INTO settings (name, value) VALUES (\"update_config\", '$arg_update_config')" box
-fi
-
-if [[ ! -z "${arg_tls_config}" ]]; then
-    mysql -u root -p${mysql_root_password} \
-        -e "REPLACE INTO settings (name, value) VALUES (\"tls_config\", '$arg_tls_config')" box
-fi
-
-echo "==> Generating dhparams (takes forever)"
-if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
-    openssl dhparam -out "${BOX_DATA_DIR}/dhparams.pem" 2048
-fi
+# do not chown the boxdata/mail directory; dovecot gets upset
+chown "${USER}:${USER}" "${BOX_DATA_DIR}"
+find "${BOX_DATA_DIR}" -mindepth 1 -maxdepth 1 -not -path "${BOX_DATA_DIR}/mail" -exec chown -R "${USER}:${USER}" {} \;
+chown "${USER}:${USER}" "${BOX_DATA_DIR}/mail"
+chown "${USER}:${USER}" -R "${BOX_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys

 set_progress "60" "Starting Cloudron"
 systemctl start cloudron.target
@@ -7,3 +7,9 @@ printf "Cloudron relies on and may break your installation. Ubuntu security upda
 printf "are automatically installed on this server every night.\n"
 printf "\n"
 printf "Read more at https://cloudron.io/documentation/security/#os-updates\n"
+
+if grep -q "^PasswordAuthentication yes" /etc/ssh/sshd_config; then
+    printf "\nPlease disable password based SSH access to secure your server. Read more at\n"
+    printf "https://cloudron.io/documentation/security/#securing-ssh-access\n"
+fi
+
@@ -1,4 +1,4 @@
-import collectd,os
+import collectd,os,subprocess

 # https://blog.dbrgn.ch/2017/3/10/write-a-collectd-python-plugin/

@@ -6,7 +6,7 @@ disks = []

 def init():
    global disks
-    lines = [s.split() for s in os.popen("df --type=ext4 --output=source,target,size,used,avail").read().splitlines()]
+    lines = [s.split() for s in subprocess.check_output(["df", "--type=ext4", "--output=source,target,size,used,avail"]).splitlines()]
    disks = lines[1:] # strip header
    collectd.info('custom df plugin initialized with %s' % disks)

@@ -14,7 +14,7 @@ def read():
    for d in disks:
        device = d[0]
        if 'devicemapper' in d[1] or not device.startswith('/dev/'): continue
-        instance = device[len('/dev/'):].replace('/', '-')
+        instance = device[len('/dev/'):].replace('/', '_') # see #348

        try:
            st = os.statvfs(d[1]) # handle disk removal
@@ -4,13 +4,54 @@ map $http_upgrade $connection_upgrade {
    '' close;
 }

+# http server
 server {
-<% if (vhost) { %>
-    listen       443 http2;
+    listen       80;
+<% if (hasIPv6) { -%>
+    listen       [::]:80;
+<% } -%>
+
+<% if (vhost) { -%>
    server_name  <%= vhost %>;
-<% } else { %>
+<% } else { -%>
+    # IP based access from collectd or initial cloudron setup. TODO: match the IPv6 address
+    server_name  "~^\d+\.\d+\.\d+\.\d+$";
+
+    # collectd
+    location /nginx_status {
+        stub_status on;
+        access_log off;
+        allow 127.0.0.1;
+        deny all;
+    }
+<% } -%>
+
+    # acme challenges (for cert renewal where the vhost config exists)
+    location /.well-known/acme-challenge/ {
+        default_type text/plain;
+        alias /home/yellowtent/platformdata/acme/;
+    }
+
+    location / {
+        # redirect everything to HTTPS
+        return 301 https://$host$request_uri;
+    }
+}
+
+# https server
+server {
+<% if (vhost) { -%>
+    server_name  <%= vhost %>;
+    listen       443 http2;
+<% if (hasIPv6) { -%>
+    listen       [::]:443 http2;
+<% } -%>
+<% } else { -%>
    listen       443 http2 default_server;
-<% } %>
+<% if (hasIPv6) { -%>
+    listen       [::]:443 http2 default_server;
+<% } -%>
+<% } -%>

    ssl                  on;
    # paths are relative to prefix and not to this file
@@ -80,7 +121,7 @@ server {
        # No buffering to temp files, it fails for large downloads
        proxy_max_temp_file_size 0;

-        # Disable check to allow unlimited body sizes
+        # Disable check to allow unlimited body sizes. this allows apps to accept whatever size they want
        client_max_body_size 0;

 <% if (robotsTxtQuoted) { %>
@@ -107,6 +148,11 @@ server {
            proxy_read_timeout 30m;
        }

+        location ~ ^/api/v1/apps/.*/upload$ {
+            proxy_pass   http://127.0.0.1:3000;
+            client_max_body_size 0;
+        }
+
        # graphite paths (uncomment block below and visit /graphite/index.html)
        # location ~ ^/(graphite|content|metrics|dashboard|render|browser|composer)/ {
        #     proxy_pass   http://127.0.0.1:8000;
@@ -36,27 +36,21 @@ http {
    # zones for rate limiting
    limit_req_zone $binary_remote_addr zone=admin_login:10m rate=10r/s; # 10 request a second

-    # HTTP server
+
+    # default http server that returns 404 for any domain we are not listening on
    server {
-        listen      80;
+        listen      80 default_server;
+        listen      [::]:80 default_server;
+        server_name does_not_match_anything;

-        # collectd
-        location /nginx_status {
-            stub_status on;
-            access_log off;
-            allow 127.0.0.1;
-            deny all;
-        }
-
-        # acme challenges
+        # acme challenges (for app installation and re-configure when the vhost config does not exist)
        location /.well-known/acme-challenge/ {
            default_type text/plain;
            alias /home/yellowtent/platformdata/acme/;
        }

        location / {
-            # redirect everything to HTTPS
-            return 301 https://$host$request_uri;
+            return 404;
        }
    }

@@ -13,8 +13,8 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reloadnginx.sh
 Defaults!/home/yellowtent/box/src/scripts/reboot.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reboot.sh

-Defaults!/home/yellowtent/box/src/scripts/reloadcollectd.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/reloadcollectd.sh
+Defaults!/home/yellowtent/box/src/scripts/configurecollectd.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/configurecollectd.sh

 Defaults!/home/yellowtent/box/src/scripts/collectlogs.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/collectlogs.sh
@@ -28,11 +28,12 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/update.sh
 Defaults!/home/yellowtent/box/src/scripts/authorized_keys.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/authorized_keys.sh

-Defaults!/home/yellowtent/box/src/scripts/node.sh env_keep="HOME BOX_ENV NODE_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/node.sh
+Defaults!/home/yellowtent/box/src/scripts/configurelogrotate.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/configurelogrotate.sh

-Defaults!/home/yellowtent/box/src/scripts/mvlogrotateconfig.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/mvlogrotateconfig.sh
+Defaults!/home/yellowtent/box/src/backuptask.js env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/backuptask.js
+
+Defaults!/home/yellowtent/box/src/scripts/restart.sh env_keep="HOME BOX_ENV"
+yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restart.sh

-Defaults!/home/yellowtent/box/src/scripts/rmlogrotateconfig.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmlogrotateconfig.sh
@@ -20,7 +20,6 @@ var appdb = require('./appdb.js'),
    async = require('async'),
    clients = require('./clients.js'),
    config = require('./config.js'),
-    constants = require('./constants.js'),
    ClientsError = clients.ClientsError,
    debug = require('debug')('box:addons'),
    docker = require('./docker.js'),
@@ -115,7 +114,7 @@ var RMAPPDIR_CMD = path.join(__dirname, 'scripts/rmappdir.sh');
 function debugApp(app, args) {
    assert(!app || typeof app === 'object');

-    var prefix = app ? (app.location || 'naked_domain') : '(no app)';
+    var prefix = app ? app.intrinsicFqdn : '(no app)';
    debug(prefix + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
 }

@@ -126,7 +125,7 @@ function setupAddons(app, addons, callback) {

    if (!addons) return callback(null);

-    debugApp(app, 'setupAddons: Settings up %j', Object.keys(addons));
+    debugApp(app, 'setupAddons: Setting up %j', Object.keys(addons));

    async.eachSeries(Object.keys(addons), function iterator(addon, iteratorCallback) {
        if (!(addon in KNOWN_ADDONS)) return iteratorCallback(new Error('No such addon:' + addon));
@@ -246,10 +245,12 @@ function setupOauth(app, options, callback) {
    assert.strictEqual(typeof options, 'object');
    assert.strictEqual(typeof callback, 'function');

+    debugApp(app, 'setupOauth');
+
    if (!app.sso) return callback(null);

    var appId = app.id;
-    var redirectURI = 'https://' + config.appFqdn(app.location);
+    var redirectURI = 'https://' + (app.altDomain || app.intrinsicFqdn);
    var scope = 'profile';

    clients.delByAppIdAndType(appId, clients.TYPE_OAUTH, function (error) { // remove existing creds
@@ -364,6 +365,7 @@ function setupSendMail(app, options, callback) {
        var env = [
            { name: 'MAIL_SMTP_SERVER', value: 'mail' },
            { name: 'MAIL_SMTP_PORT', value: '2525' },
+            { name: 'MAIL_SMTPS_PORT', value: '4650' },
            { name: 'MAIL_SMTP_USERNAME', value: mailbox.name },
            { name: 'MAIL_SMTP_PASSWORD', value: password },
            { name: 'MAIL_FROM', value: mailbox.name + '@' + config.fqdn() },
@@ -645,7 +647,10 @@ function setupRedis(app, options, callback) {
    }

    const tag = infra.images.redis.tag, redisName = 'redis-' + app.id;
+    const label = app.intrinsicFqdn;
+    // note that we do not add appId label because this interferes with the stop/start app logic
    const cmd = `docker run --restart=always -d --name=${redisName} \
+                --label=location=${label} \
                --net cloudron \
                --net-alias ${redisName} \
                -m ${memoryLimit/2} \
@@ -692,7 +697,7 @@ function teardownRedis(app, options, callback) {

       safe.fs.unlinkSync(paths.ADDON_CONFIG_DIR, 'redis-' + app.id + '_vars.sh');

-        shell.sudo('teardownRedis', [ RMAPPDIR_CMD, app.id + '/redis' ], function (error, stdout, stderr) {
+        shell.sudo('teardownRedis', [ RMAPPDIR_CMD, app.id + '/redis', true /* delete directory */ ], function (error, stdout, stderr) {
            if (error) return callback(new Error('Error removing redis data:' + error));

            appdb.unsetAddonConfig(app.id, 'redis', callback);
@@ -10,6 +10,7 @@ exports = module.exports = {
    update: update,
    getAll: getAll,
    getPortBindings: getPortBindings,
+    delPortBinding: delPortBinding,

    setAddonConfig: setAddonConfig,
    getAddonConfig: getAddonConfig,
@@ -58,9 +59,10 @@ var assert = require('assert'),
    util = require('util');

 var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.installationProgress', 'apps.runState',
-    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'apps.location', 'apps.dnsRecordId',
-    'apps.accessRestrictionJson', 'apps.lastBackupId', 'apps.oldConfigJson', 'apps.memoryLimit', 'apps.altDomain',
-    'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson', 'apps.robotsTxt' ].join(',');
+    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'apps.location', 'apps.domain', 'apps.dnsRecordId',
+    'apps.accessRestrictionJson', 'apps.restoreConfigJson', 'apps.oldConfigJson', 'apps.updateConfigJson', 'apps.memoryLimit', 
+    'apps.altDomain', 'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson', 'apps.robotsTxt', 'apps.enableBackup',
+    'apps.creationTime', 'apps.updateTime' ].join(',');

 var PORT_BINDINGS_FIELDS = [ 'hostPort', 'environmentVariable', 'appId' ].join(',');

@@ -75,6 +77,14 @@ function postProcess(result) {
    result.oldConfig = safe.JSON.parse(result.oldConfigJson);
    delete result.oldConfigJson;

+    assert(result.updateConfigJson === null || typeof result.updateConfigJson === 'string');
+    result.updateConfig = safe.JSON.parse(result.updateConfigJson);
+    delete result.updateConfigJson;
+
+    assert(result.restoreConfigJson === null || typeof result.restoreConfigJson === 'string');
+    result.restoreConfig = safe.JSON.parse(result.restoreConfigJson);
+    delete result.restoreConfigJson;
+
    assert(result.hostPorts === null || typeof result.hostPorts === 'string');
    assert(result.environmentVariables === null || typeof result.environmentVariables === 'string');

@@ -98,6 +108,7 @@ function postProcess(result) {
    result.xFrameOptions = result.xFrameOptions || 'SAMEORIGIN';

    result.sso = !!result.sso; // make it bool
+    result.enableBackup = !!result.enableBackup; // make it bool

    assert(result.debugModeJson === null || typeof result.debugModeJson === 'string');
    result.debugMode = safe.JSON.parse(result.debugModeJson);
@@ -167,12 +178,13 @@ function getAll(callback) {
    });
 }

-function add(id, appStoreId, manifest, location, portBindings, data, callback) {
+function add(id, appStoreId, manifest, location, domain, portBindings, data, callback) {
    assert.strictEqual(typeof id, 'string');
    assert.strictEqual(typeof appStoreId, 'string');
    assert(manifest && typeof manifest === 'object');
    assert.strictEqual(typeof manifest.version, 'string');
    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof portBindings, 'object');
    assert(data && typeof data === 'object');
    assert.strictEqual(typeof callback, 'function');
@@ -187,14 +199,14 @@ function add(id, appStoreId, manifest, location, portBindings, data, callback) {
    var altDomain = data.altDomain || null;
    var xFrameOptions = data.xFrameOptions || '';
    var installationState = data.installationState || exports.ISTATE_PENDING_INSTALL;
-    var lastBackupId = data.lastBackupId || null; // used when cloning
+    var restoreConfigJson = data.restoreConfig ? JSON.stringify(data.restoreConfig) : null; // used when cloning
    var sso = 'sso' in data ? data.sso : null;
    var debugModeJson = data.debugMode ? JSON.stringify(data.debugMode) : null;

    var queries = [];
    queries.push({
-        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, location, accessRestrictionJson, memoryLimit, altDomain, xFrameOptions, lastBackupId, sso, debugModeJson) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        args: [ id, appStoreId, manifestJson, installationState, location, accessRestrictionJson, memoryLimit, altDomain, xFrameOptions, lastBackupId, sso, debugModeJson ]
+        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, location, domain, accessRestrictionJson, memoryLimit, altDomain, xFrameOptions, restoreConfigJson, sso, debugModeJson) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
+        args: [ id, appStoreId, manifestJson, installationState, location, domain, accessRestrictionJson, memoryLimit, altDomain, xFrameOptions, restoreConfigJson, sso, debugModeJson ]
    });

    Object.keys(portBindings).forEach(function (env) {
@@ -207,8 +219,8 @@ function add(id, appStoreId, manifest, location, portBindings, data, callback) {
    // only allocate a mailbox if mailboxName is set
    if (data.mailboxName) {
        queries.push({
-            query: 'INSERT INTO mailboxes (name, ownerId, ownerType) VALUES (?, ?, ?)',
-            args: [ data.mailboxName, id, mailboxdb.TYPE_APP ]
+            query: 'INSERT INTO mailboxes (name, domain, ownerId, ownerType) VALUES (?, ?, ?, ?)',
+            args: [ data.mailboxName, domain, id, mailboxdb.TYPE_APP ]
        });
    }

@@ -247,6 +259,18 @@ function getPortBindings(id, callback) {
    });
 }

+function delPortBinding(hostPort, callback) {
+    assert.strictEqual(typeof hostPort, 'number');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM appPortBindings WHERE hostPort=?', [ hostPort ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
 function del(id, callback) {
    assert.strictEqual(typeof id, 'string');
    assert.strictEqual(typeof callback, 'function');
@@ -304,17 +328,8 @@ function updateWithConstraints(id, app, constraints, callback) {

    var fields = [ ], values = [ ];
    for (var p in app) {
-        if (p === 'manifest') {
-            fields.push('manifestJson = ?');
-            values.push(JSON.stringify(app[p]));
-        } else if (p === 'oldConfig') {
-            fields.push('oldConfigJson = ?');
-            values.push(JSON.stringify(app[p]));
-        } else if (p === 'accessRestriction') {
-            fields.push('accessRestrictionJson = ?');
-            values.push(JSON.stringify(app[p]));
-        } else if (p === 'debugMode') {
-            fields.push('debugModeJson = ?');
+        if (p === 'manifest' || p === 'oldConfig' || p === 'updateConfig' || p === 'restoreConfig' || p === 'accessRestriction' || p === 'debugMode') {
+            fields.push(`${p}Json = ?`);
            values.push(JSON.stringify(app[p]));
        } else if (p !== 'portBindings') {
            fields.push(p + ' = ?');
@@ -367,14 +382,14 @@ function setInstallationCommand(appId, installationState, values, callback) {
    // Rules are:
    // uninstall is allowed in any state
    // force update is allowed in any state including pending_uninstall! (for better or worse)
-    // restore is allowed from installed or error state
+    // restore is allowed from installed or error state or currently restoring
    // configure is allowed in installed state or currently configuring or in error state
    // update and backup are allowed only in installed state

    if (installationState === exports.ISTATE_PENDING_UNINSTALL || installationState === exports.ISTATE_PENDING_FORCE_UPDATE) {
        updateWithConstraints(appId, values, '', callback);
    } else if (installationState === exports.ISTATE_PENDING_RESTORE) {
-        updateWithConstraints(appId, values, 'AND (installationState = "installed" OR installationState = "error")', callback);
+        updateWithConstraints(appId, values, 'AND (installationState = "installed" OR installationState = "error" OR installationState = "pending_restore")', callback);
    } else if (installationState === exports.ISTATE_PENDING_UPDATE || installationState === exports.ISTATE_PENDING_BACKUP) {
        updateWithConstraints(appId, values, 'AND installationState = "installed"', callback);
    } else if (installationState === exports.ISTATE_PENDING_CONFIGURE) {
@@ -5,6 +5,7 @@ var appdb = require('./appdb.js'),
    assert = require('assert'),
    async = require('async'),
    DatabaseError = require('./databaseerror.js'),
+    config = require('./config.js'),
    debug = require('debug')('box:apphealthmonitor'),
    docker = require('./docker.js').connection,
    mailer = require('./mailer.js'),
@@ -25,7 +26,7 @@ var gDockerEventStream = null;
 function debugApp(app) {
    assert(!app || typeof app === 'object');

-    var prefix = app ? (app.location || 'naked_domain') : '(no app)';
+    var prefix = app ? app.intrinsicFqdn : '(no app)';
    var manifestAppId = app ? app.manifest.id : '';
    var id = app ? app.id : '';

@@ -94,19 +95,20 @@ function checkAppHealth(app, callback) {
        superagent
            .get(healthCheckUrl)
            .set('Host', app.fqdn) // required for some apache configs with rewrite rules
+            .set('User-Agent', 'Mozilla') // required for some apps (e.g. minio)
            .redirects(0)
            .timeout(HEALTHCHECK_INTERVAL)
            .end(function (error, res) {
-            if (error && !error.response) {
-                debugApp(app, 'not alive (network error): %s', error.message);
-                setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
-            } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
-                debugApp(app, 'not alive : %s', error || res.status);
-                setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
-            } else {
-                setHealth(app, appdb.HEALTH_HEALTHY, callback);
-            }
-        });
+                if (error && !error.response) {
+                    debugApp(app, 'not alive (network error): %s', error.message);
+                    setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
+                } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
+                    debugApp(app, 'not alive : %s', error || res.status);
+                    setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
+                } else {
+                    setHealth(app, appdb.HEALTH_HEALTHY, callback);
+                }
+            });
    });
 }

@@ -156,7 +158,6 @@ function processDockerEvents() {
        stream.setEncoding('utf8');
        stream.on('data', function (data) {
            var ev = JSON.parse(data);
-            debug('Container ' + ev.id + ' went OOM');
            appdb.getByContainerId(ev.id, function (error, app) { // this can error for addons
                var program = error || !app.appStoreId ? ev.id : app.appStoreId;
                var context = JSON.stringify(ev);
@@ -37,6 +37,9 @@ exports = module.exports = {

    getAppConfig: getAppConfig,

+    downloadFile: downloadFile,
+    uploadFile: uploadFile,
+
    // exported for testing
    _validateHostname: validateHostname,
    _validatePortBindings: validatePortBindings,
@@ -57,11 +60,15 @@ var addons = require('./addons.js'),
    DatabaseError = require('./databaseerror.js'),
    debug = require('debug')('box:apps'),
    docker = require('./docker.js'),
+    domaindb = require('./domaindb.js'),
+    domains = require('./domains.js'),
+    DomainError = require('./domains.js').DomainError,
    eventlog = require('./eventlog.js'),
    fs = require('fs'),
    groups = require('./groups.js'),
    mailboxdb = require('./mailboxdb.js'),
    manifestFormat = require('cloudron-manifestformat'),
+    os = require('os'),
    path = require('path'),
    paths = require('./paths.js'),
    safe = require('safetydance'),
@@ -70,10 +77,12 @@ var addons = require('./addons.js'),
    split = require('split'),
    superagent = require('superagent'),
    taskmanager = require('./taskmanager.js'),
+    tld = require('tldjs'),
+    TransformStream = require('stream').Transform,
    updateChecker = require('./updatechecker.js'),
    url = require('url'),
    util = require('util'),
-    uuid = require('node-uuid'),
+    uuid = require('uuid'),
    validator = require('validator');

 // http://dustinsenos.com/articles/customErrorsInNode
@@ -112,18 +121,34 @@ AppsError.BAD_CERTIFICATE = 'Invalid certificate';
 // Hostname validation comes from RFC 1123 (section 2.1)
 // Domain name validation comes from RFC 2181 (Name syntax)
 // https://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names
-// We are validating the validity of the location-fqdn as host name
-function validateHostname(location, fqdn) {
-    var RESERVED_LOCATIONS = [ constants.ADMIN_LOCATION, constants.API_LOCATION, constants.SMTP_LOCATION, constants.IMAP_LOCATION, constants.MAIL_LOCATION, constants.POSTMAN_LOCATION ];
+// We are validating the validity of the location-fqdn as host name (and not dns name)
+function validateHostname(location, domain, hostname) {
+    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof hostname, 'string');

+    const RESERVED_LOCATIONS = [
+        constants.API_LOCATION,
+        constants.SMTP_LOCATION,
+        constants.IMAP_LOCATION,
+        constants.POSTMAN_LOCATION
+    ];
    if (RESERVED_LOCATIONS.indexOf(location) !== -1) return new AppsError(AppsError.BAD_FIELD, location + ' is reserved');

-    if (location === '') return null; // bare location
+    if (hostname === config.adminFqdn()) return new AppsError(AppsError.BAD_FIELD, location + ' is reserved');

-    if ((location.length + 1 /*+ hyphen */ + fqdn.indexOf('.')) > 63) return new AppsError(AppsError.BAD_FIELD, 'Hostname length cannot be greater than 63');
-    if (location.match(/^[A-Za-z0-9-]+$/) === null) return new AppsError(AppsError.BAD_FIELD, 'Hostname can only contain alphanumerics and hyphen');
-    if (location[0] === '-' || location[location.length-1] === '-') return new AppsError(AppsError.BAD_FIELD, 'Hostname cannot start or end with hyphen');
-    if (location.length + 1 /* hyphen */ + fqdn.length > 253) return new AppsError(AppsError.BAD_FIELD, 'FQDN length exceeds 253 characters');
+    // workaround https://github.com/oncletom/tld.js/issues/73
+    var tmp = hostname.replace('_', '-');
+    if (!tld.isValid(tmp)) return new AppsError(AppsError.BAD_FIELD, 'Hostname is not a valid domain name');
+
+    if (hostname.length > 253) return new AppsError(AppsError.BAD_FIELD, 'Hostname length exceeds 253 characters');
+
+    if (location) {
+        // label validation
+        if (location.length > 63) return new AppsError(AppsError.BAD_FIELD, 'Subdomain exceeds 63 characters');
+        if (location.match(/^[A-Za-z0-9-]+$/) === null) return new AppsError(AppsError.BAD_FIELD, 'Subdomain can only contain alphanumerics and hyphen');
+        if (location.startsWith('-') || location.endsWith('-')) return new AppsError(AppsError.BAD_FIELD, 'Subdomain cannot start or end with hyphen');
+    }

    return null;
 }
@@ -204,7 +229,7 @@ function validateMemoryLimit(manifest, memoryLimit) {
    assert.strictEqual(typeof memoryLimit, 'number');

    var min = manifest.memoryLimit || constants.DEFAULT_MEMORY_LIMIT;
-    var max = (4096 * 1024 * 1024);
+    var max = os.totalmem() * 2; // this will overallocate since we don't allocate equal swap always (#466)

    // allow 0, which indicates that it is not set, the one from the manifest will be choosen but we don't commit any user value
    // this is needed so an app update can change the value in the manifest, and if not set by the user, the new value should be used
@@ -254,6 +279,12 @@ function validateRobotsTxt(robotsTxt) {
    return null;
 }

+function validateBackupFormat(format) {
+    if (format === 'tgz' || format == 'rsync') return null;
+
+    return new AppsError(AppsError.BAD_FIELD, 'Invalid backup format');
+}
+
 function getDuplicateErrorDetails(location, portBindings, error) {
    assert.strictEqual(typeof location, 'string');
    assert.strictEqual(typeof portBindings, 'object');
@@ -280,6 +311,8 @@ function getAppConfig(app) {
    return {
        manifest: app.manifest,
        location: app.location,
+        domain: app.domain,
+        intrinsicFqdn: app.intrinsicFqdn,
        accessRestriction: app.accessRestriction,
        portBindings: app.portBindings,
        memoryLimit: app.memoryLimit,
@@ -304,12 +337,18 @@ function hasAccessTo(app, user, callback) {
    if (app.accessRestriction.users.some(function (e) { return e === user.id; })) return callback(null, true);

    // check group access
-    if (!app.accessRestriction.groups) return callback(null, false);
+    groups.getGroups(user.id, function (error, groupIds) {
+        if (error) return callback(null, false);

-    async.some(app.accessRestriction.groups, function (groupId, iteratorDone) {
-        groups.isMember(groupId, user.id, iteratorDone);
-    }, function (error, result) {
-        callback(null, !error && result);
+        const isAdmin = groupIds.indexOf(constants.ADMIN_GROUP_ID) !== -1;
+
+        if (isAdmin) return callback(null, true); // admins can always access any app
+
+        if (!app.accessRestriction.groups) return callback(null, false);
+
+        if (app.accessRestriction.groups.some(function (gid) { return groupIds.indexOf(gid) !== -1; })) return callback(null, true);
+
+        callback(null, false);
    });
 }

@@ -321,11 +360,16 @@ function get(appId, callback) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such app'));
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-        app.iconUrl = getIconUrlSync(app);
-        app.fqdn = app.altDomain || config.appFqdn(app.location);
-        app.cnameTarget = app.altDomain ? config.appFqdn(app.location) : null;
+        domaindb.get(app.domain, function (error, result) {
+            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-        callback(null, app);
+            app.intrinsicFqdn = domains.fqdn(app.location, app.domain, result.provider);
+            app.iconUrl = getIconUrlSync(app);
+            app.fqdn = app.altDomain || app.intrinsicFqdn;
+            app.cnameTarget = app.altDomain ? app.intrinsicFqdn : null;
+
+            callback(null, app);
+        });
    });
 }

@@ -340,11 +384,16 @@ function getByIpAddress(ip, callback) {
            if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such app'));
            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-            app.iconUrl = getIconUrlSync(app);
-            app.fqdn = app.altDomain || config.appFqdn(app.location);
-            app.cnameTarget = app.altDomain ? config.appFqdn(app.location) : null;
+            domaindb.get(app.domain, function (error, result) {
+                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-            callback(null, app);
+                app.intrinsicFqdn = domains.fqdn(app.location, app.domain, result.provider);
+                app.iconUrl = getIconUrlSync(app);
+                app.fqdn = app.altDomain || app.intrinsicFqdn;
+                app.cnameTarget = app.altDomain ? app.intrinsicFqdn : null;
+
+                callback(null, app);
+            });
        });
    });
 }
@@ -355,13 +404,22 @@ function getAll(callback) {
    appdb.getAll(function (error, apps) {
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-        apps.forEach(function (app) {
-            app.iconUrl = getIconUrlSync(app);
-            app.fqdn = app.altDomain || config.appFqdn(app.location);
-            app.cnameTarget = app.altDomain ? config.appFqdn(app.location) : null;
-        });
+        async.eachSeries(apps, function (app, iteratorDone) {
+            domaindb.get(app.domain, function (error, result) {
+                if (error) return iteratorDone(new AppsError(AppsError.INTERNAL_ERROR, error));

-        callback(null, apps);
+                app.intrinsicFqdn = domains.fqdn(app.location, app.domain, result.provider);
+                app.iconUrl = getIconUrlSync(app);
+                app.fqdn = app.altDomain || app.intrinsicFqdn;
+                app.cnameTarget = app.altDomain ? app.intrinsicFqdn : null;
+
+                iteratorDone();
+            });
+        }, function (error) {
+            if (error) return callback(error);
+
+            callback(null, apps);
+        });
    });
 }

@@ -392,7 +450,7 @@ function downloadManifest(appStoreId, manifest, callback) {
    superagent.get(url).timeout(30 * 1000).end(function (error, result) {
        if (error && !error.response) return callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Network error downloading manifest:' + error.message));

-        if (result.statusCode !== 200) return callback(new AppsError(AppsError.BAD_FIELD, util.format('Failed to get app info from store.', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new AppsError(AppsError.EXTERNAL_ERROR, util.format('Failed to get app info from store.', result.statusCode, result.text)));

        callback(null, parts[0], result.body.manifest);
    });
@@ -404,6 +462,7 @@ function install(data, auditSource, callback) {
    assert.strictEqual(typeof callback, 'function');

    var location = data.location.toLowerCase(),
+        domain = data.domain.toLowerCase(),
        portBindings = data.portBindings || null,
        accessRestriction = data.accessRestriction || null,
        icon = data.icon || null,
@@ -415,7 +474,9 @@ function install(data, auditSource, callback) {
        sso = 'sso' in data ? data.sso : null,
        debugMode = data.debugMode || null,
        robotsTxt = data.robotsTxt || null,
-        backupId = data.backupId || null;
+        enableBackup = 'enableBackup' in data ? data.enableBackup : true,
+        backupId = data.backupId || null,
+        backupFormat = data.backupFormat || 'tgz';

    assert(data.appStoreId || data.manifest); // atleast one of them is required

@@ -428,9 +489,6 @@ function install(data, auditSource, callback) {
        error = checkManifestConstraints(manifest);
        if (error) return callback(error);

-        error = validateHostname(location, config.fqdn());
-        if (error) return callback(error);
-
        error = validatePortBindings(portBindings, manifest.tcpPorts);
        if (error) return callback(error);

@@ -449,6 +507,9 @@ function install(data, auditSource, callback) {
        error = validateRobotsTxt(robotsTxt);
        if (error) return callback(error);

+        error = validateBackupFormat(backupFormat);
+        if (error) return callback(error);
+
        if ('sso' in data && !('optionalSso' in manifest)) return callback(new AppsError(AppsError.BAD_FIELD, 'sso can only be specified for apps with optionalSso'));
        // if sso was unspecified, enable it by default if possible
        if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];
@@ -465,43 +526,56 @@ function install(data, auditSource, callback) {
            }
        }

-        error = certificates.validateCertificate(cert, key, config.appFqdn(location));
-        if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
+        domains.get(domain, function (error, domainObject) {
+            if (error && error.reason === DomainError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such domain'));
+            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Could not get domain info:' + error.message));

-        debug('Will install app with id : ' + appId);
+            var intrinsicFqdn = domains.fqdn(location, domain, domainObject.provider);

-        appstore.purchase(appId, appStoreId, function (error) {
-            if (error && error.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
-            if (error && error.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, error.message));
-            if (error && error.reason === AppstoreError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
-            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+            error = validateHostname(location, domain, intrinsicFqdn);
+            if (error) return callback(error);

-            var data = {
-                accessRestriction: accessRestriction,
-                memoryLimit: memoryLimit,
-                altDomain: altDomain,
-                xFrameOptions: xFrameOptions,
-                sso: sso,
-                debugMode: debugMode,
-                mailboxName: (location ? location : manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app',
-                lastBackupId: backupId
-            };
+            error = certificates.validateCertificate(cert, key, intrinsicFqdn);
+            if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));

-            appdb.add(appId, appStoreId, manifest, location, portBindings, data, function (error) {
-                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+            debug('Will install app with id : ' + appId);
+
+            appstore.purchase(appId, appStoreId, function (error) {
+                if (error && error.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
+                if (error && error.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, error.message));
+                if (error && error.reason === AppstoreError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                // save cert to boxdata/certs
-                if (cert && key) {
-                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.cert'), cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
-                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.key'), key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
-                }
+                var data = {
+                    accessRestriction: accessRestriction,
+                    memoryLimit: memoryLimit,
+                    altDomain: altDomain,
+                    xFrameOptions: xFrameOptions,
+                    sso: sso,
+                    debugMode: debugMode,
+                    mailboxName: (location ? location : manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app',
+                    restoreConfig: backupId ? { backupId: backupId, backupFormat: backupFormat } : null,
+                    enableBackup: enableBackup,
+                    robotsTxt: robotsTxt,
+                    intrinsicFqdn: intrinsicFqdn
+                };

-                taskmanager.restartAppTask(appId);
+                appdb.add(appId, appStoreId, manifest, location, domain, portBindings, data, function (error) {
+                    if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+                    if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                eventlog.add(eventlog.ACTION_APP_INSTALL, auditSource, { appId: appId, location: location, manifest: manifest, backupId: backupId });
+                    // save cert to boxdata/certs
+                    if (cert && key) {
+                        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.cert'), cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
+                        if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.key'), key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
+                    }

-                callback(null, { id : appId });
+                    taskmanager.restartAppTask(appId);
+
+                    eventlog.add(eventlog.ACTION_APP_INSTALL, auditSource, { appId: appId, location: location, domain: domain, manifest: manifest, backupId: backupId });
+
+                    callback(null, { id : appId });
+                });
            });
        });
    });
@@ -517,14 +591,12 @@ function configure(appId, data, auditSource, callback) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such app'));
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-        var location, portBindings, values = { };
-        if ('location' in data) {
-            location = values.location = data.location.toLowerCase();
-            error = validateHostname(values.location, config.fqdn());
-            if (error) return callback(error);
-        } else {
-            location = app.location;
-        }
+        var domain, location, portBindings, values = { };
+        if ('location' in data) location = values.location = data.location.toLowerCase();
+        else location = app.location;
+
+        if ('domain' in data) domain = values.domain = data.domain.toLowerCase();
+        else domain = app.domain;

        if ('accessRestriction' in data) {
            values.accessRestriction = data.accessRestriction;
@@ -569,41 +641,53 @@ function configure(appId, data, auditSource, callback) {
            if (error) return callback(error);
        }

-        // save cert to boxdata/certs. TODO: move this to apptask when we have a real task queue
-        if ('cert' in data && 'key' in data) {
-            if (data.cert && data.key) {
-                error = certificates.validateCertificate(data.cert, data.key, config.appFqdn(location));
-                if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
+        domains.get(domain, function (error, domainObject) {
+            if (error && error.reason === DomainError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such domain'));
+            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Could not get domain info:' + error.message));

-                if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.cert'), data.cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
-                if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.key'), data.key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
-            } else { // remove existing cert/key
-                if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.cert'))) debug('Error removing cert: ' + safe.error.message);
-                if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, config.appFqdn(location) + '.user.key'))) debug('Error removing key: ' + safe.error.message);
+            var intrinsicFqdn = domains.fqdn(location, domain, domainObject.provider);
+
+            error = validateHostname(location, domain, intrinsicFqdn);
+            if (error) return callback(error);
+
+            // save cert to boxdata/certs. TODO: move this to apptask when we have a real task queue
+            if ('cert' in data && 'key' in data) {
+                if (data.cert && data.key) {
+                    error = certificates.validateCertificate(data.cert, data.key, intrinsicFqdn);
+                    if (error) return callback(new AppsError(AppsError.BAD_CERTIFICATE, error.message));
+
+                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.cert'), data.cert)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving cert: ' + safe.error.message));
+                    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.key'), data.key)) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Error saving key: ' + safe.error.message));
+                } else { // remove existing cert/key
+                    if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.cert'))) debug('Error removing cert: ' + safe.error.message);
+                    if (!safe.fs.unlinkSync(path.join(paths.APP_CERTS_DIR, intrinsicFqdn + '.user.key'))) debug('Error removing key: ' + safe.error.message);
+                }
            }
-        }

-        values.oldConfig = getAppConfig(app);
+            if ('enableBackup' in data) values.enableBackup = data.enableBackup;

-        debug('Will configure app with id:%s values:%j', appId, values);
+            values.oldConfig = getAppConfig(app);

-        var oldName = (app.location ? app.location : app.manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app';
-        var newName = (location ? location : app.manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app';
-        mailboxdb.updateName(oldName, newName, function (error) {
-            if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new AppsError(AppsError.ALREADY_EXISTS, 'This mailbox is already taken'));
-            if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE));
-            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+            debug('Will configure app with id:%s values:%j', appId, values);

-            appdb.setInstallationCommand(appId, appdb.ISTATE_PENDING_CONFIGURE, values, function (error) {
-                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+            var oldName = (app.location ? app.location : app.manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app';
+            var newName = (location ? location : app.manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app';
+            mailboxdb.updateName(oldName, values.oldConfig.domain, newName, domain, function (error) {
+                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new AppsError(AppsError.ALREADY_EXISTS, 'This mailbox is already taken'));
                if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                taskmanager.restartAppTask(appId);
+                appdb.setInstallationCommand(appId, appdb.ISTATE_PENDING_CONFIGURE, values, function (error) {
+                    if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+                    if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE));
+                    if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId: appId });
+                    taskmanager.restartAppTask(appId);

-                callback(null);
+                    eventlog.add(eventlog.ACTION_APP_CONFIGURE, auditSource, { appId: appId });
+
+                    callback(null);
+                });
            });
        });
    });
@@ -620,7 +704,7 @@ function update(appId, data, auditSource, callback) {
    downloadManifest(data.appStoreId, data.manifest, function (error, appStoreId, manifest) {
        if (error) return callback(error);

-        var values = { };
+        var updateConfig = { };

        error = manifestFormat.parse(manifest);
        if (error) return callback(new AppsError(AppsError.BAD_FIELD, 'Manifest error:' + error.message));
@@ -628,13 +712,7 @@ function update(appId, data, auditSource, callback) {
        error = checkManifestConstraints(manifest);
        if (error) return callback(error);

-        values.manifest = manifest;
-
-        if ('portBindings' in data) {
-            values.portBindings = data.portBindings;
-            error = validatePortBindings(data.portBindings, values.manifest.tcpPorts);
-            if (error) return callback(error);
-        }
+        updateConfig.manifest = manifest;

        if ('icon' in data) {
            if (data.icon) {
@@ -654,26 +732,23 @@ function update(appId, data, auditSource, callback) {

            // prevent user from installing a app with different manifest id over an existing app
            // this allows cloudron install -f --app <appid> for an app installed from the appStore
-            if (app.manifest.id !== values.manifest.id) {
+            if (app.manifest.id !== updateConfig.manifest.id) {
                if (!data.force) return callback(new AppsError(AppsError.BAD_FIELD, 'manifest id does not match. force to override'));
                // clear appStoreId so that this app does not get updates anymore
-                values.appStoreId = '';
+                updateConfig.appStoreId = '';
            }

            // do not update apps in debug mode
            if (app.debugMode && !data.force) return callback(new AppsError(AppsError.BAD_STATE, 'debug mode enabled. force to override'));

            // Ensure we update the memory limit in case the new app requires more memory as a minimum
-            // 0 and -1 are special values for memory limit indicating unset and unlimited
-            if (app.memoryLimit > 0 && values.manifest.memoryLimit && app.memoryLimit < values.manifest.memoryLimit) {
-                values.memoryLimit = values.manifest.memoryLimit;
+            // 0 and -1 are special updateConfig for memory limit indicating unset and unlimited
+            if (app.memoryLimit > 0 && updateConfig.manifest.memoryLimit && app.memoryLimit < updateConfig.manifest.memoryLimit) {
+                updateConfig.memoryLimit = updateConfig.manifest.memoryLimit;
            }

-            values.oldConfig = getAppConfig(app);
-
-            appdb.setInstallationCommand(appId, data.force ? appdb.ISTATE_PENDING_FORCE_UPDATE : appdb.ISTATE_PENDING_UPDATE, values, function (error) {
+            appdb.setInstallationCommand(appId, data.force ? appdb.ISTATE_PENDING_FORCE_UPDATE : appdb.ISTATE_PENDING_UPDATE, { updateConfig: updateConfig }, function (error) {
                if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.BAD_STATE)); // might be a bad guess
-                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails('' /* location cannot conflict */, values.portBindings, error));
                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

                taskmanager.restartAppTask(appId);
@@ -754,22 +829,22 @@ function restore(appId, data, auditSource, callback) {
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

        // for empty or null backupId, use existing manifest to mimic a reinstall
-        var func = data.backupId ? backups.getRestoreConfig.bind(null, data.backupId) : function (next) { return next(null, { manifest: app.manifest }); };
+        var func = data.backupId ? backups.get.bind(null, data.backupId) : function (next) { return next(null, { manifest: app.manifest }); };

-        func(function (error, restoreConfig) {
+        func(function (error, backupInfo) {
            if (error && error.reason === BackupsError.NOT_FOUND) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
            if (error && error.reason === BackupsError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-            if (!restoreConfig) callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Could not get restore config'));
+            if (!backupInfo.manifest) callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Could not get restore manifest'));

            // re-validate because this new box version may not accept old configs
-            error = checkManifestConstraints(restoreConfig.manifest);
+            error = checkManifestConstraints(backupInfo.manifest);
            if (error) return callback(error);

            var values = {
-                lastBackupId: data.backupId || null, // when null, apptask simply reinstalls
-                manifest: restoreConfig.manifest,
+                restoreConfig: data.backupId ? { backupId: data.backupId, backupFormat: backupInfo.format } : null, // when null, apptask simply reinstalls
+                manifest: backupInfo.manifest,

                oldConfig: getAppConfig(app)
            };
@@ -797,61 +872,70 @@ function clone(appId, data, auditSource, callback) {
    debug('Will clone app with id:%s', appId);

    var location = data.location.toLowerCase(),
+        domain = data.domain.toLowerCase(),
        portBindings = data.portBindings || null,
        backupId = data.backupId;

    assert.strictEqual(typeof backupId, 'string');
    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof portBindings, 'object');

    appdb.get(appId, function (error, app) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-        backups.getRestoreConfig(backupId, function (error, restoreConfig) {
+        backups.get(backupId, function (error, backupInfo) {
            if (error && error.reason === BackupsError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
            if (error && error.reason === BackupsError.NOT_FOUND) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
            if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-            if (!restoreConfig) callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Could not get restore config'));
+            if (!backupInfo.manifest) callback(new AppsError(AppsError.EXTERNAL_ERROR, 'Could not get restore config'));

            // re-validate because this new box version may not accept old configs
-            error = checkManifestConstraints(restoreConfig.manifest);
+            error = checkManifestConstraints(backupInfo.manifest);
            if (error) return callback(error);

-            error = validateHostname(location, config.fqdn());
+            error = validatePortBindings(portBindings, backupInfo.manifest.tcpPorts);
            if (error) return callback(error);

-            error = validatePortBindings(portBindings, restoreConfig.manifest.tcpPorts);
-            if (error) return callback(error);
+            domains.get(domain, function (error, domainObject) {
+                if (error && error.reason === DomainError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND, 'No such domain'));
+                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, 'Could not get domain info:' + error.message));

-            var newAppId = uuid.v4(), appStoreId = app.appStoreId, manifest = restoreConfig.manifest;
+                var intrinsicFqdn = domains.fqdn(location, domain, domainObject.provider);

-            appstore.purchase(newAppId, appStoreId, function (error) {
-                if (error && error.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
-                if (error && error.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, error.message));
-                if (error && error.reason === AppstoreError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
-                if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));
+                error = validateHostname(location, domain, intrinsicFqdn);
+                if (error) return callback(error);

-                var data = {
-                    installationState: appdb.ISTATE_PENDING_CLONE,
-                    memoryLimit: app.memoryLimit,
-                    accessRestriction: app.accessRestriction,
-                    xFrameOptions: app.xFrameOptions,
-                    lastBackupId: backupId,
-                    sso: !!app.sso,
-                    mailboxName: (location ? location : manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app'
-                };
+                var newAppId = uuid.v4(), manifest = backupInfo.manifest;

-                appdb.add(newAppId, appStoreId, manifest, location, portBindings, data, function (error) {
-                    if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+                appstore.purchase(newAppId, app.appStoreId, function (error) {
+                    if (error && error.reason === AppstoreError.NOT_FOUND) return callback(new AppsError(AppsError.NOT_FOUND));
+                    if (error && error.reason === AppstoreError.BILLING_REQUIRED) return callback(new AppsError(AppsError.BILLING_REQUIRED, error.message));
+                    if (error && error.reason === AppstoreError.EXTERNAL_ERROR) return callback(new AppsError(AppsError.EXTERNAL_ERROR, error.message));
                    if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                    taskmanager.restartAppTask(newAppId);
+                    var data = {
+                        installationState: appdb.ISTATE_PENDING_CLONE,
+                        memoryLimit: app.memoryLimit,
+                        accessRestriction: app.accessRestriction,
+                        xFrameOptions: app.xFrameOptions,
+                        restoreConfig: { backupId: backupId, backupFormat: backupInfo.format },
+                        sso: !!app.sso,
+                        mailboxName: (location ? location : manifest.title.toLowerCase().replace(/[^a-zA-Z0-9]/g, '')) + '.app'
+                    };

-                    eventlog.add(eventlog.ACTION_APP_CLONE, auditSource, { appId: newAppId, oldAppId: appId, backupId: backupId, location: location, manifest: manifest });
+                    appdb.add(newAppId, app.appStoreId, manifest, location, domain, portBindings, data, function (error) {
+                        if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(getDuplicateErrorDetails(location, portBindings, error));
+                        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

-                    callback(null, { id : newAppId });
+                        taskmanager.restartAppTask(newAppId);
+
+                        eventlog.add(eventlog.ACTION_APP_CLONE, auditSource, { appId: newAppId, oldAppId: appId, backupId: backupId, location: location, manifest: manifest });
+
+                        callback(null, { id : newAppId });
+                    });
                });
            });
        });
@@ -1004,14 +1088,9 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
        if ((semver.major(app.manifest.version) !== 0) && (semver.major(app.manifest.version) !== semver.major(newManifest.version))) return new Error('Major version change'); // major changes are blocking

        var newTcpPorts = newManifest.tcpPorts || { };
-        var oldTcpPorts = app.manifest.tcpPorts || { };
        var portBindings = app.portBindings; // this is never null

-        for (var env in newTcpPorts) {
-            if (!(env in oldTcpPorts)) return new Error(env + ' is required from user');
-        }
-
-        for (env in portBindings) {
+        for (var env in portBindings) {
            if (!(env in newTcpPorts)) return new Error(env + ' was in use but new update removes it');
        }

@@ -1026,7 +1105,7 @@ function autoupdateApps(updateInfo, auditSource, callback) { // updateInfo is {
            if (error) {
                debug('Cannot autoupdate app %s : %s', appId, error.message);
                return iteratorDone();
-           }
+            }

            error = canAutoupdateApp(app, updateInfo[appId].manifest);
            if (error) {
@@ -1094,12 +1173,16 @@ function restoreInstalledApps(callback) {
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

        async.map(apps, function (app, iteratorDone) {
-            debug('marking %s for restore', app.location || app.id);
+            debug('marking %s for restore', app.intrinsicFqdn);

-            appdb.setInstallationCommand(app.id, appdb.ISTATE_PENDING_RESTORE, { oldConfig: null }, function (error) {
-                if (error) debug('did not mark %s for restore', app.location || app.id, error);
+            backups.getByAppIdPaged(1, 1, app.id, function (error, results) {
+                var restoreConfig = !error && results.length ? { backupId: results[0].id, backupFormat: results[0].format } : null;

-                iteratorDone(); // always succeed
+                appdb.setInstallationCommand(app.id, appdb.ISTATE_PENDING_RESTORE, { restoreConfig: restoreConfig, oldConfig: null }, function (error) {
+                    if (error) debug('did not mark %s for restore', app.intrinsicFqdn, error);
+
+                    iteratorDone(); // always succeed
+                });
            });
        }, callback);
    });
@@ -1112,13 +1195,95 @@ function configureInstalledApps(callback) {
        if (error) return callback(new AppsError(AppsError.INTERNAL_ERROR, error));

        async.map(apps, function (app, iteratorDone) {
-            debug('marking %s for reconfigure', app.location || app.id);
+            debug('marking %s for reconfigure', app.intrinsicFqdn);

            appdb.setInstallationCommand(app.id, appdb.ISTATE_PENDING_CONFIGURE, { oldConfig: null }, function (error) {
-                if (error) debug('did not mark %s for reconfigure', app.location || app.id, error);
+                if (error) debug('did not mark %s for reconfigure', app.intrinsicFqdn, error);

                iteratorDone(); // always succeed
            });
        }, callback);
    });
 }
+
+function downloadFile(appId, filePath, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof filePath, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    exec(appId, { cmd: [ 'stat', '--printf=%F-%s', filePath ], tty: true }, function (error, stream) {
+        if (error) return callback(error);
+
+        var data = '';
+        stream.setEncoding('utf8');
+        stream.on('data', function (d) { data += d; });
+        stream.on('end', function () {
+            var parts = data.split('-');
+            if (parts.length !== 2) return callback(new AppsError(AppsError.NOT_FOUND, 'file does not exist'));
+
+            var type = parts[0], filename, cmd, size;
+
+            if (type === 'regular file') {
+                cmd = [ 'cat', filePath ];
+                size = parseInt(parts[1], 10);
+                filename = path.basename(filePath);
+                if (isNaN(size)) return callback(new AppsError(AppsError.NOT_FOUND, 'file does not exist'));
+            } else if (type === 'directory') {
+                cmd = ['tar', 'zcf', '-', '-C', filePath, '.'];
+                filename = path.basename(filePath) + '.tar.gz';
+                size = 0; // unknown
+            } else {
+                return callback(new AppsError(AppsError.NOT_FOUND, 'only files or dirs can be downloaded'));
+            }
+
+            exec(appId, { cmd: cmd , tty: false }, function (error, stream) {
+                if (error) return callback(error);
+
+                var stdoutStream = new TransformStream({
+                    transform: function (chunk, ignoredEncoding, callback) {
+                        this._buffer = this._buffer ? Buffer.concat([this._buffer, chunk]) : chunk;
+
+                        while (true) {
+                            if (this._buffer.length < 8) break; // header is 8 bytes
+
+                            var type = this._buffer.readUInt8(0);
+                            var len = this._buffer.readUInt32BE(4);
+
+                            if (this._buffer.length < (8 + len)) break; // not enough
+
+                            var payload = this._buffer.slice(8, 8 + len);
+
+                            this._buffer = this._buffer.slice(8+len); // consumed
+
+                            if (type === 1) this.push(payload);
+                        }
+
+                        callback();
+                    }
+                });
+
+                stream.pipe(stdoutStream);
+
+                return callback(null, stdoutStream, { filename: filename, size: size });
+            });
+        });
+    });
+}
+
+function uploadFile(appId, sourceFilePath, destFilePath, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof sourceFilePath, 'string');
+    assert.strictEqual(typeof destFilePath, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    exec(appId, { cmd: [ 'bash', '-c', 'cat - > ' + destFilePath ], tty: false }, function (error, stream) {
+        if (error) return callback(error);
+
+        var readFile = fs.createReadStream(sourceFilePath);
+        readFile.on('error', callback);
+
+        readFile.pipe(stream);
+
+        callback(null);
+    });
+}
@@ -11,6 +11,10 @@ exports = module.exports = {
    getAppUpdate: getAppUpdate,
    getBoxUpdate: getBoxUpdate,

+    getAccount: getAccount,
+
+    sendFeedback: sendFeedback,
+
    AppstoreError: AppstoreError
 };

@@ -154,24 +158,21 @@ function sendAliveStatus(data, callback) {
            if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));

            var backendSettings = {
-                dnsConfig: {
-                    provider: result[settings.DNS_CONFIG_KEY].provider,
-                    wildcard: result[settings.DNS_CONFIG_KEY].provider === 'manual' ? result[settings.DNS_CONFIG_KEY].wildcard : undefined
-                },
                tlsConfig: {
                    provider: result[settings.TLS_CONFIG_KEY].provider
                },
                backupConfig: {
-                    provider: result[settings.BACKUP_CONFIG_KEY].provider
+                    provider: result[settings.BACKUP_CONFIG_KEY].provider,
+                    hardlinks: !result[settings.BACKUP_CONFIG_KEY].noHardlinks
                },
                mailConfig: {
                    enabled: result[settings.MAIL_CONFIG_KEY].enabled
-            },
-            mailRelay: {
-                provider: result[settings.MAIL_RELAY_KEY].provider
-            },
-            mailCatchAll: {
-                count: result[settings.CATCH_ALL_ADDRESS_KEY].length
+                },
+                mailRelay: {
+                    provider: result[settings.MAIL_RELAY_KEY].provider
+                },
+                mailCatchAll: {
+                    count: result[settings.CATCH_ALL_ADDRESS_KEY].length
                },
                autoupdatePattern: result[settings.AUTOUPDATE_PATTERN_KEY],
                timeZone: result[settings.TIME_ZONE_KEY],
@@ -180,6 +181,7 @@ function sendAliveStatus(data, callback) {
            var data = {
                domain: config.fqdn(),
                version: config.version(),
+                adminFqdn: config.adminFqdn(),
                provider: config.provider(),
                backendSettings: backendSettings,
                machine: {
@@ -245,3 +247,44 @@ function getAppUpdate(app, callback) {
        });
    });
 }
+
+function getAccount(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    getAppstoreConfig(function (error, appstoreConfig) {
+        if (error) return callback(error);
+
+        var url = config.apiServerOrigin() + '/api/v1/users/' + appstoreConfig.userId;
+
+        superagent.get(url).query({ accessToken: appstoreConfig.token }).timeout(10 * 1000).end(function (error, result) {
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+            if (result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+            // { profile: { id, email, groupId, billing, firstName, lastName, company, street, city, zip, state, country } }
+            callback(null, result.body.profile);
+        });
+    });
+}
+
+function sendFeedback(info, callback) {
+    assert.strictEqual(typeof info, 'object');
+    assert.strictEqual(typeof info.email, 'string');
+    assert.strictEqual(typeof info.displayName, 'string');
+    assert.strictEqual(typeof info.type, 'string');
+    assert.strictEqual(typeof info.subject, 'string');
+    assert.strictEqual(typeof info.description, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    getAppstoreConfig(function (error, appstoreConfig) {
+        if (error) return callback(error);
+
+        var url = config.apiServerOrigin() + '/api/v1/users/' + appstoreConfig.userId + '/cloudrons/' + appstoreConfig.cloudronId + '/feedback';
+
+        superagent.post(url).query({ accessToken: appstoreConfig.token }).send(info).timeout(10 * 1000).end(function (error, result) {
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+            if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+
+            callback(null);
+        });
+    });
+}
@@ -35,8 +35,11 @@ var addons = require('./addons.js'),
    certificates = require('./certificates.js'),
    config = require('./config.js'),
    database = require('./database.js'),
+    DatabaseError = require('./databaseerror.js'),
    debug = require('debug')('box:apptask'),
    docker = require('./docker.js'),
+    domains = require('./domains.js'),
+    DomainError = domains.DomainError,
    ejs = require('ejs'),
    fs = require('fs'),
    manifestFormat = require('cloudron-manifestformat'),
@@ -47,8 +50,6 @@ var addons = require('./addons.js'),
    paths = require('./paths.js'),
    safe = require('safetydance'),
    shell = require('./shell.js'),
-    SubdomainError = require('./subdomains.js').SubdomainError,
-    subdomains = require('./subdomains.js'),
    superagent = require('superagent'),
    sysinfo = require('./sysinfo.js'),
    tld = require('tldjs'),
@@ -56,10 +57,9 @@ var addons = require('./addons.js'),
    _ = require('underscore');

 var COLLECTD_CONFIG_EJS = fs.readFileSync(__dirname + '/collectd.config.ejs', { encoding: 'utf8' }),
-    RELOAD_COLLECTD_CMD = path.join(__dirname, 'scripts/reloadcollectd.sh'),
+    CONFIGURE_COLLECTD_CMD = path.join(__dirname, 'scripts/configurecollectd.sh'),
    LOGROTATE_CONFIG_EJS = fs.readFileSync(__dirname + '/logrotate.ejs', { encoding: 'utf8' }),
-    MV_LOGROTATE_CONFIG_CMD = path.join(__dirname, 'scripts/mvlogrotateconfig.sh'),
-    RM_LOGROTATE_CONFIG_CMD = path.join(__dirname, 'scripts/rmlogrotateconfig.sh'),
+    CONFIGURE_LOGROTATE_CMD = path.join(__dirname, 'scripts/configurelogrotate.sh'),
    RMAPPDIR_CMD = path.join(__dirname, 'scripts/rmappdir.sh'),
    CREATEAPPDIR_CMD = path.join(__dirname, 'scripts/createappdir.sh');

@@ -72,10 +72,29 @@ function initialize(callback) {
 function debugApp(app) {
    assert.strictEqual(typeof app, 'object');

-    var prefix = app ? (app.location || '(bare)') : '(no app)';
+    var prefix = app ? (app.intrinsicFqdn || '(bare)') : '(no app)';
    debug(prefix + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
 }

+// updates the app object and the database
+function updateApp(app, values, callback) {
+    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof values, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debugApp(app, 'updating app with values: %j', values);
+
+    appdb.update(app.id, values, function (error) {
+        if (error) return callback(error);
+
+        for (var value in values) {
+            app[value] = values[value];
+        }
+
+        return callback(null);
+    });
+}
+
 function reserveHttpPort(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');
@@ -131,7 +150,7 @@ function deleteContainers(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    debugApp(app, 'deleting containers');
+    debugApp(app, 'deleting app containers (app, scheduler)');

    docker.deleteContainers(app.id, function (error) {
        if (error) return callback(new Error('Error deleting container: ' + error));
@@ -147,11 +166,12 @@ function createVolume(app, callback) {
    shell.sudo('createVolume', [ CREATEAPPDIR_CMD, app.id ], callback);
 }

-function deleteVolume(app, callback) {
+function deleteVolume(app, options, callback) {
    assert.strictEqual(typeof app, 'object');
+    assert.strictEqual(typeof options, 'object');
    assert.strictEqual(typeof callback, 'function');

-    shell.sudo('deleteVolume', [ RMAPPDIR_CMD, app.id ], callback);
+    shell.sudo('deleteVolume', [ RMAPPDIR_CMD, app.id, !!options.removeDirectory ], callback);
 }

 function addCollectdProfile(app, callback) {
@@ -161,7 +181,7 @@ function addCollectdProfile(app, callback) {
    var collectdConf = ejs.render(COLLECTD_CONFIG_EJS, { appId: app.id, containerId: app.containerId });
    fs.writeFile(path.join(paths.COLLECTD_APPCONFIG_DIR, app.id + '.conf'), collectdConf, function (error) {
        if (error) return callback(error);
-        shell.sudo('addCollectdProfile', [ RELOAD_COLLECTD_CMD ], callback);
+        shell.sudo('addCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'add', app.id ], callback);
    });
 }

@@ -171,7 +191,7 @@ function removeCollectdProfile(app, callback) {

    fs.unlink(path.join(paths.COLLECTD_APPCONFIG_DIR, app.id + '.conf'), function (error) {
        if (error && error.code !== 'ENOENT') debugApp(app, 'Error removing collectd profile', error);
-        shell.sudo('removeCollectdProfile', [ RELOAD_COLLECTD_CMD ], callback);
+        shell.sudo('removeCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'remove', app.id ], callback);
    });
 }

@@ -185,11 +205,12 @@ function addLogrotateConfig(app, callback) {
        var runVolume = result.Mounts.find(function (mount) { return mount.Destination === '/run'; });
        if (!runVolume) return callback(new Error('App does not have /run mounted'));

+        // logrotate configs can have arbitrary commands, so the config files must be owned by root
        var logrotateConf = ejs.render(LOGROTATE_CONFIG_EJS, { volumePath: runVolume.Source });
        var tmpFilePath = path.join(os.tmpdir(), app.id + '.logrotate');
        fs.writeFile(tmpFilePath, logrotateConf, function (error) {
            if (error) return callback(error);
-            shell.sudo('addLogrotateConfig', [ MV_LOGROTATE_CONFIG_CMD, tmpFilePath, app.id ], callback);
+            shell.sudo('addLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'add', app.id, tmpFilePath ], callback);
        });
    });
 }
@@ -198,16 +219,13 @@ function removeLogrotateConfig(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    shell.sudo('removeLogrotateConfig', [ RM_LOGROTATE_CONFIG_CMD, app.id ], callback);
+    shell.sudo('removeLogrotateConfig', [ CONFIGURE_LOGROTATE_CMD, 'remove', app.id ], callback);
 }

-function verifyManifest(app, callback) {
-    assert.strictEqual(typeof app, 'object');
+function verifyManifest(manifest, callback) {
+    assert.strictEqual(typeof manifest, 'object');
    assert.strictEqual(typeof callback, 'function');

-    debugApp(app, 'Verifying manifest');
-
-    var manifest = app.manifest;
    var error = manifestFormat.parse(manifest);
    if (error) return callback(new Error(util.format('Manifest error: %s', error.message)));

@@ -240,7 +258,7 @@ function downloadIcon(app, callback) {
                if (!safe.fs.writeFileSync(path.join(paths.APP_ICONS_DIR, app.id + '.png'), res.body)) return retryCallback(new Error('Error saving icon:' + safe.error.message));

                retryCallback(null);
-        });
+            });
    }, callback);
 }

@@ -253,18 +271,17 @@ function registerSubdomain(app, overwrite, callback) {
        if (error) return callback(error);

        async.retry({ times: 200, interval: 5000 }, function (retryCallback) {
-            debugApp(app, 'Registering subdomain location [%s] overwrite: %s', app.location, overwrite);
+            debugApp(app, 'Registering subdomain location [%s] overwrite: %s', app.intrinsicFqdn, overwrite);

            // get the current record before updating it
-            subdomains.get(app.location, 'A', function (error, values) {
+            domains.getDNSRecords(app.location, app.domain, 'A', function (error, values) {
                if (error) return retryCallback(error);

                // refuse to update any existing DNS record for custom domains that we did not create
-                // note that the appstore sets up the naked domain for non-custom domains
-                if (config.isCustomDomain() && values.length !== 0 && !overwrite) return retryCallback(null, new Error('DNS Record already exists'));
+                if (values.length !== 0 && !overwrite) return retryCallback(null, new Error('DNS Record already exists'));

-                subdomains.upsert(app.location, 'A', [ ip ], function (error, changeId) {
-                    if (error && (error.reason === SubdomainError.STILL_BUSY || error.reason === SubdomainError.EXTERNAL_ERROR)) return retryCallback(error); // try again
+                domains.upsertDNSRecords(app.location, app.domain, 'A', [ ip ], function (error, changeId) {
+                    if (error && (error.reason === DomainError.STILL_BUSY || error.reason === DomainError.EXTERNAL_ERROR)) return retryCallback(error); // try again

                    retryCallback(null, error || changeId);
                });
@@ -272,19 +289,20 @@ function registerSubdomain(app, overwrite, callback) {
        }, function (error, result) {
            if (error || result instanceof Error) return callback(error || result);

+            // dnsRecordId tracks whether we created this DNS record so that we can unregister later
            updateApp(app, { dnsRecordId: result }, callback);
        });
    });
 }

-function unregisterSubdomain(app, location, callback) {
+function unregisterSubdomain(app, location, domain, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof location, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    // do not unregister bare domain because we show a error/cloudron info page there
-    if (!config.isCustomDomain() && location === '') {
-        debugApp(app, 'Skip unregister of empty subdomain');
+    if (!app.dnsRecordId) {
+        debugApp(app, 'Skip unregister of record not created by cloudron');
        return callback(null);
    }

@@ -292,10 +310,11 @@ function unregisterSubdomain(app, location, callback) {
        if (error) return callback(error);

        async.retry({ times: 30, interval: 5000 }, function (retryCallback) {
-            debugApp(app, 'Unregistering subdomain: %s', location);
+            debugApp(app, 'Unregistering subdomain: %s', app.intrinsicFqdn);

-            subdomains.remove(location, 'A', [ ip ], function (error) {
-                if (error && (error.reason === SubdomainError.STILL_BUSY || error.reason === SubdomainError.EXTERNAL_ERROR)) return retryCallback(error); // try again
+            domains.removeDNSRecords(location, domain, 'A', [ ip ], function (error) {
+                if (error && error.reason === DomainError.NOT_FOUND) return retryCallback(null, null); // domain can be not found if oldConfig.domain or restoreConfig.domain was removed
+                if (error && (error.reason === DomainError.STILL_BUSY || error.reason === DomainError.EXTERNAL_ERROR)) return retryCallback(error); // try again

                retryCallback(null, error);
            });
@@ -329,7 +348,7 @@ function waitForDnsPropagation(app, callback) {
    sysinfo.getPublicIp(function (error, ip) {
        if (error) return callback(error);

-        subdomains.waitForDns(config.appFqdn(app.location), ip, 'A', { interval: 5000, times: 120 }, callback);
+        domains.waitForDNSRecord(app.intrinsicFqdn, app.domain, ip, 'A', { interval: 5000, times: 120 }, callback);
    });
 }

@@ -343,32 +362,13 @@ function waitForAltDomainDnsPropagation(app, callback) {
        sysinfo.getPublicIp(function (error, ip) {
            if (error) return callback(error);

-            subdomains.waitForDns(app.altDomain, ip, 'A', { interval: 10000, times: 60 }, callback);
+            domains.waitForDNSRecord(app.altDomain, tld.getDomain(app.altDomain), ip, 'A', { interval: 10000, times: 60 }, callback);
        });
    } else {
-        subdomains.waitForDns(app.altDomain, config.appFqdn(app.location) + '.', 'CNAME', { interval: 10000, times: 60 }, callback);
+        domains.waitForDNSRecord(app.altDomain, tld.getDomain(app.altDomain), app.intrinsicFqdn + '.', 'CNAME', { interval: 10000, times: 60 }, callback);
    }
 }

-// updates the app object and the database
-function updateApp(app, values, callback) {
-    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof values, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    debugApp(app, 'updating app with values: %j', values);
-
-    appdb.update(app.id, values, function (error) {
-        if (error) return callback(error);
-
-        for (var value in values) {
-            app[value] = values[value];
-        }
-
-        return callback(null);
-    });
-}
-
 // Ordering is based on the following rationale:
 //   - configure nginx, icon, oauth
 //   - register subdomain.
@@ -379,14 +379,17 @@ function updateApp(app, values, callback) {
 //   - setup addons (requires the above volume)
 //   - setup the container (requires image, volumes, addons)
 //   - setup collectd (requires container id)
+// restore is also handled here since restore is just an install with some oldConfig to clean up
 function install(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    const backupId = app.lastBackupId, isRestoring = app.installationState === appdb.ISTATE_PENDING_RESTORE;
+    const restoreConfig = app.restoreConfig, isRestoring = app.installationState === appdb.ISTATE_PENDING_RESTORE;

    async.series([
-        verifyManifest.bind(null, app),
+        // this protects against the theoretical possibility of an app being marked for install/restore from
+        // a previous version of box code
+        verifyManifest.bind(null, app.manifest),

        // teardown for re-installs
        updateApp.bind(null, app, { installationProgress: '10, Cleaning up old install' }),
@@ -397,13 +400,13 @@ function install(app, callback) {
        deleteContainers.bind(null, app),
        // oldConfig can be null during upgrades
        addons.teardownAddons.bind(null, app, app.oldConfig ? app.oldConfig.manifest.addons : app.manifest.addons),
-        deleteVolume.bind(null, app),
+        deleteVolume.bind(null, app, { removeDirectory: false }), // do not remove any symlinked volume

        // for restore case
        function deleteImageIfChanged(done) {
-             if (!app.oldConfig || (app.oldConfig.manifest.dockerImage === app.manifest.dockerImage)) return done();
+            if (!app.oldConfig || (app.oldConfig.manifest.dockerImage === app.manifest.dockerImage)) return done();

-             docker.deleteImage(app.oldConfig.manifest, done);
+            docker.deleteImage(app.oldConfig.manifest, done);
        },

        reserveHttpPort.bind(null, app),
@@ -421,7 +424,7 @@ function install(app, callback) {
        createVolume.bind(null, app),

        function restoreFromBackup(next) {
-            if (!backupId) {
+            if (!restoreConfig) {
                async.series([
                    updateApp.bind(null, app, { installationProgress: '60, Setting up addons' }),
                    addons.setupAddons.bind(null, app, app.manifest.addons),
@@ -429,7 +432,7 @@ function install(app, callback) {
            } else {
                async.series([
                    updateApp.bind(null, app, { installationProgress: '60, Download backup and restoring addons' }),
-                    backups.restoreApp.bind(null, app, app.manifest.addons, backupId),
+                    backups.restoreApp.bind(null, app, app.manifest.addons, restoreConfig),
                ], next);
            }
        },
@@ -449,7 +452,7 @@ function install(app, callback) {
        exports._waitForDnsPropagation.bind(null, app),

        updateApp.bind(null, app, { installationProgress: '90, Waiting for External Domain setup' }),
-        exports._waitForAltDomainDnsPropagation.bind(null, app), // required when restoring and !lastBackupId
+        exports._waitForAltDomainDnsPropagation.bind(null, app), // required when restoring and !restoreConfig

        updateApp.bind(null, app, { installationProgress: '95, Configure nginx' }),
        configureNginx.bind(null, app),
@@ -472,11 +475,9 @@ function backup(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    var prefix = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
-
    async.series([
        updateApp.bind(null, app, { installationProgress: '10, Backing up' }),
-        backups.backupApp.bind(null, app, app.manifest, prefix),
+        backups.backupApp.bind(null, app),

        // done!
        function (callback) {
@@ -497,6 +498,9 @@ function configure(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

+    // oldConfig can be null during an infra update
+    var locationChanged = app.oldConfig && (app.oldConfig.intrinsicFqdn !== app.intrinsicFqdn);
+
    async.series([
        updateApp.bind(null, app, { installationProgress: '10, Cleaning up old install' }),
        unconfigureNginx.bind(null, app),
@@ -505,9 +509,10 @@ function configure(app, callback) {
        stopApp.bind(null, app),
        deleteContainers.bind(null, app),
        function (next) {
-            // oldConfig can be null during an infra update
-            if (!app.oldConfig || app.oldConfig.location === app.location) return next();
-            unregisterSubdomain(app, app.oldConfig.location, next);
+            if (!locationChanged) return next();
+
+            // the config.fqdn() fallback can be removed after 1.9
+            unregisterSubdomain(app, app.oldConfig.location, app.oldConfig.domain || config.fqdn(), next);
        },

        reserveHttpPort.bind(null, app),
@@ -516,7 +521,7 @@ function configure(app, callback) {
        downloadIcon.bind(null, app),

        updateApp.bind(null, app, { installationProgress: '35, Registering subdomain' }),
-        registerSubdomain.bind(null, app, true /* overwrite */),
+        registerSubdomain.bind(null, app, !locationChanged /* overwrite */), // if location changed, do not overwrite to detect conflicts

        updateApp.bind(null, app, { installationProgress: '40, Downloading image' }),
        docker.downloadImage.bind(null, app.manifest),
@@ -567,53 +572,80 @@ function update(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    debugApp(app, 'Updating to %s', safe.query(app, 'manifest.version'));
+    debugApp(app, `Updating to ${app.updateConfig.manifest.version}`);

    // app does not want these addons anymore
    // FIXME: this does not handle option changes (like multipleDatabases)
-    var unusedAddons = _.omit(app.oldConfig.manifest.addons, Object.keys(app.manifest.addons));
+    var unusedAddons = _.omit(app.manifest.addons, Object.keys(app.updateConfig.manifest.addons));

    async.series([
+        // this protects against the theoretical possibility of an app being marked for update from
+        // a previous version of box code
        updateApp.bind(null, app, { installationProgress: '0, Verify manifest' }),
-        verifyManifest.bind(null, app),
+        verifyManifest.bind(null, app.updateConfig.manifest),
+
+        function (next) {
+            if (app.installationState === appdb.ISTATE_PENDING_FORCE_UPDATE) return next(null);
+
+            async.series([
+                updateApp.bind(null, app, { installationProgress: '15, Backing up app' }),
+                backups.backupApp.bind(null, app)
+            ], function (error) {
+                if (error) error.backupError = true;
+                next(error);
+            });
+        },

        // download new image before app is stopped. this is so we can reduce downtime
        // and also not remove the 'common' layers when the old image is deleted
-        updateApp.bind(null, app, { installationProgress: '15, Downloading image' }),
-        docker.downloadImage.bind(null, app.manifest),
+        updateApp.bind(null, app, { installationProgress: '25, Downloading image' }),
+        docker.downloadImage.bind(null, app.updateConfig.manifest),

        // note: we cleanup first and then backup. this is done so that the app is not running should backup fail
        // we cannot easily 'recover' from backup failures because we have to revert manfest and portBindings
-        updateApp.bind(null, app, { installationProgress: '25, Cleaning up old install' }),
+        updateApp.bind(null, app, { installationProgress: '35, Cleaning up old install' }),
        removeCollectdProfile.bind(null, app),
        removeLogrotateConfig.bind(null, app),
        stopApp.bind(null, app),
        deleteContainers.bind(null, app),
        function deleteImageIfChanged(done) {
-             if (app.oldConfig.manifest.dockerImage === app.manifest.dockerImage) return done();
+            if (app.manifest.dockerImage === app.updateConfig.manifest.dockerImage) return done();

-             docker.deleteImage(app.oldConfig.manifest, done);
-        },
-
-        function (next) {
-            if (app.installationState === appdb.ISTATE_PENDING_FORCE_UPDATE) return next(null);
-
-            var prefix = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
-
-            async.series([
-                updateApp.bind(null, app, { installationProgress: '30, Backing up app' }),
-                backups.backupApp.bind(null, app, app.oldConfig.manifest, prefix)
-            ], next);
+            docker.deleteImage(app.manifest, done);
        },

        // only delete unused addons after backup
        addons.teardownAddons.bind(null, app, unusedAddons),

+        // free unused ports
+        function (next) {
+            // make sure we always have objects
+            var currentPorts = app.portBindings || {};
+            var newPorts = app.updateConfig.manifest.tcpPorts || {};
+
+            async.each(Object.keys(currentPorts), function (portName, callback) {
+                if (newPorts[portName]) return callback(); // port still in use
+
+                appdb.delPortBinding(currentPorts[portName], function (error) {
+                    if (error && error.reason === DatabaseError.NOT_FOUND) console.error('Portbinding does not exist in database.');
+                    else if (error) return next(error);
+
+                    // also delete from app object for further processing (the db is updated in the next step)
+                    delete app.portBindings[portName];
+
+                    callback();
+                });
+            }, next);
+        },
+
+        // switch over to the new config. manifest, memoryLimit, portBindings, appstoreId are updated here
+        updateApp.bind(null, app, app.updateConfig),
+
        updateApp.bind(null, app, { installationProgress: '45, Downloading icon' }),
        downloadIcon.bind(null, app),

        updateApp.bind(null, app, { installationProgress: '70, Updating addons' }),
-        addons.setupAddons.bind(null, app, app.manifest.addons),
+        addons.setupAddons.bind(null, app, app.updateConfig.manifest.addons),

        updateApp.bind(null, app, { installationProgress: '80, Creating container' }),
        createContainer.bind(null, app),
@@ -629,14 +661,18 @@ function update(app, callback) {
        // done!
        function (callback) {
            debugApp(app, 'updated');
-            updateApp(app, { installationState: appdb.ISTATE_INSTALLED, installationProgress: '', health: null }, callback);
+            updateApp(app, { installationState: appdb.ISTATE_INSTALLED, installationProgress: '', health: null, updateConfig: null, updateTime: new Date() }, callback);
        }
    ], function seriesDone(error) {
-        if (error) {
+        if (error && error.backupError) {
+            debugApp(app, 'update aborted because backup failed', error);
+            updateApp(app, { installationState: appdb.ISTATE_INSTALLED, installationProgress: '', health: null, updateConfig: null }, callback.bind(null, error));
+        } else if (error) {
            debugApp(app, 'Error updating app: %s', error);
-            return updateApp(app, { installationState: appdb.ISTATE_ERROR, installationProgress: error.message }, callback.bind(null, error));
+            updateApp(app, { installationState: appdb.ISTATE_ERROR, installationProgress: error.message, updateTime: new Date() }, callback.bind(null, error));
+        } else {
+            callback(null);
        }
-        callback(null);
    });
 }

@@ -663,13 +699,13 @@ function uninstall(app, callback) {
        addons.teardownAddons.bind(null, app, app.manifest.addons),

        updateApp.bind(null, app, { installationProgress: '40, Deleting volume' }),
-        deleteVolume.bind(null, app),
+        deleteVolume.bind(null, app, { removeDirectory: true }),

        updateApp.bind(null, app, { installationProgress: '50, Deleting image' }),
        docker.deleteImage.bind(null, app.manifest),

        updateApp.bind(null, app, { installationProgress: '60, Unregistering subdomain' }),
-        unregisterSubdomain.bind(null, app, app.location),
+        unregisterSubdomain.bind(null, app, app.location, app.domain),

        updateApp.bind(null, app, { installationProgress: '80, Cleanup icon' }),
        removeIcon.bind(null, app),
@@ -733,7 +769,7 @@ function startTask(appId, callback) {
    assert.strictEqual(typeof callback, 'function');

    // determine what to do
-    appdb.get(appId, function (error, app) {
+    apps.get(appId, function (error, app) {
        if (error) return callback(error);

        debugApp(app, 'startTask installationState: %s runState: %s', app.installationState, app.runState);
@@ -2,7 +2,9 @@

 exports = module.exports = {
    initialize: initialize,
-    uninitialize: uninitialize
+    uninitialize: uninitialize,
+
+    accessTokenAuth: accessTokenAuth
 };

 var assert = require('assert'),
@@ -23,22 +25,22 @@ var assert = require('assert'),

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');
-
+    
    passport.serializeUser(function (user, callback) {
        callback(null, user.id);
    });
-
+    
    passport.deserializeUser(function(userId, callback) {
        user.get(userId, function (error, result) {
            if (error) return callback(error);
-
+            
            var md5 = crypto.createHash('md5').update(result.alternateEmail || result.email).digest('hex');
            result.gravatar = 'https://www.gravatar.com/avatar/' + md5 + '.jpg?s=24&d=mm';
-
+            
            callback(null, result);
        });
    });
-
+    
    passport.use(new LocalStrategy(function (username, password, callback) {
        if (username.indexOf('@') === -1) {
            user.verifyWithUsername(username, password, function (error, result) {
@@ -58,7 +60,7 @@ function initialize(callback) {
            });
        }
    }));
-
+    
    passport.use(new BasicStrategy(function (username, password, callback) {
        if (username.indexOf('cid-') === 0) {
            debug('BasicStrategy: detected client id %s instead of username:password', username);
@@ -80,7 +82,7 @@ function initialize(callback) {
            });
        }
    }));
-
+    
    passport.use(new ClientPasswordStrategy(function (clientId, clientSecret, callback) {
        clients.get(clientId, function(error, client) {
            if (error && error.reason === ClientsError.NOT_FOUND) return callback(null, false);
@@ -89,30 +91,35 @@ function initialize(callback) {
            return callback(null, client);
        });
    }));
-
-    passport.use(new BearerStrategy(function (accessToken, callback) {
-        tokendb.get(accessToken, function (error, token) {
-            if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, false);
-            if (error) return callback(error);
-
-            // scopes here can define what capabilities that token carries
-            // passport put the 'info' object into req.authInfo, where we can further validate the scopes
-            var info = { scope: token.scope };
-
-            user.get(token.identifier, function (error, user) {
-                if (error && error.reason === UserError.NOT_FOUND) return callback(null, false);
-                if (error) return callback(error);
-
-                callback(null, user, info);
-            });
-        });
-    }));
-
+    
+    passport.use(new BearerStrategy(accessTokenAuth));
+    
    callback(null);
 }

 function uninitialize(callback) {
    assert.strictEqual(typeof callback, 'function');
-
+    
    callback(null);
 }
+
+function accessTokenAuth(accessToken, callback) {
+    assert.strictEqual(typeof accessToken, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    tokendb.get(accessToken, function (error, token) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, false);
+        if (error) return callback(error);
+        
+        // scopes here can define what capabilities that token carries
+        // passport put the 'info' object into req.authInfo, where we can further validate the scopes
+        var info = { scope: token.scope };
+        
+        user.get(token.identifier, function (error, user) {
+            if (error && error.reason === UserError.NOT_FOUND) return callback(null, false);
+            if (error) return callback(error);
+            
+            callback(null, user, info);
+        });
+    });
+}
@@ -6,7 +6,7 @@ var assert = require('assert'),
    safe = require('safetydance'),
    util = require('util');

-var BACKUPS_FIELDS = [ 'id', 'creationTime', 'version', 'type', 'dependsOn', 'state', 'restoreConfigJson' ];
+var BACKUPS_FIELDS = [ 'id', 'creationTime', 'version', 'type', 'dependsOn', 'state', 'manifestJson', 'format' ];

 exports = module.exports = {
    add: add,
@@ -34,8 +34,8 @@ function postProcess(result) {

    result.dependsOn = result.dependsOn ? result.dependsOn.split(',') : [ ];

-    result.restoreConfig = result.restoreConfigJson ? safe.JSON.parse(result.restoreConfigJson) : null;
-    delete result.restoreConfigJson;
+    result.manifest = result.manifestJson ? safe.JSON.parse(result.manifestJson) : null;
+    delete result.manifestJson;
 }

 function getByTypeAndStatePaged(type, state, page, perPage, callback) {
@@ -47,12 +47,12 @@ function getByTypeAndStatePaged(type, state, page, perPage, callback) {

    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
        [ type, state, (page-1)*perPage, perPage ], function (error, results) {
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

-        results.forEach(function (result) { postProcess(result); });
+            results.forEach(function (result) { postProcess(result); });

-        callback(null, results);
-    });
+            callback(null, results);
+        });
 }

 function getByTypePaged(type, page, perPage, callback) {
@@ -63,12 +63,12 @@ function getByTypePaged(type, page, perPage, callback) {

    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? ORDER BY creationTime DESC LIMIT ?,?',
        [ type, (page-1)*perPage, perPage ], function (error, results) {
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

-        results.forEach(function (result) { postProcess(result); });
+            results.forEach(function (result) { postProcess(result); });

-        callback(null, results);
-    });
+            callback(null, results);
+        });
 }

 function getByAppIdPaged(page, perPage, appId, callback) {
@@ -80,12 +80,12 @@ function getByAppIdPaged(page, perPage, appId, callback) {
    // box versions (0.93.x and below) used to use appbackup_ prefix
    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? AND id LIKE ? ORDER BY creationTime DESC LIMIT ?,?',
        [ exports.BACKUP_TYPE_APP, exports.BACKUP_STATE_NORMAL, '%app%\\_' + appId + '\\_%', (page-1)*perPage, perPage ], function (error, results) {
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

-        results.forEach(function (result) { postProcess(result); });
+            results.forEach(function (result) { postProcess(result); });

-        callback(null, results);
-    });
+            callback(null, results);
+        });
 }

 function get(id, callback) {
@@ -94,13 +94,13 @@ function get(id, callback) {

    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE id = ? ORDER BY creationTime DESC',
        [ id ], function (error, result) {
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
-        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

-        postProcess(result[0]);
+            postProcess(result[0]);

-        callback(null, result[0]);
-    });
+            callback(null, result[0]);
+        });
 }

 function add(backup, callback) {
@@ -109,20 +109,21 @@ function add(backup, callback) {
    assert.strictEqual(typeof backup.version, 'string');
    assert(backup.type === exports.BACKUP_TYPE_APP || backup.type === exports.BACKUP_TYPE_BOX);
    assert(util.isArray(backup.dependsOn));
-    assert.strictEqual(typeof backup.restoreConfig, 'object');
+    assert.strictEqual(typeof backup.manifest, 'object');
+    assert.strictEqual(typeof backup.format, 'string');
    assert.strictEqual(typeof callback, 'function');

    var creationTime = backup.creationTime || new Date(); // allow tests to set the time
-    var restoreConfig = backup.restoreConfig ? JSON.stringify(backup.restoreConfig) : '';
+    var manifestJson = JSON.stringify(backup.manifest);

-    database.query('INSERT INTO backups (id, version, type, creationTime, state, dependsOn, restoreConfigJson) VALUES (?, ?, ?, ?, ?, ?, ?)',
-        [ backup.id, backup.version, backup.type, creationTime, exports.BACKUP_STATE_NORMAL, backup.dependsOn.join(','), restoreConfig ],
+    database.query('INSERT INTO backups (id, version, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
+        [ backup.id, backup.version, backup.type, creationTime, exports.BACKUP_STATE_NORMAL, backup.dependsOn.join(','), manifestJson, backup.format ],
        function (error) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
-        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

-        callback(null);
-    });
+            callback(null);
+        });
 }

 function update(id, backup, callback) {
@@ -158,19 +159,8 @@ function del(id, callback) {
    assert.strictEqual(typeof id, 'string');
    assert.strictEqual(typeof callback, 'function');

-    get(id, function (error, result) {
-        if (error && error.reason === DatabaseError.NOT_FOUND) return callback();
-        if (error) return callback(error);
-
-        var whereClause = [ 'id=?' ], whereArgs = [ result.id ];
-        result.dependsOn.forEach(function (id) {
-            whereClause.push('id=?');
-            whereArgs.push(id);
-        });
-
-        database.query('DELETE FROM backups WHERE ' + whereClause.join(' OR '), whereArgs, function (error) {
-            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
-            callback(null);
-        });
+    database.query('DELETE FROM backups WHERE id=?', [ id ], function (error) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        callback(null);
    });
 }
@@ -1,7 +1,12 @@
-#!/usr/bin/env node
+#!/bin/bash
+':' //# comment; exec /usr/bin/env node --max_old_space_size=300 "$0" "$@"
+
+// to understand the above hack read http://sambal.org/2014/02/passing-options-node-shebang-line/

 'use strict';

+if (process.argv[2] === '--check') return console.log('OK');
+
 require('supererror')({ splatchError: true });

 // remove timestamp from debug() based output
@@ -10,29 +15,11 @@ require('debug').formatArgs = function formatArgs(args) {
 };

 var assert = require('assert'),
-    BackupsError = require('./backups.js').BackupsError,
-    caas = require('./storage/caas.js'),
+    backups = require('./backups.js'),
    database = require('./database.js'),
    debug = require('debug')('box:backuptask'),
-    filesystem = require('./storage/filesystem.js'),
-    noop = require('./storage/noop.js'),
-    path = require('path'),
    paths = require('./paths.js'),
-    s3 = require('./storage/s3.js'),
-    safe = require('safetydance'),
-    settings = require('./settings.js');
-
-function api(provider) {
-    switch (provider) {
-        case 'caas': return caas;
-        case 's3': return s3;
-        case 'filesystem': return filesystem;
-        case 'minio': return s3;
-        case 'exoscale-sos': return s3;
-        case 'noop': return noop;
-        default: return null;
-    }
-}
+    safe = require('safetydance');

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');
@@ -40,52 +27,12 @@ function initialize(callback) {
    database.initialize(callback);
 }

-function backupApp(backupId, appId, callback) {
-    assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('Start app backup with id %s for %s', backupId, appId);
-
-    settings.getBackupConfig(function (error, backupConfig) {
-        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));
-
-        var backupMapping = [{
-            source: path.join(paths.APPS_DATA_DIR, appId),
-            destination: '.'
-        }];
-
-        api(backupConfig.provider).backup(backupConfig, backupId, backupMapping, callback);
-    });
-}
-
-function backupBox(backupId, callback) {
-    assert.strictEqual(typeof backupId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    debug('Start box backup with id %s', backupId);
-
-    settings.getBackupConfig(function (error, backupConfig) {
-        if (error) return callback(new BackupsError(BackupsError.INTERNAL_ERROR, error));
-
-        var backupMapping = [{
-            source: paths.BOX_DATA_DIR,
-            destination: 'box'
-        }, {
-            source: path.join(paths.PLATFORM_DATA_DIR, 'mail'),
-            destination: 'mail'
-        }];
-
-        api(backupConfig.provider).backup(backupConfig, backupId, backupMapping, callback);
-    });
-}
-
 // Main process starts here
 var backupId = process.argv[2];
-var appId = process.argv[3];
+var format = process.argv[3];
+var dataDir = process.argv[4];

-if (appId) debug('Backuptask for the app %s with id %s', appId, backupId);
-else debug('Backuptask for the whole Cloudron with id %s', backupId);
+debug(`Backing up ${dataDir} to ${backupId}`);

 process.on('SIGTERM', function () {
    process.exit(0);
@@ -94,7 +41,9 @@ process.on('SIGTERM', function () {
 initialize(function (error) {
    if (error) throw error;

-    function resultHandler(error) {
+    safe.fs.writeFileSync(paths.BACKUP_RESULT_FILE, '');
+
+    backups.upload(backupId, format, dataDir, function resultHandler(error) {
        if (error) debug('completed with error', error);

        debug('completed');
@@ -104,8 +53,5 @@ initialize(function (error) {
        // https://nodejs.org/api/process.html are exit codes used by node. apps.js uses the value below
        // to check apptask crashes
        process.exit(error ? 50 : 0);
-    }
-
-    if (appId) backupApp(backupId, appId, resultHandler);
-    else backupBox(backupId, resultHandler);
+    });
 });
@@ -0,0 +1,194 @@
+'use strict';
+
+exports = module.exports = {
+    changePlan: changePlan,
+    upgrade: upgrade,
+    sendHeartbeat: sendHeartbeat,
+    getBoxAndUserDetails: getBoxAndUserDetails,
+    setPtrRecord: setPtrRecord
+};
+
+var assert = require('assert'),
+    backups = require('./backups.js'),
+    config = require('./config.js'),
+    debug = require('debug')('box:caas'),
+    locker = require('./locker.js'),
+    path = require('path'),
+    progress = require('./progress.js'),
+    shell = require('./shell.js'),
+    superagent = require('superagent'),
+    util = require('util'),
+    _ = require('underscore');
+
+const RETIRE_CMD = path.join(__dirname, 'scripts/retire.sh');
+var gBoxAndUserDetails = null;         // cached cloudron details like region,size...
+
+function CaasError(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(CaasError, Error);
+CaasError.BAD_FIELD = 'Field error';
+CaasError.INTERNAL_ERROR = 'Internal Error';
+CaasError.EXTERNAL_ERROR = 'External Error';
+CaasError.BAD_STATE = 'Bad state';
+
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+function retire(reason, info, callback) {
+    assert(reason === 'migrate' || reason === 'upgrade');
+    info = info || { };
+    callback = callback || NOOP_CALLBACK;
+
+    var data = {
+        apiServerOrigin: config.apiServerOrigin(),
+        adminFqdn: config.adminFqdn(),
+        fqdn: config.fqdn()
+    };
+    shell.sudo('retire', [ RETIRE_CMD, reason, JSON.stringify(info), JSON.stringify(data) ], callback);
+}
+
+function doMigrate(options, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    var error = locker.lock(locker.OP_MIGRATE);
+    if (error) return callback(new CaasError(CaasError.BAD_STATE, error.message));
+
+    function unlock(error) {
+        debug('Failed to migrate', error);
+        locker.unlock(locker.OP_MIGRATE);
+        progress.set(progress.MIGRATE, -1, 'Backup failed: ' + error.message);
+    }
+
+    progress.set(progress.MIGRATE, 10, 'Backing up for migration');
+
+    // initiate the migration in the background
+    backups.backupBoxAndApps({ userId: null, username: 'migrator' }, function (error) {
+        if (error) return unlock(error);
+
+        debug('migrate: domain: %s size %s region %s', options.domain, options.size, options.region);
+
+        superagent
+            .post(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/migrate')
+            .query({ token: config.token() })
+            .send(options)
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return unlock(error); // network error
+                if (result.statusCode === 409) return unlock(new CaasError(CaasError.BAD_STATE));
+                if (result.statusCode === 404) return unlock(new CaasError(CaasError.NOT_FOUND));
+                if (result.statusCode !== 202) return unlock(new CaasError(CaasError.EXTERNAL_ERROR, util.format('%s %j', result.status, result.body)));
+
+                progress.set(progress.MIGRATE, 10, 'Migrating');
+
+                retire('migrate', _.pick(options, 'domain', 'size', 'region'));
+            });
+    });
+
+    callback(null);
+}
+
+function changePlan(options, callback) {
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (config.isDemo()) return callback(new CaasError(CaasError.BAD_FIELD, 'Not allowed in demo mode'));
+
+    doMigrate(options, callback);
+}
+
+// this function expects a lock
+function upgrade(boxUpdateInfo, callback) {
+    assert(boxUpdateInfo !== null && typeof boxUpdateInfo === 'object');
+
+    function upgradeError(e) {
+        progress.set(progress.UPDATE, -1, e.message);
+        callback(e);
+    }
+
+    progress.set(progress.UPDATE, 5, 'Backing up for upgrade');
+
+    backups.backupBoxAndApps({ userId: null, username: 'upgrader' }, function (error) {
+        if (error) return upgradeError(error);
+
+        superagent.post(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/upgrade')
+            .query({ token: config.token() })
+            .send({ version: boxUpdateInfo.version })
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return upgradeError(new Error('Network error making upgrade request: ' + error));
+                if (result.statusCode !== 202) return upgradeError(new Error(util.format('Server not ready to upgrade. statusCode: %s body: %j', result.status, result.body)));
+
+                progress.set(progress.UPDATE, 10, 'Updating base system');
+
+                // no need to unlock since this is the last thing we ever do on this box
+                callback();
+
+                retire('upgrade');
+            });
+    });
+}
+
+function sendHeartbeat() {
+    assert(config.provider() === 'caas', 'Heartbeat is only sent for managed cloudrons');
+
+    var url = config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/heartbeat';
+    superagent.post(url).query({ token: config.token(), version: config.version() }).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) debug('Network error sending heartbeat.', error);
+        else if (result.statusCode !== 200) debug('Server responded to heartbeat with %s %s', result.statusCode, result.text);
+        else debug('Heartbeat sent to %s', url);
+    });
+}
+
+function getBoxAndUserDetails(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    if (gBoxAndUserDetails) return callback(null, gBoxAndUserDetails);
+
+    if (config.provider() !== 'caas') return callback(null, {});
+
+    superagent
+        .get(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn())
+        .query({ token: config.token() })
+        .timeout(30 * 1000)
+        .end(function (error, result) {
+            if (error && !error.response) return callback(new CaasError(CaasError.EXTERNAL_ERROR, 'Cannot reach appstore'));
+            if (result.statusCode !== 200) return callback(new CaasError(CaasError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+
+            gBoxAndUserDetails = result.body;
+
+            return callback(null, gBoxAndUserDetails);
+        });
+}
+
+function setPtrRecord(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    superagent
+        .post(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/ptr')
+        .query({ token: config.token() })
+        .send({ domain: domain })
+        .timeout(5 * 1000)
+        .end(function (error, result) {
+            if (error && !error.response) return callback(new CaasError(CaasError.EXTERNAL_ERROR, 'Cannot reach appstore'));
+            if (result.statusCode !== 202) return callback(new CaasError(CaasError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+
+            return callback(null);
+        });
+}
@@ -16,7 +16,7 @@ var assert = require('assert'),

 var CA_PROD = 'https://acme-v01.api.letsencrypt.org',
    CA_STAGING = 'https://acme-staging.api.letsencrypt.org',
-    LE_AGREEMENT = 'https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf';
+    LE_AGREEMENT = 'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf';

 exports = module.exports = {
    getCertificate: getCertificate,
@@ -17,5 +17,5 @@ function getCertificate(domain, options, callback) {

    debug('getCertificate: using fallback certificate', domain);

-    return callback(null, 'cert/host.cert', 'cert/host.key');
+    return callback(null, '', '');
 }
@@ -5,6 +5,7 @@ exports = module.exports = {

    ensureFallbackCertificate: ensureFallbackCertificate,
    setFallbackCertificate: setFallbackCertificate,
+    getFallbackCertificate: getFallbackCertificate,

    validateCertificate: validateCertificate,
    ensureCertificate: ensureCertificate,
@@ -92,7 +93,7 @@ function getApi(app, callback) {
        if (tlsConfig.provider === 'fallback') return callback(null, fallback, {});

        // use acme if we have altDomain or the tlsConfig is not caas
-        var api = (app.altDomain || tlsConfig.provider) !== 'caas' ? acme : caas;
+        var api = (app.altDomain || tlsConfig.provider !== 'caas') ? acme : caas;

        var options = { };
        if (tlsConfig.provider === 'caas') {
@@ -121,6 +122,11 @@ function ensureFallbackCertificate(callback) {
    var fallbackCertPath = path.join(paths.NGINX_CERT_DIR, 'host.cert');
    var fallbackKeyPath = path.join(paths.NGINX_CERT_DIR, 'host.key');

+    if (fs.existsSync(fallbackCertPath) && fs.existsSync(fallbackKeyPath)) {
+        debug('ensureFallbackCertificate: pre-existing fallback certs');
+        return callback();
+    }
+
    if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) { // existing custom fallback certs (when restarting, restoring, updating)
        debug('ensureFallbackCertificate: using fallback certs provided by user');
        if (!safe.child_process.execSync('cp ' + certFilePath + ' ' + fallbackCertPath)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
@@ -129,15 +135,6 @@ function ensureFallbackCertificate(callback) {
        return callback();
    }

-    if (config.tlsCert() && config.tlsKey()) {
-        // cert from CaaS or cloudron-setup. these files should _not_ be part of the backup
-        debug('ensureFallbackCertificate: using CaaS/cloudron-setup fallback certs');
-        if (!safe.fs.writeFileSync(fallbackCertPath, config.tlsCert())) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
-        if (!safe.fs.writeFileSync(fallbackKeyPath, config.tlsKey())) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
-
-        return callback();
-    }
-
    // generate a self-signed cert. it's in backup dir so that we don't create a new cert across restarts
    // FIXME: this cert does not cover the naked domain. needs SAN
    if (config.fqdn()) {
@@ -177,11 +174,11 @@ function renewAll(auditSource, callback) {
    apps.getAll(function (error, allApps) {
        if (error) return callback(error);

-        allApps.push({ location: constants.ADMIN_LOCATION }); // inject fake webadmin app
+        allApps.push({ intrinsicFqdn: config.adminFqdn() }); // inject fake webadmin app

        var expiringApps = [ ];
        for (var i = 0; i < allApps.length; i++) {
-            var appDomain = allApps[i].altDomain || config.appFqdn(allApps[i].location);
+            var appDomain = allApps[i].altDomain || allApps[i].instrincFqdn;

            var certFilePath = path.join(paths.APP_CERTS_DIR, appDomain + '.user.cert');
            var keyFilePath = path.join(paths.APP_CERTS_DIR, appDomain + '.user.key');
@@ -205,10 +202,10 @@ function renewAll(auditSource, callback) {
            }
        }

-        debug('renewAll: %j needs to be renewed', expiringApps.map(function (a) { return a.altDomain || config.appFqdn(a.location); }));
+        debug('renewAll: %j needs to be renewed', expiringApps.map(function (app) { return app.altDomain || app.intrinsicFqdn; }));

        async.eachSeries(expiringApps, function iterator(app, iteratorCallback) {
-            var domain = app.altDomain || config.appFqdn(app.location);
+            var domain = app.altDomain || app.intrinsicFqdn;

            getApi(app, function (error, api, apiOptions) {
                if (error) return callback(error);
@@ -232,14 +229,18 @@ function renewAll(auditSource, callback) {

                        debug('renewAll: using fallback certs for %s since it expires soon', domain, error);

-                        certFilePath = 'cert/host.cert';
-                        keyFilePath = 'cert/host.key';
+                        // if no cert was returned use fallback, the fallback provider will not provide any for example
+                        var fallbackCertFilePath = path.join(paths.NGINX_CERT_DIR, domain + '.cert');
+                        var fallbackKeyFilePath = path.join(paths.NGINX_CERT_DIR, domain + '.key');
+
+                        certFilePath = fs.existsSync(fallbackCertFilePath) ? fallbackCertFilePath : 'cert/host.cert';
+                        keyFilePath = fs.existsSync(fallbackKeyFilePath) ? fallbackKeyFilePath : 'cert/host.key';
                    } else {
                        debug('renewAll: certificate for %s renewed', domain);
                    }

                    // reconfigure and reload nginx. this is required for the case where we got a renewed cert after fallback
-                    var configureFunc = app.location === constants.ADMIN_LOCATION ?
+                    var configureFunc = app.intrinsicFqdn === config.adminFqdn() ?
                        nginx.configureAdmin.bind(null, certFilePath, keyFilePath, constants.NGINX_ADMIN_CONFIG_FILE_NAME, config.adminFqdn())
                        : nginx.configureApp.bind(null, app, certFilePath, keyFilePath);

@@ -276,51 +277,52 @@ function validateCertificate(cert, key, fqdn) {
    if (cert && !key) return new Error('missing key');

    var result = safe.child_process.execSync('openssl x509 -noout -checkhost "' + fqdn + '"', { encoding: 'utf8', input: cert });
-    if (!result) return new Error(util.format('could not get cert subject'));
+    if (!result) return new Error('Invalid certificate. Unable to get certificate subject.');

    // if no match, check alt names
    if (result.indexOf('does match certificate') === -1) {
        // https://github.com/drwetter/testssl.sh/pull/383
-        var cmd = `openssl x509 -noout -text | grep -A3 "Subject Alternative Name" | \
+        var cmd = 'openssl x509 -noout -text | grep -A3 "Subject Alternative Name" | \
                   grep "DNS:" | \
-                   sed -e "s/DNS://g" -e "s/ //g" -e "s/,/ /g" -e "s/othername:<unsupported>//g"`;
+                   sed -e "s/DNS://g" -e "s/ //g" -e "s/,/ /g" -e "s/othername:<unsupported>//g"';
        result = safe.child_process.execSync(cmd, { encoding: 'utf8', input: cert });
        var altNames = result ? [ ] : result.trim().split(' '); // might fail if cert has no SAN
        debug('validateCertificate: detected altNames as %j', altNames);

        // check altNames
-        if (!altNames.some(matchesDomain)) return new Error(util.format('cert is not valid for this domain. Expecting %s in %j', fqdn, altNames));
+        if (!altNames.some(matchesDomain)) return new Error(util.format('Certificate is not valid for this domain. Expecting %s in %j', fqdn, altNames));
    }

    // http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#verify
    var certModulus = safe.child_process.execSync('openssl x509 -noout -modulus', { encoding: 'utf8', input: cert });
    var keyModulus = safe.child_process.execSync('openssl rsa -noout -modulus', { encoding: 'utf8', input: key });
-    if (certModulus !== keyModulus) return new Error('key does not match the cert');
+    if (certModulus !== keyModulus) return new Error('Key does not match the certificate.');

-   // check expiration
+    // check expiration
    result = safe.child_process.execSync('openssl x509 -checkend 0', { encoding: 'utf8', input: cert });
-    if (!result) return new Error('cert expired');
+    if (!result) return new Error('Certificate is expired.');

    return null;
 }

-function setFallbackCertificate(cert, key, callback) {
+function setFallbackCertificate(cert, key, fqdn, callback) {
    assert.strictEqual(typeof cert, 'string');
    assert.strictEqual(typeof key, 'string');
+    assert.strictEqual(typeof fqdn, 'string');
    assert.strictEqual(typeof callback, 'function');

-    var error = validateCertificate(cert, key, '*.' + config.fqdn());
+    var error = validateCertificate(cert, key, '*.' + fqdn);
    if (error) return callback(new CertificatesError(CertificatesError.INVALID_CERT, error.message));

    // backup the cert
-    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, 'host.cert'), cert)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
-    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, 'host.key'), key)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
+    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, fqdn + '.cert'), cert)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
+    if (!safe.fs.writeFileSync(path.join(paths.APP_CERTS_DIR, fqdn + '.key'), key)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));

    // copy over fallback cert
-    if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, 'host.cert'), cert)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
-    if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, 'host.key'), key)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
+    if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, fqdn + '.cert'), cert)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));
+    if (!safe.fs.writeFileSync(path.join(paths.NGINX_CERT_DIR, fqdn + '.key'), key)) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, safe.error.message));

-    exports.events.emit(exports.EVENT_CERT_CHANGED, '*.' + config.fqdn());
+    exports.events.emit(exports.EVENT_CERT_CHANGED, '*.' + fqdn);

    nginx.reload(function (error) {
        if (error) return callback(new CertificatesError(CertificatesError.INTERNAL_ERROR, error));
@@ -329,11 +331,16 @@ function setFallbackCertificate(cert, key, callback) {
    });
 }

-function getFallbackCertificatePath(callback) {
+function getFallbackCertificate(fqdn, callback) {
+    assert.strictEqual(typeof fqdn, 'string');
    assert.strictEqual(typeof callback, 'function');

-    // any user fallback cert is always copied over to nginx cert dir
-    callback(null, path.join(paths.NGINX_CERT_DIR, 'host.cert'), path.join(paths.NGINX_CERT_DIR, 'host.key'));
+    var cert = safe.fs.readFileSync(path.join(paths.NGINX_CERT_DIR, fqdn + '.cert'), 'utf-8');
+    var key = safe.fs.readFileSync(path.join(paths.NGINX_CERT_DIR, fqdn + '.key'), 'utf-8');
+
+    if (!cert || !key) return callback(new CertificatesError(CertificatesError.NOT_FOUND));
+
+    callback(null, { cert: cert, key: key });
 }

 function setAdminCertificate(cert, key, callback) {
@@ -371,7 +378,8 @@ function getAdminCertificatePath(callback) {

    if (fs.existsSync(certFilePath) && fs.existsSync(keyFilePath)) return callback(null, certFilePath, keyFilePath);

-    getFallbackCertificatePath(callback);
+    // any user fallback cert is always copied over to nginx cert dir
+    callback(null, path.join(paths.NGINX_CERT_DIR, 'host.cert'), path.join(paths.NGINX_CERT_DIR, 'host.key'));
 }

 function getAdminCertificate(callback) {
@@ -394,7 +402,7 @@ function ensureCertificate(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    var domain = app.altDomain || config.appFqdn(app.location);
+    var domain = app.altDomain || app.intrinsicFqdn;

    var certFilePath = path.join(paths.APP_CERTS_DIR, domain + '.user.cert');
    var keyFilePath = path.join(paths.APP_CERTS_DIR, domain + '.user.key');
@@ -422,9 +430,15 @@ function ensureCertificate(app, callback) {
        debug('ensureCertificate: getting certificate for %s with options %j', domain, apiOptions);

        api.getCertificate(domain, apiOptions, function (error, certFilePath, keyFilePath) {
-            if (error) {
-                debug('ensureCertificate: could not get certificate. using fallback certs', error);
-                return callback(null, 'cert/host.cert', 'cert/host.key'); // use fallback certs
+            if (error) debug('ensureCertificate: could not get certificate. using fallback certs', error);
+
+            // if no cert was returned use fallback, the fallback provider will not provide any for example
+            if (!certFilePath || !keyFilePath) {
+                var fallbackCertFilePath = path.join(paths.NGINX_CERT_DIR, app.domain + '.cert');
+                var fallbackKeyFilePath = path.join(paths.NGINX_CERT_DIR, app.domain + '.key');
+
+                certFilePath = fs.existsSync(fallbackCertFilePath) ? fallbackCertFilePath : 'cert/host.cert';
+                keyFilePath = fs.existsSync(fallbackKeyFilePath) ? fallbackKeyFilePath : 'cert/host.key';
            }

            callback(null, certFilePath, keyFilePath);
@@ -18,7 +18,7 @@ exports = module.exports = {

    // keep this in sync with start.sh ADMIN_SCOPES that generates the cid-webadmin
    SCOPE_APPS: 'apps',
-    SCOPE_DEVELOPER: 'developer',
+    SCOPE_DEVELOPER: 'developer', // obsolete
    SCOPE_PROFILE: 'profile',
    SCOPE_CLOUDRON: 'cloudron',
    SCOPE_SETTINGS: 'settings',
@@ -35,7 +35,7 @@ exports = module.exports = {
    TYPE_PROXY: 'addon-proxy'
 };

-var appdb = require('./appdb.js'),
+var apps = require('./apps.js'),
    assert = require('assert'),
    async = require('async'),
    clientdb = require('./clientdb.js'),
@@ -45,7 +45,7 @@ var appdb = require('./appdb.js'),
    hat = require('hat'),
    tokendb = require('./tokendb.js'),
    util = require('util'),
-    uuid = require('node-uuid');
+    uuid = require('uuid');

 function ClientsError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
@@ -80,7 +80,7 @@ function validateName(name) {
    if (name.length < 1) return new ClientsError(ClientsError.BAD_FIELD, 'Name must be atleast 1 character');
    if (name.length > 128) return new ClientsError(ClientsError.BAD_FIELD, 'Name too long');

-    if (/[^a-zA-Z0-9\-]/.test(name)) return new ClientsError(ClientsError.BAD_FIELD, 'Username can only contain alphanumerals and dash');
+    if (/[^a-zA-Z0-9-]/.test(name)) return new ClientsError(ClientsError.BAD_FIELD, 'Username can only contain alphanumerals and dash');

    return null;
 }
@@ -183,7 +183,7 @@ function getAll(callback) {
                return callback(null);
            }

-            appdb.get(record.appId, function (error, result) {
+            apps.get(record.appId, function (error, result) {
                if (error) {
                    console.error('Failed to get app details for oauth client', record.appId, error);
                    return callback(null);  // ignore error so we continue listing clients
@@ -192,7 +192,7 @@ function getAll(callback) {
                if (record.type === exports.TYPE_PROXY) record.name = result.manifest.title + ' Website Proxy';
                if (record.type === exports.TYPE_OAUTH) record.name = result.manifest.title + ' OAuth';

-                record.location = result.location;
+                record.domain = result.altDomain || result.intrinsicFqdn;

                tmp.push(record);

@@ -325,7 +325,7 @@ function addDefaultClients(callback) {

    // The domain might have changed, therefor we have to update the record
    // !!! This needs to be in sync with the webadmin, specifically login_callback.js
-    const ADMIN_SCOPES="cloudron,developer,profile,users,apps,settings";
+    const ADMIN_SCOPES = 'cloudron,developer,profile,users,apps,settings';

    // id, appId, type, clientSecret, redirectURI, scope
    async.series([
@@ -12,12 +12,9 @@ exports = module.exports = {
    dnsSetup: dnsSetup,
    getLogs: getLogs,

-    sendHeartbeat: sendHeartbeat,
-
    updateToLatest: updateToLatest,
+    restore: restore,
    reboot: reboot,
-    retire: retire,
-    migrate: migrate,

    checkDiskSpace: checkDiskSpace,

@@ -31,6 +28,8 @@ var appdb = require('./appdb.js'),
    assert = require('assert'),
    async = require('async'),
    backups = require('./backups.js'),
+    BackupsError = require('./backups.js').BackupsError,
+    caas = require('./caas.js'),
    certificates = require('./certificates.js'),
    child_process = require('child_process'),
    clients = require('./clients.js'),
@@ -39,6 +38,8 @@ var appdb = require('./appdb.js'),
    cron = require('./cron.js'),
    debug = require('debug')('box:cloudron'),
    df = require('@sindresorhus/df'),
+    domains = require('./domains.js'),
+    DomainError = domains.DomainError,
    eventlog = require('./eventlog.js'),
    fs = require('fs'),
    locker = require('./locker.js'),
@@ -50,12 +51,13 @@ var appdb = require('./appdb.js'),
    platform = require('./platform.js'),
    progress = require('./progress.js'),
    safe = require('safetydance'),
+    semver = require('semver'),
    settings = require('./settings.js'),
+    settingsdb = require('./settingsdb.js'),
    SettingsError = settings.SettingsError,
    shell = require('./shell.js'),
    spawn = require('child_process').spawn,
    split = require('split'),
-    subdomains = require('./subdomains.js'),
    superagent = require('superagent'),
    sysinfo = require('./sysinfo.js'),
    tld = require('tldjs'),
@@ -63,31 +65,16 @@ var appdb = require('./appdb.js'),
    updateChecker = require('./updatechecker.js'),
    user = require('./user.js'),
    UserError = user.UserError,
-    user = require('./user.js'),
    util = require('util'),
    _ = require('underscore');

 var REBOOT_CMD = path.join(__dirname, 'scripts/reboot.sh'),
    UPDATE_CMD = path.join(__dirname, 'scripts/update.sh'),
-    RETIRE_CMD = path.join(__dirname, 'scripts/retire.sh');
+    RESTART_CMD = path.join(__dirname, 'scripts/restart.sh');

 var NOOP_CALLBACK = function (error) { if (error) debug(error); };

-// result to not depend on the appstore
-const BOX_AND_USER_TEMPLATE = {
-    box: {
-        region: null,
-        size: null,
-        plan: 'Custom Plan'
-    },
-    user: {
-        billing: false,
-        currency: ''
-    }
-};
-
-var gBoxAndUserDetails = null,         // cached cloudron details like region,size...
-    gWebadminStatus = { dns: false, tls: false, configuring: false };
+var gWebadminStatus = { dns: false, tls: false, configuring: false, restoring: false };

 function CloudronError(reason, errorOrMessage) {
    assert.strictEqual(typeof reason, 'string');
@@ -121,15 +108,14 @@ CloudronError.SELF_UPGRADE_NOT_SUPPORTED = 'Self upgrade not supported';
 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');

-    gWebadminStatus = { dns: false, tls: false, configuring: false };
-    gBoxAndUserDetails = null;
+    gWebadminStatus = { dns: false, tls: false, configuring: false, restoring: false };

    async.series([
        certificates.initialize,
        settings.initialize,
-        installAppBundle,
        configureDefaultServer,
-        onDomainConfigured
+        onDomainConfigured,
+        onActivated
    ], function (error) {
        if (error) return callback(error);

@@ -144,7 +130,6 @@ function uninitialize(callback) {

    async.series([
        cron.uninitialize,
-        mailer.stop,
        platform.stop,
        certificates.uninitialize,
        settings.uninitialize
@@ -160,37 +145,95 @@ function onDomainConfigured(callback) {
        clients.addDefaultClients,
        certificates.ensureFallbackCertificate,
        ensureDkimKey,
-        platform.start, // requires fallback certs for mail container
-        mailer.start, // this requires the "mail" container to be running
-        cron.initialize
+        cron.initialize // required for caas heartbeat before activation
    ], callback);
 }

-function dnsSetup(dnsConfig, domain, zoneName, callback) {
-    assert.strictEqual(typeof dnsConfig, 'object');
+function onActivated(callback) {
+    callback = callback || NOOP_CALLBACK;
+
+    // Starting the platform after a user is available means:
+    // 1. mail bounces can now be sent to the cloudron owner
+    // 2. the restore code path can run without sudo (since mail/ is non-root)
+    user.count(function (error, count) {
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
+        if (!count) return callback(); // not activated
+
+        platform.start(callback);
+    });
+}
+
+function autoprovision(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    const confJson = safe.fs.readFileSync(paths.AUTO_PROVISION_FILE, 'utf8');
+    if (!confJson) return callback();
+
+    const conf = safe.JSON.parse(confJson);
+    if (!conf) return callback();
+
+    async.eachSeries(Object.keys(conf), function (key, iteratorDone) {
+        var name;
+        switch (key) {
+        case 'dnsConfig': name = 'dns_config'; break;
+        case 'tlsConfig': name = 'tls_config'; break;
+        case 'backupConfig': name = 'backup_config'; break;
+        case 'tlsCert':
+            debug(`autoprovision: ${key}`);
+            return fs.writeFile(path.join(paths.NGINX_CERT_DIR, 'host.cert'), conf[key], iteratorDone);
+        case 'tlsKey':
+            debug(`autoprovision: ${key}`);
+            return fs.writeFile(path.join(paths.NGINX_CERT_DIR, 'host.key'), conf[key], iteratorDone);
+        default:
+            debug(`autoprovision: ${key} ignored`);
+            return iteratorDone();
+        }
+
+        debug(`autoprovision: ${name}`);
+        settingsdb.set(name, JSON.stringify(conf[key]), iteratorDone);
+    }, callback);
+}
+
+function dnsSetup(adminFqdn, domain, zoneName, provider, dnsConfig, callback) {
+    assert.strictEqual(typeof adminFqdn, 'string');
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof dnsConfig, 'object');
    assert.strictEqual(typeof callback, 'function');

    if (config.fqdn()) return callback(new CloudronError(CloudronError.ALREADY_SETUP));

-    if (!zoneName) zoneName = tld.getDomain(domain) || '';
+    if (!zoneName) zoneName = tld.getDomain(domain) || domain;

    debug('dnsSetup: Setting up Cloudron with domain %s and zone %s', domain, zoneName);

-    settings.setDnsConfig(dnsConfig, domain, zoneName, function (error) {
-        if (error && error.reason === SettingsError.BAD_FIELD) return callback(new CloudronError(CloudronError.BAD_FIELD, error.message));
+    function done(error) {
+        if (error && error.reason === DomainError.BAD_FIELD) return callback(new CloudronError(CloudronError.BAD_FIELD, error.message));
        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

-        config.setFqdn(domain); // set fqdn only after dns config is valid, otherwise cannot re-setup if we failed
-        config.setZoneName(zoneName);
+        autoprovision(function (error) {
+            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

-        async.series([ // do not block
-            onDomainConfigured,
-            configureWebadmin
-        ], NOOP_CALLBACK);
+            config.setFqdn(domain); // set fqdn only after dns config is valid, otherwise cannot re-setup if we failed
+            config.setAdminFqdn(adminFqdn);
+            config.setAdminLocation('my');
+            config.setZoneName(zoneName);

-        callback();
+            callback();
+
+            async.series([ // do not block
+                onDomainConfigured,
+                configureWebadmin
+            ], NOOP_CALLBACK);
+        });
+    }
+
+    domains.get(domain, function (error, result) {
+        if (error && error.reason !== DomainError.NOT_FOUND) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
+
+        if (!result) domains.add(domain, zoneName, provider, dnsConfig, null /* cert */, done);
+        else domains.update(domain, provider, dnsConfig, null /* cert */, done);
    });
 }

@@ -232,28 +275,36 @@ function configureWebadmin(callback) {

    function done(error) {
        gWebadminStatus.configuring = false;
-        debug('configureWebadmin: done error:%j', error);
+        debug('configureWebadmin: done error: %j', error || {});
        callback(error);
    }

-    sysinfo.getPublicIp(function (error, ip) {
-        if (error) return done(error);
+    function configureNginx(error) {
+        debug('configureNginx: dns update: %j', error || {});

-        addDnsRecords(ip, function (error) {
+        certificates.ensureCertificate({ domain: config.fqdn(), location: config.adminLocation(), intrinsicFqdn: config.adminFqdn() }, function (error, certFilePath, keyFilePath) {
            if (error) return done(error);

-            subdomains.waitForDns(config.adminFqdn(), ip, 'A', { interval: 30000, times: 50000 }, function (error) {
-                if (error) return done(error);
+            gWebadminStatus.tls = true;
+
+            nginx.configureAdmin(certFilePath, keyFilePath, constants.NGINX_ADMIN_CONFIG_FILE_NAME, config.adminFqdn(), done);
+        });
+    }
+
+    // update the DNS. configure nginx regardless of whether it succeeded so that
+    // box is accessible even if dns creds are invalid
+    sysinfo.getPublicIp(function (error, ip) {
+        if (error) return configureNginx(error);
+
+        addDnsRecords(ip, function (error) {
+            if (error) return configureNginx(error);
+
+            domains.waitForDNSRecord(config.adminFqdn(), config.fqdn(), ip, 'A', { interval: 30000, times: 50000 }, function (error) {
+                if (error) return configureNginx(error);

                gWebadminStatus.dns = true;

-                certificates.ensureCertificate({ location: constants.ADMIN_LOCATION }, function (error, certFilePath, keyFilePath) {
-                    if (error) return done(error);
-
-                    gWebadminStatus.tls = true;
-
-                    nginx.configureAdmin(certFilePath, keyFilePath, constants.NGINX_ADMIN_CONFIG_FILE_NAME, config.adminFqdn(), done);
-                });
+                configureNginx();
            });
        });
    });
@@ -314,7 +365,7 @@ function activate(username, password, email, displayName, ip, auditSource, callb

                eventlog.add(eventlog.ACTION_ACTIVATE, auditSource, { });

-                platform.createMailConfig(NOOP_CALLBACK); // bounces can now be sent to the cloudron owner
+                onActivated();

                callback(null, { token: token, expires: expires });
            });
@@ -370,32 +421,23 @@ function getDisks(callback) {
    });
 }

-function getBoxAndUserDetails(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (gBoxAndUserDetails) return callback(null, gBoxAndUserDetails);
-
-    // only supported for caas
-    if (config.provider() !== 'caas') return callback(null, {});
-
-    superagent
-        .get(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn())
-        .query({ token: config.token() })
-        .timeout(30 * 1000)
-        .end(function (error, result) {
-            if (error && !error.response) return callback(new CloudronError(CloudronError.EXTERNAL_ERROR, 'Cannot reach appstore'));
-            if (result.statusCode !== 200) return callback(new CloudronError(CloudronError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
-
-            gBoxAndUserDetails = result.body;
-
-            return callback(null, gBoxAndUserDetails);
-        });
-}
-
 function getConfig(callback) {
    assert.strictEqual(typeof callback, 'function');

-    getBoxAndUserDetails(function (error, result) {
+    // result to not depend on the appstore
+    const BOX_AND_USER_TEMPLATE = {
+        box: {
+            region: null,
+            size: null,
+            plan: 'Custom Plan'
+        },
+        user: {
+            billing: false,
+            currency: ''
+        }
+    };
+
+    caas.getBoxAndUserDetails(function (error, result) {
        if (error) debug('Failed to fetch cloudron details.', error.reason, error.message);

        result = _.extend(BOX_AND_USER_TEMPLATE, result || {});
@@ -403,44 +445,30 @@ function getConfig(callback) {
        settings.getCloudronName(function (error, cloudronName) {
            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

-            settings.getDeveloperMode(function (error, developerMode) {
-                if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-                callback(null, {
-                    apiServerOrigin: config.apiServerOrigin(),
-                    webServerOrigin: config.webServerOrigin(),
-                    fqdn: config.fqdn(),
-                    version: config.version(),
-                    update: updateChecker.getUpdateInfo(),
-                    progress: progress.get(),
-                    isCustomDomain: config.isCustomDomain(),
-                    isDemo: config.isDemo(),
-                    developerMode: developerMode,
-                    region: result.box.region,
-                    size: result.box.size,
-                    billing: !!result.user.billing,
-                    plan: result.box.plan,
-                    currency: result.user.currency,
-                    memory: os.totalmem(),
-                    provider: config.provider(),
-                    cloudronName: cloudronName
-                });
+            callback(null, {
+                apiServerOrigin: config.apiServerOrigin(),
+                webServerOrigin: config.webServerOrigin(),
+                fqdn: config.fqdn(),
+                adminLocation: config.adminLocation(),
+                adminFqdn: config.adminFqdn(),
+                mailFqdn: config.mailFqdn(),
+                version: config.version(),
+                update: updateChecker.getUpdateInfo(),
+                progress: progress.getAll(),
+                isDemo: config.isDemo(),
+                region: result.box.region,
+                size: result.box.size,
+                billing: !!result.user.billing,
+                plan: result.box.plan,
+                currency: result.user.currency,
+                memory: os.totalmem(),
+                provider: config.provider(),
+                cloudronName: cloudronName
            });
        });
    });
 }

-function sendHeartbeat() {
-    if (config.provider() !== 'caas') return;
-
-    var url = config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/heartbeat';
-    superagent.post(url).query({ token: config.token(), version: config.version() }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) debug('Network error sending heartbeat.', error);
-        else if (result.statusCode !== 200) debug('Server responded to heartbeat with %s %s', result.statusCode, result.text);
-        else debug('Heartbeat sent to %s', url);
-    });
-}
-
 function ensureDkimKey(callback) {
    assert(config.fqdn(), 'fqdn is not set');

@@ -492,7 +520,7 @@ function readDkimPublicKeySync() {
 function txtRecordsWithSpf(callback) {
    assert.strictEqual(typeof callback, 'function');

-    subdomains.get('', 'TXT', function (error, txtRecords) {
+    domains.getDNSRecords('', config.fqdn(), 'TXT', function (error, txtRecords) {
        if (error) return callback(error);

        debug('txtRecordsWithSpf: current txt records - %j', txtRecords);
@@ -531,22 +559,13 @@ function addDnsRecords(ip, callback) {
    var dkimKey = readDkimPublicKeySync();
    if (!dkimKey) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, new Error('Failed to read dkim public key')));

-    var webadminRecord = { subdomain: constants.ADMIN_LOCATION, type: 'A', values: [ ip ] };
+    var webadminRecord = { subdomain: config.adminLocation(), domain: config.fqdn(), type: 'A', values: [ ip ] };
    // t=s limits the domainkey to this domain and not it's subdomains
-    var dkimRecord = { subdomain: constants.DKIM_SELECTOR + '._domainkey', type: 'TXT', values: [ '"v=DKIM1; t=s; p=' + dkimKey + '"' ] };
+    var dkimRecord = { subdomain: config.dkimSelector() + '._domainkey', domain: config.fqdn(), type: 'TXT', values: [ '"v=DKIM1; t=s; p=' + dkimKey + '"' ] };

    var records = [ ];
-    if (config.isCustomDomain()) {
-        records.push(webadminRecord);
-        records.push(dkimRecord);
-    } else {
-        // for non-custom domains, we show a noapp.html page
-        var nakedDomainRecord = { subdomain: '', type: 'A', values: [ ip ] };
-
-        records.push(nakedDomainRecord);
-        records.push(webadminRecord);
-        records.push(dkimRecord);
-    }
+    records.push(webadminRecord);
+    records.push(dkimRecord);

    debug('addDnsRecords: %j', records);

@@ -554,12 +573,12 @@ function addDnsRecords(ip, callback) {
        txtRecordsWithSpf(function (error, txtRecords) {
            if (error) return retryCallback(error);

-            if (txtRecords) records.push({ subdomain: '', type: 'TXT', values: txtRecords });
+            if (txtRecords) records.push({ subdomain: '', domain: config.fqdn(), type: 'TXT', values: txtRecords });

            debug('addDnsRecords: will update %j', records);

            async.mapSeries(records, function (record, iteratorCallback) {
-                subdomains.upsert(record.subdomain, record.type, record.values, iteratorCallback);
+                domains.upsertDNSRecords(record.subdomain, record.domain, record.type, record.values, iteratorCallback);
            }, function (error, changeIds) {
                if (error) debug('addDnsRecords: failed to update : %s. will retry', error);
                else debug('addDnsRecords: records %j added with changeIds %j', records, changeIds);
@@ -568,12 +587,49 @@ function addDnsRecords(ip, callback) {
            });
        });
    }, function (error) {
-        debug('addDnsRecords: done updating records with error:', error);
+        if (error) debug('addDnsRecords: done updating records with error:', error);
+        else debug('addDnsRecords: done');

        callback(error);
    });
 }

+function restore(backupConfig, backupId, version, callback) {
+    assert.strictEqual(typeof backupConfig, 'object');
+    assert.strictEqual(typeof backupId, 'string');
+    assert.strictEqual(typeof version, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!semver.valid(version)) return callback(new CloudronError(CloudronError.BAD_STATE, 'version is not a valid semver'));
+    if (semver.major(config.version()) !== semver.major(version) || semver.minor(config.version()) !== semver.minor(version)) return callback(new CloudronError(CloudronError.BAD_STATE, `Run cloudron-setup with --version ${version} to restore from this backup`));
+
+    user.count(function (error, count) {
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
+        if (count) return callback(new CloudronError(CloudronError.ALREADY_PROVISIONED, 'Already activated'));
+
+        backups.testConfig(backupConfig, function (error) {
+            if (error && error.reason === BackupsError.BAD_FIELD) return callback(new CloudronError(CloudronError.BAD_FIELD, error.message));
+            if (error && error.reason === BackupsError.EXTERNAL_ERROR) return callback(new CloudronError(CloudronError.EXTERNAL_ERROR, error.message));
+            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
+
+            debug(`restore: restoring from ${backupId} from provider ${backupConfig.provider}`);
+
+            gWebadminStatus.restoring = true;
+
+            callback(null); // do no block
+
+            async.series([
+                backups.restore.bind(null, backupConfig, backupId),
+                autoprovision,
+                shell.sudo.bind(null, 'restart', [ RESTART_CMD ])
+            ], function (error) {
+                debug('restore:', error);
+                gWebadminStatus.restoring = false;
+            });
+        });
+    });
+}
+
 function reboot(callback) {
    shell.sudo('reboot', [ REBOOT_CMD ], callback);
 }
@@ -596,7 +652,7 @@ function update(boxUpdateInfo, auditSource, callback) {
    // initiate the update/upgrade but do not wait for it
    if (boxUpdateInfo.upgrade) {
        debug('Starting upgrade');
-        doUpgrade(boxUpdateInfo, function (error) {
+        caas.upgrade(boxUpdateInfo, function (error) {
            if (error) {
                debug('Upgrade failed with error:', error);
                locker.unlock(locker.OP_BOX_UPDATE);
@@ -615,7 +671,6 @@ function update(boxUpdateInfo, auditSource, callback) {
    callback(null);
 }

-
 function updateToLatest(auditSource, callback) {
    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');
@@ -648,36 +703,6 @@ function doShortCircuitUpdate(boxUpdateInfo, callback) {
    callback();
 }

-function doUpgrade(boxUpdateInfo, callback) {
-    assert(boxUpdateInfo !== null && typeof boxUpdateInfo === 'object');
-
-    function upgradeError(e) {
-        progress.set(progress.UPDATE, -1, e.message);
-        callback(e);
-    }
-
-    progress.set(progress.UPDATE, 5, 'Backing up for upgrade');
-
-    backups.backupBoxAndApps({ userId: null, username: 'upgrader' }, function (error) {
-        if (error) return upgradeError(error);
-
-        superagent.post(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/upgrade')
-          .query({ token: config.token() })
-          .send({ version: boxUpdateInfo.version })
-          .timeout(30 * 1000)
-          .end(function (error, result) {
-            if (error && !error.response) return upgradeError(new Error('Network error making upgrade request: ' + error));
-            if (result.statusCode !== 202) return upgradeError(new Error(util.format('Server not ready to upgrade. statusCode: %s body: %j', result.status, result.body)));
-
-            progress.set(progress.UPDATE, 10, 'Updating base system');
-
-            // no need to unlock since this is the last thing we ever do on this box
-            callback();
-            retire('upgrade');
-        });
-    });
-}
-
 function doUpdate(boxUpdateInfo, callback) {
    assert(boxUpdateInfo && typeof boxUpdateInfo === 'object');

@@ -698,9 +723,8 @@ function doUpdate(boxUpdateInfo, callback) {
            apiServerOrigin: config.apiServerOrigin(),
            webServerOrigin: config.webServerOrigin(),
            fqdn: config.fqdn(),
-            tlsCert: config.tlsCert(),
-            tlsKey: config.tlsKey(),
-            isCustomDomain: config.isCustomDomain(),
+            adminFqdn: config.adminFqdn(),
+            adminLocation: config.adminLocation(),
            isDemo: config.isDemo(),
            zoneName: config.zoneName(),

@@ -729,36 +753,6 @@ function doUpdate(boxUpdateInfo, callback) {
    });
 }

-function installAppBundle(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (fs.existsSync(paths.FIRST_RUN_FILE)) return callback();
-
-    var bundle = config.get('appBundle');
-    debug('initialize: installing app bundle on first run: %j', bundle);
-
-    if (!bundle || bundle.length === 0) return callback();
-
-    async.eachSeries(bundle, function (appInfo, iteratorCallback) {
-        debug('autoInstall: installing %s at %s', appInfo.appstoreId, appInfo.location);
-
-        var data = {
-            appStoreId: appInfo.appstoreId,
-            location: appInfo.location,
-            portBindings: appInfo.portBindings || null,
-            accessRestriction: appInfo.accessRestriction || null,
-        };
-
-        apps.install(data, { userId: null, username: 'autoinstaller' }, iteratorCallback);
-    }, function (error) {
-        if (error) debug('autoInstallApps: ', error);
-
-        fs.writeFileSync(paths.FIRST_RUN_FILE, 'been there, done that', 'utf8');
-
-        callback();
-    });
-}
-
 function checkDiskSpace(callback) {
    callback = callback || NOOP_CALLBACK;

@@ -801,79 +795,6 @@ function checkDiskSpace(callback) {
    });
 }

-function retire(reason, info, callback) {
-    assert(reason === 'migrate' || reason === 'upgrade');
-    info = info || { };
-    callback = callback || NOOP_CALLBACK;
-
-    var data = {
-        apiServerOrigin: config.apiServerOrigin(),
-        isCustomDomain: config.isCustomDomain(),
-        fqdn: config.fqdn()
-    };
-    shell.sudo('retire', [ RETIRE_CMD, reason, JSON.stringify(info), JSON.stringify(data) ], callback);
-}
-
-function doMigrate(options, callback) {
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    var error = locker.lock(locker.OP_MIGRATE);
-    if (error) return callback(new CloudronError(CloudronError.BAD_STATE, error.message));
-
-    function unlock(error) {
-        debug('Failed to migrate', error);
-        locker.unlock(locker.OP_MIGRATE);
-        progress.set(progress.MIGRATE, -1, 'Backup failed: ' + error.message);
-    }
-
-    progress.set(progress.MIGRATE, 10, 'Backing up for migration');
-
-    // initiate the migration in the background
-    backups.backupBoxAndApps({ userId: null, username: 'migrator' }, function (error) {
-        if (error) return unlock(error);
-
-        debug('migrate: domain: %s size %s region %s', options.domain, options.size, options.region);
-
-        superagent
-          .post(config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/migrate')
-          .query({ token: config.token() })
-          .send(options)
-          .timeout(30 * 1000)
-          .end(function (error, result) {
-            if (error && !error.response) return unlock(error); // network error
-            if (result.statusCode === 409) return unlock(new CloudronError(CloudronError.BAD_STATE));
-            if (result.statusCode === 404) return unlock(new CloudronError(CloudronError.NOT_FOUND));
-            if (result.statusCode !== 202) return unlock(new CloudronError(CloudronError.EXTERNAL_ERROR, util.format('%s %j', result.status, result.body)));
-
-            progress.set(progress.MIGRATE, 10, 'Migrating');
-
-            retire('migrate', _.pick(options, 'domain', 'size', 'region'));
-        });
-    });
-
-    callback(null);
-}
-
-function migrate(options, callback) {
-    assert.strictEqual(typeof options, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    if (config.isDemo()) return callback(new CloudronError(CloudronError.BAD_FIELD, 'Not allowed in demo mode'));
-
-    if (!options.domain) return doMigrate(options, callback);
-
-    var dnsConfig = _.pick(options, 'domain', 'provider', 'accessKeyId', 'secretAccessKey', 'region', 'endpoint', 'token', 'zoneName');
-
-    settings.setDnsConfig(dnsConfig, options.domain, options.zoneName || tld.getDomain(options.domain), function (error) {
-        if (error && error.reason === SettingsError.BAD_FIELD) return callback(new CloudronError(CloudronError.BAD_FIELD, error.message));
-        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
-
-        // TODO: should probably rollback dns config if migrate fails
-        doMigrate(options, callback);
-    });
-}
-
 // called for dynamic dns setups where we have to update the IP
 function refreshDNS(callback) {
    callback = callback || NOOP_CALLBACK;
@@ -895,7 +816,7 @@ function refreshDNS(callback) {
                    // do not change state of installing apps since apptask will error if dns record already exists
                    if (app.installationState !== appdb.ISTATE_INSTALLED) return callback();

-                    subdomains.upsert(app.location, 'A', [ ip ], callback);
+                    domains.upsertDNSRecords(app.location, app.domain, 'A', [ ip ], callback);
                }, function (error) {
                    if (error) return callback(error);

@@ -17,40 +17,44 @@ exports = module.exports = {
    apiServerOrigin: apiServerOrigin,
    webServerOrigin: webServerOrigin,
    fqdn: fqdn,
+    zoneName: zoneName,
    setFqdn: setFqdn,
+    setAdminFqdn: setAdminFqdn,
+    setAdminLocation: setAdminLocation,
    token: token,
    version: version,
    setVersion: setVersion,
-    isCustomDomain: isCustomDomain,
    database: database,

    // these values are derived
    adminOrigin: adminOrigin,
    internalAdminOrigin: internalAdminOrigin,
    sysadminOrigin: sysadminOrigin, // caas routes
+    adminLocation: adminLocation,
    adminFqdn: adminFqdn,
+    mailLocation: mailLocation,
    mailFqdn: mailFqdn,
-    appFqdn: appFqdn,
-    zoneName: zoneName,
    setZoneName: setZoneName,
+    hasIPv6: hasIPv6,
+    dkimSelector: dkimSelector,

    isDemo: isDemo,

-    tlsCert: tlsCert,
-    tlsKey: tlsKey,
-
    // for testing resets to defaults
    _reset: _reset
 };

 var assert = require('assert'),
-    constants = require('./constants.js'),
    fs = require('fs'),
    path = require('path'),
    safe = require('safetydance'),
    tld = require('tldjs'),
    _ = require('underscore');

+
+// assert on unknown environment can't proceed
+assert(exports.CLOUDRON || exports.TEST, 'Unknown environment. This should not happen!');
+
 var homeDir = process.env.HOME || process.env.HOMEPATH || process.env.USERPROFILE;

 var data = { };
@@ -62,8 +66,25 @@ function baseDir() {

 var cloudronConfigFileName = path.join(baseDir(), 'configs/cloudron.conf');

+// only tests can run without a config file on disk, they use the defaults with runtime overrides
+if (exports.CLOUDRON) assert(fs.existsSync(cloudronConfigFileName), 'No cloudron.conf found, cannot proceed');
+
 function saveSync() {
-    fs.writeFileSync(cloudronConfigFileName, JSON.stringify(data, null, 4)); // functions are ignored by JSON.stringify
+    // only save values we want to have in the cloudron.conf, see start.sh
+    var conf = {
+        version: data.version,
+        token: data.token,
+        apiServerOrigin: data.apiServerOrigin,
+        webServerOrigin: data.webServerOrigin,
+        fqdn: data.fqdn,
+        adminFqdn: data.adminFqdn,
+        zoneName: data.zoneName,
+        adminLocation: data.adminLocation,
+        provider: data.provider,
+        isDemo: data.isDemo
+    };
+
+    fs.writeFileSync(cloudronConfigFileName, JSON.stringify(conf, null, 4)); // functions are ignored by JSON.stringify
 }

 function _reset(callback) {
@@ -76,45 +97,42 @@ function _reset(callback) {

 function initConfig() {
    // setup defaults
-    data.fqdn = 'localhost';
+    data.fqdn = '';
+    data.adminFqdn = '';
    data.zoneName = '';
-
+    data.adminLocation = 'my';
+    data.port = 3000;
    data.token = null;
    data.version = null;
-    data.isCustomDomain = true;
+    data.apiServerOrigin = null;
    data.webServerOrigin = null;
-    data.smtpPort = 2525; // // this value comes from mail container
+    data.provider = 'caas';
+    data.smtpPort = 2525; // this value comes from mail container
    data.sysadminPort = 3001;
    data.ldapPort = 3002;
-    data.provider = 'caas';
-    data.appBundle = [ ];

-    if (exports.CLOUDRON) {
-        data.port = 3000;
-        data.apiServerOrigin = null;
-        data.database = null;
-    } else if (exports.TEST) {
+    // keep in sync with start.sh
+    data.database = {
+        hostname: '127.0.0.1',
+        username: 'root',
+        password: 'password',
+        port: 3306,
+        name: 'box'
+    };
+
+    // overrides for local testings
+    if (exports.TEST) {
+        data.version = '1.1.1-test';
        data.port = 5454;
-        data.apiServerOrigin = 'http://localhost:6060'; // hock doesn't support https
-        data.database = {
-            hostname: 'localhost',
-            username: 'root',
-            password: '',
-            port: 3306,
-            name: 'boxtest'
-        };
        data.token = 'APPSTORE_TOKEN';
-    } else {
-        assert(false, 'Unknown environment. This should not happen!');
+        data.apiServerOrigin = 'http://localhost:6060'; // hock doesn't support https
+        data.database.password = '';
+        data.database.name = 'boxtest';
    }

-    if (safe.fs.existsSync(cloudronConfigFileName)) {
-        var existingData = safe.JSON.parse(safe.fs.readFileSync(cloudronConfigFileName, 'utf8'));
-        _.extend(data, existingData); // overwrite defaults with saved config
-        return;
-    }
-
-    saveSync();
+    // overwrite defaults with saved config
+    var existingData = safe.JSON.parse(safe.fs.readFileSync(cloudronConfigFileName, 'utf8'));
+    _.extend(data, existingData);
 }

 initConfig();
@@ -167,24 +185,33 @@ function zoneName() {
    return tld.getDomain(fqdn()) || '';
 }

-// keep this in sync with start.sh admin.conf generation code
-function appFqdn(location) {
-    assert.strictEqual(typeof location, 'string');
+function mailLocation() {
+    return get('adminLocation'); // not a typo! should be same as admin location until we figure out certificates
+}

-    if (location === '') return fqdn();
-    return isCustomDomain() ? location + '.' + fqdn() : location + '-' + fqdn();
+function setAdminLocation(location) {
+    set('adminLocation', location);
+}
+
+
+function adminLocation() {
+    return get('adminLocation');
+}
+
+function setAdminFqdn(adminFqdn) {
+    set('adminFqdn', adminFqdn);
 }

 function adminFqdn() {
-    return appFqdn(constants.ADMIN_LOCATION);
+    return get('adminFqdn');
 }

 function mailFqdn() {
-    return appFqdn(constants.MAIL_LOCATION);
+    return adminFqdn();
 }

 function adminOrigin() {
-    return 'https://' + appFqdn(constants.ADMIN_LOCATION);
+    return 'https://' + adminFqdn();
 }

 function internalAdminOrigin() {
@@ -207,10 +234,6 @@ function setVersion(version) {
    set('version', version);
 }

-function isCustomDomain() {
-    return get('isCustomDomain');
-}
-
 function database() {
    return get('database');
 }
@@ -223,12 +246,13 @@ function provider() {
    return get('provider');
 }

-function tlsCert() {
-    var certFile = path.join(baseDir(), 'configs/host.cert');
-    return safe.fs.readFileSync(certFile, 'utf8');
+function hasIPv6() {
+    const IPV6_PROC_FILE = '/proc/net/if_inet6';
+    return fs.existsSync(IPV6_PROC_FILE);
 }

-function tlsKey() {
-    var keyFile = path.join(baseDir(), 'configs/host.key');
-    return safe.fs.readFileSync(keyFile, 'utf8');
+function dkimSelector() {
+    var loc = adminLocation();
+    return loc === 'my' ? 'cloudron' : `cloudron-${loc.replace(/\./g, '')}`;
 }
+
@@ -1,12 +1,9 @@
 'use strict';

- // default admin installation location. keep in sync with ADMIN_LOCATION in setup/start.sh and BOX_ADMIN_LOCATION in appstore constants.js
 exports = module.exports = {
-    ADMIN_LOCATION: 'my',
    API_LOCATION: 'api', // this is unused but reserved for future use (#403)
    SMTP_LOCATION: 'smtp',
    IMAP_LOCATION: 'imap',
-    MAIL_LOCATION: 'my', // not a typo! should be same as admin location until we figure out certificates
    POSTMAN_LOCATION: 'postman', // used in dovecot bounces

    // These are combined into one array because users and groups become mailboxes
@@ -22,8 +19,8 @@ exports = module.exports = {
    ADMIN_NAME: 'Settings',

    ADMIN_CLIENT_ID: 'webadmin', // oauth client id
-    ADMIN_APPID: 'admin', // admin appid (settingsdb)

+    ADMIN_GROUP_NAME: 'admin',
    ADMIN_GROUP_ID: 'admin',

    NGINX_ADMIN_CONFIG_FILE_NAME: 'admin.conf',
@@ -36,8 +33,6 @@ exports = module.exports = {

    DEMO_USERNAME: 'cloudron',

-    DKIM_SELECTOR: 'cloudron',
-
    AUTOUPDATE_PATTERN_NEVER: 'never'
 };

@@ -9,6 +9,7 @@ var apps = require('./apps.js'),
    appstore = require('./appstore.js'),
    assert = require('assert'),
    backups = require('./backups.js'),
+    caas = require('./caas.js'),
    certificates = require('./certificates.js'),
    cloudron = require('./cloudron.js'),
    config = require('./config.js'),
@@ -23,21 +24,23 @@ var apps = require('./apps.js'),
    semver = require('semver'),
    updateChecker = require('./updatechecker.js');

-var gAliveJob = null, // send periodic stats
-    gAppUpdateCheckerJob = null,
-    gAutoupdaterJob = null,
-    gBackupJob = null,
-    gBoxUpdateCheckerJob = null,
-    gCertificateRenewJob = null,
-    gCheckDiskSpaceJob = null,
-    gCleanupBackupsJob = null,
-    gCleanupEventlogJob = null,
-    gCleanupTokensJob = null,
-    gDockerVolumeCleanerJob = null,
-    gDynamicDNSJob = null,
-    gHeartbeatJob = null, // for CaaS health check
-    gSchedulerSyncJob = null,
-    gDigestEmailJob = null;
+var gJobs = {
+    alive: null, // send periodic stats
+    autoUpdater: null,
+    appUpdateChecker: null,
+    backup: null,
+    boxUpdateChecker: null,
+    caasHeartbeat: null,
+    checkDiskSpace: null,
+    certificateRenew: null,
+    cleanupBackups: null,
+    cleanupEventlog: null,
+    cleanupTokens: null,
+    digestEmail: null,
+    dockerVolumeCleaner: null,
+    dynamicDNS: null,
+    schedulerSync: null
+};

 var NOOP_CALLBACK = function (error) { if (error) console.error(error); };
 var AUDIT_SOURCE = { userId: null, username: 'cron' };
@@ -53,21 +56,21 @@ var AUDIT_SOURCE = { userId: null, username: 'cron' };
 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');

-    gHeartbeatJob = new CronJob({
-        cronTime: '00 */1 * * * *', // every minute
-        onTick: cloudron.sendHeartbeat,
-        start: false
-    });
-    // hack: send the first heartbeat only after we are running for 60 seconds
-    // required as we end up sending a heartbeat and then cloudron-setup reboots the server
-    setTimeout(function () {
-        if (!gHeartbeatJob) return; // already uninitalized
-        gHeartbeatJob.start();
-        cloudron.sendHeartbeat();
-    }, 1000 * 60);
+    if (config.provider() === 'caas') {
+        // hack: send the first heartbeat only after we are running for 60 seconds
+        // required as we end up sending a heartbeat and then cloudron-setup reboots the server
+        var seconds = (new Date()).getSeconds() - 1;
+        if (seconds === -1) seconds = 59;
+
+        gJobs.caasHeartbeat = new CronJob({
+            cronTime: `${seconds} */1 * * * *`, // every minute
+            onTick: caas.sendHeartbeat,
+            start: true
+        });
+    }

    var randomHourMinute = Math.floor(60*Math.random());
-    gAliveJob = new CronJob({
+    gJobs.alive = new CronJob({
        cronTime: '00 ' + randomHourMinute + ' * * * *', // every hour on a random minute
        onTick: appstore.sendAliveStatus,
        start: true
@@ -93,16 +96,16 @@ function recreateJobs(tz) {

    debug('Creating jobs with timezone %s', tz);

-    if (gBackupJob) gBackupJob.stop();
-    gBackupJob = new CronJob({
+    if (gJobs.backup) gJobs.backup.stop();
+    gJobs.backup = new CronJob({
        cronTime: '00 00 */6 * * *', // every 6 hours. backups.ensureBackup() will only trigger a backup once per day
        onTick: backups.ensureBackup.bind(null, AUDIT_SOURCE, NOOP_CALLBACK),
        start: true,
        timeZone: tz
    });

-    if (gCheckDiskSpaceJob) gCheckDiskSpaceJob.stop();
-    gCheckDiskSpaceJob = new CronJob({
+    if (gJobs.checkDiskSpace) gJobs.checkDiskSpace.stop();
+    gJobs.checkDiskSpace = new CronJob({
        cronTime: '00 30 */4 * * *', // every 4 hours
        onTick: cloudron.checkDiskSpace,
        start: true,
@@ -112,72 +115,72 @@ function recreateJobs(tz) {
    // randomized pattern per cloudron every hour
    var randomMinute = Math.floor(60*Math.random());

-    if (gBoxUpdateCheckerJob) gBoxUpdateCheckerJob.stop();
-    gBoxUpdateCheckerJob = new CronJob({
+    if (gJobs.boxUpdateCheckerJob) gJobs.boxUpdateCheckerJob.stop();
+    gJobs.boxUpdateCheckerJob = new CronJob({
        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
        onTick: updateChecker.checkBoxUpdates,
        start: true,
        timeZone: tz
    });

-    if (gAppUpdateCheckerJob) gAppUpdateCheckerJob.stop();
-    gAppUpdateCheckerJob = new CronJob({
+    if (gJobs.appUpdateChecker) gJobs.appUpdateChecker.stop();
+    gJobs.appUpdateChecker = new CronJob({
        cronTime: '00 ' + randomMinute + ' * * * *', // once an hour
        onTick: updateChecker.checkAppUpdates,
        start: true,
        timeZone: tz
    });

-    if (gCleanupTokensJob) gCleanupTokensJob.stop();
-    gCleanupTokensJob = new CronJob({
+    if (gJobs.cleanupTokens) gJobs.cleanupTokens.stop();
+    gJobs.cleanupTokens = new CronJob({
        cronTime: '00 */30 * * * *', // every 30 minutes
        onTick: janitor.cleanupTokens,
        start: true,
        timeZone: tz
    });

-    if (gCleanupBackupsJob) gCleanupBackupsJob.stop();
-    gCleanupBackupsJob = new CronJob({
+    if (gJobs.cleanupBackups) gJobs.cleanupBackups.stop();
+    gJobs.cleanupBackups = new CronJob({
        cronTime: '00 45 */6 * * *', // every 6 hours. try not to overlap with ensureBackup job
-        onTick: backups.cleanup,
+        onTick: backups.cleanup.bind(null, AUDIT_SOURCE, NOOP_CALLBACK),
        start: true,
        timeZone: tz
    });

-    if (gCleanupEventlogJob) gCleanupEventlogJob.stop();
-    gCleanupEventlogJob = new CronJob({
+    if (gJobs.cleanupEventlog) gJobs.cleanupEventlog.stop();
+    gJobs.cleanupEventlog = new CronJob({
        cronTime: '00 */30 * * * *', // every 30 minutes
        onTick: eventlog.cleanup,
        start: true,
        timeZone: tz
    });

-    if (gDockerVolumeCleanerJob) gDockerVolumeCleanerJob.stop();
-    gDockerVolumeCleanerJob = new CronJob({
+    if (gJobs.dockerVolumeCleaner) gJobs.dockerVolumeCleaner.stop();
+    gJobs.dockerVolumeCleaner = new CronJob({
        cronTime: '00 00 */12 * * *', // every 12 hours
        onTick: janitor.cleanupDockerVolumes,
        start: true,
        timeZone: tz
    });

-    if (gSchedulerSyncJob) gSchedulerSyncJob.stop();
-    gSchedulerSyncJob = new CronJob({
+    if (gJobs.schedulerSync) gJobs.schedulerSync.stop();
+    gJobs.schedulerSync = new CronJob({
        cronTime: config.TEST ? '*/10 * * * * *' : '00 */1 * * * *', // every minute
        onTick: scheduler.sync,
        start: true,
        timeZone: tz
    });

-    if (gCertificateRenewJob) gCertificateRenewJob.stop();
-    gCertificateRenewJob = new CronJob({
+    if (gJobs.certificateRenew) gJobs.certificateRenew.stop();
+    gJobs.certificateRenew = new CronJob({
        cronTime: '00 00 */12 * * *', // every 12 hours
        onTick: certificates.renewAll.bind(null, AUDIT_SOURCE, NOOP_CALLBACK),
        start: true,
        timeZone: tz
    });

-    if (gDigestEmailJob) gDigestEmailJob.stop();
-    gDigestEmailJob = new CronJob({
+    if (gJobs.digestEmail) gJobs.digestEmail.stop();
+    gJobs.digestEmail = new CronJob({
        cronTime: '00 00 00 * * 3', // every wednesday
        onTick: digest.maybeSend,
        start: true,
@@ -187,15 +190,15 @@ function recreateJobs(tz) {

 function autoupdatePatternChanged(pattern) {
    assert.strictEqual(typeof pattern, 'string');
-    assert(gBoxUpdateCheckerJob);
+    assert(gJobs.boxUpdateCheckerJob);

    debug('Auto update pattern changed to %s', pattern);

-    if (gAutoupdaterJob) gAutoupdaterJob.stop();
+    if (gJobs.autoUpdater) gJobs.autoUpdater.stop();

    if (pattern === constants.AUTOUPDATE_PATTERN_NEVER) return;

-    gAutoupdaterJob = new CronJob({
+    gJobs.autoUpdater = new CronJob({
        cronTime: pattern,
        onTick: function() {
            var updateInfo = updateChecker.getUpdateInfo();
@@ -214,26 +217,26 @@ function autoupdatePatternChanged(pattern) {
            }
        },
        start: true,
-        timeZone: gBoxUpdateCheckerJob.cronTime.zone // hack
+        timeZone: gJobs.boxUpdateCheckerJob.cronTime.zone // hack
    });
 }

 function dynamicDNSChanged(enabled) {
    assert.strictEqual(typeof enabled, 'boolean');
-    assert(gBoxUpdateCheckerJob);
+    assert(gJobs.boxUpdateCheckerJob);

    debug('Dynamic DNS setting changed to %s', enabled);

    if (enabled) {
-        gDynamicDNSJob = new CronJob({
+        gJobs.dynamicDNS = new CronJob({
            cronTime: '00 */10 * * * *',
            onTick: cloudron.refreshDNS,
            start: true,
-            timeZone: gBoxUpdateCheckerJob.cronTime.zone // hack
+            timeZone: gJobs.boxUpdateCheckerJob.cronTime.zone // hack
        });
    } else {
-        if (gDynamicDNSJob) gDynamicDNSJob.stop();
-        gDynamicDNSJob = null;
+        if (gJobs.dynamicDNS) gJobs.dynamicDNS.stop();
+        gJobs.dynamicDNS = null;
    }
 }

@@ -242,48 +245,13 @@ function uninitialize(callback) {

    settings.events.removeListener(settings.TIME_ZONE_KEY, recreateJobs);
    settings.events.removeListener(settings.AUTOUPDATE_PATTERN_KEY, autoupdatePatternChanged);
+    settings.events.removeListener(settings.DYNAMIC_DNS_KEY, dynamicDNSChanged);

-    if (gAutoupdaterJob) gAutoupdaterJob.stop();
-    gAutoupdaterJob = null;
-
-    if (gBoxUpdateCheckerJob) gBoxUpdateCheckerJob.stop();
-    gBoxUpdateCheckerJob = null;
-
-    if (gAppUpdateCheckerJob) gAppUpdateCheckerJob.stop();
-    gAppUpdateCheckerJob = null;
-
-    if (gHeartbeatJob) gHeartbeatJob.stop();
-    gHeartbeatJob = null;
-
-    if (gAliveJob) gAliveJob.stop();
-    gAliveJob = null;
-
-    if (gBackupJob) gBackupJob.stop();
-    gBackupJob = null;
-
-    if (gCleanupTokensJob) gCleanupTokensJob.stop();
-    gCleanupTokensJob = null;
-
-    if (gCleanupBackupsJob) gCleanupBackupsJob.stop();
-    gCleanupBackupsJob = null;
-
-    if (gCleanupEventlogJob) gCleanupEventlogJob.stop();
-    gCleanupEventlogJob = null;
-
-    if (gDockerVolumeCleanerJob) gDockerVolumeCleanerJob.stop();
-    gDockerVolumeCleanerJob = null;
-
-    if (gSchedulerSyncJob) gSchedulerSyncJob.stop();
-    gSchedulerSyncJob = null;
-
-    if (gCertificateRenewJob) gCertificateRenewJob.stop();
-    gCertificateRenewJob = null;
-
-    if (gDynamicDNSJob) gDynamicDNSJob.stop();
-    gDynamicDNSJob = null;
-
-    if (gDigestEmailJob) gDigestEmailJob.stop();
-    gDigestEmailJob = null;
+    for (var job in gJobs) {
+        if (!gJobs[job]) continue;
+        gJobs[job].stop();
+        gJobs[job] = null;
+    }

    callback();
 }
@@ -10,6 +10,9 @@ exports = module.exports = {
    rollback: rollback,
    commit: commit,

+    importFromFile: importFromFile,
+    exportToFile: exportToFile,
+
    _clear: clear
 };

@@ -101,6 +104,7 @@ function clear(callback) {
    async.series([
        child_process.exec.bind(null, cmd),
        require('./clientdb.js')._addDefaultClients,
+        require('./domaindb.js')._addDefaultDomain,
        require('./groupdb.js')._addDefaultGroups
    ], callback);
 }
@@ -183,3 +187,27 @@ function transaction(queries, callback) {
    });
 }

+function importFromFile(file, callback) {
+    assert.strictEqual(typeof file, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var password = config.database().password ? '-p' + config.database().password : '--skip-password';
+
+    var cmd = `/usr/bin/mysql -u ${config.database().username} ${password} ${config.database().name} < ${file}`;
+
+    async.series([
+        query.bind(null, 'CREATE DATABASE IF NOT EXISTS box'),
+        child_process.exec.bind(null, cmd)
+    ], callback);
+}
+
+function exportToFile(file, callback) {
+    assert.strictEqual(typeof file, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var password = config.database().password ? '-p' + config.database().password : '--skip-password';
+    var cmd = `/usr/bin/mysqldump -u root ${password} --single-transaction --routines \
+            --triggers ${config.database().name} > "${file}"`;
+
+    child_process.exec(cmd, callback);
+}
@@ -5,21 +5,14 @@
 exports = module.exports = {
    DeveloperError: DeveloperError,

-    isEnabled: isEnabled,
-    setEnabled: setEnabled,
-    issueDeveloperToken: issueDeveloperToken,
-    getNonApprovedApps: getNonApprovedApps
+    issueDeveloperToken: issueDeveloperToken
 };

 var assert = require('assert'),
    clients = require('./clients.js'),
-    config = require('./config.js'),
    constants = require('./constants.js'),
-    debug = require('debug')('box:developer'),
    eventlog = require('./eventlog.js'),
    tokendb = require('./tokendb.js'),
-    settings = require('./settings.js'),
-    superagent = require('superagent'),
    util = require('util');

 function DeveloperError(reason, errorOrMessage) {
@@ -44,29 +37,6 @@ util.inherits(DeveloperError, Error);
 DeveloperError.INTERNAL_ERROR = 'Internal Error';
 DeveloperError.EXTERNAL_ERROR = 'External Error';

-function isEnabled(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    settings.getDeveloperMode(function (error, enabled) {
-        if (error) return callback(new DeveloperError(DeveloperError.INTERNAL_ERROR, error));
-        callback(null, enabled);
-    });
-}
-
-function setEnabled(enabled, auditSource, callback) {
-    assert.strictEqual(typeof enabled, 'boolean');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof callback, 'function');
-
-    settings.setDeveloperMode(enabled, function (error) {
-        if (error) return callback(new DeveloperError(DeveloperError.INTERNAL_ERROR, error));
-
-        eventlog.add(eventlog.ACTION_CLI_MODE, auditSource, { enabled: enabled });
-
-        callback(null);
-    });
-}
-
 function issueDeveloperToken(user, auditSource, callback) {
    assert.strictEqual(typeof user, 'object');
    assert.strictEqual(typeof auditSource, 'object');
@@ -84,19 +54,3 @@ function issueDeveloperToken(user, auditSource, callback) {
        callback(null, { token: token, expiresAt: new Date(expiresAt).toISOString() });
    });
 }
-
-function getNonApprovedApps(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    var url = config.apiServerOrigin() + '/api/v1/boxes/' + config.fqdn() + '/apps';
-    superagent.get(url).query({ token: config.token(), boxVersion: config.version() }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return callback(new DeveloperError(DeveloperError.EXTERNAL_ERROR, error));
-        if (result.statusCode === 401 || result.statusCode === 403) {
-            debug('Failed to list apps in development. Appstore token invalid or missing. Returning empty list.', result.body);
-            return callback(null, []);
-        }
-        if (result.statusCode !== 200) return callback(new DeveloperError(DeveloperError.EXTERNAL_ERROR, util.format('App listing failed. %s %j', result.status, result.body)));
-
-        callback(null, result.body.apps || []);
-    });
-}
@@ -33,31 +33,38 @@ function maybeSend(callback) {

            var hasSubscription = result && result.plan.id !== 'free' && result.plan.id !== 'undecided';

-            eventlog.getByActionLastWeek(eventlog.ACTION_APP_UPDATE, function (error, appUpdates) {
+            eventlog.getByCreationTime(new Date(new Date() - 7*86400000), function (error, events) {
                if (error) return callback(error);

-                eventlog.getByActionLastWeek(eventlog.ACTION_UPDATE, function (error, boxUpdates) {
-                    if (error) return callback(error);
+                var appUpdates = events.filter(function (e) { return e.action === eventlog.ACTION_APP_UPDATE; }).map(function (e) { return e.data; });
+                var boxUpdates = events.filter(function (e) { return e.action === eventlog.ACTION_UPDATE; }).map(function (e) { return e.data; });
+                var certRenewals = events.filter(function (e) { return e.action === eventlog.ACTION_CERTIFICATE_RENEWAL; }).map(function (e) { return e.data; });
+                var usersAdded = events.filter(function (e) { return e.action === eventlog.ACTION_USER_ADD; }).map(function (e) { return e.data; });
+                var usersRemoved = events.filter(function (e) { return e.action === eventlog.ACTION_USER_REMOVE; }).map(function (e) { return e.data; });
+                var finishedBackups = events.filter(function (e) { return e.action === eventlog.ACTION_BACKUP_FINISH && !e.errorMessage; }).map(function (e) { return e.data; });

-                    var info = {
-                        hasSubscription: hasSubscription,
+                if (error) return callback(error);

-                        pendingAppUpdates: pendingAppUpdates,
-                        pendingBoxUpdate: updateInfo.box || null,
+                var info = {
+                    hasSubscription: hasSubscription,

-                        finishedAppUpdates: (appUpdates || []).map(function (e) { return e.data; }),
-                        finishedBoxUpdates: (boxUpdates || []).map(function (e) { return e.data; })
-                    };
+                    pendingAppUpdates: pendingAppUpdates,
+                    pendingBoxUpdate: updateInfo.box || null,

-                    if (info.pendingAppUpdates.length || info.pendingBoxUpdate || info.finishedAppUpdates.length || info.finishedBoxUpdates.length) {
-                        debug('maybeSend: sending digest email', info);
-                        mailer.sendDigest(info);
-                    } else {
-                        debug('maybeSend: nothing happened, NOT sending digest email');
-                    }
+                    finishedAppUpdates: appUpdates,
+                    finishedBoxUpdates: boxUpdates,

-                    callback();
-                });
+                    certRenewals: certRenewals,
+                    finishedBackups: finishedBackups, // only the successful backups
+                    usersAdded: usersAdded,
+                    usersRemoved: usersRemoved // unused because we don't have username to work with
+                };
+
+                // always send digest for backup failure notification
+                debug('maybeSend: sending digest email', info);
+                mailer.sendDigest(info);
+
+                callback();
            });
        });
    });
@@ -11,10 +11,17 @@ exports = module.exports = {
 var assert = require('assert'),
    config = require('../config.js'),
    debug = require('debug')('box:dns/caas'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
    superagent = require('superagent'),
    util = require('util');

+function getFqdn(subdomain, domain) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+
+    return (subdomain === '') ? domain : subdomain + '-' + domain;
+}
+
 function add(dnsConfig, zoneName, subdomain, type, values, callback) {
    assert.strictEqual(typeof dnsConfig, 'object');
    assert.strictEqual(typeof zoneName, 'string');
@@ -23,9 +30,9 @@ function add(dnsConfig, zoneName, subdomain, type, values, callback) {
    assert(util.isArray(values));
    assert.strictEqual(typeof callback, 'function');

-    var fqdn = subdomain !== '' && type === 'TXT' ? subdomain + '.' + config.fqdn() : config.appFqdn(subdomain);
+    var fqdn = subdomain !== '' && type === 'TXT' ? subdomain + '.' + dnsConfig.fqdn : getFqdn(subdomain, dnsConfig.fqdn);

-    debug('add: %s for zone %s of type %s with values %j', subdomain, zoneName, type, values);
+    debug('add: %s for zone %s of type %s with values %j', subdomain, dnsConfig.fqdn, type, values);

    var data = {
        type: type,
@@ -38,10 +45,10 @@ function add(dnsConfig, zoneName, subdomain, type, values, callback) {
        .send(data)
        .timeout(30 * 1000)
        .end(function (error, result) {
-            if (error && !error.response) return callback(error);
-            if (result.statusCode === 400) return callback(new SubdomainError(SubdomainError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new SubdomainError(SubdomainError.STILL_BUSY));
-            if (result.statusCode !== 201) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+            if (error && !error.response) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+            if (result.statusCode === 400) return callback(new DomainError(DomainError.BAD_FIELD, result.body.message));
+            if (result.statusCode === 420) return callback(new DomainError(DomainError.STILL_BUSY));
+            if (result.statusCode !== 201) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));

            return callback(null, result.body.changeId);
        });
@@ -54,17 +61,17 @@ function get(dnsConfig, zoneName, subdomain, type, callback) {
    assert.strictEqual(typeof type, 'string');
    assert.strictEqual(typeof callback, 'function');

-    var fqdn = subdomain !== '' && type === 'TXT' ? subdomain + '.' + config.fqdn() : config.appFqdn(subdomain);
+    var fqdn = subdomain !== '' && type === 'TXT' ? subdomain + '.' + dnsConfig.fqdn : getFqdn(subdomain, dnsConfig.fqdn);

-    debug('get: zoneName: %s subdomain: %s type: %s fqdn: %s', zoneName, subdomain, type, fqdn);
+    debug('get: zoneName: %s subdomain: %s type: %s fqdn: %s', dnsConfig.fqdn, subdomain, type, fqdn);

    superagent
        .get(config.apiServerOrigin() + '/api/v1/domains/' + fqdn)
        .query({ token: dnsConfig.token, type: type })
        .timeout(30 * 1000)
        .end(function (error, result) {
-            if (error && !error.response) return callback(error);
-            if (result.statusCode !== 200) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+            if (error && !error.response) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+            if (result.statusCode !== 200) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));

            return callback(null, result.body.values);
        });
@@ -89,7 +96,7 @@ function del(dnsConfig, zoneName, subdomain, type, values, callback) {
    assert(util.isArray(values));
    assert.strictEqual(typeof callback, 'function');

-    debug('del: %s for zone %s of type %s with values %j', subdomain, zoneName, type, values);
+    debug('del: %s for zone %s of type %s with values %j', subdomain, dnsConfig.fqdn, type, values);

    var data = {
        type: type,
@@ -97,16 +104,16 @@ function del(dnsConfig, zoneName, subdomain, type, values, callback) {
    };

    superagent
-        .del(config.apiServerOrigin() + '/api/v1/domains/' + config.appFqdn(subdomain))
+        .del(config.apiServerOrigin() + '/api/v1/domains/' + getFqdn(subdomain, dnsConfig.fqdn))
        .query({ token: dnsConfig.token })
        .send(data)
        .timeout(30 * 1000)
        .end(function (error, result) {
-            if (error && !error.response) return callback(error);
-            if (result.statusCode === 400) return callback(new SubdomainError(SubdomainError.BAD_FIELD, result.body.message));
-            if (result.statusCode === 420) return callback(new SubdomainError(SubdomainError.STILL_BUSY));
-            if (result.statusCode === 404) return callback(new SubdomainError(SubdomainError.NOT_FOUND));
-            if (result.statusCode !== 204) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+            if (error && !error.response) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+            if (result.statusCode === 400) return callback(new DomainError(DomainError.BAD_FIELD, result.body.message));
+            if (result.statusCode === 420) return callback(new DomainError(DomainError.STILL_BUSY));
+            if (result.statusCode === 404) return callback(new DomainError(DomainError.NOT_FOUND));
+            if (result.statusCode !== 204) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));

            return callback(null);
        });
@@ -120,7 +127,8 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, ip, callback) {
    assert.strictEqual(typeof callback, 'function');

    var credentials = {
-        provider: dnsConfig.provider
+        token: dnsConfig.token,
+        fqdn: domain
    };

    return callback(null, credentials);
@@ -10,12 +10,12 @@ exports = module.exports = {

 var assert = require('assert'),
    async = require('async'),
-    dns = require('dns'),
-    _ = require('underscore'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
-    superagent = require('superagent'),
    debug = require('debug')('box:dns/cloudflare'),
-    util = require('util');
+    dns = require('dns'),
+    DomainError = require('../domains.js').DomainError,
+    superagent = require('superagent'),
+    util = require('util'),
+    _ = require('underscore');

 // we are using latest v4 stable API https://api.cloudflare.com/#getting-started-endpoints
 var CLOUDFLARE_ENDPOINT = 'https://api.cloudflare.com/client/v4';
@@ -24,8 +24,8 @@ function translateRequestError(result, callback) {
    assert.strictEqual(typeof result, 'object');
    assert.strictEqual(typeof callback, 'function');

-    if (result.statusCode === 404) return callback(new SubdomainError(SubdomainError.NOT_FOUND, util.format('%s %j', result.statusCode, 'API does not exist')));
-    if (result.statusCode === 422) return callback(new SubdomainError(SubdomainError.BAD_FIELD, result.body.message));
+    if (result.statusCode === 404) return callback(new DomainError(DomainError.NOT_FOUND, util.format('%s %j', result.statusCode, 'API does not exist')));
+    if (result.statusCode === 422) return callback(new DomainError(DomainError.BAD_FIELD, result.body.message));
    if ((result.statusCode === 400 || result.statusCode === 401 || result.statusCode === 403) && result.body.errors.length > 0) {
        let error = result.body.errors[0];
        let message = error.message;
@@ -34,10 +34,10 @@ function translateRequestError(result, callback) {
            else message = 'Invalid credentials';
        }

-        return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, message));
+        return callback(new DomainError(DomainError.ACCESS_DENIED, message));
    }

-    callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
+    callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('%s %j', result.statusCode, result.body)));
 }

 function getZoneByName(dnsConfig, zoneName, callback) {
@@ -52,7 +52,7 @@ function getZoneByName(dnsConfig, zoneName, callback) {
      .end(function (error, result) {
        if (error && !error.response) return callback(error);
        if (result.statusCode !== 200 || result.body.success !== true) return translateRequestError(result, callback);
-        if (!result.body.result.length) return callback(new SubdomainError(SubdomainError.NOT_FOUND, util.format('%s %j', result.statusCode, result.body)));
+        if (!result.body.result.length) return callback(new DomainError(DomainError.NOT_FOUND, util.format('%s %j', result.statusCode, result.body)));

        callback(null, result.body.result[0]);
    });
@@ -66,18 +66,18 @@ function getDNSRecordsByZoneId(dnsConfig, zoneId, zoneName, subdomain, type, cal
    assert.strictEqual(typeof type, 'string');
    assert.strictEqual(typeof callback, 'function');

+    var fqdn = subdomain === '' ? zoneName : subdomain + '.' + zoneName;
+
    superagent.get(CLOUDFLARE_ENDPOINT + '/zones/' + zoneId + '/dns_records')
      .set('X-Auth-Key',dnsConfig.token)
      .set('X-Auth-Email',dnsConfig.email)
+      .query({ type: type, name: fqdn })
      .timeout(30 * 1000)
      .end(function (error, result) {
        if (error && !error.response) return callback(error);
        if (result.statusCode !== 200 || result.body.success !== true) return translateRequestError(result, callback);

-        var fqdn = subdomain === '' ? zoneName : subdomain + '.' + zoneName;
-        var tmp = result.body.result.filter(function (record) {
-            return (record.type === type && record.name === fqdn);
-        });
+        var tmp = result.body.result;

        return callback(null, tmp);
    });
@@ -109,10 +109,18 @@ function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {
            var i = 0;

            async.eachSeries(values, function (value, callback) {
+                var priority = null;
+
+                if (type === 'MX') {
+                    priority = value.split(' ')[0];
+                    value = value.split(' ')[1];
+                }
+
                var data = {
                    type: type,
                    name: fqdn,
                    content: value,
+                    priority: priority,
                    ttl: 120  // 1 means "automatic" (meaning 300ms) and 120 is the lowest supported
                };

@@ -225,11 +233,10 @@ function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
    assert.strictEqual(typeof ip, 'string');
    assert.strictEqual(typeof callback, 'function');

-    if (!dnsConfig.token || typeof dnsConfig.token !== 'string') return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'token must be a non-empty string'));
-    if (!dnsConfig.email || typeof dnsConfig.email !== 'string') return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'email must be a non-empty string'));
+    if (!dnsConfig.token || typeof dnsConfig.token !== 'string') return callback(new DomainError(DomainError.BAD_FIELD, 'token must be a non-empty string'));
+    if (!dnsConfig.email || typeof dnsConfig.email !== 'string') return callback(new DomainError(DomainError.BAD_FIELD, 'email must be a non-empty string'));

    var credentials = {
-        provider: dnsConfig.provider,
        token: dnsConfig.token,
        email: dnsConfig.email
    };
@@ -237,23 +244,31 @@ function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here

    dns.resolveNs(zoneName, function (error, nameservers) {
-        if (error && error.code === 'ENOTFOUND') return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
-        if (error || !nameservers) return callback(new SubdomainError(SubdomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));
+        if (error && error.code === 'ENOTFOUND') return callback(new DomainError(DomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
+        if (error || !nameservers) return callback(new DomainError(DomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));

        getZoneByName(dnsConfig, zoneName, function(error, result) {
            if (error) return callback(error);

            if (!_.isEqual(result.name_servers.sort(), nameservers.sort())) {
                debug('verifyDnsConfig: %j and %j do not match', nameservers, result.name_servers);
-                return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Domain nameservers are not set to Cloudflare'));
+                return callback(new DomainError(DomainError.BAD_FIELD, 'Domain nameservers are not set to Cloudflare'));
            }

-            upsert(credentials, zoneName, 'my', 'A', [ ip ], function (error, changeId) {
+            const testSubdomain = 'cloudrontestdns';
+
+            upsert(credentials, zoneName, testSubdomain, 'A', [ ip ], function (error, changeId) {
                if (error) return callback(error);

-                debug('verifyDnsConfig: A record added with change id %s', changeId);
+                debug('verifyDnsConfig: Test A record added with change id %s', changeId);

-                callback(null, credentials);
+                del(dnsConfig, zoneName, testSubdomain, 'A', [ ip ], function (error) {
+                    if (error) return callback(error);
+
+                    debug('verifyDnsConfig: Test A record removed again');
+
+                    callback(null, credentials);
+                });
            });
        });
    });
@@ -10,10 +10,10 @@ exports = module.exports = {

 var assert = require('assert'),
    async = require('async'),
-    constants = require('../constants.js'),
    debug = require('debug')('box:dns/digitalocean'),
    dns = require('dns'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
+    safe = require('safetydance'),
    superagent = require('superagent'),
    util = require('util');

@@ -30,22 +30,34 @@ function getInternal(dnsConfig, zoneName, subdomain, type, callback) {
    assert.strictEqual(typeof type, 'string');
    assert.strictEqual(typeof callback, 'function');

-    superagent.get(DIGITALOCEAN_ENDPOINT + '/v2/domains/' + zoneName + '/records')
-      .set('Authorization', 'Bearer ' + dnsConfig.token)
-      .timeout(30 * 1000)
-      .end(function (error, result) {
-        if (error && !error.response) return callback(error);
-        if (result.statusCode === 404) return callback(new SubdomainError(SubdomainError.NOT_FOUND, formatError(result)));
-        if (result.statusCode === 403 || result.statusCode === 401) return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, formatError(result)));
-        if (result.statusCode !== 200) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, formatError(result)));
+    var nextPage = null, matchingRecords = [];

-        var tmp = result.body.domain_records.filter(function (record) {
-            return (record.type === type && record.name === subdomain);
-        });
+    async.doWhilst(function (iteratorDone) {
+        var url = nextPage ? nextPage : DIGITALOCEAN_ENDPOINT + '/v2/domains/' + zoneName + '/records';

-        debug('getInternal: %j', tmp);
+        superagent.get(url)
+            .set('Authorization', 'Bearer ' + dnsConfig.token)
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+                if (result.statusCode === 404) return callback(new DomainError(DomainError.NOT_FOUND, formatError(result)));
+                if (result.statusCode === 403 || result.statusCode === 401) return callback(new DomainError(DomainError.ACCESS_DENIED, formatError(result)));
+                if (result.statusCode !== 200) return callback(new DomainError(DomainError.EXTERNAL_ERROR, formatError(result)));

-        return callback(null, tmp);
+                matchingRecords = matchingRecords.concat(result.body.domain_records.filter(function (record) {
+                    return (record.type === type && record.name === subdomain);
+                }));
+
+                nextPage = (result.body.links && result.body.links.pages) ? result.body.links.pages.next : null;
+
+                iteratorDone();
+            });
+    }, function () { return !!nextPage; }, function (error) {
+        if (error) return callback(error);
+
+        debug('getInternal: %j', matchingRecords);
+
+        return callback(null, matchingRecords);
    });
 }

@@ -65,9 +77,9 @@ function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {
        if (error) return callback(error);

        // used to track available records to update instead of create
-        var i = 0;
+        var i = 0, recordIds = [];

-        async.eachSeries(values, function (value, callback) {
+        async.eachSeries(values, function (value, iteratorCallback) {
            var priority = null;

            if (type === 'MX') {
@@ -85,38 +97,42 @@ function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {

            if (i >= result.length) {
                superagent.post(DIGITALOCEAN_ENDPOINT + '/v2/domains/' + zoneName + '/records')
-                  .set('Authorization', 'Bearer ' + dnsConfig.token)
-                  .send(data)
-                  .timeout(30 * 1000)
-                  .end(function (error, result) {
-                    if (error && !error.response) return callback(error);
-                    if (result.statusCode === 403 || result.statusCode === 401) return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, formatError(result)));
-                    if (result.statusCode === 422) return callback(new SubdomainError(SubdomainError.BAD_FIELD, result.body.message));
-                    if (result.statusCode !== 201) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, formatError(result)));
+                    .set('Authorization', 'Bearer ' + dnsConfig.token)
+                    .send(data)
+                    .timeout(30 * 1000)
+                    .end(function (error, result) {
+                        if (error && !error.response) return iteratorCallback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+                        if (result.statusCode === 403 || result.statusCode === 401) return iteratorCallback(new DomainError(DomainError.ACCESS_DENIED, formatError(result)));
+                        if (result.statusCode === 422) return iteratorCallback(new DomainError(DomainError.BAD_FIELD, result.body.message));
+                        if (result.statusCode !== 201) return iteratorCallback(new DomainError(DomainError.EXTERNAL_ERROR, formatError(result)));

-                    return callback(null);
-                });
+                        recordIds.push(safe.query(result.body, 'domain_record.id'));
+
+                        return iteratorCallback(null);
+                    });
            } else {
                superagent.put(DIGITALOCEAN_ENDPOINT + '/v2/domains/' + zoneName + '/records/' + result[i].id)
-                  .set('Authorization', 'Bearer ' + dnsConfig.token)
-                  .send(data)
-                  .timeout(30 * 1000)
-                  .end(function (error, result) {
+                    .set('Authorization', 'Bearer ' + dnsConfig.token)
+                    .send(data)
+                    .timeout(30 * 1000)
+                    .end(function (error, result) {
                    // increment, as we have consumed the record
-                    ++i;
+                        ++i;

-                    if (error && !error.response) return callback(error);
-                    if (result.statusCode === 403 || result.statusCode === 401) return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, formatError(result)));
-                    if (result.statusCode === 422) return callback(new SubdomainError(SubdomainError.BAD_FIELD, result.body.message));
-                    if (result.statusCode !== 200) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, formatError(result)));
+                        if (error && !error.response) return iteratorCallback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+                        if (result.statusCode === 403 || result.statusCode === 401) return iteratorCallback(new DomainError(DomainError.ACCESS_DENIED, formatError(result)));
+                        if (result.statusCode === 422) return iteratorCallback(new DomainError(DomainError.BAD_FIELD, result.body.message));
+                        if (result.statusCode !== 200) return iteratorCallback(new DomainError(DomainError.EXTERNAL_ERROR, formatError(result)));

-                    return callback(null);
-                });
+                        recordIds.push(safe.query(result.body, 'domain_record.id'));
+
+                        return iteratorCallback(null);
+                    });
            }
        }, function (error) {
            if (error) return callback(error);

-            callback(null, 'unused');
+            callback(null, '' + recordIds[0]); // DO ids are integers
        });
    });
 }
@@ -166,18 +182,18 @@ function del(dnsConfig, zoneName, subdomain, type, values, callback) {
        // FIXME we only handle the first one currently

        superagent.del(DIGITALOCEAN_ENDPOINT + '/v2/domains/' + zoneName + '/records/' + tmp[0].id)
-          .set('Authorization', 'Bearer ' + dnsConfig.token)
-          .timeout(30 * 1000)
-          .end(function (error, result) {
-            if (error && !error.response) return callback(error);
-            if (result.statusCode === 404) return callback(null);
-            if (result.statusCode === 403 || result.statusCode === 401) return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, formatError(result)));
-            if (result.statusCode !== 204) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, formatError(result)));
+            .set('Authorization', 'Bearer ' + dnsConfig.token)
+            .timeout(30 * 1000)
+            .end(function (error, result) {
+                if (error && !error.response) return callback(new DomainError(DomainError.EXTERNAL_ERROR, util.format('Network error %s', error.message)));
+                if (result.statusCode === 404) return callback(null);
+                if (result.statusCode === 403 || result.statusCode === 401) return callback(new DomainError(DomainError.ACCESS_DENIED, formatError(result)));
+                if (result.statusCode !== 204) return callback(new DomainError(DomainError.EXTERNAL_ERROR, formatError(result)));

-            debug('del: done');
+                debug('del: done');

-            return callback(null);
-        });
+                return callback(null);
+            });
    });
 }

@@ -189,29 +205,34 @@ function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
    assert.strictEqual(typeof callback, 'function');

    var credentials = {
-        provider: dnsConfig.provider,
        token: dnsConfig.token
    };

    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here

    dns.resolveNs(zoneName, function (error, nameservers) {
-        if (error && error.code === 'ENOTFOUND') return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
-        if (error || !nameservers) return callback(new SubdomainError(SubdomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));
+        if (error && error.code === 'ENOTFOUND') return callback(new DomainError(DomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
+        if (error || !nameservers) return callback(new DomainError(DomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));

        if (nameservers.map(function (n) { return n.toLowerCase(); }).indexOf('ns1.digitalocean.com') === -1) {
            debug('verifyDnsConfig: %j does not contains DO NS', nameservers);
-            return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Domain nameservers are not set to Digital Ocean'));
+            return callback(new DomainError(DomainError.BAD_FIELD, 'Domain nameservers are not set to Digital Ocean'));
        }

-        const name = constants.ADMIN_LOCATION + (fqdn === zoneName ? '' :  '.' + fqdn.slice(0, - zoneName.length - 1));
+        const testSubdomain = 'cloudrontestdns';

-        upsert(credentials, zoneName, name, 'A', [ ip ], function (error, changeId) {
+        upsert(credentials, zoneName, testSubdomain, 'A', [ ip ], function (error, changeId) {
            if (error) return callback(error);

-            debug('verifyDnsConfig: A record added with change id %s', changeId);
+            debug('verifyDnsConfig: Test A record added with change id %s', changeId);

-            callback(null, credentials);
+            del(dnsConfig, zoneName, testSubdomain, 'A', [ ip ], function (error) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record removed again');
+
+                callback(null, credentials);
+            });
        });
    });
 }
@@ -0,0 +1,206 @@
+'use strict';
+
+exports = module.exports = {
+    upsert: upsert,
+    get: get,
+    del: del,
+    waitForDns: require('./waitfordns.js'),
+    verifyDnsConfig: verifyDnsConfig
+};
+
+var assert = require('assert'),
+    config = require('../config.js'),
+    debug = require('debug')('box:dns/gcdns'),
+    dns = require('dns'),
+    DomainError = require('../domains.js').DomainError,
+    GCDNS = require('@google-cloud/dns'),
+    util = require('util'),
+    _ = require('underscore');
+
+function getDnsCredentials(dnsConfig) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+
+    var config = {
+        projectId: dnsConfig.projectId,
+        keyFilename: dnsConfig.keyFilename,
+        email: dnsConfig.email
+    };
+
+    if (dnsConfig.credentials) {
+        config.credentials = {
+            client_email: dnsConfig.credentials.client_email,
+            private_key: dnsConfig.credentials.private_key
+        };
+    }
+    return config;
+}
+
+function getZoneByName(dnsConfig, zoneName, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var gcdns = GCDNS(getDnsCredentials(dnsConfig));
+
+    gcdns.getZones(function (error, zones) {
+        if (error && error.message === 'invalid_grant') return callback(new DomainError(DomainError.ACCESS_DENIED, 'The key was probably revoked'));
+        if (error && error.reason === 'No such domain') return callback(new DomainError(DomainError.NOT_FOUND, error.message));
+        if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+        if (error && error.code === 404) return callback(new DomainError(DomainError.NOT_FOUND, error.message));
+        if (error) {
+            debug('gcdns.getZones', error);
+            return callback(new DomainError(DomainError.EXTERNAL_ERROR, error));
+        }
+
+        var zone = zones.filter(function (zone) {
+            return zone.metadata.dnsName.slice(0, -1) === zoneName;     // the zone name contains a '.' at the end
+        })[0];
+
+        if (!zone) return callback(new DomainError(DomainError.NOT_FOUND, 'no such zone'));
+
+        callback(null, zone); //zone.metadata ~= {name="", dnsName="", nameServers:[]}
+    });
+}
+
+function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('add: %s for zone %s of type %s with values %j', subdomain, zoneName, type, values);
+
+    getZoneByName(getDnsCredentials(dnsConfig), zoneName, function (error, zone) {
+        if (error) return callback(error);
+
+        var domain = (subdomain ? subdomain + '.' : '') + zoneName + '.';
+
+        zone.getRecords({ type: type, name: domain }, function (error, oldRecords) {
+            if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error) {
+                debug('upsert->zone.getRecords', error);
+                return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
+            }
+
+            var newRecord = zone.record(type, {
+                name: domain,
+                data: values,
+                ttl: 1
+            });
+
+            zone.createChange({ delete: oldRecords, add: newRecord }, function(error, change) {
+                if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+                if (error && error.code === 412) return callback(new DomainError(DomainError.STILL_BUSY, error.message));
+                if (error) {
+                    debug('upsert->zone.createChange', error);
+                    return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
+                }
+
+                callback(null, change.id);
+            });
+        });
+    });
+}
+
+function get(dnsConfig, zoneName, subdomain, type, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    getZoneByName(getDnsCredentials(dnsConfig), zoneName, function (error, zone) {
+        if (error) return callback(error);
+
+        var params = {
+            name: (subdomain ? subdomain + '.' : '') + zoneName + '.',
+            type: type
+        };
+
+        zone.getRecords(params, function (error, records) {
+            if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, error));
+            if (records.length === 0) return callback(null, [ ]);
+
+            return callback(null, records[0].data);
+        });
+    });
+}
+
+function del(dnsConfig, zoneName, subdomain, type, values, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    getZoneByName(getDnsCredentials(dnsConfig), zoneName, function (error, zone) {
+        if (error) return callback(error);
+
+        var domain = (subdomain ? subdomain + '.' : '') + zoneName + '.';
+
+        zone.getRecords({ type: type, name: domain }, function(error, oldRecords) {
+            if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error) {
+                debug('del->zone.getRecords', error);
+                return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
+            }
+
+            zone.deleteRecords(oldRecords, function (error, change) {
+                if (error && error.code === 403) return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+                if (error && error.code === 412) return callback(new DomainError(DomainError.STILL_BUSY, error.message));
+                if (error) {
+                    debug('del->zone.createChange', error);
+                    return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
+                }
+
+                callback(null, change.id);
+            });
+        });
+    });
+}
+
+function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
+    assert.strictEqual(typeof dnsConfig, 'object');
+    assert.strictEqual(typeof fqdn, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof ip, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var credentials = getDnsCredentials(dnsConfig);
+    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here
+
+    dns.resolveNs(zoneName, function (error, resolvedNS) {
+        if (error && error.code === 'ENOTFOUND') return callback(new DomainError(DomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
+        if (error || !resolvedNS) return callback(new DomainError(DomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));
+
+        getZoneByName(credentials, zoneName, function (error, zone) {
+            if (error) return callback(error);
+
+            var definedNS = zone.metadata.nameServers.sort().map(function(r) { return r.replace(/\.$/, ''); });
+            if (!_.isEqual(definedNS, resolvedNS.sort())) {
+                debug('verifyDnsConfig: %j and %j do not match', resolvedNS, definedNS);
+                return callback(new DomainError(DomainError.BAD_FIELD, 'Domain nameservers are not set to Google Cloud DNS'));
+            }
+
+            const testSubdomain = 'cloudrontestdns';
+
+            upsert(credentials, zoneName, testSubdomain, 'A', [ ip ], function (error, changeId) {
+                if (error) return callback(error);
+
+                debug('verifyDnsConfig: Test A record added with change id %s', changeId);
+
+                del(dnsConfig, zoneName, testSubdomain, 'A', [ ip ], function (error) {
+                    if (error) return callback(error);
+
+                    debug('verifyDnsConfig: Test A record removed again');
+
+                    callback(null, credentials);
+                });
+            });
+        });
+    });
+}
@@ -15,7 +15,7 @@ exports = module.exports = {
 };

 var assert = require('assert'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
    util = require('util');

 function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {
@@ -9,12 +9,9 @@ exports = module.exports = {
 };

 var assert = require('assert'),
-    async = require('async'),
-    constants = require('../constants.js'),
    debug = require('debug')('box:dns/manual'),
-    dig = require('../dig.js'),
    dns = require('dns'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
    util = require('util');

 function upsert(dnsConfig, zoneName, subdomain, type, values, callback) {
@@ -58,50 +55,10 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, ip, callback) {
    assert.strictEqual(typeof ip, 'string');
    assert.strictEqual(typeof callback, 'function');

-    var adminDomain = constants.ADMIN_LOCATION + '.' + domain;
-
+    // Very basic check if the nameservers can be fetched
    dns.resolveNs(zoneName, function (error, nameservers) {
-        if (error || !nameservers) return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Unable to get nameservers'));
+        if (error || !nameservers) return callback(new DomainError(DomainError.BAD_FIELD, 'Unable to get nameservers'));

-        async.every(nameservers, function (nameserver, everyNsCallback) {
-            // ns records cannot have cname
-            dns.resolve4(nameserver, function (error, nsIps) {
-                if (error || !nsIps || nsIps.length === 0) {
-                    return everyNsCallback(new SubdomainError(SubdomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
-                }
-
-                async.every(nsIps, function (nsIp, everyIpCallback) {
-                    dig.resolve(adminDomain, 'A', { server: nsIp, timeout: 5000 }, function (error, answer) {
-                        if (error && error.code === 'ETIMEDOUT') {
-                            debug('nameserver %s (%s) timed out when trying to resolve %s', nameserver, nsIp, adminDomain);
-                            return everyIpCallback(null, true); // should be ok if dns server is down
-                        }
-
-                        if (error) {
-                            debug('nameserver %s (%s) returned error trying to resolve %s: %s', nameserver, nsIp, adminDomain, error);
-                            return everyIpCallback(null, false);
-                        }
-
-                        if (!answer || answer.length === 0) {
-                            debug('bad answer from nameserver %s (%s) resolving %s (%s): %j', nameserver, nsIp, adminDomain, 'A', answer);
-                            return everyIpCallback(null, false);
-                        }
-
-                        debug('verifyDnsConfig: ns: %s (%s), name:%s Actual:%j Expecting:%s', nameserver, nsIp, adminDomain, answer, ip);
-
-                        var match = answer.some(function (a) { return a === ip; });
-
-                        if (match) return everyIpCallback(null, true); // done!
-
-                        everyIpCallback(null, false);
-                    });
-                }, everyNsCallback);
-            });
-        }, function (error, success) {
-            if (error) return callback(error);
-            if (!success) return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'The domain ' + adminDomain + ' does not resolve to the server\'s IP ' + ip));
-
-            callback(null, { provider: dnsConfig.provider, wildcard: !!dnsConfig.wildcard });
-        });
+        callback(null, { wildcard: !!dnsConfig.wildcard });
    });
 }
@@ -64,9 +64,5 @@ function verifyDnsConfig(dnsConfig, domain, zoneName, ip, callback) {
    assert.strictEqual(typeof ip, 'string');
    assert.strictEqual(typeof callback, 'function');

-    var credentials = {
-        provider: dnsConfig.provider
-    };
-
-    return callback(null, credentials);
+    return callback(null, { });
 }
@@ -13,10 +13,10 @@ exports = module.exports = {

 var assert = require('assert'),
    AWS = require('aws-sdk'),
-    constants = require('../constants.js'),
+    config = require('../config.js'),
    debug = require('debug')('box:dns/route53'),
    dns = require('dns'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
    util = require('util'),
    _ = require('underscore');

@@ -41,15 +41,15 @@ function getZoneByName(dnsConfig, zoneName, callback) {

    var route53 = new AWS.Route53(getDnsCredentials(dnsConfig));
    route53.listHostedZones({}, function (error, result) {
-        if (error && error.code === 'AccessDenied') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-        if (error && error.code === 'InvalidClientTokenId') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-        if (error) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, error.message));
+        if (error && error.code === 'AccessDenied') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+        if (error && error.code === 'InvalidClientTokenId') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+        if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));

        var zone = result.HostedZones.filter(function (zone) {
            return zone.Name.slice(0, -1) === zoneName;     // aws zone name contains a '.' at the end
        })[0];

-        if (!zone) return callback(new SubdomainError(SubdomainError.NOT_FOUND, 'no such zone'));
+        if (!zone) return callback(new DomainError(DomainError.NOT_FOUND, 'no such zone'));

        callback(null, zone);
    });
@@ -65,9 +65,9 @@ function getHostedZone(dnsConfig, zoneName, callback) {

        var route53 = new AWS.Route53(getDnsCredentials(dnsConfig));
        route53.getHostedZone({ Id: zone.Id }, function (error, result) {
-            if (error && error.code === 'AccessDenied') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error && error.code === 'InvalidClientTokenId') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, error.message));
+            if (error && error.code === 'AccessDenied') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'InvalidClientTokenId') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));

            callback(null, result);
        });
@@ -107,11 +107,11 @@ function add(dnsConfig, zoneName, subdomain, type, values, callback) {

        var route53 = new AWS.Route53(getDnsCredentials(dnsConfig));
        route53.changeResourceRecordSets(params, function(error, result) {
-            if (error && error.code === 'AccessDenied') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error && error.code === 'InvalidClientTokenId') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error && error.code === 'PriorRequestNotComplete') return callback(new SubdomainError(SubdomainError.STILL_BUSY, error.message));
-            if (error && error.code === 'InvalidChangeBatch') return callback(new SubdomainError(SubdomainError.BAD_FIELD, error.message));
-            if (error) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, error.message));
+            if (error && error.code === 'AccessDenied') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'InvalidClientTokenId') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'PriorRequestNotComplete') return callback(new DomainError(DomainError.STILL_BUSY, error.message));
+            if (error && error.code === 'InvalidChangeBatch') return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+            if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));

            callback(null, result.ChangeInfo.Id);
        });
@@ -148,9 +148,9 @@ function get(dnsConfig, zoneName, subdomain, type, callback) {

        var route53 = new AWS.Route53(getDnsCredentials(dnsConfig));
        route53.listResourceRecordSets(params, function (error, result) {
-            if (error && error.code === 'AccessDenied') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error && error.code === 'InvalidClientTokenId') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error) return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, error.message));
+            if (error && error.code === 'AccessDenied') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'InvalidClientTokenId') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
            if (result.ResourceRecordSets.length === 0) return callback(null, [ ]);
            if (result.ResourceRecordSets[0].Name !== params.StartRecordName || result.ResourceRecordSets[0].Type !== params.StartRecordType) return callback(null, [ ]);

@@ -194,23 +194,23 @@ function del(dnsConfig, zoneName, subdomain, type, values, callback) {

        var route53 = new AWS.Route53(getDnsCredentials(dnsConfig));
        route53.changeResourceRecordSets(params, function(error, result) {
-            if (error && error.code === 'AccessDenied') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
-            if (error && error.code === 'InvalidClientTokenId') return callback(new SubdomainError(SubdomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'AccessDenied') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
+            if (error && error.code === 'InvalidClientTokenId') return callback(new DomainError(DomainError.ACCESS_DENIED, error.message));
            if (error && error.message && error.message.indexOf('it was not found') !== -1) {
                debug('del: resource record set not found.', error);
-                return callback(new SubdomainError(SubdomainError.NOT_FOUND, error.message));
+                return callback(new DomainError(DomainError.NOT_FOUND, error.message));
            } else if (error && error.code === 'NoSuchHostedZone') {
                debug('del: hosted zone not found.', error);
-                return callback(new SubdomainError(SubdomainError.NOT_FOUND, error.message));
+                return callback(new DomainError(DomainError.NOT_FOUND, error.message));
            } else if (error && error.code === 'PriorRequestNotComplete') {
                debug('del: resource is still busy', error);
-                return callback(new SubdomainError(SubdomainError.STILL_BUSY, error.message));
+                return callback(new DomainError(DomainError.STILL_BUSY, error.message));
            } else if (error && error.code === 'InvalidChangeBatch') {
                debug('del: invalid change batch. No such record to be deleted.');
-                return callback(new SubdomainError(SubdomainError.NOT_FOUND, error.message));
+                return callback(new DomainError(DomainError.NOT_FOUND, error.message));
            } else if (error) {
                debug('del: error', error);
-                return callback(new SubdomainError(SubdomainError.EXTERNAL_ERROR, error.message));
+                return callback(new DomainError(DomainError.EXTERNAL_ERROR, error.message));
            }

            callback(null);
@@ -226,7 +226,6 @@ function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
    assert.strictEqual(typeof callback, 'function');

    var credentials = {
-        provider: dnsConfig.provider,
        accessKeyId: dnsConfig.accessKeyId,
        secretAccessKey: dnsConfig.secretAccessKey,
        region: dnsConfig.region || 'us-east-1',
@@ -236,25 +235,31 @@ function verifyDnsConfig(dnsConfig, fqdn, zoneName, ip, callback) {
    if (process.env.BOX_ENV === 'test') return callback(null, credentials); // this shouldn't be here

    dns.resolveNs(zoneName, function (error, nameservers) {
-        if (error && error.code === 'ENOTFOUND') return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
-        if (error || !nameservers) return callback(new SubdomainError(SubdomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));
+        if (error && error.code === 'ENOTFOUND') return callback(new DomainError(DomainError.BAD_FIELD, 'Unable to resolve nameservers for this domain'));
+        if (error || !nameservers) return callback(new DomainError(DomainError.BAD_FIELD, error ? error.message : 'Unable to get nameservers'));

        getHostedZone(credentials, zoneName, function (error, zone) {
            if (error) return callback(error);

            if (!_.isEqual(zone.DelegationSet.NameServers.sort(), nameservers.sort())) {
                debug('verifyDnsConfig: %j and %j do not match', nameservers, zone.DelegationSet.NameServers);
-                return callback(new SubdomainError(SubdomainError.BAD_FIELD, 'Domain nameservers are not set to Route53'));
+                return callback(new DomainError(DomainError.BAD_FIELD, 'Domain nameservers are not set to Route53'));
            }

-            const name = constants.ADMIN_LOCATION + (fqdn === zoneName ? '' :  '.' + fqdn.slice(0, - zoneName.length - 1));
+            const testSubdomain = 'cloudrontestdns';

-            upsert(credentials, zoneName, name, 'A', [ ip ], function (error, changeId) {
+            upsert(credentials, zoneName, testSubdomain, 'A', [ ip ], function (error, changeId) {
                if (error) return callback(error);

-                debug('verifyDnsConfig: A record added with change id %s', changeId);
+                debug('verifyDnsConfig: Test A record added with change id %s', changeId);

-                callback(null, credentials);
+                del(dnsConfig, zoneName, testSubdomain, 'A', [ ip ], function (error) {
+                    if (error) return callback(error);
+
+                    debug('verifyDnsConfig: Test A record removed again');
+
+                    callback(null, credentials);
+                });
            });
        });
    });
@@ -7,7 +7,7 @@ var assert = require('assert'),
    debug = require('debug')('box:dns/waitfordns'),
    dig = require('../dig.js'),
    dns = require('dns'),
-    SubdomainError = require('../subdomains.js').SubdomainError,
+    DomainError = require('../domains.js').DomainError,
    util = require('util');

 function isChangeSynced(domain, value, type, nameserver, callback) {
@@ -56,7 +56,7 @@ function isChangeSynced(domain, value, type, nameserver, callback) {
        }, callback);

    });
- }
+}

 // check if IP change has propagated to every nameserver
 function waitForDns(domain, zoneName, value, type, options, callback) {
@@ -76,22 +76,22 @@ function waitForDns(domain, zoneName, value, type, options, callback) {

    var attempt = 1;
    async.retry(options, function (retryCallback) {
-        debug('waitForDNS: %s attempt %s.', domain, attempt++);
+        debug('waitForDNS: %s (zone: %s) attempt %s.', domain, zoneName, attempt++);

        dns.resolveNs(zoneName, function (error, nameservers) {
-            if (error || !nameservers) return retryCallback(error || new SubdomainError(SubdomainError.EXTERNAL_ERROR, 'Unable to get nameservers'));
+            if (error || !nameservers) return retryCallback(error || new DomainError(DomainError.EXTERNAL_ERROR, 'Unable to get nameservers'));

            async.every(nameservers, isChangeSynced.bind(null, domain, value, type), function (error, synced) {
                debug('waitForIp: %s %s ns: %j', domain, synced ? 'done' : 'not done', nameservers);

-                retryCallback(synced ? null : new SubdomainError(SubdomainError.EXTERNAL_ERROR, 'ETRYAGAIN'));
+                retryCallback(synced ? null : new DomainError(DomainError.EXTERNAL_ERROR, 'ETRYAGAIN'));
            });
        });
    }, function retryDone(error) {
-         if (error) return callback(error);
+        if (error) return callback(error);

        debug('waitForDNS: %s done.', domain);

        callback(null);
-     });
+    });
 }
@@ -51,7 +51,7 @@ var addons = require('./addons.js'),
 function debugApp(app, args) {
    assert(!app || typeof app === 'object');

-    var prefix = app ? (app.location || '(bare)') : '(no app)';
+    var prefix = app ? app.intrinsicFqdn : '(no app)';
    debug(prefix + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)));
 }

@@ -129,7 +129,7 @@ function createSubcontainer(app, name, cmd, options, callback) {

    var manifest = app.manifest;
    var exposedPorts = {}, dockerPortBindings = { };
-    var domain = app.altDomain || config.appFqdn(app.location);
+    var domain = app.altDomain || app.intrinsicFqdn;
    var stdEnv = [
        'CLOUDRON=1',
        'WEBADMIN_ORIGIN=' + config.adminOrigin(),
@@ -186,9 +186,9 @@ function createSubcontainer(app, name, cmd, options, callback) {
                '/run': {}
            },
            Labels: {
-                "location": app.location,
-                "appId": app.id,
-                "isSubcontainer": String(!isAppContainer)
+                'fqdn': app.intrinsicFqdn,
+                'appId': app.id,
+                'isSubcontainer': String(!isAppContainer)
            },
            HostConfig: {
                Binds: addons.getBindsSync(app, app.manifest.addons),
@@ -198,20 +198,28 @@ function createSubcontainer(app, name, cmd, options, callback) {
                PublishAllPorts: false,
                ReadonlyRootfs: app.debugMode ? !!app.debugMode.readonlyRootfs : true,
                RestartPolicy: {
-                    "Name": isAppContainer ? "always" : "no",
-                    "MaximumRetryCount": 0
+                    'Name': isAppContainer ? 'always' : 'no',
+                    'MaximumRetryCount': 0
                },
                CpuShares: 512, // relative to 1024 for system processes
-                VolumesFrom: isAppContainer ? null : [ app.containerId + ":rw" ],
+                VolumesFrom: isAppContainer ? null : [ app.containerId + ':rw' ],
                NetworkMode: 'cloudron',
                Dns: ['172.18.0.1'], // use internal dns
                DnsSearch: ['.'], // use internal dns
-                SecurityOpt: enableSecurityOpt ? [ "apparmor=docker-cloudron-app" ] : null // profile available only on cloudron
+                SecurityOpt: enableSecurityOpt ? [ 'apparmor=docker-cloudron-app' ] : null // profile available only on cloudron
            }
        };
+
+        var capabilities = manifest.capabilities || [];
+        if (capabilities.includes('net_admin')) {
+            containerOptions.HostConfig.CapAdd = [
+                'NET_ADMIN'
+            ];
+        }
+
        containerOptions = _.extend(containerOptions, options);

-        debugApp(app, 'Creating container for %s with options %j', app.manifest.dockerImage, containerOptions);
+        debugApp(app, 'Creating container for %s', app.manifest.dockerImage);

        docker.createContainer(containerOptions, callback);
    });
@@ -359,31 +367,21 @@ function getContainerIdByIp(ip, callback) {
    assert.strictEqual(typeof ip, 'string');
    assert.strictEqual(typeof callback, 'function');

-    debug('get container by ip %s', ip);
-
    var docker = exports.connection;

-    docker.listNetworks({}, function (error, result) {
+    docker.getNetwork('cloudron').inspect(function (error, bridge) {
+        if (error && error.statusCode === 404) return callback(new Error('Unable to find the cloudron network'));
        if (error) return callback(error);

-        var bridge;
-        result.forEach(function (n) {
-            if (n.Name === 'cloudron') bridge = n;
-        });
-
-        if (!bridge) return callback(new Error('Unable to find the cloudron network'));
-
        var containerId;
        for (var id in bridge.Containers) {
-            if (bridge.Containers[id].IPv4Address.indexOf(ip) === 0) {
+            if (bridge.Containers[id].IPv4Address.indexOf(ip + '/16') === 0) {
                containerId = id;
                break;
            }
        }
        if (!containerId) return callback(new Error('No container with that ip'));

-        debug('found container %s with ip %s', containerId, ip);
-
        callback(null, containerId);
    });
 }
@@ -0,0 +1,125 @@
+/* jslint node:true */
+
+'use strict';
+
+exports = module.exports = {
+    add: add,
+    get: get,
+    getAll: getAll,
+    update: update,
+    upsert: upsert,
+    del: del,
+
+    _clear: clear,
+    _addDefaultDomain: addDefaultDomain
+};
+
+var assert = require('assert'),
+    database = require('./database.js'),
+    DatabaseError = require('./databaseerror'),
+    config = require('./config.js'),
+    safe = require('safetydance');
+
+function postProcess(data) {
+    data.config = safe.JSON.parse(data.configJson);
+    delete data.configJson;
+
+    return data;
+}
+
+function get(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT * FROM domains WHERE domain=?', [ domain ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        postProcess(result[0]);
+
+        callback(null, result[0]);
+    });
+}
+
+function getAll(callback) {
+    database.query('SELECT * FROM domains ORDER BY domain', function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        results.forEach(postProcess);
+
+        callback(null, results);
+    });
+}
+
+function add(domain, zoneName, provider, config, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof config, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('INSERT INTO domains (domain, zoneName, provider, configJson) VALUES (?, ?, ?, ?)', [ domain, zoneName, provider, JSON.stringify(config) ], function (error) {
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function upsert(domain, zoneName, provider, config, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof config, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('REPLACE INTO domains (domain, zoneName, provider, configJson) VALUES (?, ?, ?, ?)', [ domain, zoneName, provider, JSON.stringify(config) ], function (error) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function update(domain, provider, config, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof config, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('UPDATE domains SET provider=?, configJson=? WHERE domain=?', [ provider, JSON.stringify(config), domain ], function (error) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function del(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM domains WHERE domain=?', [ domain ], function (error, result) {
+        if (error && error.code === 'ER_ROW_IS_REFERENCED_2') return callback(new DatabaseError(DatabaseError.IN_USE));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
+function clear(callback) {
+    database.query('DELETE FROM domains', function (error) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(error);
+    });
+}
+
+function addDefaultDomain(callback) {
+    assert(config.fqdn(), 'no fqdn set in config, cannot continue');
+
+    add(config.fqdn(), config.zoneName(), 'manual', { }, function (error) {
+        if (error && error.reason !== DatabaseError.ALREADY_EXISTS) return callback(error);
+        callback();
+    });
+}
@@ -0,0 +1,335 @@
+'use strict';
+
+module.exports = exports = {
+    add: add,
+    get: get,
+    getAll: getAll,
+    update: update,
+    del: del,
+
+    fqdn: fqdn,
+    setAdmin: setAdmin,
+
+    getDNSRecords: getDNSRecords,
+    upsertDNSRecords: upsertDNSRecords,
+    removeDNSRecords: removeDNSRecords,
+
+    waitForDNSRecord: waitForDNSRecord,
+
+    DomainError: DomainError
+};
+
+var assert = require('assert'),
+    caas = require('./caas.js'),
+    config = require('./config.js'),
+    certificates = require('./certificates.js'),
+    CertificatesError = certificates.CertificatesError,
+    DatabaseError = require('./databaseerror.js'),
+    debug = require('debug')('box:domains'),
+    domaindb = require('./domaindb.js'),
+    path = require('path'),
+    shell = require('./shell.js'),
+    sysinfo = require('./sysinfo.js'),
+    tld = require('tldjs'),
+    util = require('util');
+
+var RESTART_CMD = path.join(__dirname, 'scripts/restart.sh');
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+function DomainError(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(DomainError, Error);
+
+DomainError.NOT_FOUND = 'No such domain';
+DomainError.ALREADY_EXISTS = 'Domain already exists';
+DomainError.EXTERNAL_ERROR = 'External error';
+DomainError.BAD_FIELD = 'Bad Field';
+DomainError.STILL_BUSY = 'Still busy';
+DomainError.IN_USE = 'In Use';
+DomainError.INTERNAL_ERROR = 'Internal error';
+DomainError.ACCESS_DENIED = 'Access denied';
+DomainError.INVALID_PROVIDER = 'provider must be route53, gcdns, digitalocean, cloudflare, noop, manual or caas';
+
+// choose which subdomain backend we use for test purpose we use route53
+function api(provider) {
+    assert.strictEqual(typeof provider, 'string');
+
+    switch (provider) {
+    case 'caas': return require('./dns/caas.js');
+    case 'cloudflare': return require('./dns/cloudflare.js');
+    case 'route53': return require('./dns/route53.js');
+    case 'gcdns': return require('./dns/gcdns.js');
+    case 'digitalocean': return require('./dns/digitalocean.js');
+    case 'noop': return require('./dns/noop.js');
+    case 'manual': return require('./dns/manual.js');
+    default: return null;
+    }
+}
+
+// TODO make it return a DomainError instead of DomainError
+function verifyDnsConfig(config, domain, zoneName, provider, ip, callback) {
+    assert(config && typeof config === 'object'); // the dns config to test with
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof ip, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var backend = api(provider);
+    if (!backend) return callback(new DomainError(DomainError.INVALID_PROVIDER));
+
+    api(provider).verifyDnsConfig(config, domain, zoneName, ip, callback);
+}
+
+
+function add(domain, zoneName, provider, config, fallbackCertificate, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof zoneName, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof config, 'object');
+    assert.strictEqual(typeof fallbackCertificate, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (!tld.isValid(domain)) return callback(new DomainError(DomainError.BAD_FIELD, 'Invalid domain'));
+    if (!tld.isValid(zoneName)) return callback(new DomainError(DomainError.BAD_FIELD, 'Invalid zoneName'));
+
+    if (fallbackCertificate) {
+        let error = certificates.validateCertificate(fallbackCertificate.cert, fallbackCertificate.key, domain);
+        if (error) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+    }
+
+    sysinfo.getPublicIp(function (error, ip) {
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, 'Error getting IP:' + error.message));
+
+        verifyDnsConfig(config, domain, zoneName, provider, ip, function (error, result) {
+            if (error && error.reason === DomainError.ACCESS_DENIED) return callback(new DomainError(DomainError.BAD_FIELD, 'Error adding A record. Access denied'));
+            if (error && error.reason === DomainError.NOT_FOUND) return callback(new DomainError(DomainError.BAD_FIELD, 'Zone not found'));
+            if (error && error.reason === DomainError.EXTERNAL_ERROR) return callback(new DomainError(DomainError.BAD_FIELD, 'Error adding A record:' + error.message));
+            if (error && error.reason === DomainError.BAD_FIELD) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+            if (error && error.reason === DomainError.INVALID_PROVIDER) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+            if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+            domaindb.add(domain, zoneName, provider, result, function (error) {
+                if (error && error.reason === DatabaseError.ALREADY_EXISTS) return callback(new DomainError(DomainError.ALREADY_EXISTS));
+                if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+                if (!fallbackCertificate) return callback();
+
+                // cert validation already happened above no need to check all errors again
+                certificates.setFallbackCertificate(fallbackCertificate.cert, fallbackCertificate.key, domain, function (error) {
+                    if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+                    callback();
+                });
+            });
+        });
+    });
+}
+
+function get(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    domaindb.get(domain, function (error, result) {
+        // TODO try to find subdomain entries maybe based on zoneNames or so
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainError(DomainError.NOT_FOUND));
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        certificates.getFallbackCertificate(domain, function (error, fallbackCertificate) {
+            if (error && error.reason !== CertificatesError.NOT_FOUND) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+            if (fallbackCertificate) result.fallbackCertificate = fallbackCertificate;
+
+            return callback(null, result);
+        });
+    });
+}
+
+function getAll(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    domaindb.getAll(function (error, result) {
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        return callback(null, result);
+    });
+}
+
+function update(domain, provider, config, fallbackCertificate, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof provider, 'string');
+    assert.strictEqual(typeof config, 'object');
+    assert.strictEqual(typeof fallbackCertificate, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    domaindb.get(domain, function (error, result) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainError(DomainError.NOT_FOUND));
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        if (fallbackCertificate) {
+            let error = certificates.validateCertificate(fallbackCertificate.cert, fallbackCertificate.key, domain);
+            if (error) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+        }
+
+        sysinfo.getPublicIp(function (error, ip) {
+            if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, 'Error getting IP:' + error.message));
+
+            verifyDnsConfig(config, domain, result.zoneName, provider, ip, function (error, result) {
+                if (error && error.reason === DomainError.ACCESS_DENIED) return callback(new DomainError(DomainError.BAD_FIELD, 'Error adding A record. Access denied'));
+                if (error && error.reason === DomainError.NOT_FOUND) return callback(new DomainError(DomainError.BAD_FIELD, 'Zone not found'));
+                if (error && error.reason === DomainError.EXTERNAL_ERROR) return callback(new DomainError(DomainError.BAD_FIELD, 'Error adding A record:' + error.message));
+                if (error && error.reason === DomainError.BAD_FIELD) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+                if (error && error.reason === DomainError.INVALID_PROVIDER) return callback(new DomainError(DomainError.BAD_FIELD, error.message));
+                if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+                domaindb.update(domain, provider, result, function (error) {
+                    if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainError(DomainError.NOT_FOUND));
+                    if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+                    if (!fallbackCertificate) return callback();
+
+                    // cert validation already happened above no need to check all errors again
+                    certificates.setFallbackCertificate(fallbackCertificate.cert, fallbackCertificate.key, domain, function (error) {
+                        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+                        callback();
+                    });
+                });
+            });
+        });
+    });
+}
+
+function del(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    domaindb.del(domain, function (error) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DomainError(DomainError.NOT_FOUND));
+        if (error && error.reason === DatabaseError.IN_USE) return callback(new DomainError(DomainError.IN_USE));
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        return callback(null);
+    });
+}
+
+function getDNSRecords(subdomain, domain, type, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    get(domain, function (error, result) {
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        api(result.provider).get(result.config, result.zoneName, subdomain, type, function (error, values) {
+            if (error) return callback(error);
+
+            callback(null, values);
+        });
+    });
+}
+
+function upsertDNSRecords(subdomain, domain, type, values, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('upsertDNSRecord: %s on %s type %s values', subdomain, domain, type, values);
+
+    get(domain, function (error, result) {
+        if (error) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        api(result.provider).upsert(result.config, result.zoneName, subdomain, type, values, function (error, changeId) {
+            if (error) return callback(error);
+
+            callback(null, changeId);
+        });
+    });
+}
+
+function removeDNSRecords(subdomain, domain, type, values, callback) {
+    assert.strictEqual(typeof subdomain, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert(util.isArray(values));
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('removeDNSRecord: %s on %s type %s values', subdomain, domain, type, values);
+
+    get(domain, function (error, result) {
+        if (error) return callback(error);
+
+        api(result.provider).del(result.config, result.zoneName, subdomain, type, values, function (error) {
+            if (error && error.reason !== DomainError.NOT_FOUND) return callback(error);
+
+            callback(null);
+        });
+    });
+}
+
+function waitForDNSRecord(fqdn, domain, value, type, options, callback) {
+    assert.strictEqual(typeof fqdn, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert(typeof value === 'string' || util.isRegExp(value));
+    assert(type === 'A' || type === 'CNAME' || type === 'TXT');
+    assert(options && typeof options === 'object'); // { interval: 5000, times: 50000 }
+    assert.strictEqual(typeof callback, 'function');
+
+    get(domain, function (error, result) {
+        // domain can be not found when waiting for altDomain. When we migrate altDomain, this can never happen
+        if (error && error.reason !== DomainError.NOT_FOUND) return callback(new DomainError(DomainError.INTERNAL_ERROR, error));
+
+        // hack for lack of provider with altDomain. When we migrate altDomain, this will be automatically "manual"
+        const provider = result ? result.provider : 'manual';
+
+        api(provider).waitForDns(fqdn, result ? result.zoneName : domain, value, type, options, callback);
+    });
+}
+
+function setAdmin(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('setAdmin domain:%s', domain);
+
+    get(domain, function (error, result) {
+        if (error) return callback(error);
+
+        var setPtrRecord = config.provider() === 'caas' ? caas.setPtrRecord : function (d, next) { next(); };
+
+        setPtrRecord(domain, function (error) {
+            if (error) return callback(new DomainError(DomainError.EXTERNAL_ERROR, 'Error setting PTR record:' + error.message));
+
+            config.setFqdn(result.domain);
+            config.setAdminLocation('my');
+            config.setAdminFqdn('my' + (result.provider === 'caas' ? '-' : '.') + result.domain);
+            config.setZoneName(result.zoneName);
+
+            callback();
+
+            shell.sudo('restart', [ RESTART_CMD ], NOOP_CALLBACK);
+        });
+    });
+}
+
+function fqdn(location, domain, provider) {
+    return location + (location ? (provider !== 'caas' ? '.' : '-') : '') + domain;
+}
+
@@ -3,6 +3,7 @@
 exports = module.exports = {
    verifyRelay: verifyRelay,
    getStatus: getStatus,
+    checkRblStatus: checkRblStatus,

    EmailError: EmailError
 };
@@ -11,7 +12,6 @@ var assert = require('assert'),
    async = require('async'),
    cloudron = require('./cloudron.js'),
    config = require('./config.js'),
-    constants = require('./constants.js'),
    debug = require('debug')('box:email'),
    dig = require('./dig.js'),
    net = require('net'),
@@ -109,7 +109,7 @@ function checkSmtpRelay(relay, callback) {
            return callback(error, result);
        }

-       callback(null, result);
+        callback(null, result);
    });
 }

@@ -128,7 +128,7 @@ function verifyRelay(relay, callback) {

 function checkDkim(callback) {
    var dkim = {
-        domain: constants.DKIM_SELECTOR + '._domainkey.' + config.fqdn(),
+        domain: config.dkimSelector() + '._domainkey.' + config.fqdn(),
        type: 'TXT',
        expected: null,
        value: null,
@@ -259,6 +259,100 @@ function checkPtr(callback) {
    });
 }

+// https://raw.githubusercontent.com/jawsome/node-dnsbl/master/list.json
+const RBL_LIST = [
+    {
+        'name': 'Barracuda',
+        'dns': 'b.barracudacentral.org',
+        'site': 'http://www.barracudacentral.org/rbl/removal-request'
+    },
+    {
+        'name': 'SpamCop',
+        'dns': 'bl.spamcop.net',
+        'site': 'http://spamcop.net'
+    },
+    {
+        'name': 'Sorbs Aggregate Zone',
+        'dns': 'dnsbl.sorbs.net',
+        'site': 'http://dnsbl.sorbs.net/'
+    },
+    {
+        'name': 'Sorbs spam.dnsbl Zone',
+        'dns': 'spam.dnsbl.sorbs.net',
+        'site': 'http://sorbs.net'
+    },
+    {
+        'name': 'Composite Blocking List',
+        'dns': 'cbl.abuseat.org',
+        'site': 'http://www.abuseat.org'
+    },
+    {
+        'name': 'SpamHaus Zen',
+        'dns': 'zen.spamhaus.org',
+        'site': 'http://spamhaus.org'
+    },
+    {
+        'name': 'Multi SURBL',
+        'dns': 'multi.surbl.org',
+        'site': 'http://www.surbl.org'
+    },
+    {
+        'name': 'Spam Cannibal',
+        'dns': 'bl.spamcannibal.org',
+        'site': 'http://www.spamcannibal.org/cannibal.cgi'
+    },
+    {
+        'name': 'dnsbl.abuse.ch',
+        'dns': 'spam.abuse.ch',
+        'site': 'http://dnsbl.abuse.ch/'
+    },
+    {
+        'name': 'The Unsubscribe Blacklist(UBL)',
+        'dns': 'ubl.unsubscore.com ',
+        'site': 'http://www.lashback.com/blacklist/'
+    },
+    {
+        'name': 'UCEPROTECT Network',
+        'dns': 'dnsbl-1.uceprotect.net',
+        'site': 'http://www.uceprotect.net/en'
+    }
+];
+
+function checkRblStatus(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    sysinfo.getPublicIp(function (error, ip) {
+        if (error) return callback(error, ip);
+
+        var flippedIp = ip.split('.').reverse().join('.');
+
+        // https://tools.ietf.org/html/rfc5782
+        async.map(RBL_LIST, function (rblServer, iteratorDone) {
+            dig.resolve(flippedIp + '.' + rblServer.dns, 'A', digOptions, function (error, records) {
+                if (error || !records) return iteratorDone(null, null);    // not listed
+
+                debug('checkRblStatus: %s (ip: %s) is in the blacklist of %j', config.fqdn(), flippedIp, rblServer);
+
+                var result = _.extend({ }, rblServer);
+
+                dig.resolve(flippedIp + '.' + rblServer.dns, 'TXT', digOptions, function (error, txtRecords) {
+                    result.txtRecords = error || !txtRecords ? 'No txt record' : txtRecords;
+
+                    debug('checkRblStatus: %s (error: %s) (txtRecords: %j)', config.fqdn(), error, txtRecords);
+
+                    return iteratorDone(null, result);
+                });
+            });
+        }, function (ignoredError, blacklistedServers) {
+            blacklistedServers = blacklistedServers.filter(function(b) { return b !== null; });
+
+            debug('checkRblStatus: %s (ip: %s) servers: %j', config.fqdn(), ip, blacklistedServers);
+
+            return callback(null, { status: blacklistedServers.length === 0, ip: ip, servers: blacklistedServers });
+        });
+    });
+}
+
 function getStatus(callback) {
    assert.strictEqual(typeof callback, 'function');

@@ -290,7 +384,8 @@ function getStatus(callback) {
                recordResult('dns.spf', checkSpf),
                recordResult('dns.dkim', checkDkim),
                recordResult('dns.ptr', checkPtr),
-                recordResult('relay', checkOutboundPort25)
+                recordResult('relay', checkOutboundPort25),
+                recordResult('rbl', checkRblStatus)
            );
        } else {
            checks.push(recordResult('relay', checkSmtpRelay.bind(null, relay)));
@@ -6,7 +6,7 @@ exports = module.exports = {
    add: add,
    get: get,
    getAllPaged: getAllPaged,
-    getByActionLastWeek: getByActionLastWeek,
+    getByCreationTime: getByCreationTime,
    cleanup: cleanup,

    // keep in sync with webadmin index.js filter and CLI tool
@@ -20,6 +20,7 @@ exports = module.exports = {
    ACTION_APP_LOGIN: 'app.login',
    ACTION_BACKUP_FINISH: 'backup.finish',
    ACTION_BACKUP_START: 'backup.start',
+    ACTION_BACKUP_CLEANUP: 'backup.cleanup',
    ACTION_CERTIFICATE_RENEWAL: 'certificate.renew',
    ACTION_CLI_MODE: 'settings.climode',
    ACTION_START: 'cloudron.start',
@@ -35,7 +36,7 @@ var assert = require('assert'),
    debug = require('debug')('box:eventlog'),
    eventlogdb = require('./eventlogdb.js'),
    util = require('util'),
-    uuid = require('node-uuid');
+    uuid = require('uuid');

 var NOOP_CALLBACK = function (error) { if (error) debug(error); };

@@ -97,21 +98,21 @@ function getAllPaged(action, search, page, perPage, callback) {
    assert.strictEqual(typeof perPage, 'number');
    assert.strictEqual(typeof callback, 'function');

-    eventlogdb.getAllPaged(action, search, page, perPage, function (error, boxes) {
+    eventlogdb.getAllPaged(action, search, page, perPage, function (error, events) {
        if (error) return callback(new EventLogError(EventLogError.INTERNAL_ERROR, error));

-        callback(null, boxes);
+        callback(null, events);
    });
 }

-function getByActionLastWeek(action, callback) {
-    assert(typeof action === 'string' || action === null);
+function getByCreationTime(creationTime, callback) {
+    assert(util.isDate(creationTime));
    assert.strictEqual(typeof callback, 'function');

-    eventlogdb.getByActionLastWeek(action, function (error, boxes) {
+    eventlogdb.getByCreationTime(creationTime, function (error, events) {
        if (error) return callback(new EventLogError(EventLogError.INTERNAL_ERROR, error));

-        callback(null, boxes);
+        callback(null, events);
    });
 }

@@ -119,7 +120,7 @@ function cleanup(callback) {
    callback = callback || NOOP_CALLBACK;

    var d = new Date();
-    d.setDate(d.getDate() - 7); // 7 days ago
+    d.setDate(d.getDate() - 10); // 10 days ago

    // only cleanup high frequency events
    var actions = [
@@ -3,7 +3,7 @@
 exports = module.exports = {
    get: get,
    getAllPaged: getAllPaged,
-    getByActionLastWeek: getByActionLastWeek,
+    getByCreationTime: getByCreationTime,
    add: add,
    count: count,
    delByCreationTime: delByCreationTime,
@@ -73,12 +73,12 @@ function getAllPaged(action, search, page, perPage, callback) {
    });
 }

-function getByActionLastWeek(action, callback) {
-    assert(typeof action === 'string' || action === null);
+function getByCreationTime(creationTime, callback) {
+    assert(util.isDate(creationTime));
    assert.strictEqual(typeof callback, 'function');

-    var query = 'SELECT ' + EVENTLOGS_FIELDS + ' FROM eventlog WHERE action=? AND creationTime >= DATE_SUB(NOW(), INTERVAL 1 WEEK) ORDER BY creationTime DESC';
-    database.query(query, [ action ], function (error, results) {
+    var query = 'SELECT ' + EVENTLOGS_FIELDS + ' FROM eventlog WHERE creationTime >= ? ORDER BY creationTime DESC';
+    database.query(query, [ creationTime ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        results.forEach(postProcess);
@@ -24,6 +24,7 @@ exports = module.exports = {

 var assert = require('assert'),
    constants = require('./constants.js'),
+    config = require('./config.js'),
    database = require('./database.js'),
    DatabaseError = require('./databaseerror'),
    mailboxdb = require('./mailboxdb.js');
@@ -88,10 +89,8 @@ function add(id, name, callback) {
    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof callback, 'function');

-    var data = [ id, name ];
-
    var queries = [];
-    queries.push({ query: 'INSERT INTO mailboxes (name, ownerId, ownerType) VALUES (?, ?, ?)', args: [ name, id, mailboxdb.TYPE_GROUP ] });
+    queries.push({ query: 'INSERT INTO mailboxes (name, domain, ownerId, ownerType) VALUES (?, ?, ?, ?)', args: [ name, config.fqdn(), id, mailboxdb.TYPE_GROUP ] });
    queries.push({ query: 'INSERT INTO groups (id, name) VALUES (?, ?)', args: [ id, name ] });

    database.transaction(queries, function (error, result) {
@@ -25,7 +25,7 @@ var assert = require('assert'),
    DatabaseError = require('./databaseerror.js'),
    groupdb = require('./groupdb.js'),
    util = require('util'),
-    uuid = require('node-uuid');
+    uuid = require('uuid');

 // http://dustinsenos.com/articles/customErrorsInNode
 // http://code.google.com/p/v8/wiki/JavaScriptStackTraceApi
@@ -5,9 +5,9 @@
 // Do not require anything here!

 exports = module.exports = {
-    // a major version makes all apps restore from backup
+    // a major version makes all apps restore from backup. #451 must be fixed before we do this.
    // a minor version makes all apps re-configure themselves
-    'version': '48.5.0',
+    'version': '48.8.0',

    'baseImages': [ 'cloudron/base:0.10.0' ],

@@ -15,10 +15,10 @@ exports = module.exports = {
    // This is because we upgrade using dumps instead of mysql_upgrade, pg_upgrade etc
    'images': {
        'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:0.18.0' },
-        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:0.17.0' },
+        'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:0.17.1' },
        'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:0.13.0' },
        'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:0.11.0' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:0.36.2' },
-        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:0.11.0' }
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:0.40.0' },
+        'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:0.12.0' }
    }
 };
@@ -70,14 +70,13 @@ function cleanupTmpVolume(containerInfo, callback) {
    docker.getContainer(containerInfo.Id).exec({ Cmd: cmd, AttachStdout: true, AttachStderr: true, Tty: false }, function (error, execContainer) {
        if (error) return callback(new Error('Failed to exec container : ' + error.message));

-        execContainer.start(function(err, stream) {
+        execContainer.start({ hijack: true }, function (error, stream) {
            if (error) return callback(new Error('Failed to start exec container : ' + error.message));

            stream.on('error', callback);
            stream.on('end', callback);

-            stream.setEncoding('utf8');
-            stream.pipe(process.stdout);
+            docker.modem.demuxStream(stream, process.stdout, process.stderr);
        });
    });
 }
@@ -61,12 +61,74 @@ function getUsersWithAccessToApp(req, callback) {
    });
 }

+// helper function to deal with pagination
+function finalSend(results, req, res, next) {
+    var min = 0;
+    var max = results.length;
+    var cookie = null;
+    var pageSize = 0;
+
+    // check if this is a paging request, if so get the cookie for session info
+    req.controls.forEach(function (control) {
+        if (control.type === ldap.PagedResultsControl.OID) {
+            pageSize = control.value.size;
+            cookie = control.value.cookie;
+        }
+    });
+
+    function sendPagedResults(start, end) {
+        start = (start < min) ? min : start;
+        end = (end > max || end < min) ? max : end;
+        var i;
+
+        for (i = start; i < end; i++) {
+            res.send(results[i]);
+        }
+
+        return i;
+    }
+
+    if (cookie && Buffer.isBuffer(cookie)) {
+        // we have pagination
+        var first = min;
+        if (cookie.length !== 0) {
+            first = parseInt(cookie.toString(), 10);
+        }
+        var last = sendPagedResults(first, first + pageSize);
+
+        var resultCookie;
+        if (last < max) {
+            resultCookie = new Buffer(last.toString());
+        } else {
+            resultCookie = new Buffer('');
+        }
+
+        res.controls.push(new ldap.PagedResultsControl({
+            value: {
+                size: pageSize, // correctness not required here
+                cookie: resultCookie
+            }
+        }));
+    } else {
+        // no pagination simply send all
+        results.forEach(function (result) {
+            res.send(result);
+        });
+    }
+
+    // all done
+    res.end();
+    next();
+}
+
 function userSearch(req, res, next) {
    debug('user search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);

    getUsersWithAccessToApp(req, function (error, result) {
        if (error) return next(error);

+        var results = [];
+
        // send user objects
        result.forEach(function (entry) {
            // skip entries with empty username. Some apps like owncloud can't deal with this
@@ -109,11 +171,11 @@ function userSearch(req, res, next) {
            if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));

            if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
-                res.send(obj);
+                results.push(obj);
            }
        });

-        res.end();
+        finalSend(results, req, res, next);
    });
 }

@@ -123,6 +185,8 @@ function groupSearch(req, res, next) {
    getUsersWithAccessToApp(req, function (error, result) {
        if (error) return next(error);

+        var results = [];
+
        var groups = [{
            name: 'users',
            admin: false
@@ -149,11 +213,43 @@ function groupSearch(req, res, next) {
            if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));

            if ((req.dn.equals(dn) || req.dn.parentOf(dn)) && lowerCaseFilter.matches(obj.attributes)) {
-                res.send(obj);
+                results.push(obj);
            }
        });

-        res.end();
+        finalSend(results, req, res, next);
+    });
+}
+
+function groupUsersCompare(req, res, next) {
+    debug('group users compare: dn %s, attribute %s, value %s (from %s)', req.dn.toString(), req.attribute, req.value, req.connection.ldap.id);
+
+    getUsersWithAccessToApp(req, function (error, result) {
+        if (error) return next(error);
+
+        // we only support memberuid here, if we add new group attributes later add them here
+        if (req.attribute === 'memberuid') {
+            var found = result.find(function (u) { return u.id === req.value; });
+            if (found) return res.end(true);
+        }
+
+        res.end(false);
+    });
+}
+
+function groupAdminsCompare(req, res, next) {
+    debug('group admins compare: dn %s, attribute %s, value %s (from %s)', req.dn.toString(), req.attribute, req.value, req.connection.ldap.id);
+
+    getUsersWithAccessToApp(req, function (error, result) {
+        if (error) return next(error);
+
+        // we only support memberuid here, if we add new group attributes later add them here
+        if (req.attribute === 'memberuid') {
+            var found = result.find(function (u) { return u.id === req.value; });
+            if (found && found.admin) return res.end(true);
+        }
+
+        res.end(false);
    });
 }

@@ -161,6 +257,7 @@ function mailboxSearch(req, res, next) {
    debug('mailbox search: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);

    if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
+
    var name = req.dn.rdns[0].attrs.cn.value.toLowerCase();
    // allow login via email
    var parts = name.split('@');
@@ -168,7 +265,7 @@ function mailboxSearch(req, res, next) {
        name = parts[0];
    }

-    mailboxdb.getMailbox(name, function (error, mailbox) {
+    mailboxdb.getMailbox(name, config.fqdn(), function (error, mailbox) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
        if (error) return next(new ldap.OperationsError(error.toString()));

@@ -188,9 +285,11 @@ function mailboxSearch(req, res, next) {
        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));

-        if (lowerCaseFilter.matches(obj.attributes)) res.send(obj);
-
-        res.end();
+        if (lowerCaseFilter.matches(obj.attributes)) {
+            finalSend([ obj ], req, res, next);
+        } else {
+            res.end();
+        }
    });
 }

@@ -198,7 +297,8 @@ function mailAliasSearch(req, res, next) {
    debug('mail alias get: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);

    if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-    mailboxdb.getAlias(req.dn.rdns[0].attrs.cn.value.toLowerCase(), function (error, alias) {
+
+    mailboxdb.getAlias(req.dn.rdns[0].attrs.cn.value.toLowerCase(), config.fqdn(), function (error, alias) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
        if (error) return next(new ldap.OperationsError(error.toString()));

@@ -218,9 +318,11 @@ function mailAliasSearch(req, res, next) {
        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));

-        if (lowerCaseFilter.matches(obj.attributes)) res.send(obj);
-
-        res.end();
+        if (lowerCaseFilter.matches(obj.attributes)) {
+            finalSend([ obj ], req, res, next);
+        } else {
+            res.end();
+        }
    });
 }

@@ -228,7 +330,8 @@ function mailingListSearch(req, res, next) {
    debug('mailing list get: dn %s, scope %s, filter %s (from %s)', req.dn.toString(), req.scope, req.filter.toString(), req.connection.ldap.id);

    if (!req.dn.rdns[0].attrs.cn) return next(new ldap.NoSuchObjectError(req.dn.toString()));
-    mailboxdb.getGroup(req.dn.rdns[0].attrs.cn.value.toLowerCase(), function (error, group) {
+
+    mailboxdb.getGroup(req.dn.rdns[0].attrs.cn.value.toLowerCase(), config.fqdn(), function (error, group) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
        if (error) return next(new ldap.OperationsError(error.toString()));

@@ -248,9 +351,11 @@ function mailingListSearch(req, res, next) {
        var lowerCaseFilter = safe(function () { return ldap.parseFilter(req.filter.toString().toLowerCase()); }, null);
        if (!lowerCaseFilter) return next(new ldap.OperationsError(safe.error.toString()));

-        if (lowerCaseFilter.matches(obj.attributes)) res.send(obj);
-
-        res.end();
+        if (lowerCaseFilter.matches(obj.attributes)) {
+            finalSend([ obj ], req, res, next);
+        } else {
+            res.end();
+        }
    });
 }

@@ -314,7 +419,7 @@ function authenticateMailbox(req, res, next) {
        name = parts[0];
    }

-    mailboxdb.getMailbox(name, function (error, mailbox) {
+    mailboxdb.getMailbox(name, config.fqdn(), function (error, mailbox) {
        if (error && error.reason === DatabaseError.NOT_FOUND) return next(new ldap.NoSuchObjectError(req.dn.toString()));
        if (error) return next(new ldap.OperationsError(error.message));

@@ -370,14 +475,17 @@ function start(callback) {
    gServer.bind('ou=recvmail,dc=cloudron', authenticateMailbox);
    gServer.bind('ou=sendmail,dc=cloudron', authenticateMailbox);

+    gServer.compare('cn=users,ou=groups,dc=cloudron', groupUsersCompare);
+    gServer.compare('cn=admins,ou=groups,dc=cloudron', groupAdminsCompare);
+
    // this is the bind for addons (after bind, they might search and authenticate)
-    gServer.bind('ou=addons,dc=cloudron', function(req, res, next) {
+    gServer.bind('ou=addons,dc=cloudron', function(req, res /*, next */) {
        debug('addons bind: %s', req.dn.toString()); // note: cn can be email or id
        res.end();
    });

    // this is the bind for apps (after bind, they might search and authenticate user)
-    gServer.bind('ou=apps,dc=cloudron', function(req, res, next) {
+    gServer.bind('ou=apps,dc=cloudron', function(req, res /*, next */) {
        // TODO: validate password
        debug('application bind: %s', req.dn.toString());
        res.end();
@@ -15,6 +15,7 @@ util.inherits(Locker, EventEmitter);

 // these are mutually exclusive operations
 Locker.prototype.OP_BOX_UPDATE = 'box_update';
+Locker.prototype.OP_PLATFORM_START = 'platform_start';
 Locker.prototype.OP_FULL_BACKUP = 'full_backup';
 Locker.prototype.OP_APPTASK = 'apptask';
 Locker.prototype.OP_MIGRATE = 'migrate';
@@ -4,7 +4,7 @@
  rotate 7
  daily
  compress
-  size=1M
+  maxsize=1M
  missingok
  delaycompress
  copytruncate
@@ -2,13 +2,14 @@

 Dear Cloudron Admin,

-a new version <%= updateInfo.manifest.version %> of the app '<%= app.manifest.title %>' installed at <%= app.fqdn %> is available!
-
-The app will update automatically tonight. Alternately, update immediately at <%= webadminUrl %>.
+A new version <%= updateInfo.manifest.version %> of the app '<%= app.manifest.title %>' installed at <%= app.fqdn %> is available!

 Changes:
 <%= updateInfo.manifest.changelog %>

+<% if (!hasSubscription) { -%>
+*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
+<% } -%>

 Powered by https://cloudron.io

@@ -16,4 +17,35 @@ Sent at: <%= new Date().toUTCString() %>

 <% } else { %>

+    <center>
+
+    <img src="<%= cloudronAvatarUrl %>" width="128px" height="128px"/>
+
+    <h3>Dear <%= cloudronName %> Admin,</h3>
+
+    <div style="width: 650px; text-align: left;">
+        <p>
+            A new version <%= updateInfo.manifest.version %> of the app '<%= app.manifest.title %>' installed at <%= app.fqdn %> is available!
+        </p>
+
+        <h5>Changelog:</h5>
+        <%- changelogHTML %>
+
+        <br/>
+
+        <% if (!hasSubscription) { %>
+        <p>Keep your Cloudron automatically up-to-date and secure by upgrading to a <a href="<%= webadminUrl %>/#/settings">paid plan</a>.</p>
+        <% } %>
+
+        <br/>
+    </div>
+
+    <div style="font-size: 10px; color: #333333; background: #ffffff;">
+        Powered by <a href="https://cloudron.io">Cloudron</a>.
+    </div>
+
+    </center>
+
+    <img src="https://analytics.cloudron.io/piwik.php?idsite=2&rec=1&e_c=CloudronEmail&e_a=update" style="border:0" alt="" />
+
 <% } %>
@@ -4,15 +4,18 @@ Dear <%= cloudronName %> Admin,

 Version <%= newBoxVersion %> for Cloudron <%= fqdn %> is now available!

-Your Cloudron will update automatically tonight. Alternately, update immediately at <%= webadminUrl %>.
-
 Changelog:
 <% for (var i = 0; i < changelog.length; i++) { %>
  * <%- changelog[i] %>
 <% } %>

-Thank you,
-your Cloudron
+<% if (!hasSubscription) { -%>
+*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
+<% } -%>
+
+Powered by https://cloudron.io
+
+Sent at: <%= new Date().toUTCString() %>

 <% } else { %>

@@ -27,11 +30,6 @@ your Cloudron
        Version <b><%= newBoxVersion %></b> for Cloudron <%= fqdn %> is now available!
    </p>

-    <p>
-        Your Cloudron will update automatically tonight.<br/>
-        Alternately, update immediately <a href="<%= webadminUrl %>">here</a>.
-    </p>
-
    <h5>Changelog:</h5>
    <ul>
    <% for (var i = 0; i < changelogHTML.length; i++) { %>
@@ -40,6 +38,11 @@ your Cloudron
    </ul>

    <br/>
+
+    <% if (!hasSubscription) { %>
+    <p>Keep your Cloudron automatically up-to-date and secure by upgrading to a <a href="<%= webadminUrl %>/#/settings">paid plan</a>.</p>
+    <% } %>
+
    <br/>
 </div>

@@ -2,7 +2,19 @@

 Dear <%= cloudronName %> Admin,

-This is the weekly summary of activities on your Cloudron <%= fqdn %>.
+This is a summary of the activities on your Cloudron <%= fqdn %>.
+<% if (info.usersAdded.length) { -%>
+
+The following users were added:
+<% for (var i = 0; i < info.usersAdded.length; i++) { -%>
+    * <%- info.usersAdded[i].email %>
+<% }} -%>
+<% if (info.certRenewals.length) { -%>
+
+The certificates of the following apps was renewed:
+<% for (var i = 0; i < info.certRenewals.length; i++) { -%>
+    * <%- info.certRenewals[i].domain %> - <%- info.certRenewals[i].errorMessage || 'Success' %>
+<% }} -%>
 <% if (info.pendingBoxUpdate) { -%>

 Cloudron v<%- info.pendingBoxUpdate.version %> is available:
@@ -33,6 +45,14 @@ The following apps were updated:
 <% for (var j = 0; j < info.finishedAppUpdates[i].toManifest.changelog.trim().split('\n').length; j++) { -%>
        <%= info.finishedAppUpdates[i].toManifest.changelog.trim().split('\n')[j] %>
 <% }}} -%>
+<% if (info.finishedBackups.length) { -%>
+
+Last successful backup: <%- info.finishedBackups[0].backupId || info.finishedBackups[0].filename %>
+<% } else { -%>
+
+This Cloudron did **not** backup successfully in the last week!
+<% } -%>
+
 <% if (!info.hasSubscription) { -%>

 *Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
@@ -52,9 +72,25 @@ Sent at: <%= new Date().toUTCString() %>

        <br/>

-        <p>Weekly summary of activities on your Cloudron <a href="<%= webadminUrl %>"><%= cloudronName %></a>:</p>
+        <p>This is a summary of the activities on your Cloudron <a href="<%= webadminUrl %>"><%= cloudronName %></a> last week.</p>

-        <br/>
+    <% if (info.usersAdded.length) { -%>
+        <p><b>The following users were added:</b></p>
+        <ul>
+        <% for (var i = 0; i < info.usersAdded.length; i++) { %>
+            <li><%- info.usersAdded[i].email %></li>
+        <% } %>
+        </ul>
+    <% } %>
+
+    <% if (info.certRenewals.length) { -%>
+        <p><b>The certificates of the following apps were renewed:</b></p>
+        <ul>
+            <% for (var i = 0; i < info.certRenewals.length; i++) { %>
+                <li><%- info.certRenewals[i].domain %> - <%- info.certRenewals[i].errorMessage || 'Success' %></li>
+            <% } %>
+        </ul>
+    <% } %>

    <% if (info.pendingBoxUpdate) { -%>
        <p><b>Cloudron v<%- info.pendingBoxUpdate.version %> is available:</b></p>
@@ -113,6 +149,12 @@ Sent at: <%= new Date().toUTCString() %>
        </ul>
    <% } %>

+    <% if (info.finishedBackups.length) { %>
+        <p><b>Last successful backup : </b> <%= info.finishedBackups[0].backupId || info.finishedBackups[0].filename %> </p>
+    <% } else { %>
+        <p><b>This Cloudron did not backup successfully in the last week!</b></p>
+    <% } %>
+
        <br/>

    <% if (!info.hasSubscription) { %>
@@ -123,12 +165,12 @@ Sent at: <%= new Date().toUTCString() %>
        <br/>
        <br/>

-        <p style="text-align: right;">
+        <center>
            <small>
                Powered by <a href="https://cloudron.io">Cloudron</a><br/>
                Sent on <%= new Date().toUTCString() %>
            </small>
-        </p>
+        </center>
    </div>
 </center>

@@ -1,14 +0,0 @@
-<%if (format === 'text') { %>
-
-New <%= type %> from <%= fqdn %>.
-
-Sender:  <%= user.email %>
-Sent at: <%= new Date().toUTCString() %>
-
-Subject: <%= subject %>
-----------------------------------------------------------
-<%= description %>
-
-<% } else { %>
-
-<% } %>
@@ -0,0 +1,11 @@
+<%if (format === 'text') { %>
+
+Test email from <%= fqdn %>,
+
+If you can read this, your Cloudron email settings are good.
+
+Sent at: <%= new Date().toUTCString() %>
+
+<% } else { %>
+
+<% } %>
@@ -31,15 +31,16 @@ var assert = require('assert'),
    DatabaseError = require('./databaseerror.js'),
    util = require('util');

-var MAILBOX_FIELDS = [ 'name', 'ownerId', 'ownerType', 'aliasTarget', 'creationTime' ].join(',');
+var MAILBOX_FIELDS = [ 'name', 'ownerId', 'ownerType', 'aliasTarget', 'creationTime', 'domain' ].join(',');

-function add(name, ownerId, ownerType, callback) {
+function add(name, domain, ownerId, ownerType, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof ownerId, 'string');
    assert.strictEqual(typeof ownerType, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('INSERT INTO mailboxes (name, ownerId, ownerType) VALUES (?, ?, ?)', [ name, ownerId, ownerType ], function (error) {
+    database.query('INSERT INTO mailboxes (name, domain, ownerId, ownerType) VALUES (?, ?, ?, ?)', [ name, domain, ownerId, ownerType ], function (error) {
        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'mailbox already exists'));
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

@@ -56,12 +57,13 @@ function clear(callback) {
    });
 }

-function del(name, callback) {
+function del(name, domain, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

    // deletes aliases as well
-    database.query('DELETE FROM mailboxes WHERE name=? OR aliasTarget = ?', [ name, name ], function (error, result) {
+    database.query('DELETE FROM mailboxes WHERE (name=? OR aliasTarget = ?) AND domain = ?', [ name, name, domain ], function (error, result) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (result.affectedRows === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -81,15 +83,17 @@ function delByOwnerId(id, callback) {
    });
 }

-function updateName(oldName, newName, callback) {
+function updateName(oldName, oldDomain, newName, newDomain, callback) {
    assert.strictEqual(typeof oldName, 'string');
+    assert.strictEqual(typeof oldDomain, 'string');
    assert.strictEqual(typeof newName, 'string');
+    assert.strictEqual(typeof newDomain, 'string');
    assert.strictEqual(typeof callback, 'function');

    // skip if no changes
-    if (oldName === newName) return callback(null);
+    if (oldName === newName && oldDomain === newDomain) return callback(null);

-    database.query('UPDATE mailboxes SET name=? WHERE name=?', [ newName, oldName ], function (error, result) {
+    database.query('UPDATE mailboxes SET name=?, domain=? WHERE name=? AND domain = ?', [ newName, newDomain, oldName, oldDomain ], function (error, result) {
        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, 'mailbox already exists'));
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
@@ -98,11 +102,12 @@ function updateName(oldName, newName, callback) {
    });
 }

-function getMailbox(name, callback) {
+function getMailbox(name, domain, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND (ownerType = ? OR ownerType = ?) AND aliasTarget IS NULL', [ name, exports.TYPE_APP, exports.TYPE_USER ], function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND domain = ? AND (ownerType = ? OR ownerType = ?) AND aliasTarget IS NULL', [ name, domain, exports.TYPE_APP, exports.TYPE_USER ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -110,18 +115,20 @@ function getMailbox(name, callback) {
    });
 }

-function listMailboxes(callback) {
+function listMailboxes(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE (ownerType = ? OR ownerType = ?) AND aliasTarget IS NULL ORDER BY name', [ exports.TYPE_APP, exports.TYPE_USER ], function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE domain = ? AND (ownerType = ? OR ownerType = ?) AND aliasTarget IS NULL ORDER BY name', [ domain, exports.TYPE_APP, exports.TYPE_USER ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
    });
 }

-function getGroup(name, callback) {
+function getGroup(name, domain, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

    // This can be merged into a single query but cannot get 'not found' information
@@ -130,11 +137,12 @@ function getGroup(name, callback) {
    //    INNER JOIN users ON groupMembers.userId = users.id
    //    WHERE mailboxes.name = <name>

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND ownerType = ? AND aliasTarget IS NULL', [ name, exports.TYPE_GROUP ], function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND domain = ? AND ownerType = ? AND aliasTarget IS NULL', [ name, domain, exports.TYPE_GROUP ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

-        database.query('SELECT users.username FROM groupMembers INNER JOIN users ON groupMembers.userId = users.id WHERE groupMembers.groupId = ?', [ results[0].ownerId ], function (error, memberList) {
+        // username can be null if the user has not signed up with the invite yet
+        database.query('SELECT users.username FROM groupMembers INNER JOIN users ON groupMembers.userId = users.id WHERE groupMembers.groupId = ? AND users.username IS NOT NULL', [ results[0].ownerId ], function (error, memberList) {
            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            results[0].members = memberList.map(function (m) { return m.username; });
@@ -156,20 +164,21 @@ function getByOwnerId(ownerId, callback) {
    });
 }

-function setAliasesForName(name, aliases, callback) {
+function setAliasesForName(name, domain, aliases, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert(util.isArray(aliases));
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? ', [ name ], function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND domain = ?', [ name, domain ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        var queries = [];
-        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasTarget = ?', args: [ name ] });
+        queries.push({ query: 'DELETE FROM mailboxes WHERE aliasTarget = ? AND domain = ?', args: [ name, domain ] });
        aliases.forEach(function (alias) {
-            queries.push({ query: 'INSERT INTO mailboxes (name, aliasTarget, ownerId, ownerType) VALUES (?, ?, ?, ?)',
-                         args: [ alias, name, results[0].ownerId, results[0].ownerType ] });
+            queries.push({ query: 'INSERT INTO mailboxes (name, domain, aliasTarget, ownerId, ownerType) VALUES (?, ?, ?, ?, ?)',
+                         args: [ alias, domain, name, results[0].ownerId, results[0].ownerType ] });
        });

        database.transaction(queries, function (error) {
@@ -181,11 +190,12 @@ function setAliasesForName(name, aliases, callback) {
    });
 }

-function getAliasesForName(name, callback) {
+function getAliasesForName(name, domain, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT name FROM mailboxes WHERE aliasTarget=? ORDER BY name', [ name ], function (error, results) {
+    database.query('SELECT name FROM mailboxes WHERE aliasTarget = ? AND domain = ? ORDER BY name', [ name, domain ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        results = results.map(function (r) { return r.name; });
@@ -193,21 +203,23 @@ function getAliasesForName(name, callback) {
    });
 }

-function listAliases(callback) {
+function listAliases(domain, callback) {
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE aliasTarget IS NOT NULL ORDER BY name', function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE domain = ? AND aliasTarget IS NOT NULL ORDER BY name', [ domain ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
    });
 }

-function getAlias(name, callback) {
+function getAlias(name, domain, callback) {
    assert.strictEqual(typeof name, 'string');
+    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND aliasTarget IS NOT NULL', [ name ], function (error, results) {
+    database.query('SELECT ' + MAILBOX_FIELDS + ' FROM mailboxes WHERE name = ? AND domain = ? AND aliasTarget IS NOT NULL', [ name, domain ], function (error, results) {
        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

@@ -1,9 +1,6 @@
 'use strict';

 exports = module.exports = {
-    start: start,
-    stop: stop,
-
    userAdded: userAdded,
    userRemoved: userRemoved,
    adminChanged: adminChanged,
@@ -23,12 +20,7 @@ exports = module.exports = {

    certificateRenewalError: certificateRenewalError,

-    FEEDBACK_TYPE_FEEDBACK: 'feedback',
-    FEEDBACK_TYPE_TICKET: 'ticket',
-    FEEDBACK_TYPE_APP_MISSING: 'app_missing',
-    FEEDBACK_TYPE_APP_ERROR: 'app_error',
-    FEEDBACK_TYPE_UPGRADE_REQUEST: 'upgrade_request',
-    sendFeedback: sendFeedback,
+    sendTestMail: sendTestMail,

    _getMailQueue: _getMailQueue,
    _clearMailQueue: _clearMailQueue
@@ -54,8 +46,7 @@ var NOOP_CALLBACK = function (error) { if (error) debug(error); };

 var MAIL_TEMPLATES_DIR = path.join(__dirname, 'mail_templates');

-var gMailQueue = [ ],
-    gPaused = false;
+var gMailQueue = [ ];

 function splatchError(error) {
    var result = { };
@@ -68,25 +59,6 @@ function splatchError(error) {
    return util.inspect(result, { depth: null, showHidden: true });
 }

-function start(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (process.env.BOX_ENV === 'test') gPaused = true;
-
-    callback(null);
-}
-
-function stop(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    // TODO: interrupt processQueue as well
-
-    debug(gMailQueue.length + ' mail items dropped');
-    gMailQueue = [ ];
-
-    callback(null);
-}
-
 function mailConfig() {
    return {
        from: '"Cloudron" <no-reply@' + config.fqdn() + '>'
@@ -94,8 +66,6 @@ function mailConfig() {
 }

 function processQueue() {
-    assert(!gPaused);
-
    sendMails(gMailQueue);
    gMailQueue = [ ];
 }
@@ -142,7 +112,7 @@ function enqueue(mailOptions) {
    debug('Queued mail for ' + mailOptions.from + ' to ' + mailOptions.to);
    gMailQueue.push(mailOptions);

-    if (!gPaused) processQueue();
+    if (process.env.BOX_ENV !== 'test') processQueue();
 }

 function render(templateFile, params) {
@@ -167,6 +137,7 @@ function getAdminEmails(callback) {
        if (admins.length === 0) return callback(new Error('No admins on this cloudron')); // box not activated yet

        var adminEmails = [ ];
+        if (admins[0].alternateEmail) adminEmails.push(admins[0].alternateEmail);
        admins.forEach(function (admin) { adminEmails.push(admin.email); });

        callback(null, adminEmails);
@@ -240,7 +211,7 @@ function userAdded(user, inviteSent) {
    debug('Sending mail for userAdded %s including invite link', inviteSent ? 'not' : '');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        adminEmails = _.difference(adminEmails, [ user.email ]);

@@ -337,7 +308,7 @@ function appDied(app) {
    debug('Sending mail for app %s @ %s died', app.id, app.fqdn);

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        var mailOptions = {
            from: mailConfig().from,
@@ -350,12 +321,13 @@ function appDied(app) {
    });
 }

-function boxUpdateAvailable(newBoxVersion, changelog) {
+function boxUpdateAvailable(hasSubscription, newBoxVersion, changelog) {
+    assert.strictEqual(typeof hasSubscription, 'boolean');
    assert.strictEqual(typeof newBoxVersion, 'string');
    assert(util.isArray(changelog));

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        settings.getCloudronName(function (error, cloudronName) {
            if (error) {
@@ -369,6 +341,7 @@ function boxUpdateAvailable(newBoxVersion, changelog) {
                fqdn: config.fqdn(),
                webadminUrl: config.adminOrigin(),
                newBoxVersion: newBoxVersion,
+                hasSubscription: hasSubscription,
                changelog: changelog,
                changelogHTML: changelog.map(function (e) { return converter.makeHtml(e); }),
                cloudronName: cloudronName,
@@ -381,7 +354,7 @@ function boxUpdateAvailable(newBoxVersion, changelog) {
            var templateDataHTML = JSON.parse(JSON.stringify(templateData));
            templateDataHTML.format = 'html';

-             var mailOptions = {
+            var mailOptions = {
                from: mailConfig().from,
                to: adminEmails.join(', '),
                subject: util.format('%s has a new update available', config.fqdn()),
@@ -394,21 +367,49 @@ function boxUpdateAvailable(newBoxVersion, changelog) {
    });
 }

-function appUpdateAvailable(app, updateInfo) {
+function appUpdateAvailable(app, hasSubscription, info) {
    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof updateInfo, 'object');
+    assert.strictEqual(typeof hasSubscription, 'boolean');
+    assert.strictEqual(typeof info, 'object');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

-         var mailOptions = {
-            from: mailConfig().from,
-            to: adminEmails.join(', '),
-            subject: util.format('[%s] Update available for %s', config.fqdn(), app.fqdn),
-            text: render('app_update_available.ejs', { fqdn: config.fqdn(), webadminUrl: config.adminOrigin(), app: app, updateInfo: updateInfo, format: 'text' })
-        };
+        settings.getCloudronName(function (error, cloudronName) {
+            if (error) {
+                debug(error);
+                cloudronName = 'Cloudron';
+            }

-        enqueue(mailOptions);
+            var converter = new showdown.Converter();
+
+            var templateData = {
+                fqdn: config.fqdn(),
+                webadminUrl: config.adminOrigin(),
+                hasSubscription: hasSubscription,
+                app: app,
+                updateInfo: info,
+                changelogHTML: converter.makeHtml(info.manifest.changelog),
+                cloudronName: cloudronName,
+                cloudronAvatarUrl: config.adminOrigin() + '/api/v1/cloudron/avatar'
+            };
+
+            var templateDataText = JSON.parse(JSON.stringify(templateData));
+            templateDataText.format = 'text';
+
+            var templateDataHTML = JSON.parse(JSON.stringify(templateData));
+            templateDataHTML.format = 'html';
+
+            var mailOptions = {
+                from: mailConfig().from,
+                to: adminEmails.join(', '),
+                subject: util.format('App %s has a new update available', app.fqdn),
+                text: render('app_update_available.ejs', templateDataText),
+                html: render('app_update_available.ejs', templateDataHTML)
+            };
+
+            enqueue(mailOptions);
+        });
    });
 }

@@ -416,7 +417,7 @@ function sendDigest(info) {
    assert.strictEqual(typeof info, 'object');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        settings.getCloudronName(function (error, cloudronName) {
            if (error) {
@@ -455,7 +456,7 @@ function outOfDiskSpace(message) {
    assert.strictEqual(typeof message, 'string');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        var mailOptions = {
            from: mailConfig().from,
@@ -472,7 +473,7 @@ function backupFailed(error) {
    var message = splatchError(error);

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        var mailOptions = {
            from: mailConfig().from,
@@ -490,7 +491,7 @@ function certificateRenewalError(domain, message) {
    assert.strictEqual(typeof message, 'string');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        var mailOptions = {
            from: mailConfig().from,
@@ -508,7 +509,7 @@ function oomEvent(program, context) {
    assert.strictEqual(typeof context, 'string');

    getAdminEmails(function (error, adminEmails) {
-        if (error) return console.log('Error getting admins', error);
+        if (error) return debug('Error getting admins', error);

        var mailOptions = {
            from: mailConfig().from,
@@ -540,23 +541,14 @@ function unexpectedExit(program, context, callback) {
    sendMails([ mailOptions ], callback);
 }

-function sendFeedback(user, type, subject, description) {
-    assert.strictEqual(typeof user, 'object');
-    assert.strictEqual(typeof type, 'string');
-    assert.strictEqual(typeof subject, 'string');
-    assert.strictEqual(typeof description, 'string');
-
-    assert(type === exports.FEEDBACK_TYPE_TICKET ||
-        type === exports.FEEDBACK_TYPE_FEEDBACK ||
-        type === exports.FEEDBACK_TYPE_APP_MISSING ||
-        type === exports.FEEDBACK_TYPE_UPGRADE_REQUEST ||
-        type === exports.FEEDBACK_TYPE_APP_ERROR);
+function sendTestMail(email) {
+    assert.strictEqual(typeof email, 'string');

    var mailOptions = {
        from: mailConfig().from,
-        to: 'support@cloudron.io',
-        subject: util.format('[%s] %s - %s', type, config.fqdn(), subject),
-        text: render('feedback.ejs', { fqdn: config.fqdn(), type: type, user: user, subject: subject, description: description, format: 'text'})
+        to: email,
+        subject: util.format('Test Email from %s', config.fqdn()),
+        text: render('test.ejs', { fqdn: config.fqdn(), format: 'text'})
    };

    enqueue(mailOptions);
@@ -32,6 +32,7 @@ function configureAdmin(certFilePath, keyFilePath, configFileName, vhost, callba
        sourceDir: path.resolve(__dirname, '..'),
        adminOrigin: config.adminOrigin(),
        vhost: vhost, // if vhost is empty it will become the default_server
+        hasIPv6: config.hasIPv6(),
        endpoint: 'admin',
        certFilePath: certFilePath,
        keyFilePath: keyFilePath,
@@ -54,12 +55,13 @@ function configureApp(app, certFilePath, keyFilePath, callback) {

    var sourceDir = path.resolve(__dirname, '..');
    var endpoint = 'app';
-    var vhost = app.altDomain || config.appFqdn(app.location);
+    var vhost = app.altDomain || app.intrinsicFqdn;

    var data = {
        sourceDir: sourceDir,
        adminOrigin: config.adminOrigin(),
        vhost: vhost,
+        hasIPv6: config.hasIPv6(),
        port: app.httpPort,
        endpoint: endpoint,
        certFilePath: certFilePath,
@@ -84,7 +86,7 @@ function unconfigureApp(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    var vhost = app.altDomain || config.appFqdn(app.location);
+    var vhost = app.altDomain || app.intrinsicFqdn;

    var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, app.id + '.conf');
    if (!safe.fs.unlinkSync(nginxConfigFilename)) {
@@ -15,64 +15,67 @@ app.controller('Controller', ['$scope', function ($scope) {

 </script>

-<center>
+<div class="layout-content">
+
+  <center>
    <br/>
    <h4>Hello <%= (user && user.email) ? user.email : '' %>, welcome to <%= cloudronName %>.</h4>
    <h2>Setup your account and password.</h2>
-</center>
+  </center>

-<div class="container" ng-app="Application" ng-controller="Controller">
+  <div class="container" ng-app="Application" ng-controller="Controller">
    <div class="row">
-        <div class="col-md-6 col-md-offset-3">
-            <form action="/api/v1/session/account/setup" method="post" name="setupForm" autocomplete="off" role="form" novalidate>
-                <input type="password" style="display: none;">
-                <input type="hidden" name="_csrf" value="<%= csrf %>"/>
-                <input type="hidden" name="resetToken" value="<%= resetToken %>"/>
+      <div class="col-md-6 col-md-offset-3">
+        <form action="/api/v1/session/account/setup" method="post" name="setupForm" autocomplete="off" role="form" novalidate>
+          <input type="password" style="display: none;">
+          <input type="hidden" name="_csrf" value="<%= csrf %>"/>
+          <input type="hidden" name="resetToken" value="<%= resetToken %>"/>

-                <center><p class="has-error"><%= error %></p></center>
+          <center><p class="has-error"><%= error %></p></center>

 <% if (user && user.username) { %>
-                <div class="form-group"">
-                    <label class="control-label">Username</label>
-                    <input type="text" class="form-control" ng-model="username" name="username" readonly required>
-                </div>
+          <div class="form-group"">
+            <label class="control-label">Username</label>
+            <input type="text" class="form-control" ng-model="username" name="username" readonly required>
+          </div>
 <% } else { %>
-                <div class="form-group" ng-class="{ 'has-error': (setupForm.username.$dirty && setupForm.username.$invalid) }">
-                    <label class="control-label">Username</label>
-                    <div class="control-label" ng-show="setupForm.username.$dirty && setupForm.username.$invalid">
-                        <small ng-show="setupForm.username.$error.minlength">The username is too short</small>
-                        <small ng-show="setupForm.username.$error.maxlength">The username is too long</small>
-                        <small ng-show="setupForm.username.$dirty && setupForm.username.$invalid">Not a valid username</small>
-                    </div>
-                    <input type="text" class="form-control" ng-model="username" name="username" required autofocus>
-                </div>
+          <div class="form-group" ng-class="{ 'has-error': (setupForm.username.$dirty && setupForm.username.$invalid) }">
+            <label class="control-label">Username</label>
+            <div class="control-label" ng-show="setupForm.username.$dirty && setupForm.username.$invalid">
+              <small ng-show="setupForm.username.$error.minlength">The username is too short</small>
+              <small ng-show="setupForm.username.$error.maxlength">The username is too long</small>
+              <small ng-show="setupForm.username.$dirty && setupForm.username.$invalid">Not a valid username</small>
+            </div>
+            <input type="text" class="form-control" ng-model="username" name="username" required autofocus>
+          </div>
 <% } %>

-                <div class="form-group">
-                    <label class="control-label">Display Name</label>
-                    <input type="displayName" class="form-control" ng-model="displayName" name="displayName" required>
-                </div>
+          <div class="form-group">
+            <label class="control-label">Display Name</label>
+            <input type="displayName" class="form-control" ng-model="displayName" name="displayName" required>
+          </div>

-                <div class="form-group" ng-class="{ 'has-error': (setupForm.password.$dirty && setupForm.password.$invalid) }">
-                    <label class="control-label">New Password</label>
-                    <div class="control-label" ng-show="setupForm.password.$dirty && setupForm.password.$invalid">
-                        <small ng-show="setupForm.password.$dirty && setupForm.password.$invalid">Password must be 8-30 character with at least one uppercase, one numeric and one special character</small>
-                    </div>
-                    <input type="password" class="form-control" ng-model="password" name="password" ng-pattern="/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[^a-zA-Z0-9])(?!.*\s).{8,30}$/" required>
-                </div>
+          <div class="form-group" ng-class="{ 'has-error': (setupForm.password.$dirty && setupForm.password.$invalid) }">
+            <label class="control-label">New Password</label>
+            <div class="control-label" ng-show="setupForm.password.$dirty && setupForm.password.$invalid">
+              <small ng-show="setupForm.password.$dirty && setupForm.password.$invalid">Password must be 8-30 character with at least one uppercase, one numeric and one special character</small>
+            </div>
+            <input type="password" class="form-control" ng-model="password" name="password" ng-pattern="/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[^a-zA-Z0-9])(?!.*\s).{8,30}$/" required>
+          </div>

-                <div class="form-group" ng-class="{ 'has-error': (setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)) }">
-                    <label class="control-label">Repeat Password</label>
-                    <div class="control-label" ng-show="setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)">
-                        <small ng-show="setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)">Passwords don't match</small>
-                    </div>
-                    <input type="password" class="form-control" ng-model="passwordRepeat" name="passwordRepeat" required>
-                </div>
+          <div class="form-group" ng-class="{ 'has-error': (setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)) }">
+            <label class="control-label">Repeat Password</label>
+            <div class="control-label" ng-show="setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)">
+              <small ng-show="setupForm.passwordRepeat.$dirty && (password !== passwordRepeat)">Passwords don't match</small>
+            </div>
+            <input type="password" class="form-control" ng-model="passwordRepeat" name="passwordRepeat" required>
+          </div>

-                <input class="btn btn-primary btn-outline pull-right" type="submit" value="Create" ng-disabled="setupForm.$invalid || password !== passwordRepeat"/>
-            </form>
-        </div>
+          <input class="btn btn-primary btn-outline pull-right" type="submit" value="Create" ng-disabled="setupForm.$invalid || password !== passwordRepeat"/>
+        </form>
+      </div>
    </div>
+  </div>
 </div>

 <% include footer %>
@@ -2,25 +2,26 @@

 <!-- error tester -->

-<br/>
+<div class="layout-content">

-<div class="container">
+  <div class="container" style="margin-top: 50px;">
    <div class="row">
-        <div class="col-md-2"></div>
-        <div class="col-md-8">
-            <div class="alert alert-danger">
-                <%- message %>
-            </div>
+      <div class="col-md-2"></div>
+      <div class="col-md-8">
+        <div class="alert alert-danger">
+          <%- message %>
        </div>
-        <div class="col-md-2"></div>
+      </div>
+      <div class="col-md-2"></div>
    </div>
    <div class="row">
-        <div class="col-md-2"></div>
-        <div class="col-md-8 text-center">
-            <a href="<%- adminOrigin %>">Back</a>
-        </div>
-        <div class="col-md-2"></div>
+      <div class="col-md-2"></div>
+      <div class="col-md-8 text-center">
+        <a href="<%- adminOrigin %>">Back</a>
+      </div>
+      <div class="col-md-2"></div>
    </div>
+  </div>
 </div>

 <% include footer %>
@@ -1,9 +1,11 @@

    <footer class="text-center">
-        <span class="text-muted">&copy; 2017 <a href="https://cloudron.io" target="_blank">Cloudron</a></span>
-        <span class="text-muted"><a href="https://twitter.com/cloudron_io" target="_blank">Twitter <i class="fa fa-twitter"></i></a></span>
-        <span class="text-muted"><a href="https://chat.cloudron.io" target="_blank">Chat <i class="fa fa-comments"></i></a></span>
+      <span class="text-muted">&copy; 2017 <a href="https://cloudron.io" target="_blank">Cloudron</a></span>
+      <span class="text-muted"><a href="https://twitter.com/cloudron_io" target="_blank">Twitter <i class="fa fa-twitter"></i></a></span>
+      <span class="text-muted"><a href="https://chat.cloudron.io" target="_blank">Chat <i class="fa fa-comments"></i></a></span>
    </footer>

-    </body>
+	</div>
+
+</body>
 </html>
@@ -27,14 +27,15 @@

 </head>

-<body class="oauth">
+<body>
+
+  <div class="layout-root">

-    <!-- Navigation -->
    <nav class="navbar navbar-default navbar-static-top shadow" role="navigation" style="margin-bottom: 0">
-        <div class="container-fluid">
-            <div class="navbar-header">
-                <a href="/" class="navbar-brand navbar-brand-icon"><img src="/api/v1/cloudron/avatar?<%= Math.random() %>" width="40" height="40"/></a>
-                <a href="/" class="navbar-brand"><%= cloudronName %></a>
-            </div>
+      <div class="container-fluid">
+        <div class="navbar-header">
+          <a href="/" class="navbar-brand navbar-brand-icon"><img src="/api/v1/cloudron/avatar?<%= Math.random() %>" width="40" height="40"/></a>
+          <a href="/" class="navbar-brand"><%= cloudronName %></a>
        </div>
+      </div>
    </nav>
@@ -2,45 +2,41 @@

 <!-- login tester -->

-<div class="container">
+<div class="layout-content">
+  <div class="card" style="padding: 20px; margin-top: 50px; max-width: 620px;">
    <div class="row">
-        <div class="col-md-6 col-md-offset-3">
-            <div class="card">
-                <div class="row">
-                    <div class="col-md-12" style="text-align: center;">
-                        <img width="128" height="128" src="<%= applicationLogo %>?<%= Math.random() %>"/>
-                        <h1><small>Login to</small> <%= applicationName %></h1>
-                        <br/>
-                    </div>
-                </div>
-                <br/>
-            <% if (error) { %>
-                <div class="row">
-                    <div class="col-md-12">
-                        <h4 class="has-error"><%= error %></h4>
-                    </div>
-                </div>
-            <% } %>
-                <div class="row">
-                    <div class="col-md-12">
-                        <form id="loginForm" action="" method="post">
-                            <input type="hidden" name="_csrf" value="<%= csrf %>"/>
-                            <div class="form-group">
-                                <label class="control-label" for="inputUsername">Username or Email</label>
-                                <input type="text" class="form-control" id="inputUsername" name="username" value="<%= username %>" autofocus required>
-                            </div>
-                            <div class="form-group">
-                                <label class="control-label" for="inputPassword">Password</label>
-                                <input type="password" class="form-control" name="password" id="inputPassword" value="<%= password %>" required>
-                            </div>
-                            <input class="btn btn-primary btn-outline pull-right" type="submit" value="Sign in"/>
-                        </form>
-                        <a href="/api/v1/session/password/resetRequest.html">Reset your password</a>
-                    </div>
-                </div>
-            </div>
-        </div>
+      <div class="col-md-12" style="text-align: center;">
+        <img width="128" height="128" src="<%= applicationLogo %>?<%= Math.random() %>"/>
+        <h1><small>Login to</small> <%= applicationName %></h1>
+        <br/>
+      </div>
    </div>
+    <br/>
+<% if (error) { -%>
+    <div class="row">
+      <div class="col-md-12">
+        <h4 class="has-error"><%= error %></h4>
+      </div>
+    </div>
+<% } -%>
+    <div class="row">
+      <div class="col-md-12">
+        <form id="loginForm" action="" method="post">
+          <input type="hidden" name="_csrf" value="<%= csrf %>"/>
+          <div class="form-group">
+            <label class="control-label" for="inputUsername">Username or Email</label>
+            <input type="text" class="form-control" id="inputUsername" name="username" value="<%= username %>" autofocus required>
+          </div>
+          <div class="form-group">
+            <label class="control-label" for="inputPassword">Password</label>
+            <input type="password" class="form-control" name="password" id="inputPassword" value="<%= password %>" required>
+          </div>
+          <input class="btn btn-primary btn-outline pull-right" type="submit" value="Sign in"/>
+        </form>
+        <a href="/api/v1/session/password/resetRequest.html">Reset your password</a>
+      </div>
+    </div>
+  </div>
 </div>

 <script>
@@ -12,36 +12,41 @@ app.controller('Controller', [function () {}]);

 </script>

-<center>
-    <h1>Hello <%= user.username %>, set a new password</h1>
-</center>
+<div class="layout-content">

-<div class="container" ng-app="Application" ng-controller="Controller">
+  <center>
+    <h2>Hello <%= user.username %>, set a new password</h2>
+  </center>
+
+  <br/>
+
+  <div class="container" ng-app="Application" ng-controller="Controller">
    <div class="row">
-        <div class="col-md-6 col-md-offset-3">
-            <form action="/api/v1/session/password/reset" method="post" name="resetForm" autocomplete="off" role="form" novalidate>
-                <input type="password" style="display: none;">
-                <input type="hidden" name="_csrf" value="<%= csrf %>"/>
-                <input type="hidden" name="resetToken" value="<%= resetToken %>"/>
+      <div class="col-md-6 col-md-offset-3">
+        <form action="/api/v1/session/password/reset" method="post" name="resetForm" autocomplete="off" role="form" novalidate>
+          <input type="password" style="display: none;">
+          <input type="hidden" name="_csrf" value="<%= csrf %>"/>
+          <input type="hidden" name="resetToken" value="<%= resetToken %>"/>

-                <div class="form-group" ng-class="{ 'has-error': resetForm.password.$dirty && resetForm.password.$invalid }">
-                    <label class="control-label" for="inputPassword">New Password</label>
-                    <div class="control-label" ng-show="resetForm.password.$dirty && resetForm.password.$invalid">
-                        <small ng-show="resetForm.password.$dirty && resetForm.password.$invalid">Password must be 8-30 character with at least one uppercase, one numeric and one special character</small>
-                    </div>
-                    <input type="password" class="form-control" id="inputPassword" ng-model="password" name="password" ng-pattern="/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[^a-zA-Z0-9])(?!.*\s).{8,30}$/" autofocus required>
-                </div>
-                <div class="form-group" ng-class="{ 'has-error': resetForm.passwordRepeat.$dirty && (password !== passwordRepeat) }">
-                    <label class="control-label" for="inputPasswordRepeat">Repeat Password</label>
-                    <div class="control-label" ng-show="resetForm.passwordRepeat.$dirty && (password !== passwordRepeat)">
-                        <small ng-show="resetForm.passwordRepeat.$dirty && (password !== passwordRepeat)">Passwords don't match</small>
-                    </div>
-                    <input type="password" class="form-control" id="inputPasswordRepeat" ng-model="passwordRepeat" name="passwordRepeat" required>
-                </div>
-                <input class="btn btn-primary btn-outline pull-right" type="submit" value="Create" ng-disabled="resetForm.$invalid || password !== passwordRepeat"/>
-            </form>
-        </div>
+          <div class="form-group" ng-class="{ 'has-error': resetForm.password.$dirty && resetForm.password.$invalid }">
+            <label class="control-label" for="inputPassword">New Password</label>
+            <div class="control-label" ng-show="resetForm.password.$dirty && resetForm.password.$invalid">
+              <small ng-show="resetForm.password.$dirty && resetForm.password.$invalid">Password must be 8-30 character with at least one uppercase, one numeric and one special character</small>
+            </div>
+            <input type="password" class="form-control" id="inputPassword" ng-model="password" name="password" ng-pattern="/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[^a-zA-Z0-9])(?!.*\s).{8,30}$/" autofocus required>
+          </div>
+          <div class="form-group" ng-class="{ 'has-error': resetForm.passwordRepeat.$dirty && (password !== passwordRepeat) }">
+            <label class="control-label" for="inputPasswordRepeat">Repeat Password</label>
+            <div class="control-label" ng-show="resetForm.passwordRepeat.$dirty && (password !== passwordRepeat)">
+              <small ng-show="resetForm.passwordRepeat.$dirty && (password !== passwordRepeat)">Passwords don't match</small>
+            </div>
+            <input type="password" class="form-control" id="inputPasswordRepeat" ng-model="passwordRepeat" name="passwordRepeat" required>
+          </div>
+          <input class="btn btn-primary btn-outline pull-right" type="submit" value="Create" ng-disabled="resetForm.$invalid || password !== passwordRepeat"/>
+        </form>
+      </div>
    </div>
+  </div>
 </div>

 <% include footer %>
--- a/Show More
+++ b/Show More