Ensure notifications are sorted by time descending

(cherry picked from commit 63310c44c0)
4.0.3 changes
2019-05-14 09:49:26 -07:00 · 2019-05-14 16:57:45 +02:00 · 2019-05-12 18:14:17 -07:00 · 2019-05-12 14:04:26 -07:00 · 2019-05-12 13:59:07 -07:00 · 2019-05-12 12:57:06 -07:00
288 changed files with 19396 additions and 21242 deletions
@@ -1600,494 +1600,3 @@
 [4.0.3]
 * Fix dashboard issue for non-admins

-[4.1.0]
-* Remove password requirement for uninstalling apps and users
-* Hosting provider edition
-* Enforce limits in mail container
-* Fix crash when using unauthenticated relay
-* Fix domain and tag filtering
-* Customizable app icons
-* Remove obsolete X-Frame-Options from nginx configs
-* Give SFTP access based on access restriction
-
-[4.1.1]
-* Add UI hint about SFTP access restriction
-
-[4.1.2]
-* Accept incoming mail from a private relay
-* Fix issue where unused addon images were not pruned
-* Add UI for redirect from multiple domains
-* Allow apps to be relocated to custom data directory
-* Make all cloudron env vars have CLOUDRON_ prefix
-* Update manifest version to 2
-* Fix issue where DKIM keys were inaccessible
-* Fix DKIM selector conflict when adding same domain across multiple cloudrons
-* Fix name.com DNS backend issue for naked domains
-* Add DigitalOcean Frankfurt (fra1) region for backup storage
-
-[4.1.3]
-* Update manifest format package
-
-[4.1.4]
-* Add CLOUDRON_ prefix to MySQL addon variables
-
-[4.1.5]
-* Make the terminal addon button inject variables based on manifest version
-* Preserve addon passwords correctly when using v2 manifest
-* Show error message instead of logging out user when invalid 2FA token is provided
-* Ensure redis vars are renamed with manifest v2
-* Add missing Scaleway Object Storage to restore UI
-* Fix Exoscale endpoints in restore UI
-* Reset the app icon when showing the configure UI
-
-[4.1.6]
-* Fix issue where CLOUDRON_APP_HOSTNAME was incorrectly set
-* Remove chat link from the footer of login screen
-* Add support for oplog tailing in mongodb
-* Fix LDAP not accessible via scheduler containers
-
-[4.1.7]
-* Fix issue where login looped when admin bit was removed
-
-[4.2.0]
-* Fix issue where tar backups with files > 8GB was corrupt
-* Add SparkPost as mail relay backend
-* Add Wasabi storage backend
-* TOTP tokens are now checked for with +- 60 seconds
-* IP based restore
-* Fix issue where task logs were not getting rotated correctly
-* Add notification for box update
-* User enable/disable flag
-* Check disk space before various operations like install, update, backup etc
-* Collect per app du information
-* Set Cloudron specific UA for healthchecks
-* Show message why an app task is 'pending'
-* Rework app task system so that we can now pass dynamic arguments
-* Add external LDAP server integration
-
-[4.2.1]
-* Rework the app configuration routes & UI
-* Fine grained eventlog for app configuration
-* Update Haraka to 2.8.24
-* Set sieve_max_redirects to 64
-* SRS support for mail forwarding
-* Fix issue where sieve responses were not sent via the relay
-* File based session store
-* Fix API token error reporting for namecheap backend
-
-[4.2.2]
-* Fix typos in migration
-
-[4.2.3]
-* Remove flicker of custom icon
-* Preserve PROVIDER setting from cloudron.conf
-* Add Skip backup option when updating an app
-* Fix bug where nginx was not reloaded on cert renewal
-
-[4.2.4]
-* Fix demo settings state regression
-
-[4.2.5]
-* Fix the demo settins fix
-
-[4.2.6]
-* Fix configuration of empty app location (subdomain)
-
-[4.2.7]
-* Fix issue where the icon for normal users was displayed incorrectly
-* Kill stuck backup processes after 12 hours and notify admins
-* Reconfigure email apps when mail domain is added/removed
-* Fix crash when only udp ports are defined
-
-[4.3.0]
-* Add timeout to kill long running tasks in case they get stuck
-* email: Auto-subscribe to Spam folder
-* Allow setting a custom CSP policy
-* ticket: when email is down, add a field to provide alternate contact email
-* Re-work app import flow
-* Add pagination and search to mailbox and mail alias listing
-* Add UI and workflow to add a private registry
-* Show external LDAP connector
-* Network view: Allow IP address detection to be configurable
-* Add support for custom docker registry
-* Resolve any lists and aliases in a mailing list
-* Rename Accounts view to Profile
-* Add search for groups and user association UI
-
-[4.3.1]
-* Make logout from all button logout from all sessions
-* List unstable apps by default
-* Fix crash when listing mailboxes
-
-[4.3.2]
-* Update manifestformat module
-
-[4.3.3]
-* Fix bug where stopped containers got started on server restart
-* Fix external LDAP UI and syncing
-* Fix timeout being too low in docker proxy
-* Make manifest.id optional for custom apps
-* Fix registry detection in private images
-* Make mailbox domain configurable for apps
-
-[4.3.4]
-* Do not error if fallback certs went missing
-* Add 'New Apps' section to Appstore view
-* Fix issue where graphs of some apps were not appearing
-
-[4.4.0]
-* Show swap in graphs
-* Make avatars customizable
-* Hide access tokens from logs
-* Add missing '@' sign for email address in app mailbox
-* Add app fqdn to backup progress message
-* import: add option to import app in-place
-* import: add option to import app from arbitrary backup config
-* Show download progress for rsync backups
-* Fix various repair workflows
-* acme2: Implement post-as-get
-
-[4.4.1]
-* ami: fix AWS provider validation
-
-[4.4.2]
-* Fix crash when reporting that DKIM is not setup correctly
-* Stopped apps cannot be updated or auto-updated
-* eventlog: track support ticket creation and remote support status
-
-[4.4.3]
-* Add restart button in recovery section
-* Fix issue where memory usage was not computed correctly
-* cloudflare: support API tokens
-
-[4.4.4]
-* Fix bug where restart button in terminal was not working
-* Add search field in apps view
-* Make app view tags and domain filter persistent
-* Add timezone UI
-
-[4.4.5]
-* Fix user listing regression in group edit dialog
-* Do not show error page for 503
-* Add mail list and mail box update events
-* Certs of stopped apps are not renewed anymore
-* Fix broken memory sliders in the services UI
-* Set CPU Shares
-* Update nodejs to 12.14.1
-* Update MySQL addon packet size to 64M
-
-[5.0.0]
-* Show backup disk usage in graphs
-* Add per-user app passwords
-* Make app not responding page customizable
-* Make footer customizable
-* Add UI to import backups
-* Display timestamps in browser timezone in the UI
-* Mail eventlog and usage
-* Add user roles - owner, admin, user manager and user
-* Setup logrotate configs for collectd since upstream does not set it up
-* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
-* linode: add object storage backend
-* restore: carefully replace backup config
-* spam: add default corpus and global db
-
-[5.0.1]
-* Show backup disk usage in graphs
-* Add per-user app passwords
-* Make app not responding page customizable
-* Make footer customizable
-* Add UI to import backups
-* Display timestamps in browser timezone in the UI
-* Mail eventlog and usage
-* Add user roles - owner, admin, user manager and user
-* Setup logrotate configs for collectd since upstream does not set it up
-* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
-* linode: add object storage backend
-* restore: carefully replace backup config
-* spam: add default corpus and global db
-
-[5.0.2]
-* Show backup disk usage in graphs
-* Add per-user app passwords
-* Make app not responding page customizable
-* Make footer customizable
-* Add UI to import backups
-* Display timestamps in browser timezone in the UI
-* Mail eventlog and usage
-* Add user roles - owner, admin, user manager and user
-* Setup logrotate configs for collectd since upstream does not set it up
-* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
-* linode: add object storage backend
-* restore: carefully replace backup config
-* spam: per mailbox bayes db and training
-
-[5.0.3]
-* Show backup disk usage in graphs
-* Add per-user app passwords
-* Make app not responding page customizable
-* Make footer customizable
-* Add UI to import backups
-* Display timestamps in browser timezone in the UI
-* Mail eventlog and usage
-* Add user roles - owner, admin, user manager and user
-* Setup logrotate configs for collectd since upstream does not set it up
-* mail: Add X-Envelope-To and X-Envelope-From headers for incoming mails
-* linode: add object storage backend
-* restore: carefully replace backup config
-* spam: per mailbox bayes db and training
-
-[5.0.4]
-* Fix potential previlige escalation because of ghost file
-* linode: dns backend
-* make branding routes owner only
-* add branding API
-* Add app start/stop/restart events
-* Use the primary email for LE account
-* make mail eventlog more descriptive
-
-[5.0.5]
-* Fix bug where incoming mail from dynamic hostnames was rejected
-* Increase token expiry
-* Fix bug in tag UI where tag removal did not work
-
-[5.0.6]
-* Make mail eventlog only visible to owners
-* Make app password work with sftp
-
-[5.1.0]
-* Add turn addon
-* Fix disk usage display
-* Drop support for TLSv1 and TLSv1.1
-* Make cert validation work for ECC certs
-* Add type filter to mail eventlog
-* mail: Fix listing of mailboxes and aliases in the UI
-* branding: fix login page title
-* Only a Cloudron owner can install/update/exec apps with the docker addon
-* security: reset tokens are only valid for a day
-* mail: fix eventlog db perms
-* Fix various bugs in the disk graphs
-
-[5.1.1]
-* Add turn addon
-* Fix disk usage display
-* Drop support for TLSv1 and TLSv1.1
-* Make cert validation work for ECC certs
-* Add type filter to mail eventlog
-* mail: Fix listing of mailboxes and aliases in the UI
-* branding: fix login page title
-* Only a Cloudron owner can install/update/exec apps with the docker addon
-* security: reset tokens are only valid for a day
-* mail: fix eventlog db perms
-* Fix various bugs in the disk graphs
-* Fix collectd installation
-* graphs: sort disk contents by usage
-* backups: show apps that are not automatically backed up in backup view
-
-[5.1.2]
-* Add turn addon
-* Fix disk usage display
-* Drop support for TLSv1 and TLSv1.1
-* Make cert validation work for ECC certs
-* Add type filter to mail eventlog
-* mail: Fix listing of mailboxes and aliases in the UI
-* branding: fix login page title
-* Only a Cloudron owner can install/update/exec apps with the docker addon
-* security: reset tokens are only valid for a day
-* mail: fix eventlog db perms
-* Fix various bugs in the disk graphs
-* Fix collectd installation
-* graphs: sort disk contents by usage
-* backups: show apps that are not automatically backed up in backup view
-* turn: deny local address peers https://www.rtcsec.com/2020/04/01-slack-webrtc-turn-compromise/
-
-[5.1.3]
-* Fix crash with misconfigured reverse proxy
-* Fix issue where invitation links are not working anymore
-
-[5.1.4]
-* Add support for custom .well-known documents to be served
-* Add ECDHE-RSA-AES128-SHA256 to cipher list
-* Fix GPG signature verification
-
-[5.1.5]
-* Check for .well-known routes upstream as fallback. This broke nextcloud's caldav/carddav
-
-[5.2.0]
-* acme: request ECC certs
-* less-strict DKIM check to allow users to set a stronger DKIM key
-* Add members only flag to mailing list
-* oauth: add backward compat layer for backup and uninstall
-* fix bug in disk usage sorting
-* mail: aliases can be across domains
-* mail: allow an external MX to be set
-* Add UI to download backup config as JSON (and import it)
-* Ensure stopped apps are getting backed up
-* Add OVH Object Storage backend
-* Add per-app redis status and configuration to Services
-* spam: large emails were not scanned
-* mail relay: fix delivery event log
-* manual update check always gets the latest updates
-* graphs: fix issue where large number of apps would crash the box code (query param limit exceeded)
-* backups: fix various security issues in encypted backups (thanks @mehdi)
-* graphs: add app graphs
-* older encrypted backups cannot be used in this version
-* Add backup listing UI
-* stopping an app will stop dependent services
-* Add new wasabi s3 storage region us-east-2
-* mail: Fix bug where SRS translation was done on the main domain instead of mailing list domain
-* backups: add retention policy
-* Drop `NET_RAW` caps from container preventing sniffing of network traffic
-
-[5.2.1]
-* Fix app disk graphs
-* restart apps on addon container change
-
-[5.2.2]
-* regression: import UI
-* Mbps -> MBps
-* Remove verbose logs
-* Set dmode in tar extract
-* mail: fix crash in audit logs
-* import: fix crash because encryption is unset
-* create redis with the correct label
-
-[5.2.3]
-* Do not restart stopped apps
-
-[5.2.4]
-* mail: enable/disable incoming mail was showing an error
-* Do not trigger backup of stopped apps. Instead, we will just retain it's existing backups
-  based on retention policy
-* remove broken disk graphs
-* fix OVH backups
-
-[5.3.0]
-* better nginx config for higher loads
-* backups: add CIFS storage provider
-* backups: add SSHFS storage provider
-* backups: add NFS storage provider
-* s3: use vhost style
-* Fix crash when redis config was set
-* Update schedule was unselected in the UI
-* cloudron-setup: --provider is now optional
-* show warning for unstable updates
-* add forumUrl to app manifest
-* postgresql: add unaccent extension for peertube
-* mail: Add Auto-Submitted header to NDRs
-* backups: ensure that the latest backup of installed apps is always preserved
-* add nginx logs
-* mail: make authentication case insensitive
-* Fix timeout issues in postgresql and mysql addon
-* Do not count stopped apps for memory use
-* LDAP group synchronization
-
-[5.3.1]
-* better nginx config for higher loads
-* backups: add CIFS storage provider
-* backups: add SSHFS storage provider
-* backups: add NFS storage provider
-* s3: use vhost style
-* Fix crash when redis config was set
-* Update schedule was unselected in the UI
-* cloudron-setup: --provider is now optional
-* show warning for unstable updates
-* add forumUrl to app manifest
-* postgresql: add unaccent extension for peertube
-* mail: Add Auto-Submitted header to NDRs
-* backups: ensure that the latest backup of installed apps is always preserved
-* add nginx logs
-* mail: make authentication case insensitive
-* Fix timeout issues in postgresql and mysql addon
-* Do not count stopped apps for memory use
-* LDAP group synchronization
-
-[5.3.2]
-* Do not install sshfs package
-* 'provider' is not required anymore in various API calls
-* redis: Set maxmemory and maxmemory-policy
-* Add mlock capability to manifest (for vault app)
-
-[5.3.3]
-* Fix issue where some postinstall messages where causing angular to infinite loop
-
-[5.3.4]
-* Fix issue in database error handling
-
-[5.4.0]
-* Update nginx to 1.18 for various security fixes
-* Add ping capability (for statping app)
-* Fix bug where aliases were displayed incorrectly in SOGo
-* Add univention as LDAP provider
-* Bump max_connection for postgres addon to 200
-* mail: Add pagination to mailing list API
-* Allow admin to lock email and display name of users
-* Allow admin to ensure all users have 2FA setup
-* ami: fix regression where we didn't send provider as part of get status call
-* nginx: hide version
-* backups: add b2 provider
-* Add filemanager webinterface
-* Add darkmode
-* Add note that password reset and invite links expire in 24 hours
-
-[5.4.1]
-* Update nginx to 1.18 for various security fixes
-* Add ping capability (for statping app)
-* Fix bug where aliases were displayed incorrectly in SOGo
-* Add univention as LDAP provider
-* Bump max_connection for postgres addon to 200
-* mail: Add pagination to mailing list API
-* Allow admin to lock email and display name of users
-* Allow admin to ensure all users have 2FA setup
-* ami: fix regression where we didn't send provider as part of get status call
-* nginx: hide version
-* backups: add b2 provider
-* Add filemanager webinterface
-* Add darkmode
-* Add note that password reset and invite links expire in 24 hours
-
-[5.5.0]
-* postgresql: update to PostgreSQL 11
-* postgresql: add citext extension to whitelist for loomio
-* postgresql: add btree_gist,postgres_fdw,pg_stat_statements,plpgsql extensions for gitlab
-* SFTP/Filebrowser: fix access of external data directories
-* Fix contrast issues in dark mode
-* Add option to delete mailbox data when mailbox is delete
-* Allow days/hours of backups and updates to be configurable
-* backup cleaner: fix issue where referenced backups where not counted against time periods
-* route53: fix issue where verification failed if user had more than 100 zones
-* rework task workers to run them in a separate cgroup
-* backups: now much faster thanks to reworking of task worker
-* When custom fallback cert is set, make sure it's used over LE certs
-* mongodb: update to MongoDB 4.0.19
-* List groups ordered by name
-* Invite links are now valid for a week
-* Update release GPG key
-* Add pre-defined variables ($CLOUDRON_APPID) for better post install messages
-* filemanager: show folder first
-
-[5.6.0]
-* Remove IP nginx configuration that redirects to dashboard after activation
-* dashboard: looks for search string in app title as well
-* Add vaapi caps for transcoding
-* Fix issue where the long mongodb database names where causing app indices of rocket.chat to overflow (> 127)
-* Do not resize swap if swap file exists. This means that users can now control how swap is allocated on their own.
-* SFTP: fix issue where parallel rebuilds would cause an error
-* backups: make part size configurable
-* mail: set max email size
-* mail: allow mail server location to be set
-* spamassassin: custom configs and wl/bl
-* Do not automatically update to unstable release
-* scheduler: reduce container churn
-* mail: add API to set banner
-* Fix bug where systemd 237 ignores --nice value in systemd-run
-* postgresql: enable uuid-ossp extension
-* firewall: add blocklist
-* HTTP URLs now redirect directly to the HTTPS of the final domain
-* linode: Add singapore region
-* ovh: add sydney region
-* s3: makes multi-part copies in parallel
-
-[5.6.1]
-* Blocklists are now stored in a text file instead of json
-* regenerate nginx configs
-
@@ -1,5 +1,5 @@
 The Cloudron Subscription license
-Copyright (c) 2020 Cloudron UG
+Copyright (c) 2019 Cloudron UG

 With regard to the Cloudron Software:

@@ -29,9 +29,9 @@ anyone to effortlessly host web applications on their server on their own terms.
 * Trivially migrate to another server keeping your apps and data (for example, switch your
  infrastructure provider or move to a bigger server).

-* Comprehensive [REST API](https://docs.cloudron.io/api/).
+* Comprehensive [REST API](https://cloudron.io/documentation/developer/api/).

-* [CLI](https://docs.cloudron.io/custom-apps/cli/) to configure apps.
+* [CLI](https://cloudron.io/documentation/cli/) to configure apps.

 * Alerts, audit logs, graphs, dns management ... and much more

@@ -41,15 +41,30 @@ Try our demo at https://my.demo.cloudron.io (username: cloudron password: cloudr

 ## Installing

-[Install script](https://docs.cloudron.io/installation/) - [Pricing](https://cloudron.io/pricing.html)
+You can install the Cloudron platform on your own server or get a managed server
+from cloudron.io. In either case, the Cloudron platform will keep your server and
+apps up-to-date and secure.
+
+* [Selfhosting](https://cloudron.io/documentation/installation/) - [Pricing](https://cloudron.io/pricing.html)
+* [Managed Hosting](https://cloudron.io/managed.html)

 **Note:** This repo is a small part of what gets installed on your server - there is
 the dashboard, database addons, graph container, base image etc. Cloudron also relies
 on external services such as the App Store for apps to be installed. As such, don't
 clone this repo and npm install and expect something to work.

-## Support
+## Documentation

-* [Documentation](https://docs.cloudron.io/)
+* [Documentation](https://cloudron.io/documentation/)
+
+## Related repos
+
+The [base image repo](https://git.cloudron.io/cloudron/docker-base-image) is the parent image of all
+the containers in the Cloudron.
+
+## Community
+
+* [Chat](https://chat.cloudron.io)
 * [Forum](https://forum.cloudron.io/)
+* [Support](mailto:support@cloudron.io)

@@ -4,7 +4,8 @@ set -euv -o pipefail

 readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

-readonly arg_infraversionpath="${SOURCE_DIR}/../src"
+readonly arg_provider="${1:-generic}"
+readonly arg_infraversionpath="${SOURCE_DIR}/${2:-}"

 function die {
    echo $1
@@ -13,9 +14,6 @@ function die {

 export DEBIAN_FRONTEND=noninteractive

-readonly ubuntu_codename=$(lsb_release -cs)
-readonly ubuntu_version=$(lsb_release -rs)
-
 # hold grub since updating it breaks on some VPS providers. also, dist-upgrade will trigger it
 apt-mark hold grub* >/dev/null
 apt-get -o Dpkg::Options::="--force-confdef" update -y
@@ -29,49 +27,40 @@ debconf-set-selections <<< 'mysql-server mysql-server/root_password_again passwo

 # this enables automatic security upgrades (https://help.ubuntu.com/community/AutomaticSecurityUpdates)
 # resolvconf is needed for unbound to work property after disabling systemd-resolved in 18.04
+ubuntu_version=$(lsb_release -rs)
+ubuntu_codename=$(lsb_release -cs)
 gpg_package=$([[ "${ubuntu_version}" == "16.04" ]] && echo "gnupg" || echo "gpg")
 apt-get -y install \
    acl \
    build-essential \
-    cifs-utils \
    cron \
    curl \
    debconf-utils \
    dmsetup \
    $gpg_package \
-    ipset \
    iptables \
    libpython2.7 \
-    linux-generic \
    logrotate \
    mysql-server-5.7 \
+    nginx-full \
    openssh-server \
    pwgen \
    resolvconf \
+    sudo \
    swaks \
-    tzdata \
    unattended-upgrades \
    unbound \
    xfsprogs

-echo "==> installing nginx for xenial for TLSv3 support"
-curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
-# apt install with install deps (as opposed to dpkg -i)
-apt install -y /tmp/nginx.deb
-rm /tmp/nginx.deb
-
-# on some providers like scaleway the sudo file is changed and we want to keep the old one
-apt-get -o Dpkg::Options::="--force-confold" install -y sudo
-
 # this ensures that unattended upgades are enabled, if it was disabled during ubuntu install time (see #346)
 # debconf-set-selection of unattended-upgrades/enable_auto_updates + dpkg-reconfigure does not work
 cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades

 echo "==> Installing node.js"
-mkdir -p /usr/local/node-10.18.1
-curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxf - --strip-components=1 -C /usr/local/node-10.18.1
-ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
+mkdir -p /usr/local/node-10.15.1
+curl -sL https://nodejs.org/dist/v10.15.1/node-v10.15.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.15.1
+ln -sf /usr/local/node-10.15.1/bin/node /usr/bin/node
+ln -sf /usr/local/node-10.15.1/bin/npm /usr/bin/npm
 apt-get install -y python   # Install python which is required for npm rebuild
 [[ "$(python --version 2>&1)" == "Python 2.7."* ]] || die "Expecting python version to be 2.7.x"

@@ -117,7 +106,7 @@ for image in ${images}; do
 done

 echo "==> Install collectd"
-if ! apt-get install -y libcurl3-gnutls collectd collectd-utils; then
+if ! apt-get install -y collectd collectd-utils; then
    # FQDNLookup is true in default debian config. The box code has a custom collectd.conf that fixes this
    echo "Failed to install collectd. Presumably because of http://mailman.verplant.org/pipermail/collectd/2015-March/006491.html"
    sed -e 's/^FQDNLookup true/FQDNLookup false/' -i /etc/collectd/collectd.conf
@@ -129,15 +118,6 @@ timedatectl set-ntp 1
 # mysql follows the system timezone
 timedatectl set-timezone UTC

-echo "==> Adding sshd configuration warning"
-sed -e '/Port 22/ i # NOTE: Cloudron only supports moving SSH to port 202. See https://docs.cloudron.io/security/#securing-ssh-access' -i /etc/ssh/sshd_config
-
-# https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1701068
-echo "==> Disabling motd news"
-if [ -f "/etc/default/motd-news" ]; then
-    sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
-fi
-
 # Disable bind for good measure (on online.net, kimsufi servers these are pre-installed and conflicts with unbound)
 systemctl stop bind9 || true
 systemctl disable bind9 || true
@@ -2,61 +2,68 @@

 'use strict';

+// prefix all output with a timestamp
+// debug() already prefixes and uses process.stderr NOT console.*
+['log', 'info', 'warn', 'debug', 'error'].forEach(function (log) {
+    var orig = console[log];
+    console[log] = function () {
+        orig.apply(console, [new Date().toISOString()].concat(Array.prototype.slice.call(arguments)));
+    };
+});
+
+require('supererror')({ splatchError: true });
+
 let async = require('async'),
-    dockerProxy = require('./src/dockerproxy.js'),
-    fs = require('fs'),
+    config = require('./src/config.js'),
    ldap = require('./src/ldap.js'),
-    paths = require('./src/paths.js'),
+    dockerProxy = require('./src/dockerproxy.js'),
    server = require('./src/server.js');

-const NOOP_CALLBACK = function () { };
-
-function setupLogging(callback) {
-    if (process.env.BOX_ENV === 'test') return callback();
-
-    var logfileStream = fs.createWriteStream(paths.BOX_LOG_FILE, { flags:'a' });
-    process.stdout.write = process.stderr.write = logfileStream.write.bind(logfileStream);
-
-    callback();
-}
+console.log();
+console.log('==========================================');
+console.log(' Cloudron will use the following settings ');
+console.log('==========================================');
+console.log();
+console.log(' Environment:                    ', config.CLOUDRON ? 'CLOUDRON' : 'TEST');
+console.log(' Version:                        ', config.version());
+console.log(' Admin Origin:                   ', config.adminOrigin());
+console.log(' Appstore API server origin:     ', config.apiServerOrigin());
+console.log(' Appstore Web server origin:     ', config.webServerOrigin());
+console.log(' SysAdmin Port:                  ', config.get('sysadminPort'));
+console.log(' LDAP Server Port:               ', config.get('ldapPort'));
+console.log(' Docker Proxy Port:              ', config.get('dockerProxyPort'));
+console.log();
+console.log('==========================================');
+console.log();

 async.series([
-    setupLogging,
    server.start,
    ldap.start,
    dockerProxy.start
 ], function (error) {
    if (error) {
-        console.log('Error starting server', error);
+        console.error('Error starting server', error);
        process.exit(1);
    }
-
-    // require those here so that logging handler is already setup
-    require('supererror');
-    const debug = require('debug')('box:box');
-
-    process.on('SIGINT', function () {
-        debug('Received SIGINT. Shutting down.');
-
-        server.stop(NOOP_CALLBACK);
-        ldap.stop(NOOP_CALLBACK);
-        dockerProxy.stop(NOOP_CALLBACK);
-        setTimeout(process.exit.bind(process), 3000);
-    });
-
-    process.on('SIGTERM', function () {
-        debug('Received SIGTERM. Shutting down.');
-
-        server.stop(NOOP_CALLBACK);
-        ldap.stop(NOOP_CALLBACK);
-        dockerProxy.stop(NOOP_CALLBACK);
-        setTimeout(process.exit.bind(process), 3000);
-    });
-
-    process.on('uncaughtException', function (error) {
-        console.error((error && error.stack) ? error.stack : error);
-        setTimeout(process.exit.bind(process, 1), 3000);
-    });
-
-    console.log(`Cloudron is up and running. Logs are at ${paths.BOX_LOG_FILE}`); // this goes to journalctl
+    console.log('Cloudron is up and running');
+});
+
+var NOOP_CALLBACK = function () { };
+
+process.on('SIGINT', function () {
+    console.log('Received SIGINT. Shutting down.');
+
+    server.stop(NOOP_CALLBACK);
+    ldap.stop(NOOP_CALLBACK);
+    dockerProxy.stop(NOOP_CALLBACK);
+    setTimeout(process.exit.bind(process), 3000);
+});
+
+process.on('SIGTERM', function () {
+    console.log('Received SIGTERM. Shutting down.');
+
+    server.stop(NOOP_CALLBACK);
+    ldap.stop(NOOP_CALLBACK);
+    dockerProxy.stop(NOOP_CALLBACK);
+    setTimeout(process.exit.bind(process), 3000);
 });
@@ -12,6 +12,8 @@ exports.up = function(db, callback) {
 			db.all('SELECT * FROM users WHERE admin=1', function (error, results) {
 				if (error) return done(error);

+				console.dir(results);
+
 				async.eachSeries(results, function (r, next) {
 					db.runSql('INSERT INTO groupMembers (groupId, userId) VALUES (?, ?)', [ ADMIN_GROUP_ID, r.id ], next);
 				}, done);
@@ -1,6 +1,12 @@
 'use strict';

-var async = require('async');
+var async = require('async'),
+    crypto = require('crypto'),
+    fs = require('fs'),
+    os = require('os'),
+    path = require('path'),
+    safe = require('safetydance'),
+    tldjs = require('tldjs');

 exports.up = function(db, callback) {
    db.all('SELECT * FROM apps, subdomains WHERE apps.id=subdomains.appId AND type="primary"', function (error, apps) {
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE mail ADD COLUMN dkimSelector VARCHAR(128) NOT NULL DEFAULT "cloudron"', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE mail DROP COLUMN dkimSelector', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,14 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps DROP FOREIGN KEY apps_owner_constraint'),
-        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN ownerId')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,29 +0,0 @@
-'use strict';
-
-var async = require('async'),
-    fs = require('fs');
-
-exports.up = function(db, callback) {
-    if (!fs.existsSync('/etc/cloudron/cloudron.conf')) {
-        console.log('Unable to locate cloudron.conf');
-        return callback();
-    }
-
-    const config = JSON.parse(fs.readFileSync('/etc/cloudron/cloudron.conf', 'utf8'));
-
-    async.series([
-        fs.writeFile.bind(null, '/etc/cloudron/PROVIDER', config.provider, 'utf8'),
-        db.runSql.bind(db, 'START TRANSACTION;'),
-        // we use replace instead of insert because the cloudron-setup adds api/web_server_origin even for legacy setups
-        db.runSql.bind(db, 'REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'api_server_origin', config.apiServerOrigin ]),
-        db.runSql.bind(db, 'REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'web_server_origin', config.webServerOrigin ]),
-        db.runSql.bind(db, 'REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'admin_domain', config.adminDomain ]),
-        db.runSql.bind(db, 'REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'admin_fqdn', config.adminFqdn ]),
-        db.runSql.bind(db, 'REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'demo', config.isDemo ]),
-        db.runSql.bind(db, 'COMMIT')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE users ADD COLUMN active BOOLEAN DEFAULT 1', function (error) {
-        if (error) return callback(error);
-
-        callback();
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN active', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -1,17 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps ADD COLUMN taskId INTEGER'),
-        db.runSql.bind(db, 'ALTER TABLE apps ADD CONSTRAINT apps_task_constraint FOREIGN KEY(taskId) REFERENCES tasks(id)')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE app DROP FOREIGN KEY apps_task_constraint'),
-        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN taskId'),
-    ], callback);
-};
@@ -1,12 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps DROP updateConfigJson, DROP restoreConfigJson, DROP oldConfigJson', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps CHANGE installationProgress errorJson TEXT', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps CHANGE errorJson installationProgress TEXT', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE users ADD COLUMN source VARCHAR(128) DEFAULT ""', function (error) {
-        if (error) return callback(error);
-
-        callback();
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN source', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -1,26 +0,0 @@
-'use strict';
-
-let async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE tasks CHANGE errorMessage errorJson TEXT', [], function (error) {
-        if (error) console.error(error);
-
-        // convert error messages into json
-        db.all('SELECT id, errorJson FROM apps', function (error, apps) {
-            async.eachSeries(apps, function (app, iteratorDone) {
-                if (app.errorJson === 'null') return iteratorDone();
-                if (app.errorJson === null) return iteratorDone();
-
-                db.runSql('UPDATE apps SET errorJson = ? WHERE id = ?', [ JSON.stringify({ message: app.errorJson }), app.id ], iteratorDone);
-            }, callback);
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE tasks CHANGE errorJson errorMessage TEXT', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,21 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-// imports mailbox entries for existing users
-exports.up = function(db, callback) {
-    db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
-        async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
-            if (!mailbox.membersJson) return iteratorDone();
-
-            let members = JSON.parse(mailbox.membersJson);
-            members = members.map((m) => m && m.indexOf('@') === -1 ? `${m}@${mailbox.domain}` : m); // only because we don't do things in a xction
-
-            db.runSql('UPDATE mailboxes SET membersJson=? WHERE name=? AND domain=?', [ JSON.stringify(members), mailbox.name, mailbox.domain ], iteratorDone);
-        }, callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,19 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('UPDATE apps SET runState=? WHERE runState IS NULL', [ 'running' ], function (error) {
-        if (error) return callback(error);
-
-        db.runSql('ALTER TABLE apps MODIFY runState VARCHAR(512) NOT NULL', [], function (error) {
-            if (error) console.error(error);
-            callback(error);
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE app MODIFY runState VARCHAR(512)', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,10 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    // We clear all demo state in the Cloudron...the demo cloudron needs manual intervention afterwards
-    db.runSql('REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'demo', '' ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,30 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps ADD COLUMN reverseProxyConfigJson TEXT', function (error) {
-        if (error) return callback(error);
-
-        db.all('SELECT id, robotsTxt FROM apps', function (error, apps) {
-            if (error) return callback(error);
-
-            async.eachSeries(apps, function (app, iteratorDone) {
-                if (!app.robotsTxt) return iteratorDone();
-
-                db.runSql('UPDATE apps SET reverseProxyConfigJson=? WHERE id=?', [ JSON.stringify({ robotsTxt: JSON.stringify(app.robotsTxt) }), app.id ], iteratorDone);
-            }, function (error) {
-                if (error) return callback(error);
-
-                db.runSql('ALTER TABLE apps DROP COLUMN robotsTxt', callback);
-            });
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN reverseProxyConfigJson'),
-    ], callback);
-};
-
@@ -1,13 +0,0 @@
-'use strict';
-
-var fs = require('fs');
-
-exports.up = function(db, callback) {
-    let sysinfoConfig = { provider: 'generic' };
-
-    db.runSql('REPLACE INTO settings (name, value) VALUES(?, ?)', [ 'sysinfo_config', JSON.stringify(sysinfoConfig) ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,27 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE apps ADD COLUMN mailboxDomain VARCHAR(128)'),
-        function setDefaultMailboxDomain(done) {
-            db.all('SELECT * FROM apps, subdomains WHERE apps.id=subdomains.appId AND type="primary"', function (error, apps) {
-                if (error) return done(error);
-
-                async.eachSeries(apps, function (app, iteratorDone) {
-                    db.runSql('UPDATE apps SET mailboxDomain=? WHERE id=?', [ app.domain, app.id ], iteratorDone);
-                }, done);
-            });
-        },
-        db.runSql.bind(db, 'ALTER TABLE apps MODIFY COLUMN mailboxDomain VARCHAR(128) NOT NULL'),
-        db.runSql.bind(db, 'ALTER TABLE apps ADD CONSTRAINT apps_mailDomain_constraint FOREIGN KEY(mailboxDomain) REFERENCES domains(domain)'),
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE app DROP FOREIGN KEY apps_mailDomain_constraint'),
-        db.runSql.bind(db, 'ALTER TABLE apps DROP COLUMN mailboxDomain'),
-    ], callback);
-};
@@ -1,22 +0,0 @@
-'use strict';
-
-let async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('SELECT * FROM domains', function (error, domains) {
-        if (error) return callback(error);
-
-        async.eachSeries(domains, function (domain, iteratorCallback) {
-            if (domain.provider !== 'cloudflare') return iteratorCallback();
-
-            let config = JSON.parse(domain.configJson);
-            config.tokenType = 'GlobalApiKey';
-
-            db.runSql('UPDATE domains SET configJson = ? WHERE domain = ?', [ JSON.stringify(config), domain.domain ], iteratorCallback);
-        }, callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps ADD COLUMN cpuShares INTEGER DEFAULT 512', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps DROP COLUMN cpuShares', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,26 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-	var cmd = 'CREATE TABLE appPasswords(' +
-                'id VARCHAR(128) NOT NULL UNIQUE,' +
-				'name VARCHAR(128) NOT NULL,' +
-                'userId VARCHAR(128) NOT NULL,' +
-                'identifier VARCHAR(128) NOT NULL,' +
-                'hashedPassword VARCHAR(1024) NOT NULL,' +
-				'creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,' +
-                'FOREIGN KEY(userId) REFERENCES users(id),' +
-                'UNIQUE (name, userId),' +
-				'PRIMARY KEY (id)) CHARACTER SET utf8 COLLATE utf8_bin';
-
-    db.runSql(cmd, function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('DROP TABLE appPasswords', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,22 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('DROP TABLE authcodes', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    var cmd = `CREATE TABLE IF NOT EXISTS authcodes(
-        authCode VARCHAR(128) NOT NULL UNIQUE,
-        userId VARCHAR(128) NOT NULL,
-        clientId VARCHAR(128) NOT NULL,
-        expiresAt BIGINT NOT NULL,
-        PRIMARY KEY(authCode)) CHARACTER SET utf8 COLLATE utf8_bin`;
-
-    db.runSql(cmd, function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,24 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('DROP TABLE clients', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    var cmd = `CREATE TABLE IF NOT EXISTS clients(
-      id VARCHAR(128) NOT NULL UNIQUE,
-      appId VARCHAR(128) NOT NULL,
-      type VARCHAR(16) NOT NULL,
-      clientSecret VARCHAR(512) NOT NULL,
-      redirectURI VARCHAR(512) NOT NULL,
-      scope VARCHAR(512) NOT NULL,
-      PRIMARY KEY(id)) CHARACTER SET utf8 COLLATE utf8_bin`;
-
-    db.runSql(cmd, function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE domains DROP COLUMN locked', function (error) {
-        if (error) return callback(error);
-
-        callback();
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE domains ADD COLUMN locked BOOLEAN DEFAULT 0', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -1,40 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'START TRANSACTION;'),
-        db.runSql.bind(db, 'ALTER TABLE users ADD COLUMN role VARCHAR(32)'),
-        function migrateAdminFlag(done) {
-            db.all('SELECT * FROM users ORDER BY createdAt', function (error, results) {
-                if (error) return done(error);
-                let ownerFound = false;
-
-                async.eachSeries(results, function (user, next) {
-                    let role;
-                    if (!ownerFound && user.admin) {
-                        role = 'owner';
-                        ownerFound = true;
-                        console.log(`Designating ${user.username} ${user.email} ${user.id} as the owner of this cloudron`);
-                    } else {
-                        role = user.admin ? 'admin' : 'user';
-                    }
-                    db.runSql('UPDATE users SET role=? WHERE id=?', [ role, user.id ], next);
-                }, done);
-            });
-        },
-        db.runSql.bind(db, 'ALTER TABLE users DROP COLUMN admin'),
-        db.runSql.bind(db, 'ALTER TABLE users MODIFY role VARCHAR(32) NOT NULL'),
-        db.runSql.bind(db, 'COMMIT')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN role', function (error) {
-        if (error) console.error(error);
-
-        callback(error);
-    });
-};
-
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE users ADD COLUMN resetTokenCreationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN resetTokenCreationTime', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,28 +0,0 @@
-'use strict';
-
-let async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps MODIFY mailboxDomain VARCHAR(128)', [], function (error) { // make it nullable
-        if (error) console.error(error);
-
-        // clear mailboxName/Domain for apps that do not use mail addons
-        db.all('SELECT * FROM apps', function (error, apps) {
-            if (error) return callback(error);
-
-            async.eachSeries(apps, function (app, iteratorDone) {
-                var manifest = JSON.parse(app.manifestJson);
-                if (manifest.addons['sendmail'] || manifest.addons['recvmail']) return iteratorDone();
-
-                db.runSql('UPDATE apps SET mailboxName=?, mailboxDomain=? WHERE id=?', [ null, null, app.id ], iteratorDone);
-            }, callback);
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps MODIFY manifestJson VARCHAR(128) NOT NULL', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE mailboxes ADD COLUMN membersOnly BOOLEAN DEFAULT 0', function (error) {
-        if (error) return callback(error);
-
-        callback();
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN membersOnly', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -1,28 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD COLUMN aliasDomain VARCHAR(128)'),
-        function setAliasDomain(done) {
-            db.all('SELECT * FROM mailboxes', function (error, mailboxes) {
-                async.eachSeries(mailboxes, function (mailbox, iteratorDone) {
-                    if (!mailbox.aliasTarget) return iteratorDone();
-
-                    db.runSql('UPDATE mailboxes SET aliasDomain=? WHERE name=? AND domain=?', [ mailbox.domain, mailbox.name, mailbox.domain ], iteratorDone);
-                }, done);
-            });
-        },
-        db.runSql.bind(db, 'ALTER TABLE mailboxes ADD CONSTRAINT mailboxes_aliasDomain_constraint FOREIGN KEY(aliasDomain) REFERENCES mail(domain)'),
-        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasTarget aliasName VARCHAR(128)')
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP FOREIGN KEY mailboxes_aliasDomain_constraint'),
-        db.runSql.bind(db, 'ALTER TABLE mailboxes DROP COLUMN aliasDomain'),
-        db.runSql.bind(db, 'ALTER TABLE mailboxes CHANGE aliasName aliasTarget VARCHAR(128)')
-    ], callback);
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps ADD COLUMN servicesConfigJson TEXT', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps DROP COLUMN servicesConfigJson', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE apps ADD COLUMN bindsJson TEXT', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE apps DROP COLUMN bindsJson', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,35 +0,0 @@
-'use strict';
-
-const backups = require('../src/backups.js'),
-    fs = require('fs');
-
-exports.up = function(db, callback) {
-    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
-        if (error || results.length === 0) return callback(error);
-
-        var backupConfig = JSON.parse(results[0].value);
-        if (backupConfig.key) {
-            backupConfig.encryption = backups.generateEncryptionKeysSync(backupConfig.key);
-            backups.cleanupCacheFilesSync();
-
-            fs.writeFileSync('/home/yellowtent/platformdata/BACKUP_PASSWORD',
-                'This file contains your Cloudron backup password.\nBefore Cloudron v5.2, this was saved in the database.' +
-                'From Cloudron 5.2, this password is not required anymore. We generate strong keys based off this password and use those keys to encrypt the backups.\n' +
-                'This means that the password is only required at decryption/restore time.\n\n' +
-                'This file can be safely removed and only exists for the off-chance that you do not remember your backup password.\n\n' +
-                `Password: ${backupConfig.key}\n`,
-                'utf8');
-
-        } else {
-            backupConfig.encryption = null;
-        }
-
-        delete backupConfig.key;
-
-        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE backups CHANGE version packageVersion VARCHAR(128) NOT NULL', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE backups CHANGE packageVersion version VARCHAR(128) NOT NULL', [], function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,24 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE backups ADD COLUMN encryptionVersion INTEGER', function (error) {
-        if (error) return callback(error);
-
-        db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
-            if (error || results.length === 0) return callback(error);
-
-            var backupConfig = JSON.parse(results[0].value);
-            if (!backupConfig.encryption) return callback(null);
-
-            // mark old encrypted backups as v1
-            db.runSql('UPDATE backups SET encryptionVersion=1', callback);
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE backups DROP COLUMN encryptionVersion', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,18 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
-        if (error || results.length === 0) return callback(error);
-
-        var backupConfig = JSON.parse(results[0].value);
-        backupConfig.retentionPolicy = { keepWithinSecs: backupConfig.retentionSecs };
-        delete backupConfig.retentionSecs;
-
-        // mark old encrypted backups as v1
-        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,18 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
-        if (error || results.length === 0) return callback(error);
-
-        var backupConfig = JSON.parse(results[0].value);
-        if (backupConfig.provider !== 'minio' && backupConfig.provider !== 's3-v4-compat') return callback();
-
-        backupConfig.s3ForcePathStyle = true; // usually minio is self-hosted. s3 v4 compat, we don't know
-
-        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-
-    // http://stackoverflow.com/questions/386294/what-is-the-maximum-length-of-a-valid-email-address
-
-    async.series([
-        db.runSql.bind(db, 'ALTER TABLE appPasswords DROP INDEX name'),
-        db.runSql.bind(db, 'ALTER TABLE appPasswords ADD CONSTRAINT appPasswords_name_userId_identifier UNIQUE (name, userId, identifier)'),
-    ], callback);
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,17 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE userGroups ADD COLUMN source VARCHAR(128) DEFAULT ""', function (error) {
-        if (error) return callback(error);
-
-        callback();
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE userGroups DROP COLUMN source', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
@@ -1,38 +0,0 @@
-'use strict';
-
-const async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE backups ADD COLUMN identifier VARCHAR(128)', function (error) {
-        if (error) return callback(error);
-
-
-        db.all('SELECT * FROM backups', function (error, backups) {
-            if (error) return callback(error);
-
-            async.eachSeries(backups, function (backup, next) {
-                let identifier = 'unknown';
-
-                if (backup.type === 'box') {
-                    identifier = 'box';
-                } else {
-                    const match = backup.id.match(/app_(.+?)_.+/);
-                    if (match) identifier = match[1];
-                }
-
-                db.runSql('UPDATE backups SET identifier=? WHERE id=?', [ identifier, backup.id ], next);
-            }, function (error) {
-                if (error) return callback(error);
-
-                db.runSql('ALTER TABLE backups MODIFY COLUMN identifier VARCHAR(128) NOT NULL', callback);
-            });
-        });
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE backups DROP COLUMN identifier', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,16 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE users ADD COLUMN ts TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP', function (error) {
-        if (error) console.error(error);
-
-        db.runSql('ALTER TABLE users DROP COLUMN modifiedAt', callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE users DROP COLUMN ts', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,29 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.all('SELECT value FROM settings WHERE name="backup_config"', function (error, results) {
-        if (error || results.length === 0) return callback(error);
-
-        var backupConfig = JSON.parse(results[0].value);
-        if (backupConfig.intervalSecs === 6 * 60 * 60) { // every 6 hours
-            backupConfig.schedulePattern = '00 00 5,11,17,23 * * *';
-        } else if (backupConfig.intervalSecs === 12 * 60 * 60) { // every 12 hours
-            backupConfig.schedulePattern = '00 00 5,17 * * *';
-        } else if (backupConfig.intervalSecs === 24 * 60 * 60) { // every day
-            backupConfig.schedulePattern = '00 00 23 * * *';
-        } else if (backupConfig.intervalSecs === 3 * 24 * 60 * 60) { // every 3 days (based on day)
-            backupConfig.schedulePattern = '00 00 23 * * 1,3,5';
-        } else if (backupConfig.intervalSecs === 7 * 24 * 60 * 60) { // every week (saturday)
-            backupConfig.schedulePattern = '00 00 23 * * 6';
-        } else { // default to everyday
-            backupConfig.schedulePattern = '00 00 23 * * *';
-        }
-
-        delete backupConfig.intervalSecs;
-        db.runSql('UPDATE settings SET value=? WHERE name="backup_config"', [ JSON.stringify(backupConfig) ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,23 +0,0 @@
-'use strict';
-
-const async = require('async');
-
-exports.up = function(db, callback) {
-    db.all('SELECT value FROM settings WHERE name="admin_domain"', function (error, results) {
-        if (error || results.length === 0) return callback(error);
-
-        const adminDomain = results[0].value;
-
-        async.series([
-            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_domain', adminDomain ]),
-            db.runSql.bind(db, 'INSERT INTO settings (name, value) VALUES (?, ?)', [ 'mail_fqdn', `my.${adminDomain}` ])
-        ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    async.series([
-        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_domain"'),
-        db.runSql.bind(db, 'DELETE FROM settings WHERE name="mail_fqdn"'),
-    ], callback);
-};
@@ -1,22 +0,0 @@
-'use strict';
-
-var async = require('async');
-
-exports.up = function(db, callback) {
-    db.runSql('SELECT * FROM settings WHERE name=?', ['app_autoupdate_pattern'], function (error, results) {
-        if (error || results.length === 0) return callback(error); // will use defaults from box code
-
-        var updatePattern = results[0].value; // use app auto update patter for the box as well
-
-        async.series([
-            db.runSql.bind(db, 'START TRANSACTION;'),
-            db.runSql.bind(db, 'DELETE FROM settings WHERE name=? OR name=?', ['app_autoupdate_pattern', 'box_autoupdate_pattern']),
-            db.runSql.bind(db, 'INSERT settings (name, value) VALUES(?, ?)', ['autoupdate_pattern', updatePattern]),
-            db.runSql.bind(db, 'COMMIT')
-        ], callback);
-    });
-};
-
-exports.down = function(db, callback) {
-    callback();
-};
@@ -1,15 +0,0 @@
-'use strict';
-
-exports.up = function(db, callback) {
-    db.runSql('ALTER TABLE mail ADD COLUMN bannerJson TEXT', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
-
-exports.down = function(db, callback) {
-    db.runSql('ALTER TABLE mail DROP COLUMN bannerJson', function (error) {
-        if (error) console.error(error);
-        callback(error);
-    });
-};
@@ -1,27 +0,0 @@
-'use strict';
-
-const OLD_FIREWALL_CONFIG_JSON = '/home/yellowtent/boxdata/firewall-config.json';
-const PORTS_FILE = '/home/yellowtent/boxdata/firewall/ports.json';
-const BLOCKLIST_FILE = '/home/yellowtent/boxdata/firewall/blocklist.txt';
-
-const fs = require('fs');
-
-exports.up = function (db, callback) {
-    if (!fs.existsSync(OLD_FIREWALL_CONFIG_JSON)) return callback();
-
-    try {
-        const dataJson = fs.readFileSync(OLD_FIREWALL_CONFIG_JSON, 'utf8');
-        const data = JSON.parse(dataJson);
-        fs.writeFileSync(BLOCKLIST_FILE, data.blocklist.join('\n') + '\n', 'utf8');
-        fs.writeFileSync(PORTS_FILE, JSON.stringify({ allowed_tcp_ports: data.allowed_tcp_ports }, null, 4), 'utf8');
-        fs.unlinkSync(OLD_FIREWALL_CONFIG_JSON);
-    } catch (error) {
-        console.log('Error migrating old firewall config', error);
-    }
-
-    callback();
-};
-
-exports.down = function (db, callback) {
-    callback();
-};
@@ -21,23 +21,18 @@ CREATE TABLE IF NOT EXISTS users(
    password VARCHAR(1024) NOT NULL,
    salt VARCHAR(512) NOT NULL,
    createdAt VARCHAR(512) NOT NULL,
-    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    modifiedAt VARCHAR(512) NOT NULL,
    displayName VARCHAR(512) DEFAULT "",
    fallbackEmail VARCHAR(512) DEFAULT "",
    twoFactorAuthenticationSecret VARCHAR(128) DEFAULT "",
    twoFactorAuthenticationEnabled BOOLEAN DEFAULT false,
-    source VARCHAR(128) DEFAULT "",
-    role VARCHAR(32),
-    resetToken VARCHAR(128) DEFAULT "",
-    resetTokenCreationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    active BOOLEAN DEFAULT 1,
+    admin BOOLEAN DEFAULT false,

    PRIMARY KEY(id));

 CREATE TABLE IF NOT EXISTS userGroups(
    id VARCHAR(128) NOT NULL UNIQUE,
    name VARCHAR(254) NOT NULL UNIQUE,
-    source VARCHAR(128) DEFAULT "",
    PRIMARY KEY(id));

 CREATE TABLE IF NOT EXISTS groupMembers(
@@ -56,40 +51,51 @@ CREATE TABLE IF NOT EXISTS tokens(
    expires BIGINT NOT NULL, // FIXME: make this a timestamp
    PRIMARY KEY(accessToken));

+CREATE TABLE IF NOT EXISTS clients(
+    id VARCHAR(128) NOT NULL UNIQUE, // prefixed with cid- to identify token easily in auth routes
+    appId VARCHAR(128) NOT NULL,     // name of the client (for external apps) or id of app (for built-in apps)
+    type VARCHAR(16) NOT NULL,
+    clientSecret VARCHAR(512) NOT NULL,
+    redirectURI VARCHAR(512) NOT NULL,
+    scope VARCHAR(512) NOT NULL,
+    PRIMARY KEY(id));
+
 CREATE TABLE IF NOT EXISTS apps(
    id VARCHAR(128) NOT NULL UNIQUE,
-    appStoreId VARCHAR(128) NOT NULL, // empty for custom apps
-    installationState VARCHAR(512) NOT NULL, // the active task on the app
-    runState VARCHAR(512) NOT NULL,   // if the app is stopped
+    appStoreId VARCHAR(128) NOT NULL,
+    installationState VARCHAR(512) NOT NULL,
+    installationProgress TEXT,
+    runState VARCHAR(512),
    health VARCHAR(128),
    healthTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app last responded
    containerId VARCHAR(128),
    manifestJson TEXT,
    httpPort INTEGER,                        // this is the nginx proxy port and not manifest.httpPort
+    location VARCHAR(128) NOT NULL,
+    domain VARCHAR(128) NOT NULL,
    accessRestrictionJson TEXT, // { users: [ ], groups: [ ] }
    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, // when the app was installed
    updateTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,   // when the last app update was done
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, // when this db record was updated (useful for UI caching)
    memoryLimit BIGINT DEFAULT 0,
-    cpuShares INTEGER DEFAULT 512,
    xFrameOptions VARCHAR(512),
    sso BOOLEAN DEFAULT 1, // whether user chose to enable SSO
    debugModeJson TEXT, // options for development mode
-    reverseProxyConfigJson TEXT, // { robotsTxt, csp }
+    robotsTxt TEXT,
    enableBackup BOOLEAN DEFAULT 1, // misnomer: controls automatic daily backups
    enableAutomaticUpdate BOOLEAN DEFAULT 1,
-    mailboxName VARCHAR(128), // mailbox of this app
-    mailboxDomain VARCHAR(128), // mailbox domain of this apps
+    mailboxName VARCHAR(128), // mailbox of this app. default allocated as '.app'
    label VARCHAR(128),     // display name
    tagsJson VARCHAR(2048), // array of tags
-    dataDir VARCHAR(256) UNIQUE,
-    taskId INTEGER, // current task
-    errorJson TEXT,
-    bindsJson TEXT, // bind mounts
-    servicesConfigJson TEXT, // app services configuration

-    FOREIGN KEY(mailboxDomain) REFERENCES domains(domain),
-    FOREIGN KEY(taskId) REFERENCES tasks(id),
+    // the following fields do not belong here, they can be removed when we use a queue for apptask
+    restoreConfigJson VARCHAR(256), // used to pass backupId to restore from to apptask
+    oldConfigJson TEXT, // used to pass old config to apptask (configure, restore)
+    updateConfigJson TEXT, // used to pass new config to apptask (update)
+
+    ownerId VARCHAR(128),
+
+    FOREIGN KEY(ownerId) REFERENCES users(id),
    PRIMARY KEY(id));

 CREATE TABLE IF NOT EXISTS appPortBindings(
@@ -100,6 +106,13 @@ CREATE TABLE IF NOT EXISTS appPortBindings(
    FOREIGN KEY(appId) REFERENCES apps(id),
    PRIMARY KEY(hostPort));

+CREATE TABLE IF NOT EXISTS authcodes(
+    authCode VARCHAR(128) NOT NULL UNIQUE,
+    userId VARCHAR(128) NOT NULL,
+    clientId VARCHAR(128) NOT NULL,
+    expiresAt BIGINT NOT NULL, // ## FIXME: make this a timestamp
+    PRIMARY KEY(authCode));
+
 CREATE TABLE IF NOT EXISTS settings(
    name VARCHAR(128) NOT NULL UNIQUE,
    value TEXT,
@@ -121,10 +134,8 @@ CREATE TABLE IF NOT EXISTS appEnvVars(
 CREATE TABLE IF NOT EXISTS backups(
    id VARCHAR(128) NOT NULL,
    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    packageVersion VARCHAR(128) NOT NULL, /* app version or box version */
-    encryptionVersion INTEGER, /* when null, unencrypted backup */
+    version VARCHAR(128) NOT NULL, /* app version or box version */
    type VARCHAR(16) NOT NULL, /* 'box' or 'app' */
-    identifier VARCHAR(128) NOT NULL, /* 'box' or the app id */
    dependsOn TEXT, /* comma separate list of objects this backup depends on */
    state VARCHAR(16) NOT NULL,
    manifestJson TEXT, /* to validate if the app can be installed in this version of box */
@@ -148,6 +159,7 @@ CREATE TABLE IF NOT EXISTS domains(
    provider VARCHAR(16) NOT NULL,
    configJson TEXT, /* JSON containing the dns backend provider config */
    tlsConfigJson TEXT, /* JSON containing the tls provider config */
+    locked BOOLEAN,

    PRIMARY KEY (domain))

@@ -161,9 +173,6 @@ CREATE TABLE IF NOT EXISTS mail(
    mailFromValidation BOOLEAN DEFAULT 1,
    catchAllJson TEXT,
    relayJson TEXT,
-    bannerJson TEXT,
-
-    dkimSelector VARCHAR(128) NOT NULL DEFAULT "cloudron",

    FOREIGN KEY(domain) REFERENCES domains(domain),
    PRIMARY KEY(domain))
@@ -181,22 +190,19 @@ CREATE TABLE IF NOT EXISTS mailboxes(
    name VARCHAR(128) NOT NULL,
    type VARCHAR(16) NOT NULL, /* 'mailbox', 'alias', 'list' */
    ownerId VARCHAR(128) NOT NULL, /* user id */
-    aliasName VARCHAR(128), /* the target name type is an alias */
-    aliasDomain VARCHAR(128), /* the target domain */
-    membersJson TEXT, /* members of a group. fully qualified */
-    membersOnly BOOLEAN DEFAULT false,
+    aliasTarget VARCHAR(128), /* the target name type is an alias */
+    membersJson TEXT, /* members of a group */
    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    domain VARCHAR(128),

    FOREIGN KEY(domain) REFERENCES mail(domain),
-    FOREIGN KEY(aliasDomain) REFERENCES mail(domain),
    UNIQUE (name, domain));

 CREATE TABLE IF NOT EXISTS subdomains(
    appId VARCHAR(128) NOT NULL,
    domain VARCHAR(128) NOT NULL,
    subdomain VARCHAR(128) NOT NULL,
-    type VARCHAR(128) NOT NULL, /* primary or redirect */
+    type VARCHAR(128) NOT NULL,

    FOREIGN KEY(domain) REFERENCES domains(domain),
    FOREIGN KEY(appId) REFERENCES apps(id),
@@ -207,8 +213,8 @@ CREATE TABLE IF NOT EXISTS tasks(
    type VARCHAR(32) NOT NULL,
    percent INTEGER DEFAULT 0,
    message TEXT,
-    errorJson TEXT,
-    resultJson TEXT,
+    errorMessage TEXT,
+    result TEXT,
    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (id));
@@ -221,18 +227,6 @@ CREATE TABLE IF NOT EXISTS notifications(
    message TEXT,
    acknowledged BOOLEAN DEFAULT false,
    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    UNIQUE KEY appPasswords_name_appId_identifier (name, userId, identifier),
-    PRIMARY KEY (id)
-);
-
-CREATE TABLE IF NOT EXISTS appPasswords(
-    id VARCHAR(128) NOT NULL UNIQUE,
-    name VARCHAR(128) NOT NULL,
-    userId VARCHAR(128) NOT NULL,
-    identifier VARCHAR(128) NOT NULL, // resourceId: app id or mail or webadmin
-    hashedPassword VARCHAR(1024) NOT NULL,
-    creationTime TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY(userId) REFERENCES users(id),

    PRIMARY KEY (id)
 );
@@ -14,73 +14,82 @@
    "node": ">=4.0.0 <=4.1.1"
  },
  "dependencies": {
-    "@google-cloud/dns": "^1.1.0",
+    "@google-cloud/dns": "^0.9.2",
    "@google-cloud/storage": "^2.5.0",
-    "@sindresorhus/df": "git+https://github.com/cloudron-io/df.git#type",
-    "async": "^2.6.3",
-    "aws-sdk": "^2.685.0",
-    "body-parser": "^1.19.0",
-    "cloudron-manifestformat": "^5.6.0",
-    "connect": "^3.7.0",
-    "connect-lastmile": "^2.0.0",
+    "@sindresorhus/df": "^3.1.0",
+    "async": "^2.6.2",
+    "aws-sdk": "^2.441.0",
+    "body-parser": "^1.18.3",
+    "cloudron-manifestformat": "^2.14.2",
+    "connect": "^3.6.6",
+    "connect-ensure-login": "^0.1.1",
+    "connect-lastmile": "^1.0.2",
    "connect-timeout": "^1.9.0",
-    "cookie-session": "^1.4.0",
-    "cron": "^1.8.2",
-    "db-migrate": "^0.11.11",
+    "cookie-parser": "^1.4.4",
+    "cookie-session": "^1.3.3",
+    "cron": "^1.7.0",
+    "csurf": "^1.9.0",
+    "db-migrate": "^0.11.5",
    "db-migrate-mysql": "^1.1.10",
    "debug": "^4.1.1",
    "dockerode": "^2.5.8",
    "ejs": "^2.6.1",
-    "ejs-cli": "^2.2.0",
-    "express": "^4.17.1",
-    "ipaddr.js": "^2.0.0",
-    "js-yaml": "^3.14.0",
+    "ejs-cli": "^2.0.1",
+    "express": "^4.16.4",
+    "express-session": "^1.16.1",
+    "js-yaml": "^3.13.1",
    "json": "^9.0.6",
    "ldapjs": "^1.0.2",
-    "lodash": "^4.17.15",
    "lodash.chunk": "^4.2.0",
-    "mime": "^2.4.6",
-    "moment": "^2.26.0",
-    "moment-timezone": "^0.5.31",
-    "morgan": "^1.10.0",
+    "mime": "^2.4.2",
+    "moment-timezone": "^0.5.25",
+    "morgan": "^1.9.1",
    "multiparty": "^4.2.1",
-    "mysql": "^2.18.1",
-    "nodemailer": "^6.4.6",
+    "mysql": "^2.17.1",
+    "namecheap": "github:joshuakarjala/node-namecheap#464a952",
+    "nodemailer": "^6.1.1",
    "nodemailer-smtp-transport": "^2.7.4",
+    "oauth2orize": "^1.11.0",
    "once": "^1.4.0",
-    "pretty-bytes": "^5.3.0",
+    "parse-links": "^0.1.0",
+    "passport": "^0.4.0",
+    "passport-http": "^0.3.0",
+    "passport-http-bearer": "^1.0.1",
+    "passport-local": "^1.0.0",
+    "passport-oauth2-client-password": "^0.1.2",
    "progress-stream": "^2.0.0",
    "proxy-middleware": "^0.15.0",
-    "qrcode": "^1.4.4",
-    "readdirp": "^3.4.0",
-    "request": "^2.88.2",
+    "qrcode": "^1.3.3",
+    "readdirp": "^3.0.0",
+    "request": "^2.88.0",
    "rimraf": "^2.6.3",
    "s3-block-read-stream": "^0.5.0",
-    "safetydance": "^1.1.1",
-    "semver": "^6.1.1",
-    "showdown": "^1.9.1",
+    "safetydance": "^0.7.1",
+    "semver": "^6.0.0",
+    "showdown": "^1.9.0",
    "speakeasy": "^2.0.0",
    "split": "^1.0.1",
-    "superagent": "^5.2.2",
+    "superagent": "^5.0.2",
    "supererror": "^0.7.2",
    "tar-fs": "github:cloudron-io/tar-fs#ignore_stat_error",
-    "tar-stream": "^2.1.2",
+    "tar-stream": "^2.0.1",
    "tldjs": "^2.3.1",
-    "underscore": "^1.10.2",
-    "uuid": "^3.4.0",
-    "validator": "^11.0.0",
-    "ws": "^7.3.0",
-    "xml2js": "^0.4.23"
+    "underscore": "^1.9.1",
+    "uuid": "^3.3.2",
+    "valid-url": "^1.0.9",
+    "validator": "^10.11.0",
+    "ws": "^6.2.1"
  },
  "devDependencies": {
    "expect.js": "*",
-    "hock": "^1.4.1",
-    "js2xmlparser": "^4.0.1",
+    "hock": "^1.3.3",
+    "js2xmlparser": "^4.0.0",
    "mocha": "^6.1.4",
    "mock-aws-s3": "git+https://github.com/cloudron-io/mock-aws-s3.git",
    "nock": "^10.0.6",
-    "node-sass": "^4.14.1",
-    "recursive-readdir": "^2.2.2"
+    "node-sass": "^4.11.0",
+    "recursive-readdir": "^2.2.2",
+    "sinon": "^7.3.2"
  },
  "scripts": {
    "test": "./runTests",
@@ -22,7 +22,7 @@ fi
 mkdir -p ${DATA_DIR}
 cd ${DATA_DIR}
 mkdir -p appsdata
-mkdir -p boxdata/profileicons boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com
+mkdir -p boxdata/appicons boxdata/mail boxdata/certs boxdata/mail/dkim/localhost boxdata/mail/dkim/foobar.com
 mkdir -p platformdata/addons/mail platformdata/nginx/cert platformdata/nginx/applications platformdata/collectd/collectd.conf.d platformdata/addons platformdata/logrotate.d platformdata/backup platformdata/logs/tasks

 # put cert
@@ -0,0 +1,106 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400 --http1.1"
+
+ip=""
+dns_config=""
+tls_cert_file=""
+tls_key_file=""
+license_file=""
+backup_config=""
+
+args=$(getopt -o "" -l "ip:,backup-config:,license:,dns-config:,tls-cert:,tls-key:" -n "$0" -- "$@")
+eval set -- "${args}"
+
+while true; do
+    case "$1" in
+    --ip) ip="$2"; shift 2;;
+    --dns-config) dns_config="$2"; shift 2;;
+    --tls-cert) tls_cert_file="$2"; shift 2;;
+    --tls-key) tls_key_file="$2"; shift 2;;
+    --license) license_file="$2"; shift 2;;
+    --backup-config) backup_config="$2"; shift 2;;
+    --) break;;
+    *) echo "Unknown option $1"; exit 1;;
+    esac
+done
+
+# validate arguments in the absence of data
+if [[ -z "${ip}" ]]; then
+    echo "--ip is required"
+    exit 1
+fi
+
+if [[ -z "${dns_config}" ]]; then
+    echo "--dns-config is required"
+    exit 1
+fi
+
+if [[ ! -f "${license_file}" ]]; then
+    echo "--license must be a valid license file"
+    exit 1
+fi
+
+function get_status() {
+    key="$1"
+    if status=$($curl -q -f -k "https://${ip}/api/v1/cloudron/status" 2>/dev/null); then
+        currentValue=$(echo "${status}" | python3 -c 'import sys, json; print(json.dumps(json.load(sys.stdin)[sys.argv[1]]))' "${key}")
+        echo "${currentValue}"
+        return 0
+    fi
+
+    return 1
+}
+
+function wait_for_status() {
+    key="$1"
+    expectedValue="$2"
+
+    echo "wait_for_status: $key to be $expectedValue"
+    while true; do
+        if currentValue=$(get_status "${key}"); then
+            echo "wait_for_status: $key is current: $currentValue expecting: $expectedValue"
+            if [[ "${currentValue}" == $expectedValue ]]; then
+                break
+            fi
+        fi
+        sleep 3
+    done
+}
+
+echo "=> Waiting for cloudron to be ready"
+wait_for_status "version" '*'
+
+domain=$(echo "${dns_config}" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["domain"])')
+
+echo "Provisioning Cloudron ${domain}"
+if [[ -n "${tls_cert_file}" && -n "${tls_key_file}" ]]; then
+    tls_cert=$(cat "${tls_cert_file}" | awk '{printf "%s\\n", $0}')
+    tls_key=$(cat "${tls_key_file}" | awk '{printf "%s\\n", $0}')
+    fallback_cert=$(printf '{ "cert": "%s", "key": "%s", "provider": "fallback", "restricted": true }' "${tls_cert}" "${tls_key}")
+else
+    fallback_cert=None
+fi
+
+tls_config='{ "provider": "fallback" }'
+dns_config=$(echo "${dns_config}" | python3 -c "import json,sys;obj=json.load(sys.stdin);obj.update(tlsConfig=${tls_config});obj.update(fallbackCertficate=${fallback_cert});print(json.dumps(obj))")
+
+license=$(cat "${license_file}")
+
+if [[ -z "${backup_config:-}" ]]; then
+    backup_config='{ "provider": "filesystem", "backupFolder": "/var/backups", "format": "tgz" }'
+fi
+
+setupData=$(printf '{ "dnsConfig": %s, "autoconf": { "appstoreConfig": %s, "backupConfig": %s } }' "${dns_config}" "${license}" "${backup_config}")
+
+if ! setupResult=$($curl -kq -X POST -H "Content-Type: application/json" -d "${setupData}" https://${ip}/api/v1/cloudron/setup); then
+    echo "Failed to setup with ${setupData} ${setupResult}"
+    exit 1
+fi
+
+wait_for_status "webadminStatus" '*"tls": true*'
+
+echo "Cloudron is ready at https://my-${domain}"
+
@@ -41,19 +41,21 @@ if systemctl -q is-active box; then
 fi

 initBaseImage="true"
-provider="generic"
+# provisioning data
+provider=""
 requestedVersion=""
 apiServerOrigin="https://api.cloudron.io"
 webServerOrigin="https://cloudron.io"
 sourceTarballUrl=""
 rebootServer="true"
+license=""

-args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,skip-baseimage-init,provider:,version:,env:,skip-reboot,license:" -n "$0" -- "$@")
 eval set -- "${args}"

 while true; do
    case "$1" in
-    --help) echo "See https://docs.cloudron.io/installation/ on how to install Cloudron"; exit 0;;
+    --help) echo "See https://cloudron.io/documentation/installation/ on how to install Cloudron"; exit 0;;
    --provider) provider="$2"; shift 2;;
    --version) requestedVersion="$2"; shift 2;;
    --env)
@@ -65,6 +67,7 @@ while true; do
            webServerOrigin="https://staging.cloudron.io"
        fi
        shift 2;;
+    --license) license="$2"; shift 2;;
    --skip-baseimage-init) initBaseImage="false"; shift;;
    --skip-reboot) rebootServer="false"; shift;;
    --) break;;
@@ -88,6 +91,41 @@ fi
 # Can only write after we have confirmed script has root access
 echo "Running cloudron-setup with args : $@" > "${LOG_FILE}"

+# validate arguments in the absence of data
+if [[ -z "${provider}" ]]; then
+    echo "--provider is required (azure, contabo, digitalocean, ec2, exoscale, galaxygate, gce, hetzner, lightsail, linode, netcup, ovh, rosehosting, scaleway, upcloud, vultr or generic)"
+    exit 1
+elif [[  \
+            "${provider}" != "ami" && \
+            "${provider}" != "azure" && \
+            "${provider}" != "caas" && \
+            "${provider}" != "cloudscale" && \
+            "${provider}" != "contabo" && \
+            "${provider}" != "digitalocean" && \
+            "${provider}" != "digitalocean-mp" && \
+            "${provider}" != "ec2" && \
+            "${provider}" != "exoscale" && \
+            "${provider}" != "galaxygate" && \
+            "${provider}" != "digitalocean" && \
+            "${provider}" != "gce" && \
+            "${provider}" != "hetzner" && \
+            "${provider}" != "lightsail" && \
+            "${provider}" != "linode" && \
+            "${provider}" != "linode-stackscript" && \
+            "${provider}" != "netcup" && \
+            "${provider}" != "netcup-image" && \
+            "${provider}" != "ovh" && \
+            "${provider}" != "rosehosting" && \
+            "${provider}" != "scaleway" && \
+            "${provider}" != "upcloud" && \
+            "${provider}" != "upcloud-image" && \
+            "${provider}" != "vultr" && \
+            "${provider}" != "generic" \
+        ]]; then
+    echo "--provider must be one of: azure, cloudscale.ch, contabo, digitalocean, ec2, exoscale, galaxygate, gce, hetzner, lightsail, linode, netcup, ovh, rosehosting, scaleway, upcloud, vultr or generic"
+    exit 1
+fi
+
 echo ""
 echo "##############################################"
 echo "         Cloudron Setup (${requestedVersion:-latest})"
@@ -106,13 +144,19 @@ if [[ "${initBaseImage}" == "true" ]]; then
        exit 1
    fi

+    echo "=> Ensure required apt sources"
+    if ! add-apt-repository universe &>> "${LOG_FILE}"; then
+        echo "Could not add required apt sources (for nginx-full). See ${LOG_FILE}"
+        exit 1
+    fi
+
    echo "=> Updating apt and installing script dependencies"
    if ! apt-get update &>> "${LOG_FILE}"; then
        echo "Could not update package repositories. See ${LOG_FILE}"
        exit 1
    fi

-    if ! DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install curl python3 ubuntu-standard -y &>> "${LOG_FILE}"; then
+    if ! apt-get install curl python3 ubuntu-standard -y &>> "${LOG_FILE}"; then
        echo "Could not install setup dependencies (curl). See ${LOG_FILE}"
        exit 1
    fi
@@ -145,40 +189,41 @@ fi

 if [[ "${initBaseImage}" == "true" ]]; then
    echo -n "=> Installing base dependencies and downloading docker images (this takes some time) ..."
-    # initializeBaseUbuntuImage.sh args (provider, infraversion path) are only to support installation of pre 5.3 Cloudrons
-    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "generic" "../src" &>> "${LOG_FILE}"; then
+    if ! /bin/bash "${box_src_tmp_dir}/baseimage/initializeBaseUbuntuImage.sh" "${provider}" "../src" &>> "${LOG_FILE}"; then
        echo "Init script failed. See ${LOG_FILE} for details"
        exit 1
    fi
    echo ""
 fi

-# The provider flag is still used for marketplace images
+# NOTE: this install script only supports 3.x and above
 echo "=> Installing version ${version} (this takes some time) ..."
 mkdir -p /etc/cloudron
-echo "${provider}" > /etc/cloudron/PROVIDER
+cat > "/etc/cloudron/cloudron.conf" <<CONF_END
+{
+    "apiServerOrigin": "${apiServerOrigin}",
+    "webServerOrigin": "${webServerOrigin}",
+    "provider": "${provider}"
+}
+CONF_END
+
+[[ -n "${license}" ]] && echo -n "$license" > /etc/cloudron/LICENSE

 if ! /bin/bash "${box_src_tmp_dir}/scripts/installer.sh" &>> "${LOG_FILE}"; then
    echo "Failed to install cloudron. See ${LOG_FILE} for details"
    exit 1
 fi

-mysql -uroot -ppassword -e "REPLACE INTO box.settings (name, value) VALUES ('api_server_origin', '${apiServerOrigin}');" 2>/dev/null
-mysql -uroot -ppassword -e "REPLACE INTO box.settings (name, value) VALUES ('web_server_origin', '${webServerOrigin}');" 2>/dev/null
-
 echo -n "=> Waiting for cloudron to be ready (this takes some time) ..."
 while true; do
    echo -n "."
-    if status=$($curl -s -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
+    if status=$($curl -q -f "http://localhost:3000/api/v1/cloudron/status" 2>/dev/null); then
        break # we are up and running
    fi
    sleep 10
 done

-if ! ip=$(curl -s --fail --connect-timeout 2 --max-time 2 https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
-    ip='<IP>'
-fi
-echo -e "\n\n${GREEN}Visit https://${ip} and accept the self-signed certificate to finish setup.${DONE}\n"
+echo -e "\n\n${GREEN}Visit https://<IP> and accept the self-signed certificate to finish setup.${DONE}\n"

 if [[ "${rebootServer}" == "true" ]]; then
    systemctl stop box mysql # sometimes mysql ends up having corrupt privilege tables
@@ -1,7 +1,5 @@
 #!/bin/bash

-set -eu -o pipefail
-
 # This script collects diagnostic information to help debug server related issues
 # It also enables SSH access for the cloudron support team

@@ -13,38 +11,25 @@ HELP_MESSAGE="
 This script collects diagnostic information to help debug server related issues

 Options:
-   --owner-login   Login as owner
   --enable-ssh    Enable SSH access for the Cloudron support team
   --help          Show this message
 "

 # We require root
 if [[ ${EUID} -ne 0 ]]; then
-    echo "This script should be run as root. Run with sudo"
+    echo "This script should be run as root." > /dev/stderr
    exit 1
 fi

 enableSSH="false"

-args=$(getopt -o "" -l "help,enable-ssh,admin-login,owner-login" -n "$0" -- "$@")
+args=$(getopt -o "" -l "help,enable-ssh" -n "$0" -- "$@")
 eval set -- "${args}"

 while true; do
    case "$1" in
    --help) echo -e "${HELP_MESSAGE}"; exit 0;;
    --enable-ssh) enableSSH="true"; shift;;
-    --admin-login)
-        # fall through
-        ;&
-    --owner-login)
-        admin_username=$(mysql -NB -uroot -ppassword -e "SELECT username FROM box.users WHERE role='owner' LIMIT 1" 2>/dev/null)
-        admin_password=$(pwgen -1s 12)
-        ghost_file=/home/yellowtent/platformdata/cloudron_ghost.json
-        printf '{"%s":"%s"}\n' "${admin_username}" "${admin_password}" > "${ghost_file}"
-        chown yellowtent:yellowtent "${ghost_file}" && chmod o-r,g-r "${ghost_file}"
-        echo "Login as ${admin_username} / ${admin_password} . Remove ${ghost_file} when done."
-        exit 0
-        ;;
    --) break;;
    *) echo "Unknown option $1"; exit 1;;
    esac
@@ -57,7 +42,7 @@ if [[ "`df --output="avail" / | sed -n 2p`" -lt "10240" ]]; then
    echo ""
    df -h
    echo ""
-    echo "To recover from a full disk, follow the guide at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full"
+    echo "To recover from a full disk, follow the guide at https://cloudron.io/documentation/server/#recovery-after-disk-full"
    exit 1
 fi

@@ -73,8 +58,23 @@ echo -n "Generating Cloudron Support stats..."
 # clear file
 rm -rf $OUT

-echo -e $LINE"PROVIDER"$LINE >> $OUT
-cat /etc/cloudron/PROVIDER &>> $OUT || true
+ssh_port=$(cat /etc/ssh/sshd_config | grep "Port " | sed -e "s/.*Port //")
+if [[ $SUDO_USER == "" ]]; then
+    ssh_user="root"
+    ssh_folder="/root/.ssh/"
+    authorized_key_file="${ssh_folder}/authorized_keys"
+else
+    ssh_user="$SUDO_USER"
+    ssh_folder="/home/$SUDO_USER/.ssh/"
+    authorized_key_file="${ssh_folder}/authorized_keys"
+fi
+
+echo -e $LINE"SSH"$LINE >> $OUT
+echo "Username: ${ssh_user}" >> $OUT
+echo "Port:     ${ssh_port}" >> $OUT
+
+echo -e $LINE"cloudron.conf"$LINE >> $OUT
+cat /etc/cloudron/cloudron.conf &>> $OUT

 echo -e $LINE"Docker container"$LINE >> $OUT
 if ! timeout --kill-after 10s 15s docker ps -a &>> $OUT 2>&1; then
@@ -85,64 +85,39 @@ echo -e $LINE"Filesystem stats"$LINE >> $OUT
 df -h &>> $OUT

 echo -e $LINE"Appsdata stats"$LINE >> $OUT
-du -hcsL /home/yellowtent/appsdata/* &>> $OUT || true
+du -hcsL /home/yellowtent/appsdata/* &>> $OUT

 echo -e $LINE"Boxdata stats"$LINE >> $OUT
 du -hcsL /home/yellowtent/boxdata/* &>> $OUT

 echo -e $LINE"Backup stats (possibly misleading)"$LINE >> $OUT
-du -hcsL /var/backups/* &>> $OUT || true
+du -hcsL /var/backups/* &>> $OUT

 echo -e $LINE"System daemon status"$LINE >> $OUT
-systemctl status --lines=100 box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT
+systemctl status --lines=100 cloudron.target box mysql unbound cloudron-syslog nginx collectd docker &>> $OUT

 echo -e $LINE"Box logs"$LINE >> $OUT
 tail -n 100 /home/yellowtent/platformdata/logs/box.log &>> $OUT

-echo -e $LINE"Firewall chains"$LINE >> $OUT
-ip addr &>> $OUT
-
 echo -e $LINE"Firewall chains"$LINE >> $OUT
 iptables -L &>> $OUT

 echo "Done"

-if [[ "${enableSSH}" == "true" ]]; then
-    ssh_port=$(cat /etc/ssh/sshd_config | grep "Port " | sed -e "s/.*Port //")
-    permit_root_login=$(grep -q ^PermitRootLogin.*yes /etc/ssh/sshd_config && echo "yes" || echo "no")
-
-    # support.js uses similar logic
-    if [[ -d /home/ubuntu ]]; then
-        ssh_user="ubuntu"
-        keys_file="/home/ubuntu/.ssh/authorized_keys"
-    else
-        ssh_user="root"
-        keys_file="/root/.ssh/authorized_keys"
-    fi
-
-    echo -e $LINE"SSH"$LINE >> $OUT
-    echo "Username: ${ssh_user}" >> $OUT
-    echo "Port:     ${ssh_port}" >> $OUT
-    echo "PermitRootLogin: ${permit_root_login}" >> $OUT
-    echo "Key file: ${keys_file}" >> $OUT
-
-    echo -n "Enabling ssh access for the Cloudron support team..."
-    mkdir -p $(dirname "${keys_file}")       # .ssh does not exist sometimes
-    touch "${keys_file}"                # required for concat to work
-    if ! grep -q "${CLOUDRON_SUPPORT_PUBLIC_KEY}" "${keys_file}"; then
-        echo -e "\n${CLOUDRON_SUPPORT_PUBLIC_KEY}" >> "${keys_file}"
-        chmod 600 "${keys_file}"
-        chown "${ssh_user}" "${keys_file}"
-    fi
-
-    echo "Done"
-fi
-
 echo -n "Uploading information..."
 # for some reason not using $(cat $OUT) will not contain newlines!?
 paste_key=$(curl -X POST ${PASTEBIN}/documents --silent -d "$(cat $OUT)" | python3 -c "import sys, json; print(json.load(sys.stdin)['key'])")
 echo "Done"

+if [[ "${enableSSH}" == "true" ]]; then
+    echo -n "Enabling ssh access for the Cloudron support team..."
+    mkdir -p "${ssh_folder}"
+    echo -e "\n${CLOUDRON_SUPPORT_PUBLIC_KEY}" >> ${authorized_key_file}
+    chown -R ${ssh_user} "${ssh_folder}"
+    chmod 600 "${authorized_key_file}"
+    echo "Done"
+fi
+
 echo ""
 echo "Please email the following link to support@cloudron.io"
 echo ""
@@ -41,8 +41,8 @@ if ! $(cd "${SOURCE_DIR}/../dashboard" && git diff --exit-code >/dev/null); then
    exit 1
 fi

-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    echo "This script requires node 10.18.1"
+if [[ "$(node --version)" != "v10.15.1" ]]; then
+    echo "This script requires node 10.15.1"
    exit 1
 fi

@@ -11,8 +11,9 @@ if [[ ${EUID} -ne 0 ]]; then
    exit 1
 fi

-readonly user=yellowtent
-readonly box_src_dir=/home/${user}/box
+readonly USER=yellowtent
+readonly BOX_SRC_DIR=/home/${USER}/box
+readonly BASE_DATA_DIR=/home/${USER}

 readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
 readonly script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -23,8 +24,6 @@ readonly ubuntu_codename=$(lsb_release -cs)

 readonly is_update=$(systemctl is-active box && echo "yes" || echo "no")

-echo "==> installer: Updating from $(cat $box_src_dir/VERSION) to $(cat $box_src_tmp_dir/VERSION) <=="
-
 echo "==> installer: updating docker"

 if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
@@ -57,27 +56,13 @@ if [[ $(docker version --format {{.Client.Version}}) != "18.09.2" ]]; then
    rm /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
 fi

-readonly nginx_version=$(nginx -v 2>&1)
-if [[ "${nginx_version}" != *"1.18."* ]]; then
-    echo "==> installer: installing nginx 1.18"
-    curl -sL http://nginx.org/packages/ubuntu/pool/nginx/n/nginx/nginx_1.18.0-1~${ubuntu_codename}_amd64.deb -o /tmp/nginx.deb
-    # apt install with install deps (as opposed to dpkg -i)
-    apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes /tmp/nginx.deb
-    rm /tmp/nginx.deb
-fi
-
-if ! which ipset; then
-    echo "==> installer: installing ipset"
-    apt install -y ipset
-fi
-
 echo "==> installer: updating node"
-if [[ "$(node --version)" != "v10.18.1" ]]; then
-    mkdir -p /usr/local/node-10.18.1
-    $curl -sL https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.18.1
-    ln -sf /usr/local/node-10.18.1/bin/node /usr/bin/node
-    ln -sf /usr/local/node-10.18.1/bin/npm /usr/bin/npm
-    rm -rf /usr/local/node-10.15.1
+if [[ "$(node --version)" != "v10.15.1" ]]; then
+    mkdir -p /usr/local/node-10.15.1
+    $curl -sL https://nodejs.org/dist/v10.15.1/node-v10.15.1-linux-x64.tar.gz | tar zxvf - --strip-components=1 -C /usr/local/node-10.15.1
+    ln -sf /usr/local/node-10.15.1/bin/node /usr/bin/node
+    ln -sf /usr/local/node-10.15.1/bin/npm /usr/bin/npm
+    rm -rf /usr/local/node-8.11.2 /usr/local/node-8.9.3
 fi

 # this is here (and not in updater.js) because rebuild requires the above node
@@ -124,22 +109,22 @@ while [[ ! -f "${CLOUDRON_SYSLOG}" || "$(${CLOUDRON_SYSLOG} --version)" != ${CLO
    sleep 5
 done

-if ! id "${user}" 2>/dev/null; then
-    useradd "${user}" -m
+if ! id "${USER}" 2>/dev/null; then
+    useradd "${USER}" -m
 fi

 if [[ "${is_update}" == "yes" ]]; then
-    echo "==> installer: stop box service for update"
-    ${box_src_dir}/setup/stop.sh
+    echo "==> installer: stop cloudron.target service for update"
+    ${BOX_SRC_DIR}/setup/stop.sh
 fi

 # ensure we are not inside the source directory, which we will remove now
 cd /root

 echo "==> installer: switching the box code"
-rm -rf "${box_src_dir}"
-mv "${box_src_tmp_dir}" "${box_src_dir}"
-chown -R "${user}:${user}" "${box_src_dir}"
+rm -rf "${BOX_SRC_DIR}"
+mv "${box_src_tmp_dir}" "${BOX_SRC_DIR}"
+chown -R "${USER}:${USER}" "${BOX_SRC_DIR}"

 echo "==> installer: calling box setup script"
-"${box_src_dir}/setup/start.sh"
+"${BOX_SRC_DIR}/setup/start.sh"
@@ -20,11 +20,6 @@ readonly ubuntu_version=$(lsb_release -rs)

 cp -f "${script_dir}/../scripts/cloudron-support" /usr/bin/cloudron-support

-# this needs to match the cloudron/base:2.0.0 gid
-if ! getent group media; then
-    addgroup --gid 500 --system media
-fi
-
 echo "==> Configuring docker"
 cp "${script_dir}/start/docker-cloudron-app.apparmor" /etc/apparmor.d/docker-cloudron-app
 systemctl enable apparmor
@@ -44,7 +39,7 @@ mkdir -p "${PLATFORM_DATA_DIR}/mysql"
 mkdir -p "${PLATFORM_DATA_DIR}/postgresql"
 mkdir -p "${PLATFORM_DATA_DIR}/mongodb"
 mkdir -p "${PLATFORM_DATA_DIR}/redis"
-mkdir -p "${PLATFORM_DATA_DIR}/addons/mail/banner"
+mkdir -p "${PLATFORM_DATA_DIR}/addons/mail"
 mkdir -p "${PLATFORM_DATA_DIR}/collectd/collectd.conf.d"
 mkdir -p "${PLATFORM_DATA_DIR}/logrotate.d"
 mkdir -p "${PLATFORM_DATA_DIR}/acme"
@@ -52,17 +47,13 @@ mkdir -p "${PLATFORM_DATA_DIR}/backup"
 mkdir -p "${PLATFORM_DATA_DIR}/logs/backup" \
         "${PLATFORM_DATA_DIR}/logs/updater" \
         "${PLATFORM_DATA_DIR}/logs/tasks" \
-         "${PLATFORM_DATA_DIR}/logs/crash" \
-         "${PLATFORM_DATA_DIR}/logs/collectd"
+         "${PLATFORM_DATA_DIR}/logs/crash"
 mkdir -p "${PLATFORM_DATA_DIR}/update"

 mkdir -p "${BOX_DATA_DIR}/appicons"
-mkdir -p "${BOX_DATA_DIR}/firewall"
-mkdir -p "${BOX_DATA_DIR}/profileicons"
 mkdir -p "${BOX_DATA_DIR}/certs"
 mkdir -p "${BOX_DATA_DIR}/acme" # acme keys
 mkdir -p "${BOX_DATA_DIR}/mail/dkim"
-mkdir -p "${BOX_DATA_DIR}/well-known" # .well-known documents

 # ensure backups folder exists and is writeable
 mkdir -p /var/backups
@@ -86,28 +77,23 @@ systemctl daemon-reload
 systemctl restart systemd-journald
 setfacl -n -m u:${USER}:r /var/log/journal/*/system.journal

-# Give user access to nginx logs (uses adm group)
-usermod -a -G adm ${USER}
-
 echo "==> Setting up unbound"
 # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org)
 # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)
 # We listen on 0.0.0.0 because there is no way control ordering of docker (which creates the 172.18.0.0/16) and unbound
 # If IP6 is not enabled, dns queries seem to fail on some hosts. -s returns false if file missing or 0 size
 ip6=$([[ -s /proc/net/if_inet6 ]] && echo "yes" || echo "no")
-cp -f "${script_dir}/start/unbound.conf" /etc/unbound/unbound.conf.d/cloudron-network.conf
+echo -e "server:\n\tinterface: 0.0.0.0\n\tdo-ip6: ${ip6}\n\taccess-control: 127.0.0.1 allow\n\taccess-control: 172.18.0.1/16 allow\n\tcache-max-negative-ttl: 30\n\tcache-max-ttl: 300\n\t#logfile: /var/log/unbound.log\n\t#verbosity: 10" > /etc/unbound/unbound.conf.d/cloudron-network.conf
 # update the root anchor after a out-of-disk-space situation (see #269)
 unbound-anchor -a /var/lib/unbound/root.key

 echo "==> Adding systemd services"
 cp -r "${script_dir}/start/systemd/." /etc/systemd/system/
-systemctl disable cloudron.target || true
-rm -f /etc/systemd/system/cloudron.target
 [[ "${ubuntu_version}" == "16.04" ]] && sed -e 's/MemoryMax/MemoryLimit/g' -i /etc/systemd/system/box.service
 systemctl daemon-reload
 systemctl enable unbound
 systemctl enable cloudron-syslog
-systemctl enable box
+systemctl enable cloudron.target
 systemctl enable cloudron-firewall

 # update firewall rules
@@ -122,12 +108,17 @@ systemctl restart unbound
 # ensure cloudron-syslog runs
 systemctl restart cloudron-syslog

+$json -f /etc/cloudron/cloudron.conf -I -e "delete this.edition" # can be removed after 4.0
+
+echo "==> Clearing custom.yml"
+rm -f /etc/cloudron/custom.yml
+
 echo "==> Configuring sudoers"
 rm -f /etc/sudoers.d/${USER}
 cp "${script_dir}/start/sudoers" /etc/sudoers.d/${USER}

 echo "==> Configuring collectd"
-rm -rf /etc/collectd /var/log/collectd.log
+rm -rf /etc/collectd
 ln -sfF "${PLATFORM_DATA_DIR}/collectd" /etc/collectd
 cp "${script_dir}/start/collectd/collectd.conf" "${PLATFORM_DATA_DIR}/collectd/collectd.conf"
 systemctl restart collectd
@@ -136,8 +127,8 @@ echo "==> Configuring logrotate"
 if ! grep -q "^include ${PLATFORM_DATA_DIR}/logrotate.d" /etc/logrotate.conf; then
    echo -e "\ninclude ${PLATFORM_DATA_DIR}/logrotate.d\n" >> /etc/logrotate.conf
 fi
-rm -f "${PLATFORM_DATA_DIR}/logrotate.d/"*
 cp "${script_dir}/start/logrotate/"* "${PLATFORM_DATA_DIR}/logrotate.d/"
+rm -f "${PLATFORM_DATA_DIR}/logrotate.d/box-logrotate" "${PLATFORM_DATA_DIR}/logrotate.d/app-logrotate" # remove pre 3.6 config files

 # logrotate files have to be owned by root, this is here to fixup existing installations where we were resetting the owner to yellowtent
 chown root:root "${PLATFORM_DATA_DIR}/logrotate.d/"
@@ -156,15 +147,8 @@ cp "${script_dir}/start/nginx/mime.types" "${PLATFORM_DATA_DIR}/nginx/mime.types
 if ! grep -q "^Restart=" /etc/systemd/system/multi-user.target.wants/nginx.service; then
    # default nginx service file does not restart on crash
    echo -e "\n[Service]\nRestart=always\n" >> /etc/systemd/system/multi-user.target.wants/nginx.service
+    systemctl daemon-reload
 fi
-
-# worker_rlimit_nofile in nginx config can be max this number
-mkdir -p /etc/systemd/system/nginx.service.d
-if ! grep -q "^LimitNOFILE=" /etc/systemd/system/nginx.service.d/cloudron.conf; then
-    echo -e "[Service]\nLimitNOFILE=16384\n" > /etc/systemd/system/nginx.service.d/cloudron.conf
-fi
-
-systemctl daemon-reload
 systemctl start nginx

 # restart mysql to make sure it has latest config
@@ -189,16 +173,8 @@ readonly mysql_root_password="password"
 mysqladmin -u root -ppassword password password # reset default root password
 mysql -u root -p${mysql_root_password} -e 'CREATE DATABASE IF NOT EXISTS box'

-# set HOME explicity, because it's not set when the installer calls it. this is done because
-# paths.js uses this env var and some of the migrate code requires box code
 echo "==> Migrating data"
-cd "${BOX_SRC_DIR}"
-if ! HOME=${HOME_DIR} BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up; then
-    echo "DB migration failed"
-    exit 1
-fi
-
-rm -f /etc/cloudron/cloudron.conf
+(cd "${BOX_SRC_DIR}" && BOX_ENV=cloudron DATABASE_URL=mysql://root:${mysql_root_password}@127.0.0.1/box "${BOX_SRC_DIR}/node_modules/.bin/db-migrate" up)

 if [[ ! -f "${BOX_DATA_DIR}/dhparams.pem" ]]; then
    echo "==> Generating dhparams (takes forever)"
@@ -212,9 +188,6 @@ fi
 echo "==> Cleaning up stale redis directories"
 find "${APPS_DATA_DIR}" -maxdepth 2 -type d -name redis -exec rm -rf {} +

-echo "==> Cleaning up old logs"
-rm -f /home/yellowtent/platformdata/logs/*/*.log.* || true
-
 echo "==> Changing ownership"
 # be careful of what is chown'ed here. subdirs like mysql,redis etc are owned by the containers and will stop working if perms change
 chown -R "${USER}" /etc/cloudron
@@ -230,7 +203,7 @@ chown "${USER}:${USER}" "${BOX_DATA_DIR}/mail"
 chown "${USER}:${USER}" -R "${BOX_DATA_DIR}/mail/dkim" # this is owned by box currently since it generates the keys

 echo "==> Starting Cloudron"
-systemctl start box
+systemctl start cloudron.target

 sleep 2 # give systemd sometime to start the processes

@@ -6,29 +6,11 @@ echo "==> Setting up firewall"
 iptables -t filter -N CLOUDRON || true
 iptables -t filter -F CLOUDRON # empty any existing rules

-# first setup any user IP block lists
-ipset create cloudron_blocklist hash:net || true
-/home/yellowtent/box/src/scripts/setblocklist.sh
-
-iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP
-if ! iptables -t filter -C FORWARD -m set --match-set cloudron_blocklist src -j DROP; then
-    iptables -t filter -A FORWARD -m set --match-set cloudron_blocklist src -j DROP
-fi
-
-# allow related and establisted connections
-iptables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
-iptables -t filter -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443 -j ACCEPT # 202 is the alternate ssh port
-
-# whitelist any user ports
-ports_json="/home/yellowtent/boxdata/firewall/ports.json"
-if allowed_tcp_ports=$(node -e "console.log(JSON.parse(fs.readFileSync('${ports_json}', 'utf8')).allowed_tcp_ports.join(','))" 2>/dev/null); then
-    [[ -n "${allowed_tcp_ports}" ]] && iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports "${allowed_tcp_ports}" -j ACCEPT
-fi
-
-# turn and stun service
-iptables -t filter -A CLOUDRON -p tcp -m multiport --dports 3478,5349 -j ACCEPT
-iptables -t filter -A CLOUDRON -p udp -m multiport --dports 3478,5349 -j ACCEPT
-iptables -t filter -A CLOUDRON -p udp -m multiport --dports 50000:51000 -j ACCEPT
+# NOTE: keep these in sync with src/apps.js validatePortBindings
+# allow ssh, http, https, ping, dns
+iptables -t filter -I CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT
+# ssh is allowed alternately on port 202
+iptables -A CLOUDRON -p tcp -m tcp -m multiport --dports 22,25,80,202,443,587,993,4190 -j ACCEPT

 iptables -t filter -A CLOUDRON -p icmp --icmp-type echo-request -j ACCEPT
 iptables -t filter -A CLOUDRON -p icmp --icmp-type echo-reply -j ACCEPT
@@ -3,19 +3,12 @@
 printf "**********************************************************************\n\n"

 if [[ -z "$(ls -A /home/yellowtent/boxdata/mail/dkim)" ]]; then
-    if [[ -f /tmp/.cloudron-motd-cache ]]; then
-        ip=$(cat /tmp/.cloudron-motd-cache)
-    elif ! ip=$(curl --fail --connect-timeout 2 --max-time 2 -q https://api.cloudron.io/api/v1/helper/public_ip | sed -n -e 's/.*"ip": "\(.*\)"/\1/p'); then
-        ip='<IP>'
-    fi
-    echo "${ip}" > /tmp/.cloudron-motd-cache
-
    printf "\t\t\tWELCOME TO CLOUDRON\n"
    printf "\t\t\t-------------------\n"

-    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://${ip} on your browser and accept the self-signed certificate to finish setup."
-    printf "Cloudron overview - https://docs.cloudron.io/ \n"
-    printf "Cloudron setup - https://docs.cloudron.io/installation/#setup \n"
+    printf '\n\e[1;32m%-6s\e[m\n\n' "Visit https://<IP> on your browser and accept the self-signed certificate to finish setup."
+    printf "Cloudron overview - https://cloudron.io/documentation/ \n"
+    printf "Cloudron setup - https://cloudron.io/documentation/installation/#setup \n"
 else
    printf "\t\t\tNOTE TO CLOUDRON ADMINS\n"
    printf "\t\t\t-----------------------\n"
@@ -23,7 +16,7 @@ else
    printf "Cloudron relies on and may break your installation. Ubuntu security updates\n"
    printf "are automatically installed on this server every night.\n"
    printf "\n"
-    printf "Read more at https://docs.cloudron.io/security/#os-updates\n"
+    printf "Read more at https://cloudron.io/documentation/security/#os-updates\n"
 fi

 printf "\nFor help and more information, visit https://forum.cloudron.io\n\n"
@@ -4,11 +4,6 @@ set -eu -o pipefail

 readonly APPS_SWAP_FILE="/apps.swap"

-if [[ -f "${APPS_SWAP_FILE}" ]]; then
-    echo "Swap file already exists at /apps.swap . Skipping"
-    exit
-fi
-
 # all sizes are in mb
 readonly physical_memory=$(LC_ALL=C free -m | awk '/Mem:/ { print $2 }')
 readonly swap_size=$((${physical_memory} > 4096 ? 4096 : ${physical_memory})) # min(RAM, 4GB) if you change this, fix enoughResourcesAvailable() in client.js
@@ -57,7 +57,7 @@ LoadPlugin logfile

 <Plugin logfile>
   LogLevel "info"
-   File "/home/yellowtent/platformdata/logs/collectd/collectd.log"
+   File "/var/log/collectd.log"
   Timestamp true
   PrintSeverity false
 </Plugin>
@@ -240,23 +240,8 @@ LoadPlugin write_graphite
    Interactive false

    Import "df"
-
-    Import "du"
-    <Module du>
-        <Path>
-            Instance maildata
-            Dir "/home/yellowtent/boxdata/mail"
-        </Path>
-        <Path>
-            Instance boxdata
-            Dir "/home/yellowtent/boxdata"
-            Exclude "mail"
-        </Path>
-        <Path>
-            Instance platformdata
-            Dir "/home/yellowtent/platformdata"
-        </Path>
-    </Module>
+    # <Module df>
+    # </Module>
 </Plugin>

 <Plugin write_graphite>
@@ -21,7 +21,6 @@ def read():
        except:
            continue

-        # type comes from https://github.com/collectd/collectd/blob/master/src/types.db
        val = collectd.Values(type='df_complex', plugin='df', plugin_instance=instance)

        free = st.f_bavail * st.f_frsize # bavail is for non-root user. bfree is total
@@ -1,82 +0,0 @@
-import collectd,os,subprocess,sys,re,time
-
-# https://www.programcreek.com/python/example/106897/collectd.register_read
-
-PATHS = [] # { name, dir, exclude }
-# there is a pattern in carbon/storage-schemas.conf which stores values every 12h for a year
-INTERVAL = 60 * 60 * 12 # twice a day. change values in docker-graphite if you change this
-
-def du(pathinfo):
-    # -B1 makes du print block sizes and not apparent sizes (to match df which also uses block sizes)
-    cmd = 'timeout 1800 du -DsB1 "{}"'.format(pathinfo['dir'])
-    if pathinfo['exclude'] != '':
-        cmd += ' --exclude "{}"'.format(pathinfo['exclude'])
-
-    collectd.info('computing size with command: %s' % cmd);
-    try:
-        size = subprocess.check_output(cmd, shell=True).split()[0].decode('utf-8')
-        collectd.info('\tsize of %s is %s (time: %i)' % (pathinfo['dir'], size, int(time.time())))
-        return size
-    except Exception as e:
-        collectd.info('\terror getting the size of %s: %s' % (pathinfo['dir'], str(e)))
-        return 0
-
-def parseSize(size):
-    units = {"B": 1, "KB": 10**3, "MB": 10**6, "GB": 10**9, "TB": 10**12}
-    number, unit, _ = re.split('([a-zA-Z]+)', size.upper())
-    return int(float(number)*units[unit])
-
-def dockerSize():
-    # use --format '{{json .}}' to dump the string. '{{if eq .Type "Images"}}{{.Size}}{{end}}' still creates newlines
-    # https://godoc.org/github.com/docker/go-units#HumanSize is used. so it's 1000 (KB) and not 1024 (KiB)
-    cmd = 'timeout 1800 docker system df --format "{{.Size}}" | head -n1'
-    try:
-        size = subprocess.check_output(cmd, shell=True).strip().decode('utf-8')
-        collectd.info('size of docker images is %s (%s) (time: %i)' % (size, parseSize(size), int(time.time())))
-        return parseSize(size)
-    except Exception as e:
-        collectd.info('error getting docker images size : %s' % str(e))
-        return 0
-
-# configure is called for each module block. this is called before init
-def configure(config):
-    global PATHS
-
-    for child in config.children:
-        if child.key != 'Path':
-            collectd.info('du plugin: Unknown config key "%s"' % key)
-            continue
-
-        pathinfo = { 'name': '', 'dir': '', 'exclude': '' }
-        for node in child.children:
-            if node.key == 'Instance':
-                pathinfo['name'] = node.values[0]
-            elif node.key == 'Dir':
-                pathinfo['dir'] = node.values[0]
-            elif node.key == 'Exclude':
-                pathinfo['exclude'] = node.values[0]
-
-        PATHS.append(pathinfo);
-        collectd.info('du plugin: monitoring %s' % pathinfo['dir']);
-
-def init():
-    global PATHS
-    collectd.info('custom du plugin initialized with %s %s' % (PATHS, sys.version))
-
-def read():
-    for pathinfo in PATHS:
-        size = du(pathinfo)
-
-        # type comes from https://github.com/collectd/collectd/blob/master/src/types.db
-        val = collectd.Values(type='capacity', plugin='du', plugin_instance=pathinfo['name'])
-        val.dispatch(values=[size], type_instance='usage')
-
-    size = dockerSize()
-    val = collectd.Values(type='capacity', plugin='du', plugin_instance='docker')
-    val.dispatch(values=[size], type_instance='usage')
-
-
-
-collectd.register_init(init)
-collectd.register_config(configure)
-collectd.register_read(read, INTERVAL)
@@ -0,0 +1,16 @@
+# add customizations here
+# after making changes run "sudo systemctl restart box"
+
+# features:
+#   configureBackup: true
+#   dynamicDns: true
+#   subscription: true
+#   remoteSupport: true
+# 
+# support:
+#   email: support@cloudron.io
+# 
+# alerts:
+#   email: support@cloudron.io
+#   notifyCloudronAdmins: false
+
@@ -0,0 +1,18 @@
+# logrotate config for app, crash, addon and task logs
+
+# man 7 glob
+/home/yellowtent/platformdata/logs/[!t][!a][!s][!k][!s]/*.log {
+    # only keep one rotated file, we currently do not send that over the api
+    rotate 1
+    size 10M
+    # we never compress so we can simply tail the files
+    nocompress
+    copytruncate
+}
+
+/home/yellowtent/platformdata/logs/tasks/*.log {
+    monthly
+    rotate 0
+    missingok
+}
+
@@ -1,8 +1,7 @@
 # logrotate config for box logs

-# keep upto 5 logs of size 10M each
 /home/yellowtent/platformdata/logs/box.log {
-    rotate 5
+    rotate 10
    size 10M
    # we never compress so we can simply tail the files
    nocompress
@@ -1,34 +0,0 @@
-# logrotate config for app, crash, addon and task logs
-
-# man 7 glob
-/home/yellowtent/platformdata/logs/graphite/*.log
-/home/yellowtent/platformdata/logs/mail/*.log
-/home/yellowtent/platformdata/logs/mysql/*.log
-/home/yellowtent/platformdata/logs/mongodb/*.log
-/home/yellowtent/platformdata/logs/postgresql/*.log
-/home/yellowtent/platformdata/logs/sftp/*.log
-/home/yellowtent/platformdata/logs/redis-*/*.log
-/home/yellowtent/platformdata/logs/crash/*.log
-/home/yellowtent/platformdata/logs/collectd/*.log
-/home/yellowtent/platformdata/logs/turn/*.log
-/home/yellowtent/platformdata/logs/updater/*.log {
-    # only keep one rotated file, we currently do not send that over the api
-    rotate 1
-    size 10M
-    missingok
-    # we never compress so we can simply tail the files
-    nocompress
-    # this truncates the original log file and not the rotated one
-    copytruncate
-}
-
-# keep task logs for a week. the 'nocreate' option ensures empty log files are not
-# created post rotation
-/home/yellowtent/platformdata/logs/tasks/*.log {
-    minage 7
-    daily
-    rotate 0
-    missingok
-    nocreate
-}
-
@@ -1,18 +1,11 @@
 user www-data;

-# detect based on available CPU cores
-worker_processes  auto;
-
-# this is 4096 by default. See /proc/<PID>/limits and /etc/security/limits.conf
-# usually twice the worker_connections (one for uptsream, one for downstream)
-# see also LimitNOFILE=16384 in systemd drop-in
-worker_rlimit_nofile 8192; 
+worker_processes  1;

 pid /run/nginx.pid;

 events {
-    # a single worker has these many simultaneous connections max
-    worker_connections  4096;
+    worker_connections  1024;
 }

 http {
@@ -50,15 +50,3 @@ yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartdocker.s
 Defaults!/home/yellowtent/box/src/scripts/restartunbound.sh env_keep="HOME BOX_ENV"
 yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/restartunbound.sh

-Defaults!/home/yellowtent/box/src/scripts/rmmailbox.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/rmmailbox.sh
-
-Defaults!/home/yellowtent/box/src/scripts/starttask.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD:SETENV: /home/yellowtent/box/src/scripts/starttask.sh
-
-Defaults!/home/yellowtent/box/src/scripts/stoptask.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/stoptask.sh
-
-Defaults!/home/yellowtent/box/src/scripts/setblocklist.sh env_keep="HOME BOX_ENV"
-yellowtent ALL=(root) NOPASSWD: /home/yellowtent/box/src/scripts/setblocklist.sh
-
@@ -1,21 +1,20 @@
 [Unit]
 Description=Cloudron Admin
 OnFailure=crashnotifier@%n.service
+StopWhenUnneeded=true
 ; journald crashes result in a EPIPE in node. Cannot ignore it as it results in loss of logs.
 BindsTo=systemd-journald.service
 After=mysql.service nginx.service
 ; As cloudron-resize-fs is a one-shot, the Wants= automatically ensures that the service *finishes*
 Wants=cloudron-resize-fs.service

-[Install]
-WantedBy=multi-user.target
-
 [Service]
 Type=idle
 WorkingDirectory=/home/yellowtent/box
 Restart=always
-ExecStart=/home/yellowtent/box/box.js
-Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box:*,connect-lastmile,-box:ldap" "BOX_ENV=cloudron" "NODE_ENV=production"
+; Systemd does not append logs when logging to files, we spawn a shell first and exec to replace it after setting up the pipes
+ExecStart=/bin/sh -c 'echo "Logging to /home/yellowtent/platformdata/logs/box.log"; exec /usr/bin/node --max_old_space_size=150 /home/yellowtent/box/box.js >> /home/yellowtent/platformdata/logs/box.log 2>&1'
+Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box*,connect-lastmile" "BOX_ENV=cloudron" "NODE_ENV=production"
 ; kill apptask processes as well
 KillMode=control-group
 ; Do not kill this process on OOM. Children inherit this score. Do not set it to -1000 so that MemoryMax can keep working
@@ -0,0 +1,10 @@
+[Unit]
+Description=Cloudron Smartserver
+Documentation=https://cloudron.io/documentation.html
+StopWhenUnneeded=true
+Requires=box.service
+After=box.service
+# AllowIsolate=yes
+
+[Install]
+WantedBy=multi-user.target
@@ -1,11 +0,0 @@
-server:
-        interface: 0.0.0.0
-        do-ip6: no
-        access-control: 127.0.0.1 allow
-        access-control: 172.18.0.1/16 allow
-        cache-max-negative-ttl: 30
-        cache-max-ttl: 300
-        # enable below for logging to journalctl -u unbound
-        # verbosity: 5
-        # log-queries: yes
-
@@ -4,4 +4,4 @@ set -eu -o pipefail

 echo "Stopping cloudron"

-systemctl stop box
+systemctl stop cloudron.target
@@ -1,29 +1,144 @@
 'use strict';

 exports = module.exports = {
-    verifyToken: verifyToken
+    SCOPE_APPS_READ: 'apps:read',
+    SCOPE_APPS_MANAGE: 'apps:manage',
+    SCOPE_APPSTORE: 'appstore',
+    SCOPE_CLIENTS: 'clients',
+    SCOPE_CLOUDRON: 'cloudron',
+    SCOPE_DOMAINS_READ: 'domains:read',
+    SCOPE_DOMAINS_MANAGE: 'domains:manage',
+    SCOPE_MAIL: 'mail',
+    SCOPE_PROFILE: 'profile',
+    SCOPE_SETTINGS: 'settings',
+    SCOPE_SUBSCRIPTION: 'subscription',
+    SCOPE_USERS_READ: 'users:read',
+    SCOPE_USERS_MANAGE: 'users:manage',
+    VALID_SCOPES: [ 'apps', 'appstore', 'clients', 'cloudron', 'domains', 'mail', 'profile', 'settings', 'subscription', 'users' ], // keep this sorted
+
+    SCOPE_ANY: '*',
+
+    validateScopeString: validateScopeString,
+    hasScopes: hasScopes,
+    canonicalScopeString: canonicalScopeString,
+    intersectScopes: intersectScopes,
+    validateToken: validateToken,
+    scopesForUser: scopesForUser
 };

 var assert = require('assert'),
-    BoxError = require('./boxerror.js'),
+    config = require('./config.js'),
+    DatabaseError = require('./databaseerror.js'),
+    debug = require('debug')('box:accesscontrol'),
    tokendb = require('./tokendb.js'),
-    users = require('./users.js');
+    users = require('./users.js'),
+    UsersError = users.UsersError,
+    _ = require('underscore');

-function verifyToken(accessToken, callback) {
+// returns scopes that does not have wildcards and is sorted
+function canonicalScopeString(scope) {
+    if (scope === exports.SCOPE_ANY) return exports.VALID_SCOPES.join(',');
+
+    return scope.split(',').sort().join(',');
+}
+
+function intersectScopes(allowedScopes, wantedScopes) {
+    assert(Array.isArray(allowedScopes), 'Expecting sorted array');
+    assert(Array.isArray(wantedScopes), 'Expecting sorted array');
+
+    if (_.isEqual(allowedScopes, wantedScopes)) return allowedScopes; // quick path
+
+    let wantedScopesMap = new Map();
+    let results = [];
+
+    // make a map of scope -> [ subscopes ]
+    for (let w of wantedScopes) {
+        let parts = w.split(':');
+        let subscopes = wantedScopesMap.get(parts[0]) || new Set();
+        subscopes.add(parts[1] || '*');
+        wantedScopesMap.set(parts[0], subscopes);
+    }
+
+    for (let a of allowedScopes) {
+        let parts = a.split(':');
+        let as = parts[1] || '*';
+
+        let subscopes = wantedScopesMap.get(parts[0]);
+        if (!subscopes) continue;
+
+        if (subscopes.has('*') || subscopes.has(as)) {
+            results.push(a);
+        } else if (as === '*') {
+            results = results.concat(Array.from(subscopes).map(function (ss) { return `${a}:${ss}`; }));
+        }
+    }
+
+    return results;
+}
+
+function validateScopeString(scope) {
+    assert.strictEqual(typeof scope, 'string');
+
+    if (scope === '') return new Error('Empty scope not allowed');
+
+    // NOTE: this function intentionally does not allow '*'. This is only allowed in the db to allow
+    // us not write a migration script every time we add a new scope
+    var allValid = scope.split(',').every(function (s) { return exports.VALID_SCOPES.indexOf(s.split(':')[0]) !== -1; });
+    if (!allValid) return new Error('Invalid scope. Available scopes are ' + exports.VALID_SCOPES.join(', '));
+
+    return null;
+}
+
+// tests if all requiredScopes are attached to the request
+function hasScopes(authorizedScopes, requiredScopes) {
+    assert(Array.isArray(authorizedScopes), 'Expecting array');
+    assert(Array.isArray(requiredScopes), 'Expecting array');
+
+    if (authorizedScopes.indexOf(exports.SCOPE_ANY) !== -1) return null;
+
+    for (var i = 0; i < requiredScopes.length; ++i) {
+        const scopeParts = requiredScopes[i].split(':');
+
+        // this allows apps:write if the token has a higher apps scope
+        if (authorizedScopes.indexOf(requiredScopes[i]) === -1 && authorizedScopes.indexOf(scopeParts[0]) === -1) {
+            debug('scope: missing scope "%s".', requiredScopes[i]);
+            return new Error('Missing required scope "' + requiredScopes[i] + '"');
+        }
+    }
+
+    return null;
+}
+
+function scopesForUser(user, callback) {
+    assert.strictEqual(typeof user, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    if (user.admin) return callback(null, exports.VALID_SCOPES);
+
+    callback(null, [ 'profile', 'apps:read' ]);
+}
+
+function validateToken(accessToken, callback) {
    assert.strictEqual(typeof accessToken, 'string');
    assert.strictEqual(typeof callback, 'function');

    tokendb.getByAccessToken(accessToken, function (error, token) {
-        if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-        if (error) return callback(error);
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, null /* user */, 'Invalid Token'); // will end up as a 401
+        if (error) return callback(error); // this triggers 'internal error' in passport

        users.get(token.identifier, function (error, user) {
-            if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+            if (error && error.reason === UsersError.NOT_FOUND) return callback(null, null /* user */, 'Invalid Token'); // will end up as a 401
            if (error) return callback(error);

-            if (!user.active) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
+            scopesForUser(user, function (error, userScopes) {
+                if (error) return callback(error);

-            callback(null, user);
+                var authorizedScopes = intersectScopes(userScopes, token.scope.split(','));
+                const skipPasswordVerification = token.clientId === 'cid-sdk' || token.clientId === 'cid-cli'; // these clients do not require password checks unlike UI
+                var info = { authorizedScopes: authorizedScopes, skipPasswordVerification: skipPasswordVerification }; // ends up in req.authInfo
+
+                callback(null, user, info);
+            });
        });
    });
 }
@@ -7,7 +7,6 @@ map $http_upgrade $connection_upgrade {
 # http server
 server {
    listen       80;
-    server_tokens off; # hide version
 <% if (hasIPv6) { -%>
    listen       [::]:80;
 <% } -%>
@@ -35,14 +34,7 @@ server {

    location / {
        # redirect everything to HTTPS
-<% if ( endpoint === 'admin' ) { %>
        return 301 https://$host$request_uri;
-<% } else if ( endpoint === 'app' ) { %>
-        return 301 https://$host$request_uri;
-<% } else if ( endpoint === 'redirect' ) { %>
-        return 301 https://<%= redirectTo %>$request_uri;
-<% } %>
-
    }
 }

@@ -50,36 +42,40 @@ server {
 server {
 <% if (vhost) { -%>
    server_name  <%= vhost %>;
-    listen       443 ssl http2;
+    listen       443 http2;
 <% if (hasIPv6) { -%>
-    listen       [::]:443 ssl http2;
+    listen       [::]:443 http2;
 <% } -%>
 <% } else { -%>
-    listen       443 ssl http2 default_server;
+    listen       443 http2 default_server;
 <% if (hasIPv6) { -%>
-    listen       [::]:443 ssl http2 default_server;
+    listen       [::]:443 http2 default_server;
 <% } -%>
 <% } -%>

-    server_tokens off; # hide version
-
+    ssl                  on;
    # paths are relative to prefix and not to this file
    ssl_certificate      <%= certFilePath %>;
    ssl_certificate_key  <%= keyFilePath %>;
    ssl_session_timeout  5m;
-    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
-    ssl_session_tickets off;
+    ssl_session_cache shared:SSL:50m;

+    # https://bettercrypto.org/static/applied-crypto-hardening.pdf
+    # https://mozilla.github.io/server-side-tls/ssl-config-generator/
+    # https://cipherli.st/
    # https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
-    # https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#25-use-forward-secrecy
-    # ciphers according to https://ssl-config.mozilla.org/#server=nginx&version=1.14.0&config=intermediate&openssl=1.1.1&guideline=5.4
-    ssl_protocols TLSv1.2 TLSv1.3;
-    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256;
-    ssl_prefer_server_ciphers off;
+    ssl_prefer_server_ciphers on;
+    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don't use SSLv3 ref: POODLE

+    # ciphers according to https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=nginx-1.10.3&openssl=1.0.2g&hsts=yes&profile=modern
+    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_dhparam /home/yellowtent/boxdata/dhparams.pem;
    add_header Strict-Transport-Security "max-age=15768000";

+    # https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options
+    add_header X-Frame-Options "<%= xFrameOptions %>";
+    proxy_hide_header X-Frame-Options;
+
    # https://github.com/twitter/secureheaders
    # https://www.owasp.org/index.php/OWASP_Secure_Headers_Project#tab=Compatibility_Matrix
    # https://wiki.mozilla.org/Security/Guidelines/Web_Security
@@ -104,21 +100,11 @@ server {

 <% if ( endpoint === 'admin' ) { -%>
    # CSP headers for the admin/dashboard resources
-    add_header Content-Security-Policy "default-src 'none'; frame-src 'self' cloudron.io *.cloudron.io; connect-src wss: https: 'self' *.cloudron.io; script-src https: 'self' 'unsafe-inline' 'unsafe-eval'; img-src * data:; style-src https: 'unsafe-inline'; object-src 'none'; font-src https: 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'self';";
-<% } else { %>
-    <% if (cspQuoted) { %>
-    add_header Content-Security-Policy <%- cspQuoted %>;
-    <% } %>
-
-    <% for (var i = 0; i < hideHeaders.length; i++) { -%>
-    proxy_hide_header <%- hideHeaders[i] %>;
-    <% } %>
+    add_header Content-Security-Policy "default-src 'none'; connect-src wss: https: 'self' *.cloudron.io; script-src https: 'self' 'unsafe-inline' 'unsafe-eval'; img-src * data:; style-src https: 'unsafe-inline'; object-src 'none'; font-src https: 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'self';";
 <% } -%>

    proxy_http_version 1.1;
-    # intercept errors (>= 400) and use the error_page handler
    proxy_intercept_errors on;
-    # nginx will return 504 on connect/timeout errors
    proxy_read_timeout       3500;
    proxy_connect_timeout    3250;

@@ -135,32 +121,11 @@ server {

    # only serve up the status page if we get proxy gateway errors
    root <%= sourceDir %>/dashboard/dist;
-    # some apps use 503 to indicate updating or maintenance
-    error_page 502 504 /app_error_page;
-    location /app_error_page {
-        root /home/yellowtent/boxdata;
-        # the first argument looks for file under the root
-        try_files /custom_pages/$request_uri /custom_pages/app_not_responding.html /appstatus.html;
-        # internal means this is for internal routing and cannot be accessed as URL from browser
+    error_page 502 503 504 /appstatus.html;
+    location /appstatus.html {
        internal;
    }

-    location @wellknown-upstream {
-<% if ( endpoint === 'admin' ) { %>
-        proxy_pass http://127.0.0.1:3000;
-<% } else if ( endpoint === 'app' ) { %>
-        proxy_pass http://127.0.0.1:<%= port %>;
-<% } else if ( endpoint === 'redirect' ) { %>
-        return 302 https://<%= redirectTo %>$request_uri;
-<% } %>
-    }
-
-    # user defined .well-known resources
-    location ~ ^/.well-known/(.*)$ {
-        root /home/yellowtent/boxdata/well-known/$host;
-        try_files /$1 @wellknown-upstream;
-    }
-
    location / {
        # increase the proxy buffer sizes to not run into buffer issues (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers)
        proxy_buffer_size       128k;
@@ -202,14 +167,9 @@ server {
            client_max_body_size 0;
        }

-        location ~ ^/api/v1/apps/.*/files/ {
-            proxy_pass   http://127.0.0.1:3000;
-            client_max_body_size 0;
-        }
-
-        # graphite paths (uncomment block below and visit /graphite-web/dashboard)
+        # graphite paths (uncomment block below and visit /graphite/index.html)
        # remember to comment out the CSP policy as well to access the graphite dashboard
-        # location ~ ^/graphite-web/ {
+        # location ~ ^/(graphite|content|metrics|dashboard|render|browser|composer)/ {
        #     proxy_pass   http://127.0.0.1:8417;
        #     client_max_body_size 1m;
        # }
@@ -21,9 +21,36 @@ exports = module.exports = {
    getAppIdByAddonConfigValue: getAppIdByAddonConfigValue,

    setHealth: setHealth,
-    setTask: setTask,
+    setInstallationCommand: setInstallationCommand,
+    setRunCommand: setRunCommand,
    getAppStoreIds: getAppStoreIds,

+    setOwner: setOwner,
+    transferOwnership: transferOwnership,
+
+    // installation codes (keep in sync in UI)
+    ISTATE_PENDING_INSTALL: 'pending_install', // installs and fresh reinstalls
+    ISTATE_PENDING_CLONE: 'pending_clone', // clone
+    ISTATE_PENDING_CONFIGURE: 'pending_configure', // config (location, port) changes and on infra update
+    ISTATE_PENDING_UNINSTALL: 'pending_uninstall', // uninstallation
+    ISTATE_PENDING_RESTORE: 'pending_restore', // restore to previous backup or on upgrade
+    ISTATE_PENDING_UPDATE: 'pending_update', // update from installed state preserving data
+    ISTATE_PENDING_FORCE_UPDATE: 'pending_force_update', // update from any state preserving data
+    ISTATE_PENDING_BACKUP: 'pending_backup', // backup the app
+    ISTATE_ERROR: 'error', // error executing last pending_* command
+    ISTATE_INSTALLED: 'installed', // app is installed
+
+    RSTATE_RUNNING: 'running',
+    RSTATE_PENDING_START: 'pending_start',
+    RSTATE_PENDING_STOP: 'pending_stop',
+    RSTATE_STOPPED: 'stopped', // app stopped by us
+
+    // run codes (keep in sync in UI)
+    HEALTH_HEALTHY: 'healthy',
+    HEALTH_UNHEALTHY: 'unhealthy',
+    HEALTH_ERROR: 'error',
+    HEALTH_DEAD: 'dead',
+
    // subdomain table types
    SUBDOMAIN_TYPE_PRIMARY: 'primary',
    SUBDOMAIN_TYPE_REDIRECT: 'redirect',
@@ -33,17 +60,17 @@ exports = module.exports = {

 var assert = require('assert'),
    async = require('async'),
-    BoxError = require('./boxerror.js'),
    database = require('./database.js'),
+    DatabaseError = require('./databaseerror'),
    safe = require('safetydance'),
    util = require('util');

-var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.errorJson', 'apps.runState',
+var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.installationProgress', 'apps.runState',
    'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'subdomains.subdomain AS location', 'subdomains.domain',
-    'apps.accessRestrictionJson', 'apps.memoryLimit', 'apps.cpuShares',
-    'apps.label', 'apps.tagsJson', 'apps.taskId', 'apps.reverseProxyConfigJson', 'apps.servicesConfigJson', 'apps.bindsJson',
-    'apps.sso', 'apps.debugModeJson', 'apps.enableBackup',
-    'apps.creationTime', 'apps.updateTime', 'apps.mailboxName', 'apps.mailboxDomain', 'apps.enableAutomaticUpdate',
+    'apps.accessRestrictionJson', 'apps.restoreConfigJson', 'apps.oldConfigJson', 'apps.updateConfigJson', 'apps.memoryLimit',
+    'apps.label', 'apps.tagsJson',
+    'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson', 'apps.robotsTxt', 'apps.enableBackup',
+    'apps.creationTime', 'apps.updateTime', 'apps.ownerId', 'apps.mailboxName', 'apps.enableAutomaticUpdate',
    'apps.dataDir', 'apps.ts', 'apps.healthTime' ].join(',');

 var PORT_BINDINGS_FIELDS = [ 'hostPort', 'type', 'environmentVariable', 'appId' ].join(',');
@@ -57,14 +84,22 @@ function postProcess(result) {
    result.manifest = safe.JSON.parse(result.manifestJson);
    delete result.manifestJson;

+    assert(result.oldConfigJson === null || typeof result.oldConfigJson === 'string');
+    result.oldConfig = safe.JSON.parse(result.oldConfigJson);
+    delete result.oldConfigJson;
+
+    assert(result.updateConfigJson === null || typeof result.updateConfigJson === 'string');
+    result.updateConfig = safe.JSON.parse(result.updateConfigJson);
+    delete result.updateConfigJson;
+
+    assert(result.restoreConfigJson === null || typeof result.restoreConfigJson === 'string');
+    result.restoreConfig = safe.JSON.parse(result.restoreConfigJson);
+    delete result.restoreConfigJson;
+
    assert(result.tagsJson === null || typeof result.tagsJson === 'string');
    result.tags = safe.JSON.parse(result.tagsJson) || [];
    delete result.tagsJson;

-    assert(result.reverseProxyConfigJson === null || typeof result.reverseProxyConfigJson === 'string');
-    result.reverseProxyConfig = safe.JSON.parse(result.reverseProxyConfigJson) || {};
-    delete result.reverseProxyConfigJson;
-
    assert(result.hostPorts === null || typeof result.hostPorts === 'string');
    assert(result.environmentVariables === null || typeof result.environmentVariables === 'string');

@@ -86,6 +121,9 @@ function postProcess(result) {
    if (result.accessRestriction && !result.accessRestriction.users) result.accessRestriction.users = [];
    delete result.accessRestrictionJson;

+    // TODO remove later once all apps have this attribute
+    result.xFrameOptions = result.xFrameOptions || 'SAMEORIGIN';
+
    result.sso = !!result.sso; // make it bool
    result.enableBackup = !!result.enableBackup; // make it bool
    result.enableAutomaticUpdate = !!result.enableAutomaticUpdate; // make it bool
@@ -94,14 +132,6 @@ function postProcess(result) {
    result.debugMode = safe.JSON.parse(result.debugModeJson);
    delete result.debugModeJson;

-    assert(result.servicesConfigJson === null || typeof result.servicesConfigJson === 'string');
-    result.servicesConfig = safe.JSON.parse(result.servicesConfigJson) || {};
-    delete result.servicesConfigJson;
-
-    assert(result.bindsJson === null || typeof result.bindsJson === 'string');
-    result.binds = safe.JSON.parse(result.bindsJson) || {};
-    delete result.bindsJson;
-
    result.alternateDomains = result.alternateDomains || [];
    result.alternateDomains.forEach(function (d) {
        delete d.appId;
@@ -116,10 +146,8 @@ function postProcess(result) {
        if (envNames[i]) result.env[envNames[i]] = envValues[i];
    }

-    result.error = safe.JSON.parse(result.errorJson);
-    delete result.errorJson;
-
-    result.taskId = result.taskId ? String(result.taskId) : null;
+    // in the db, we store dataDir as unique/nullable
+    result.dataDir = result.dataDir || '';
 }

 function get(id, callback) {
@@ -134,11 +162,11 @@ function get(id, callback) {
        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE apps.id = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, id ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            result[0].alternateDomains = alternateDomains;

@@ -161,11 +189,11 @@ function getByHttpPort(httpPort, callback) {
        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE httpPort = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, httpPort ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            result[0].alternateDomains = alternateDomains;
            postProcess(result[0]);
@@ -187,11 +215,11 @@ function getByContainerId(containerId, callback) {
        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' WHERE containerId = ? GROUP BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY, containerId ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE appId = ? AND type = ?', [ result[0].id, exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            result[0].alternateDomains = alternateDomains;
            postProcess(result[0]);
@@ -212,10 +240,10 @@ function getAll(callback) {
        + '  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId'
        + '  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = ?'
        + ' GROUP BY apps.id ORDER BY apps.id', [ exports.SUBDOMAIN_TYPE_PRIMARY ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        database.query('SELECT ' + SUBDOMAIN_FIELDS + ' FROM subdomains WHERE type = ?', [ exports.SUBDOMAIN_TYPE_REDIRECT ], function (error, alternateDomains) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            alternateDomains.forEach(function (d) {
                var domain = results.find(function (a) { return d.appId === a.id; });
@@ -232,13 +260,14 @@ function getAll(callback) {
    });
 }

-function add(id, appStoreId, manifest, location, domain, portBindings, data, callback) {
+function add(id, appStoreId, manifest, location, domain, ownerId, portBindings, data, callback) {
    assert.strictEqual(typeof id, 'string');
    assert.strictEqual(typeof appStoreId, 'string');
    assert(manifest && typeof manifest === 'object');
    assert.strictEqual(typeof manifest.version, 'string');
    assert.strictEqual(typeof location, 'string');
    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
    assert.strictEqual(typeof portBindings, 'object');
    assert(data && typeof data === 'object');
    assert.strictEqual(typeof callback, 'function');
@@ -250,26 +279,25 @@ function add(id, appStoreId, manifest, location, domain, portBindings, data, cal
    const accessRestriction = data.accessRestriction || null;
    const accessRestrictionJson = JSON.stringify(accessRestriction);
    const memoryLimit = data.memoryLimit || 0;
-    const cpuShares = data.cpuShares || 512;
-    const installationState = data.installationState;
-    const runState = data.runState;
+    const xFrameOptions = data.xFrameOptions || '';
+    const installationState = data.installationState || exports.ISTATE_PENDING_INSTALL;
+    const restoreConfigJson = data.restoreConfig ? JSON.stringify(data.restoreConfig) : null; // used when cloning
    const sso = 'sso' in data ? data.sso : null;
+    const robotsTxt = 'robotsTxt' in data ? data.robotsTxt : null;
    const debugModeJson = data.debugMode ? JSON.stringify(data.debugMode) : null;
    const env = data.env || {};
    const label = data.label || null;
    const tagsJson = data.tags ? JSON.stringify(data.tags) : null;
    const mailboxName = data.mailboxName || null;
-    const mailboxDomain = data.mailboxDomain || null;
-    const reverseProxyConfigJson = data.reverseProxyConfig ? JSON.stringify(data.reverseProxyConfig) : null;

    var queries = [];

    queries.push({
-        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, runState, accessRestrictionJson, memoryLimit, cpuShares, '
-            + 'sso, debugModeJson, mailboxName, mailboxDomain, label, tagsJson, reverseProxyConfigJson) '
+        query: 'INSERT INTO apps (id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions,'
+            + 'restoreConfigJson, sso, debugModeJson, robotsTxt, ownerId, mailboxName, label, tagsJson) '
            + ' VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        args: [ id, appStoreId, manifestJson, installationState, runState, accessRestrictionJson, memoryLimit, cpuShares,
-            sso, debugModeJson, mailboxName, mailboxDomain, label, tagsJson, reverseProxyConfigJson ]
+        args: [ id, appStoreId, manifestJson, installationState, accessRestrictionJson, memoryLimit, xFrameOptions, restoreConfigJson,
+            sso, debugModeJson, robotsTxt, ownerId, mailboxName, label, tagsJson ]
    });

    queries.push({
@@ -301,9 +329,9 @@ function add(id, appStoreId, manifest, location, domain, portBindings, data, cal
    }

    database.transaction(queries, function (error) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
-        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new BoxError(BoxError.NOT_FOUND, 'no such domain'));
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, error.message));
+        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new DatabaseError(DatabaseError.NOT_FOUND, 'no such domain'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null);
    });
@@ -314,7 +342,7 @@ function exists(id, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT 1 FROM apps WHERE id=?', [ id ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        return callback(null, result.length !== 0);
    });
@@ -325,7 +353,7 @@ function getPortBindings(id, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + PORT_BINDINGS_FIELDS + ' FROM appPortBindings WHERE appId = ?', [ id ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        var portBindings = { };
        for (var i = 0; i < results.length; i++) {
@@ -342,8 +370,8 @@ function delPortBinding(hostPort, type, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('DELETE FROM appPortBindings WHERE hostPort=? AND type=?', [ hostPort, type ], function (error, result) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (result.affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        callback(null);
    });
@@ -357,13 +385,12 @@ function del(id, callback) {
        { query: 'DELETE FROM subdomains WHERE appId = ?', args: [ id ] },
        { query: 'DELETE FROM appPortBindings WHERE appId = ?', args: [ id ] },
        { query: 'DELETE FROM appEnvVars WHERE appId = ?', args: [ id ] },
-        { query: 'DELETE FROM appPasswords WHERE identifier = ?', args: [ id ] },
        { query: 'DELETE FROM apps WHERE id = ?', args: [ id ] }
    ];

    database.transaction(queries, function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[4].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (results[3].affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        callback(null);
    });
@@ -379,7 +406,7 @@ function clear(callback) {
        database.query.bind(null, 'DELETE FROM appEnvVars'),
        database.query.bind(null, 'DELETE FROM apps')
    ], function (error) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        return callback(null);
    });
 }
@@ -435,7 +462,7 @@ function updateWithConstraints(id, app, constraints, callback) {

    var fields = [ ], values = [ ];
    for (var p in app) {
-        if (p === 'manifest' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode' || p === 'error' || p === 'reverseProxyConfig' || p === 'servicesConfig' || p === 'binds') {
+        if (p === 'manifest' || p === 'oldConfig' || p === 'updateConfig' || p === 'restoreConfig' || p === 'tags' || p === 'accessRestriction' || p === 'debugMode') {
            fields.push(`${p}Json = ?`);
            values.push(JSON.stringify(app[p]));
        } else if (p !== 'portBindings' && p !== 'location' && p !== 'domain' && p !== 'alternateDomains' && p !== 'env') {
@@ -450,14 +477,15 @@ function updateWithConstraints(id, app, constraints, callback) {
    }

    database.transaction(queries, function (error, results) {
-        if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results[results.length - 1].affectedRows !== 1) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS, error.message));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (results[results.length - 1].affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        return callback(null);
    });
 }

+// not sure if health should influence runState
 function setHealth(appId, health, healthTime, callback) {
    assert.strictEqual(typeof appId, 'string');
    assert.strictEqual(typeof health, 'string');
@@ -466,29 +494,60 @@ function setHealth(appId, health, healthTime, callback) {

    var values = { health, healthTime };

-    updateWithConstraints(appId, values, '', callback);
+    var constraints = 'AND runState NOT LIKE "pending_%" AND installationState = "installed"';
+
+    updateWithConstraints(appId, values, constraints, callback);
 }

-function setTask(appId, values, options, callback) {
+function setInstallationCommand(appId, installationState, values, callback) {
    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof values, 'object');
-    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof installationState, 'string');
+
+    if (typeof values === 'function') {
+        callback = values;
+        values = { };
+    } else {
+        assert.strictEqual(typeof values, 'object');
+        assert.strictEqual(typeof callback, 'function');
+    }
+
+    values.installationState = installationState;
+    values.installationProgress = '';
+
+    // Rules are:
+    // uninstall is allowed in any state
+    // force update is allowed in any state including pending_uninstall! (for better or worse)
+    // restore is allowed from installed or error state or currently restoring
+    // configure is allowed in installed state or currently configuring or in error state
+    // update and backup are allowed only in installed state
+
+    if (installationState === exports.ISTATE_PENDING_UNINSTALL || installationState === exports.ISTATE_PENDING_FORCE_UPDATE) {
+        updateWithConstraints(appId, values, '', callback);
+    } else if (installationState === exports.ISTATE_PENDING_RESTORE) {
+        updateWithConstraints(appId, values, 'AND (installationState = "installed" OR installationState = "error" OR installationState = "pending_restore")', callback);
+    } else if (installationState === exports.ISTATE_PENDING_UPDATE || installationState === exports.ISTATE_PENDING_BACKUP) {
+        updateWithConstraints(appId, values, 'AND installationState = "installed"', callback);
+    } else if (installationState === exports.ISTATE_PENDING_CONFIGURE) {
+        updateWithConstraints(appId, values, 'AND (installationState = "installed" OR installationState = "pending_configure" OR installationState = "error")', callback);
+    } else {
+        callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, 'invalid installationState'));
+    }
+}
+
+function setRunCommand(appId, runState, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof runState, 'string');
    assert.strictEqual(typeof callback, 'function');

-    if (!options.requireNullTaskId)  return updateWithConstraints(appId, values, '', callback);
-
-    if (options.requiredState === null) {
-        updateWithConstraints(appId, values, 'AND taskId IS NULL', callback);
-    } else {
-        updateWithConstraints(appId, values, `AND taskId IS NULL AND installationState = "${options.requiredState}"`, callback);
-    }
+    var values = { runState: runState };
+    updateWithConstraints(appId, values, 'AND runState NOT LIKE "pending_%" AND installationState = "installed"', callback);
 }

 function getAppStoreIds(callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT id, appStoreId FROM apps', function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
    });
@@ -513,7 +572,7 @@ function setAddonConfig(appId, addonId, env, callback) {
        }

        database.query(query + queryArgs.join(','), args, function (error) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            return callback(null);
        });
@@ -526,7 +585,7 @@ function unsetAddonConfig(appId, addonId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('DELETE FROM appAddonConfigs WHERE appId = ? AND addonId = ?', [ appId, addonId ], function (error) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null);
    });
@@ -537,7 +596,7 @@ function unsetAddonConfigByAppId(appId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('DELETE FROM appAddonConfigs WHERE appId = ?', [ appId ], function (error) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null);
    });
@@ -549,7 +608,7 @@ function getAddonConfig(appId, addonId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT name, value FROM appAddonConfigs WHERE appId = ? AND addonId = ?', [ appId, addonId ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
    });
@@ -560,36 +619,64 @@ function getAddonConfigByAppId(appId, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT name, value FROM appAddonConfigs WHERE appId = ?', [ appId ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null, results);
    });
 }

-function getAppIdByAddonConfigValue(addonId, namePattern, value, callback) {
+function getAppIdByAddonConfigValue(addonId, name, value, callback) {
    assert.strictEqual(typeof addonId, 'string');
-    assert.strictEqual(typeof namePattern, 'string');
+    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof value, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT appId FROM appAddonConfigs WHERE addonId = ? AND name LIKE ? AND value = ?', [ addonId, namePattern, value ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+    database.query('SELECT appId FROM appAddonConfigs WHERE addonId = ? AND name = ? AND value = ?', [ addonId, name, value ], function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        callback(null, results[0].appId);
    });
 }

-function getAddonConfigByName(appId, addonId, namePattern, callback) {
+function getAddonConfigByName(appId, addonId, name, callback) {
    assert.strictEqual(typeof appId, 'string');
    assert.strictEqual(typeof addonId, 'string');
-    assert.strictEqual(typeof namePattern, 'string');
+    assert.strictEqual(typeof name, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT value FROM appAddonConfigs WHERE appId = ? AND addonId = ? AND name LIKE ?', [ appId, addonId, namePattern ], function (error, results) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-        if (results.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'App not found'));
+    database.query('SELECT value FROM appAddonConfigs WHERE appId = ? AND addonId = ? AND name = ?', [ appId, addonId, name ], function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

        callback(null, results[0].value);
    });
 }
+
+function setOwner(appId, ownerId, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof ownerId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('UPDATE apps SET ownerId=? WHERE appId=?', [ ownerId, appId ], function (error, results) {
+        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new DatabaseError(DatabaseError.NOT_FOUND, 'No such user'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        if (results.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND, 'No such app'));
+
+        callback(null);
+    });
+}
+
+function transferOwnership(oldOwnerId, newOwnerId, callback) {
+    assert.strictEqual(typeof oldOwnerId, 'string');
+    assert.strictEqual(typeof newOwnerId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('UPDATE apps SET ownerId=? WHERE ownerId=?', [ newOwnerId, oldOwnerId ], function (error) {
+        if (error && error.code === 'ER_NO_REFERENCED_ROW_2') return callback(new DatabaseError(DatabaseError.NOT_FOUND, 'No such user'));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
@@ -5,7 +5,7 @@ var appdb = require('./appdb.js'),
    assert = require('assert'),
    async = require('async'),
    auditSource = require('./auditsource.js'),
-    BoxError = require('./boxerror.js'),
+    DatabaseError = require('./databaseerror.js'),
    debug = require('debug')('box:apphealthmonitor'),
    docker = require('./docker.js'),
    eventlog = require('./eventlog.js'),
@@ -26,7 +26,7 @@ let gLastOomMailTime = Date.now() - (5 * 60 * 1000); // pretend we sent email 5
 function debugApp(app) {
    assert(typeof app === 'object');

-    debug(app.fqdn + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)) + ' - ' + app.id);
+    debug(app.fqdn + ' ' + app.manifest.id + ' ' + util.format.apply(util, Array.prototype.slice.call(arguments, 1)) + ' - ' + app.id);
 }

 function setHealth(app, health, callback) {
@@ -36,16 +36,16 @@ function setHealth(app, health, callback) {

    let now = new Date(), healthTime = app.healthTime, curHealth = app.health;

-    if (health === apps.HEALTH_HEALTHY) {
+    if (health === appdb.HEALTH_HEALTHY) {
        healthTime = now;
-        if (curHealth && curHealth !== apps.HEALTH_HEALTHY) { // app starts out with null health
+        if (curHealth && curHealth !== appdb.HEALTH_HEALTHY) { // app starts out with null health
            debugApp(app, 'app switched from %s to healthy', curHealth);

            // do not send mails for dev apps
            if (!app.debugMode) eventlog.add(eventlog.ACTION_APP_UP, auditSource.HEALTH_MONITOR, { app: app });
        }
    } else if (Math.abs(now - healthTime) > UNHEALTHY_THRESHOLD) {
-        if (curHealth === apps.HEALTH_HEALTHY) {
+        if (curHealth === appdb.HEALTH_HEALTHY) {
            debugApp(app, 'marking as unhealthy since not seen for more than %s minutes', UNHEALTHY_THRESHOLD/(60 * 1000));

            // do not send mails for dev apps
@@ -57,7 +57,7 @@ function setHealth(app, health, callback) {
    }

    appdb.setHealth(app.id, health, healthTime, function (error) {
-        if (error && error.reason === BoxError.NOT_FOUND) return callback(null); // app uninstalled?
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null); // app uninstalled?
        if (error) return callback(error);

        app.health = health;
@@ -72,34 +72,44 @@ function checkAppHealth(app, callback) {
    assert.strictEqual(typeof app, 'object');
    assert.strictEqual(typeof callback, 'function');

-    if (app.installationState !== apps.ISTATE_INSTALLED || app.runState !== apps.RSTATE_RUNNING) {
+    if (app.installationState !== appdb.ISTATE_INSTALLED || app.runState !== appdb.RSTATE_RUNNING) {
+        debugApp(app, 'skipped. istate:%s rstate:%s', app.installationState, app.runState);
        return callback(null);
    }

    const manifest = app.manifest;

    docker.inspect(app.containerId, function (error, data) {
-        if (error || !data || !data.State) return setHealth(app, apps.HEALTH_ERROR, callback);
-        if (data.State.Running !== true) return setHealth(app, apps.HEALTH_DEAD, callback);
+        if (error || !data || !data.State) {
+            debugApp(app, 'Error inspecting container');
+            return setHealth(app, appdb.HEALTH_ERROR, callback);
+        }
+
+        if (data.State.Running !== true) {
+            debugApp(app, 'exited');
+            return setHealth(app, appdb.HEALTH_DEAD, callback);
+        }

        // non-appstore apps may not have healthCheckPath
-        if (!manifest.healthCheckPath) return setHealth(app, apps.HEALTH_HEALTHY, callback);
+        if (!manifest.healthCheckPath) return setHealth(app, appdb.HEALTH_HEALTHY, callback);

        // poll through docker network instead of nginx to bypass any potential oauth proxy
        var healthCheckUrl = 'http://127.0.0.1:' + app.httpPort + manifest.healthCheckPath;
        superagent
            .get(healthCheckUrl)
            .set('Host', app.fqdn) // required for some apache configs with rewrite rules
-            .set('User-Agent', 'Mozilla (CloudronHealth)') // required for some apps (e.g. minio)
+            .set('User-Agent', 'Mozilla') // required for some apps (e.g. minio)
            .redirects(0)
            .timeout(HEALTHCHECK_INTERVAL)
            .end(function (error, res) {
                if (error && !error.response) {
-                    setHealth(app, apps.HEALTH_UNHEALTHY, callback);
+                    debugApp(app, 'not alive (network error): %s', error.message);
+                    setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
                } else if (res.statusCode >= 400) { // 2xx and 3xx are ok
-                    setHealth(app, apps.HEALTH_UNHEALTHY, callback);
+                    debugApp(app, 'not alive : %s', error || res.status);
+                    setHealth(app, appdb.HEALTH_UNHEALTHY, callback);
                } else {
-                    setHealth(app, apps.HEALTH_HEALTHY, callback);
+                    setHealth(app, appdb.HEALTH_HEALTHY, callback);
                }
            });
    });
@@ -170,14 +180,17 @@ function processDockerEvents(intervalSecs, callback) {
 function processApp(callback) {
    assert.strictEqual(typeof callback, 'function');

-    apps.getAll(function (error, allApps) {
+    apps.getAll(function (error, result) {
        if (error) return callback(error);

-        async.each(allApps, checkAppHealth, function (error) {
-            const alive = allApps
-                .filter(function (a) { return a.installationState === apps.ISTATE_INSTALLED && a.runState === apps.RSTATE_RUNNING && a.health === apps.HEALTH_HEALTHY; });
+        async.each(result, checkAppHealth, function (error) {
+            if (error) console.error(error);

-            debug(`app health: ${alive.length} alive / ${allApps.length - alive.length} dead.` + (error ? ` ${error.reason}` : ''));
+            var alive = result
+                .filter(function (a) { return a.installationState === appdb.ISTATE_INSTALLED && a.runState === appdb.RSTATE_RUNNING && a.health === appdb.HEALTH_HEALTHY; })
+                .map(function (a) { return (a.location || 'naked_domain') + '|' + a.manifest.id; }).join(', ');
+
+            debug('apps alive: [%s]', alive);

            callback(null);
        });
@@ -192,7 +205,7 @@ function run(intervalSecs, callback) {
        processApp, // this is first because docker.getEvents seems to get 'stuck' sometimes
        processDockerEvents.bind(null, intervalSecs)
    ], function (error) {
-        if (error) debug(`run: could not check app health. ${error.message}`);
+        if (error) debug(error);

        callback();
    });
@@ -1,83 +1,83 @@
 'use strict';

 exports = module.exports = {
-    getFeatures: getFeatures,
-
    getApps: getApps,
    getApp: getApp,
    getAppVersion: getAppVersion,

-    trackBeginSetup: trackBeginSetup,
-    trackFinishedSetup: trackFinishedSetup,
-
    registerWithLoginCredentials: registerWithLoginCredentials,
+    registerWithLicense: registerWithLicense,

    purchaseApp: purchaseApp,
    unpurchaseApp: unpurchaseApp,

-    getUserToken: getUserToken,
    getSubscription: getSubscription,
    isFreePlan: isFreePlan,

+    sendAliveStatus: sendAliveStatus,
+
    getAppUpdate: getAppUpdate,
    getBoxUpdate: getBoxUpdate,

-    createTicket: createTicket
+    createTicket: createTicket,
+
+    AppstoreError: AppstoreError
 };

 var apps = require('./apps.js'),
    assert = require('assert'),
    async = require('async'),
-    BoxError = require('./boxerror.js'),
-    constants = require('./constants.js'),
+    config = require('./config.js'),
+    custom = require('./custom.js'),
    debug = require('debug')('box:appstore'),
+    domains = require('./domains.js'),
    eventlog = require('./eventlog.js'),
-    paths = require('./paths.js'),
+    mail = require('./mail.js'),
+    os = require('os'),
    safe = require('safetydance'),
    semver = require('semver'),
    settings = require('./settings.js'),
    superagent = require('superagent'),
    util = require('util');

-// These are the default options and will be adjusted once a subscription state is obtained
-// Keep in sync with appstore/routes/cloudrons.js
-let gFeatures = {
-    userMaxCount: 5,
-    domainMaxCount: 1,
-    externalLdap: false,
-    privateDockerRegistry: false,
-    branding: false,
-    support: false,
-    directoryConfig: false,
-    mailboxMaxCount: 5,
-    emailPremium: false
-};
+function AppstoreError(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');

-// attempt to load feature cache in case appstore would be down
-let tmp = safe.JSON.parse(safe.fs.readFileSync(paths.FEATURES_INFO_FILE, 'utf8'));
-if (tmp) gFeatures = tmp;
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);

-function getFeatures() {
-    return gFeatures;
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
 }
+util.inherits(AppstoreError, Error);
+AppstoreError.INTERNAL_ERROR = 'Internal Error';
+AppstoreError.EXTERNAL_ERROR = 'External Error';
+AppstoreError.ALREADY_EXISTS = 'Already Exists';
+AppstoreError.ACCESS_DENIED = 'Access Denied';
+AppstoreError.NOT_FOUND = 'Not Found';
+AppstoreError.PLAN_LIMIT = 'Plan limit reached'; // upstream 402 (subsciption_expired and subscription_required)
+AppstoreError.LICENSE_ERROR = 'License Error'; // upstream 422 (no license, invalid license)
+AppstoreError.INVALID_TOKEN = 'Invalid token'; // upstream 401 (invalid token)
+AppstoreError.NOT_REGISTERED = 'Not registered'; // upstream 412 (no token, not set yet)
+AppstoreError.ALREADY_REGISTERED = 'Already registered';

-function isAppAllowed(appstoreId, listingConfig) {
-    assert.strictEqual(typeof listingConfig, 'object');
-    assert.strictEqual(typeof appstoreId, 'string');
-
-    if (listingConfig.blacklist && listingConfig.blacklist.includes(appstoreId)) return false;
-
-    if (listingConfig.whitelist) return listingConfig.whitelist.includes(appstoreId);
-
-    return true;
-}
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };

 function getCloudronToken(callback) {
    assert.strictEqual(typeof callback, 'function');

    settings.getCloudronToken(function (error, token) {
-        if (error) return callback(error);
-        if (!token) return callback(new BoxError(BoxError.LICENSE_ERROR, 'Missing token'));
+        if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+        if (!token) return callback(new AppstoreError(AppstoreError.NOT_REGISTERED));

        callback(null, token);
    });
@@ -95,11 +95,11 @@ function login(email, password, totpToken, callback) {
        totpToken: totpToken
    };

-    const url = settings.apiServerOrigin() + '/api/v1/login';
+    const url = config.apiServerOrigin() + '/api/v1/login';
    superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `login status code: ${result.statusCode}`));
+        if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+        if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.ACCESS_DENIED));
+        if (result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, `login status code: ${result.statusCode}`));

        callback(null, result.body); // { userId, accessToken }
    });
@@ -115,54 +115,31 @@ function registerUser(email, password, callback) {
        password: password,
    };

-    const url = settings.apiServerOrigin() + '/api/v1/register_user';
+    const url = config.apiServerOrigin() + '/api/v1/register_user';
    superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode === 409) return callback(new BoxError(BoxError.ALREADY_EXISTS, error.message));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `register status code: ${result.statusCode}`));
+        if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+        if (result.statusCode === 409) return callback(new AppstoreError(AppstoreError.ALREADY_EXISTS));
+        if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, `register status code: ${result.statusCode}`));

        callback(null);
    });
 }

-function getUserToken(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    if (settings.isDemo()) return callback(new BoxError(BoxError.BAD_FIELD, 'Not allowed in demo mode'));
-
-    getCloudronToken(function (error, token) {
-        if (error) return callback(error);
-
-        const url = `${settings.apiServerOrigin()}/api/v1/user_token`;
-
-        superagent.post(url).send({}).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `getUserToken status code: ${result.status}`));
-
-            callback(null, result.body.accessToken);
-        });
-    });
-}
-
 function getSubscription(callback) {
    assert.strictEqual(typeof callback, 'function');

    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        const url = settings.apiServerOrigin() + '/api/v1/subscription';
+        const url = config.apiServerOrigin() + '/api/v1/subscription';
        superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-            if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR));
-            if (result.statusCode === 502) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Stripe error: ${error.message}`));
-            if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unknown error: ${error.message}`));
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+            if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR));
+            if (result.statusCode === 502) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, `Stripe error: ${error.message}`));
+            if (result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, `Unknown error: ${error.message}`));

-            // update the features cache
-            gFeatures = result.body.features;
-            safe.fs.writeFileSync(paths.FEATURES_INFO_FILE, JSON.stringify(gFeatures), 'utf8');
-
-            callback(null, result.body);
+            callback(null, result.body); // { email, subscription }
        });
    });
 }
@@ -181,16 +158,16 @@ function purchaseApp(data, callback) {
    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        const url = `${settings.apiServerOrigin()}/api/v1/cloudronapps`;
+        const url = `${config.apiServerOrigin()}/api/v1/cloudronapps`;

        superagent.post(url).send(data).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND)); // appstoreId does not exist
-            if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-            if (result.statusCode === 402) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-            if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+            if (result.statusCode === 404) return callback(new AppstoreError(AppstoreError.NOT_FOUND)); // appstoreId does not exist
+            if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 402) return callback(new AppstoreError(AppstoreError.PLAN_LIMIT, result.body.message));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
            // 200 if already purchased, 201 is newly purchased
-            if (result.statusCode !== 201 && result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('App purchase failed. %s %j', result.status, result.body)));
+            if (result.statusCode !== 201 && result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App purchase failed. %s %j', result.status, result.body)));

            callback(null);
        });
@@ -206,19 +183,19 @@ function unpurchaseApp(appId, data, callback) {
    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        const url = `${settings.apiServerOrigin()}/api/v1/cloudronapps/${appId}`;
+        const url = `${config.apiServerOrigin()}/api/v1/cloudronapps/${appId}`;

        superagent.get(url).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
            if (result.statusCode === 404) return callback(null);   // was never purchased
-            if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-            if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-            if (result.statusCode !== 201 && result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));
+            if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
+            if (result.statusCode !== 201 && result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));

            superagent.del(url).send(data).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode !== 204) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));
+                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+                if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+                if (result.statusCode !== 204) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App unpurchase failed. %s %j', result.status, result.body)));

                callback(null);
            });
@@ -226,70 +203,143 @@ function unpurchaseApp(appId, data, callback) {
    });
 }

-function getBoxUpdate(options, callback) {
-    assert.strictEqual(typeof options, 'object');
+function sendAliveStatus(callback) {
+    callback = callback || NOOP_CALLBACK;
+
+    var allSettings, allDomains, mailDomains, loginEvents;
+
+    async.series([
+        function (callback) {
+            settings.getAll(function (error, result) {
+                if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+                allSettings = result;
+                callback();
+            });
+        },
+        function (callback) {
+            domains.getAll(function (error, result) {
+                if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+                allDomains = result;
+                callback();
+            });
+        },
+        function (callback) {
+            mail.getDomains(function (error, result) {
+                if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+                mailDomains = result;
+                callback();
+            });
+        },
+        function (callback) {
+            eventlog.getAllPaged([ eventlog.ACTION_USER_LOGIN ], null, 1, 1, function (error, result) {
+                if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+                loginEvents = result;
+                callback();
+            });
+        }
+    ], function (error) {
+        if (error) return callback(error);
+
+        var backendSettings = {
+            backupConfig: {
+                provider: allSettings[settings.BACKUP_CONFIG_KEY].provider,
+                hardlinks: !allSettings[settings.BACKUP_CONFIG_KEY].noHardlinks
+            },
+            domainConfig: {
+                count: allDomains.length,
+                domains: Array.from(new Set(allDomains.map(function (d) { return { domain: d.domain, provider: d.provider }; })))
+            },
+            mailConfig: {
+                outboundCount: mailDomains.length,
+                inboundCount: mailDomains.filter(function (d) { return d.enabled; }).length,
+                catchAllCount: mailDomains.filter(function (d) { return d.catchAll.length !== 0; }).length,
+                relayProviders: Array.from(new Set(mailDomains.map(function (d) { return d.relay.provider; })))
+            },
+            appAutoupdatePattern: allSettings[settings.APP_AUTOUPDATE_PATTERN_KEY],
+            boxAutoupdatePattern: allSettings[settings.BOX_AUTOUPDATE_PATTERN_KEY],
+            timeZone: allSettings[settings.TIME_ZONE_KEY],
+        };
+
+        var data = {
+            version: config.version(),
+            adminFqdn: config.adminFqdn(),
+            provider: config.provider(),
+            backendSettings: backendSettings,
+            machine: {
+                cpus: os.cpus(),
+                totalmem: os.totalmem()
+            },
+            events: {
+                lastLogin: loginEvents[0] ? (new Date(loginEvents[0].creationTime).getTime()) : 0
+            }
+        };
+
+        getCloudronToken(function (error, token) {
+            if (error) return callback(error);
+
+            const url = `${config.apiServerOrigin()}/api/v1/alive`;
+            superagent.post(url).send(data).query({ accessToken: token }).timeout(30 * 1000).end(function (error, result) {
+                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+                if (result.statusCode === 404) return callback(new AppstoreError(AppstoreError.NOT_FOUND));
+                if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+                if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
+                if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Sending alive status failed. %s %j', result.status, result.body)));
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function getBoxUpdate(callback) {
    assert.strictEqual(typeof callback, 'function');

    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        const url = `${settings.apiServerOrigin()}/api/v1/boxupdate`;
+        const url = `${config.apiServerOrigin()}/api/v1/boxupdate`;

-        const query = {
-            accessToken: token,
-            boxVersion: constants.VERSION,
-            automatic: options.automatic
-        };
-
-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-            if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-            if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+        superagent.get(url).query({ accessToken: token, boxVersion: config.version() }).timeout(10 * 1000).end(function (error, result) {
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+            if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
            if (result.statusCode === 204) return callback(null); // no update
-            if (result.statusCode !== 200 || !result.body) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+            if (result.statusCode !== 200 || !result.body) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));

            var updateInfo = result.body;

-            if (!semver.valid(updateInfo.version) || semver.gt(constants.VERSION, updateInfo.version)) {
-                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Invalid update version: %s %s', result.statusCode, result.text)));
+            if (!semver.valid(updateInfo.version) || semver.gt(config.version(), updateInfo.version)) {
+                return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Invalid update version: %s %s', result.statusCode, result.text)));
            }

            // updateInfo: { version, changelog, sourceTarballUrl, sourceTarballSigUrl, boxVersionsUrl, boxVersionsSigUrl }
-            if (!updateInfo.version || typeof updateInfo.version !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad version): %s %s', result.statusCode, result.text)));
-            if (!updateInfo.changelog || !Array.isArray(updateInfo.changelog)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad version): %s %s', result.statusCode, result.text)));
-            if (!updateInfo.sourceTarballUrl || typeof updateInfo.sourceTarballUrl !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad sourceTarballUrl): %s %s', result.statusCode, result.text)));
-            if (!updateInfo.sourceTarballSigUrl || typeof updateInfo.sourceTarballSigUrl !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad sourceTarballSigUrl): %s %s', result.statusCode, result.text)));
-            if (!updateInfo.boxVersionsUrl || typeof updateInfo.boxVersionsUrl !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad boxVersionsUrl): %s %s', result.statusCode, result.text)));
-            if (!updateInfo.boxVersionsSigUrl || typeof updateInfo.boxVersionsSigUrl !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response (bad boxVersionsSigUrl): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.version || typeof updateInfo.version !== 'string') return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad version): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.changelog || !Array.isArray(updateInfo.changelog)) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad version): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.sourceTarballUrl || typeof updateInfo.sourceTarballUrl !== 'string') return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad sourceTarballUrl): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.sourceTarballSigUrl || typeof updateInfo.sourceTarballSigUrl !== 'string') return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad sourceTarballSigUrl): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.boxVersionsUrl || typeof updateInfo.boxVersionsUrl !== 'string') return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad boxVersionsUrl): %s %s', result.statusCode, result.text)));
+            if (!updateInfo.boxVersionsSigUrl || typeof updateInfo.boxVersionsSigUrl !== 'string') return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response (bad boxVersionsSigUrl): %s %s', result.statusCode, result.text)));

            callback(null, updateInfo);
        });
    });
 }

-function getAppUpdate(app, options, callback) {
+function getAppUpdate(app, callback) {
    assert.strictEqual(typeof app, 'object');
-    assert.strictEqual(typeof options, 'object');
    assert.strictEqual(typeof callback, 'function');

    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        const url = `${settings.apiServerOrigin()}/api/v1/appupdate`;
-        const query = {
-            accessToken: token,
-            boxVersion: constants.VERSION,
-            appId: app.appStoreId,
-            appVersion: app.manifest.version,
-            automatic: options.automatic
-        };
+        const url = `${config.apiServerOrigin()}/api/v1/appupdate`;

-        superagent.get(url).query(query).timeout(10 * 1000).end(function (error, result) {
-            if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error));
-            if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-            if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
+        superagent.get(url).query({ accessToken: token, boxVersion: config.version(), appId: app.appStoreId, appVersion: app.manifest.version }).timeout(10 * 1000).end(function (error, result) {
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+            if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
            if (result.statusCode === 204) return callback(null); // no update
-            if (result.statusCode !== 200 || !result.body) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+            if (result.statusCode !== 200 || !result.body) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));

            const updateInfo = result.body;

@@ -299,12 +349,10 @@ function getAppUpdate(app, options, callback) {
            // do some sanity checks
            if (!safe.query(updateInfo, 'manifest.version') || semver.gt(curAppVersion, safe.query(updateInfo, 'manifest.version'))) {
                debug('Skipping malformed update of app %s version: %s. got %j', app.id, curAppVersion, updateInfo);
-                return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Malformed update: %s %s', result.statusCode, result.text)));
+                return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Malformed update: %s %s', result.statusCode, result.text)));
            }

-            updateInfo.unstable = !!updateInfo.unstable;
-
-            // { id, creationDate, manifest, unstable }
+            // { id, creationDate, manifest }
            callback(null, updateInfo);
        });
    });
@@ -314,23 +362,23 @@ function registerCloudron(data, callback) {
    assert.strictEqual(typeof data, 'object');
    assert.strictEqual(typeof callback, 'function');

-    const url = `${settings.apiServerOrigin()}/api/v1/register_cloudron`;
+    const url = `${config.apiServerOrigin()}/api/v1/register_cloudron`;

    superagent.post(url).send(data).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Unable to register cloudron: ${result.statusCode} ${error.message}`));
+        if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+        if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, `Unable to register cloudron: ${error.message}`));

        // cloudronId, token, licenseKey
-        if (!result.body.cloudronId) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response - no cloudron id'));
-        if (!result.body.cloudronToken) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response - no token'));
-        if (!result.body.licenseKey) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response - no license'));
+        if (!result.body.cloudronId) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, 'Invalid response - no cloudron id'));
+        if (!result.body.cloudronToken) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, 'Invalid response - no token'));
+        if (!result.body.licenseKey) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, 'Invalid response - no license'));

        async.series([
            settings.setCloudronId.bind(null, result.body.cloudronId),
            settings.setCloudronToken.bind(null, result.body.cloudronToken),
            settings.setLicenseKey.bind(null, result.body.licenseKey),
        ], function (error) {
-            if (error) return callback(error);
+            if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));

            debug(`registerCloudron: Cloudron registered with id ${result.body.cloudronId}`);

@@ -339,30 +387,14 @@ function registerCloudron(data, callback) {
    });
 }

-// This works without a Cloudron token as this Cloudron was not yet registered
-let gBeginSetupAlreadyTracked = false;
-function trackBeginSetup() {
-    // avoid browser reload double tracking, not perfect since box might restart, but covers most cases and is simple
-    if (gBeginSetupAlreadyTracked) return;
-    gBeginSetupAlreadyTracked = true;
+function registerWithLicense(license, callback) {
+    assert.strictEqual(typeof license, 'string');
+    assert.strictEqual(typeof callback, 'function');

-    const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_begin`;
+    getCloudronToken(function (error, token) {
+        if (token) return callback(new AppstoreError(AppstoreError.ALREADY_REGISTERED));

-    superagent.post(url).send({}).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return debug(`trackBeginSetup: ${error.message}`);
-        if (result.statusCode !== 200) return debug(`trackBeginSetup: ${result.statusCode} ${error.message}`);
-    });
-}
-
-// This works without a Cloudron token as this Cloudron was not yet registered
-function trackFinishedSetup(domain) {
-    assert.strictEqual(typeof domain, 'string');
-
-    const url = `${settings.apiServerOrigin()}/api/v1/helper/setup_finished`;
-
-    superagent.post(url).send({ domain }).timeout(30 * 1000).end(function (error, result) {
-        if (error && !error.response) return debug(`trackFinishedSetup: ${error.message}`);
-        if (result.statusCode !== 200) return debug(`trackFinishedSetup: ${result.statusCode} ${error.message}`);
+        registerCloudron({ license }, callback);
    });
 }

@@ -377,7 +409,7 @@ function registerWithLoginCredentials(options, callback) {
    }

    getCloudronToken(function (error, token) {
-        if (token) return callback(new BoxError(BoxError.CONFLICT, 'Cloudron is already registered'));
+        if (token) return callback(new AppstoreError(AppstoreError.ALREADY_REGISTERED));

        maybeSignup(function (error) {
            if (error) return callback(error);
@@ -385,20 +417,19 @@ function registerWithLoginCredentials(options, callback) {
            login(options.email, options.password, options.totpToken || '', function (error, result) {
                if (error) return callback(error);

-                registerCloudron({ domain: settings.adminDomain(), accessToken: result.accessToken, version: constants.VERSION, purpose: options.purpose || '' }, callback);
+                registerCloudron({ domain: config.adminDomain(), accessToken: result.accessToken }, callback);
            });
        });
    });
 }

-function createTicket(info, auditSource, callback) {
+function createTicket(info, callback) {
    assert.strictEqual(typeof info, 'object');
    assert.strictEqual(typeof info.email, 'string');
    assert.strictEqual(typeof info.displayName, 'string');
    assert.strictEqual(typeof info.type, 'string');
    assert.strictEqual(typeof info.subject, 'string');
    assert.strictEqual(typeof info.description, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

    function collectAppInfoIfNeeded(callback) {
@@ -410,22 +441,20 @@ function createTicket(info, auditSource, callback) {
        if (error) return callback(error);

        collectAppInfoIfNeeded(function (error, result) {
-            if (error) return callback(error);
+            if (error) console.error('Unable to get app info', error);
            if (result) info.app = result;

-            let url = settings.apiServerOrigin() + '/api/v1/ticket';
+            let url = config.apiServerOrigin() + '/api/v1/ticket';

-            info.supportEmail = constants.SUPPORT_EMAIL; // destination address for tickets
+            info.supportEmail = custom.supportEmail(); // destination address for tickets

            superagent.post(url).query({ accessToken: token }).send(info).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
+                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+                if (result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+                if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
+                if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));

-                eventlog.add(eventlog.ACTION_SUPPORT_TICKET, auditSource, info);
-
-                callback(null, { message: `An email for sent to ${constants.SUPPORT_EMAIL}. We will get back shortly!` });
+                callback(null);
            });
        });
    });
@@ -438,23 +467,16 @@ function getApps(callback) {
        if (error) return callback(error);

        settings.getUnstableAppsConfig(function (error, unstable) {
-            if (error) return callback(error);
+            if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+            const url = `${config.apiServerOrigin()}/api/v1/apps`;
+            superagent.get(url).query({ accessToken: token, boxVersion: config.version(), unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
+                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+                if (result.statusCode === 403 || result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+                if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
+                if (result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App listing failed. %s %j', result.status, result.body)));
+                if (!result.body.apps) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));

-            const url = `${settings.apiServerOrigin()}/api/v1/apps`;
-            superagent.get(url).query({ accessToken: token, boxVersion: constants.VERSION, unstable: unstable }).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('App listing failed. %s %j', result.status, result.body)));
-                if (!result.body.apps) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Bad response: %s %s', result.statusCode, result.text)));
-
-                settings.getAppstoreListingConfig(function (error, listingConfig) {
-                    if (error) return callback(error);
-
-                    const filteredApps = result.body.apps.filter(app => isAppAllowed(app.id, listingConfig));
-
-                    callback(null, filteredApps);
-                });
+                callback(null, result.body.apps);
            });
        });
    });
@@ -465,26 +487,20 @@ function getAppVersion(appId, version, callback) {
    assert.strictEqual(typeof version, 'string');
    assert.strictEqual(typeof callback, 'function');

-    settings.getAppstoreListingConfig(function (error, listingConfig) {
+    getCloudronToken(function (error, token) {
        if (error) return callback(error);

-        if (!isAppAllowed(appId, listingConfig)) return callback(new BoxError(BoxError.FEATURE_DISABLED));
+        let url = `${config.apiServerOrigin()}/api/v1/apps/${appId}`;
+        if (version !== 'latest') url += `/versions/${version}`;

-        getCloudronToken(function (error, token) {
-            if (error) return callback(error);
+        superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
+            if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error.message));
+            if (result.statusCode === 403 || result.statusCode === 401) return callback(new AppstoreError(AppstoreError.INVALID_TOKEN));
+            if (result.statusCode === 404) return callback(new AppstoreError(AppstoreError.NOT_FOUND));
+            if (result.statusCode === 422) return callback(new AppstoreError(AppstoreError.LICENSE_ERROR, result.body.message));
+            if (result.statusCode !== 200) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('App fetch failed. %s %j', result.status, result.body)));

-            let url = `${settings.apiServerOrigin()}/api/v1/apps/${appId}`;
-            if (version !== 'latest') url += `/versions/${version}`;
-
-            superagent.get(url).query({ accessToken: token }).timeout(10 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new BoxError(BoxError.NETWORK_ERROR, error.message));
-                if (result.statusCode === 403 || result.statusCode === 401) return callback(new BoxError(BoxError.INVALID_CREDENTIALS));
-                if (result.statusCode === 404) return callback(new BoxError(BoxError.NOT_FOUND));
-                if (result.statusCode === 422) return callback(new BoxError(BoxError.LICENSE_ERROR, result.body.message));
-                if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('App fetch failed. %s %j', result.status, result.body)));
-
-                callback(null, result.body);
-            });
+            callback(null, result.body);
        });
    });
 }
@@ -1,94 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    scheduleTask: scheduleTask
-};
-
-let assert = require('assert'),
-    BoxError = require('./boxerror.js'),
-    debug = require('debug')('box:apptaskmanager'),
-    fs = require('fs'),
-    locker = require('./locker.js'),
-    safe = require('safetydance'),
-    path = require('path'),
-    paths = require('./paths.js'),
-    sftp = require('./sftp.js'),
-    tasks = require('./tasks.js');
-
-let gActiveTasks = { }; // indexed by app id
-let gPendingTasks = [ ];
-let gInitialized = false;
-
-const TASK_CONCURRENCY = 3;
-const NOOP_CALLBACK = function (error) { if (error) debug(error); };
-
-function waitText(lockOperation) {
-    if (lockOperation === locker.OP_BOX_UPDATE) return 'Waiting for Cloudron to finish updating. See the Settings view';
-    if (lockOperation === locker.OP_PLATFORM_START) return 'Waiting for Cloudron to initialize';
-    if (lockOperation === locker.OP_FULL_BACKUP) return 'Wait for Cloudron to finish backup. See the Backups view';
-
-    return ''; // cannot happen
-}
-
-function initializeSync() {
-    gInitialized = true;
-    locker.on('unlocked', startNextTask);
-}
-
-// callback is called when task is finished
-function scheduleTask(appId, taskId, callback) {
-    assert.strictEqual(typeof appId, 'string');
-    assert.strictEqual(typeof taskId, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    if (!gInitialized) initializeSync();
-
-    if (appId in gActiveTasks) {
-        return callback(new BoxError(BoxError.CONFLICT, `Task for %s is already active: ${appId}`));
-    }
-
-    if (Object.keys(gActiveTasks).length >= TASK_CONCURRENCY) {
-        debug(`Reached concurrency limit, queueing task id ${taskId}`);
-        tasks.update(taskId, { percent: 1, message: 'Waiting for other app tasks to complete' }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
-        return;
-    }
-
-    var lockError = locker.recursiveLock(locker.OP_APPTASK);
-
-    if (lockError) {
-        debug(`Could not get lock. ${lockError.message}, queueing task id ${taskId}`);
-        tasks.update(taskId, { percent: 1, message: waitText(lockError.operation) }, NOOP_CALLBACK);
-        gPendingTasks.push({ appId, taskId, callback });
-        return;
-    }
-
-    gActiveTasks[appId] = {};
-
-    const logFile = path.join(paths.LOG_DIR, appId, 'apptask.log');
-
-    if (!fs.existsSync(path.dirname(logFile))) safe.fs.mkdirSync(path.dirname(logFile)); // ensure directory
-
-    // TODO: set memory limit for app backup task
-    tasks.startTask(taskId, { logFile, timeout: 20 * 60 * 60 * 1000 /* 20 hours */, nice: 15 }, function (error, result) {
-        callback(error, result);
-
-        delete gActiveTasks[appId];
-        locker.unlock(locker.OP_APPTASK); // unlock event will trigger next task
-
-        // post app task hooks
-        sftp.rebuild(function (error) {
-            if (error) console.error('Unable to rebuild sftp:', error);
-        });
-    });
-}
-
-function startNextTask() {
-    if (gPendingTasks.length === 0) return;
-
-    assert(Object.keys(gActiveTasks).length < TASK_CONCURRENCY);
-
-    const t = gPendingTasks.shift();
-    scheduleTask(t.appId, t.taskId, t.callback);
-}
-
@@ -3,9 +3,9 @@
 exports = module.exports = {
    CRON:  { userId: null, username: 'cron' },
    HEALTH_MONITOR: { userId: null, username: 'healthmonitor' },
+    SYSADMIN: { userId: null, username: 'sysadmin' },
+    TASK_MANAGER: { userId: null, username: 'taskmanager' },
    APP_TASK: { userId: null, username: 'apptask' },
-    EXTERNAL_LDAP_TASK: { userId: null, username: 'externalldap' },
-    EXTERNAL_LDAP_AUTO_CREATE: { userId: null, username: 'externalldap' },

    fromRequest: fromRequest
 };
@@ -0,0 +1,78 @@
+/* jslint node:true */
+
+'use strict';
+
+exports = module.exports = {
+    get: get,
+    add: add,
+    del: del,
+    delExpired: delExpired,
+
+    _clear: clear
+};
+
+var assert = require('assert'),
+    database = require('./database.js'),
+    DatabaseError = require('./databaseerror');
+
+var AUTHCODES_FIELDS = [ 'authCode', 'userId', 'clientId', 'expiresAt' ].join(',');
+
+function get(authCode, callback) {
+    assert.strictEqual(typeof authCode, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + AUTHCODES_FIELDS + ' FROM authcodes WHERE authCode = ? AND expiresAt > ?', [ authCode, Date.now() ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null, result[0]);
+    });
+}
+
+function add(authCode, clientId, userId, expiresAt, callback) {
+    assert.strictEqual(typeof authCode, 'string');
+    assert.strictEqual(typeof clientId, 'string');
+    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof expiresAt, 'number');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('INSERT INTO authcodes (authCode, clientId, userId, expiresAt) VALUES (?, ?, ?, ?)',
+            [ authCode, clientId, userId, expiresAt ], function (error, result) {
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
+        if (error || result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function del(authCode, callback) {
+    assert.strictEqual(typeof authCode, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM authcodes WHERE authCode = ?', [ authCode ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
+function delExpired(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM authcodes WHERE expiresAt <= ?', [ Date.now() ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        return callback(null, result.affectedRows);
+    });
+}
+
+function clear(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM authcodes', function (error) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
@@ -1,25 +1,32 @@
 'use strict';

 var assert = require('assert'),
-    BoxError = require('./boxerror.js'),
    database = require('./database.js'),
+    DatabaseError = require('./databaseerror.js'),
    safe = require('safetydance'),
    util = require('util');

-var BACKUPS_FIELDS = [ 'id', 'identifier', 'creationTime', 'packageVersion', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs', 'encryptionVersion' ];
+var BACKUPS_FIELDS = [ 'id', 'creationTime', 'version', 'type', 'dependsOn', 'state', 'manifestJson', 'format', 'preserveSecs' ];

 exports = module.exports = {
-    add,
+    add: add,

-    getByTypePaged,
-    getByIdentifierPaged,
-    getByIdentifierAndStatePaged,
+    getByTypeAndStatePaged: getByTypeAndStatePaged,
+    getByTypePaged: getByTypePaged,

-    get,
-    del,
-    update,
+    get: get,
+    del: del,
+    update: update,
+    getByAppIdPaged: getByAppIdPaged,

-    _clear: clear
+    _clear: clear,
+
+    BACKUP_TYPE_APP: 'app',
+    BACKUP_TYPE_BOX: 'box',
+
+    BACKUP_STATE_NORMAL: 'normal', // should rename to created to avoid listing in UI?
+    BACKUP_STATE_CREATING: 'creating',
+    BACKUP_STATE_ERROR: 'error'
 };

 function postProcess(result) {
@@ -31,16 +38,16 @@ function postProcess(result) {
    delete result.manifestJson;
 }

-function getByIdentifierAndStatePaged(identifier, state, page, perPage, callback) {
-    assert.strictEqual(typeof identifier, 'string');
+function getByTypeAndStatePaged(type, state, page, perPage, callback) {
+    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
    assert.strictEqual(typeof state, 'string');
    assert(typeof page === 'number' && page > 0);
    assert(typeof perPage === 'number' && perPage > 0);
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ identifier, state, (page-1)*perPage, perPage ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ type, state, (page-1)*perPage, perPage ], function (error, results) {
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            results.forEach(function (result) { postProcess(result); });

@@ -49,14 +56,14 @@ function getByIdentifierAndStatePaged(identifier, state, page, perPage, callback
 }

 function getByTypePaged(type, page, perPage, callback) {
-    assert.strictEqual(typeof type, 'string');
+    assert(type === exports.BACKUP_TYPE_APP || type === exports.BACKUP_TYPE_BOX);
    assert(typeof page === 'number' && page > 0);
    assert(typeof perPage === 'number' && perPage > 0);
    assert.strictEqual(typeof callback, 'function');

    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? ORDER BY creationTime DESC LIMIT ?,?',
        [ type, (page-1)*perPage, perPage ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            results.forEach(function (result) { postProcess(result); });

@@ -64,15 +71,16 @@ function getByTypePaged(type, page, perPage, callback) {
        });
 }

-function getByIdentifierPaged(identifier, page, perPage, callback) {
-    assert.strictEqual(typeof identifier, 'string');
+function getByAppIdPaged(page, perPage, appId, callback) {
    assert(typeof page === 'number' && page > 0);
    assert(typeof perPage === 'number' && perPage > 0);
+    assert.strictEqual(typeof appId, 'string');
    assert.strictEqual(typeof callback, 'function');

-    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE identifier = ? ORDER BY creationTime DESC LIMIT ?,?',
-        [ identifier, (page-1)*perPage, perPage ], function (error, results) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+    // box versions (0.93.x and below) used to use appbackup_ prefix
+    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE type = ? AND state = ? AND id LIKE ? ORDER BY creationTime DESC LIMIT ?,?',
+        [ exports.BACKUP_TYPE_APP, exports.BACKUP_STATE_NORMAL, '%app%\\_' + appId + '\\_%', (page-1)*perPage, perPage ], function (error, results) {
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            results.forEach(function (result) { postProcess(result); });

@@ -86,8 +94,8 @@ function get(id, callback) {

    database.query('SELECT ' + BACKUPS_FIELDS + ' FROM backups WHERE id = ? ORDER BY creationTime DESC',
        [ id ], function (error, result) {
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
-            if (result.length === 0) return callback(new BoxError(BoxError.NOT_FOUND, 'Backup not found'));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+            if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));

            postProcess(result[0]);

@@ -98,11 +106,8 @@ function get(id, callback) {
 function add(id, data, callback) {
    assert(data && typeof data === 'object');
    assert.strictEqual(typeof id, 'string');
-    assert(data.encryptionVersion === null || typeof data.encryptionVersion === 'number');
-    assert.strictEqual(typeof data.packageVersion, 'string');
-    assert.strictEqual(typeof data.type, 'string');
-    assert.strictEqual(typeof data.identifier, 'string');
-    assert.strictEqual(typeof data.state, 'string');
+    assert.strictEqual(typeof data.version, 'string');
+    assert(data.type === exports.BACKUP_TYPE_APP || data.type === exports.BACKUP_TYPE_BOX);
    assert(util.isArray(data.dependsOn));
    assert.strictEqual(typeof data.manifest, 'object');
    assert.strictEqual(typeof data.format, 'string');
@@ -111,11 +116,11 @@ function add(id, data, callback) {
    var creationTime = data.creationTime || new Date(); // allow tests to set the time
    var manifestJson = JSON.stringify(data.manifest);

-    database.query('INSERT INTO backups (id, identifier, encryptionVersion, packageVersion, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        [ id, data.identifier, data.encryptionVersion, data.packageVersion, data.type, creationTime, data.state, data.dependsOn.join(','), manifestJson, data.format ],
+    database.query('INSERT INTO backups (id, version, type, creationTime, state, dependsOn, manifestJson, format) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
+        [ id, data.version, data.type, creationTime, exports.BACKUP_STATE_NORMAL, data.dependsOn.join(','), manifestJson, data.format ],
        function (error) {
-            if (error && error.code === 'ER_DUP_ENTRY') return callback(new BoxError(BoxError.ALREADY_EXISTS));
-            if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+            if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
+            if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

            callback(null);
        });
@@ -134,8 +139,8 @@ function update(id, backup, callback) {
    values.push(id);

    database.query('UPDATE backups SET ' + fields.join(', ') + ' WHERE id = ?', values, function (error) {
-        if (error && error.reason === BoxError.NOT_FOUND) return callback(new BoxError(BoxError.NOT_FOUND, 'Backup not found'));
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));

        callback(null);
    });
@@ -145,7 +150,7 @@ function clear(callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('TRUNCATE TABLE backups', [], function (error) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        callback(null);
    });
 }
@@ -155,7 +160,7 @@ function del(id, callback) {
    assert.strictEqual(typeof callback, 'function');

    database.query('DELETE FROM backups WHERE id=?', [ id ], function (error) {
-        if (error) return callback(new BoxError(BoxError.DATABASE_ERROR, error));
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
        callback(null);
    });
 }
@@ -1,103 +0,0 @@
-/* jslint node:true */
-
-'use strict';
-
-const assert = require('assert'),
-    HttpError = require('connect-lastmile').HttpError,
-    util = require('util'),
-    _ = require('underscore');
-
-exports = module.exports = BoxError;
-
-function BoxError(reason, errorOrMessage, details) {
-    assert.strictEqual(typeof reason, 'string');
-    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
-    assert(typeof details === 'object' || typeof details === 'undefined');
-
-    Error.call(this);
-    Error.captureStackTrace(this, this.constructor);
-
-    this.name = this.constructor.name;
-    this.reason = reason;
-    this.details = details || {};
-
-    if (typeof errorOrMessage === 'undefined') {
-        this.message = reason;
-    } else if (typeof errorOrMessage === 'string') {
-        this.message = errorOrMessage;
-    } else { // error object
-        this.message = errorOrMessage.message;
-        this.nestedError = errorOrMessage;
-        _.extend(this.details, errorOrMessage); // copy enumerable properies
-    }
-}
-util.inherits(BoxError, Error);
-BoxError.ACCESS_DENIED = 'Access Denied';
-BoxError.ADDONS_ERROR = 'Addons Error';
-BoxError.ALREADY_EXISTS = 'Already Exists';
-BoxError.BAD_FIELD = 'Bad Field';
-BoxError.BAD_STATE = 'Bad State';
-BoxError.BUSY = 'Busy';
-BoxError.COLLECTD_ERROR = 'Collectd Error';
-BoxError.CONFLICT = 'Conflict';
-BoxError.CRYPTO_ERROR = 'Crypto Error';
-BoxError.DATABASE_ERROR = 'Database Error';
-BoxError.DNS_ERROR = 'DNS Error';
-BoxError.DOCKER_ERROR = 'Docker Error';
-BoxError.EXTERNAL_ERROR = 'External Error'; // use this for external API errors
-BoxError.FEATURE_DISABLED = 'Feature Disabled';
-BoxError.FS_ERROR = 'FileSystem Error';
-BoxError.INACTIVE = 'Inactive';
-BoxError.INTERNAL_ERROR = 'Internal Error';
-BoxError.INVALID_CREDENTIALS = 'Invalid Credentials';
-BoxError.IPTABLES_ERROR = 'IPTables Error';
-BoxError.LICENSE_ERROR = 'License Error';
-BoxError.LOGROTATE_ERROR = 'Logrotate Error';
-BoxError.MAIL_ERROR = 'Mail Error';
-BoxError.NETWORK_ERROR = 'Network Error';
-BoxError.NGINX_ERROR = 'Nginx Error';
-BoxError.NOT_FOUND = 'Not found';
-BoxError.NOT_IMPLEMENTED = 'Not implemented';
-BoxError.NOT_SIGNED = 'Not Signed';
-BoxError.OPENSSL_ERROR = 'OpenSSL Error';
-BoxError.PLAN_LIMIT = 'Plan Limit';
-BoxError.SPAWN_ERROR = 'Spawn Error';
-BoxError.TASK_ERROR = 'Task Error';
-BoxError.TIMEOUT = 'Timeout';
-BoxError.TRY_AGAIN = 'Try Again';
-
-BoxError.prototype.toPlainObject = function () {
-    return _.extend({}, { message: this.message, reason: this.reason }, this.details);
-};
-
-// this is a class method for now in case error is not a BoxError
-BoxError.toHttpError = function (error) {
-    switch (error.reason) {
-    case BoxError.BAD_FIELD:
-        return new HttpError(400, error);
-    case BoxError.LICENSE_ERROR:
-        return new HttpError(402, error);
-    case BoxError.NOT_FOUND:
-        return new HttpError(404, error);
-    case BoxError.FEATURE_DISABLED:
-        return new HttpError(405, error);
-    case BoxError.ALREADY_EXISTS:
-    case BoxError.BAD_STATE:
-    case BoxError.CONFLICT:
-        return new HttpError(409, error);
-    case BoxError.INVALID_CREDENTIALS:
-        return new HttpError(412, error);
-    case BoxError.EXTERNAL_ERROR:
-    case BoxError.NETWORK_ERROR:
-    case BoxError.FS_ERROR:
-    case BoxError.MAIL_ERROR:
-    case BoxError.DOCKER_ERROR:
-    case BoxError.ADDONS_ERROR:
-    case BoxError.IPTABLES_ERROR:
-        return new HttpError(424, error);
-    case BoxError.DATABASE_ERROR:
-    case BoxError.INTERNAL_ERROR:
-    default:
-        return new HttpError(500, error);
-    }
-};
@@ -2,15 +2,14 @@

 var assert = require('assert'),
    async = require('async'),
-    BoxError = require('../boxerror.js'),
    crypto = require('crypto'),
    debug = require('debug')('box:cert/acme2'),
    domains = require('../domains.js'),
    fs = require('fs'),
    path = require('path'),
    paths = require('../paths.js'),
-    request = require('request'),
    safe = require('safetydance'),
+    superagent = require('superagent'),
    util = require('util'),
    _ = require('underscore');

@@ -25,6 +24,31 @@ exports = module.exports = {
    _getChallengeSubdomain: getChallengeSubdomain
 };

+function Acme2Error(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(Acme2Error, Error);
+Acme2Error.INTERNAL_ERROR = 'Internal Error';
+Acme2Error.EXTERNAL_ERROR = 'External Error';
+Acme2Error.ALREADY_EXISTS = 'Already Exists';
+Acme2Error.NOT_COMPLETED = 'Not Completed';
+Acme2Error.FORBIDDEN = 'Forbidden';
+
 // http://jose.readthedocs.org/en/latest/
 // https://www.ietf.org/proceedings/92/slides/slides-92-acme-1.pdf
 // https://community.letsencrypt.org/t/list-of-client-implementations/2103
@@ -41,6 +65,15 @@ function Acme2(options) {
    this.wildcard = !!options.wildcard;
 }

+Acme2.prototype.getNonce = function (callback) {
+    superagent.get(this.directory.newNonce).timeout(30 * 1000).end(function (error, response) {
+        if (error && !error.response) return callback(error);
+        if (response.statusCode !== 204) return callback(new Error('Invalid response code when fetching nonce : ' + response.statusCode));
+
+        return callback(null, response.headers['Replay-Nonce'.toLowerCase()]);
+    });
+};
+
 // urlsafe base64 encoding (jose)
 function urlBase64Encode(string) {
    return string.replace(/\+/g, '-').replace(/\//g, '_').replace(/=/g, '');
@@ -87,12 +120,8 @@ Acme2.prototype.sendSignedRequest = function (url, payload, callback) {

    var payload64 = b64(payload);

-    request.get(this.directory.newNonce, { json: true, timeout: 30000 }, function (error, response) {
-        if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Network error sending signed request: ${error.message}`));
-        if (response.statusCode !== 204) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response code when fetching nonce : ' + response.statusCode));
-
-        const nonce = response.headers['Replay-Nonce'.toLowerCase()];
-        if (!nonce) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'No nonce in response'));
+    this.getNonce(function (error, nonce) {
+        if (error) return callback(error);

        debug('sendSignedRequest: using nonce %s for url %s', nonce, url);

@@ -108,23 +137,14 @@ Acme2.prototype.sendSignedRequest = function (url, payload, callback) {
            signature: signature64
        };

-        request.post(url, { headers: { 'Content-Type': 'application/jose+json', 'User-Agent': 'acme-cloudron' }, body: JSON.stringify(data), timeout: 30000 }, function (error, response) {
-            if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Network error sending signed request: ${error.message}`)); // network error
+        superagent.post(url).set('Content-Type', 'application/jose+json').set('User-Agent', 'acme-cloudron').send(JSON.stringify(data)).timeout(30 * 1000).end(function (error, res) {
+            if (error && !error.response) return callback(error); // network errors

-            // we don't set json: true in request because it ends up mangling the content-type
-            // we don't set json: true in request because it ends up mangling the content-type
-            if (response.headers['content-type'] === 'application/json') response.body = safe.JSON.parse(response.body);
-
-            callback(null, response);
+            callback(null, res);
        });
    });
 };

-// https://tools.ietf.org/html/rfc8555#section-6.3
-Acme2.prototype.postAsGet = function (url, callback) {
-    this.sendSignedRequest(url, '', callback);
-};
-
 Acme2.prototype.updateContact = function (registrationUri, callback) {
    assert.strictEqual(typeof registrationUri, 'string');
    assert.strictEqual(typeof callback, 'function');
@@ -138,8 +158,8 @@ Acme2.prototype.updateContact = function (registrationUri, callback) {

    const that = this;
    this.sendSignedRequest(registrationUri, JSON.stringify(payload), function (error, result) {
-        if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when registering user: ' + error.message));
+        if (result.statusCode !== 200) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to update contact. Expecting 200, got %s %s', result.statusCode, result.text)));

        debug(`updateContact: contact of user updated to ${that.email}`);

@@ -158,9 +178,9 @@ Acme2.prototype.registerUser = function (callback) {

    var that = this;
    this.sendSignedRequest(this.directory.newAccount, JSON.stringify(payload), function (error, result) {
-        if (error) return callback(error);
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when registering new account: ' + error.message));
        // 200 if already exists. 201 for new accounts
-        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register new account. Expecting 200 or 201, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200 && result.statusCode !== 201) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to register new account. Expecting 200 or 201, got %s %s', result.statusCode, result.text)));

        debug(`registerUser: user registered keyid: ${result.headers.location}`);

@@ -184,17 +204,17 @@ Acme2.prototype.newOrder = function (domain, callback) {
    debug('newOrder: %s', domain);

    this.sendSignedRequest(this.directory.newOrder, JSON.stringify(payload), function (error, result) {
-        if (error) return callback(error);
-        if (result.statusCode === 403) return callback(new BoxError(BoxError.ACCESS_DENIED, `Forbidden sending signed request: ${result.body.detail}`));
-        if (result.statusCode !== 201) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when registering domain: ' + error.message));
+        if (result.statusCode === 403) return callback(new Acme2Error(Acme2Error.FORBIDDEN, result.body.detail));
+        if (result.statusCode !== 201) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to register user. Expecting 201, got %s %s', result.statusCode, result.text)));

        debug('newOrder: created order %s %j', domain, result.body);

        const order = result.body, orderUrl = result.headers.location;

-        if (!Array.isArray(order.authorizations)) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'invalid authorizations in order'));
-        if (typeof order.finalize !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'invalid finalize in order'));
-        if (typeof orderUrl !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'invalid order location in order header'));
+        if (!Array.isArray(order.authorizations)) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'invalid authorizations in order'));
+        if (typeof order.finalize !== 'string') return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'invalid finalize in order'));
+        if (typeof orderUrl !== 'string') return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'invalid order location in order header'));

        callback(null, order, orderUrl);
    });
@@ -205,26 +225,25 @@ Acme2.prototype.waitForOrder = function (orderUrl, callback) {
    assert.strictEqual(typeof callback, 'function');

    debug(`waitForOrder: ${orderUrl}`);
-    const that = this;

    async.retry({ times: 15, interval: 20000 }, function (retryCallback) {
        debug('waitForOrder: getting status');

-        that.postAsGet(orderUrl, function (error, result) {
-            if (error) {
+        superagent.get(orderUrl).timeout(30 * 1000).end(function (error, result) {
+            if (error && !error.response) {
                debug('waitForOrder: network error getting uri %s', orderUrl);
-                return retryCallback(error);
+                return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message)); // network error
            }
            if (result.statusCode !== 200) {
                debug('waitForOrder: invalid response code getting uri %s', result.statusCode);
-                return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, 'Bad response code:' + result.statusCode));
+                return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Bad response code:' + result.statusCode));
            }

            debug('waitForOrder: status is "%s %j', result.body.status, result.body);

-            if (result.body.status === 'pending' || result.body.status === 'processing') return retryCallback(new BoxError(BoxError.TRY_AGAIN, `Request is in ${result.body.status} state`));
+            if (result.body.status === 'pending' || result.body.status === 'processing') return retryCallback(new Acme2Error(Acme2Error.NOT_COMPLETED));
            else if (result.body.status === 'valid' && result.body.certificate) return retryCallback(null, result.body.certificate);
-            else return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, 'Unexpected status or invalid response: ' + result.body));
+            else return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Unexpected status or invalid response: ' + result.body));
        });
    }, callback);
 };
@@ -258,8 +277,8 @@ Acme2.prototype.notifyChallengeReady = function (challenge, callback) {
    };

    this.sendSignedRequest(challenge.url, JSON.stringify(payload), function (error, result) {
-        if (error) return callback(error);
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when notifying challenge: ' + error.message));
+        if (result.statusCode !== 200) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to notify challenge. Expecting 200, got %s %s', result.statusCode, result.text)));

        callback();
    });
@@ -270,26 +289,25 @@ Acme2.prototype.waitForChallenge = function (challenge, callback) {
    assert.strictEqual(typeof callback, 'function');

    debug('waitingForChallenge: %j', challenge);
-    const that = this;

    async.retry({ times: 15, interval: 20000 }, function (retryCallback) {
        debug('waitingForChallenge: getting status');

-        that.postAsGet(challenge.url, function (error, result) {
-            if (error) {
+        superagent.get(challenge.url).timeout(30 * 1000).end(function (error, result) {
+            if (error && !error.response) {
                debug('waitForChallenge: network error getting uri %s', challenge.url);
-                return retryCallback(error);
+                return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message)); // network error
            }
            if (result.statusCode !== 200) {
                debug('waitForChallenge: invalid response code getting uri %s', result.statusCode);
-                return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, 'Bad response code:' + result.statusCode));
+                return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Bad response code:' + result.statusCode));
            }

-            debug('waitForChallenge: status is "%s" %j', result.body.status, result.body);
+            debug('waitForChallenge: status is "%s %j', result.body.status, result.body);

-            if (result.body.status === 'pending') return retryCallback(new BoxError(BoxError.TRY_AGAIN));
+            if (result.body.status === 'pending') return retryCallback(new Acme2Error(Acme2Error.NOT_COMPLETED));
            else if (result.body.status === 'valid') return retryCallback();
-            else return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, 'Unexpected status: ' + result.body.status));
+            else return retryCallback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Unexpected status: ' + result.body.status));
        });
    }, function retryFinished(error) {
        // async.retry will pass 'undefined' as second arg making it unusable with async.waterfall()
@@ -311,9 +329,9 @@ Acme2.prototype.signCertificate = function (domain, finalizationUrl, csrDer, cal
    debug('signCertificate: sending sign request');

    this.sendSignedRequest(finalizationUrl, JSON.stringify(payload), function (error, result) {
-        if (error) return callback(error);
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when signing certificate: ' + error.message));
        // 429 means we reached the cert limit for this domain
-        if (result.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 200, got %s %s', result.statusCode, result.text)));
+        if (result.statusCode !== 200) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to sign certificate. Expecting 200, got %s %s', result.statusCode, result.text)));

        return callback(null);
    });
@@ -332,16 +350,16 @@ Acme2.prototype.createKeyAndCsr = function (hostname, callback) {
        // in some old releases, csr file was corrupt. so always regenerate it
        debug('createKeyAndCsr: reuse the key for renewal at %s', privateKeyFile);
    } else {
-        var key = safe.child_process.execSync('openssl ecparam -genkey -name secp384r1'); // openssl ecparam -list_curves
-        if (!key) return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
-        if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new BoxError(BoxError.FS_ERROR, safe.error));
+        var key = safe.child_process.execSync('openssl genrsa 4096');
+        if (!key) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));
+        if (!safe.fs.writeFileSync(privateKeyFile, key)) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));

        debug('createKeyAndCsr: key file saved at %s', privateKeyFile);
    }

    var csrDer = safe.child_process.execSync(`openssl req -new -key ${privateKeyFile} -outform DER -subj /CN=${hostname}`);
-    if (!csrDer) return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
-    if (!safe.fs.writeFileSync(csrFile, csrDer)) return callback(new BoxError(BoxError.FS_ERROR, safe.error)); // bookkeeping
+    if (!csrDer) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));
+    if (!safe.fs.writeFileSync(csrFile, csrDer)) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error)); // bookkeeping

    debug('createKeyAndCsr: csr file (DER) saved at %s', csrFile);

@@ -354,27 +372,26 @@ Acme2.prototype.downloadCertificate = function (hostname, certUrl, callback) {
    assert.strictEqual(typeof callback, 'function');

    var outdir = paths.APP_CERTS_DIR;
-    const that = this;

-    async.retry({ times: 5, interval: 20000 }, function (retryCallback) {
-        debug('downloadCertificate: downloading certificate');
+    superagent.get(certUrl).buffer().parse(function (res, done) {
+        var data = [ ];
+        res.on('data', function(chunk) { data.push(chunk); });
+        res.on('end', function () { res.text = Buffer.concat(data); done(); });
+    }).timeout(30 * 1000).end(function (error, result) {
+        if (error && !error.response) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'Network error when downloading certificate'));
+        if (result.statusCode === 202) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, 'Retry not implemented yet'));
+        if (result.statusCode !== 200) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));

-        that.postAsGet(certUrl, function (error, result) {
-            if (error) return retryCallback(new BoxError(BoxError.NETWORK_ERROR, `Network error when downloading certificate: ${error.message}`));
-            if (result.statusCode === 202) return retryCallback(new BoxError(BoxError.TRY_AGAIN, 'Retry downloading certificate'));
-            if (result.statusCode !== 200) return retryCallback(new BoxError(BoxError.EXTERNAL_ERROR, util.format('Failed to get cert. Expecting 200, got %s %s', result.statusCode, result.text)));
+        const fullChainPem = result.text;

-            const fullChainPem = result.body; // buffer
+        const certName = hostname.replace('*.', '_.');
+        var certificateFile = path.join(outdir, `${certName}.cert`);
+        if (!safe.fs.writeFileSync(certificateFile, fullChainPem)) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));

-            const certName = hostname.replace('*.', '_.');
-            var certificateFile = path.join(outdir, `${certName}.cert`);
-            if (!safe.fs.writeFileSync(certificateFile, fullChainPem)) return retryCallback(new BoxError(BoxError.FS_ERROR, safe.error));
+        debug('downloadCertificate: cert file for %s saved at %s', hostname, certificateFile);

-            debug('downloadCertificate: cert file for %s saved at %s', hostname, certificateFile);
-
-            retryCallback(null);
-        });
-    }, callback);
+        callback();
+    });
 };

 Acme2.prototype.prepareHttpChallenge = function (hostname, domain, authorization, callback) {
@@ -385,7 +402,7 @@ Acme2.prototype.prepareHttpChallenge = function (hostname, domain, authorization

    debug('acmeFlow: challenges: %j', authorization);
    let httpChallenges = authorization.challenges.filter(function(x) { return x.type === 'http-01'; });
-    if (httpChallenges.length === 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'no http challenges'));
+    if (httpChallenges.length === 0) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'no http challenges'));
    let challenge = httpChallenges[0];

    debug('prepareHttpChallenge: preparing for challenge %j', challenge);
@@ -395,7 +412,7 @@ Acme2.prototype.prepareHttpChallenge = function (hostname, domain, authorization
    debug('prepareHttpChallenge: writing %s to %s', keyAuthorization, path.join(paths.ACME_CHALLENGES_DIR, challenge.token));

    fs.writeFile(path.join(paths.ACME_CHALLENGES_DIR, challenge.token), keyAuthorization, function (error) {
-        if (error) return callback(new BoxError(BoxError.FS_ERROR, error));
+        if (error) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, error));

        callback(null, challenge);
    });
@@ -437,7 +454,7 @@ Acme2.prototype.prepareDnsChallenge = function (hostname, domain, authorization,

    debug('acmeFlow: challenges: %j', authorization);
    let dnsChallenges = authorization.challenges.filter(function(x) { return x.type === 'dns-01'; });
-    if (dnsChallenges.length === 0) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'no dns challenges'));
+    if (dnsChallenges.length === 0) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, 'no dns challenges'));
    let challenge = dnsChallenges[0];

    const keyAuthorization = this.getKeyAuthorization(challenge.token);
@@ -450,10 +467,10 @@ Acme2.prototype.prepareDnsChallenge = function (hostname, domain, authorization,
    debug(`prepareDnsChallenge: update ${challengeSubdomain} with ${txtValue}`);

    domains.upsertDnsRecords(challengeSubdomain, domain, 'TXT', [ `"${txtValue}"` ], function (error) {
-        if (error) return callback(error);
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message));

-        domains.waitForDnsRecord(challengeSubdomain, domain, 'TXT', txtValue, { times: 200 }, function (error) {
-            if (error) return callback(error);
+        domains.waitForDnsRecord(challengeSubdomain, domain, 'TXT', txtValue, { interval: 5000, times: 200 }, function (error) {
+            if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error.message));

            callback(null, challenge);
        });
@@ -476,7 +493,7 @@ Acme2.prototype.cleanupDnsChallenge = function (hostname, domain, challenge, cal
    debug(`cleanupDnsChallenge: remove ${challengeSubdomain} with ${txtValue}`);

    domains.removeDnsRecords(challengeSubdomain, domain, 'TXT', [ `"${txtValue}"` ], function (error) {
-        if (error) return callback(error);
+        if (error) return callback(new Acme2Error(Acme2Error.EXTERNAL_ERROR, error));

        callback(null);
    });
@@ -488,12 +505,10 @@ Acme2.prototype.prepareChallenge = function (hostname, domain, authorizationUrl,
    assert.strictEqual(typeof authorizationUrl, 'string');
    assert.strictEqual(typeof callback, 'function');

-    debug(`prepareChallenge: http: ${this.performHttpAuthorization}`);
-
    const that = this;
-    this.postAsGet(authorizationUrl, function (error, response) {
-        if (error) return callback(error);
-        if (response.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response code getting authorization : ' + response.statusCode));
+    superagent.get(authorizationUrl).timeout(30 * 1000).end(function (error, response) {
+        if (error && !error.response) return callback(error);
+        if (response.statusCode !== 200) return callback(new Error('Invalid response code getting authorization : ' + response.statusCode));

        const authorization = response.body;

@@ -511,8 +526,6 @@ Acme2.prototype.cleanupChallenge = function (hostname, domain, challenge, callba
    assert.strictEqual(typeof challenge, 'object');
    assert.strictEqual(typeof callback, 'function');

-    debug(`cleanupChallenge: http: ${this.performHttpAuthorization}`);
-
    if (this.performHttpAuthorization) {
        this.cleanupHttpChallenge(hostname, domain, challenge, callback);
    } else {
@@ -528,7 +541,7 @@ Acme2.prototype.acmeFlow = function (hostname, domain, callback) {
    if (!fs.existsSync(paths.ACME_ACCOUNT_KEY_FILE)) {
        debug('getCertificate: generating acme account key on first run');
        this.accountKeyPem = safe.child_process.execSync('openssl genrsa 4096');
-        if (!this.accountKeyPem) return callback(new BoxError(BoxError.OPENSSL_ERROR, safe.error));
+        if (!this.accountKeyPem) return callback(new Acme2Error(Acme2Error.INTERNAL_ERROR, safe.error));

        safe.fs.writeFileSync(paths.ACME_ACCOUNT_KEY_FILE, this.accountKeyPem);
    } else {
@@ -572,13 +585,13 @@ Acme2.prototype.acmeFlow = function (hostname, domain, callback) {
 Acme2.prototype.getDirectory = function (callback) {
    const that = this;

-    request.get(this.caDirectory, { json: true, timeout: 30000 }, function (error, response) {
-        if (error) return callback(new BoxError(BoxError.NETWORK_ERROR, `Network error getting directory: ${error.message}`));
-        if (response.statusCode !== 200) return callback(new BoxError(BoxError.EXTERNAL_ERROR, 'Invalid response code when fetching directory : ' + response.statusCode));
+    superagent.get(this.caDirectory).timeout(30 * 1000).end(function (error, response) {
+        if (error && !error.response) return callback(error);
+        if (response.statusCode !== 200) return callback(new Error('Invalid response code when fetching directory : ' + response.statusCode));

        if (typeof response.body.newNonce !== 'string' ||
            typeof response.body.newOrder !== 'string' ||
-            typeof response.body.newAccount !== 'string') return callback(new BoxError(BoxError.EXTERNAL_ERROR, `Invalid response body : ${response.body}`));
+            typeof response.body.newAccount !== 'string') return callback(new Error(`Invalid response body : ${response.body}`));

        that.directory = response.body;

@@ -618,11 +631,6 @@ function getCertificate(hostname, domain, options, callback) {
    assert.strictEqual(typeof options, 'object');
    assert.strictEqual(typeof callback, 'function');

-    let attempt = 1;
-    async.retry({ times: 3, interval: 0 }, function (retryCallback) {
-        debug(`getCertificate: attempt ${attempt++}`);
-
-        let acme = new Acme2(options || { });
-        acme.getCertificate(hostname, domain, retryCallback);
-    }, callback);
+    var acme = new Acme2(options || { });
+    acme.getCertificate(hostname, domain, callback);
 }
@@ -0,0 +1,22 @@
+'use strict';
+
+exports = module.exports = {
+    getCertificate: getCertificate,
+
+    // testing
+    _name: 'caas'
+};
+
+var assert = require('assert'),
+    debug = require('debug')('box:cert/caas.js');
+
+function getCertificate(hostname, domain, options, callback) {
+    assert.strictEqual(typeof hostname, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('getCertificate: using fallback certificate', hostname);
+
+    return callback(null, '', '');
+}
@@ -0,0 +1,22 @@
+'use strict';
+
+exports = module.exports = {
+    getCertificate: getCertificate,
+
+    // testing
+    _name: 'fallback'
+};
+
+var assert = require('assert'),
+    debug = require('debug')('box:cert/fallback.js');
+
+function getCertificate(hostname, domain, options, callback) {
+    assert.strictEqual(typeof hostname, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('getCertificate: using fallback certificate', hostname);
+
+    return callback(null, '', '');
+}
@@ -0,0 +1,23 @@
+'use strict';
+
+// -------------------------------------------
+//  This file just describes the interface
+//
+//  New backends can start from here
+// -------------------------------------------
+
+exports = module.exports = {
+    getCertificate: getCertificate
+};
+
+var assert = require('assert');
+
+function getCertificate(hostname, domain, options, callback) {
+    assert.strictEqual(typeof hostname, 'string');
+    assert.strictEqual(typeof domain, 'string');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    return callback(new Error('Not implemented'));
+}
+
@@ -1,38 +0,0 @@
-'use strict';
-
-let assert = require('assert'),
-    fs = require('fs'),
-    path = require('path');
-
-exports = module.exports = {
-    getChanges: getChanges
-};
-
-function getChanges(version) {
-    assert.strictEqual(typeof version, 'string');
-
-    let changelog = [ ];
-    const lines = fs.readFileSync(path.join(__dirname, '../CHANGES'), 'utf8').split('\n');
-
-    version = version.replace(/[+-].*/, ''); // strip prerelease
-
-    let i;
-    for (i = 0; i < lines.length; i++) {
-        if (lines[i] === '[' + version + ']') break;
-    }
-
-    for (i = i + 1; i < lines.length; i++) {
-        if (lines[i] === '') continue;
-        if (lines[i][0] === '[') break;
-
-        lines[i] = lines[i].trim();
-
-        // detect and remove list style - and * in changelog lines
-        if (lines[i].indexOf('-') === 0) lines[i] = lines[i].slice(1).trim();
-        if (lines[i].indexOf('*') === 0) lines[i] = lines[i].slice(1).trim();
-
-        changelog.push(lines[i]);
-    }
-
-    return changelog;
-}
@@ -0,0 +1,189 @@
+'use strict';
+
+exports = module.exports = {
+    get: get,
+    getAll: getAll,
+    getAllWithTokenCount: getAllWithTokenCount,
+    getAllWithTokenCountByIdentifier: getAllWithTokenCountByIdentifier,
+    add: add,
+    del: del,
+    getByAppId: getByAppId,
+    getByAppIdAndType: getByAppIdAndType,
+
+    upsert: upsert,
+
+    delByAppId: delByAppId,
+    delByAppIdAndType: delByAppIdAndType,
+
+    _clear: clear,
+    _addDefaultClients: addDefaultClients
+};
+
+var assert = require('assert'),
+    async = require('async'),
+    database = require('./database.js'),
+    DatabaseError = require('./databaseerror.js');
+
+var CLIENTS_FIELDS = [ 'id', 'appId', 'type', 'clientSecret', 'redirectURI', 'scope' ].join(',');
+var CLIENTS_FIELDS_PREFIXED = [ 'clients.id', 'clients.appId', 'clients.type', 'clients.clientSecret', 'clients.redirectURI', 'clients.scope' ].join(',');
+
+function get(id, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS + ' FROM clients WHERE id = ?', [ id ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null, result[0]);
+    });
+}
+
+function getAll(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS + ' FROM clients ORDER BY appId', function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null, results);
+    });
+}
+
+function getAllWithTokenCount(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS_PREFIXED + ',COUNT(tokens.clientId) AS tokenCount FROM clients LEFT OUTER JOIN tokens ON clients.id=tokens.clientId GROUP BY clients.id', [], function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null, results);
+    });
+}
+
+function getAllWithTokenCountByIdentifier(identifier, callback) {
+    assert.strictEqual(typeof identifier, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS_PREFIXED + ',COUNT(tokens.clientId) AS tokenCount FROM clients LEFT OUTER JOIN tokens ON clients.id=tokens.clientId WHERE tokens.identifier=? GROUP BY clients.id', [ identifier ], function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null, results);
+    });
+}
+
+function getByAppId(appId, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS + ' FROM clients WHERE appId = ? LIMIT 1', [ appId ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null, result[0]);
+    });
+}
+
+function getByAppIdAndType(appId, type, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('SELECT ' + CLIENTS_FIELDS + ' FROM clients WHERE appId = ? AND type = ? LIMIT 1', [ appId, type ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.length === 0) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null, result[0]);
+    });
+}
+
+function add(id, appId, type, clientSecret, redirectURI, scope, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof clientSecret, 'string');
+    assert.strictEqual(typeof redirectURI, 'string');
+    assert.strictEqual(typeof scope, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var data = [ id, appId, type, clientSecret, redirectURI, scope ];
+
+    database.query('INSERT INTO clients (id, appId, type, clientSecret, redirectURI, scope) VALUES (?, ?, ?, ?, ?, ?)', data, function (error, result) {
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
+        if (error || result.affectedRows === 0) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function upsert(id, appId, type, clientSecret, redirectURI, scope, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof clientSecret, 'string');
+    assert.strictEqual(typeof redirectURI, 'string');
+    assert.strictEqual(typeof scope, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var data = [ id, appId, type, clientSecret, redirectURI, scope ];
+
+    database.query('REPLACE INTO clients (id, appId, type, clientSecret, redirectURI, scope) VALUES (?, ?, ?, ?, ?, ?)', data, function (error, result) {
+        if (error && error.code === 'ER_DUP_ENTRY') return callback(new DatabaseError(DatabaseError.ALREADY_EXISTS));
+        if (error || result.affectedRows === 0) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function del(id, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM clients WHERE id = ?', [ id ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
+function delByAppId(appId, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM clients WHERE appId=?', [ appId ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
+function delByAppIdAndType(appId, type, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM clients WHERE appId=? AND type=?', [ appId, type ], function (error, result) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+        if (result.affectedRows !== 1) return callback(new DatabaseError(DatabaseError.NOT_FOUND));
+
+        callback(null);
+    });
+}
+
+function clear(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    database.query('DELETE FROM clients WHERE id!="cid-webadmin" AND id!="cid-sdk" AND id!="cid-cli"', function (error) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        callback(null);
+    });
+}
+
+function addDefaultClients(callback) {
+    async.series([
+        add.bind(null, 'cid-webadmin', 'Settings', 'built-in', 'secret-webadmin', 'https://admin-localhost', '*'),
+        add.bind(null, 'cid-sdk', 'SDK', 'built-in', 'secret-sdk', 'https://admin-localhost', '*'),
+        add.bind(null, 'cid-cli', 'Cloudron Tool', 'built-in', 'secret-cli', 'https://admin-localhost', '*')
+    ], callback);
+}
@@ -0,0 +1,359 @@
+'use strict';
+
+exports = module.exports = {
+    ClientsError: ClientsError,
+
+    add: add,
+    get: get,
+    del: del,
+    getAll: getAll,
+    getByAppIdAndType: getByAppIdAndType,
+    getTokensByUserId: getTokensByUserId,
+    delTokensByUserId: delTokensByUserId,
+    delByAppIdAndType: delByAppIdAndType,
+    addTokenByUserId: addTokenByUserId,
+    delToken: delToken,
+
+    issueDeveloperToken: issueDeveloperToken,
+
+    addDefaultClients: addDefaultClients,
+
+    removeTokenPrivateFields: removeTokenPrivateFields,
+
+    // client type enums
+    TYPE_EXTERNAL: 'external',
+    TYPE_BUILT_IN: 'built-in',
+    TYPE_OAUTH: 'addon-oauth',
+    TYPE_PROXY: 'addon-proxy'
+};
+
+var apps = require('./apps.js'),
+    assert = require('assert'),
+    async = require('async'),
+    clientdb = require('./clientdb.js'),
+    constants = require('./constants.js'),
+    DatabaseError = require('./databaseerror.js'),
+    debug = require('debug')('box:clients'),
+    eventlog = require('./eventlog.js'),
+    hat = require('./hat.js'),
+    accesscontrol = require('./accesscontrol.js'),
+    tokendb = require('./tokendb.js'),
+    users = require('./users.js'),
+    UsersError = users.UsersError,
+    util = require('util'),
+    uuid = require('uuid'),
+    _ = require('underscore');
+
+function ClientsError(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(ClientsError, Error);
+ClientsError.INVALID_SCOPE = 'Invalid scope';
+ClientsError.INVALID_CLIENT = 'Invalid client';
+ClientsError.INVALID_TOKEN = 'Invalid token';
+ClientsError.BAD_FIELD = 'Bad field';
+ClientsError.NOT_FOUND = 'Not found';
+ClientsError.INTERNAL_ERROR = 'Internal Error';
+ClientsError.NOT_ALLOWED = 'Not allowed to remove this client';
+
+function validateClientName(name) {
+    assert.strictEqual(typeof name, 'string');
+
+    if (name.length < 1) return new ClientsError(ClientsError.BAD_FIELD, 'Name must be atleast 1 character');
+    if (name.length > 128) return new ClientsError(ClientsError.BAD_FIELD, 'Name too long');
+
+    if (/[^a-zA-Z0-9-]/.test(name)) return new ClientsError(ClientsError.BAD_FIELD, 'Username can only contain alphanumerals and dash');
+
+    return null;
+}
+
+function validateTokenName(name) {
+    assert.strictEqual(typeof name, 'string');
+
+    if (name.length > 64) return new ClientsError(ClientsError.BAD_FIELD, 'Name too long');
+
+    return null;
+}
+
+function add(appId, type, redirectURI, scope, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof redirectURI, 'string');
+    assert.strictEqual(typeof scope, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    var error = accesscontrol.validateScopeString(scope);
+    if (error) return callback(new ClientsError(ClientsError.INVALID_SCOPE, error.message));
+
+    error = validateClientName(appId);
+    if (error) return callback(error);
+
+    var id = 'cid-' + uuid.v4();
+    var clientSecret = hat(8 * 128);
+
+    clientdb.add(id, appId, type, clientSecret, redirectURI, scope, function (error) {
+        if (error) return callback(error);
+
+        var client = {
+            id: id,
+            appId: appId,
+            type: type,
+            clientSecret: clientSecret,
+            redirectURI: redirectURI,
+            scope: scope
+        };
+
+        callback(null, client);
+    });
+}
+
+function get(id, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    clientdb.get(id, function (error, result) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.NOT_FOUND, 'No such client'));
+        if (error) return callback(error);
+        callback(null, result);
+    });
+}
+
+function del(id, callback) {
+    assert.strictEqual(typeof id, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    clientdb.del(id, function (error, result) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.NOT_FOUND, 'No such client'));
+        if (error) return callback(error);
+        callback(null, result);
+    });
+}
+
+function getAll(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    clientdb.getAll(function (error, results) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, []);
+        if (error) return callback(error);
+
+        var tmp = [];
+        async.each(results, function (record, callback) {
+            if (record.type === exports.TYPE_EXTERNAL || record.type === exports.TYPE_BUILT_IN) {
+                // the appId in this case holds the name
+                record.name = record.appId;
+
+                tmp.push(record);
+
+                return callback(null);
+            }
+
+            apps.get(record.appId, function (error, result) {
+                if (error) {
+                    console.error('Failed to get app details for oauth client', record.appId, error);
+                    return callback(null);  // ignore error so we continue listing clients
+                }
+
+                if (record.type === exports.TYPE_PROXY) record.name = result.manifest.title + ' Website Proxy';
+                if (record.type === exports.TYPE_OAUTH) record.name = result.manifest.title + ' OAuth';
+
+                record.domain = result.fqdn;
+
+                tmp.push(record);
+
+                callback(null);
+            });
+        }, function (error) {
+            if (error) return callback(error);
+            callback(null, tmp);
+        });
+    });
+}
+
+function getByAppIdAndType(appId, type, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    clientdb.getByAppIdAndType(appId, type, function (error, result) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.NOT_FOUND, 'No such client'));
+        if (error) return callback(error);
+        callback(null, result);
+    });
+}
+
+function getTokensByUserId(clientId, userId, callback) {
+    assert.strictEqual(typeof clientId, 'string');
+    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    tokendb.getByIdentifierAndClientId(userId, clientId, function (error, result) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) {
+            // this can mean either that there are no tokens or the clientId is actually unknown
+            get(clientId, function (error/*, result*/) {
+                if (error) return callback(error);
+                callback(null, []);
+            });
+            return;
+        }
+        if (error) return callback(error);
+        callback(null, result || []);
+    });
+}
+
+function delTokensByUserId(clientId, userId, callback) {
+    assert.strictEqual(typeof clientId, 'string');
+    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    tokendb.delByIdentifierAndClientId(userId, clientId, function (error) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) {
+            // this can mean either that there are no tokens or the clientId is actually unknown
+            get(clientId, function (error/*, result*/) {
+                if (error) return callback(error);
+                callback(null);
+            });
+            return;
+        }
+        if (error) return callback(error);
+        callback(null);
+    });
+}
+
+function delByAppIdAndType(appId, type, callback) {
+    assert.strictEqual(typeof appId, 'string');
+    assert.strictEqual(typeof type, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    getByAppIdAndType(appId, type, function (error, result) {
+        if (error) return callback(error);
+
+        tokendb.delByClientId(result.id, function (error) {
+            if (error && error.reason !== DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.INTERNAL_ERROR, error));
+
+            clientdb.delByAppIdAndType(appId, type, function (error) {
+                if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.NOT_FOUND, 'No such client'));
+                if (error) return callback(error);
+
+                callback(null);
+            });
+        });
+    });
+}
+
+function addTokenByUserId(clientId, userId, expiresAt, options, callback) {
+    assert.strictEqual(typeof clientId, 'string');
+    assert.strictEqual(typeof userId, 'string');
+    assert.strictEqual(typeof expiresAt, 'number');
+    assert.strictEqual(typeof options, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const name = options.name || '';
+    let error = validateTokenName(name);
+    if (error) return callback(error);
+
+    get(clientId, function (error, result) {
+        if (error) return callback(error);
+
+        users.get(userId, function (error, user) {
+            if (error && error.reason === UsersError.NOT_FOUND) return callback(new ClientsError(ClientsError.NOT_FOUND, 'No such user'));
+            if (error) return callback(new ClientsError(ClientsError.INTERNAL_ERROR, error));
+
+            accesscontrol.scopesForUser(user, function (error, userScopes) {
+                if (error) return callback(new ClientsError(ClientsError.INTERNAL_ERROR, error));
+
+                const scope = accesscontrol.canonicalScopeString(result.scope);
+                const authorizedScopes = accesscontrol.intersectScopes(userScopes, scope.split(','));
+
+                const token = {
+                    id: 'tid-' + uuid.v4(),
+                    accessToken: hat(8 * 32),
+                    identifier: userId,
+                    clientId: result.id,
+                    expires: expiresAt,
+                    scope: authorizedScopes.join(','),
+                    name: name
+                };
+
+                tokendb.add(token, function (error) {
+                    if (error) return callback(new ClientsError(ClientsError.INTERNAL_ERROR, error));
+
+                    callback(null, {
+                        accessToken: token.accessToken,
+                        tokenScopes: authorizedScopes,
+                        identifier: userId,
+                        clientId: result.id,
+                        expires: expiresAt
+                    });
+                });
+            });
+        });
+    });
+}
+
+// this issues a cid-cli token that does not require a password in various routes
+function issueDeveloperToken(userObject, auditSource, callback) {
+    assert.strictEqual(typeof userObject, 'object');
+    assert.strictEqual(typeof auditSource, 'object');
+    assert.strictEqual(typeof callback, 'function');
+
+    const expiresAt = Date.now() + constants.DEFAULT_TOKEN_EXPIRATION;
+
+    addTokenByUserId('cid-cli', userObject.id, expiresAt, {}, function (error, result) {
+        if (error) return callback(error);
+
+        eventlog.add(eventlog.ACTION_USER_LOGIN, auditSource, { userId: userObject.id, user: users.removePrivateFields(userObject) });
+
+        callback(null, result);
+    });
+}
+
+function delToken(clientId, tokenId, callback) {
+    assert.strictEqual(typeof clientId, 'string');
+    assert.strictEqual(typeof tokenId, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    get(clientId, function (error) {
+        if (error) return callback(error);
+
+        tokendb.del(tokenId, function (error) {
+            if (error && error.reason === DatabaseError.NOT_FOUND) return callback(new ClientsError(ClientsError.INVALID_TOKEN, 'Invalid token'));
+            if (error) return callback(new ClientsError(ClientsError.INTERNAL_ERROR, error));
+
+            callback(null);
+        });
+    });
+}
+
+function addDefaultClients(origin, callback) {
+    assert.strictEqual(typeof origin, 'string');
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('Adding default clients');
+
+    // The domain might have changed, therefor we have to update the record
+    // id, appId, type, clientSecret, redirectURI, scope
+    async.series([
+        clientdb.upsert.bind(null, 'cid-webadmin', 'Settings', 'built-in', 'secret-webadmin', origin, '*'),
+        clientdb.upsert.bind(null, 'cid-sdk', 'SDK', 'built-in', 'secret-sdk', origin, '*'),
+        clientdb.upsert.bind(null, 'cid-cli', 'Cloudron Tool', 'built-in', 'secret-cli', origin, '*')
+    ], callback);
+}
+
+function removeTokenPrivateFields(token) {
+    return _.pick(token, 'id', 'identifier', 'clientId', 'scope', 'expires', 'name');
+}
@@ -1,65 +1,95 @@
 'use strict';

 exports = module.exports = {
-    initialize,
-    uninitialize,
-    getConfig,
-    getLogs,
+    CloudronError: CloudronError,

-    reboot,
-    isRebootRequired,
+    initialize: initialize,
+    uninitialize: uninitialize,
+    getConfig: getConfig,
+    getDisks: getDisks,
+    getLogs: getLogs,

-    onActivated,
+    reboot: reboot,
+    isRebootRequired: isRebootRequired,

-    setupDnsAndCert,
+    onActivated: onActivated,

-    prepareDashboardDomain,
-    setDashboardDomain,
-    updateDashboardDomain,
-    renewCerts,
+    prepareDashboardDomain: prepareDashboardDomain,
+    setDashboardDomain: setDashboardDomain,
+    setDashboardAndMailDomain: setDashboardAndMailDomain,
+    renewCerts: renewCerts,

-    runSystemChecks
+    runSystemChecks: runSystemChecks,
+
+    // exposed for testing
+    _checkDiskSpace: checkDiskSpace
 };

-var addons = require('./addons.js'),
-    apps = require('./apps.js'),
-    appstore = require('./appstore.js'),
+var apps = require('./apps.js'),
    assert = require('assert'),
    async = require('async'),
-    auditSource = require('./auditsource.js'),
    backups = require('./backups.js'),
-    BoxError = require('./boxerror.js'),
+    clients = require('./clients.js'),
+    config = require('./config.js'),
    constants = require('./constants.js'),
    cron = require('./cron.js'),
    debug = require('debug')('box:cloudron'),
    domains = require('./domains.js'),
+    DomainsError = require('./domains.js').DomainsError,
+    df = require('@sindresorhus/df'),
    eventlog = require('./eventlog.js'),
+    custom = require('./custom.js'),
    fs = require('fs'),
    mail = require('./mail.js'),
    notifications = require('./notifications.js'),
+    os = require('os'),
    path = require('path'),
    paths = require('./paths.js'),
    platform = require('./platform.js'),
    reverseProxy = require('./reverseproxy.js'),
-    safe = require('safetydance'),
    settings = require('./settings.js'),
    shell = require('./shell.js'),
    spawn = require('child_process').spawn,
    split = require('split'),
-    sysinfo = require('./sysinfo.js'),
    tasks = require('./tasks.js'),
-    users = require('./users.js');
+    users = require('./users.js'),
+    util = require('util');

 var REBOOT_CMD = path.join(__dirname, 'scripts/reboot.sh');

-const NOOP_CALLBACK = function (error) { if (error) debug(error); };
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+function CloudronError(reason, errorOrMessage) {
+    assert.strictEqual(typeof reason, 'string');
+    assert(errorOrMessage instanceof Error || typeof errorOrMessage === 'string' || typeof errorOrMessage === 'undefined');
+
+    Error.call(this);
+    Error.captureStackTrace(this, this.constructor);
+
+    this.name = this.constructor.name;
+    this.reason = reason;
+    if (typeof errorOrMessage === 'undefined') {
+        this.message = reason;
+    } else if (typeof errorOrMessage === 'string') {
+        this.message = errorOrMessage;
+    } else {
+        this.message = 'Internal error';
+        this.nestedError = errorOrMessage;
+    }
+}
+util.inherits(CloudronError, Error);
+CloudronError.BAD_FIELD = 'Field error';
+CloudronError.INTERNAL_ERROR = 'Internal Error';
+CloudronError.EXTERNAL_ERROR = 'External Error';
+CloudronError.BAD_STATE = 'Bad state';
+CloudronError.ALREADY_UPTODATE = 'No Update Available';

 function initialize(callback) {
    assert.strictEqual(typeof callback, 'function');

    runStartupTasks();

-    notifyUpdate(callback);
+    callback();
 }

 function uninitialize(callback) {
@@ -67,7 +97,7 @@ function uninitialize(callback) {

    async.series([
        cron.stopJobs,
-        platform.stopAllTasks
+        platform.stop
    ], callback);
 }

@@ -78,82 +108,51 @@ function onActivated(callback) {
    // 1. mail bounces can now be sent to the cloudron owner
    // 2. the restore code path can run without sudo (since mail/ is non-root)
    async.series([
-        reverseProxy.removeDefaultConfig, // remove IP based nginx config once user is created
        platform.start,
-        cron.startJobs,
-        function checkBackupConfiguration(callback) {
-            backups.checkConfiguration(function (error, message) {
-                if (error) return callback(error);
-                notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
-            });
-        }
+        cron.startJobs
    ], callback);
 }

-function notifyUpdate(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
-    const version = safe.fs.readFileSync(paths.VERSION_FILE, 'utf8');
-    if (version === constants.VERSION) return callback();
-
-    eventlog.add(eventlog.ACTION_UPDATE_FINISH, auditSource.CRON, { errorMessage: '', oldVersion: version || 'dev', newVersion: constants.VERSION }, function (error) {
-        if (error) return callback(error);
-
-        tasks.setCompletedByType(tasks.TASK_UPDATE, { error: null }, function (error) {
-            if (error && error.reason !== BoxError.NOT_FOUND) return callback(error); // when hotfixing, task may not exist
-
-            safe.fs.writeFileSync(paths.VERSION_FILE, constants.VERSION, 'utf8');
-
-            callback();
-        });
-    });
-}
-
 // each of these tasks can fail. we will add some routes to fix/re-run them
 function runStartupTasks() {
-    const tasks = [
-        // stop all the systemd tasks
-        platform.stopAllTasks,
+    // configure nginx to be reachable by IP
+    reverseProxy.configureDefaultServer(NOOP_CALLBACK);

-        // this configures collectd to collect backup storage metrics if filesystem is used. This is also triggerd when the settings change with the rest api
-        function (callback) {
-            settings.getBackupConfig(function (error, backupConfig) {
-                if (error) return callback(error);
+    // always generate webadmin config since we have no versioning mechanism for the ejs
+    if (config.adminDomain()) reverseProxy.writeAdminConfig(config.adminDomain(), NOOP_CALLBACK);

-                backups.configureCollectd(backupConfig, callback);
-            });
-        },
+    // check activation state and start the platform
+    users.isActivated(function (error, activated) {
+        if (error) return debug(error);
+        if (!activated) return debug('initialize: not activated yet'); // not activated

-        // always generate webadmin config since we have no versioning mechanism for the ejs
-        function (callback) {
-            if (!settings.adminDomain()) return callback();
+        onActivated(NOOP_CALLBACK);
+    });
+}

-            reverseProxy.writeAdminConfig(settings.adminDomain(), callback);
-        },
+function getDisks(callback) {
+    assert.strictEqual(typeof callback, 'function');

-        // check activation state and start the platform
-        function (callback) {
-            users.isActivated(function (error, activated) {
-                if (error) return callback(error);
+    var disks = {
+        boxDataDisk: null,
+        platformDataDisk: null,
+        appsDataDisk: null
+    };

-                // configure nginx to be reachable by IP when not activated. for the moment, the IP based redirect exists even after domain is setup
-                // just in case user forgot or some network error happenned in the middle (then browser refresh takes you to activation page)
-                // we remove the config as a simple security measure to not expose IP <-> domain
-                if (!activated) {
-                    debug('runStartupTasks: not activated. generating IP based redirection config');
-                    return reverseProxy.writeDefaultConfig(callback);
-                }
+    df.file(paths.BOX_DATA_DIR).then(function (result) {
+        disks.boxDataDisk = result.filesystem;

-                onActivated(callback);
-            });
-        }
-    ];
+        return df.file(paths.PLATFORM_DATA_DIR);
+    }).then(function (result) {
+        disks.platformDataDisk = result.filesystem;

-    // we used to run tasks in parallel but simultaneous nginx reloads was causing issues
-    async.series(async.reflectAll(tasks), function (error, results) {
-        results.forEach((result, idx) => {
-            if (result.error) debug(`Startup task at index ${idx} failed: ${result.error.message}`);
-        });
+        return df.file(paths.APPS_DATA_DIR);
+    }).then(function (result) {
+        disks.appsDataDisk = result.filesystem;
+
+        callback(null, disks);
+    }).catch(function (error) {
+        callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));
    });
 }

@@ -161,32 +160,28 @@ function getConfig(callback) {
    assert.strictEqual(typeof callback, 'function');

    settings.getAll(function (error, allSettings) {
-        if (error) return callback(error);
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

        // be picky about what we send out here since this is sent for 'normal' users as well
        callback(null, {
-            apiServerOrigin: settings.apiServerOrigin(),
-            webServerOrigin: settings.webServerOrigin(),
-            adminDomain: settings.adminDomain(),
-            adminFqdn: settings.adminFqdn(),
-            mailFqdn: settings.mailFqdn(),
-            version: constants.VERSION,
-            isDemo: settings.isDemo(),
+            apiServerOrigin: config.apiServerOrigin(),
+            webServerOrigin: config.webServerOrigin(),
+            adminDomain: config.adminDomain(),
+            adminFqdn: config.adminFqdn(),
+            mailFqdn: config.mailFqdn(),
+            version: config.version(),
+            isDemo: config.isDemo(),
+            memory: os.totalmem(),
+            provider: config.provider(),
            cloudronName: allSettings[settings.CLOUDRON_NAME_KEY],
-            footer: allSettings[settings.FOOTER_KEY] || constants.FOOTER,
-            features: appstore.getFeatures(),
-            profileLocked: allSettings[settings.DIRECTORY_CONFIG_KEY].lockUserProfiles,
-            mandatory2FA: allSettings[settings.DIRECTORY_CONFIG_KEY].mandatory2FA
+            features: custom.features(),
+            supportEmail: custom.supportEmail()
        });
    });
 }

 function reboot(callback) {
-    notifications.alert(notifications.ALERT_REBOOT, 'Reboot Required', '', function (error) {
-        if (error) debug('reboot: failed to clear reboot notification.', error);
-
-        shell.sudo('reboot', [ REBOOT_CMD ], {}, callback);
-    });
+    shell.sudo('reboot', [ REBOOT_CMD ], {}, callback);
 }

 function isRebootRequired(callback) {
@@ -197,13 +192,66 @@ function isRebootRequired(callback) {
 }

 // called from cron.js
-function runSystemChecks(callback) {
-    assert.strictEqual(typeof callback, 'function');
-
+function runSystemChecks() {
    async.parallel([
+        checkBackupConfiguration,
+        checkDiskSpace,
        checkMailStatus,
        checkRebootRequired
-    ], callback);
+    ], function (error) {
+        debug('runSystemChecks: done', error);
+    });
+}
+
+function checkBackupConfiguration(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('Checking backup configuration');
+
+    backups.checkConfiguration(function (error, message) {
+        if (error) return callback(error);
+
+        notifications.alert(notifications.ALERT_BACKUP_CONFIG, 'Backup configuration is unsafe', message, callback);
+    });
+}
+
+function checkDiskSpace(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    debug('Checking disk space');
+
+    getDisks(function (error, disks) {
+        if (error) {
+            debug('df error %s', error.message);
+            return callback();
+        }
+
+        df().then(function (entries) {
+            /*
+                [{
+                filesystem: '/dev/disk1',
+                size: 499046809600,
+                used: 443222245376,
+                available: 55562420224,
+                capacity: 0.89,
+                mountpoint: '/'
+            }, ...]
+            */
+            var oos = entries.some(function (entry) {
+                // ignore other filesystems but where box, app and platform data is
+                if (entry.filesystem !== disks.boxDataDisk && entry.filesystem !== disks.platformDataDisk && entry.filesystem !== disks.appsDataDisk) return false;
+
+                return (entry.available <= (1.25 * 1024 * 1024 * 1024)); // 1.5G
+            });
+
+            debug('Disk space checked. ok: %s', !oos);
+
+            notifications.alert(notifications.ALERT_DISK_SPACE, 'Server is running out of disk space', oos ? JSON.stringify(entries, null, 4) : '', callback);
+        }).catch(function (error) {
+            if (error) console.error(error);
+            callback();
+        });
+    });
 }

 function checkMailStatus(callback) {
@@ -226,7 +274,7 @@ function checkRebootRequired(callback) {
    isRebootRequired(function (error, rebootRequired) {
        if (error) return callback(error);

-        notifications.alert(notifications.ALERT_REBOOT, 'Reboot Required', rebootRequired ? 'To finish ubuntu security updates, a reboot is necessary.' : '', callback);
+        notifications.alert(notifications.ALERT_REBOOT, 'Reboot Required', rebootRequired ? 'To finish security updates, a [reboot](/#/system) is necessary.' : '', callback);
    });
 }

@@ -251,7 +299,7 @@ function getLogs(unit, options, callback) {
    // need to handle box.log without subdir
    if (unit === 'box') args.push(path.join(paths.LOG_DIR, 'box.log'));
    else if (unit.startsWith('crash-')) args.push(path.join(paths.CRASH_LOG_DIR, unit.slice(6) + '.log'));
-    else return callback(new BoxError(BoxError.BAD_FIELD, 'No such unit', { field: 'unit' }));
+    else return callback(new CloudronError(CloudronError.BAD_FIELD, 'No such unit'));

    var cp = spawn('/usr/bin/tail', args);

@@ -283,26 +331,21 @@ function prepareDashboardDomain(domain, auditSource, callback) {

    debug(`prepareDashboardDomain: ${domain}`);

-    if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
-
    domains.get(domain, function (error, domainObject) {
-        if (error) return callback(error);
+        if (error && error.reason === DomainsError.NOT_FOUND) return callback(new CloudronError(CloudronError.BAD_FIELD, 'No such domain'));
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

        const fqdn = domains.fqdn(constants.ADMIN_LOCATION, domainObject);

        apps.getAll(function (error, result) {
-            if (error) return callback(error);
+            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

            const conflict = result.filter(app => app.fqdn === fqdn);
-            if (conflict.length) return callback(new BoxError(BoxError.BAD_STATE, 'Dashboard location conflicts with an existing app'));
+            if (conflict.length) return callback(new CloudronError(CloudronError.BAD_STATE, 'Dashboard location conflicts with an existing app'));

-            tasks.add(tasks.TASK_SETUP_DNS_AND_CERT, [ constants.ADMIN_LOCATION, domain, auditSource ], function (error, taskId) {
-                if (error) return callback(error);
-
-                tasks.startTask(taskId, {}, NOOP_CALLBACK);
-
-                callback(null, taskId);
-            });
+            let task = tasks.startTask(tasks.TASK_PREPARE_DASHBOARD_DOMAIN, [ domain, auditSource ]);
+            task.on('error', (error) => callback(new CloudronError(CloudronError.INTERNAL_ERROR, error)));
+            task.on('start', (taskId) => callback(null, taskId));
        });
    });
 }
@@ -316,15 +359,19 @@ function setDashboardDomain(domain, auditSource, callback) {
    debug(`setDashboardDomain: ${domain}`);

    domains.get(domain, function (error, domainObject) {
-        if (error) return callback(error);
+        if (error && error.reason === DomainsError.NOT_FOUND) return callback(new CloudronError(CloudronError.BAD_FIELD, 'No such domain'));
+        if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

        reverseProxy.writeAdminConfig(domain, function (error) {
-            if (error) return callback(error);
+            if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

            const fqdn = domains.fqdn(constants.ADMIN_LOCATION, domainObject);

-            settings.setAdminLocation(domain, fqdn, function (error) {
-                if (error) return callback(error);
+            config.setAdminDomain(domain);
+            config.setAdminFqdn(fqdn);
+
+            clients.addDefaultClients(config.adminOrigin(), function (error) {
+                if (error) return callback(new CloudronError(CloudronError.INTERNAL_ERROR, error));

                eventlog.add(eventlog.ACTION_DASHBOARD_DOMAIN_UPDATE, auditSource, { domain: domain, fqdn: fqdn });

@@ -335,19 +382,17 @@ function setDashboardDomain(domain, auditSource, callback) {
 }

 // call this only post activation because it will restart mail server
-function updateDashboardDomain(domain, auditSource, callback) {
+function setDashboardAndMailDomain(domain, auditSource, callback) {
    assert.strictEqual(typeof domain, 'string');
    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

-    debug(`updateDashboardDomain: ${domain}`);
-
-    if (settings.isDemo()) return callback(new BoxError(BoxError.CONFLICT, 'Not allowed in demo mode'));
+    debug(`setDashboardAndMailDomain: ${domain}`);

    setDashboardDomain(domain, auditSource, function (error) {
        if (error) return callback(error);

-        addons.restartService('turn', NOOP_CALLBACK); // to update the realm variable
+        mail.onMailFqdnChanged(NOOP_CALLBACK); // this will update dns and re-configure mail server

        callback(null);
    });
@@ -358,42 +403,7 @@ function renewCerts(options, auditSource, callback) {
    assert.strictEqual(typeof auditSource, 'object');
    assert.strictEqual(typeof callback, 'function');

-    tasks.add(tasks.TASK_RENEW_CERTS, [ options, auditSource ], function (error, taskId) {
-        if (error) return callback(error);
-
-        tasks.startTask(taskId, {}, NOOP_CALLBACK);
-
-        callback(null, taskId);
-    });
-}
-
-function setupDnsAndCert(subdomain, domain, auditSource, progressCallback, callback) {
-    assert.strictEqual(typeof subdomain, 'string');
-    assert.strictEqual(typeof domain, 'string');
-    assert.strictEqual(typeof auditSource, 'object');
-    assert.strictEqual(typeof progressCallback, 'function');
-    assert.strictEqual(typeof callback, 'function');
-
-    domains.get(domain, function (error, domainObject) {
-        if (error) return callback(error);
-
-        const adminFqdn = domains.fqdn(subdomain, domainObject);
-
-        sysinfo.getServerIp(function (error, ip) {
-            if (error) return callback(error);
-
-            async.series([
-                (done) => { progressCallback({ message: `Updating DNS of ${adminFqdn}` }); done(); },
-                domains.upsertDnsRecords.bind(null, subdomain, domain, 'A', [ ip ]),
-                (done) => { progressCallback({ message: `Waiting for DNS of ${adminFqdn}` }); done(); },
-                domains.waitForDnsRecord.bind(null, subdomain, domain, 'A', ip, { interval: 30000, times: 50000 }),
-                (done) => { progressCallback({ message: `Getting certificate of ${adminFqdn}` }); done(); },
-                reverseProxy.ensureCertificate.bind(null, domains.fqdn(subdomain, domainObject), domain, auditSource)
-            ], function (error) {
-                if (error) return callback(error);
-
-                callback(null);
-            });
-        });
-    });
+    let task = tasks.startTask(tasks.TASK_RENEW_CERTS, [ options, auditSource ]);
+    task.on('error', (error) => callback(new CloudronError(CloudronError.INTERNAL_ERROR, error)));
+    task.on('start', (taskId) => callback(null, taskId));
 }
@@ -30,13 +30,3 @@ LoadPlugin "table"
    </Result>
  </Table>
 </Plugin>
-
-<Plugin python>
-  <Module du>
-    <Path>
-        Instance "<%= appId %>"
-        Dir "<%= appDataDir %>"
-    </Path>
-  </Module>
-</Plugin>
-
@@ -1,55 +0,0 @@
-'use strict';
-
-exports = module.exports = {
-    addProfile,
-    removeProfile
-};
-
-const assert = require('assert'),
-    BoxError = require('./boxerror.js'),
-    debug = require('debug')('collectd'),
-    fs = require('fs'),
-    path = require('path'),
-    paths = require('./paths.js'),
-    safe = require('safetydance'),
-    shell = require('./shell.js');
-
-const CONFIGURE_COLLECTD_CMD = path.join(__dirname, 'scripts/configurecollectd.sh');
-
-function addProfile(name, profile, callback) {
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof profile, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    const configFilePath = path.join(paths.COLLECTD_APPCONFIG_DIR, `${name}.conf`);
-
-    // skip restarting collectd if the profile already exists with the same contents
-    const currentProfile = safe.fs.readFileSync(configFilePath, 'utf8') || '';
-    if (currentProfile === profile) return callback(null);
-
-    fs.writeFile(configFilePath, profile, function (error) {
-        if (error) return callback(new BoxError(BoxError.FS_ERROR, `Error writing collectd config: ${error.message}`));
-
-        shell.sudo('addCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'add', name ], {}, function (error) {
-            if (error) return callback(new BoxError(BoxError.COLLECTD_ERROR, 'Could not add collectd config'));
-
-            callback(null);
-        });
-    });
-
-}
-
-function removeProfile(name, callback) {
-    assert.strictEqual(typeof name, 'string');
-    assert.strictEqual(typeof callback, 'function');
-
-    fs.unlink(path.join(paths.COLLECTD_APPCONFIG_DIR, `${name}.conf`), function (error) {
-        if (error && error.code !== 'ENOENT') debug('Error removing collectd profile', error);
-
-        shell.sudo('removeCollectdProfile', [ CONFIGURE_COLLECTD_CMD, 'remove', name ], {}, function (error) {
-            if (error) return callback(new BoxError(BoxError.COLLECTD_ERROR, 'Could not remove collectd config'));
-
-            callback(null);
-        });
-    });
-}
@@ -1,9 +0,0 @@
-<Plugin python>
-  <Module du>
-    <Path>
-        Instance "cloudron-backup"
-        Dir "<%= backupDir %>"
-    </Path>
-  </Module>
-</Plugin>
-
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Johannes Zellner	51a69cce41	Ensure notifications are sorted by time descending (cherry picked from commit `63310c44c0`)	2019-05-14 09:49:26 -07:00
Johannes Zellner	8686832bd1	4.0.3 changes	2019-05-14 16:57:45 +02:00
Girish Ramakrishnan	c93c06ba88	gcs: fix crash (cherry picked from commit `05d3f8a667`)	2019-05-12 18:14:17 -07:00
Girish Ramakrishnan	36d64b3566	4.0.2 changes (cherry picked from commit `3fa45ea728`)	2019-05-12 14:04:26 -07:00
Girish Ramakrishnan	e0c531564c	Add option to skip backup before update (cherry picked from commit `a7d2098f09`)	2019-05-12 13:59:07 -07:00
Girish Ramakrishnan	1e0ec75f0a	gcdns: fix crash (cherry picked from commit `e1ecb49d59`)	2019-05-12 12:57:06 -07:00
Girish Ramakrishnan	36ac02d29f	Fix crash because params was undefined (cherry picked from commit `800e25a7a7`)	2019-05-10 13:08:26 -07:00