Use -%> for newline slurping

Fixes #383

CHANGES

@@ -902,3 +902,11 @@
 * Fix issue where mail container does not cleanup LDAP connections properly
 * Update node to 6.11.1
 
+[1.3.0]
+* Add option to configure robots.txt for each app from the web interface
+* Make sure zoneName is not lost across updates
+* Save manually triggered app backups under a datetime prefix
+* Optionally disable FROM validation check in the mail container. This will allow apps to send emails with arbitrary FROM addresses
+* Set X-Forwarded-Port in the reverse proxy. This fixes a problem with plugins of certain apps (like Jetpack)
+* Send a weekly activity digest about pending and applied Cloudron and app updates
+

docs/references/api.md

@@ -119,7 +119,8 @@ Request:
     memoryLimit: <number>,           // memory constraint in bytes
     backupId: <string>,              // initialize the app from this backup
     altDomain: <string>,             // alternate domain from which this app can be reached
-    xFrameOptions: <string>          // set X-Frame-Options header, to control which websites can embed this app
+    xFrameOptions: <string>,         // set X-Frame-Options header, to control which websites can embed this app
+    robotsTxt: <string>              // robots.txt file content
 }
 ```
 
@@ -152,11 +153,14 @@ If `altDomain` is set, the app can be accessed from `https://<altDomain>`.
 * `SAMEORIGIN` - allows embedding from the same domain as the app. This is the default.
 * `ALLOW-FROM https://example.com/` - allows this app to be embedded from example.com
 
+Read more about the options at [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options).
+
 `memoryLimit` is the maximum memory this app can use (in bytes) including swap. If set to 0, the app uses the `memoryLimit` value set in the manifest. If set to -1, the app gets unlimited memory.
 
-If `backupId` is provided the app will be initialized with the data from the backup.
+If `robotsTxt` if set, it will be returned as the response for `/robots.txt`. You can read about the
+[Robots Exclustion Protocol](http://www.robotstxt.org/robotstxt.html) site.
 
-Read more about the options at [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options).
+If `backupId` is provided the app will be initialized with the data from the backup.
 
 Response (200):
 
@@ -214,7 +218,8 @@ Response (200):
     },
     iconUrl: <url>,                  // a relative url providing the icon
     memoryLimit: <number>,           // memory constraint in bytes
-    sso: <boolean>                   // Enable single sign-on
+    sso: <boolean>,                  // Enable single sign-on
+    robotsTxt: <string>              // robots.txt file content
 }
 ```
 
@@ -474,7 +479,8 @@ Request:
     key: <string>,                  // pem encoded TLS key
     memoryLimit: <number>,          // memory constraint in bytes
     altDomain: <string>,            // alternate domain from which this app can be reached
-    xFrameOptions: <string>         // set X-Frame-Options header, to control which websites can embed this app
+    xFrameOptions: <string>,        // set X-Frame-Options header, to control which websites can embed this app
+    robotsTxt: <string>             // robots.txt file content
 ```
 
 All values are optional. See [Install app](/references/api.html#install-app) API for field descriptions.
@@ -1317,6 +1323,46 @@ Request:
 ```
 -->
 
+### Get mail from validation
+
+GET `/api/v1/settings/mail_from_validation` <scope>admin</scope>
+
+Gets the configuration of mail from header check for outbound mails.
+
+Cloudron only allows authenticated users and apps to send outbound mail. After authentication, it ensures
+that the SMTP MAIL FROM header matches either the authenticated username or the aliases of the username. This
+prevents apps and users from impersonating using other email ids.
+
+You can disable this to skip the MAIL FROM header check. Do so only if you completely trust your apps and users.
+By default, this value is true.
+
+```
+Response (200):
+{
+    enabled: <boolean>
+}
+```
+
+### Set mail from validation
+
+POST  `/api/v1/settings/mail_from_validation` <scope>admin</scope>
+
+Enables or disables the mail from header check for outbound mails.
+
+Cloudron only allows authenticated users and apps to send outbound mail. After authentication, it ensures
+that the SMTP MAIL FROM header matches either the authenticated username or the aliases of the username. This
+prevents apps and users from impersonating using other email ids.
+
+You can disable this to skip the MAIL FROM header check. Note that the Cloudron will never send out emails
+if the FROM domain does not match the Cloudron's domain regardless of this setting.
+
+Request:
+```
+{
+    enabled: <boolean>
+}
+```
+
 ### Get name
 
 GET `/api/v1/settings/cloudron_name` <scope>admin</scope>

docs/references/selfhosting.md

@@ -108,9 +108,13 @@ IP address (`https://ip`) to complete the installation.
 The setup website will show a certificate warning. Accept the self-signed certificate
 and proceed to the domain setup.
 
-Currently, only subdomains of the [Public Suffix List](https://publicsuffix.org/) are supported.
-For example, `example.com`,  `example.co.uk` will work fine. Choosing other non-registrable
-domain names like `cloudron.example.com` will not work.
+Cloudron requires a subdomain of the [Public Suffix List](https://publicsuffix.org/).
+For example, `example.com`,  `example.co.uk` will work fine.
+
+If you want to install Cloudron on a non-registrable domain like `cloudron.example.com`,
+you must purchase an Enterprise subscription. This allows for setups where you can host
+multiple Cloudrons under the same top level domain like `customer1.company.com`,
+`customer2.company.com` and so on.
 
 ### Route 53
 

docs/references/usermanual.md

@@ -138,6 +138,35 @@ Note that all data associated with the app will be immediately removed from the
 persist in your old backups and the [CLI tool](https://git.cloudron.io/cloudron/cloudron-cli) provides a way to
 restore from those old backups should it be required.
 
+## Migration
+
+Migrating apps from one Cloudron to another works by first creating a new backup of the app on the old Cloudron,
+copying the backup tarball onto the new Cloudron's backup storage and then installing a new app, based on the backup on
+the new Cloudron.
+
+**Both Cloudrons have to use the same backup encryption key!**
+
+The following step will migrate an app:
+* Against the old Cloudron
+```
+cloudron backup create --app <subdomain/appid>
+```
+* Copy the new backup from the old Cloudron's backup storage to the new one. This can be done via the s3 webinterface
+  or scp if the filesystem backend is used. The backup can be located by its backup ID, which can be seen with:
+```
+cloudron backup list --app <subdomain/appid>
+```
+* Make note of the app's appstore id and version from:
+```
+cloudron list
+```
+* Then login to the new Cloudron and install the new app based on the created backup:
+```
+cloudron login <new Cloudron domain>
+cloudron install --appstore-id=<apps appstore id>@<specific version if required> --backup <backupId>
+```
+The backupId usually also includes a path prefix and looks like: `2017-07-17-121412-248/app_2d7f2a6a-4c17-43a6-80bc-0bd47a99727f_2017-07-17-121412-269_v4.1.1.tar.gz`
+
 ## Embedding Apps
 
 It is possible to embed Cloudron apps into other websites. By default, this is disabled to prevent
@@ -298,6 +327,13 @@ your mail client but login using the Cloudron credentials.
 Emails addressed to `<username>+tag@<domain>` will be delivered to the `username` mailbox. You can use this feature to give out emails of the form
 `username+kayak@<domain>`, `username+aws@<domain>` and so on and have them all delivered to your mailbox.
 
+## Catch All
+
+A Catch-all mailbox is one that will "catch all" of the emails addressed to non-existent addresses. You can forward
+such emails to one or more user mailboxes in the Email section. Note that if you do not select any mailbox (the default), Cloudron will send a bounce.
+
+<img src="/docs/img/catch-all-mailbox.png" width="500" class="shadow">
+
 ## Forwarding addresses
 
 Each group on the Cloudron is also a forwarding address. Mails can be addressed to `group@<domain>` and the mail will
@@ -308,6 +344,17 @@ be sent to each user who is part of the group.
 The spam detection agent on the Cloudron requires training to identify spam. To do this, simply move your junk mails
 to a pre-created folder named `Spam`. Most mail clients have a Junk or Spam button which does this automatically.
 
+## Mail relay
+
+By default, Cloudron's built-in email server sends out email directly to recipients. You can instead configure
+the Cloudron to hand all outgoing emails to a 'mail relay' and have the relay deliver it to recipients. Such a
+setup is useful when the Cloudron server does not have a good IP reputation for mail delivery or if server service provider does not allow sending email via port 25 (which is the case with Google Cloud and Amazon EC2).
+
+Cloudron can be configured to send all outbound email via Amazon SES, Google, Mailgun, Postmark,
+Sendgrid or any other external SMTP server. To setup a relay, enter the relay credentials in the Email section. Cloudron only supports relaying via the STARTTLS mechanism (usually port 587).
+
+<img src="/docs/img/email-relay.png" width=500 class="shadow">
+
 # Graphs
 
 The Graphs view shows an overview of the disk and memory usage on your Cloudron.

migrations/20170714170908-appdb-add-robotsTxt.js

@@ -0,0 +1,15 @@
+'use strict';
+
+exports.up = function(db, callback) {
+    db.runSql('ALTER TABLE apps ADD COLUMN robotsTxt TEXT', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};
+
+exports.down = function(db, callback) {
+    db.runSql('ALTER TABLE apps DROP COLUMN robotsTxt', function (error) {
+        if (error) console.error(error);
+        callback(error);
+    });
+};

migrations/schema.sql

@@ -68,6 +68,7 @@ CREATE TABLE IF NOT EXISTS apps(
     xFrameOptions VARCHAR(512),
     sso BOOLEAN DEFAULT 1, // whether user chose to enable SSO
     debugModeJson TEXT, // options for development mode
+    robotsTxt TEXT,
 
     // the following fields do not belong here, they can be removed when we use a queue for apptask
     lastBackupId VARCHAR(128), // used to pass backupId to restore from to apptask

scripts/cloudron-setup

@@ -21,11 +21,17 @@ readonly MINIMUM_MEMORY="974"      # this is mostly reported for 1GB main memory
 readonly curl="curl --fail --connect-timeout 20 --retry 10 --retry-delay 2 --max-time 2400"
 
 # copied from cloudron-resize-fs.sh
+readonly rootfs_type=$(LC_ALL=C df --output=fstype / | tail -n1)
 readonly physical_memory=$(LC_ALL=C free -m | awk '/Mem:/ { print $2 }')
 readonly disk_size_bytes=$(LC_ALL=C df --output=size / | tail -n1)
 readonly disk_size_gb=$((${disk_size_bytes}/1024/1024))
 
 # verify the system has minimum requirements met
+if [[ "${rootfs_type}" != "ext4" ]]; then
+    echo "Error: Cloudron requires '/' to be ext4" # see #364
+    exit 1
+fi
+
 if [[ "${physical_memory}" -lt "${MINIMUM_MEMORY}" ]]; then
     echo "Error: Cloudron requires atleast 1GB physical memory"
     exit 1
@@ -39,6 +45,7 @@ fi
 initBaseImage="true"
 # provisioning data
 domain=""
+zoneName=""
 provider=""
 encryptionKey=""
 restoreUrl=""
@@ -188,6 +195,7 @@ if [[ -z "${dataJson}" ]]; then
     {
         "boxVersionsUrl": "${versionsUrl}",
         "fqdn": "${domain}",
+        "zoneName": "${zoneName}",
         "provider": "${provider}",
         "apiServerOrigin": "${apiServerOrigin}",
         "webServerOrigin": "${webServerOrigin}",
@@ -215,6 +223,7 @@ EOF
     {
         "boxVersionsUrl": "${versionsUrl}",
         "fqdn": "${domain}",
+        "zoneName": "${zoneName}",
         "provider": "${provider}",
         "apiServerOrigin": "${apiServerOrigin}",
         "webServerOrigin": "${webServerOrigin}",

setup/argparser.sh

@@ -6,6 +6,7 @@ json="${source_dir}/../node_modules/.bin/json"
 # IMPORTANT: Fix cloudron.js:doUpdate if you add/remove any arg. keep these sorted for readability
 arg_api_server_origin=""
 arg_fqdn=""
+arg_zone_name=""
 arg_is_custom_domain="false"
 arg_restore_key=""
 arg_restore_url=""
@@ -40,6 +41,7 @@ while true; do
     --data)
         # these params must be valid in all cases
         arg_fqdn=$(echo "$2" | $json fqdn)
+        arg_zone_name=$(echo "$2" | $json zoneName)
 
         arg_is_custom_domain=$(echo "$2" | $json isCustomDomain)
         [[ "${arg_is_custom_domain}" == "" ]] && arg_is_custom_domain="true"

setup/splashpage.sh

@@ -34,11 +34,11 @@ if [[ "${arg_retire_reason}" != "" || "${existing_infra}" != "${current_infra}"
     echo "Showing progress bar on all subdomains in retired mode or infra update. retire: ${arg_retire_reason} existing: ${existing_infra} current: ${current_infra}"
     rm -f ${PLATFORM_DATA_DIR}/nginx/applications/*
     ${box_src_dir}/node_modules/.bin/ejs-cli -f "${script_dir}/start/nginx/appconfig.ejs" \
-        -O "{ \"vhost\": \"~^(.+)\$\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\" }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
+        -O "{ \"vhost\": \"~^(.+)\$\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
 else
     echo "Show progress bar only on admin domain for normal update"
     ${box_src_dir}/node_modules/.bin/ejs-cli -f "${script_dir}/start/nginx/appconfig.ejs" \
-        -O "{ \"vhost\": \"${admin_fqdn}\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\" }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
+        -O "{ \"vhost\": \"${admin_fqdn}\", \"adminOrigin\": \"${admin_origin}\", \"endpoint\": \"splash\", \"sourceDir\": \"${SETUP_WEBSITE_DIR}\", \"certFilePath\": \"cert/host.cert\", \"keyFilePath\": \"cert/host.key\", \"xFrameOptions\": \"SAMEORIGIN\", \"robotsTxtQuoted\": null }" > "${PLATFORM_DATA_DIR}/nginx/applications/admin.conf"
 fi
 
 if [[ "${arg_retire_reason}" == "migrate" ]]; then

setup/start.sh

@@ -249,6 +249,7 @@ cat > "${CONFIG_DIR}/cloudron.conf" <<CONF_END
     "apiServerOrigin": "${arg_api_server_origin}",
     "webServerOrigin": "${arg_web_server_origin}",
     "fqdn": "${arg_fqdn}",
+    "zoneName": "${arg_zone_name}",
     "isCustomDomain": ${arg_is_custom_domain},
     "provider": "${arg_provider}",
     "isDemo": ${arg_is_demo},

setup/start/nginx/appconfig.ejs

@@ -56,6 +56,7 @@ server {
     proxy_set_header   Host               $host;
     proxy_set_header   X-Forwarded-For    $remote_addr;
     proxy_set_header   X-Forwarded-Host   $host;
+    proxy_set_header   X-Forwarded-Port   $server_port;
     proxy_set_header   X-Forwarded-Proto  https;
     proxy_set_header   X-Forwarded-Ssl    on;
 
@@ -82,6 +83,12 @@ server {
         # Disable check to allow unlimited body sizes
         client_max_body_size 0;
 
+<% if (robotsTxtQuoted) { %>
+        location = /robots.txt {
+            return 200 <%- robotsTxtQuoted %>;
+        }
+<% } %>
+
 <% if ( endpoint === 'admin' ) { %>
         location /api/ {
             proxy_pass   http://127.0.0.1:3000;

setup/start/nginx/nginx.conf

@@ -15,12 +15,12 @@ http {
     # the collectd config depends on this log format
     log_format combined2 '$remote_addr - [$time_local] '
         '"$request" $status $body_bytes_sent $request_time '
-        '"$http_referer" "$http_user_agent"';
+        '"$http_referer" "$host" "$http_user_agent"';
 
     # required for long host names
     server_names_hash_bucket_size 128;
 
-    access_log access.log combined2;
+    access_log /var/log/nginx/access.log combined2;
 
     sendfile        on;
 

src/appdb.js

@@ -60,7 +60,7 @@ var assert = require('assert'),
 var APPS_FIELDS_PREFIXED = [ 'apps.id', 'apps.appStoreId', 'apps.installationState', 'apps.installationProgress', 'apps.runState',
     'apps.health', 'apps.containerId', 'apps.manifestJson', 'apps.httpPort', 'apps.location', 'apps.dnsRecordId',
     'apps.accessRestrictionJson', 'apps.lastBackupId', 'apps.oldConfigJson', 'apps.memoryLimit', 'apps.altDomain',
-    'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson' ].join(',');
+    'apps.xFrameOptions', 'apps.sso', 'apps.debugModeJson', 'apps.robotsTxt' ].join(',');
 
 var PORT_BINDINGS_FIELDS = [ 'hostPort', 'environmentVariable', 'appId' ].join(',');
 

src/apps.js

@@ -243,6 +243,17 @@ function validateDebugMode(debugMode) {
     return null;
 }
 
+function validateRobotsTxt(robotsTxt) {
+    if (robotsTxt === null) return null;
+
+    // this is the nginx limit on inline strings. if we really hit this, we have to generate a file
+    if (robotsTxt.length > 4096) return new AppsError(AppsError.BAD_FIELD, 'robotsTxt must be less than 4096');
+
+    // TODO: validate the robots file? we escape the string when templating the nginx config right now
+
+    return null;
+}
+
 function getDuplicateErrorDetails(location, portBindings, error) {
     assert.strictEqual(typeof location, 'string');
     assert.strictEqual(typeof portBindings, 'object');
@@ -403,6 +414,7 @@ function install(data, auditSource, callback) {
         xFrameOptions = data.xFrameOptions || 'SAMEORIGIN',
         sso = 'sso' in data ? data.sso : null,
         debugMode = data.debugMode || null,
+        robotsTxt = data.robotsTxt || null,
         backupId = data.backupId || null;
 
     assert(data.appStoreId || data.manifest); // atleast one of them is required
@@ -434,6 +446,9 @@ function install(data, auditSource, callback) {
         error = validateDebugMode(debugMode);
         if (error) return callback(error);
 
+        error = validateRobotsTxt(robotsTxt);
+        if (error) return callback(error);
+
         if ('sso' in data && !('optionalSso' in manifest)) return callback(new AppsError(AppsError.BAD_FIELD, 'sso can only be specified for apps with optionalSso'));
         // if sso was unspecified, enable it by default if possible
         if (sso === null) sso = !!manifest.addons['ldap'] || !!manifest.addons['oauth'];
@@ -548,6 +563,12 @@ function configure(appId, data, auditSource, callback) {
             if (error) return callback(error);
         }
 
+        if ('robotsTxt' in data) {
+            values.robotsTxt = data.robotsTxt || null;
+            error = validateRobotsTxt(values.robotsTxt);
+            if (error) return callback(error);
+        }
+
         // save cert to boxdata/certs. TODO: move this to apptask when we have a real task queue
         if ('cert' in data && 'key' in data) {
             if (data.cert && data.key) {

src/appstore.js

@@ -17,6 +17,7 @@ exports = module.exports = {
 var assert = require('assert'),
     config = require('./config.js'),
     debug = require('debug')('box:appstore'),
+    eventlog = require('./eventlog.js'),
     os = require('os'),
     settings = require('./settings.js'),
     superagent = require('superagent'),
@@ -149,51 +150,58 @@ function sendAliveStatus(data, callback) {
     settings.getAll(function (error, result) {
         if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
 
-        var backendSettings = {
-            dnsConfig: {
-                provider: result[settings.DNS_CONFIG_KEY].provider,
-                wildcard: result[settings.DNS_CONFIG_KEY].provider === 'manual' ? result[settings.DNS_CONFIG_KEY].wildcard : undefined
-            },
-            tlsConfig: {
-                provider: result[settings.TLS_CONFIG_KEY].provider
-            },
-            backupConfig: {
-                provider: result[settings.BACKUP_CONFIG_KEY].provider
-            },
-            mailConfig: {
-                enabled: result[settings.MAIL_CONFIG_KEY].enabled
+        eventlog.getAllPaged(eventlog.ACTION_USER_LOGIN, null, 1, 1, function (error, loginEvents) {
+            if (error) return callback(new AppstoreError(AppstoreError.INTERNAL_ERROR, error));
+
+            var backendSettings = {
+                dnsConfig: {
+                    provider: result[settings.DNS_CONFIG_KEY].provider,
+                    wildcard: result[settings.DNS_CONFIG_KEY].provider === 'manual' ? result[settings.DNS_CONFIG_KEY].wildcard : undefined
+                },
+                tlsConfig: {
+                    provider: result[settings.TLS_CONFIG_KEY].provider
+                },
+                backupConfig: {
+                    provider: result[settings.BACKUP_CONFIG_KEY].provider
+                },
+                mailConfig: {
+                    enabled: result[settings.MAIL_CONFIG_KEY].enabled
             },
             mailRelay: {
                 provider: result[settings.MAIL_RELAY_KEY].provider
             },
             mailCatchAll: {
-                count: result[settings.CATCH_ALL_ADDRESS].length
-            },
-            autoupdatePattern: result[settings.AUTOUPDATE_PATTERN_KEY],
-            timeZone: result[settings.TIME_ZONE_KEY]
-        };
+                count: result[settings.CATCH_ALL_ADDRESS_KEY].length
+                },
+                autoupdatePattern: result[settings.AUTOUPDATE_PATTERN_KEY],
+                timeZone: result[settings.TIME_ZONE_KEY],
+            };
 
-        var data = {
-            domain: config.fqdn(),
-            version: config.version(),
-            provider: config.provider(),
-            backendSettings: backendSettings,
-            machine: {
-                cpus: os.cpus(),
-                totalmem: os.totalmem()
-            }
-        };
+            var data = {
+                domain: config.fqdn(),
+                version: config.version(),
+                provider: config.provider(),
+                backendSettings: backendSettings,
+                machine: {
+                    cpus: os.cpus(),
+                    totalmem: os.totalmem()
+                },
+                events: {
+                    lastLogin: loginEvents[0] ? (new Date(loginEvents[0].creationTime).getTime()) : 0
+                }
+            };
 
-        getAppstoreConfig(function (error, appstoreConfig) {
-            if (error) return callback(error);
+            getAppstoreConfig(function (error, appstoreConfig) {
+                if (error) return callback(error);
 
-            var url = config.apiServerOrigin() + '/api/v1/users/' + appstoreConfig.userId + '/cloudrons/' + appstoreConfig.cloudronId + '/alive';
-            superagent.post(url).send(data).query({ accessToken: appstoreConfig.token }).timeout(30 * 1000).end(function (error, result) {
-                if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
-                if (result.statusCode === 404) return callback(new AppstoreError(AppstoreError.NOT_FOUND));
-                if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Sending alive status failed. %s %j', result.status, result.body)));
+                var url = config.apiServerOrigin() + '/api/v1/users/' + appstoreConfig.userId + '/cloudrons/' + appstoreConfig.cloudronId + '/alive';
+                superagent.post(url).send(data).query({ accessToken: appstoreConfig.token }).timeout(30 * 1000).end(function (error, result) {
+                    if (error && !error.response) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, error));
+                    if (result.statusCode === 404) return callback(new AppstoreError(AppstoreError.NOT_FOUND));
+                    if (result.statusCode !== 201) return callback(new AppstoreError(AppstoreError.EXTERNAL_ERROR, util.format('Sending alive status failed. %s %j', result.status, result.body)));
 
-                callback(null);
+                    callback(null);
+                });
             });
         });
     });

src/apptask.js

@@ -438,9 +438,11 @@ function backup(app, callback) {
     assert.strictEqual(typeof app, 'object');
     assert.strictEqual(typeof callback, 'function');
 
+    var prefix = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
+
     async.series([
         updateApp.bind(null, app, { installationProgress: '10, Backing up' }),
-        backups.backupApp.bind(null, app, app.manifest, 'appbackups' /* tag */),
+        backups.backupApp.bind(null, app, app.manifest, prefix),
 
         // done!
         function (callback) {
@@ -557,9 +559,11 @@ function update(app, callback) {
         function (next) {
             if (app.installationState === appdb.ISTATE_PENDING_FORCE_UPDATE) return next(null);
 
+            var prefix = (new Date()).toISOString().replace(/[T.]/g, '-').replace(/[:Z]/g,'');
+
             async.series([
                 updateApp.bind(null, app, { installationProgress: '30, Backing up app' }),
-                backups.backupApp.bind(null, app, app.oldConfig.manifest, 'appbackups' /* tag */)
+                backups.backupApp.bind(null, app, app.oldConfig.manifest, prefix)
             ], next);
         },
 

src/cloudron.js

@@ -702,6 +702,7 @@ function doUpdate(boxUpdateInfo, callback) {
             tlsKey: config.tlsKey(),
             isCustomDomain: config.isCustomDomain(),
             isDemo: config.isDemo(),
+            zoneName: config.zoneName(),
 
             appstore: {
                 token: config.token(),

src/cron.js

@@ -15,6 +15,7 @@ var apps = require('./apps.js'),
     constants = require('./constants.js'),
     CronJob = require('cron').CronJob,
     debug = require('debug')('box:cron'),
+    digest = require('./digest.js'),
     eventlog = require('./eventlog.js'),
     janitor = require('./janitor.js'),
     scheduler = require('./scheduler.js'),
@@ -22,20 +23,21 @@ var apps = require('./apps.js'),
     semver = require('semver'),
     updateChecker = require('./updatechecker.js');
 
-var gAutoupdaterJob = null,
-    gBoxUpdateCheckerJob = null,
+var gAliveJob = null, // send periodic stats
     gAppUpdateCheckerJob = null,
-    gHeartbeatJob = null, // for CaaS health check
-    gAliveJob = null, // send periodic stats
+    gAutoupdaterJob = null,
     gBackupJob = null,
-    gCleanupTokensJob = null,
-    gCleanupBackupsJob = null,
-    gDockerVolumeCleanerJob = null,
-    gSchedulerSyncJob = null,
+    gBoxUpdateCheckerJob = null,
     gCertificateRenewJob = null,
     gCheckDiskSpaceJob = null,
+    gCleanupBackupsJob = null,
     gCleanupEventlogJob = null,
-    gDynamicDNSJob = null;
+    gCleanupTokensJob = null,
+    gDockerVolumeCleanerJob = null,
+    gDynamicDNSJob = null,
+    gHeartbeatJob = null, // for CaaS health check
+    gSchedulerSyncJob = null,
+    gDigestEmailJob = null;
 
 var NOOP_CALLBACK = function (error) { if (error) console.error(error); };
 var AUDIT_SOURCE = { userId: null, username: 'cron' };
@@ -173,6 +175,14 @@ function recreateJobs(tz) {
         start: true,
         timeZone: tz
     });
+
+    if (gDigestEmailJob) gDigestEmailJob.stop();
+    gDigestEmailJob = new CronJob({
+        cronTime: '00 00 * * * 3', // every tuesday
+        onTick: digest.maybeSend,
+        start: true,
+        timeZone: tz
+    });
 }
 
 function autoupdatePatternChanged(pattern) {
@@ -272,5 +282,8 @@ function uninitialize(callback) {
     if (gDynamicDNSJob) gDynamicDNSJob.stop();
     gDynamicDNSJob = null;
 
+    if (gDigestEmailJob) gDigestEmailJob.stop();
+    gDigestEmailJob = null;
+
     callback();
 }

src/digest.js

@@ -0,0 +1,64 @@
+'use strict';
+
+var appstore = require('./appstore.js'),
+    debug = require('debug')('box:digest'),
+    eventlog = require('./eventlog.js'),
+    updatechecker = require('./updatechecker.js'),
+    mailer = require('./mailer.js'),
+    settings = require('./settings.js');
+
+var NOOP_CALLBACK = function (error) { if (error) debug(error); };
+
+exports = module.exports = {
+    maybeSend: maybeSend
+};
+
+function maybeSend(callback) {
+    callback = callback || NOOP_CALLBACK;
+
+    settings.getEmailDigest(function (error, enabled) {
+        if (error) return callback(error);
+
+        if (!enabled) {
+            debug('Email digest is disabled');
+            return callback();
+        }
+
+        var updateInfo = updatechecker.getUpdateInfo();
+        var pendingAppUpdates = updateInfo.apps || {};
+        pendingAppUpdates = Object.keys(pendingAppUpdates).map(function (key) { return pendingAppUpdates[key]; });
+
+        appstore.getSubscription(function (error, result) {
+            if (error) debug('Error getting subscription:', error);
+
+            var hasSubscription = result && result.plan.id !== 'free' && result.plan.id !== 'undecided';
+
+            eventlog.getByActionLastWeek(eventlog.ACTION_APP_UPDATE, function (error, appUpdates) {
+                if (error) return callback(error);
+
+                eventlog.getByActionLastWeek(eventlog.ACTION_UPDATE, function (error, boxUpdates) {
+                    if (error) return callback(error);
+
+                    var info = {
+                        hasSubscription: hasSubscription,
+
+                        pendingAppUpdates: pendingAppUpdates,
+                        pendingBoxUpdate: updateInfo.box || null,
+
+                        finishedAppUpdates: (appUpdates || []).map(function (e) { return e.data; }),
+                        finishedBoxUpdates: (boxUpdates || []).map(function (e) { return e.data; })
+                    };
+
+                    if (info.pendingAppUpdates.length || info.pendingBoxUpdate || info.finishedAppUpdates.length || info.finishedBoxUpdates.length) {
+                        debug('maybeSend: sending digest email', info);
+                        mailer.sendDigest(info);
+                    } else {
+                        debug('maybeSend: nothing happened, NOT sending digest email');
+                    }
+
+                    callback();
+                });
+            });
+        });
+    });
+}

src/eventlog.js

@@ -6,6 +6,7 @@ exports = module.exports = {
     add: add,
     get: get,
     getAllPaged: getAllPaged,
+    getByActionLastWeek: getByActionLastWeek,
     cleanup: cleanup,
 
     // keep in sync with webadmin index.js filter and CLI tool
@@ -103,6 +104,17 @@ function getAllPaged(action, search, page, perPage, callback) {
     });
 }
 
+function getByActionLastWeek(action, callback) {
+    assert(typeof action === 'string' || action === null);
+    assert.strictEqual(typeof callback, 'function');
+
+    eventlogdb.getByActionLastWeek(action, function (error, boxes) {
+        if (error) return callback(new EventLogError(EventLogError.INTERNAL_ERROR, error));
+
+        callback(null, boxes);
+    });
+}
+
 function cleanup(callback) {
     callback = callback || NOOP_CALLBACK;
 

src/eventlogdb.js

@@ -3,6 +3,7 @@
 exports = module.exports = {
     get: get,
     getAllPaged: getAllPaged,
+    getByActionLastWeek: getByActionLastWeek,
     add: add,
     count: count,
     delByCreationTime: delByCreationTime,
@@ -71,6 +72,20 @@ function getAllPaged(action, search, page, perPage, callback) {
     });
 }
 
+function getByActionLastWeek(action, callback) {
+    assert(typeof action === 'string' || action === null);
+    assert.strictEqual(typeof callback, 'function');
+
+    var query = 'SELECT ' + EVENTLOGS_FIELDS + ' FROM eventlog WHERE action=? AND creationTime >= DATE_SUB(NOW(), INTERVAL 1 WEEK) ORDER BY creationTime DESC';
+    database.query(query, [ action ], function (error, results) {
+        if (error) return callback(new DatabaseError(DatabaseError.INTERNAL_ERROR, error));
+
+        results.forEach(postProcess);
+
+        callback(null, results);
+    });
+}
+
 function add(id, action, source, data, callback) {
     assert.strictEqual(typeof id, 'string');
     assert.strictEqual(typeof action, 'string');

src/infra_version.js

@@ -7,7 +7,7 @@
 exports = module.exports = {
     // a major version makes all apps restore from backup
     // a minor version makes all apps re-configure themselves
-    'version': '48.4.0',
+    'version': '48.5.0',
 
     'baseImages': [ 'cloudron/base:0.10.0' ],
 
@@ -18,7 +18,7 @@ exports = module.exports = {
         'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:0.17.0' },
         'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:0.13.0' },
         'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:0.11.0' },
-        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:0.34.2' },
+        'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:0.35.0' },
         'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:0.11.0' }
     }
 };

src/mail_templates/digest.ejs

@@ -0,0 +1,47 @@
+<% if (format === 'text') { -%>
+
+Dear <%= cloudronName %> Admin,
+
+This is the weekly summary of activities on your Cloudron <%= fqdn %>.
+<% if (info.pendingBoxUpdate) { -%>
+
+Cloudron v<%- info.pendingBoxUpdate.version %> is available:
+<% for (var i = 0; i < info.pendingBoxUpdate.changelog.length; i++) { -%>
+    * <%- info.pendingBoxUpdate.changelog[i] %>
+<% }} -%>
+<% if (info.pendingAppUpdates.length) { -%>
+
+One or more app updates are available:
+<% for (var i = 0; i < info.pendingAppUpdates.length; i++) { -%>
+    - <%= info.pendingAppUpdates[i].manifest.title %> package v<%= info.pendingAppUpdates[i].manifest.version %>
+<% for (var j = 0; j < info.pendingAppUpdates[i].manifest.changelog.trim().split('\n').length; j++) { -%>
+        <%= info.pendingAppUpdates[i].manifest.changelog.trim().split('\n')[j] %>
+<% }}} -%>
+<% if (info.finishedBoxUpdates.length) { -%>
+
+Cloudron was updated with the following releases:
+<% for (var i = 0; i < info.finishedBoxUpdates.length; i++) { -%>
+    - Version <%= info.finishedBoxUpdates[i].boxUpdateInfo.version %>
+<% for (var j = 0; j < info.finishedBoxUpdates[i].boxUpdateInfo.changelog.length; j++) { -%>
+        * <%= info.finishedBoxUpdates[i].boxUpdateInfo.changelog[j] %>
+<% }}} -%>
+<% if (info.finishedAppUpdates.length) { -%>
+
+The following apps were updated:
+<% for (var i = 0; i < info.finishedAppUpdates.length; i++) { -%>
+    - <%= info.finishedAppUpdates[i].toManifest.title %> package v<%= info.finishedAppUpdates[i].toManifest.version %>
+<% for (var j = 0; j < info.finishedAppUpdates[i].toManifest.changelog.trim().split('\n').length; j++) { -%>
+        <%= info.finishedAppUpdates[i].toManifest.changelog.trim().split('\n')[j] %>
+<% }}} -%>
+<% if (!info.hasSubscription) { -%>
+
+*Keep your Cloudron automatically up-to-date and secure by upgrading to a paid plan at* <%= webadminUrl %>/#/settings
+<% } -%>
+
+Powered by https://cloudron.io
+
+Sent at: <%= new Date().toUTCString() %>
+
+<% } else { %>
+
+<% } %>

src/mailer.js

@@ -10,6 +10,7 @@ exports = module.exports = {
     passwordReset: passwordReset,
     boxUpdateAvailable: boxUpdateAvailable,
     appUpdateAvailable: appUpdateAvailable,
+    sendDigest: sendDigest,
 
     sendInvite: sendInvite,
     unexpectedExit: unexpectedExit,
@@ -403,6 +404,30 @@ function appUpdateAvailable(app, updateInfo) {
     });
 }
 
+function sendDigest(info) {
+    assert.strictEqual(typeof info, 'object');
+
+    getAdminEmails(function (error, adminEmails) {
+        if (error) return console.log('Error getting admins', error);
+
+        settings.getCloudronName(function (error, cloudronName) {
+            if (error) {
+                debug(error);
+                cloudronName = 'Cloudron';
+            }
+
+             var mailOptions = {
+                from: mailConfig().from,
+                to: adminEmails.join(', '),
+                subject: util.format('[%s] Cloudron - Weekly activity digest', config.fqdn()),
+                text: render('digest.ejs', { fqdn: config.fqdn(), webadminUrl: config.adminOrigin(), cloudronName: cloudronName, info: info, format: 'text' })
+            };
+
+            enqueue(mailOptions);
+        });
+    });
+}
+
 function outOfDiskSpace(message) {
     assert.strictEqual(typeof message, 'string');
 

src/nginx.js

@@ -35,7 +35,8 @@ function configureAdmin(certFilePath, keyFilePath, configFileName, vhost, callba
         endpoint: 'admin',
         certFilePath: certFilePath,
         keyFilePath: keyFilePath,
-        xFrameOptions: 'SAMEORIGIN'
+        xFrameOptions: 'SAMEORIGIN',
+        robotsTxtQuoted: JSON.stringify('User-agent: *\nDisallow: /\n')
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);
     var nginxConfigFilename = path.join(paths.NGINX_APPCONFIG_DIR, configFileName);
@@ -63,6 +64,7 @@ function configureApp(app, certFilePath, keyFilePath, callback) {
         endpoint: endpoint,
         certFilePath: certFilePath,
         keyFilePath: keyFilePath,
+        robotsTxtQuoted: app.robotsTxt ? JSON.stringify(app.robotsTxt) : null,
         xFrameOptions: app.xFrameOptions || 'SAMEORIGIN'    // once all apps have been updated/
     };
     var nginxConf = ejs.render(NGINX_APPCONFIG_EJS, data);

src/platform.js

@@ -247,32 +247,31 @@ function createMailConfig(callback) {
         var alertsTo = config.provider() === 'caas' ? [ 'support@cloudron.io' ] : [ ];
         alertsTo.concat(error ? [] : owner.email).join(','); // owner may not exist yet
 
-        settings.getCatchAllAddress(function (error, address) {
+        settings.getAll(function (error, result) {
             if (error) return callback(error);
 
-            var catchAll = address.join(',');
+            var catchAll = result[settings.CATCH_ALL_ADDRESS_KEY].join(',');
+            var mailFromValidation = result[settings.MAIL_FROM_VALIDATION_KEY];
 
             if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/mail/mail.ini',
-                `mail_domain=${fqdn}\nmail_server_name=${mailFqdn}\nalerts_from=${alertsFrom}\nalerts_to=${alertsTo}\ncatch_all=${catchAll}\n`, 'utf8')) {
+                `mail_domain=${fqdn}\nmail_server_name=${mailFqdn}\nalerts_from=${alertsFrom}\nalerts_to=${alertsTo}\ncatch_all=${catchAll}\nmail_from_validation=${mailFromValidation}\n`, 'utf8')) {
                 return callback(new Error('Could not create mail var file:' + safe.error.message));
             }
 
-            settings.getMailRelay(function (error, relay) {
-                if (error) return callback(error);
+            var relay = result[settings.MAIL_RELAY_KEY];
 
-                const enabled = relay.provider !== 'cloudron-smtp' ? true : false,
-                      host = relay.host || '',
-                      port = relay.port || 25,
-                      username = relay.username || '',
-                      password = relay.password || '';
+            const enabled = relay.provider !== 'cloudron-smtp' ? true : false,
+                  host = relay.host || '',
+                  port = relay.port || 25,
+                  username = relay.username || '',
+                  password = relay.password || '';
 
-                if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/mail/smtp_forward.ini',
-                    `enable_outbound=${enabled}\nhost=${host}\nport=${port}\nenable_tls=true\nauth_type=plain\nauth_user=${username}\nauth_pass=${password}`, 'utf8')) {
-                    return callback(new Error('Could not create mail var file:' + safe.error.message));
-                }
+            if (!safe.fs.writeFileSync(paths.ADDON_CONFIG_DIR + '/mail/smtp_forward.ini',
+                `enable_outbound=${enabled}\nhost=${host}\nport=${port}\nenable_tls=true\nauth_type=plain\nauth_user=${username}\nauth_pass=${password}`, 'utf8')) {
+                return callback(new Error('Could not create mail var file:' + safe.error.message));
+            }
 
-                callback();
-            });
+            callback();
         });
     });
 }

src/routes/apps.js

@@ -57,7 +57,8 @@ function removeInternalAppFields(app) {
         cnameTarget: app.cnameTarget,
         xFrameOptions: app.xFrameOptions,
         sso: app.sso,
-        debugMode: app.debugMode
+        debugMode: app.debugMode,
+        robotsTxt: app.robotsTxt
     };
 }
 
@@ -132,6 +133,8 @@ function installApp(req, res, next) {
 
     if (('debugMode' in data) && typeof data.debugMode !== 'object') return next(new HttpError(400, 'debugMode must be an object'));
 
+    if (data.robotsTxt && typeof data.robotsTxt !== 'string') return next(new HttpError(400, 'robotsTxt must be a string'));
+
     debug('Installing app :%j', data);
 
     apps.install(data, auditSource(req), function (error, app) {
@@ -170,6 +173,8 @@ function configureApp(req, res, next) {
 
     if (('debugMode' in data) && typeof data.debugMode !== 'object') return next(new HttpError(400, 'debugMode must be an object'));
 
+    if (data.robotsTxt && typeof data.robotsTxt !== 'string') return next(new HttpError(400, 'robotsTxt must be a string'));
+
     debug('Configuring app id:%s data:%j', req.params.id, data);
 
     apps.configure(req.params.id, data, auditSource(req), function (error) {

src/routes/settings.js

@@ -30,6 +30,9 @@ exports = module.exports = {
     getCatchAllAddress: getCatchAllAddress,
     setCatchAllAddress: setCatchAllAddress,
 
+    getMailFromValidation: getMailFromValidation,
+    setMailFromValidation: setMailFromValidation,
+
     getAppstoreConfig: getAppstoreConfig,
     setAppstoreConfig: setAppstoreConfig,
 
@@ -132,6 +135,27 @@ function setMailConfig(req, res, next) {
     });
 }
 
+function getMailFromValidation(req, res, next) {
+    settings.getMailFromValidation(function (error, enabled) {
+        if (error) return next(new HttpError(500, error));
+
+        next(new HttpSuccess(200, { enabled: enabled }));
+    });
+}
+
+function setMailFromValidation(req, res, next) {
+    assert.strictEqual(typeof req.body, 'object');
+
+    if (typeof req.body.enabled !== 'boolean') return next(new HttpError(400, 'enabled is required'));
+
+    settings.setMailFromValidation(req.body.enabled, function (error) {
+        if (error && error.reason === SettingsError.BAD_FIELD) return next(new HttpError(400, error.message));
+        if (error) return next(new HttpError(500, error));
+
+        next(new HttpSuccess(202));
+    });
+}
+
 function getMailRelay(req, res, next) {
     settings.getMailRelay(function (error, mail) {
         if (error) return next(new HttpError(500, error));

src/routes/test/apps-test.js

@@ -41,7 +41,7 @@ var SERVER_URL = 'http://localhost:' + config.get('port');
 
 // Test image information
 var TEST_IMAGE_REPO = 'cloudron/test';
-var TEST_IMAGE_TAG = '23.0.0';
+var TEST_IMAGE_TAG = '24.0.1';
 var TEST_IMAGE = TEST_IMAGE_REPO + ':' + TEST_IMAGE_TAG;
 // var TEST_IMAGE_ID = child_process.execSync('docker inspect --format={{.Id}} ' + TEST_IMAGE).toString('utf8').trim();
 

src/routes/test/settings-test.js

@@ -884,4 +884,36 @@ describe('Settings API', function () {
             });
         });
     });
+
+    describe('mail from validation', function () {
+        it('get mail from validation succeeds', function (done) {
+            superagent.get(SERVER_URL + '/api/v1/settings/mail_from_validation')
+                   .query({ access_token: token })
+                   .end(function (err, res) {
+                expect(res.statusCode).to.equal(200);
+                expect(res.body).to.eql({ enabled: true });
+                done();
+            });
+        });
+
+        it('cannot set without enabled field', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/settings/mail_from_validation')
+                   .query({ access_token: token })
+                   .send({ })
+                   .end(function (err, res) {
+                expect(res.statusCode).to.equal(400);
+                done();
+            });
+        });
+
+        it('can set with enabled field', function (done) {
+            superagent.post(SERVER_URL + '/api/v1/settings/mail_from_validation')
+                   .query({ access_token: token })
+                   .send({ enabled: false })
+                   .end(function (err, res) {
+                expect(res.statusCode).to.equal(202);
+                done();
+            });
+        });
+    });
 });

src/server.js

@@ -217,6 +217,8 @@ function initializeExpressSync() {
     router.post('/api/v1/settings/mail_relay',         settingsScope, routes.user.requireAdmin, routes.settings.setMailRelay);
     router.get ('/api/v1/settings/catch_all_address',  settingsScope, routes.user.requireAdmin, routes.settings.getCatchAllAddress);
     router.put ('/api/v1/settings/catch_all_address',  settingsScope, routes.user.requireAdmin, routes.settings.setCatchAllAddress);
+    router.get ('/api/v1/settings/mail_from_validation', settingsScope, routes.user.requireAdmin, routes.settings.getMailFromValidation);
+    router.post('/api/v1/settings/mail_from_validation', settingsScope, routes.user.requireAdmin, routes.settings.setMailFromValidation);
 
     // feedback
     router.post('/api/v1/feedback', usersScope, routes.cloudron.feedback);

src/settings.js

@@ -42,28 +42,40 @@ exports = module.exports = {
     getMailConfig: getMailConfig,
     setMailConfig: setMailConfig,
 
+    getMailFromValidation: getMailFromValidation,
+    setMailFromValidation: setMailFromValidation,
+
     getMailRelay: getMailRelay,
     setMailRelay: setMailRelay,
 
     setCatchAllAddress: setCatchAllAddress,
     getCatchAllAddress: getCatchAllAddress,
 
-    getDefaultSync: getDefaultSync,
+    getEmailDigest: getEmailDigest,
+    setEmailDigest: setEmailDigest,
+
     getAll: getAll,
 
-    AUTOUPDATE_PATTERN_KEY: 'autoupdate_pattern',
-    TIME_ZONE_KEY: 'time_zone',
-    CLOUDRON_NAME_KEY: 'cloudron_name',
+    // booleans. if you add an entry here, be sure to fix getAll
     DEVELOPER_MODE_KEY: 'developer_mode',
-    DNS_CONFIG_KEY: 'dns_config',
     DYNAMIC_DNS_KEY: 'dynamic_dns',
+    MAIL_FROM_VALIDATION_KEY: 'mail_from_validation',
+    EMAIL_DIGEST: 'email_digest',
+
+    // json. if you add an entry here, be sure to fix getAll
+    DNS_CONFIG_KEY: 'dns_config',
     BACKUP_CONFIG_KEY: 'backup_config',
     TLS_CONFIG_KEY: 'tls_config',
     UPDATE_CONFIG_KEY: 'update_config',
     APPSTORE_CONFIG_KEY: 'appstore_config',
     MAIL_CONFIG_KEY: 'mail_config',
     MAIL_RELAY_KEY: 'mail_relay',
-    CATCH_ALL_ADDRESS: 'catch_all_address',
+    CATCH_ALL_ADDRESS_KEY: 'catch_all_address',
+
+    // strings
+    AUTOUPDATE_PATTERN_KEY: 'autoupdate_pattern',
+    TIME_ZONE_KEY: 'time_zone',
+    CLOUDRON_NAME_KEY: 'cloudron_name',
 
     events: null
 };
@@ -110,7 +122,9 @@ var gDefaults = (function () {
     result[exports.APPSTORE_CONFIG_KEY] = {};
     result[exports.MAIL_CONFIG_KEY] = { enabled: false };
     result[exports.MAIL_RELAY_KEY] = { provider: 'cloudron-smtp' };
-    result[exports.CATCH_ALL_ADDRESS] = [ ];
+    result[exports.CATCH_ALL_ADDRESS_KEY] = [ ];
+    result[exports.MAIL_FROM_VALIDATION_KEY] = true;
+    result[exports.EMAIL_DIGEST] = true;
 
     return result;
 })();
@@ -269,8 +283,7 @@ function getDeveloperMode(callback) {
         if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.DEVELOPER_MODE_KEY]);
         if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
 
-        // settingsdb holds string values only
-        callback(null, !!enabled);
+        callback(null, !!enabled); // settingsdb holds string values only
     });
 }
 
@@ -336,8 +349,7 @@ function getDynamicDnsConfig(callback) {
         if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.DYNAMIC_DNS_KEY]);
         if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
 
-        // settingsdb holds string values only
-        callback(null, !!enabled);
+        callback(null, !!enabled); // settingsdb holds string values only
     });
 }
 
@@ -461,6 +473,56 @@ function setMailConfig(mailConfig, callback) {
     });
 }
 
+function getMailFromValidation(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    settingsdb.get(exports.MAIL_FROM_VALIDATION_KEY, function (error, enabled) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.MAIL_FROM_VALIDATION_KEY]);
+        if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
+
+        callback(null, !!enabled); // settingsdb holds string values only
+    });
+}
+
+function setMailFromValidation(enabled, callback) {
+    assert.strictEqual(typeof enabled, 'boolean');
+    assert.strictEqual(typeof callback, 'function');
+
+    settingsdb.set(exports.MAIL_FROM_VALIDATION_KEY, enabled ? 'enabled' : '', function (error) {
+        if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
+
+        exports.events.emit(exports.MAIL_FROM_VALIDATION_KEY, enabled);
+
+        platform.createMailConfig(NOOP_CALLBACK);
+
+        callback(null);
+    });
+}
+
+function getEmailDigest(callback) {
+    assert.strictEqual(typeof callback, 'function');
+
+    settingsdb.get(exports.EMAIL_DIGEST, function (error, enabled) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.EMAIL_DIGEST]);
+        if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
+
+        callback(null, !!enabled); // settingsdb holds string values only
+    });
+}
+
+function setEmailDigest(enabled, callback) {
+    assert.strictEqual(typeof enabled, 'boolean');
+    assert.strictEqual(typeof callback, 'function');
+
+    settingsdb.set(exports.EMAIL_DIGEST, enabled ? 'enabled' : '', function (error) {
+        if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
+
+        exports.events.emit(exports.EMAIL_DIGEST, enabled);
+
+        callback(null);
+    });
+}
+
 function getMailRelay(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -493,8 +555,8 @@ function setMailRelay(relay, callback) {
 function getCatchAllAddress(callback) {
     assert.strictEqual(typeof callback, 'function');
 
-    settingsdb.get(exports.CATCH_ALL_ADDRESS, function (error, value) {
-        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.CATCH_ALL_ADDRESS]);
+    settingsdb.get(exports.CATCH_ALL_ADDRESS_KEY, function (error, value) {
+        if (error && error.reason === DatabaseError.NOT_FOUND) return callback(null, gDefaults[exports.CATCH_ALL_ADDRESS_KEY]);
         if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
 
         callback(null, JSON.parse(value));
@@ -505,10 +567,10 @@ function setCatchAllAddress(address, callback) {
     assert(Array.isArray(address));
     assert.strictEqual(typeof callback, 'function');
 
-    settingsdb.set(exports.CATCH_ALL_ADDRESS, JSON.stringify(address), function (error) {
+    settingsdb.set(exports.CATCH_ALL_ADDRESS_KEY, JSON.stringify(address), function (error) {
         if (error) return callback(new SettingsError(SettingsError.INTERNAL_ERROR, error));
 
-        exports.events.emit(exports.CATCH_ALL_ADDRESS, address);
+        exports.events.emit(exports.CATCH_ALL_ADDRESS_KEY, address);
 
         platform.createMailConfig(NOOP_CALLBACK);
 
@@ -586,12 +648,6 @@ function setAppstoreConfig(appstoreConfig, callback) {
 
 }
 
-function getDefaultSync(name) {
-    assert.strictEqual(typeof name, 'string');
-
-    return gDefaults[name];
-}
-
 function getAll(callback) {
     assert.strictEqual(typeof callback, 'function');
 
@@ -604,9 +660,11 @@ function getAll(callback) {
         // convert booleans
         result[exports.DEVELOPER_MODE_KEY] = !!result[exports.DEVELOPER_MODE_KEY];
         result[exports.DYNAMIC_DNS_KEY] = !!result[exports.DYNAMIC_DNS_KEY];
+        result[exports.MAIL_FROM_VALIDATION_KEY] = !!result[exports.MAIL_FROM_VALIDATION_KEY];
 
         // convert JSON objects
-        [exports.DNS_CONFIG_KEY, exports.TLS_CONFIG_KEY, exports.BACKUP_CONFIG_KEY, exports.MAIL_CONFIG_KEY].forEach(function (key) {
+        [exports.DNS_CONFIG_KEY, exports.TLS_CONFIG_KEY, exports.BACKUP_CONFIG_KEY, exports.MAIL_CONFIG_KEY,
+         exports.UPDATE_CONFIG_KEY, exports.APPSTORE_CONFIG_KEY, exports.MAIL_RELAY_KEY, exports.CATCH_ALL_ADDRESS_KEY].forEach(function (key) {
             result[key] = typeof result[key] === 'object' ? result[key] : safe.JSON.parse(result[key]);
         });
 

src/storage/s3.js

@@ -214,7 +214,7 @@ function testConfig(apiConfig, callback) {
 
         var params = {
             Bucket: apiConfig.bucket,
-            Key: apiConfig.prefix + '/cloudron-testfile',
+            Key: path.join(apiConfig.prefix, 'cloudron-testfile'),
             Body: 'testcontent'
         };
 
@@ -224,7 +224,7 @@ function testConfig(apiConfig, callback) {
 
             var params = {
                 Bucket: apiConfig.bucket,
-                Key: apiConfig.prefix + '/cloudron-testfile'
+                Key: path.join(apiConfig.prefix, 'cloudron-testfile')
             };
 
             s3.deleteObject(params, function (error) {

src/test/apptask-test.js

@@ -30,7 +30,7 @@ var MANIFEST = {
   "contactEmail": "support@cloudron.io",
   "version": "0.1.0",
   "manifestVersion": 1,
-  "dockerImage": "cloudron/test:23.0.0",
+  "dockerImage": "cloudron/test:24.0.1",
   "healthCheckPath": "/",
   "httpPort": 7777,
   "tcpPorts": {

src/test/checkInstall

@@ -3,7 +3,7 @@
 set -eu
 
 readonly SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-readonly TEST_IMAGE="cloudron/test:23.0.0"
+readonly TEST_IMAGE="cloudron/test:24.0.1"
 
 # reset sudo timestamp to avoid wrong success
 sudo -k || sudo --reset-timestamp

src/test/database-test.js

@@ -542,7 +542,8 @@ describe('database', function () {
             altDomain: null,
             xFrameOptions: 'DENY',
             sso: true,
-            debugMode: null
+            debugMode: null,
+            robotsTxt: null
         };
         var APP_1 = {
             id: 'appid-1',
@@ -564,7 +565,8 @@ describe('database', function () {
             altDomain: null,
             xFrameOptions: 'SAMEORIGIN',
             sso: true,
-            debugMode: null
+            debugMode: null,
+            robotsTxt: null
         };
 
         it('add fails due to missing arguments', function () {

src/test/digest-test.js

@@ -0,0 +1,110 @@
+/* global it:false */
+/* global describe:false */
+/* global before:false */
+/* global after:false */
+
+'use strict';
+
+var async = require('async'),
+    config = require('../config.js'),
+    database = require('../database.js'),
+    digest = require('../digest.js'),
+    eventlog = require('../eventlog.js'),
+    expect = require('expect.js'),
+    mailer = require('../mailer.js'),
+    paths = require('../paths.js'),
+    safe = require('safetydance'),
+    settings = require('../settings.js'),
+    updatechecker = require('../updatechecker.js'),
+    user = require('../user.js');
+
+// owner
+var USER_0 = {
+    username: 'username0',
+    password: 'Username0pass?1234',
+    email: 'user0@email.com',
+    displayName: 'User 0'
+};
+
+var AUDIT_SOURCE = {
+    ip: '1.2.3.4'
+};
+
+function checkMails(number, done) {
+    // mails are enqueued async
+    setTimeout(function () {
+        expect(mailer._getMailQueue().length).to.equal(number);
+        mailer._clearMailQueue();
+        done();
+    }, 500);
+}
+
+describe('digest', function () {
+    function cleanup(done) {
+        mailer._clearMailQueue();
+        safe.fs.unlinkSync(paths.UPDATE_CHECKER_FILE);
+
+        async.series([
+            settings.uninitialize,
+            database._clear
+        ], done);
+    }
+
+    before(function (done) {
+        config._reset();
+        config.set('version', '1.0.0');
+        config.set('apiServerOrigin', 'http://localhost:4444');
+        config.set('provider', 'notcaas');
+        safe.fs.unlinkSync(paths.UPDATE_CHECKER_FILE);
+
+        async.series([
+            database.initialize,
+            database._clear,
+            settings.initialize,
+            user.createOwner.bind(null, USER_0.username, USER_0.password, USER_0.email, USER_0.displayName, AUDIT_SOURCE),
+            eventlog.add.bind(null, eventlog.ACTION_UPDATE, AUDIT_SOURCE, { boxUpdateInfo: { sourceTarballUrl: 'xx', version: '1.2.3', changelog: [ 'good stuff' ] } }),
+            mailer.start,
+            mailer._clearMailQueue
+        ], done);
+    });
+
+    after(cleanup);
+
+    describe('disabled', function () {
+        before(function (done) {
+            settings.setEmailDigest(false, done);
+        });
+
+        it('does not send mail with digest disabled', function (done) {
+            digest.maybeSend(function (error) {
+                if (error) return done(error);
+                checkMails(0, done);
+            });
+        });
+
+    });
+
+    describe('enabled', function () {
+        before(function (done) {
+            settings.setEmailDigest(true, done);
+        });
+
+        it('sends mail for box update', function (done) {
+            digest.maybeSend(function (error) {
+                if (error) return done(error);
+
+                checkMails(1, done);
+            });
+        });
+
+        it('sends mail for pending update', function (done) {
+            updatechecker._setUpdateInfo({ box: null, apps: { 'appid': { manifest: { version: '1.2.5', changelog: 'noop\nreally' } } } });
+
+            digest.maybeSend(function (error) {
+                if (error) return done(error);
+
+                checkMails(1, done);
+            });
+        });
+    });
+});

src/test/settings-test.js

@@ -182,6 +182,36 @@ describe('Settings', function () {
             });
         });
 
+        it('can set mail from validation', function (done) {
+            settings.setMailFromValidation(true, function (error) {
+                expect(error).to.be(null);
+                done();
+            });
+        });
+
+        it('can get mail from validation', function (done) {
+            settings.getMailFromValidation(function (error, enabled) {
+                expect(error).to.be(null);
+                expect(enabled).to.be(true);
+                done();
+            });
+        });
+
+        it('can enable mail digest', function (done) {
+            settings.setEmailDigest(true, function (error) {
+                expect(error).to.be(null);
+                done();
+            });
+        });
+
+        it('can get mail digest', function (done) {
+            settings.getEmailDigest(function (error, enabled) {
+                expect(error).to.be(null);
+                expect(enabled).to.be(true);
+                done();
+            });
+        });
+
         it('can get mail relay', function (done) {
             settings.getMailRelay(function (error, address) {
                 expect(error).to.be(null);

src/updatechecker.js

@@ -6,7 +6,9 @@ exports = module.exports = {
 
     getUpdateInfo: getUpdateInfo,
     resetUpdateInfo: resetUpdateInfo,
-    resetAppUpdateInfo: resetAppUpdateInfo
+    resetAppUpdateInfo: resetAppUpdateInfo,
+
+    _setUpdateInfo: setUpdateInfo
 };
 
 var apps = require('./apps.js'),
@@ -41,6 +43,11 @@ function getUpdateInfo() {
     };
 }
 
+function setUpdateInfo(info) {
+    gBoxUpdateInfo = info.box;
+    gAppUpdateInfo = info.apps;
+}
+
 function resetUpdateInfo() {
     gBoxUpdateInfo = null;
     resetAppUpdateInfo();

webadmin/src/js/client.js

@@ -336,7 +336,8 @@ angular.module('Application').service('Client', ['$http', 'md5', 'Notification',
             key: config.key,
             memoryLimit: config.memoryLimit,
             altDomain: config.altDomain || null,
-            xFrameOptions: config.xFrameOptions
+            xFrameOptions: config.xFrameOptions,
+            robotsTxt: config.robotsTxt || null
         };
 
         post('/api/v1/apps/' + id + '/configure', data).success(function (data, status) {

webadmin/src/views/apps.html

@@ -144,6 +144,13 @@
                                 <input type="text" class="form-control" id="appConfigureXFrameOptionsInput" name="xFrameOptions" placeholder="https://example.com" ng-model="appConfigure.xFrameOptions" uib-tooltip="Leave blank to not allow embedding">
                             </div>
 
+                            <br/>
+
+                            <div class="form-group">
+                                <label class="control-label">Specify robots.txt file content</label>
+                                <textarea ng-model="appConfigure.robotsTxt" placeholder="Leave empty to allow all bots to index this app." class="form-control" rows="3"></textarea>
+                            </div>
+
                             <div class="hide">
                                 <label class="control-label" for="appConfigureCertificateInput" ng-show="config.isCustomDomain">Certificate (optional)</label>
                                 <div class="has-error text-center" ng-show="appConfigure.error.cert && config.isCustomDomain">{{ appConfigure.error.cert }}</div>

webadmin/src/views/apps.js

@@ -24,6 +24,7 @@ angular.module('Application').controller('AppsController', ['$scope', '$location
         portBindings: {},
         portBindingsEnabled: {},
         portBindingsInfo: {},
+        robotsTxt: '',
         certificateFile: null,
         certificateFileName: '',
         keyFile: null,
@@ -112,6 +113,7 @@ angular.module('Application').controller('AppsController', ['$scope', '$location
         $scope.appConfigure.accessRestriction = { users: [], groups: [] };
         $scope.appConfigure.xFrameOptions = '';
         $scope.appConfigure.customAuth = false;
+        $scope.appConfigure.robotsTxt = '';
 
         $scope.appConfigureForm.$setPristine();
         $scope.appConfigureForm.$setUntouched();
@@ -204,6 +206,7 @@ angular.module('Application').controller('AppsController', ['$scope', '$location
         $scope.appConfigure.memoryLimit = app.memoryLimit || app.manifest.memoryLimit || (256 * 1024 * 1024);
         $scope.appConfigure.xFrameOptions = app.xFrameOptions.indexOf('ALLOW-FROM') === 0 ? app.xFrameOptions.split(' ')[1] : '';
         $scope.appConfigure.customAuth = !(app.manifest.addons['ldap'] || app.manifest.addons['oauth']);
+        $scope.appConfigure.robotsTxt = app.robotsTxt;
 
         // create ticks starting from manifest memory limit
         $scope.appConfigure.memoryTicks = [
@@ -256,7 +259,8 @@ angular.module('Application').controller('AppsController', ['$scope', '$location
             cert: $scope.appConfigure.certificateFile,
             key: $scope.appConfigure.keyFile,
             xFrameOptions: $scope.appConfigure.xFrameOptions ? ('ALLOW-FROM ' + $scope.appConfigure.xFrameOptions) : 'SAMEORIGIN',
-            memoryLimit: $scope.appConfigure.memoryLimit === $scope.appConfigure.memoryTicks[0] ? 0 : $scope.appConfigure.memoryLimit
+            memoryLimit: $scope.appConfigure.memoryLimit === $scope.appConfigure.memoryTicks[0] ? 0 : $scope.appConfigure.memoryLimit,
+            robotsTxt: $scope.appConfigure.robotsTxt
         };
 
         Client.configureApp($scope.appConfigure.app.id, $scope.appConfigure.password, data, function (error) {

webadmin/src/views/settings.html

@@ -320,6 +320,14 @@
             <a ng-href="{{ config.webServerOrigin + '/console.html#/userprofile?email=' + appstoreConfig.profile.email }}" target="_blank">{{ appstoreConfig.profile.email }}</a>
         </div>
     </div>
+    <div class="row">
+        <div class="col-xs-6">
+            <span class="text-muted">Cloudron ID</span>
+        </div>
+        <div class="col-xs-6 text-right">
+            <span>{{ appstoreConfig.cloudronId }}</span>
+        </div>
+    </div>
     <div class="row">
         <div class="col-xs-6">
             <span class="text-muted">Subscription</span>