Girish Ramakrishnan acd00222e5 Allow per-app configuration of robots.txt ...

https://developers.google.com/search/reference/robots_txt has
the specification

Part of #344

2017-07-14 15:25:05 -05:00

..

src

Allow per-app configuration of robots.txt

2017-07-14 15:25:05 -05:00