Part of #123
This allows us to test the exact same usage of the api through the cli tool, not the javascript api