manymjolnir appservice (#364)

Mjolnir can now be run as an application service,
meaning it will host multiple independent mjolnirs that can be requested by users.
If the user is on the same homeserver as the appservice is deployed on,
then they can provision a mjolnir via a widget https://github.com/matrix-org/mjolnir-widget.
Otherwise they can invite the appservice bot to a room they want to protect.
This will create them a mjolnir, a management room and a policy list.

The appservice shares the same docker image as the bot,
but is started slightly differently by specifying "appservice"
as the first argument to docker run (this s managed by `mjolnir-entrypoint.sh`. 
We could have used another Dockerfile for the appservice,
extending the existing one but we decided not to because there
would have been lots of fiddling around the entrypoint
and logistics involved around adding a tag for it via github actions.
Not to mention that this would be duplicating the image
just to run it with a different binary.

A list of followup issues can be found here https://github.com/issues?q=is%3Aopen+is%3Aissue+author%3AGnuxie+archived%3Afalse+label%3AA-Appservice.

Somewhat relevant and squashed commit messages(regrettably squashing because frankly these won't make sense in isolation): 

* draft widget backend

* add `managementRoomId` to `provisionNewMjolnir`

* remove ratelimits from appservice mjolnirs

* add /join endpoint to api backend


* tighter guard around room type in PolicyList

matrix-bot-sdk imporved the types for this

* enable esModuleInterop

* launch and use postgres in a container whilst using mx-tester


* limited access control

policy list used for access control

* Redesign initialization API of many mjolnir.

It's much harder to forget to initialize the components now that you have to in order to construct them in the first place.


* Ammend config not to clash with existing CI

this means that the appsrvice bot is now called 'mjolnir-bot' by default
which was easier than going through old code base and renaming


* Change entrypoint in Dockerfile so that we can start the appservice.

We could have used another Dockerfile for the appservice,
extending the exising one but we decided not to because there
would have been lots of fiddling around the entrypoint
and logistics involved around adding a tag for it via github actions.
Not to mention that this would be duplicating the image
just to run it with a different binary.

This solution is much simpler, backwards compatible, and conscious about the future.


Co-authored-by: gnuxie <gnuxie@element.io>

.github/workflows/mjolnir.yml

@@ -56,3 +56,25 @@ jobs:
       run: RUST_LOG=debug,hyper=info,rusttls=info mx-tester run
     - name: Cleanup
       run: mx-tester down
+  appservice-integration:
+    name: Application Service Integration tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-node@v3
+      with:
+        node-version: '16'
+    - name: Fetch and build mx-tester (cached across runs)
+      uses: baptiste0928/cargo-install@v1
+      with:
+        crate: mx-tester
+        version: "0.3.3"
+    - name: Setup image
+      run: RUST_LOG=debug,hyper=info,rusttls=info mx-tester build up
+    - name: Setup dependencies
+      run: yarn install
+    - name: Run tests
+      run: yarn test:appservice:integration
+    - name: Cleanup
+      run: mx-tester down

Dockerfile

@@ -1,15 +1,18 @@
-FROM node:16-alpine
+# We can't use alpine anymore because crypto has rust deps.
+FROM node:16-slim
 COPY . /tmp/src
 RUN cd /tmp/src \
     && yarn install \
     && yarn build \
     && mv lib/ /mjolnir/ \
     && mv node_modules / \
+    && mv mjolnir-entrypoint.sh / \
     && cd / \
     && rm -rf /tmp/*
 
 ENV NODE_ENV=production
 ENV NODE_CONFIG_DIR=/data/config
 
-CMD node /mjolnir/index.js
+CMD ["bot"]
+ENTRYPOINT ["./mjolnir-entrypoint.sh"]
 VOLUME ["/data"]

docs/appservice.md

@@ -0,0 +1,32 @@
+Mjolnir can be run as an appservice, allowing users you trust or on your homeserver to run their own Mjolnir without hosting anything themselves.
+This module is currently alpha quality and is subject to rapid changes,
+it is not recommended currently and support will be limited.
+
+# Prerequisites
+
+This guide assumes you will be using Docker and that you are able to provide a postgres database for Mjolnir to connect to in application service mode.
+
+# Setup
+
+1. Create a new Matrix room that will act as a policy list for who can use the appservice.
+   FIXME: Currently required to be aliased.
+   FIXME: Should really be created and managed by the admin room, but waiting for command refactor before doing that. 
+
+2. Decide on a spare local TCP port number to use that will listen for messages from the matrix homeserver. Take care to configure firewalls appropriately. We will call this port `$MATRIX_PORT` in the remaining instructions.
+
+3. Create a `config/config.appservice.yaml` file that can be copied from the example in `src/appservice/config/config.example.yaml`. Your config file needs to be accessible to the docker container later on. To do this you could create a directory called `mjolnir-data` so we can map it to a volume when we launch the container later on.
+
+4. Generate the appservice registration file. This will be used by both the appservice and your homeserver.
+   Here, you must specify the direct link the Matrix Homeserver can use to access the appservice, including the Matrix port it will send messages through (if this bridge runs on the same machine you can use `localhost` as the `$HOST` name):
+   
+   `docker run -rm -v /your/path/to/mjolnir-data:/data matrixdotorg/mjolnir appservice -r -u "http://$HOST:$MATRIX_PORT" -f /data/config/mjolnir-registration.yaml`
+
+5. Step 4 created an application service bot. This will be a bot iwth the mxid specified in `mjolnir-registration.yaml` under `sender_localpart`. You now need to invite it in the access control room that you have created in Step 1.
+   
+6. Start the application service `docker run -v /your/path/to/mjolnir-data/:/data/ matrixdotorg/mjolnir appservice -c /data/config/config.appservice.yaml -f /data/config/mjolnir-registration.yaml -p $MATRIX_PORT`
+
+7. Copy the `mjolnir-registration.yaml` to your matrix homeserver and refer to it in `homeserver.yaml` like so:
+```
+  app_service_config_files:
+    - "/data/mjolnir-registration.yaml"
+```

mjolnir-entrypoint.sh

@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# This is used as the entrypoint in the mjolnir Dockerfile.
+# We want to transition away form people running the image without specifying `bot` or `appservice`.
+# So if eventually cli arguments are provided for the bot version, we want this to be the opportunity to move to `bot`.
+# Therefore using arguments without specifying `bot` (or appservice) is unsupported.
+# We maintain the behaviour where if it looks like someone is providing an executable to `docker run`, then we will execute that instead.
+# This aids configuration and debugging of the image if for example node needed to be started via another method.
+case "$1" in
+  bot) shift; set -- node /mjolnir/index.js "$@";;
+  appservice) shift; set -- node /mjolnir/appservice/cli.js "$@";;
+esac
+
+exec "$@";

mx-tester.yml

@@ -2,8 +2,16 @@ name: mjolnir
 
 up:
   before:
+    - docker run --rm --network $MX_TEST_NETWORK_NAME --name mjolnir-test-postgres --domainname mjolnir-test-postgres -e POSTGRES_PASSWORD=mjolnir-test -e POSTGRES_USER=mjolnir-tester -e POSTGRES_DB=mjolnir-test-db -d -p 127.0.0.1:8083:5432 postgres
+    # Wait until postgresql is ready
+    - until psql postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db -c ""; do echo "Waiting for psql..."; sleep 1s; done
+    # Make table in postgres
+    - psql postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db -c "CREATE TABLE mjolnir (local_part VARCHAR(255), owner VARCHAR(255), management_room TEXT)"
     # Launch the reverse proxy, listening for connections *only* on the local host.
     - docker run --rm --network host --name mjolnir-test-reverse-proxy -p 127.0.0.1:8081:80 -v $MX_TEST_CWD/test/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
+    - yarn install
+    - npx ts-node src/appservice/cli.ts -r -u "http://host.docker.internal:9000"
+    - cp mjolnir-registration.yaml $MX_TEST_SYNAPSE_DIR/data/
   after:
     # Wait until Synapse is ready
     - until curl localhost:9999 --stderr /dev/null > /dev/null; do echo "Waiting for Synapse..."; sleep 1s; done
@@ -14,12 +22,13 @@ run:
 
 down:
   finally:
+    - docker stop mjolnir-test-postgres || true
     - docker stop mjolnir-test-reverse-proxy || true
 
 modules:
   - name: mjolnir
     build:
-      - cp -r synapse_antispam $MX_TEST_MODULE_DIR
+      - cp -r synapse_antispam $MX_TEST_MODULE_DIR/
     config:
       module: mjolnir.Module
       config: {}
@@ -34,6 +43,9 @@ homeserver:
   enable_registration: true
   enable_registration_without_verification: true
 
+  app_service_config_files:
+    - "/data/mjolnir-registration.yaml"
+
   # We remove rc_message so we can test rate limiting,
   # but we keep the others because of https://github.com/matrix-org/synapse/issues/11785
   # and we don't want to slow integration tests down.

package.json

@@ -14,17 +14,22 @@
     "start:dev": "yarn build && node --async-stack-traces lib/index.js",
     "test": "ts-mocha --project ./tsconfig.json test/commands/**/*.ts",
     "test:integration": "NODE_ENV=harness ts-mocha --async-stack-traces --require test/integration/fixtures.ts --timeout 300000 --project ./tsconfig.json \"test/integration/**/*Test.ts\"",
+    "test:appservice:integration": "NODE_ENV=harness ts-mocha --async-stack-traces --timeout 300000 --project ./tsconfig.json \"test/appservice/integration/**/*Test.ts\"",
     "test:manual": "NODE_ENV=harness ts-node test/integration/manualLaunchScript.ts",
     "version": "sed -i '/# version automated/s/[0-9][0-9]*\\.[0-9][0-9]*\\.[0-9][0-9]*/'$npm_package_version'/' synapse_antispam/setup.py && git add synapse_antispam/setup.py && cat synapse_antispam/setup.py"
   },
   "devDependencies": {
     "@types/crypto-js": "^4.0.2",
+    "@types/express": "^4.17.13",
     "@types/html-to-text": "^8.0.1",
     "@types/humanize-duration": "^3.27.1",
     "@types/js-yaml": "^4.0.5",
     "@types/jsdom": "^16.2.11",
     "@types/mocha": "^9.0.0",
+    "@types/nedb": "^1.8.12",
     "@types/node": "^16.7.10",
+    "@types/pg": "^8.6.5",
+    "@types/request": "^2.48.8",
     "@types/shell-quote": "1.7.1",
     "crypto-js": "^4.1.1",
     "eslint": "^7.32",
@@ -32,19 +37,21 @@
     "mocha": "^9.0.1",
     "ts-mocha": "^9.0.2",
     "tslint": "^6.1.3",
-    "typescript": "^4.3.5",
+    "typescript": "^4.8.4",
     "typescript-formatter": "^7.2"
   },
   "dependencies": {
     "await-lock": "^2.2.2",
+    "body-parser": "^1.20.1",
     "express": "^4.17",
     "html-to-text": "^8.0.0",
     "humanize-duration": "^3.27.1",
     "humanize-duration-ts": "^2.1.1",
     "js-yaml": "^4.1.0",
     "jsdom": "^16.6.0",
-    "matrix-bot-sdk": "^0.5.19",
+    "matrix-appservice-bridge": "^5.0.0",
     "parse-duration": "^1.0.2",
+    "pg": "^8.8.0",
     "shell-quote": "^1.7.3",
     "yaml": "^2.1.1"
   },

src/appservice/AccessControl.ts

@@ -0,0 +1,66 @@
+/*
+Copyright 2022 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { Bridge } from "matrix-appservice-bridge";
+import AccessControlUnit, { EntityAccess } from "../models/AccessControlUnit";
+import PolicyList from "../models/PolicyList";
+import { Permalinks } from "matrix-bot-sdk";
+
+/**
+ * Utility to manage which users have access to the application service,
+ * meaning whether a user is able to provision a mjolnir or continue to use one.
+ * Internally we use a policy list within matrix to determine who has access via the `AccessControlUnit`.
+ */
+export class AccessControl {
+
+    private constructor(
+        private readonly accessControlList: PolicyList,
+        private readonly accessControlUnit: AccessControlUnit
+        ) {
+    }
+
+    /**
+     * Construct and initialize access control for the `MjolnirAppService`.
+     * @param accessControlListId The room id of a policy list used to manage access to the appservice (who can provision & use mjolniren)
+     * @param bridge The matrix-appservice-bridge, used to get the appservice bot.
+     * @returns A new instance of `AccessControl` to be used by `MjolnirAppService`.
+     */
+    public static async setupAccessControl(
+        /** The room id for the access control list. */
+        accessControlListId: string,
+        bridge: Bridge,
+    ): Promise<AccessControl> {
+        await bridge.getBot().getClient().joinRoom(accessControlListId);
+        const accessControlList = new PolicyList(
+            accessControlListId,
+            Permalinks.forRoom(accessControlListId),
+            bridge.getBot().getClient()
+        );
+        const accessControlUnit = new AccessControlUnit([accessControlList]);
+        await accessControlList.updateList();
+        return new AccessControl(accessControlList, accessControlUnit);
+    }
+
+    public handleEvent(roomId: string, event: any) {
+        if (roomId === this.accessControlList.roomId) {
+            this.accessControlList.updateForEvent(event);
+        }
+    }
+
+    public getUserAccess(mxid: string): EntityAccess {
+        return this.accessControlUnit.getAccessForUser(mxid, "CHECK_SERVER");
+    }
+}

src/appservice/Api.ts

@@ -0,0 +1,204 @@
+import request from "request";
+import express from "express";
+import * as bodyParser from "body-parser";
+import { MjolnirManager } from "./MjolnirManager";
+import * as http from "http";
+
+/**
+ * This provides a web api that is designed to power the mjolnir widget https://github.com/matrix-org/mjolnir-widget.
+ */
+export class Api {
+    private httpdConfig: express.Express = express();
+    private httpServer?: http.Server;
+
+    constructor(
+        private homeserver: string,
+        private mjolnirManager: MjolnirManager,
+    ) {}
+
+    /**
+     * Resolves an open id access token to find a matching user that the token is valid for.
+     * @param accessToken An openID token.
+     * @returns The mxid of the user that this token belongs to or null if the token could not be authenticated.
+     */
+    private resolveAccessToken(accessToken: string): Promise<string|null> {
+        return new Promise((resolve, reject) => {
+            request({
+                url: `${this.homeserver}/_matrix/federation/v1/openid/userinfo`,
+                qs: { access_token: accessToken },
+            }, (err, homeserver_response, body) => {
+                if (err) {
+                    console.error(`Error resolving openID token from ${this.homeserver}`, err);
+                    reject(null);
+                }
+
+                let response: { sub: string};
+                try {
+                    response = JSON.parse(body);
+                } catch (e) {
+                    console.error(`Received ill formed response from ${this.homeserver} when resolving an openID token`, e);
+                    reject(null);
+                    return;
+                }
+
+                resolve(response.sub);
+            });
+        });
+    }
+
+    public async close(): Promise<void> {
+        await new Promise((resolve, reject) => {
+            if (!this.httpServer) {
+                throw new TypeError("Server was never started");
+            }
+            this.httpServer.close(error => error ? reject(error) : resolve(undefined))
+        });
+    }
+
+    public start(port: number) {
+        if (this.httpServer) {
+            throw new TypeError("server already started");
+        }
+        this.httpdConfig.use(bodyParser.json());
+
+        this.httpdConfig.get("/get", this.pathGet.bind(this));
+        this.httpdConfig.get("/list", this.pathList.bind(this));
+        this.httpdConfig.post("/create", this.pathCreate.bind(this));
+        this.httpdConfig.post("/join", this.pathJoin.bind(this));
+
+        this.httpServer = this.httpdConfig.listen(port);
+    }
+
+    /**
+     * Finds the management room for a mjolnir.
+     * @param req.body.openId An OpenID token to verify that the sender of the request owns the mjolnir described in `req.body.mxid`.
+     * @param req.body.mxid   The mxid of the mjolnir we want to find the management room for.
+     */
+    private async pathGet(req: express.Request, response: express.Response) {
+        const accessToken = req.body["openId"];
+        if (accessToken === undefined) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const userId = await this.resolveAccessToken(accessToken);
+        if (userId === null) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const mjolnirId = req.body["mxid"];
+        if (mjolnirId === undefined) {
+            response.status(400).send("invalid request");
+            return;
+        }
+
+        // TODO: getMjolnir can fail if the ownerId doesn't match the requesting userId.
+        // https://github.com/matrix-org/mjolnir/issues/408
+        const mjolnir = this.mjolnirManager.getMjolnir(mjolnirId, userId);
+        if (mjolnir === undefined) {
+            response.status(400).send("unknown mjolnir mxid");
+            return;
+        }
+
+        response.status(200).json({ managementRoom: mjolnir.managementRoomId });
+    }
+
+    /**
+     * Return the mxids of mjolnirs that this user has provisioned.
+     * @param req.body.openId An OpenID token to find the sender of the request with and find their provisioned mjolnirs.
+     */
+    private async pathList(req: express.Request, response: express.Response) {
+        const accessToken = req.body["openId"];
+        if (accessToken === undefined) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const userId = await this.resolveAccessToken(accessToken);
+        if (userId === null) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const existing = this.mjolnirManager.getOwnedMjolnirs(userId)
+        response.status(200).json(existing);
+    }
+
+    /**
+     * Creates a new mjolnir for the requesting user and protects their first room.
+     * @param req.body.roomId The room id that the request to create a mjolnir originates from.
+     * This is so that mjolnir can protect the room once the authenticity of the request has been verified.
+     * @param req.body.openId An OpenID token to find the sender of the request with.
+     */
+    private async pathCreate(req: express.Request, response: express.Response) {
+        const accessToken = req.body["openId"];
+        if (accessToken === undefined) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const roomId = req.body["roomId"];
+        if (roomId === undefined) {
+            response.status(400).send("invalid request");
+            return;
+        }
+
+        const userId = await this.resolveAccessToken(accessToken);
+        if (userId === null) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        // TODO: provisionNewMjolnir will throw if it fails...
+        // https://github.com/matrix-org/mjolnir/issues/408
+        const [mjolnirId, managementRoom] = await this.mjolnirManager.provisionNewMjolnir(userId);
+
+        response.status(200).json({ mxid: mjolnirId, roomId: managementRoom });
+    }
+
+    /**
+     * Request a mjolnir to join and protect a room.
+     * @param req.body.openId An OpenID token to find the sender of the request with and that they own the mjolnir described in `req.body.mxid`.
+     * @param req.body.mxid   The mxid of the mjolnir that should join the room.
+     * @param req.body.roomId The room that this mjolnir should join and protect.
+     */
+    private async pathJoin(req: express.Request, response: express.Response) {
+        const accessToken = req.body["openId"];
+        if (accessToken === undefined) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const userId = await this.resolveAccessToken(accessToken);
+        if (userId === null) {
+            response.status(401).send("unauthorised");
+            return;
+        }
+
+        const mjolnirId = req.body["mxid"];
+        if (mjolnirId === undefined) {
+            response.status(400).send("invalid request");
+            return;
+        }
+
+        const roomId = req.body["roomId"];
+        if (roomId === undefined) {
+            response.status(400).send("invalid request");
+            return;
+        }
+
+        // TODO: getMjolnir can fail if the ownerId doesn't match the requesting userId.
+        // https://github.com/matrix-org/mjolnir/issues/408
+        const mjolnir = this.mjolnirManager.getMjolnir(mjolnirId, userId);
+        if (mjolnir === undefined) {
+            response.status(400).send("unknown mjolnir mxid");
+            return;
+        }
+
+        await mjolnir.joinRoom(roomId);
+        await mjolnir.addProtectedRoom(roomId);
+
+        response.status(200).json({});
+    }
+}

src/appservice/AppService.ts

@@ -0,0 +1,162 @@
+/*
+Copyright 2022 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { AppServiceRegistration, Bridge, Request, WeakEvent, BridgeContext, MatrixUser } from "matrix-appservice-bridge";
+import { MjolnirManager } from ".//MjolnirManager";
+import { DataStore, PgDataStore } from ".//datastore";
+import { Api } from "./Api";
+import { IConfig } from "./config/config";
+import { AccessControl } from "./AccessControl";
+
+/**
+ * Responsible for setting up listeners and delegating functionality to a matrix-appservice-bridge `Bridge` for
+ * the entrypoint of the application.
+ */
+export class MjolnirAppService {
+
+    private readonly api: Api;
+
+    /**
+     * The constructor is private because we want to ensure intialization steps are followed,
+     * use `makeMjolnirAppService`.
+     */
+    private constructor(
+        public readonly config: IConfig,
+        public readonly bridge: Bridge,
+        private readonly mjolnirManager: MjolnirManager,
+        private readonly accessControl: AccessControl,
+        private readonly dataStore: DataStore,
+    ) {
+        this.api = new Api(config.homeserver.url, mjolnirManager);
+    }
+
+    /**
+     * Make and initialize the app service from the config, ready to be started.
+     * @param config The appservice's config, not mjolnir's, see `src/appservice/config`.
+     * @param dataStore A datastore to persist infomration about the mjolniren to.
+     * @param registrationFilePath A file path to the registration file to read the namespace and tokens from.
+     * @returns A new `MjolnirAppService`.
+     */
+    public static async makeMjolnirAppService(config: IConfig, dataStore: DataStore, registrationFilePath: string) {
+        const bridge = new Bridge({
+            homeserverUrl: config.homeserver.url,
+            domain: config.homeserver.domain,
+            registration: registrationFilePath,
+            // We lazily initialize the controller to avoid null checks
+            // It also allows us to combine constructor/initialize logic
+            // to make the code base much simpler. A small hack to pay for an overall less hacky code base.
+            controller: {
+                onUserQuery: () => {throw new Error("Mjolnir uninitialized")},
+                onEvent: () => {throw new Error("Mjolnir uninitialized")},
+            },
+            suppressEcho: false,
+        });
+        await bridge.initalise();
+        const accessControlListId = await bridge.getBot().getClient().resolveRoom(config.accessControlList);
+        const accessControl = await AccessControl.setupAccessControl(accessControlListId, bridge);
+        const mjolnirManager = await MjolnirManager.makeMjolnirManager(dataStore, bridge, accessControl);
+        const appService = new MjolnirAppService(
+            config,
+            bridge,
+            mjolnirManager,
+            accessControl,
+            dataStore
+        );
+        bridge.opts.controller = {
+            onUserQuery: appService.onUserQuery.bind(appService),
+            onEvent: appService.onEvent.bind(appService),
+        };
+        return appService;
+    }
+
+    /**
+     * Start the appservice for the end user with the appropriate settings from their config and registration file.
+     * @param port The port to make the appservice listen for transactions from the homeserver on (usually sourced from the cli).
+     * @param config The parsed configuration file.
+     * @param registrationFilePath A path to their homeserver registration file.
+     */
+     public static async run(port: number, config: IConfig, registrationFilePath: string) {
+        const dataStore = new PgDataStore(config.db.connectionString);
+        await dataStore.init();
+        const service = await MjolnirAppService.makeMjolnirAppService(config, dataStore, registrationFilePath);
+        // The call to `start` MUST happen last. As it needs the datastore, and the mjolnir manager to be initialized before it can process events from the homeserver.
+        await service.start(port);
+    }
+
+    public onUserQuery (queriedUser: MatrixUser) {
+        return {}; // auto-provision users with no additonal data
+    }
+
+    /**
+     * Handle an individual event pushed by the homeserver to us.
+     * This function is async (and anything downstream would be anyway), which does mean that events can be processed out of order.
+     * Not a huge problem for us, but is something to be aware of.
+     * @param request A matrix-appservice-bridge request encapsulating a Matrix event.
+     * @param context Additional context for the Matrix event.
+     */
+    public async onEvent(request: Request<WeakEvent>, context: BridgeContext) {
+        const mxEvent = request.getData();
+        // Provision a new mjolnir for the invitee when the appservice bot (designated by this.bridge.botUserId) is invited to a room.
+        // Acts as an alternative to the web api provided for the widget.
+        if ('m.room.member' === mxEvent.type) {
+            if ('invite' === mxEvent.content['membership'] && mxEvent.state_key === this.bridge.botUserId) {
+                await this.mjolnirManager.provisionNewMjolnir(mxEvent.sender);
+                // reject the invite to keep the room clean and make sure the invetee doesn't get confused and think this is their mjolnir.
+                this.bridge.getBot().getClient().leaveRoom(mxEvent.room_id).catch(e => {
+                    console.warn("Unable to reject an invite to a room", e)
+                });
+            }
+        }
+        this.accessControl.handleEvent(mxEvent['room_id'], mxEvent);
+        this.mjolnirManager.onEvent(request, context);
+    }
+
+    /**
+     * Start the appservice. See `run`.
+     * @param port The port that the appservice should listen on to receive transactions from the homeserver.
+     */
+    private async start(port: number) {
+        console.log("Starting MjolnirAppService, Matrix-side to listen on port %s", port);
+        this.api.start(this.config.webAPI.port);
+        await this.bridge.listen(port);
+        console.log("MjolnirAppService started successfully");
+    }
+
+    /**
+     * Stop listening to requests from both the homeserver and web api and disconnect from the datastore.
+     */
+    public async close(): Promise<void> {
+        await this.bridge.close();
+        await this.dataStore.close();
+        await this.api.close();
+    }
+
+    /**
+     * Generate a registration file for a fresh deployment of the appservice.
+     * Included to satisfy `matrix-appservice-bridge`'s `Cli` utility which allows a registration file to be registered when setting up a deployment of an appservice.
+     * @param reg Any existing paramaters to be included in the registration, to be mutated by this method.
+     * @param callback To call when the registration has been generated with the final registration.
+     */
+    public static generateRegistration(reg: AppServiceRegistration, callback: (finalRegistration: AppServiceRegistration) => void) {
+        reg.setId(AppServiceRegistration.generateToken());
+        reg.setHomeserverToken(AppServiceRegistration.generateToken());
+        reg.setAppServiceToken(AppServiceRegistration.generateToken());
+        reg.setSenderLocalpart("mjolnir-bot");
+        reg.addRegexPattern("users", "@mjolnir_.*", true);
+        reg.setRateLimited(false);
+        callback(reg);
+    }
+}

src/appservice/MjolnirManager.ts

@@ -0,0 +1,225 @@
+import { Mjolnir } from "../Mjolnir";
+import { Request, WeakEvent, BridgeContext, Bridge, Intent } from "matrix-appservice-bridge";
+import { IConfig, read as configRead } from "../config";
+import PolicyList from "../models/PolicyList";
+import { Permalinks, MatrixClient } from "matrix-bot-sdk";
+import { DataStore } from "./datastore";
+import { AccessControl } from "./AccessControl";
+import { Access } from "../models/AccessControlUnit";
+import { randomUUID } from "crypto";
+
+/**
+ * The MjolnirManager is responsible for:
+ * * Provisioning new mjolnir instances.
+ * * Starting mjolnirs when the appservice is brought online.
+ * * Informing mjolnirs about new events.
+ */
+export class MjolnirManager {
+    private readonly mjolnirs: Map</*the user id of the mjolnir*/string, ManagedMjolnir> = new Map();
+
+    private constructor(
+        private readonly dataStore: DataStore,
+        private readonly bridge: Bridge,
+        private readonly accessControl: AccessControl
+    ) {
+
+    }
+
+    /**
+     * Create the mjolnir manager from the datastore and the access control.
+     * @param dataStore The data store interface that has the details for provisioned mjolnirs.
+     * @param bridge The bridge abstraction that encapsulates details about the appservice.
+     * @param accessControl Who has access to the bridge.
+     * @returns A new mjolnir manager.
+     */
+    public static async makeMjolnirManager(dataStore: DataStore, bridge: Bridge, accessControl: AccessControl): Promise<MjolnirManager> {
+        const mjolnirManager = new MjolnirManager(dataStore, bridge, accessControl);
+        await mjolnirManager.createMjolnirsFromDataStore();
+        return mjolnirManager;
+    }
+
+    /**
+     * Gets the default config to give the newly provisioned mjolnirs.
+     * @param managementRoomId A room that has been created to serve as the mjolnir's management room for the owner.
+     * @returns A config that can be directly used by the new mjolnir.
+     */
+    private getDefaultMjolnirConfig(managementRoomId: string): IConfig {
+        let config = configRead();
+        config.managementRoom = managementRoomId;
+        config.protectedRooms = [];
+        return config;
+    }
+
+    /**
+     * Creates a new mjolnir for a user.
+     * @param requestingUserId The user that is requesting this mjolnir and who will own it.
+     * @param managementRoomId An existing matrix room to act as the management room.
+     * @param client A client for the appservice virtual user that the new mjolnir should use.
+     * @returns A new managed mjolnir.
+     */
+    public async makeInstance(requestingUserId: string, managementRoomId: string, client: MatrixClient): Promise<ManagedMjolnir> {
+        const managedMjolnir = new ManagedMjolnir(
+            requestingUserId,
+            await Mjolnir.setupMjolnirFromConfig(client, this.getDefaultMjolnirConfig(managementRoomId))
+        );
+        this.mjolnirs.set(await client.getUserId(), managedMjolnir);
+        return managedMjolnir;
+    }
+
+    /**
+     * Gets a mjolnir for the corresponding mxid that is owned by a specific user.
+     * @param mjolnirId The mxid of the mjolnir we are trying to get.
+     * @param ownerId The owner of the mjolnir. We ask for it explicitly to not leak access to another user's mjolnir.
+     * @returns The matching managed mjolnir instance.
+     */
+    public getMjolnir(mjolnirId: string, ownerId: string): ManagedMjolnir|undefined {
+        const mjolnir = this.mjolnirs.get(mjolnirId);
+        if (mjolnir) {
+            if (mjolnir.ownerId !== ownerId) {
+                throw new Error(`${mjolnirId} is owned by a different user to ${ownerId}`);
+            } else {
+                return mjolnir;
+            }
+        } else {
+            return undefined;
+        }
+    }
+
+    /**
+     * Find all of the mjolnirs that are owned by this specific user.
+     * @param ownerId An owner of multiple mjolnirs.
+     * @returns Any mjolnirs that they own.
+     */
+    public getOwnedMjolnirs(ownerId: string): ManagedMjolnir[] {
+        // TODO we need to use the database for this but also provide the utility
+        // for going from a MjolnirRecord to a ManagedMjolnir.
+        // https://github.com/matrix-org/mjolnir/issues/409
+        return [...this.mjolnirs.values()].filter(mjolnir => mjolnir.ownerId !== ownerId);
+    }
+
+    /**
+     * Listener that should be setup and called by `MjolnirAppService` while listening to the bridge abstraction provided by matrix-appservice-bridge.
+     */
+    public onEvent(request: Request<WeakEvent>, context: BridgeContext) {
+        // TODO We need a way to map a room id (that the event is from) to a set of managed mjolnirs that should be informed.
+        // https://github.com/matrix-org/mjolnir/issues/412
+        [...this.mjolnirs.values()].forEach((mj: ManagedMjolnir) => mj.onEvent(request));
+    }
+
+    /**
+     * provision a new mjolnir for a matrix user.
+     * @param requestingUserId The mxid of the user we are creating a mjolnir for.
+     * @returns The matrix id of the new mjolnir and its management room.
+     */
+    public async provisionNewMjolnir(requestingUserId: string): Promise<[string, string]> {
+        const access = this.accessControl.getUserAccess(requestingUserId);
+        if (access.outcome !== Access.Allowed) {
+            throw new Error(`${requestingUserId} tried to provision a mjolnir when they do not have access ${access.outcome} ${access.rule?.reason ?? 'no reason specified'}`);
+        }
+        const provisionedMjolnirs = await this.dataStore.lookupByOwner(requestingUserId);
+        if (provisionedMjolnirs.length === 0) {
+            const mjolnirLocalPart = `mjolnir_${randomUUID()}`;
+            const mjIntent = await this.makeMatrixIntent(mjolnirLocalPart);
+
+            const managementRoomId = await mjIntent.matrixClient.createRoom({
+                preset: 'private_chat',
+                invite: [requestingUserId],
+                name: `${requestingUserId}'s mjolnir`
+            });
+
+            const mjolnir = await this.makeInstance(requestingUserId, managementRoomId, mjIntent.matrixClient);
+            await mjolnir.createFirstList(requestingUserId, "list");
+
+            await this.dataStore.store({
+                local_part: mjolnirLocalPart,
+                owner: requestingUserId,
+                management_room: managementRoomId,
+            });
+
+            return [mjIntent.userId, managementRoomId];
+        } else {
+            throw new Error(`User: ${requestingUserId} has already provisioned ${provisionedMjolnirs.length} mjolnirs.`);
+        }
+    }
+
+    /**
+     * Utility that creates a matrix client for a virtual user on our homeserver with the specified loclapart.
+     * @param localPart The localpart of the virtual user we need a client for.
+     * @returns A bridge intent with the complete mxid of the virtual user and a MatrixClient.
+     */
+    private async makeMatrixIntent(localPart: string): Promise<Intent> {
+        const mjIntent = this.bridge.getIntentFromLocalpart(localPart);
+        await mjIntent.ensureRegistered();
+        return mjIntent;
+    }
+
+    // TODO: We need to check that an owner still has access to the appservice each time they send a command to the mjolnir or use the web api.
+    // https://github.com/matrix-org/mjolnir/issues/410
+    /**
+     * Used at startup to create all the ManagedMjolnir instances and start them so that they will respond to users.
+     */
+    private async createMjolnirsFromDataStore() {
+        for (const mjolnirRecord of await this.dataStore.list()) {
+            const mjIntent = await this.makeMatrixIntent(mjolnirRecord.local_part);
+            const access = this.accessControl.getUserAccess(mjolnirRecord.owner);
+            if (access.outcome !== Access.Allowed) {
+                // Don't await, we don't want to clobber initialization just because we can't tell someone they're no longer allowed.
+                mjIntent.matrixClient.sendNotice(mjolnirRecord.management_room, `Your mjolnir has been disabled by the administrator: ${access.rule?.reason ?? "no reason supplied"}`);
+            } else {
+                await this.makeInstance(
+                    mjolnirRecord.owner,
+                    mjolnirRecord.management_room,
+                    mjIntent.matrixClient,
+                );
+            }
+        }
+    }
+}
+
+export class ManagedMjolnir {
+    public constructor(
+        public readonly ownerId: string,
+        private readonly mjolnir: Mjolnir,
+    ) { }
+
+    public async onEvent(request: Request<WeakEvent>) {
+        // Emulate the client syncing.
+        // https://github.com/matrix-org/mjolnir/issues/411
+        const mxEvent = request.getData();
+        if (mxEvent['type'] !== undefined) {
+            this.mjolnir.client.emit('room.event', mxEvent.room_id, mxEvent);
+            if (mxEvent.type === 'm.room.message') {
+                this.mjolnir.client.emit('room.message', mxEvent.room_id, mxEvent);
+            }
+            // TODO: We need to figure out how to inform the mjolnir of `room.join`.
+            // https://github.com/matrix-org/mjolnir/issues/411
+        }
+        if (mxEvent['type'] === 'm.room.member') {
+            if (mxEvent['content']['membership'] === 'invite' && mxEvent.state_key === await this.mjolnir.client.getUserId()) {
+                this.mjolnir.client.emit('room.invite', mxEvent.room_id, mxEvent);
+            }
+        }
+    }
+
+    public async joinRoom(roomId: string) {
+        await this.mjolnir.client.joinRoom(roomId);
+    }
+    public async addProtectedRoom(roomId: string) {
+        await this.mjolnir.addProtectedRoom(roomId);
+    }
+
+    public async createFirstList(mjolnirOwnerId: string, shortcode: string) {
+        const listRoomId = await PolicyList.createList(
+            this.mjolnir.client,
+            shortcode,
+            [mjolnirOwnerId],
+            { name: `${mjolnirOwnerId}'s policy room` }
+        );
+        const roomRef = Permalinks.forRoom(listRoomId);
+        return await this.mjolnir.watchList(roomRef);
+    }
+
+    public get managementRoomId(): string {
+        return this.mjolnir.managementRoomId;
+    }
+}

src/appservice/cli.ts

@@ -0,0 +1,27 @@
+import { Cli } from "matrix-appservice-bridge";
+import { MjolnirAppService } from "./AppService";
+import { IConfig } from "./config/config";
+
+/**
+ * This file provides the entrypoint for the appservice mode for mjolnir.
+ * A registration file can be generated `ts-node src/appservice/cli.ts -r -u "http://host.docker.internal:9000"`
+ * and the appservice can be started with `ts-node src/appservice/cli -p 9000 -c your-confg.yaml`.
+ */
+const cli = new Cli({
+    registrationPath: "mjolnir-registration.yaml",
+    bridgeConfig: {
+        schema: {},
+        affectsRegistration: false,
+        defaults: {}
+    },
+    generateRegistration: MjolnirAppService.generateRegistration,
+    run: async function(port: number) {
+        const config: IConfig | null = cli.getConfig() as any;
+        if (config === null) {
+            throw new Error("Couldn't load config");
+        }
+        await MjolnirAppService.run(port, config, cli.getRegistrationFilePath());
+    }
+});
+
+cli.run();

src/appservice/config/config.example.yaml

@@ -0,0 +1,19 @@
+
+homeserver:
+  # The Matrix server name, this will be the name of the server in your matrix id.
+  domain: "localhost:9999"
+  # The url for the appservice to call the client server API from.
+  url: http://localhost:8081
+
+# Database configuration for storing which Mjolnirs have been provisioned.
+db:
+  engine: "postgres"
+  connectionString: "postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db"
+
+# A room you have created that scopes who can access the appservice.
+# See docs/access_control.md
+accessControlList: "#access-control-list:localhost:9999"
+
+# This is a web api that the widget connects to in order to interact with the appservice.
+webAPI:
+  port: 9001

src/appservice/config/config.harness.yaml

@@ -0,0 +1,13 @@
+
+homeserver:
+  domain: "localhost:9999"
+  url: http://localhost:8081
+
+db:
+  engine: "postgres"
+  connectionString: "postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db"
+
+accessControlList: "#access-control-list:localhost:9999"
+
+webAPI:
+  port: 9001

src/appservice/config/config.ts

@@ -0,0 +1,46 @@
+/*
+Copyright 2022 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import * as fs from "fs";
+import { load } from "js-yaml";
+
+export interface IConfig {
+    /** Details for the homeserver the appservice will be serving */
+    homeserver: {
+        /** The domain of the homeserver that is found at the end of mxids */
+        domain: string,
+        /** The url to use to acccess the client server api e.g. "https://matrix-client.matrix.org" */
+        url: string
+    },
+    /** Details for the database backend */
+    db: {
+        /** Postgres connection string  */
+        connectionString: string
+    },
+    /** Config for the web api used to access the appservice via the widget */
+    webAPI: {
+        port: number
+    },
+    /** A policy room for controlling access to the appservice */
+    accessControlList: string,
+}
+
+export function read(configPath: string): IConfig {
+    const content = fs.readFileSync(configPath, "utf8");
+    const parsed = load(content);
+    const config = (parsed as object) as IConfig;
+    return config;
+}

src/appservice/datastore.ts

@@ -0,0 +1,108 @@
+/*
+Copyright 2022 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+import { Client } from "pg";
+
+export interface MjolnirRecord {
+    local_part: string,
+    owner: string,
+    management_room: string,
+}
+
+/**
+ * Used to persist mjolnirs that have been provisioned by the mjolnir manager.
+ */
+export interface DataStore {
+    /**
+     * Initialize any resources that the datastore needs to function.
+     */
+    init(): Promise<void>;
+
+    /**
+     * Close any resources that the datastore is using.
+     */
+    close(): Promise<void>;
+
+    /**
+     * List all of the mjolnirs we have provisioned.
+     */
+    list(): Promise<MjolnirRecord[]>;
+
+    /**
+     * Persist a new `MjolnirRecord`.
+     */
+    store(mjolnirRecord: MjolnirRecord): Promise<void>;
+
+    /**
+     * @param owner The mxid of the user who provisioned this mjolnir.
+     */
+    lookupByOwner(owner: string): Promise<MjolnirRecord[]>;
+
+    /**
+     * @param localPart the mxid of the provisioned mjolnir.
+     */
+    lookupByLocalPart(localPart: string): Promise<MjolnirRecord[]>;
+}
+
+export class PgDataStore implements DataStore {
+    private pgClient: Client;
+
+    constructor(connectionString: string) {
+        this.pgClient = new Client({ connectionString: connectionString });
+    }
+
+    public async init(): Promise<void> {
+        await this.pgClient.connect();
+    }
+
+    public async close(): Promise<void> {
+        await this.pgClient.end()
+    }
+
+    public async list(): Promise<MjolnirRecord[]> {
+        const result = await this.pgClient.query<MjolnirRecord>("SELECT local_part, owner, management_room FROM mjolnir");
+
+        if (!result.rowCount) {
+            return [];
+        }
+
+        return result.rows;
+    }
+
+    public async store(mjolnirRecord: MjolnirRecord): Promise<void> {
+        await this.pgClient.query(
+            "INSERT INTO mjolnir (local_part, owner, management_room) VALUES ($1, $2, $3)",
+            [mjolnirRecord.local_part, mjolnirRecord.owner, mjolnirRecord.management_room],
+        );
+    }
+
+    public async lookupByOwner(owner: string): Promise<MjolnirRecord[]> {
+        const result = await this.pgClient.query<MjolnirRecord>(
+            "SELECT local_part, owner, management_room FROM mjolnir WHERE owner = $1",
+            [owner],
+        );
+
+        return result.rows;
+    }
+
+    public async lookupByLocalPart(localPart: string): Promise<MjolnirRecord[]> {
+        const result = await this.pgClient.query<MjolnirRecord>(
+            "SELECT local_part, owner, management_room FROM mjolnir WHERE local_part = $1",
+            [localPart],
+        );
+
+        return result.rows;
+    }
+}

src/models/PolicyList.ts

@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-import { extractRequestError, LogService, MatrixClient, UserID } from "matrix-bot-sdk";
+import { extractRequestError, LogService, MatrixClient, RoomCreateOptions, UserID } from "matrix-bot-sdk";
 import { EventEmitter } from "events";
 import { ALL_RULE_TYPES, EntityType, ListRule, Recommendation, ROOM_RULE_TYPES, RULE_ROOM, RULE_SERVER, RULE_USER, SERVER_RULE_TYPES, USER_RULE_TYPES } from "./ListRule";
 
@@ -121,7 +121,7 @@ class PolicyList extends EventEmitter {
         client: MatrixClient,
         shortcode: string,
         invite: string[],
-        createRoomOptions = {}
+        createRoomOptions: RoomCreateOptions = {}
     ): Promise<string /* room id */> {
         const powerLevels: { [key: string]: any } = {
             "ban": 50,
@@ -143,7 +143,7 @@ class PolicyList extends EventEmitter {
             },
             "users_default": 0,
         };
-        const finalRoomCreateOptions = {
+        const finalRoomCreateOptions: RoomCreateOptions = {
             // Support for MSC3784.
             creation_content: {
                 type: PolicyList.ROOM_TYPE
@@ -161,7 +161,8 @@ class PolicyList extends EventEmitter {
             ...createRoomOptions
         };
         // Guard room type in case someone overwrites it when declaring custom creation_content in future code.
-        if (!PolicyList.ROOM_TYPE_VARIANTS.includes(finalRoomCreateOptions.creation_content.type)) {
+        const roomType = finalRoomCreateOptions.creation_content?.type;
+        if (typeof roomType !== 'string' || !PolicyList.ROOM_TYPE_VARIANTS.includes(roomType)) {
             throw new TypeError(`Creating a policy room with a type other than the policy room type is not supported, you probably don't want to do this.`);
         }
         const listRoomId = await client.createRoom(finalRoomCreateOptions);

src/webapis/WebAPIs.ts

@@ -15,7 +15,7 @@ limitations under the License.
 */
 
 import { Server } from "http";
-import * as express from "express";
+import express from "express";
 import { LogService, MatrixClient } from "matrix-bot-sdk";
 import RuleServer from "../models/RuleServer";
 import { ReportManager } from "../report/ReportManager";

test/appservice/integration/provisionTest.ts

@@ -0,0 +1,48 @@
+import { isPolicyRoom, readTestConfig, setupHarness } from "../utils/harness";
+import { newTestUser } from "../../integration/clientHelper";
+import { getFirstReply } from "../../integration/commands/commandUtils";
+import { MatrixClient } from "matrix-bot-sdk";
+import { MjolnirAppService } from "../../../src/appservice/AppService";
+
+interface Context extends Mocha.Context {
+    moderator?: MatrixClient,
+    appservice?:  MjolnirAppService
+}
+
+describe("Test that the app service can provision a mjolnir on invite of the appservice bot", function () {
+    afterEach(function(this: Context) {
+        this.moderator?.stop();
+        if (this.appservice) {
+            return this.appservice.close();
+        } else {
+            console.warn("Missing Appservice in this context, so cannot stop it.")
+        }
+    });
+    it("A moderator that requests a mjolnir via a matrix invitation will be invited to a new policy and management room", async function (this: Context) {
+        const config = readTestConfig();
+        this.appservice = await setupHarness();
+        // create a user to act as the moderator
+        const moderator = await newTestUser(config.homeserver.url, { name: { contains: "test" } });
+        const roomWeWantProtecting = await moderator.createRoom();
+        // have the moderator invite the appservice bot in order to request a new mjolnir
+        this.moderator = moderator;
+        const roomsInvitedTo: string[] = [];
+        await new Promise(async resolve => {
+            moderator.on('room.invite', (roomId: string) => {
+                roomsInvitedTo.push(roomId)
+                // the appservice should invite the moderator to a policy room and a management room.
+                if (roomsInvitedTo.length === 2) {
+                    resolve(null);
+                }
+            });
+            await moderator.start();
+            await moderator.inviteUser(this.appservice!.bridge.getBot().getUserId(), roomWeWantProtecting);
+        });
+        await Promise.all(roomsInvitedTo.map(roomId => moderator.joinRoom(roomId)));
+        const managementRoomId = roomsInvitedTo.filter(async roomId => !(await isPolicyRoom(moderator, roomId)))[0];
+        // Check that the newly provisioned mjolnir is actually responsive.
+        await getFirstReply(moderator, managementRoomId, () => {
+            return moderator.sendMessage(managementRoomId, { body: `!mjolnir status`, msgtype: 'm.text' });
+        })
+    })
+})

test/appservice/integration/webAPITest.ts

@@ -0,0 +1,56 @@
+import { MjolnirAppService } from "../../../src/appservice/AppService";
+import { newTestUser } from "../../integration/clientHelper";
+import { isPolicyRoom, readTestConfig, setupHarness } from "../utils/harness";
+import { CreateMjolnirResponse, MjolnirWebAPIClient } from "../utils/webAPIClient";
+import { MatrixClient } from "matrix-bot-sdk";
+import { getFirstReply } from "../../integration/commands/commandUtils";
+import expect from "expect";
+
+
+interface Context extends Mocha.Context {
+    appservice?: MjolnirAppService
+    moderator?: MatrixClient
+}
+
+
+describe("Test that the app service can provision a mjolnir when requested from the web API", function () {
+    afterEach(function(this: Context) {
+        this.moderator?.stop();
+        if (this.appservice) {
+            return this.appservice.close();
+        } else {
+            console.warn("Missing Appservice in this context, so cannot stop it.")
+        }
+    });
+    it("A moderator that requests a mjolnir via a matrix invitation will be invited to a new policy and management room", async function (this: Context) {
+        const config = readTestConfig();
+        this.appservice = await setupHarness();
+        // create a moderator
+        const moderator = await newTestUser(config.homeserver.url, { name: { contains: "test" } });
+        const apiClient = await MjolnirWebAPIClient.makeClient(moderator, "http://localhost:9001");
+        const roomToProtectId = await moderator.createRoom({ preset: "public_chat" });
+
+        // have the moderator invite the appservice bot in order to request a new mjolnir
+        this.moderator = moderator;
+        const roomsInvitedTo: string[] = [];
+        const mjolnirDetails: CreateMjolnirResponse = await new Promise(async resolve => {
+            const mjolnirDetailsPromise = apiClient.createMjolnir(roomToProtectId);
+            moderator.on('room.invite', (roomId: string) => {
+                roomsInvitedTo.push(roomId)
+                // the appservice should invite it to a policy room and a management room.
+                if (roomsInvitedTo.length === 2) {
+                    mjolnirDetailsPromise.then(resolve);
+                }
+            });
+            await moderator.start();
+        });
+        await Promise.all(roomsInvitedTo.map(roomId => moderator.joinRoom(roomId)));
+        const managementRoomId = roomsInvitedTo.filter(async roomId => !(await isPolicyRoom(moderator, roomId)))[0];
+        expect(managementRoomId).toBe(mjolnirDetails.managementRoomId);
+        // Check that the newly provisioned mjolnir is actually responsive.
+        const event = await getFirstReply(moderator, managementRoomId, () => {
+            return moderator.sendMessage(managementRoomId, { body: `!mjolnir status`, msgtype: 'm.text' });
+        })
+        expect(event.sender).toBe(mjolnirDetails.mjolnirUserId);
+    })
+})

test/appservice/utils/harness.ts

@@ -0,0 +1,29 @@
+import path from "path";
+import { MjolnirAppService } from "../../../src/appservice/AppService";
+import { ensureAliasedRoomExists } from "../../integration/mjolnirSetupUtils";
+import { read as configRead, IConfig } from "../../../src/appservice/config/config";
+import { PgDataStore } from "../../../src/appservice/datastore";
+import { newTestUser } from "../../integration/clientHelper";
+import PolicyList from "../../../src/models/PolicyList";
+import { CreateEvent, MatrixClient } from "matrix-bot-sdk";
+
+export function readTestConfig(): IConfig {
+    return configRead(path.join(__dirname, "../../../src/appservice/config/config.harness.yaml"));
+}
+
+export async function setupHarness(): Promise<MjolnirAppService> {
+    const config = readTestConfig();
+    const utilityUser = await newTestUser(config.homeserver.url, { name: { contains: "utility" }});
+    await ensureAliasedRoomExists(utilityUser, config.accessControlList);
+    const dataStore = new PgDataStore(config.db.connectionString);
+    await dataStore.init();
+    const appservice = await MjolnirAppService.makeMjolnirAppService(config, dataStore, "mjolnir-registration.yaml");
+    await appservice.start(9000);
+    return appservice;
+}
+
+export async function isPolicyRoom(user: MatrixClient, roomId: string): Promise<boolean> {
+    const createEvent = new CreateEvent(await user.getRoomStateEvent(roomId, "m.room.create", ""));
+    return PolicyList.ROOM_TYPE_VARIANTS.includes(createEvent.type);
+}
+

test/appservice/utils/webAPIClient.ts

@@ -0,0 +1,50 @@
+import * as request from "request";
+import { MatrixClient } from "matrix-bot-sdk";
+
+interface OpenIDTokenInfo {
+    access_token: string,
+    expires_in: number,
+    matrix_server_name: string,
+    token_type: string
+}
+
+async function getOpenIDToken(client: MatrixClient): Promise<string> {
+    const tokenInfo: OpenIDTokenInfo = await client.doRequest("POST", `/_matrix/client/v3/user/${await client.getUserId()}/openid/request_token`, undefined, {});
+    return tokenInfo.access_token;
+}
+
+export interface CreateMjolnirResponse {
+    mjolnirUserId: string,
+    managementRoomId: string,
+}
+
+export class MjolnirWebAPIClient {
+
+    private constructor(
+        private readonly matrixClient: MatrixClient,
+        private readonly openIDToken: string,
+        private readonly baseURL: string,
+    ) {
+
+    }
+
+    public static async makeClient(client: MatrixClient, baseUrl: string): Promise<MjolnirWebAPIClient> {
+        const token = await getOpenIDToken(client);
+        return new MjolnirWebAPIClient(client, token, baseUrl);
+    }
+
+    public async createMjolnir(roomToProtectId: string): Promise<CreateMjolnirResponse> {
+        const body: { mxid: string, roomId: string } = await new Promise((resolve, reject) => {
+            request.post(`${this.baseURL}/create`, {
+                json: {
+                    openId: this.openIDToken,
+                    roomId: roomToProtectId,
+                },
+            }, (error, response) => error ? reject(error) : resolve(response.body))
+        });
+        return {
+            mjolnirUserId: body.mxid,
+            managementRoomId: body.roomId
+        }
+    }
+}

test/commands/UnbanBanCommandTest.ts

@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-import * as expect from "expect";
+import expect from "expect";
 import { Mjolnir } from "../../src/Mjolnir";
 import { DEFAULT_LIST_EVENT_TYPE } from "../../src/commands/SetDefaultBanListCommand";
 import { parseArguments } from "../../src/commands/UnbanBanCommand";

tsconfig.json

@@ -1,6 +1,7 @@
 {
     "compilerOptions": {
         "alwaysStrict": true,
+        "esModuleInterop": true,
         "experimentalDecorators": true,
         "emitDecoratorMetadata": true,
         "module": "commonjs",
@@ -20,6 +21,7 @@
     },
     "include": [
         "./src/**/*",
+	    "./test/appservice/*",
         "./test/integration/manualLaunchScript.ts",
         "./test/integration/roomMembersTest.ts",
         "./test/integration/banListTest.ts",