Commit Graph

88 Commits

Author SHA1 Message Date
Shay
8bf628e5bd
Bump node version to 18 (#496)
* try win dns res

* add dns hack to application service tests

* fix unused import

* update types

* add note about why dns resolution order is necessary
2023-11-14 12:52:53 -08:00
David Teller
5b509a226a Setting up Decentralized Abuse Reports automatically by default 2023-01-10 20:46:56 +01:00
David Teller
fa5fbee229 A first implementation of report-to-moderator
Traditionally, when a user clicks "report" in a Matrix client, this goes
to the homeserver administrator, who often is the wrong person for the job.

MSC3215 introduces a mechanism to let clients cooperate with a bot to send
the report to the moderator instead. Client support has landed in Element Web
(behind a Labs flag) in in 2021. This allows Mjölnir to serve as the partner
bot.
2023-01-10 20:46:56 +01:00
David Teller
c3cb22bf36
Very basic support for OpenMetrics (aka Prometheus) (#442)
This PR:

- creates an OpenMetrics server that enables collecting performance data from this process by e.g. a Prometheus server;
- exposes as metrics the performance of http requests with MatrixBot.

Further metrics may of course be added.
2023-01-05 08:37:54 +01:00
David Teller
cff9b43207
This should hopefully fix some startup woes (#462)
Splitting PolicyListManager from Mjolnir, making it more resilient to startup errors
2022-12-21 19:32:27 +01:00
Gnuxie
433ff7eadd
A look at PolicyList.update (#454)
This started out as just a way to find out why mjolnir was syncing with lists several times for each update to a policy list.

The main changes are

- Verbosity was irrelevant to the sync command but for some reason was an option.
  Unfortunately all this did was suppress whether to tell you when it had finished, meaning it wouldn't
  when verbose logging was disabled. Historically this was probably a parameter that got passed through
  to applyServerAcl/applyUserBans, which can be horribly verbose, but they access the config directly.

- Stop emitting `'PolicyList.update'` when there are no changes.
- Include a revision ID for the `'PolicyList.update'`method and event.
- Use the revision ID in the `ProtectedRoomsSet` so that we don't unnecessarily resynchronize all rooms when the `'PolicyList.update'` event is received. Though not when the `sync` command is used. Since this is supposed to `sync` in the case when there is a state reset or otherwise or the user has changed some room settings.
- insert an await lock around the `PolicyList.update` method to avoid a race condition where a call can be started and finished within the extent of an existing call (via another task, this can happen if the server is slow with handling one request). `PolicyList.udpate` now has a helper that is synchronous to be called directly after requesting the room state. The reason for this is to enforce that no one `await`s while updating the policy list's cache of rules. Which is important because it is one of the biggest methods that I tolerate and visually checking for `await` is impossible.
- The revision ID uses a ULID, but this is unnecessary and could have just been a "dumb counter".

closes https://github.com/matrix-org/mjolnir/issues/447
2022-12-08 16:09:55 +00:00
Gnuxie
704bb660c2
Refactor how we listen for matrix events. (#446)
* Refactor Matrix event listener in Mjolnir and ManagedMjolnir.

closes https://github.com/matrix-org/mjolnir/issues/411.

Issue #411 says that we have to be careful about room.join,
but this was before we figured how to make matrix-appservice-bridge
echo events sent by its own intents.

* Remove MatrixClientListener since it isn't actually needed.

* Protect which config values can be used for ManagedMjolnirs.

* Introduce MatrixSendClient

so listeners aren't accidentally added to a MatrixClient instead
of MatrixEmitter.

* doc

* Move provisioned mjolnir config to src/config.

This just aids maintance so whenever someone goes to change the config
of the bot they will see this and update it.

* doc for matrix intent listener.
2022-12-06 17:17:40 +00:00
David Teller
2915757b7d
Very basic support for Sentry. (#398)
The Sentry package is very useful for monitoring runtime errors. With this PR,
we simply add the necessary mechanism to:

- log to sentry any uncaught error that reaches the toplevel, including startup errors.
2022-11-30 16:06:02 +01:00
David Teller
81cd91c250
Unbitrotting ruleserver tests (#418) 2022-11-15 13:06:41 +01:00
gnuxie
a970733d0b Update roomMemberTest for removal of mjolnir.protectedRooms object.
https://github.com/matrix-org/mjolnir/issues/370.
2022-10-19 15:21:51 +01:00
gnuxie
c537f2067e protected rooms config test
https://github.com/matrix-org/mjolnir/issues/370
2022-10-19 15:21:51 +01:00
Gnuxie
da084328a9
Refactor list creation with MSC3784 support. (#386)
https://github.com/matrix-org/matrix-spec-proposals/pull/3784

This was extracted from the appservice mjolnir work to reduce review burden.
2022-10-19 10:33:12 +01:00
Gnuxie
5bd23ced9b
Access Control Unit (#378)
The ACL unit allows you to combine an policy lists and conveniently test users and servers against them.
The main motivation for this work is provide access control on who can provision and continue to use mjolnir instances in the appservice component.
We include a new recommendation type org.matrix.mjolnir.allow which can be used with user and server entity types to create allow lists.
We have also replaced the destructing of policy lists in applyServerACL and applyMemberBans (in ProtectedRooms.ts) with calls to the AccessControlUnit.
Adding commands to add/remove allowed entities is not something i want to do at the moment.
2022-10-18 16:28:11 +01:00
David Teller
7b0edadd17
Update tests and instructions to intercept reports also with v3 endpoint (#388)
In both our instructions and our tests, we use the r0 endpoint to intercept abuse reports. This endpoint is deprecated and not implemented by all clients. This PR updates the instructions and tests to the new endpoint.
2022-10-18 15:48:39 +02:00
Gnuxie
77ad40e27a
Refactor protected rooms. (#371)
* Attempt to factor out protected rooms from Mjolnir.

This is useful to the appservice because it means we don't
have to wrap a Mjolnir that is designed to sync.

It's also useful if we later on want to have specific
settings per space.

It's also just a nice seperation between Mjolnir's needs while
syncing via client-server and the behaviour of syncing policy rooms.

 ### Things that have changed

- `ErrorCache` no longer a static class (phew), gets used by `ProtectedRooms`.
- `ManagementRoomOutput` class gets created to handle logging back to the management room.
- Responsibilities for syncing member bans and server ACL are handled by `ProtectedRooms`.
- Responsibilities for watched lists should be moved to `ProtectedRooms` if they haven't been.
- `EventRedactionQueue` is moved to `ProtectedRooms` since this needs to happen after
  member bans.
- ApplyServerAcls moved to `ProtectedRooms`
- ApplyMemberBans move to `ProtectedRooms`
- `logMessage` and `replaceRoomIdsWithPills` moved to `ManagementRoomOutput`.
- `resyncJoinedRooms` has been made a little more clear, though I am concerned about how often it does run because it does seem expensive.


* ProtectedRooms is not supposed to track joined rooms.

The reason is because it is supposed to represent a specific
set of rooms to protect, not do horrible logic
for working out what rooms mjolnir is supposed to protect.
2022-09-29 14:49:09 +01:00
Jess Porter
f108935d07
support compound consequences, switch WordList to consequences (#351) 2022-09-26 16:57:21 +01:00
Gnuxie
4d5447cb50
Rework the banning and unbanning of entities in PolicyLists. (#345)
* Rework the banning and unbanning of entities in PolicyLists.

1. We keep track of the event that created a list rule so that we
can remove the rule by having a way to determine the original state key for the rule.
This is because the state key of rules can be anything and should not be
relied on by Mjolnir to unban things (which it was doing).

2. The old scheme for producing a state key was causing for some entities to escape bans
https://github.com/matrix-org/mjolnir/issues/322.

We could have used a hash or something similar, but we know that
the reason for the `rule:${entity}` scheme existed was for ease of debugging
and finding rules in devtools. So instead we have followed a scheme simalar to
bridges where the first character of an mxid is replaced with an underscore.
Everything else just gets put into the state key. Since domains can't have '@'
and room ids, aliases can't either.

3. We have stopped the need for Mjolnir to wait for the next response from sync after banning,
unbanning an entity so that we can apply ACL's sooner.

* Use PolicyList's `banEntity` method to create imported rules.
2022-08-19 13:09:08 +01:00
Gnuxie
8bafa16495
Missing awaits on floating promises in tests. (#359)
If you add `"no-floating-promises": true` it's very easy
to find where this is done accidentally.
Not sure we can keep that on all the time yet though..
2022-08-18 13:09:03 +01:00
Jess Porter
10b7233e48
matrix.to urls for aliases don't need a via param (#348) 2022-08-18 09:37:30 +01:00
Gnuxie
9bcb0b7a59
Replace acceptInvitesFromGroup with acceptInvitesFromSpace. (#338)
Replace acceptInvitesFromGroup with acceptInvitesFromSpace.

https://github.com/matrix-org/mjolnir/issues/125
https://github.com/matrix-org/mjolnir/issues/99

acceptInvitesFromGroup was implemented with an experimental api
that was a precursor to spaces which was refereed to
as either communities or groups.
Support for communities/groups ended in Synapse 1.61.0
https://github.com/matrix-org/synapse/releases/tag/v1.61.0.

To test we just edit the config dynamically which changes how the join room listener functions
though idk, shouldn't we have just made a new mjolnir instance
for this test, or changed the config before the test started somehow?


Co-authored-by: jesopo <github@lolnerd.net>
2022-08-17 10:05:23 +01:00
Jess Porter
4376679b99
load config yaml manually, remove more references to static config (#347) 2022-08-16 15:51:18 +01:00
gnuxie
899a8bd784 Reduce number of rooms involved in banListTest.ts
apparently this takes over 5 minutes on the little github acitons VM.
Starting to wonder if this is really a sustainable setup.
2022-08-16 12:55:50 +01:00
gnuxie
b9284f0167 Reduce the throttle test theshold even more.
The implementation is rubbish, as it doesn't avoid the exponential backoff

Remove default rate limit testing.

It doesn't work. No there really isn't more to say about it
you're welcome to dispute it if you're going to do the work investigating. I'm not.

We used to have a test here that tested whether Mjolnir was going to carry out a redact order the default limits in a reasonable time scale.
Now I think that's never going to happen without writing a new algorithm for respecting rate limiting.
Which is not something there is time for.

https://github.com/matrix-org/synapse/pull/13018

Synapse rate limits were broken and very permitting so that's why the current hack worked so well.
Now it is not broken, so our rate limit handling is.

b850e4554c

Honestly I don't think we can expect anyone to be able to use Mjolnir under default rate limits.

well, it's not quite simple as broken, but it is broken. With the default level in synapse (which is what matrix.org uses) it is struggling to redact 15 messages within 5 minutes. that means 5 messages over the burst count. This is ofc ontop mjolnir sending reactions / responding to replies (which isn't much but... enough to mess with the rate limiter since ofc, Synapse tells requests to wait x amount of time before trying again, but that doesn't help for concurrent requests since ofc there's only 1 slot available at that future time.  This means Synapse just wacks everything with exponentially longer shit without many (or any?) events going through
it used to be fine
because rate limiting in synapse used to be a lot more liberal because it was "broken" or something, that's not me saying it's broken that's just what synapse devs say which is probably true.
if all requests went into a queue then yeah you could eliminate one problem
but that's a lot of work and i don't think we should be doing it
cos no one uses mjolnir like this anyways
2022-08-16 12:55:30 +01:00
gnuxie
d5171bd299 Activity tracker wouldn't update for recently joined/parted protected rooms. 2022-08-15 13:29:19 +01:00
Gnuxie
21aabc879a
Stop the config being global (in almost all contexts). (#334)
* Stop the config being global (in almost all contexts).

* make sure unit test has a config

* Make failing word list more visible

* Only use Healthz from index.ts

Not really sure how useful it is anyways?
2022-08-09 11:29:27 +01:00
Gnuxie
121d4cf98f
Mjolnir would apply stale ACL to rooms during batching (#331)
* banListTest would applyACL before rules appeared in `/state`.

Mjolnir will call applyServerAcls several times while a policy list is being updated, sometimes concurrently. This means a request to set a server ACL in a room which has old data can finish after a more recent recent request with the correct ACL. This means that the old ACL gets applied to the rooms (for a moment).

This is a follow up from 551065815e

* Only allow one invocation of applyServerAcls at a time as to not conflict with each other by using a promise chain.

We don't use the throttle queue because we don't want to be blocked by other background tasks.
We don't make another throttle queue because we don't want throttling and we don't want to delay the ACL application, which can happen even with throttle time of 0.
2022-08-09 10:57:38 +01:00
David Teller
829e1bd0aa
Towards opinions in PolicyLists. (#336)
Towards opinions in PolicyLists.

This changeset is part of an ongoing effort to implement "opinions"
within policy lists, as per MSC3847.

For the time being:
- we rename BanList into PolicyList;
- we cleanup a little dead code;
- we replace a few `string`s with `enum`;
- `ListRule` becomes an abstract class with two concrete subclasses `ListRuleBan` and `ListRuleOpinion`.
2022-07-26 21:47:26 +02:00
Gnuxie
b850e4554c
Remove debug leftovers from a test. (#314)
* Remove debug leftovers from a test.

This is really terrible and has meant whenever anyone has run `yarn test:integration` they have only been running this test.
💀💀💀
https://www.youtube.com/watch?v=jmX-tzSOFE0

* Set a default timeout for integration tests that is 5 minutes long.

Seriously, I don't think there is much to gain by making people guess
a reasnoble time for a test to complete in all the time, especially
with how much Synapse changes in response time and all of the machines
involved in running these tests.

* Warn when giving up on being throttled

* For some reason it takes longer for events to appear in /state

no i am not going to track down why yet.

* Rate limiting got a lot more aggresive.

https://github.com/matrix-org/synapse/pull/13018

Rate limiting in Synapse used to reset the burst count and remove
the backoff when you were spamming continuously, now it doesn't.
Ideally we'd rewrite the rate limiting logic to back off for longer
than suggested so we could get burst again, but for now
lets just unblock CI by reducing the number of events we send in these
tests.
2022-07-06 14:20:25 +01:00
Gnuxie
ac2e736e96
Report polling test temporality (#325)
So the test before sometimes sent the report *before*
the protection (that is used to check whether we have received
the report) was registered.
This of course meant that we didn't ever receive the report from
the perspectivee of the test.
This PR should now mean we always send the report after
registering the protection.
2022-07-05 16:58:29 +01:00
David Teller
6e5d520566
Fix: roomMemberTest off-by-one error (#324) 2022-07-05 15:33:53 +02:00
David Teller
cb34af02c6
Revert "Fix: roomMemberTest off-by-one error (#319)" (#323)
This reverts commit d8aac434f1.
2022-07-05 15:29:01 +02:00
David Teller
d8aac434f1
Fix: roomMemberTest off-by-one error (#319) 2022-07-05 14:38:53 +02:00
Jess Porter
ed68e02c4e
implement polling reports in synapse (#259) 2022-07-04 15:06:36 +01:00
David Teller
0eea04bd69
A background queue for kicking (#296)
A background queue for kicking
2022-06-08 11:49:43 +02:00
David Teller
74d8caa7e7
Since command: adding the ability to mute (#272) 2022-05-10 17:19:16 +02:00
Gnuxie
c8caf744c5
Apply members and server bans to the most recently active rooms first. (#274)
* Apply members and server bans to the most recently active rooms first.

https://github.com/matrix-org/mjolnir/issues/273
2022-05-03 12:36:53 +01:00
Jonathan de Jong
cc9f393ed7
fix CI (#282)
ts-mocha has also been updated to make running locally possible (else it gives a confusing error that tsconfig.json is not "json" or the likes)
2022-05-03 11:20:40 +01:00
David Teller
e05616b327
New command !mjolnir since <date or duration> <kick | ban | show> <limit> [reason] [...rooms] (#238)
A new command `since` to affect all users who have joined a protected room since a given date.
2022-03-21 10:39:15 +01:00
Jess Porter
1880287ac4
standard protection consequences (#232)
* standard protection consequences

* add integration test to make sure good users aren't banned

* the less far `event` propagates, the better

* better document consequence.ts

* improve innocent user integration test

* switch to room.event emit
2022-03-18 10:11:23 +00:00
David Teller
26ae55cd24
A command to show when users in a given room have joined (#225) 2022-03-07 11:34:25 +01:00
Marco Cirillo
97df4d5f61
Add command to elevate a user (or the bot) as room administrator (#219) 2022-03-07 10:14:06 +01:00
jesopo
65af82d46f send whole channel shutdown reason, not just the first word 2022-03-02 10:20:43 +00:00
Gnuxie
17dd0aa173
Fix the test script yarn test:manual and add it to tsconfig (#234)
* Add test:manual launch script to tsconfig

this is so we won't keep breaking it
2022-02-24 14:46:15 +00:00
Gnuxie
7353ba4919
Make limit for ACL batcher more leniant for CI (#235) 2022-02-24 14:27:53 +00:00
David Teller
82a2e63d23
A room Protection designed to measure lag in a room (#217) 2022-02-24 13:43:31 +01:00
Gnuxie
332da15d0d
Remove old rules when unbanning entities from BanLists. (#227)
* Remove old rules when unbanning entities from BanLists.

Fixes #220
2022-02-21 16:51:14 +00:00
Jess Porter
a58c7d3f1a
move LogProxy.logMessage to Mjolnir.logMessage (#194) 2022-02-15 15:44:41 +00:00
Gnuxie
e9dff8fd5a
Batch events from ban lists together during sync (#221)
* Test for batching ACL.

* Batch events from sync within BanList.

* Introduce the BanList.batch event to the BanList emitter to let Mjolnir sync after new events have been added from sync.

Fixes #203
2022-02-15 13:51:20 +00:00
Gnuxie
ff9a7db159
Make ACL safe so that Mjolnir will not ban itself. (#213) 2022-02-07 17:02:06 +00:00
jesopo
813741c42c refactor (and simplify) how protections are stored in-memory 2022-02-07 13:48:32 +00:00