Commit Graph

416 Commits

Author SHA1 Message Date
David Teller
c4b61e0ba3 WIP: Comments 2022-10-18 15:47:54 +02:00
David Teller
bf72563db8 Update tests and instructions to intercept reports also with v3 endpoint
In both our instructions and our tests, we use the r0 endpoint to intercept abuse reports. This endpoint is deprecated and not implemented by all clients. This PR updates the instructions and tests to the new endpoint.
2022-10-18 12:43:32 +02:00
David Teller
28265cec2e
CI: Let's try and cache mx-tester installs (#383) 2022-10-10 17:17:22 +02:00
Than Harrison
0379cc45fb
Fix small typo (#382)
* src/commands/CommandHandler.ts:
 - fix small typo that's bugging me in `!status` response.
   change "projection" -> "protection"

Signed-off-by: Than Harrison <infosecvoid@proton.me>

Signed-off-by: Than Harrison <infosecvoid@proton.me>
2022-10-10 12:35:35 +02:00
Gnuxie
77ad40e27a
Refactor protected rooms. (#371)
* Attempt to factor out protected rooms from Mjolnir.

This is useful to the appservice because it means we don't
have to wrap a Mjolnir that is designed to sync.

It's also useful if we later on want to have specific
settings per space.

It's also just a nice seperation between Mjolnir's needs while
syncing via client-server and the behaviour of syncing policy rooms.

 ### Things that have changed

- `ErrorCache` no longer a static class (phew), gets used by `ProtectedRooms`.
- `ManagementRoomOutput` class gets created to handle logging back to the management room.
- Responsibilities for syncing member bans and server ACL are handled by `ProtectedRooms`.
- Responsibilities for watched lists should be moved to `ProtectedRooms` if they haven't been.
- `EventRedactionQueue` is moved to `ProtectedRooms` since this needs to happen after
  member bans.
- ApplyServerAcls moved to `ProtectedRooms`
- ApplyMemberBans move to `ProtectedRooms`
- `logMessage` and `replaceRoomIdsWithPills` moved to `ManagementRoomOutput`.
- `resyncJoinedRooms` has been made a little more clear, though I am concerned about how often it does run because it does seem expensive.


* ProtectedRooms is not supposed to track joined rooms.

The reason is because it is supposed to represent a specific
set of rooms to protect, not do horrible logic
for working out what rooms mjolnir is supposed to protect.
2022-09-29 14:49:09 +01:00
Jess Porter
f108935d07
support compound consequences, switch WordList to consequences (#351) 2022-09-26 16:57:21 +01:00
Denis Lisov
89b7ec1a18
Add list shortcodes to status command output (#312) 2022-09-14 10:32:14 +01:00
Gnuxie
4d5447cb50
Rework the banning and unbanning of entities in PolicyLists. (#345)
* Rework the banning and unbanning of entities in PolicyLists.

1. We keep track of the event that created a list rule so that we
can remove the rule by having a way to determine the original state key for the rule.
This is because the state key of rules can be anything and should not be
relied on by Mjolnir to unban things (which it was doing).

2. The old scheme for producing a state key was causing for some entities to escape bans
https://github.com/matrix-org/mjolnir/issues/322.

We could have used a hash or something similar, but we know that
the reason for the `rule:${entity}` scheme existed was for ease of debugging
and finding rules in devtools. So instead we have followed a scheme simalar to
bridges where the first character of an mxid is replaced with an underscore.
Everything else just gets put into the state key. Since domains can't have '@'
and room ids, aliases can't either.

3. We have stopped the need for Mjolnir to wait for the next response from sync after banning,
unbanning an entity so that we can apply ACL's sooner.

* Use PolicyList's `banEntity` method to create imported rules.
2022-08-19 13:09:08 +01:00
Gnuxie
8bafa16495
Missing awaits on floating promises in tests. (#359)
If you add `"no-floating-promises": true` it's very easy
to find where this is done accidentally.
Not sure we can keep that on all the time yet though..
2022-08-18 13:09:03 +01:00
Erik Johnston
a2d919b57a
Fix example to have correct incantation (#358) 2022-08-18 11:25:07 +01:00
Jess Porter
10b7233e48
matrix.to urls for aliases don't need a via param (#348) 2022-08-18 09:37:30 +01:00
Jess Porter
625b62ce6d
use requested path to configs (#357) 2022-08-17 16:17:23 +01:00
Gnuxie
9bcb0b7a59
Replace acceptInvitesFromGroup with acceptInvitesFromSpace. (#338)
Replace acceptInvitesFromGroup with acceptInvitesFromSpace.

https://github.com/matrix-org/mjolnir/issues/125
https://github.com/matrix-org/mjolnir/issues/99

acceptInvitesFromGroup was implemented with an experimental api
that was a precursor to spaces which was refereed to
as either communities or groups.
Support for communities/groups ended in Synapse 1.61.0
https://github.com/matrix-org/synapse/releases/tag/v1.61.0.

To test we just edit the config dynamically which changes how the join room listener functions
though idk, shouldn't we have just made a new mjolnir instance
for this test, or changed the config before the test started somehow?


Co-authored-by: jesopo <github@lolnerd.net>
2022-08-17 10:05:23 +01:00
Gnuxie
f5a1a39861
audit yarn.lock (#356)
### Auditing the lock file

```
npm install --package-lock-only
npm audit fix
rm yarn.lock
yarn import
```

```
npm audit

json-schema  <0.4.0
Severity: critical
json-schema is vulnerable to Prototype Pollution - https://github.com/advisories/GHSA-896r-f27r-55mw
fix available via `npm audit fix`
node_modules/json-schema
  jsprim  0.3.0 - 1.4.1 || 2.0.0 - 2.0.1
  Depends on vulnerable versions of json-schema
  node_modules/jsprim

minimist  <1.2.6
Severity: critical
Prototype Pollution in minimist - https://github.com/advisories/GHSA-xvch-5gv4-984h
fix available via `npm audit fix`
node_modules/minimist

nanoid  3.0.0 - 3.1.30
Severity: moderate
Exposure of Sensitive Information to an Unauthorized Actor in nanoid - https://github.com/advisories/GHSA-qrpm-p2h7-hrv2
fix available via `npm audit fix`
node_modules/nanoid
node_modules/postcss/node_modules/nanoid
  mocha  8.2.0 - 9.1.4
  Depends on vulnerable versions of nanoid
  node_modules/mocha

5 vulnerabilities (2 moderate, 3 critical)

To address all issues, run:
  npm audit fix
```

### minimist

minimist@1.2.5
used by mocha, tslint and matrix-bot-sdk@0.5.19

via
```
MatrixClient::replyHtmlText
MatrixClient::replyHtmlNotice
MatrixClient::sendHtmlNotice
MatrixClient::sendHtmlTex
```

none of which we use.

### nanoid

As for nanoid this is used by mocha.
It's also used by postcss vis the bot sdk

```
├─┬ matrix-bot-sdk@0.5.19
│ └─┬ sanitize-html@2.7.1
│   └─┬ postcss@8.4.16
│     ├── nanoid@3.3.4

```
though unless i'm missing something nanoid@3.3.4 doesn't fit into the vulnerable versions  `3.0.0 - 3.1.30`


### json-schema

As for json-schema, it is used by jsprim@1.4.2 within 'validateJsonObjectJS'.
fortunately we depend on jsprim via the http-signatures@1.2.0 package which only use jsprim for rfc1123.
(which request depends upon in the matrix-bot-sdk).

```
├─┬ matrix-bot-sdk@0.5.19
│ ├─┬ request@2.88.2
│ │ ├─┬ http-signature@1.2.0
│ │ │ ├─┬ jsprim@1.4.2
│ │ │ │ ├── json-schema@0.4.0
2022-08-16 18:49:33 +01:00
Jess Porter
4376679b99
load config yaml manually, remove more references to static config (#347) 2022-08-16 15:51:18 +01:00
Gnuxie
64c26e55f4
Merge pull request #352 from matrix-org/gnuxie/room-activity
Fix Intermittent integration test failures
2022-08-16 13:21:05 +01:00
gnuxie
899a8bd784 Reduce number of rooms involved in banListTest.ts
apparently this takes over 5 minutes on the little github acitons VM.
Starting to wonder if this is really a sustainable setup.
2022-08-16 12:55:50 +01:00
gnuxie
b9284f0167 Reduce the throttle test theshold even more.
The implementation is rubbish, as it doesn't avoid the exponential backoff

Remove default rate limit testing.

It doesn't work. No there really isn't more to say about it
you're welcome to dispute it if you're going to do the work investigating. I'm not.

We used to have a test here that tested whether Mjolnir was going to carry out a redact order the default limits in a reasonable time scale.
Now I think that's never going to happen without writing a new algorithm for respecting rate limiting.
Which is not something there is time for.

https://github.com/matrix-org/synapse/pull/13018

Synapse rate limits were broken and very permitting so that's why the current hack worked so well.
Now it is not broken, so our rate limit handling is.

b850e4554c

Honestly I don't think we can expect anyone to be able to use Mjolnir under default rate limits.

well, it's not quite simple as broken, but it is broken. With the default level in synapse (which is what matrix.org uses) it is struggling to redact 15 messages within 5 minutes. that means 5 messages over the burst count. This is ofc ontop mjolnir sending reactions / responding to replies (which isn't much but... enough to mess with the rate limiter since ofc, Synapse tells requests to wait x amount of time before trying again, but that doesn't help for concurrent requests since ofc there's only 1 slot available at that future time.  This means Synapse just wacks everything with exponentially longer shit without many (or any?) events going through
it used to be fine
because rate limiting in synapse used to be a lot more liberal because it was "broken" or something, that's not me saying it's broken that's just what synapse devs say which is probably true.
if all requests went into a queue then yeah you could eliminate one problem
but that's a lot of work and i don't think we should be doing it
cos no one uses mjolnir like this anyways
2022-08-16 12:55:30 +01:00
Jess Porter
1e67eed30b
change report polling to read oldest first, so we paginate to end and wait (#353) 2022-08-15 15:58:25 +01:00
gnuxie
d5171bd299 Activity tracker wouldn't update for recently joined/parted protected rooms. 2022-08-15 13:29:19 +01:00
Gnuxie
21aabc879a
Stop the config being global (in almost all contexts). (#334)
* Stop the config being global (in almost all contexts).

* make sure unit test has a config

* Make failing word list more visible

* Only use Healthz from index.ts

Not really sure how useful it is anyways?
2022-08-09 11:29:27 +01:00
Gnuxie
121d4cf98f
Mjolnir would apply stale ACL to rooms during batching (#331)
* banListTest would applyACL before rules appeared in `/state`.

Mjolnir will call applyServerAcls several times while a policy list is being updated, sometimes concurrently. This means a request to set a server ACL in a room which has old data can finish after a more recent recent request with the correct ACL. This means that the old ACL gets applied to the rooms (for a moment).

This is a follow up from 551065815e

* Only allow one invocation of applyServerAcls at a time as to not conflict with each other by using a promise chain.

We don't use the throttle queue because we don't want to be blocked by other background tasks.
We don't make another throttle queue because we don't want throttling and we don't want to delay the ACL application, which can happen even with throttle time of 0.
2022-08-09 10:57:38 +01:00
David Teller
829e1bd0aa
Towards opinions in PolicyLists. (#336)
Towards opinions in PolicyLists.

This changeset is part of an ongoing effort to implement "opinions"
within policy lists, as per MSC3847.

For the time being:
- we rename BanList into PolicyList;
- we cleanup a little dead code;
- we replace a few `string`s with `enum`;
- `ListRule` becomes an abstract class with two concrete subclasses `ListRuleBan` and `ListRuleOpinion`.
2022-07-26 21:47:26 +02:00
Will Hunt
4aad5c455d
Update tsconfig.json touse ES2021 (#337)
Node16+ allows usage of ES2021 features, and frankly ES2015 is very old now.
2022-07-26 15:33:43 +01:00
Jess Porter
eb2c25da7b
docs/moderators.md: add section on Trusted Reporters (#332)
* docs/moderators.md: add section on Trusted Reporters

* Update docs/moderators.md

Co-authored-by: Thibault Martin <thibaultamartin@users.noreply.github.com>

* clarify that alertThreshold is not the same as displayReports

Co-authored-by: Thibault Martin <thibaultamartin@users.noreply.github.com>
2022-07-13 13:22:18 +01:00
DeepBlueV7.X
9f872f78e1
Use stable prefixes for policy recommendations (#329)
They have been in the spec for ~3 years now and most mjolnirs should be
able to handle them. Let's use the stable endpoint now, so that other
moderation tools don't need to implement the legacy identifier to handle
new bans at some point.

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2022-07-11 14:28:57 +01:00
gnuxie
b48904bc2b v1.5.0 2022-07-07 13:07:27 +01:00
Gnuxie
941cc32ddd
Add !mjolnir rules matching <entity> to search watched lists. (#307)
* Add `!mjolnir rules matching <entity> to search watched lists.

Lists all the rules that will match the entity.
2022-07-07 13:03:03 +01:00
Jess Porter
b03d81dcc4
ability to disable displaying new reports in moderation room (#320) 2022-07-07 12:55:03 +01:00
Gnuxie
84ffb36494
Bump minimum node version to 16. (#316)
* Bump minimum node version to 16.

We already made the mistake of using features from 16 without realising
Previously: https://github.com/matrix-org/mjolnir/pull/192

* Make sure CI uses specific version of node and we also lint.

https://github.com/matrix-org/pipelines/blob/master/mjolnir/pipeline.yml#L13
2022-07-06 14:50:33 +01:00
Gnuxie
65f52fef3a
Timeout integration tests CI after 1 hour. (#317) 2022-07-06 14:21:33 +01:00
Gnuxie
b850e4554c
Remove debug leftovers from a test. (#314)
* Remove debug leftovers from a test.

This is really terrible and has meant whenever anyone has run `yarn test:integration` they have only been running this test.
💀💀💀
https://www.youtube.com/watch?v=jmX-tzSOFE0

* Set a default timeout for integration tests that is 5 minutes long.

Seriously, I don't think there is much to gain by making people guess
a reasnoble time for a test to complete in all the time, especially
with how much Synapse changes in response time and all of the machines
involved in running these tests.

* Warn when giving up on being throttled

* For some reason it takes longer for events to appear in /state

no i am not going to track down why yet.

* Rate limiting got a lot more aggresive.

https://github.com/matrix-org/synapse/pull/13018

Rate limiting in Synapse used to reset the burst count and remove
the backoff when you were spamming continuously, now it doesn't.
Ideally we'd rewrite the rate limiting logic to back off for longer
than suggested so we could get burst again, but for now
lets just unblock CI by reducing the number of events we send in these
tests.
2022-07-06 14:20:25 +01:00
Gnuxie
ac2e736e96
Report polling test temporality (#325)
So the test before sometimes sent the report *before*
the protection (that is used to check whether we have received
the report) was registered.
This of course meant that we didn't ever receive the report from
the perspectivee of the test.
This PR should now mean we always send the report after
registering the protection.
2022-07-05 16:58:29 +01:00
David Teller
6e5d520566
Fix: roomMemberTest off-by-one error (#324) 2022-07-05 15:33:53 +02:00
David Teller
cb34af02c6
Revert "Fix: roomMemberTest off-by-one error (#319)" (#323)
This reverts commit d8aac434f1.
2022-07-05 15:29:01 +02:00
David Teller
d8aac434f1
Fix: roomMemberTest off-by-one error (#319) 2022-07-05 14:38:53 +02:00
Jess Porter
ed68e02c4e
implement polling reports in synapse (#259) 2022-07-04 15:06:36 +01:00
gnuxie
d7b846cdb3 Remove duplicated kick from kick command (merging cleanup)
Accidentally introduced while merging https://github.com/matrix-org/mjolnir/pull/291.
2022-06-15 12:34:59 +01:00
Jae Lo Presti
a876a05520
Glob kick command (#291)
This pull requests adds for glob support in the `!mjolnir kick` command.

## Example
```
!mjolnir kick @*:domain.tld <reason> --force
```
This command will kick every user having a mxid matching `domain.tld`.  
You can also still kick a particular user:
```
!mjolnir kick @user:domain.tld <reason>
```

## Tests:
Tested on the Furry Tech room (`vRGLvqJYlFvzpThbxI:matrix.org`) after a spam wave.  
It kicked over 13k bots in a matter of hours without putting too much strain on the homeserver.  
For instance, this command was matching `@spam*`:
![image](https://user-images.githubusercontent.com/76598503/167320002-f0575f50-4b54-41d1-8220-f67d72ccaf16.png)

  
  
Signed-off-by: Jae Lo Presti <me@jae.fi>
2022-06-15 12:20:27 +01:00
David Teller
0eea04bd69
A background queue for kicking (#296)
A background queue for kicking
2022-06-08 11:49:43 +02:00
Gnuxie
558cbb3cae
Remove the need to call /initialSync in getMessagesByUserIn. (#297)
* Remove the need to call `/initialSync` in `getMessagesByUserIn`.

At the moment we call `/initialSync` to give a `from` token to `/messages`.
In this PR we instead do not provide a `from` token when calling `/messages`,
which has recently been permitted in the spec
Technically this is still unstable in the spec
https://spec.matrix.org/unstable/client-server-api/#get_matrixclientv3roomsroomidmessages
https://github.com/matrix-org/matrix-spec/pull/1002

Synapse has supported this for over 2 years and Element web depends on it for threads.
https://github.com/matrix-org/matrix-js-sdk/pull/2065

Given that redactions are super heavy in Mjolnir already and have been reported
as barely functional on matrix.org I believe we should also adopt this approach as
if for some reason the spec did change before the next release (1.3) (extremely unlikely) we can revert this commit.
2022-05-24 11:16:52 +01:00
Jonathan de Jong
bf7f1318af
Improve documentation (#277)
* improve documentation

* incorperate documentation of configuration into sample file
2022-05-20 12:19:26 +01:00
David Teller
bcc3405e51
Revert "Fix traceback on Abuse Report. (#230)" (#293)
This reverts commit 27ae3ec625.
2022-05-13 15:29:49 +02:00
Marco Cirillo
27ae3ec625
Fix traceback on Abuse Report. (#230)
* Have the callback work on the packed event Obj.

And also explicitly pack the Mjolnir object inside the event.

* Add VS ignores.

* Commit suggested review changes.

- Add bind to callback
- Add type cast for event packed object

* Fix missing parenthesis.

* Don't pass Mjolnir obj to handleReport().

Co-authored-by: David Teller <davidt@element.io>
2022-05-13 15:18:30 +02:00
David Teller
74d8caa7e7
Since command: adding the ability to mute (#272) 2022-05-10 17:19:16 +02:00
gnuxie
a88fc64a07 v1.4.2 2022-05-06 11:25:49 +01:00
Jonathan de Jong
9fce35c000
Add Join Wave Short-Circuit Protection (#280)
* Add Short Circuit Protection

* fix module name

* change to dynamic timescales

* address feedback
2022-05-06 09:24:34 +01:00
Gnuxie
c8caf744c5
Apply members and server bans to the most recently active rooms first. (#274)
* Apply members and server bans to the most recently active rooms first.

https://github.com/matrix-org/mjolnir/issues/273
2022-05-03 12:36:53 +01:00
Jonathan de Jong
cc9f393ed7
fix CI (#282)
ts-mocha has also been updated to make running locally possible (else it gives a confusing error that tsconfig.json is not "json" or the likes)
2022-05-03 11:20:40 +01:00
Gnuxie
3d10f998b7
Change antispam Python module verison with mjolnir version. (#264)
The reason we want this is so that people do not forget to change the version number in the synapse module.
The precedent is that the version number has been 0.1.0 since the begging until now.
While this solution does mean that there may be new version of the module where
nothing has actually changed, this is still better than not changing the version at all.
Another version scheme for the module would be inconsistent to
the git repository tags and that has the potential to cause much more confusion
than "blank" version bumps.
If this is a problem, then antispam must be extracted to another repository.

In order to test this, run `yarn version --patch` observe the changes with `git log` and `git diff HEAD~`,
then YOU MUST delete the tag with `git tag --delete vd.d.d` when you are finished.

* more robust
2022-04-12 16:36:35 +01:00