Optimise the lookup for an invite from another homeserver when trying to reject it.