Rather than enforcing a 200 status code, the instance query is deemed a
success if the status code is <400. Various services return 200-399
status codes that don't necessarily indicate an error, but may have to
do with how the instance was configured.
The FARSIDE_NO_ROUTER variable wasn't terribly useful after refactoring
the app to include the update routine internally (rather than available
externally as an elixir script).
Now the only supported environment variable is FARSIDE_TEST, which is
still useful for tests and quick validation of functionality.
Although the current list of queries is only 2 elements ("time" and
"weather"), this allows searches for Whoogle and Searx to be slightly
more random, and potentially avoid any issues with their parent engine
rate limiting them.
* Create nightly update workflow for instances
A nightly GitHub Actions CI workflow has been added to fetch new
instances of supported services within Farside.
Currently only Searx is supported, but obviously others could be added
if there are similarly easy ways to fetch and filter instances
programmatically.
services.json has also been updated with the initial results of the
workflow script.
* Set headers for every HTTPoison request
This serves as a workaround for bot blocking via filtron.
* Expand filtering of searx instances
New filter enforces:
- No Cloudflare
- Good TLS config
- Good HTTP header config
- Vanilla instances or forks
- Instances with 100% search success
Rather than requiring a traditional crontab install, the app now
leverages quantum-core (link below) to schedule the instance update/sync
task every 5 minutes. Some updates as a result:
- The new job is scheduled at runtime in server.ex.
- The update.exs script was refactored to be compiled along with the
rest of the app as instances.ex.
- Scheduler and Server modules were added for creating and executing
the new update task
- All shell scripts were removed, as they are no longer needed
https://github.com/quantum-elixir/quantum-core
Setting the aforementioned env var skips creation of the app router,
which is useful for running update.exs when the main app is already
running (otherwise there's a port conflict).
Query params ("/watch?v=dQw4w9WgXcQ" for instance) would previously be
lost in Farside redirects. This now includes them if they were included
in the original request.
This introduces a way of throttling requests in a way that makes sense
for the purpose of the app. The app only supports redirecting to one
particular service when browsing, which would seldom be required more
than once per second for normal "human" browsing.
Without this, the service could easily be used to DOS multiple instances
at once. That being said, anyone concerned about someone DOS-ing
multiple instances at once should be aware that this would be trivial to
do with a simple bash script. This is simply a preventative measure to
hopefully deter people from trying to attack all public instances of
private frontends using farside.link.
Note that this throttling applies to all routes in the app, including
the homepage. This could be updated to exclude the homepage I guess,
but I'm not really sure what the use case would be for that.
The process of selecting a random (working) instance for a specified
service has been moved out of the router and into lib/farside.ex. Moving
forward, the router itself should have very simple and easy to follow
logic for all paths.
This introduces a number of new changes:
- Services are now inserted into redis with a prefix prepended to the
key name. This allows for easier filtering to get only live instances.
- The home page now uses an eex template for displaying all live
instances for every service, determined by the last update
- A "last_updated" field was added
- farside.ex was added to contain all functionality related to querying
for instances (WIP)
- Other improvements
The name of the project is being refactored from Privacy Revolver to
Farside. The reasoning behind this is:
1. A shorter name is easier to remember
2. It can stand for "FOSS alternative redirecting service" (which I know
doesn't encapsulate all letters from "farside", but it's close enough).
This commit also includes improvements to the update script for
determining how far along the script is.
