Git-Mirrors/farside
1
0
Fork 0
mirror of https://github.com/benbusby/farside.git synced 2025-03-13 10:46:35 -04:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Rewrite project, add daily update of services list

Browse Source 
The project was rewritten from Elixir to Go, primarily because:

- I don't write Elixir anymore and don't want to maintain a project in a
  language I no longer write
- I already write Go for other projects, including my day job, so it's
  a safer bet for a project that I want to maintain long term
- Go allows me to build portable executables that will make it easier
  for others to run farside on their own machines

The Go version of Farsside also has a built in task to fetch the latest
services{-full}.json file from the repo and ingest it, which makes
running a farside server a lot simpler.

It also automatically fetches the latest instance state from
https://farside.link unless configured as a primary farside node, which
will allow others to use farside without increasing traffic to all
instances that are queried by farside (just to the farside node itself).
This commit is contained in:
Ben Busby 2025-01-21 13:46:29 -07:00
parent e0e395f3c8
commit b5bad4defc
No known key found for this signature in database
GPG Key ID: B9B7231E01D924A1
31 changed files with 1031 additions and 768 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
.github/workflows/elixir.yml vendored
View File

@ -1,38 +0,0 @@
name: Elixir CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
name: Build and test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Elixir
uses: erlef/setup-beam@v1
with:
elixir-version: '1.12.3'
otp-version: '24'
- name: Restore dependencies cache
uses: actions/cache@v3
with:
path: deps
key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
restore-keys: ${{ runner.os }}-mix-
- name: Install dependencies
run: mix deps.get
- name: Initialize services
run: FARSIDE_TEST=1 FARSIDE_SERVICES_JSON=services-full.json mix run -e Farside.Instances.sync
- name: Run tests
run: FARSIDE_TEST=1 FARSIDE_SERVICES_JSON=services-full.json mix test --trace

19
.github/workflows/tests.yml vendored Normal file
View File

@ -0,0 +1,19 @@
on: [push, pull_request]
name: Tests
jobs:
test:
strategy:
matrix:
go-version: [1.21.x, 1.22.x, 1.23.x]
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: Test
run: go test -v ./...

20
.gitignore vendored
View File

@ -1,18 +1,2 @@
/_build
/cover
/deps
/doc
/.fetch
erl_crash.dump
*.ez
*.beam
/config/*.secret.exs
.elixir_ls/
# Ignore results from update script
.update-result*
*.rdb
.idea/
*.iml
*.cub
badger-db
farside

40
README.md
View File

@ -7,7 +7,7 @@
[![Latest Release](https://img.shields.io/github/v/release/benbusby/farside?label=Release)](https://github.com/benbusby/farside/releases)
[![MIT License](https://img.shields.io/github/license/benbusby/earthbound-themes.svg)](http://opensource.org/licenses/MIT)
[![Elixir CI](https://github.com/benbusby/privacy-revolver/actions/workflows/elixir.yml/badge.svg)](https://github.com/benbusby/privacy-revolver/actions/workflows/elixir.yml)
[![Tests](https://github.com/benbusby/farside/actions/workflows/tests.yml/badge.svg)](https://github.com/benbusby/farside/actions/workflows/tests.yml)
<table>
<tr>
@ -24,7 +24,6 @@ Contents
3. [How It Works](#how-it-works)
4. [Cloudflare](#regarding-cloudflare)
5. [Development](#development)
1. [Compiling](#compiling)
1. [Environment Variables](#environment-variables)
## About
@ -187,30 +186,9 @@ that their mission to centralize the entire web behind their service ultimately
goes against what Farside is trying to solve. Use at your own discretion.
## Development
- Install [elixir](https://elixir-lang.org/install.html)
- (on Debian systems) Install [erlang-dev](https://packages.debian.org/sid/erlang-dev)
To run Farside without compiling, you can perform the following steps:
- Install dependencies: `mix deps.get`
- Initialize db contents: `FARSIDE_CRON=0 mix run -e Farside.Instances.sync`
- Run Farside: `mix run --no-halt`
- Uses localhost:4001
### Compiling
You can create a standalone Farside app using the steps below. In the example, the
Farside executable is copied to `/usr/local/bin`, but can be moved to any preferred
destination. Note that the executable still depends on the C runtime of the machine
it is built on, so if you want a more portable binary, you should build Farside on a
system with older library versions.
```
MIX_ENV=cli && mix deps.get && mix release
cp _build/cli/rel/bakeware/farside /usr/local/bin
sudo chmod +x /usr/local/bin/farside
farside
```
- Install [Go](https://go.dev/doc/install)
- Compile with `go build`
### Environment Variables
@ -221,23 +199,23 @@ farside
</tr>
<tr>
<td>FARSIDE_TEST</td>
<td>If enabled, bypasses the instance availability check and adds all instances to the pool.</td>
<td>If enabled, bypasses the instance availability check and adds all instances to the pool</td>
</tr>
<tr>
<td>FARSIDE_PORT</td>
<td>The port to run Farside on (default: `4001`)</td>
</tr>
<tr>
<td>FARSIDE_DATA_DIR</td>
<td>The path to the directory to use for storing instance data (default: `/tmp`)</td>
<td>FARSIDE_DB_DIR</td>
<td>The path to the directory to use for storing instance data (default: `./`)</td>
</tr>
<tr>
<td>FARSIDE_SERVICES_JSON</td>
<td>The JSON file to use for selecting instances (default: `services.json`)</td>
<td>FARSIDE_CF_ENABLED</td>
<td>Set to 1 to enable redirecting to instances behind cloudflare</td>
</tr>
<tr>
<td>FARSIDE_CRON</td>
<td>Set to 0 to deactivate the scheduled instance availability check (default on).</td>
<td>Set to 0 to deactivate the periodic instance availability check</td>
</tr>
</table>

19
config/config.exs
View File

@ -1,19 +0,0 @@
import Config
config :farside,
update_file: ".update-results",
service_prefix: "service-",
fallback_suffix: "-fallback",
previous_suffix: "-previous",
index: "index.eex",
route: "route.eex",
headers: [
{"User-Agent", "Mozilla/5.0 (compatible; Farside/0.1.0; +https://farside.link)"},
{"Accept", "text/html"},
{"Accept-Language", "en-US,en;q=0.5"},
{"Accept-Encoding", "gzip, deflate, br"}
],
queries: [
"weather",
"time"
]

6
config/runtime.exs
View File

@ -1,6 +0,0 @@
import Config
config :farside,
port: System.get_env("FARSIDE_PORT", "4001"),
services_json: System.get_env("FARSIDE_SERVICES_JSON", "services.json"),
data_dir: System.get_env("FARSIDE_DATA_DIR", File.cwd!)

139
db/cron.go Normal file
View File

@ -0,0 +1,139 @@
package db
import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strings"
"time"
"github.com/benbusby/farside/services"
"github.com/robfig/cron/v3"
)
const defaultPrimary = "https://farside.link/state"
const defaultCFPrimary = "https://cf.farside.link/state"
var LastUpdate time.Time
func InitCronTasks() {
log.Println("Initializing cron tasks...")
cronDisabled := os.Getenv("FARSIDE_CRON")
if len(cronDisabled) == 0 || cronDisabled == "1" {
c := cron.New()
c.AddFunc("@every 10m", queryServiceInstances)
c.AddFunc("@daily", updateServiceList)
c.Start()
}
queryServiceInstances()
}
func updateServiceList() {
fileName := services.GetServicesFileName()
_, _ = services.FetchServicesFile(fileName)
services.InitializeServices()
}
func queryServiceInstances() {
log.Println("Starting instance queries...")
isPrimary := os.Getenv("FARSIDE_PRIMARY")
if len(isPrimary) == 0 || isPrimary != "1" {
remoteServices, err := fetchInstancesFromPrimary()
if err != nil {
log.Println("Unable to fetch instances from primary", err)
}
for _, service := range remoteServices {
SetInstances(service.Type, service.Instances)
}
return
}
for _, service := range services.ServiceList {
fmt.Printf("===== %s =====\n", service.Type)
var instances []string
for _, instance := range service.Instances {
testURL := strings.ReplaceAll(
service.TestURL,
"<%=query%>",
"current+weather")
available := queryServiceInstance(
instance,
testURL,
)
if available {
instances = append(instances, instance)
}
}
SetInstances(service.Type, instances)
}
LastUpdate = time.Now().UTC()
}
func fetchInstancesFromPrimary() ([]services.Service, error) {
primaryURL := defaultPrimary
useCF := os.Getenv("FARSIDE_CF_ENABLED")
if len(useCF) > 0 && useCF == "1" {
primaryURL = defaultCFPrimary
}
resp, err := http.Get(primaryURL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var serviceList []services.Service
err = json.Unmarshal(bodyBytes, &serviceList)
return serviceList, err
}
func queryServiceInstance(instance, testURL string) bool {
testMode := os.Getenv("FARSIDE_TEST")
if len(testMode) > 0 && testMode == "1" {
return true
}
ua := "Mozilla/5.0 (compatible; Farside/1.0.0; +https://farside.link)"
url := instance + testURL
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
fmt.Println(" [ERRO] Failed to create new http request!", err)
return false
}
req.Header.Set("User-Agent", ua)
client := &http.Client{
Timeout: 10 * time.Second,
}
resp, err := client.Do(req)
if err != nil {
fmt.Println(" [ERRO] Error fetching instance:", err)
return false
} else if resp.StatusCode != http.StatusOK {
fmt.Printf(" [WARN] Received non-200 status for %s\n", url)
return false
} else {
fmt.Printf(" [INFO] Received 200 status for %s\n", url)
}
return true
}

150
db/db.go Normal file
View File

@ -0,0 +1,150 @@
package db
import (
"encoding/json"
"errors"
"log"
"math/rand"
"os"
"slices"
"time"
"github.com/benbusby/farside/services"
"github.com/dgraph-io/badger/v4"
)
var (
badgerDB *badger.DB
selectionMap map[string]string
cachedServiceList []services.Service
cacheUpdated time.Time
)
func InitializeDB() error {
var err error
dbDir := os.Getenv("FARSIDE_DB_DIR")
if len(dbDir) == 0 {
dbDir = "./badger-db"
}
badgerDB, err = badger.Open(badger.DefaultOptions(dbDir))
if err != nil {
return err
}
return nil
}
func SetInstances(service string, instances []string) error {
instancesBytes, err := json.Marshal(instances)
if err != nil {
return err
}
err = badgerDB.Update(func(txn *badger.Txn) error {
err := txn.Set([]byte(service), instancesBytes)
return err
})
if err != nil {
return err
}
return nil
}
func GetInstance(service string) (string, error) {
instances, err := GetAllInstances(service)
if err != nil || len(instances) == 0 {
if err != nil {
log.Println("DB err:", err)
}
link, ok := services.FallbackMap[service]
if !ok {
return "", errors.New("invalid service")
}
return link, nil
}
previous, ok := selectionMap[service]
if ok && len(instances) > 2 {
instances = slices.DeleteFunc(instances, func(i string) bool {
return i == previous
})
}
index := rand.Intn(len(instances))
value := instances[index]
selectionMap[service] = value
return value, nil
}
func GetAllInstances(service string) ([]string, error) {
var instances []string
err := badgerDB.View(func(txn *badger.Txn) error {
item, err := txn.Get([]byte(service))
if err != nil {
return err
}
err = item.Value(func(val []byte) error {
err := json.Unmarshal(val, &instances)
return err
})
return err
})
return instances, err
}
func GetServiceList() []services.Service {
if cacheUpdated.Add(5 * time.Minute).After(time.Now().UTC()) {
return cachedServiceList
}
canCache := true
var serviceList []services.Service
for _, service := range services.ServiceList {
instances, err := GetAllInstances(service.Type)
if err != nil {
canCache = false
instances = []string{service.Fallback}
}
storedService := services.Service{
Type: service.Type,
Instances: instances,
}
serviceList = append(serviceList, storedService)
}
if canCache {
cachedServiceList = serviceList
cacheUpdated = time.Now().UTC()
}
return serviceList
}
func CloseDB() error {
log.Println("Closing database...")
err := badgerDB.Close()
if err != nil {
log.Println("Error closing database", err)
return err
}
log.Println("Database closed!")
return nil
}
func init() {
selectionMap = make(map[string]string)
}

60
db/db_test.go Normal file
View File

@ -0,0 +1,60 @@
package db
import (
"log"
"os"
"slices"
"testing"
)
func TestMain(m *testing.M) {
err := InitializeDB()
if err != nil {
log.Fatalln("Failed to initialize database")
}
exitCode := m.Run()
_ = CloseDB()
os.Exit(exitCode)
}
func TestDatabase(t *testing.T) {
var (
service = "test"
siteA = "a.com"
siteB = "b.com"
siteC = "c.com"
)
instances := []string{siteA, siteB, siteC}
err := SetInstances(service, instances)
if err != nil {
t.Fatalf("Failed to set instances: %v\n", err)
}
dbInstances, err := GetAllInstances(service)
if err != nil {
t.Fatalf("Failed to retrieve instances: %v\n", err)
}
for _, instance := range instances {
idx := slices.Index(dbInstances, instance)
if idx < 0 {
t.Fatalf("Failed to find instance in list")
}
}
firstInstance, err := GetInstance(service)
if err != nil {
t.Fatalf("Failed to fetch single instance: %v\n", err)
}
secondInstance, err := GetInstance(service)
if err != nil {
t.Fatalf("Failed to fetch single instance (second): %v\n", err)
} else if firstInstance == secondInstance {
t.Fatalf("Same instance was selected twice")
}
_ = CloseDB()
}

29
go.mod Normal file
View File

@ -0,0 +1,29 @@
module github.com/benbusby/farside
go 1.23.4
require (
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/charmbracelet/lipgloss v0.10.0 // indirect
github.com/dgraph-io/badger/v4 v4.5.0 // indirect
github.com/dgraph-io/ristretto/v2 v2.0.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mattn/go-isatty v0.0.18 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.15.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/robfig/cron/v3 v3.0.0 // indirect
go.opencensus.io v0.24.0 // indirect
golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
golang.org/x/net v0.31.0 // indirect
golang.org/x/sys v0.27.0 // indirect
google.golang.org/protobuf v1.33.0 // indirect
)

141
go.sum Normal file
View File

@ -0,0 +1,141 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/charmbracelet/lipgloss v0.10.0 h1:KWeXFSexGcfahHX+54URiZGkBFazf70JNMtwg/AFW3s=
github.com/charmbracelet/lipgloss v0.10.0/go.mod h1:Wig9DSfvANsxqkRsqj6x87irdy123SR4dOXlKa91ciE=
github.com/charmbracelet/log v0.4.0 h1:G9bQAcx8rWA2T3pWvx7YtPTPwgqpk7D68BX21IRW8ZM=
github.com/charmbracelet/log v0.4.0/go.mod h1:63bXt/djrizTec0l11H20t8FDSvA4CRZJ1KH22MdptM=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgraph-io/badger/v4 v4.5.0 h1:TeJE3I1pIWLBjYhIYCA1+uxrjWEoJXImFBMEBVSm16g=
github.com/dgraph-io/badger/v4 v4.5.0/go.mod h1:ysgYmIeG8dS/E8kwxT7xHyc7MkmwNYLRoYnFbr7387A=
github.com/dgraph-io/ristretto/v2 v2.0.0 h1:l0yiSOtlJvc0otkqyMaDNysg8E9/F/TYZwMbxscNOAQ=
github.com/dgraph-io/ristretto/v2 v2.0.0/go.mod h1:FVFokF2dRqXyPyeMnK1YDy8Fc6aTe0IKgbcd03CYeEk=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/go-logfmt/logfmt v0.6.0 h1:wGYYu3uicYdqXVgoYbvnkrPVXkuLM1p1ifugDMEdRi4=
github.com/go-logfmt/logfmt v0.6.0/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/flatbuffers v24.3.25+incompatible h1:CX395cjN9Kke9mmalRoL3d81AtFUxJM+yDthflgJGkI=
github.com/google/flatbuffers v24.3.25+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.18 h1:DOKFKCQ7FNG2L1rbrmstDN4QVRdS89Nkh85u68Uwp98=
github.com/mattn/go-isatty v0.0.18/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=
github.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKtnHY/8=
github.com/muesli/termenv v0.15.2 h1:GohcuySI0QmI3wN8Ok9PtKGkgkFIk7y6Vpb5PvrY+Wo=
github.com/muesli/termenv v0.15.2/go.mod h1:Epx+iuz8sNs7mNKhxzH4fWXGNpZwUaJKRS1noLXviQ8=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/robfig/cron/v3 v3.0.0 h1:kQ6Cb7aHOHTSzNVNEhmp8EcWKLb4CbiMW9h9VyIhO4E=
github.com/robfig/cron/v3 v3.0.0/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.31.0 h1:68CPQngjLL0r2AlUKiSxtQFKvzRVbnzLwMUn5SzcLHo=
golang.org/x/net v0.31.0/go.mod h1:P4fl1q7dY2hnZFxEk4pPSkDHF+QqjitcnDjUQyMM+pM=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.27.0 h1:wBqf8DvsY9Y/2P8gAfPDEYNuS30J4lPHJxXSb/nJZ+s=
golang.org/x/sys v0.27.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

142
lib/farside.ex
View File

@ -1,142 +0,0 @@
defmodule Farside do
@service_prefix Application.compile_env!(:farside, :service_prefix)
@fallback_suffix Application.compile_env!(:farside, :fallback_suffix)
@previous_suffix Application.compile_env!(:farside, :previous_suffix)
# Define relation between available services and their parent service.
# This enables Farside to redirect with links such as:
# farside.link/https://www.youtube.com/watch?v=dQw4w9WgXcQ
@youtube_regex ~r/youtu(.be|be.com)|invidious|piped/
@twitter_regex ~r/twitter.com|x.com|nitter/
@reddit_regex ~r/reddit.com|libreddit|redlib/
@instagram_regex ~r/instagram.com|proxigram/
@wikipedia_regex ~r/wikipedia.org|wikiless/
@medium_regex ~r/medium.com|scribe/
@odysee_regex ~r/odysee.com|librarian/
@imgur_regex ~r/imgur.com|rimgo/
@gtranslate_regex ~r/translate.google.com|lingva/
@tiktok_regex ~r/tiktok.com|proxitok/
@imdb_regex ~r/imdb.com|libremdb/
@quora_regex ~r/quora.com|quetre/
@gsearch_regex ~r/google.com\/search|whoogle/
@fandom_regex ~r/fandom.com|breezewiki/
@github_regex ~r/github.com|gothub/
@stackoverflow_regex ~r/stackoverflow.com|anonymousoverflow/
@parent_services %{
@youtube_regex => ["invidious", "piped"],
@reddit_regex => ["libreddit", "redlib"],
@instagram_regex => ["proxigram"],
@twitter_regex => ["nitter"],
@wikipedia_regex => ["wikiless"],
@medium_regex => ["scribe"],
@odysee_regex => ["librarian"],
@imgur_regex => ["rimgo"],
@gtranslate_regex => ["lingva"],
@tiktok_regex => ["proxitok"],
@imdb_regex => ["libremdb"],
@quora_regex => ["quetre"],
@gsearch_regex => ["whoogle"],
@fandom_regex => ["breezewiki"],
@github_regex => ["gothub"],
@stackoverflow_regex => ["anonymousoverflow"]
}
def get_services_map do
service_list = CubDB.select(CubDB)
|> Stream.map(fn {key, _value} -> key end)
|> Stream.filter(fn key -> String.starts_with?(key, @service_prefix) end)
|> Enum.to_list
# Match service name to list of available instances
Enum.reduce(service_list, %{}, fn service, acc ->
instance_list = CubDB.get(CubDB, service)
Map.put(
acc,
String.replace_prefix(
service,
@service_prefix,
""
),
instance_list
)
end)
end
def get_service(service) do
# Check if service has an entry in the db, otherwise try to
# match against available parent services
service_name = cond do
!check_service(service) ->
Enum.find_value(
@parent_services,
fn {k, v} ->
String.match?(service, k) && Enum.random(v)
end)
true ->
service
end
service_name
end
def check_service(service) do
# Checks to see if a specific service has instances available
instances = CubDB.get(CubDB, "#{@service_prefix}#{service}")
instances != nil && Enum.count(instances) > 0
end
def last_instance(service) do
# Fetches the last selected instance for a particular service
CubDB.get(CubDB, "#{service}#{@previous_suffix}")
end
def pick_instance(service) do
instances = CubDB.get(CubDB, "#{@service_prefix}#{service}")
# Either pick a random available instance,
# or fall back to the default one
instance =
if instances != nil && Enum.count(instances) > 0 do
if Enum.count(instances) == 1 do
# If there's only one instance, just return that one...
List.first(instances)
else
# ...otherwise pick a random one from the list, ensuring
# that the same instance is never picked twice in a row.
instance =
Enum.filter(instances, &(&1 != last_instance(service)))
|> Enum.random()
CubDB.put(CubDB, "#{service}#{@previous_suffix}", instance)
instance
end
else
CubDB.get(CubDB, "#{service}#{@fallback_suffix}")
end
instance
end
def amend_instance(instance, service, path) do
cond do
String.match?(service, @fandom_regex) ->
# Fandom links require the subdomain to be preserved, otherwise the
# requested path won't work.
if String.contains?(service, ".fandom.com") do
wiki = String.replace(service, ".fandom.com", "")
"#{instance}/#{wiki}"
else
instance
end
true ->
instance
end
end
def get_last_updated do
CubDB.get(CubDB, "last_updated")
end
end

29
lib/farside/application.ex
View File

@ -1,29 +0,0 @@
defmodule Farside.Application do
@moduledoc false
use Application
@impl true
def start(_type, _args) do
farside_port = Application.fetch_env!(:farside, :port)
data_dir = Application.fetch_env!(:farside, :data_dir)
IO.puts "Running on http://localhost:#{farside_port}"
children = [
Plug.Cowboy.child_spec(
scheme: :http,
plug: Farside.Router,
options: [
port: String.to_integer(farside_port)
]
),
{PlugAttack.Storage.Ets, name: Farside.Throttle.Storage, clean_period: 60_000},
{CubDB, [data_dir: data_dir, name: CubDB, auto_compact: true]},
Farside.Scheduler,
Farside.Server
]
opts = [strategy: :one_for_one, name: Farside.Supervisor]
Supervisor.start_link(children, opts)
end
end

134
lib/farside/instances.ex
View File

@ -1,134 +0,0 @@
defmodule Farside.Instances do
@fallback_suffix Application.fetch_env!(:farside, :fallback_suffix)
@update_file Application.fetch_env!(:farside, :update_file)
@service_prefix Application.fetch_env!(:farside, :service_prefix)
@headers Application.fetch_env!(:farside, :headers)
@queries Application.fetch_env!(:farside, :queries)
@debug_header "======== "
@debug_spacer " "
# These instance uptimes are inspected as part of the nightly Farside build,
# and should not be included in the constant periodic update.
@skip_service_updates ["searxng", "nitter"]
def sync() do
File.rename(@update_file, "#{@update_file}-prev")
update()
# Add UTC time of last update
CubDB.put(CubDB, "last_updated", Calendar.strftime(DateTime.utc_now(), "%c"))
end
def request(url) do
IO.puts("#{@debug_spacer}#{url}")
cond do
System.get_env("FARSIDE_TEST") ->
:good
true ->
HTTPoison.get(url, @headers)
|> then(&elem(&1, 1))
|> Map.get(:status_code)
|> case do
n when n < 300 ->
IO.puts("#{@debug_spacer}✓ [#{n}]")
:good
n ->
IO.puts("#{@debug_spacer}x [#{(n && n) || "error"}]")
:bad
end
end
end
def update() do
services_json = Application.fetch_env!(:farside, :services_json)
{:ok, file} = File.read(services_json)
{:ok, json} = Jason.decode(file)
# Loop through all instances and check each for availability
for service_json <- json do
service_atom = for {key, val} <- service_json, into: %{} do
{String.to_existing_atom(key), val}
end
service = struct(%Service{}, service_atom)
IO.puts("#{@debug_header}#{service.type}")
result = cond do
Enum.member?(@skip_service_updates, service.type) ->
get_service_vals(service.instances)
true ->
Enum.filter(service.instances, fn instance_url ->
test_url = get_test_val(instance_url)
test_path = get_test_val(service.test_url)
test_request_url = gen_validation_url(test_url, test_path)
service_url = get_service_val(instance_url)
service_path = get_service_val(service.test_url)
service_request_url = gen_validation_url(service_url, service_path)
cond do
service_url != test_url ->
service_up = request(service_request_url)
test_up = request(test_request_url)
service_up == :good && test_up == :good
true ->
request(test_request_url) == :good
end
end)
end
add_to_db(service, result)
log_results(service.type, result)
end
end
def add_to_db(service, instances) do
# Ensure only service URLs are inserted, not test URLs (separated by "|")
instances = get_service_vals(instances)
# Remove previous list of instances
CubDB.delete(CubDB, "#{@service_prefix}#{service.type}")
# Update with new list of available instances
CubDB.put(CubDB, "#{@service_prefix}#{service.type}", instances)
# Set fallback to one of the available instances,
# or the default instance if all are "down"
if Enum.count(instances) > 0 do
CubDB.put(CubDB, "#{service.type}#{@fallback_suffix}", Enum.random(instances))
else
CubDB.put(CubDB, "#{service.type}#{@fallback_suffix}", service.fallback)
end
end
def log_results(service_name, results) do
{:ok, file} = File.open(@update_file, [:append, {:delayed_write, 100, 20}])
IO.write(file, "#{service_name}: #{inspect(results)}\n")
File.close(file)
end
def gen_validation_url(url, path) do
url <> EEx.eval_string(path, query: Enum.random(@queries))
end
def get_service_vals(services) do
Enum.map(services, fn x -> get_service_val(x) end)
end
def get_service_val(service) do
String.split(service, "|") |> List.first
end
def get_test_vals(services) do
Enum.map(services, fn x -> get_test_val(x) end)
end
def get_test_val(service) do
String.split(service, "|") |> List.last
end
end

78
lib/farside/router.ex
View File

@ -1,78 +0,0 @@
defmodule Farside.Router do
@index Application.fetch_env!(:farside, :index)
@route Application.fetch_env!(:farside, :route)
use Plug.Router
plug(RemoteIp)
plug(Farside.Throttle)
plug(:match)
plug(:dispatch)
def get_query_params(conn) do
cond do
String.length(conn.query_string) > 0 ->
"?#{conn.query_string}"
true ->
""
end
end
match "/" do
resp =
EEx.eval_file(
@index,
last_updated: Farside.get_last_updated(),
services: Farside.get_services_map()
)
put_resp_header(conn, "content-type", "text/html")
|> send_resp(200, resp)
end
match "/_/:service/*glob" do
r_path = String.slice(conn.request_path, 2..-1)
resp =
EEx.eval_file(
@route,
instance_url: "#{r_path}#{get_query_params(conn)}"
)
send_resp(conn, 200, resp)
end
match "/:service/*glob" do
service_name = cond do
service =~ "http" ->
List.first(glob)
true ->
service
end
path = cond do
service_name != service ->
Enum.join(Enum.slice(glob, 1..-1), "/")
true ->
Enum.join(glob, "/")
end
cond do
conn.assigns[:throttle] != nil ->
send_resp(conn, :too_many_requests, "Too many requests - max request rate is 1 per second")
true ->
instance = Farside.get_service(service_name)
|> Farside.pick_instance
|> Farside.amend_instance(service_name, path)
# Redirect to the available instance
conn
|> Plug.Conn.resp(:found, "")
|> Plug.Conn.put_resp_header(
"location",
"#{instance}/#{path}#{get_query_params(conn)}"
)
end
end
end

3
lib/farside/scheduler.ex
View File

@ -1,3 +0,0 @@
defmodule Farside.Scheduler do
use Quantum, otp_app: :farside
end

25
lib/farside/server.ex
View File

@ -1,25 +0,0 @@
defmodule Farside.Server do
use GenServer
import Crontab.CronExpression
def init(init_arg) do
{:ok, init_arg}
end
def start_link(arg) do
test = System.get_env("FARSIDE_TEST")
cron = System.get_env("FARSIDE_CRON")
if test == "1" || cron == "0" do
IO.puts("Skipping sync job setup...")
else
Farside.Scheduler.new_job()
|> Quantum.Job.set_name(:sync)
|> Quantum.Job.set_schedule(~e[*/5 * * * *])
|> Quantum.Job.set_task(fn -> Farside.Instances.sync() end)
|> Farside.Scheduler.add_job()
end
GenServer.start_link(__MODULE__, arg)
end
end

20
lib/farside/throttle.ex
View File

@ -1,20 +0,0 @@
defmodule Farside.Throttle do
import Plug.Conn
use PlugAttack
rule "throttle per ip", conn do
# throttle to 1 request per second
throttle(conn.remote_ip,
period: 1_000,
limit: 1,
storage: {PlugAttack.Storage.Ets, Farside.Throttle.Storage}
)
end
def allow_action(conn, _data, _opts), do: conn
def block_action(conn, _data, _opts) do
conn = assign(conn, :throttle, 1)
conn
end
end

6
lib/service.ex
View File

@ -1,6 +0,0 @@
defmodule Service do
defstruct type: nil,
test_url: nil,
fallback: nil,
instances: []
end

37
main.go Normal file
View File

@ -0,0 +1,37 @@
package main
import (
"log"
"os"
"os/signal"
"syscall"
"github.com/benbusby/farside/db"
"github.com/benbusby/farside/server"
"github.com/benbusby/farside/services"
)
func main() {
err := db.InitializeDB()
if err != nil {
log.Fatal(err)
}
err = services.InitializeServices()
if err != nil {
log.Fatal(err)
}
db.InitCronTasks()
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, os.Interrupt, syscall.SIGTERM)
go func() {
<-signalChan
_ = db.CloseDB()
os.Exit(0)
}()
server.RunServer()
}

74
mix.exs
View File

@ -1,74 +0,0 @@
defmodule Farside.MixProject do
use Mix.Project
@source_url "https://github.com/benbusby/farside.git"
@version "0.1.1"
@app :farside
def project do
[
app: @app,
version: @version,
name: "farside",
elixir: "~> 1.8",
source_url: @source_url,
start_permanent: Mix.env() == :prod || Mix.env() == :cli,
deps: deps(),
aliases: aliases(),
description: description(),
package: package(),
releases: [{@app, release()}],
preferred_cli_env: [release: :cli]
]
end
defp aliases do
[]
end
# Run "mix help compile.app" to learn about applications.
def application do
[
extra_applications: [:logger],
mod: {Farside.Application, []}
]
end
# Run "mix help deps" to learn about dependencies.
defp deps do
[
{:httpoison, "~> 1.8"},
{:jason, "~> 1.1"},
{:plug_attack, "~> 0.4.2"},
{:plug_cowboy, "~> 2.0"},
{:quantum, "~> 3.0"},
{:remote_ip, "~> 1.1"},
{:cubdb, "~> 2.0.1"},
{:bakeware, "~> 0.2.4"}
]
end
defp description() do
"A redirecting service for FOSS alternative frontends."
end
defp package() do
[
name: "farside",
files: ["lib", "mix.exs", "README*"],
maintainers: ["Ben Busby"],
licenses: ["MIT"],
links: %{"GitHub" => "https://github.com/benbusby/farside"}
]
end
defp release() do
[
overwrite: true,
cookie: "#{@app}_cookie",
quiet: true,
steps: [:assemble, &Bakeware.assemble/1],
strip_beams: Mix.env() == :cli
]
end
end

31
mix.lock
View File

@ -1,31 +0,0 @@
%{
"bakeware": {:hex, :bakeware, "0.2.4", "0aaf49b34f4bab2aa433f9ff1485d9401e421603160abd6d269c469fc7b65212", [:make, :mix], [{:elixir_make, "~> 0.6", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "7b97bcf6fbeee53bb32441d6c495bf478d26f9575633cfef6831e421e86ada6d"},
"certifi": {:hex, :certifi, "2.9.0", "6f2a475689dd47f19fb74334859d460a2dc4e3252a3324bd2111b8f0429e7e21", [:rebar3], [], "hexpm", "266da46bdb06d6c6d35fde799bcb28d36d985d424ad7c08b5bb48f5b5cdd4641"},
"combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"},
"cowboy": {:hex, :cowboy, "2.9.0", "865dd8b6607e14cf03282e10e934023a1bd8be6f6bacf921a7e2a96d800cd452", [:make, :rebar3], [{:cowlib, "2.11.0", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, "1.8.0", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm", "2c729f934b4e1aa149aff882f57c6372c15399a20d54f65c8d67bef583021bde"},
"cowboy_telemetry": {:hex, :cowboy_telemetry, "0.4.0", "f239f68b588efa7707abce16a84d0d2acf3a0f50571f8bb7f56a15865aae820c", [:rebar3], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "7d98bac1ee4565d31b62d59f8823dfd8356a169e7fcbb83831b8a5397404c9de"},
"cowlib": {:hex, :cowlib, "2.11.0", "0b9ff9c346629256c42ebe1eeb769a83c6cb771a6ee5960bd110ab0b9b872063", [:make, :rebar3], [], "hexpm", "2b3e9da0b21c4565751a6d4901c20d1b4cc25cbb7fd50d91d2ab6dd287bc86a9"},
"crontab": {:hex, :crontab, "1.1.11", "4028ced51b813a5061f85b689d4391ef0c27550c8ab09aaf139e4295c3d93ea4", [:mix], [{:ecto, "~> 1.0 or ~> 2.0 or ~> 3.0", [hex: :ecto, repo: "hexpm", optional: true]}], "hexpm", "ecb045f9ac14a3e2990e54368f70cdb6e2f2abafc5bc329d6c31f0c74b653787"},
"cubdb": {:hex, :cubdb, "2.0.1", "24cab8fb4128df704c52ed641f5ed70af352f7a3a80cebbb44c3bbadc3fd5f45", [:mix], [], "hexpm", "57cf25aebfc34f4580d9075da06882b4fe3e0739f5353d4dcc213e9cc1b10cdf"},
"elixir_make": {:hex, :elixir_make, "0.7.6", "67716309dc5d43e16b5abbd00c01b8df6a0c2ab54a8f595468035a50189f9169", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}], "hexpm", "5a0569756b0f7873a77687800c164cca6dfc03a09418e6fcf853d78991f49940"},
"gen_stage": {:hex, :gen_stage, "1.1.2", "b1656cd4ba431ed02c5656fe10cb5423820847113a07218da68eae5d6a260c23", [:mix], [], "hexpm", "9e39af23140f704e2b07a3e29d8f05fd21c2aaf4088ff43cb82be4b9e3148d02"},
"hackney": {:hex, :hackney, "1.18.1", "f48bf88f521f2a229fc7bae88cf4f85adc9cd9bcf23b5dc8eb6a1788c662c4f6", [:rebar3], [{:certifi, "~>2.9.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~>6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~>1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~>1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.3.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~>1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~>0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "a4ecdaff44297e9b5894ae499e9a070ea1888c84afdd1fd9b7b2bc384950128e"},
"httpoison": {:hex, :httpoison, "1.8.2", "9eb9c63ae289296a544842ef816a85d881d4a31f518a0fec089aaa744beae290", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "2bb350d26972e30c96e2ca74a1aaf8293d61d0742ff17f01e0279fef11599921"},
"idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~>0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"},
"jason": {:hex, :jason, "1.4.0", "e855647bc964a44e2f67df589ccf49105ae039d4179db7f6271dfd3843dc27e6", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "79a3791085b2a0f743ca04cec0f7be26443738779d09302e01318f97bdb82121"},
"metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"},
"mime": {:hex, :mime, "2.0.3", "3676436d3d1f7b81b5a2d2bd8405f412c677558c81b1c92be58c00562bb59095", [:mix], [], "hexpm", "27a30bf0db44d25eecba73755acf4068cbfe26a4372f9eb3e4ea3a45956bff6b"},
"mimerl": {:hex, :mimerl, "1.2.0", "67e2d3f571088d5cfd3e550c383094b47159f3eee8ffa08e64106cdf5e981be3", [:rebar3], [], "hexpm", "f278585650aa581986264638ebf698f8bb19df297f66ad91b18910dfc6e19323"},
"parse_trans": {:hex, :parse_trans, "3.3.1", "16328ab840cc09919bd10dab29e431da3af9e9e7e7e6f0089dd5a2d2820011d8", [:rebar3], [], "hexpm", "07cd9577885f56362d414e8c4c4e6bdf10d43a8767abb92d24cbe8b24c54888b"},
"plug": {:hex, :plug, "1.14.0", "ba4f558468f69cbd9f6b356d25443d0b796fbdc887e03fa89001384a9cac638f", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "bf020432c7d4feb7b3af16a0c2701455cbbbb95e5b6866132cb09eb0c29adc14"},
"plug_attack": {:hex, :plug_attack, "0.4.3", "88e6c464d68b1491aa083a0347d59d58ba71a7e591a7f8e1b675e8c7792a0ba8", [:mix], [{:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "9ed6fb8a6f613a36040f2875130a21187126c5625092f24bc851f7f12a8cbdc1"},
"plug_cowboy": {:hex, :plug_cowboy, "2.6.0", "d1cf12ff96a1ca4f52207c5271a6c351a4733f413803488d75b70ccf44aebec2", [:mix], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:cowboy_telemetry, "~> 0.3", [hex: :cowboy_telemetry, repo: "hexpm", optional: false]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "073cf20b753ce6682ed72905cd62a2d4bd9bad1bf9f7feb02a1b8e525bd94fa6"},
"plug_crypto": {:hex, :plug_crypto, "1.2.3", "8f77d13aeb32bfd9e654cb68f0af517b371fb34c56c9f2b58fe3df1235c1251a", [:mix], [], "hexpm", "b5672099c6ad5c202c45f5a403f21a3411247f164e4a8fab056e5cd8a290f4a2"},
"quantum": {:hex, :quantum, "3.5.0", "8d2c5ba68c55991e8975aca368e3ab844ba01f4b87c4185a7403280e2c99cf34", [:mix], [{:crontab, "~> 1.1", [hex: :crontab, repo: "hexpm", optional: false]}, {:gen_stage, "~> 0.14 or ~> 1.0", [hex: :gen_stage, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:telemetry_registry, "~> 0.2", [hex: :telemetry_registry, repo: "hexpm", optional: false]}], "hexpm", "cab737d1d9779f43cb1d701f46dd05ea58146fd96238d91c9e0da662c1982bb6"},
"ranch": {:hex, :ranch, "1.8.0", "8c7a100a139fd57f17327b6413e4167ac559fbc04ca7448e9be9057311597a1d", [:make, :rebar3], [], "hexpm", "49fbcfd3682fab1f5d109351b61257676da1a2fdbe295904176d5e521a2ddfe5"},
"remote_ip": {:hex, :remote_ip, "1.1.0", "cb308841595d15df3f9073b7c39243a1dd6ca56e5020295cb012c76fbec50f2d", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "616ffdf66aaad6a72fc546dabf42eed87e2a99e97b09cbd92b10cc180d02ed74"},
"ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
"telemetry": {:hex, :telemetry, "1.1.0", "a589817034a27eab11144ad24d5c0f9fab1f58173274b1e9bae7074af9cbee51", [:rebar3], [], "hexpm", "b727b2a1f75614774cff2d7565b64d0dfa5bd52ba517f16543e6fc7efcc0df48"},
"telemetry_registry": {:hex, :telemetry_registry, "0.3.0", "6768f151ea53fc0fbca70dbff5b20a8d663ee4e0c0b2ae589590e08658e76f1e", [:mix, :rebar3], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "492e2adbc609f3e79ece7f29fec363a97a2c484ac78a83098535d6564781e917"},
"unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"},
}

10
route.eex
View File

@ -1,10 +0,0 @@
<head>
<title>Farside Redirect</title>
<meta http-equiv="refresh" content="1; url=<%= instance_url %>">
<script>
history.pushState({page: 1}, "Farside Redirect");
</script>
</head>
<body>
<span>Redirecting to <%= instance_url %>...</span>
</body>

24
index.eex → server/index.html
View File

@ -1,3 +1,6 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>Farside</title>
<style>
@ -44,19 +47,20 @@
<div id="child-div">
<h1>Farside [<a href="https://sr.ht/~benbusby/farside">SourceHut</a>, <a href="https://github.com/benbusby/farside">GitHub</a>]</h1>
<hr>
<h3>Last synced <%= last_updated %> UTC</h2>
<h3>Updated: {{ .LastUpdated }}</h2>
<div>
<ul>
<%= for {service, instance_list} <- services do %>
<li><a href="/<%= service %>"><%= service %></a></li>
<ul>
<%= for url <- instance_list do %>
<li><a href="<%= url %>"><%= url %></a></li>
<% end%>
</ul>
<% end %>
</ul>
{{ range $i, $service := .ServiceList }}
<li><a href="/{{ $service.Type }}">{{ $service.Type }}</a></li>
<ul>
{{ range $j, $instance := $service.Instances }}
<li><a href="{{ $instance }}">{{ $instance }}</li>
{{ end }}
</ul>
{{ end }}
</ul>
</div>
</div>
</div>
</body>

10
server/route.html Normal file
View File

@ -0,0 +1,10 @@
<head>
<title>Farside Redirect</title>
<meta http-equiv="refresh" content="1; url={{ .InstanceURL }}">
<script>
history.pushState({page: 1}, "Farside Redirect");
</script>
</head>
<body>
<span>Redirecting to {{ .InstanceURL }}...</span>
</body>

138
server/server.go Normal file
View File

@ -0,0 +1,138 @@
package server
import (
_ "embed"
"encoding/json"
"html/template"
"log"
"net/http"
"net/url"
"os"
"strings"
"time"
"github.com/benbusby/farside/db"
"github.com/benbusby/farside/services"
)
//go:embed index.html
var indexHTML string
//go:embed route.html
var routeHTML string
type indexData struct {
LastUpdated time.Time
ServiceList []services.Service
}
type routeData struct {
InstanceURL string
}
func home(w http.ResponseWriter, r *http.Request) {
serviceList := db.GetServiceList()
data := indexData{
LastUpdated: db.LastUpdate,
ServiceList: serviceList,
}
tmpl, err := template.New("").Parse(indexHTML)
if err != nil {
log.Println(err)
http.Error(w, "Error parsing template", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html")
err = tmpl.Execute(w, data)
if err != nil {
log.Println(err)
http.Error(w, "Error executing template", http.StatusInternalServerError)
}
}
func state(w http.ResponseWriter, r *http.Request) {
storedServices := db.GetServiceList()
jsonData, _ := json.Marshal(storedServices)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(jsonData)
}
func baseRouting(w http.ResponseWriter, r *http.Request) {
routing(w, r, false)
}
func jsRouting(w http.ResponseWriter, r *http.Request) {
r.URL.Path = strings.Replace(r.URL.Path, "/_", "", 1)
routing(w, r, true)
}
func routing(w http.ResponseWriter, r *http.Request, jsEnabled bool) {
value := r.PathValue("routing")
if len(value) == 0 {
value = r.URL.Path
}
url, _ := url.Parse(value)
path := strings.TrimPrefix(url.Path, "/")
segments := strings.Split(path, "/")
target, err := services.MatchRequest(segments[0])
if err != nil {
log.Printf("Error during match request: %v\n", err)
http.Error(w, "No routing found for "+target, http.StatusBadRequest)
return
}
instance, err := db.GetInstance(target)
if err != nil {
log.Printf("Error fetching instance from db: %v\n", err)
http.Error(
w,
"Error fetching instance for "+target,
http.StatusInternalServerError)
return
}
if len(segments) > 1 {
targetPath := strings.Join(segments[1:], "/")
instance = instance + "/" + targetPath
}
w.Header().Set("Cache-Control", "no-store, no-cache, must-revalidate, max-age=0")
w.Header().Set("Pragma", "no-cache")
w.Header().Set("Expires", "0")
if jsEnabled {
data := routeData{
InstanceURL: instance,
}
tmpl, _ := template.New("").Parse(routeHTML)
w.Header().Set("Content-Type", "text/html")
_ = tmpl.Execute(w, data)
} else {
http.Redirect(w, r, instance, http.StatusFound)
}
}
func RunServer() {
mux := http.NewServeMux()
mux.HandleFunc("/{$}", home)
mux.HandleFunc("/state/{$}", state)
mux.HandleFunc("/{routing...}", baseRouting)
mux.HandleFunc("/_/{routing...}", jsRouting)
port := os.Getenv("FARSIDE_PORT")
if len(port) == 0 {
port = "4001"
}
log.Println("Starting server on http://localhost:" + port)
err := http.ListenAndServe(":"+port, mux)
if err != nil {
log.Fatal(err)
}
}

80
server/server_test.go Normal file
View File

@ -0,0 +1,80 @@
package server
import (
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strings"
"testing"
"github.com/benbusby/farside/db"
)
const breezewikiTestSite = "https://breezewikitest.com"
func TestMain(m *testing.M) {
err := db.InitializeDB()
if err != nil {
log.Fatalln("Failed to initialize database", err)
}
err = db.SetInstances("breezewiki", []string{breezewikiTestSite})
if err != nil {
log.Fatalln("Failed to set instances in db")
}
exitCode := m.Run()
_ = db.CloseDB()
os.Exit(exitCode)
}
func TestBaseRouting(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/fandom.com", nil)
w := httptest.NewRecorder()
baseRouting(w, req)
res := w.Result()
defer res.Body.Close()
if res.StatusCode != http.StatusFound {
t.Fatalf("Incorrect resp code (%d) in base routing", res.StatusCode)
}
expectedHost, _ := url.Parse(breezewikiTestSite)
redirect, err := res.Location()
if err != nil {
t.Fatalf("Error retrieving direct from request: %v\n", err)
} else if redirect.Host != expectedHost.Host {
t.Fatalf("Incorrect redirect site -- expected: %s, actual: %s\n",
expectedHost.Host,
redirect.Host)
}
}
func TestJSRouting(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/_/fandom.com", nil)
w := httptest.NewRecorder()
jsRouting(w, req)
res := w.Result()
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
t.Fatalf("Incorrect resp code (%d) in base routing", res.StatusCode)
}
data, err := io.ReadAll(res.Body)
if err != nil {
t.Fatalf("Error reading response body: %v", err)
}
if !strings.Contains(string(data), breezewikiTestSite) {
t.Fatalf("%s not found in response body (%s)", breezewikiTestSite, string(data))
}
}

110
services/mappings.go Normal file
View File

@ -0,0 +1,110 @@
package services
import (
"errors"
"math/rand"
"regexp"
)
type RegexMapping struct {
Pattern *regexp.Regexp
Targets []string
}
var regexMap = []RegexMapping{
{
// YouTube
Pattern: regexp.MustCompile(`youtu(\.be|be\.com)|invidious|piped`),
Targets: []string{"piped", "invidious"},
},
{
// Twitter / X
Pattern: regexp.MustCompile(`twitter\.com|x\.com|nitter`),
Targets: []string{"nitter"},
},
{
// Reddit
Pattern: regexp.MustCompile(`reddit\.com|libreddit|redlib`),
Targets: []string{"libreddit", "redlib"},
},
{
// Google Search
Pattern: regexp.MustCompile(`google\.com|whoogle|searx|searxng`),
Targets: []string{"whoogle", "searx", "searxng"},
},
{
// Instagram
Pattern: regexp.MustCompile(`instagram\.com|proxigram`),
Targets: []string{"proxigram"},
},
{
// Wikipedia
Pattern: regexp.MustCompile(`wikipedia\.org|wikiless`),
Targets: []string{"wikiless"},
},
{
// Medium
Pattern: regexp.MustCompile(`medium\.com|scribe`),
Targets: []string{"scribe"},
},
{
// Odysee
Pattern: regexp.MustCompile(`odysee\.com|librarian`),
Targets: []string{"librarian"},
},
{
// Imgur
Pattern: regexp.MustCompile(`imgur\.com|rimgo`),
Targets: []string{"rimgo"},
},
{
// Google Translate
Pattern: regexp.MustCompile(`translate\.google\.com|lingva`),
Targets: []string{"lingva"},
},
{
// TikTok
Pattern: regexp.MustCompile(`tiktok\.com|proxitok`),
Targets: []string{"proxitok"},
},
{
// Fandom
Pattern: regexp.MustCompile(`fandom\.com|breezewiki`),
Targets: []string{"breezewiki"},
},
{
// IMDB
Pattern: regexp.MustCompile(`imdb\.com|libremdb`),
Targets: []string{"libremdb"},
},
{
// Quora
Pattern: regexp.MustCompile(`quora\.com|quetre`),
Targets: []string{"quetre"},
},
{
// GitHub
Pattern: regexp.MustCompile(`github\.com|gothub`),
Targets: []string{"gothub"},
},
{
// StackOverflow
Pattern: regexp.MustCompile(`stackoverflow\.com|anonymousoverflow`),
Targets: []string{"anonymousoverflow"},
},
}
func MatchRequest(service string) (string, error) {
for _, mapping := range regexMap {
hasMatch := mapping.Pattern.MatchString(service)
if !hasMatch {
continue
}
index := rand.Intn(len(mapping.Targets))
value := mapping.Targets[index]
return value, nil
}
return "", errors.New("no match found")
}

93
services/services.go Normal file
View File

@ -0,0 +1,93 @@
package services
import (
"encoding/json"
"io"
"net/http"
"os"
)
var (
ServiceList []Service
FallbackMap map[string]string
)
const (
baseRepoLink = "https://git.sr.ht/~benbusby/farside/blob/main/"
noCFServicesJSON = "services.json"
fullServicesJSON = "services-full.json"
)
type Service struct {
Type string `json:"type"`
TestURL string `json:"test_url,omitempty"`
Fallback string `json:"fallback,omimtempty"`
Instances []string `json:"instances"`
}
func ingestServicesList(servicesBytes []byte) error {
err := json.Unmarshal(servicesBytes, &ServiceList)
return err
}
func GetServicesFileName() string {
cloudflareEnabled := false
cfEnabledVar := os.Getenv("FARSIDE_CF_ENABLED")
if len(cfEnabledVar) > 0 && cfEnabledVar == "1" {
cloudflareEnabled = true
}
serviceJSON := noCFServicesJSON
if cloudflareEnabled {
serviceJSON = fullServicesJSON
}
return serviceJSON
}
func FetchServicesFile(serviceJSON string) ([]byte, error) {
resp, err := http.Get(baseRepoLink + serviceJSON)
if err != nil {
return nil, err
}
defer resp.Body.Close()
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
err = os.WriteFile(serviceJSON, bodyBytes, 0666)
if err != nil {
return nil, err
}
return bodyBytes, nil
}
func InitializeServices() error {
serviceJSON := GetServicesFileName()
fileBytes, err := os.ReadFile(serviceJSON)
if err != nil {
fileBytes, err = FetchServicesFile(serviceJSON)
if err != nil {
return err
}
}
err = ingestServicesList(fileBytes)
if err != nil {
return err
}
FallbackMap = make(map[string]string)
for _, serviceElement := range ServiceList {
FallbackMap[serviceElement.Type] = serviceElement.Fallback
}
return nil
}

93
test/farside_test.exs
View File

@ -1,93 +0,0 @@
defmodule FarsideTest do
use ExUnit.Case
use Plug.Test
alias Farside.Router
@opts Router.init([])
def test_conn(path) do
:timer.sleep(1000)
:get
|> conn(path, "")
|> Router.call(@opts)
end
test "throttle" do
first_conn =
:get
|> conn("/", "")
|> Router.call(@opts)
first_redirect = elem(List.last(first_conn.resp_headers), 1)
throttled_conn =
:get
|> conn("/", "")
|> Router.call(@opts)
throttled_redirect = elem(List.last(first_conn.resp_headers), 1)
assert throttled_conn.state == :sent
assert throttled_redirect == first_redirect
end
test "/" do
conn = test_conn("/")
assert conn.state == :sent
assert conn.status == 200
end
test "/:service" do
services_json = Application.fetch_env!(:farside, :services_json)
{:ok, file} = File.read(services_json)
{:ok, service_list} = Jason.decode(file)
service_names =
Enum.map(
service_list,
fn service -> service["type"] end
)
IO.puts("")
service_names |>
Enum.filter(fn service_name -> service_name != "nitter" end) |>
Enum.map(fn service_name ->
conn = test_conn("/#{service_name}")
first_redirect = elem(List.last(conn.resp_headers), 1)
IO.puts(" /#{service_name} (#1) -- #{first_redirect}")
assert conn.state == :set
assert conn.status == 302
conn = test_conn("/#{service_name}")
second_redirect = elem(List.last(conn.resp_headers), 1)
IO.puts(" /#{service_name} (#2) -- #{second_redirect}")
assert conn.state == :set
assert conn.status == 302
assert first_redirect != second_redirect
end)
end
test "/https://..." do
parent_service = "https://www.youtube.com"
parent_path = "watch?v=dQw4w9WgXcQ"
conn = test_conn("/#{parent_service}/#{parent_path}")
redirect = elem(List.last(conn.resp_headers), 1)
IO.puts("")
IO.puts(" /#{parent_service}/#{parent_path}")
IO.puts(" redirected to")
IO.puts(" #{redirect}")
assert conn.state == :set
assert conn.status == 302
assert redirect =~ parent_path
assert !(redirect =~ parent_service)
end
end

1
test/test_helper.exs
View File

@ -1 +0,0 @@
ExUnit.start()