auto-update on Sat Apr 9 22:09:26 UTC 2022

2024-10-01 01:06:18 -04:00 · 2022-04-09 22:09:26 +00:00 · 2022-04-09 22:09:26 +00:00 · 8350340058
commit 8350340058
parent ed07b8cfd9
5 changed files with 38 additions and 19 deletions
--- a/.02-footnotes.md.swp
+++ b/.02-footnotes.md.swp
--- a/01-preamble.md
+++ b/01-preamble.md
@ -9,6 +9,8 @@ This is a list of substantial, commercial-or-social-good mainstream websites whi
 - licensed: cc-by-sa
 - author/editor: alec muffett

+## Legend/Key for Symbols
+
 You can find techical details and the legend/key for symbols in the [footnotes section](#footnotes), below.

 ## Regarding Updates and Suggestions
--- a/02-footnotes.md
+++ b/02-footnotes.md
@ -14,8 +14,7 @@
 - :eight_spoked_asterisk: site up, and redirected to another page
 - :no_entry_sign: site up, but could not access the page
 - :stop_sign: site up, but reported a system error
- :sos: site returned no data, or is down, or curl experienced a transient
-  network error (may be a problem with the RWOS server connection)
+- :sos: site returned no data, or is down, or curl experienced a transient or permanent network error; may also reflect a problem with the RWOS server connection
 - :new: site is newly added, no data yet

 You can also see the [history of updates](https://github.com/alecmuffett/real-world-onion-sites/commits/master/README.md).
@ -26,7 +25,9 @@ Mouse-over the icons for details of HTTP codes, curl exit statuses,
 and the number of attempts made on each site.

 - codes [are from HTTP and are documented elsewhere](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes); RWOS-internal ones include:
-  - `901`, `902`, `903` - malformed HTTP response
+  - `901` - malformed HTTP response
+  - `902` - malformed HTTP response
+  - `903` - malformed HTTP response, commonly including (e.g.) invalid HTTPS certificate
  - `904` - HTTP status code parse error
  - `910` - connection timeout
 - exits [are from Curl and are documented elsewhere](https://curl.haxx.se/libcurl/c/libcurl-errors.html); common ones include:
--- a/README.md
+++ b/README.md
@ -9,6 +9,8 @@ This is a list of substantial, commercial-or-social-good mainstream websites whi
 - licensed: cc-by-sa
 - author/editor: alec muffett

+## Legend/Key for Symbols
+
 You can find techical details and the legend/key for symbols in the [footnotes section](#footnotes), below.

 ## Regarding Updates and Suggestions
@ -46,7 +48,7 @@ You can find techical details and the legend/key for symbols in the [footnotes s
 * link: [https://lxwu7pwyszfevhglxfgaukjqjdk2belosfvsl2ekzx3vrboacvewc7qd.onion/](https://lxwu7pwyszfevhglxfgaukjqjdk2belosfvsl2ekzx3vrboacvewc7qd.onion/)
 * plain: `https://lxwu7pwyszfevhglxfgaukjqjdk2belosfvsl2ekzx3vrboacvewc7qd.onion/`
 * proof: [link](https://ahf.me)
-* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:26+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:23:20+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:23:38+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:16+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:24+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:30+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:11:38+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:45:34+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:45:31+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:20:30+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:20:29+00:00">:sos:</span>
+* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:26+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:23:20+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:23:38+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:22+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:16+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:24+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:30+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:11:38+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:45:34+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:45:31+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:20:30+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:20:29+00:00">:old_key:</span>

 ### [Ctrl blog](http://v65ngaoj2nyaiq2ltf4uzota254gnasarrkuj4aqndi2bb5lw6frt3ad.onion/)
 * transport: :small_red_triangle: **HTTP**
@ -67,7 +69,7 @@ You can find techical details and the legend/key for symbols in the [footnotes s
 * link: [https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion/](https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion/)
 * plain: `https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion/`
 * proof: [link](https://kushaldas.in/)
-* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:43+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:53+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:26:38+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:24:02+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:46+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:52+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:53+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:18:28+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:52:01+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:51:57+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:27:09+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:26:57+00:00">:sos:</span>
+* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:43+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:53+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:26:38+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:24:02+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:46+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:52+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:53+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:18:28+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:52:01+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:51:57+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:27:09+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:26:57+00:00">:old_key:</span>

 ### [Michael Altfield](http://michaelahgu3sqef5yz3u242nok2uczduq5oxqfkwq646tvjhdnl35id.onion/)
 * transport: :small_red_triangle: **HTTP**
@ -88,7 +90,7 @@ You can find techical details and the legend/key for symbols in the [footnotes s
 * link: [https://nickf43ab43xxf3yqgzy5uedsjij6h473rmbyzq6inohcnr3lohlu3yd.onion/](https://nickf43ab43xxf3yqgzy5uedsjij6h473rmbyzq6inohcnr3lohlu3yd.onion/)
 * plain: `https://nickf43ab43xxf3yqgzy5uedsjij6h473rmbyzq6inohcnr3lohlu3yd.onion/`
 * proof: [link](https://frichetten.com)
-* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:34+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:52+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:23:32+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:30:57+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:33+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:27+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:24+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:35+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 07:06:42+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:45:46+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:45:49+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:33:22+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:20:32+00:00">:sos:</span>
+* check: <span title="attempts=1 code=200 exit=0 time=2022-04-09 20:49:34+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-09 08:49:52+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 17:23:32+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-08 05:30:57+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 13:57:33+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-07 01:57:27+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 22:32:24+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-06 10:32:37+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 19:06:35+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-05 07:06:42+00:00">:white_check_mark:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:45:46+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:45:49+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 12:33:22+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-03 00:20:32+00:00">:old_key:</span>

 ### [Shen's Essays](https://shen.hongio267dx4o2ofkn4ddsztu4ok2vq4euc7sxumbi7kcfd64ije62ad.onion/)
 * transport: :closed_lock_with_key: **HTTPS**
@ -903,7 +905,7 @@ You can find techical details and the legend/key for symbols in the [footnotes s
 * link: [https://hzwjmjimhr7bdmfv2doll4upibt5ojjmpo3pbp5ctwcg37n3hyk7qzid.onion/](https://hzwjmjimhr7bdmfv2doll4upibt5ojjmpo3pbp5ctwcg37n3hyk7qzid.onion/)
 * plain: `https://hzwjmjimhr7bdmfv2doll4upibt5ojjmpo3pbp5ctwcg37n3hyk7qzid.onion/`
 * proof: :lock: see tls/ssl certificate
-* check: <span title="attempts=6 code=903 exit=60 time=2022-04-09 20:55:33+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-09 09:20:06+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-08 17:30:27+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-08 05:30:10+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-07 14:03:40+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-07 02:03:24+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-06 22:38:24+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-06 10:39:08+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 19:12:35+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:13:22+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:46:29+00:00">:sos:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:46:44+00:00">:sos:</span><span title="attempts=1 code=200 exit=0 time=2022-04-03 12:17:20+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-03 00:16:23+00:00">:white_check_mark:</span>
+* check: <span title="attempts=6 code=903 exit=60 time=2022-04-09 20:55:33+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-09 09:20:06+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-08 17:30:27+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-08 05:30:10+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-07 14:03:40+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-07 02:03:24+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-06 22:38:24+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-06 10:39:08+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 19:12:35+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-05 07:13:22+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 15:46:29+00:00">:old_key:</span><span title="attempts=6 code=903 exit=60 time=2022-04-04 03:46:44+00:00">:old_key:</span><span title="attempts=1 code=200 exit=0 time=2022-04-03 12:17:20+00:00">:white_check_mark:</span><span title="attempts=1 code=200 exit=0 time=2022-04-03 00:16:23+00:00">:white_check_mark:</span>

 ### [DEF CON Groups](http://jrw32khnmfehvdsvwdf34mywoqj5emvxh4mzbkls6jk2cb3thcgz6nid.onion/)
 * transport: :small_red_triangle: **HTTP**
@ -1411,8 +1413,7 @@ These sites have apparently stopped responding.
 - :eight_spoked_asterisk: site up, and redirected to another page
 - :no_entry_sign: site up, but could not access the page
 - :stop_sign: site up, but reported a system error
- :sos: site returned no data, or is down, or curl experienced a transient
-  network error (may be a problem with the RWOS server connection)
+- :sos: site returned no data, or is down, or curl experienced a transient or permanent network error; may also reflect a problem with the RWOS server connection
 - :new: site is newly added, no data yet

 You can also see the [history of updates](https://github.com/alecmuffett/real-world-onion-sites/commits/master/README.md).
@ -1423,7 +1424,9 @@ Mouse-over the icons for details of HTTP codes, curl exit statuses,
 and the number of attempts made on each site.

 - codes [are from HTTP and are documented elsewhere](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes); RWOS-internal ones include:
-  - `901`, `902`, `903` - malformed HTTP response
+  - `901` - malformed HTTP response
+  - `902` - malformed HTTP response
+  - `903` - malformed HTTP response, commonly including (e.g.) invalid HTTPS certificate
  - `904` - HTTP status code parse error
  - `910` - connection timeout
 - exits [are from Curl and are documented elsewhere](https://curl.haxx.se/libcurl/c/libcurl-errors.html); common ones include:
--- a/rwos-db.py
+++ b/rwos-db.py
@ -39,6 +39,7 @@ EMOJI_4xx = ':no_entry_sign:'
 EMOJI_5xx = ':stop_sign:'
 EMOJI_DEAD = ':sos:'
 EMOJI_NO_DATA = ':new:'
+EMOJI_BAD_CERT = ':old_key:'

 H1 = '#'
 H2 = '##'
@ -77,7 +78,7 @@ VALUES (:run, :url, :attempt, :out, :err, :http_code, :curl_exit)
 '''

 SUMMARY_SQL = '''
-SELECT foo.ctime, foo.attempt, foo.http_code, foo.curl_exit
+SELECT foo.ctime, foo.attempt, foo.http_code, foo.curl_exit, foo.err
 FROM fetches foo
 INNER JOIN (
  SELECT url, run, MAX(attempt) AS pivot
@ -113,10 +114,23 @@ def extract_hcode(s): # static
        code = BADNESS + 4
    return code

+def placeholder(s):
+    if s == '': return PLACEHOLDER
+    if s == None: return PLACEHOLDER
+    return s
+
+def unicode_cleanup(x):
+    x = placeholder(x) # canonicalise blanks and None
+    if isinstance(x, str): # native python3 utf-8 string
+        result = x
+    else: # is byte array
+        result = x.decode('utf-8', 'ignore')
+    return result
+
 class Database:
    def __init__(self, filename):
        self.connection = sqlite3.connect(filename)
-        self.connection.text_factory = lambda x: unicode(x, UTF8, 'ignore') # ignore bad unicode shit
+        self.connection.text_factory = lambda x: unicode_cleanup(x)
        self.cursor = self.connection.cursor()
        self.cursor.executescript(SCHEMA_SQL)
        self.now = time.strftime('%Y%m%d%H%M%S', time.gmtime())
@ -187,11 +201,6 @@ class URL:
            if self.last_code < BADNESS: return
            time.sleep(RETRY_SLEEP)

-def placeholder(s):
-    if s == '': return PLACEHOLDER
-    if s == None: return PLACEHOLDER
-    return s
-
 def caps(s):
    return ' '.join([w.capitalize() for w in s.lower().split()])

@ -229,7 +238,8 @@ def get_summary(url):
    if len(rows) == 0:
        return ( EMOJI_NO_DATA, )
    result = []
-    for when, attempt, hcode, ecode in rows:
+    for when, attempt, hcode, ecode, errstr in rows:
+        errstr = unicode_cleanup(errstr) # THIS SHOULD NOT BE NEEDED, WHY? PERHAPS BECAUSE MULTI-LINE?
        emoji = EMOJI_UNSET
        if hcode >= 200 and hcode < 300:
            emoji = EMOJI_2xx
@ -240,7 +250,10 @@ def get_summary(url):
        elif hcode >= 500 and hcode < 600:
            emoji = EMOJI_5xx
        elif hcode >= BADNESS:
-            emoji = EMOJI_DEAD
+            if 'SSL certificate' in errstr:
+                emoji = EMOJI_BAD_CERT
+            else:
+                emoji = EMOJI_DEAD
        t = datetime.fromtimestamp(when, timezone.utc)
        result.append('<span title="attempts={1} code={2} exit={3} time={4}">{0}</span>'.format(emoji, attempt, hcode, ecode, t))
    return result