The `www.parrotsec.org` Web server has moved behind CloudFlare, which
returns an HTTP 503 error for the `awesome_bot` Web crawler, causing a
build failure. The domain is live and still works, but the CloudFlare
WAF does not permit the crawler to verify this.
In this commit, the primary change is alphabetizing both the table of
contents as well as the line items for each category. This is done in
order to make it easier for readers to locate their desired information
with their naked eyes. The list is long, and as such should at least
have a consistent scheme for ordering the items within it.
Alphabetization also side-steps the issue of favoritism since the sort
order is lexicographical.
Additionally, this commit changes several headings to more clearly
describe its contents. For example, most of the subheadings under the
"Online Resources" category have been renamed to "Online [Topic]
Resources", where "[Topic]" was the old heading. Similarly, I split the
Docker Container section into two, one for distros and for tools, since
the previous section muddled those two distinct categories of containers
together. (The main list does not do this, so that was anomalous.)
Another major change is the removal of the top-level "Tools" section.
This section had clearly become a catch-all and also prevented us from
being able to use sub-headings to more intelligently categorize the
individual tools without running afoul of the Awesome List guidelines
that restrict us to one level of subheading per category. This continues
the work that was begun in #290 of moving, e.g., the "Network Tools"
section to its own top-level heading.
Further, I have removed several tools that are strictly either forensics
or malware analysis utilities, such as cuckoo sandbox. I feel that this
more accurately aligns this list with its stated purpose: Penetration
Testing. While related, listing forensics of malware analysis tools that
cannot also be used for vulnerability discovery or exploit development
seems like an invitation to suffer from scope creep. Instead of listing
those tools directly, I have therefore added "See also" lines with links
to more appropriate places (often other Awesome lists) for their topic.
Finally, several links were upgraded from their listed HTTP to HTTPS
versions, after I confirmed that those Web servers did indeed respond to
TLS requests. I also removed `www.defcon.org` from the `awesome_bot`'s
white list, since that link works just fine for me as well.
This commit also adds GhostProject.fr to the whitelist, as they use
CloudFlare's JavaScript DDoS detection and return an HTTP 503 error.
This commit also removes `zoomeye.org` from the whitelist,
because they seem to be returning HTTP 200 OK responses reliably now.
This commit fixes numerous CI build issues related to stale or broken
links. These include:
* Removal of Zoom username enumeration tool, covered by WPScan anyway.
* Removal of old Google dork database that is unmaintained/has vanished.
* Removal of `OSVDB.org` zone, which no longer resolves via DNS.
* Fix link to NoSQLmap tool (domain expired, use GitHub.com link now).
* Update link to Social Engineering in IT book from legacy URL.
* Update link to OWASP's AppSecUSA conference; now uses second-level domain.
Further, this commit simplifies the `.travis.yml` file in order to use a
plainer (more standard) certificates bundle. Two URLs have been added to
the whitelist: `www.shodan.io`, which returns a 403 Forbidden error when
accessed by Awesome Bot, and `www.mhprofessional.com`, which generates
an SSLv3 certificate validation error.
Prior to this commit, a custom SSL certificate bundle was generated and
then placed in the `/tmp` directory for use, but this is no longer
required as the latest `ca-certificates` bundle shipped with Ubuntu
contains the root certificates needed for the domains that once required
this custom bundle to be used.
* Add CVE List to Vulnerability Databases section, since it was missing.
* Style guide compliance pass focused on Vulnerability Databases section.
* Whitelist the Inj3ct0r URLs.
The `0day.today` website sits behind an extremely aggressive Cloudflare
anti-bot checker, which causes `awesome-bot` to trigger an HTTP 503
response. This fails the build but is actually normal behavior.
Similarly, the Onion service is inaccessible except over Tor and our
Travis CI configuration does not (yet?) support checking Onion service
links. (Although, perhaps it should be updated to do so in a future PR.)
