diff --git a/README.md b/README.md index e83d012..edd9ff5 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,7 @@ This list of tools and software is intended to briefly describe some of the most * [ArchiveTools](https://github.com/recrm/ArchiveTools) - Collection of tools to extract and interact with WARC files (Python). +* [cdx-toolkit](https://pypi.org/project/cdx-toolkit/) - Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. *(Stable)* * [Go Get Crawl](https://github.com/karust/gogetcrawl) - Extract web archive data using [Wayback Machine](https://web.archive.org/) and [Common Crawl](https://commoncrawl.org/). *(Stable)* * [gowarcserver](https://github.com/nlnwa/gowarcserver) - [BadgerDB](https://github.com/dgraph-io/badger)-based capture index (CDX) and WARC record server, used to index and serve WARC files (Go). * [har2warc](https://github.com/webrecorder/har2warc) - Convert HTTP Archive (HAR) -> Web Archive (WARC) format (Python). @@ -161,7 +162,7 @@ This list of tools and software is intended to briefly describe some of the most * [Sparkling](https://github.com/internetarchive/Sparkling) - Internet Archive's Sparkling Data Processing Library. *(Stable)* * [Unwarcit](https://github.com/emmadickson/unwarcit) - Command line interface to unzip WARC and WACZ files (Python). * [Warcat](https://github.com/chfoo/warcat) - Tool and library for handling Web ARChive (WARC) files (Python). *(Stable)* -* [warcio](https://github.com/webrecorder/warcio) - Streaming WARC/ARC library for fast web archive IO (Python). +* [warcio](https://github.com/webrecorder/warcio) - Streaming WARC/ARC library for fast web archive IO (Python). *(Stable)* * [warctools](https://github.com/internetarchive/warctools) - Library to work with ARC and WARC files (Python). * [webarchive](https://github.com/richardlehane/webarchive) - Golang readers for ARC and WARC webarchive formats (Golang). @@ -175,6 +176,7 @@ This list of tools and software is intended to briefly describe some of the most * [Common Crawl Web Graph](https://commoncrawl.org/category/web-graph/) - A host or domain-level graph of the web, with ranking information. *(Stable)* * [Common Crawl Jupyter notebooks](https://github.com/commoncrawl/cc-notebooks) - A collection of notebooks using Common Crawl's various datasets. *(Stable)* * [Tweet Archvies Unleashed Toolkit](https://github.com/archivesunleashed/twut) - An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark. *(In Development)* +* [Web Data Commons](http://webdatacommons.org/) - Structured data extracted from Common Crawl. *(Stable)* ### Quality Assurance @@ -215,6 +217,7 @@ This list of tools and software is intended to briefly describe some of the most ### Mailing Lists +* [Common Crawl](https://groups.google.com/g/common-crawl) * [IIPC](http://netpreserve.org/about-us/iipc-mailing-list/) * [OpenWayback](https://groups.google.com/g/openwayback-dev) * [WASAPI](https://groups.google.com/g/wasapi-community)