Awesome-Mirrors
/
awesome-web-archiving
mirror of https://github.com/iipc/awesome-web-archiving.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add cdx-toolkit (#135) 

Co-authored-by: Greg Lindahl <greg@commomncrawl.org>
Co-authored-by: Andy Jackson <Andrew.Jackson@bl.uk>
This commit is contained in:
Greg Lindahl 2023-07-04 01:37:05 -07:00 committed by GitHub
parent d395bb1b44
commit d4673d008e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@ -129,6 +129,7 @@ This list of tools and software is intended to briefly describe some of the most
* [ArchiveTools](https://github.com/recrm/ArchiveTools) - Collection of tools to extract and interact with WARC files (Python).
<!--lint ignore double-link-->
* [cdx-toolkit](https://pypi.org/project/cdx-toolkit/) - Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. *(Stable)*
* [Go Get Crawl](https://github.com/karust/gogetcrawl) - Extract web archive data using [Wayback Machine](https://web.archive.org/) and [Common Crawl](https://commoncrawl.org/). *(Stable)*
* [gowarcserver](https://github.com/nlnwa/gowarcserver) - [BadgerDB](https://github.com/dgraph-io/badger)-based capture index (CDX) and WARC record server, used to index and serve WARC files (Go).
* [har2warc](https://github.com/webrecorder/har2warc) - Convert HTTP Archive (HAR) -> Web Archive (WARC) format (Python).