Awesome-Mirrors
/
awesome-web-archiving
mirror of https://github.com/iipc/awesome-web-archiving.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: Awesome-Mirrors:82b0b4c90d23700517ae9a5833fe03c5ab4aaf05
Branches Tags
Awesome-Mirrors:main
...
compare: Awesome-Mirrors:0bf399b6957a4ba041ced5375b7bd441f189d91a
Branches Tags
Awesome-Mirrors:main

3 Commits
82b0b4c90d ... 0bf399b695

Author SHA1 Message Date
Andy Jackson 0bf399b695
Merge branch 'main' into add-cc-mailing-list 2023-07-04 09:34:57 +01:00
Greg Lindahl bf9664ff45
add web data commons (#137) 
Co-authored-by: Greg Lindahl <greg@commomncrawl.org>
Co-authored-by: Andy Jackson <Andrew.Jackson@bl.uk>
 2023-07-04 09:34:33 +01:00
Greg Lindahl 54110410bf
warcio was stable a long time ago (#134) 
Co-authored-by: Greg Lindahl <greg@commomncrawl.org>
 2023-07-04 09:33:09 +01:00
1 changed files with 2 additions and 1 deletions
Show Stats Expand all files Collapse all files

3
README.md
View File

@ -161,7 +161,7 @@ This list of tools and software is intended to briefly describe some of the most
* [Sparkling](https://github.com/internetarchive/Sparkling) - Internet Archive's Sparkling Data Processing Library. *(Stable)*
* [Unwarcit](https://github.com/emmadickson/unwarcit) - Command line interface to unzip WARC and WACZ files (Python).
* [Warcat](https://github.com/chfoo/warcat) - Tool and library for handling Web ARChive (WARC) files (Python). *(Stable)*
* [warcio](https://github.com/webrecorder/warcio) - Streaming WARC/ARC library for fast web archive IO (Python).
* [warcio](https://github.com/webrecorder/warcio) - Streaming WARC/ARC library for fast web archive IO (Python). *(Stable)*
* [warctools](https://github.com/internetarchive/warctools) - Library to work with ARC and WARC files (Python).
* [webarchive](https://github.com/richardlehane/webarchive) - Golang readers for ARC and WARC webarchive formats (Golang).
@ -172,6 +172,7 @@ This list of tools and software is intended to briefly describe some of the most
* [Archives Unleashed Notebooks](https://github.com/archivesunleashed/notebooks) - Notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit. *(Stable)*
* [Archives Unleashed Toolkit](https://github.com/archivesunleashed/aut) - Archives Unleashed Toolkit (AUT) is an open-source platform for analyzing web archives with Apache Spark. *(Stable)*
* [Tweet Archvies Unleashed Toolkit](https://github.com/archivesunleashed/twut) - An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark. *(In Development)*
* [Web Data Commons](http://webdatacommons.org/) - Structured data extracted from Common Crawl. *(Stable)*
### Quality Assurance