Git-Mirrors/PushshiftDumps
1
0
Fork 0
mirror of https://github.com/Watchful1/PushshiftDumps.git synced 2025-07-23 23:00:40 -04:00
Code Issues Projects Releases Packages Wiki Activity

Clean up

Browse source
This commit is contained in:
Watchful1 2021-09-09 22:24:14 -07:00
parent bd7378ff91
commit dd12687141
4 changed files with 40 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

4
scripts/iterate_folder.py
View file

@ -1,3 +1,7 @@
# this is an example of iterating over all zst files in a single folder,
# decompressing them and reading the created_utc field to make sure the files
# are intact. It has no output other than the number of lines
import zstandard
import os
import json