Git-Mirrors/PushshiftDumps
1
0
Fork 0
mirror of https://github.com/Watchful1/PushshiftDumps.git synced 2025-07-23 14:50:35 -04:00
Code Issues Projects Releases Packages Wiki Activity

Initial work on filter file

Browse source
This commit is contained in:
Watchful1 2023-03-02 18:41:50 -08:00
parent c7aa694631
commit 8dcc65abf7
2 changed files with 193 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
scripts/single_file.py
View file

@ -31,7 +31,6 @@ def read_lines_zst(file_name):
with open(file_name, 'rb') as file_handle:
buffer = ''
reader = zstandard.ZstdDecompressor(max_window_size=2**31).stream_reader(file_handle)
#reader.read(40000000000)
while True:
chunk = read_and_decode(reader, 2**27, (2**29) * 2)