Git-Mirrors/PushshiftDumps
1
0
Fork 0
mirror of https://github.com/Watchful1/PushshiftDumps.git synced 2025-07-23 14:50:35 -04:00
Code Issues Projects Releases Packages Wiki Activity

Initial implementation of multiprocess recompress script

Browse source
This commit is contained in:
Watchful1 2024-01-25 22:01:17 -08:00
parent 199a967888
commit 7727934db0
3 changed files with 355 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
scripts/filter_file.py
View file

@ -28,8 +28,8 @@ single_field = None
write_bad_lines = True
# only output items between these two dates
from_date = datetime.strptime("2022-01-01", "%Y-%m-%d")
to_date = datetime.strptime("2022-12-31", "%Y-%m-%d")
from_date = datetime.strptime("2005-01-01", "%Y-%m-%d")
to_date = datetime.strptime("2025-12-31", "%Y-%m-%d")
# the field to filter on, the values to filter with and whether it should be an exact match
# some examples: