From bd4e52fab60b04c9e56197150e5c316933585588 Mon Sep 17 00:00:00 2001 From: nil0x42 Date: Sun, 13 Oct 2019 20:15:10 +0000 Subject: [PATCH] Add duplicut, C Tool to remove duplicates, without changing the order, and without getting OOM on huge wordlists. duplicut is a C tool, highly optimized for a single task: Removing duplicate entries from a wordlist, without changing the order, and without getting OOM on huge wordlists whose size exceeds available memory. It's trivial to remove duplicates by sorting, but duplicut is the only tool capable of removing them without changing the order, to assist the creation of statictically optimized wordlists for password cracking purposes. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 999e5ea..ffc779f 100644 --- a/README.md +++ b/README.md @@ -303,6 +303,7 @@ See also [DEF CON Suggested Reading](https://www.defcon.org/html/links/book-list * [BruteForce Wallet](https://github.com/glv2/bruteforce-wallet) - Find the password of an encrypted wallet file (i.e. `wallet.dat`). * [CeWL](https://digi.ninja/projects/cewl.php) - Generates custom wordlists by spidering a target's website and collecting unique words. +* [duplicut](https://github.com/nil0x42/duplicut) - Quickly remove duplicates, without changing the order, and without getting OOM on huge wordlists. * [GoCrack](https://github.com/fireeye/gocrack) - Management Web frontend for distributed password cracking sessions using hashcat (or other supported tools) written in Go. * [Hashcat](http://hashcat.net/hashcat/) - The more fast hash cracker. * [hate_crack](https://github.com/trustedsec/hate_crack) - Tool for automating cracking methodologies through Hashcat.