WebHackersWeapons/categorize/tags/crawl.md

## Tools for crawl

| Type | Name | Description | Star | Tags | Badges |
| --- | --- | --- | --- | --- | --- |
|Army-Knife|[BurpSuite](https://portswigger.net/burp)|The BurpSuite Project||[`mitmproxy`](/categorize/tags/mitmproxy.md) [`live-audit`](/categorize/tags/live-audit.md) [`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)![burp](/images/burp.png)[![Java](/images/java.png)](/categorize/langs/Java.md)|
|Army-Knife|[ZAP](https://github.com/zaproxy/zaproxy)|The ZAP core project|![](https://img.shields.io/github/stars/zaproxy/zaproxy?label=%20)|[`mitmproxy`](/categorize/tags/mitmproxy.md) [`live-audit`](/categorize/tags/live-audit.md) [`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)![zap](/images/zap.png)[![Java](/images/java.png)](/categorize/langs/Java.md)|
|Recon|[Photon](https://github.com/s0md3v/Photon)|Incredibly fast crawler designed for OSINT. |![](https://img.shields.io/github/stars/s0md3v/Photon?label=%20)|[`osint`](/categorize/tags/osint.md) [`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Python](/images/python.png)](/categorize/langs/Python.md)|
|Recon|[gospider](https://github.com/jaeles-project/gospider)|Gospider - Fast web spider written in Go |![](https://img.shields.io/github/stars/jaeles-project/gospider?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Go](/images/go.png)](/categorize/langs/Go.md)|
|Recon|[katana](https://github.com/projectdiscovery/katana)|A next-generation crawling and spidering framework.|![](https://img.shields.io/github/stars/projectdiscovery/katana?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Go](/images/go.png)](/categorize/langs/Go.md)|
|Recon|[cariddi](https://github.com/edoardottt/cariddi)|Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more|![](https://img.shields.io/github/stars/edoardottt/cariddi?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Go](/images/go.png)](/categorize/langs/Go.md)|
|Recon|[hakrawler](https://github.com/hakluke/hakrawler)|Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application |![](https://img.shields.io/github/stars/hakluke/hakrawler?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Go](/images/go.png)](/categorize/langs/Go.md)|
|Recon|[crawlergo](https://github.com/Qianlitp/crawlergo)|A powerful browser crawler for web vulnerability scanners|![](https://img.shields.io/github/stars/Qianlitp/crawlergo?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![Go](/images/go.png)](/categorize/langs/Go.md)|
|Utils|[security-crawl-maze](https://github.com/google/security-crawl-maze)|Security Crawl Maze is a comprehensive testbed for web security crawlers. It contains pages representing many ways in which one can link resources from a valid HTML document.|![](https://img.shields.io/github/stars/google/security-crawl-maze?label=%20)|[`crawl`](/categorize/tags/crawl.md)|![linux](/images/linux.png)![macos](/images/apple.png)![windows](/images/windows.png)[![HTML](/images/html.png)](/categorize/langs/HTML.md)|