Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-04-19 07:15:52 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

robots.txt for testing

Browse Source
This commit is contained in:
Noah Levitt 2016-11-16 12:12:17 -08:00
parent 3aead6de93
commit 24cc8377fb
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tests/htdocs/robots.txt Normal file
View File

@ -0,0 +1,2 @@
User-agent: badbot
Disallow: /