Anant Shrivastava : Techno Enthusiast

Most of us at one point come to a conclusion that spam bot and Leachers are some of the annoyance of web which needs to be dealt  with severly.

so here is one way of doing this if you have Apache Webserver.

.htaccess is the file which can help you on this.

I have collected a list of some most common bot user agent strings used and based on these strings we can block them just by using .htaccess file.

if any of the below mentioned user agent crawls to our website it will get a Blank page.

Hope this list can help you all.

# Abuse Agent Blocking
RewriteEngine On  
<IfModule mod_rewrite.c>
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]  
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]  
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]  
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]  
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]  
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]  
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]  
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]  
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]  
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]  
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]  
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]  
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]  
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]  
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]  
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]  
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]  
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]  
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]  
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]  
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]  
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]  
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]  
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]  
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]  
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]  
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]  
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]  
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]  
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]  
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]  
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]  
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]  
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]  
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WWW-Mechanize [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]  
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]  
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]  
RewriteCond %{HTTP_USER_AGENT} ^Toata\ dragostea\ mea\ pentru\ diavola [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus  
RewriteRule ^.* – [F,L]
</IfModule>

# Abuse bot blocking rule end

Download the file directly from here

You just need to add  the file content at the bottom of the .htaccess file at the root of your website or if the file doesn’t exist then please rename this file as .htaccess

For more details on Apache Mod Re-write refer Apache – Mod-Rewrite (slideshare.net