WebDevVote - Blocking not Robots.txt Compliant Crawlers
2votes
voted
Blocking not Robots.txt Compliant Crawlers Blocking not Robots.txt Compliant Crawlers
ethilik ethilik 1 year, 3 months ago
blog.sb2.fr — Today an article talking about blocking Crawlers which are not Robots.txt Compliant. Number of web sites grows exponentially day by day and crawlers too ! But there is a lot of Web Crawlers that don't follow Robots.txt Disallow Rules or totally ignore it ... original link
| 63 views | get vote counter code
| add tag

new Add a live vote counter to your blog >> liveImage

You can even customize the image by choosing your own colors, and then clicking the button below to update the preview and the html code:

  • "Vote it" text
  • "Vote it" Background
  • Vote count text
  • Vote count background
  • Border

Simply copy and paste this HTML into your blog post.

Users who voted this story

ethilik ethilik - oliver oliver

Comments

No comments so far!



information Thank you for your visit. Please create an account to share and vote!

Login or Sign up to add your comment.
  • Web Dev Vote's Facebook Profile
  • Follow us on Twitter
  • RSS



Technorati Profile