A robots.txt file is the means by which any administrator of a Web page tells search engines what the different files can crawl, index.
The truth is that originally few Web publishers who know the existence of this powerful, and important tool. So that, over time, have been aware that search engines like Google or Yahoo! end up indexing documents and files private theoretically, hosted on the server and should not have publicly indexed.
If we consider that the document robots.txt first file is the appropriate one when accessing robots and introduced into a particular Web page, there is no doubt that having a good robots.txt is simply critical to save headaches and, why not maintain adequate communication with search engines and enjoy an optimal configuration.
For this reason, on a previous occasion I explained how to use the robots.txt file, Telling you some important elements, such as the different codes used to specify the search engines by which folders or files can "move", which should be indexed and which not.
Now have known it is essential that the robots.txt never exceed 500KB Because, apparently, Google has a limit not to prosecute those files exceed this weight. And, of course, we run the risk that GoogleBot is left at the gates of our server and do not access or indexing our content.
The truth is that originally few Web publishers who know the existence of this powerful, and important tool. So that, over time, have been aware that search engines like Google or Yahoo! end up indexing documents and files private theoretically, hosted on the server and should not have publicly indexed.
If we consider that the document robots.txt first file is the appropriate one when accessing robots and introduced into a particular Web page, there is no doubt that having a good robots.txt is simply critical to save headaches and, why not maintain adequate communication with search engines and enjoy an optimal configuration.
For this reason, on a previous occasion I explained how to use the robots.txt file, Telling you some important elements, such as the different codes used to specify the search engines by which folders or files can "move", which should be indexed and which not.
Now have known it is essential that the robots.txt never exceed 500KB Because, apparently, Google has a limit not to prosecute those files exceed this weight. And, of course, we run the risk that GoogleBot is left at the gates of our server and do not access or indexing our content.
+ comments + 1 comments
Hey,
very nice blog!I’m an instant fan, I have bookmarked you and I’ll be checking back on a regular.See u.
Cruciate ligament reconstruction
Post a Comment