Website not getting any traffic from Google, Yahoo or BingÂ
Forever promoting your site but not seeing the fruition of your marketing work?
You need to implement Search Engine Optimization (SEO)
Search Engine Optimization (SEO) is at the core of any domain that ranks well on Google.
Ensuring your site has all the necessary files and scripts is the initial part of your endeavor.
Let's discuss the robots.txt.
What is the robots.txt?
Robots.txt is a page that any crawler will access in order to understand how you want your site to be displayed in results.
It is uploaded to your website as a file so it can be accessed as a page by a web crawler i.e www.yourwebsitename/sitemap.xml
The script has many functions
This is what the script will generally look like
user-agent: *
Allow: /
What is written here has told all search engine (SE) crawlers that there isn't any pages on this website that have sensitive material or inappropriate content.
What we have basically done is said that we want every page on this site listed in its index.
Now that may sound confusing without understanding how the script works so let's break it down a bit.
user-agent: *
This is where you say which search engines this robots.txt will effect. Entering Googlebot, Bingbot etc. in this field will limit that particular SE and there crawlers specifically, any other crawler or SE will not be effected.
Placing the * means that you want every search engine to adhere to this conduct.
allow:
Pretty self explanatory this one. This is where you say what is allowed for display.
Placing the / here means everything is allowed and nothing should be disregarded.
Alternatively you can use the disallow attribute like so..
disallow: /
This would disallow every page on your website.
Appending extra on the end of the / can limit it to sub-directories such as images, videos and URL's.
For example. If a website wanted to disregard a contact us page they would do it like this.
disallow: /contact-us.html
Removing images is a little bit different as we actually place the search engines image crawler bot in the user-agent, however it is still the same concept.
Modifying the user-agent to not crawl images is done like so.
user-agent: Googlebot-Image, Bingbot-Image etc...
disallow: /
This will remove all images displayed in a website from that search engines index.
To remove an image from a certain page that is indexed is now similar to the previous example.
user-agent: Googlebot-Images, Bingbot-Image etc...
disallow: /images/nissanskyline.jpg
This will remove a specific image from your website.
In a way we can also optimize a robots script.
It is good to place your sitemap in your robots.txt script to let the search engine know that you have one.
Place your sitemap at the bottom like so.
user-agent: *
allow: /
sitemap: http://www.yoursite.com/sitemap.xml
Simply put, adding your sitemap location in your robots.txt is good manners to a search engine crawler.
When saving your file it has to be saved with this exact name without the quotation marks.. "robots.txt"
It then it needs to be uploaded to your website as a page so a crawler can access it.
For further DIY SEO help, links can be found in the description.
Author: Martin Heeremans
Forever promoting your site but not seeing the fruition of your marketing work?
You need to implement Search Engine Optimization (SEO)
Search Engine Optimization (SEO) is at the core of any domain that ranks well on Google.
Ensuring your site has all the necessary files and scripts is the initial part of your endeavor.
Let's discuss the robots.txt.
What is the robots.txt?
Robots.txt is a page that any crawler will access in order to understand how you want your site to be displayed in results.
It is uploaded to your website as a file so it can be accessed as a page by a web crawler i.e www.yourwebsitename/sitemap.xml
The script has many functions
This is what the script will generally look like
user-agent: *
Allow: /
What is written here has told all search engine (SE) crawlers that there isn't any pages on this website that have sensitive material or inappropriate content.
What we have basically done is said that we want every page on this site listed in its index.
Now that may sound confusing without understanding how the script works so let's break it down a bit.
user-agent: *
This is where you say which search engines this robots.txt will effect. Entering Googlebot, Bingbot etc. in this field will limit that particular SE and there crawlers specifically, any other crawler or SE will not be effected.
Placing the * means that you want every search engine to adhere to this conduct.
allow:
Pretty self explanatory this one. This is where you say what is allowed for display.
Placing the / here means everything is allowed and nothing should be disregarded.
Alternatively you can use the disallow attribute like so..
disallow: /
This would disallow every page on your website.
Appending extra on the end of the / can limit it to sub-directories such as images, videos and URL's.
For example. If a website wanted to disregard a contact us page they would do it like this.
disallow: /contact-us.html
Removing images is a little bit different as we actually place the search engines image crawler bot in the user-agent, however it is still the same concept.
Modifying the user-agent to not crawl images is done like so.
user-agent: Googlebot-Image, Bingbot-Image etc...
disallow: /
This will remove all images displayed in a website from that search engines index.
To remove an image from a certain page that is indexed is now similar to the previous example.
user-agent: Googlebot-Images, Bingbot-Image etc...
disallow: /images/nissanskyline.jpg
This will remove a specific image from your website.
In a way we can also optimize a robots script.
It is good to place your sitemap in your robots.txt script to let the search engine know that you have one.
Place your sitemap at the bottom like so.
user-agent: *
allow: /
sitemap: http://www.yoursite.com/sitemap.xml
Simply put, adding your sitemap location in your robots.txt is good manners to a search engine crawler.
When saving your file it has to be saved with this exact name without the quotation marks.. "robots.txt"
It then it needs to be uploaded to your website as a page so a crawler can access it.
For further DIY SEO help, links can be found in the description.
Author: Martin Heeremans