This is the first post in our two-part AdSense Crawler Errors series.
There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there's another straightforward optimization tip that many publishers often overlook?
A bit of context
Your site's robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don't have permission are not able to view or index specific sections of your site, depending on what you've specified.
AdSense ads are displayed through the use of an AdSense web crawler. That crawler scans your page's content and determines which ads to display, according to specific keywords. If our AdSense crawler is being blocked by your robots.txt file, we're going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.
How you can help yourself
View the contents of your robots.txt file by going to [yourdomain.com]/robots.txt. (If you have a subdomain, it likely has a robots.txt file as well, located at [sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:
If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on 'Account Settings' from the home page. From there, click on 'View errors' under 'Access and Authorization.'
Stay tuned for the second post in our AdSense Crawler Errors series, where we'll cover crawler login issues and how you can solve them.
Posted by Andrew Boni - Inside AdSense Team
There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there's another straightforward optimization tip that many publishers often overlook?
A bit of context
Your site's robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don't have permission are not able to view or index specific sections of your site, depending on what you've specified.
AdSense ads are displayed through the use of an AdSense web crawler. That crawler scans your page's content and determines which ads to display, according to specific keywords. If our AdSense crawler is being blocked by your robots.txt file, we're going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.
How you can help yourself
View the contents of your robots.txt file by going to [yourdomain.com]/robots.txt. (If you have a subdomain, it likely has a robots.txt file as well, located at [sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:
User-agent: Mediapartners-GoogleThis will ensure that our AdSense ad crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. Please note that making this change will not impact your Google search rankings. Adding these two lines to your robots.txt file will only help to deliver better, more relevant ads to pages with AdSense code already on them. Pages that don't have AdSense ad code will not be affected.
Disallow:
If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on 'Account Settings' from the home page. From there, click on 'View errors' under 'Access and Authorization.'
Posted by Andrew Boni - Inside AdSense Team
--
Posted By Inside AdSense Team to Inside AdSense at 11/17/2011 09:00:00 AM --
Unable to view the links or images in the message above? Want to comment on this post? Visit http://adsense.blogspot.com/ .
You received this message because you are subscribed to the Google Groups "Inside AdSense" group.
To unsubscribe from this group, send email to Inside-AdSense+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/Inside-AdSense
Hеllο would you minԁ stating which
ReplyDeleteblog plаtform you're working with? I'm goіng tο start my own blog in the neаr future but
I'm having a tough time deciding between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your design and style seems different then most blogs and I'm looκing foг
something unique. Ρ.S Apologies foг getting οff-tοpic
but I had to ask!
my web blog ... reputation management
Hi there, just became alert to your blog through Google,
ReplyDeleteand found that it's truly informative. I am gonna watch out for brussels. I'll be
grateful if you continue this in future. A lot of people will be benefited from your writing.
Cheers!
My web page ... パーツ