I am trying to figure out what I am doing wrong. My website is not being indexed by Bing/Yahoo and I seem to be stuck in the middle of two companies.. Here is the communications from Bing..
Good day to you!
Thank you for contacting Bing Technical Support. My name is Rizzalyn and I will be assisting you today.
You have mentioned that your website is not being indexed in Bing and that you have your robots.txt in your root directory. You also have mentioned that according to Webmaster Tools, the robots.txt cannot be downloaded. Allow me to discuss this with you.
This issue is caused by the current instance of Bingbot not being able to fetch the robots.txt file itself. This could be due to a server-side IP block or some other configuration that prevents (at least some of) our Bingbot machines to successfully retrieve this file.
You might be able to repro this by using Fetch as Bingbot on the robots.txt file itself (http://www.fordthunderbird.net/robots.txt). When checking internally this fetch request simply expires, meaning that our Bingbot crawler is not getting a valid response back within a reasonable amount of time. When we cannot get a valid response for robots.txt (such as 200 or 404), we cannot proceed with crawling the website, which is why the other fetches are failing. We recommend you check for server-side code or configuration settings - especially checking for IPs ranges - that may be the root cause for this. Since we appear to have successfully fetched and indexed pages from your site previously, the issue may not affect all of our machine IPs.
I hope this information helps. Should you have further questions regarding this matter, please let us know.
Thank you and Have a nice day!
Bing Technical Support
Here is what I got back from Dave at WHB
Thank you for contacting us. We have investigated the problem you have encountered thoroughly. Robot.txt file you have created is absolutely correct.
If you need any additional information on how robot.txt file can be built, please check our fairly concise wiki article: http://www.webhostingbuzz.com/wiki/u...ff/robots-txt/
Most likely Bing IP range was blocked earlier. As we can see, Bing bots' IP range was removed from the temporary block on the 29th of January 2013. Please check that:
- root@rs17 [/home/thunder/public_html]# zgrep 157.55 /var/log/lfd.log*
/var/log/lfd.log.1.gz:Jan 29 07:17:29 rs17 lfd: Incoming IP 184.108.40.206/16 temporary block removed
- root@rs17 [/home/thunder/public_html]# csf -g 157.55.16
Chain num pkts bytes target prot opt in out source destination
No matches found for 157.55.16 in iptables
So, Bing bots are not blocked by our server firewall rules. We kindly recommend you to contact Bing support once again and provide them with the information from this ticket. It may be useful for them to perform in-depth investigation on their end.
If you have any questions, don't hesitate to contact us for further assistance (we provide our customers with 24x7 customer and technical support and you can reach us in the Live Chats, tickets and over the phone).
Level 2 Technical Team
I did send the above to Bing but haven't had a response yet. What am I doing wrong? I am losing a bit of traffic and revenue from my site because Google is the only place to find me. I guess if I had a choice, I would rather be on Google, but my content is worth being put out on both search engines.
Has any other webmaster experienced this? Why would WHB block BingBot to begin with? I get that there are bad bots out there, but blocking it completely isn't cool.