Results 1 to 9 of 9

Thread: Bingbot being blocked??

  1. #1
    CometGSR is offline New Bee
    Join Date
    Dec 2009
    Posts
    7
    WHB Points this Month
    0.00
    WHB Points
    0.00
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default Bingbot being blocked??

    I am trying to figure out what I am doing wrong. My website is not being indexed by Bing/Yahoo and I seem to be stuck in the middle of two companies.. Here is the communications from Bing..

    ---

    Good day to you!

    Thank you for contacting Bing Technical Support. My name is Rizzalyn and I will be assisting you today.

    You have mentioned that your website is not being indexed in Bing and that you have your robots.txt in your root directory. You also have mentioned that according to Webmaster Tools, the robots.txt cannot be downloaded. Allow me to discuss this with you.


    This issue is caused by the current instance of Bingbot not being able to fetch the robots.txt file itself. This could be due to a server-side IP block or some other configuration that prevents (at least some of) our Bingbot machines to successfully retrieve this file.

    You might be able to repro this by using Fetch as Bingbot on the robots.txt file itself (http://www.fordthunderbird.net/robots.txt). When checking internally this fetch request simply expires, meaning that our Bingbot crawler is not getting a valid response back within a reasonable amount of time. When we cannot get a valid response for robots.txt (such as 200 or 404), we cannot proceed with crawling the website, which is why the other fetches are failing. We recommend you check for server-side code or configuration settings - especially checking for IPs ranges - that may be the root cause for this. Since we appear to have successfully fetched and indexed pages from your site previously, the issue may not affect all of our machine IPs.


    I hope this information helps. Should you have further questions regarding this matter, please let us know.

    Thank you and Have a nice day!

    Best regards,

    Rizzalyn
    Bing Technical Support
    ----

    Here is what I got back from Dave at WHB
    -

    Hello James,

    Thank you for contacting us. We have investigated the problem you have encountered thoroughly. Robot.txt file you have created is absolutely correct.

    If you need any additional information on how robot.txt file can be built, please check our fairly concise wiki article: http://www.webhostingbuzz.com/wiki/u...ff/robots-txt/

    Most likely Bing IP range was blocked earlier. As we can see, Bing bots' IP range was removed from the temporary block on the 29th of January 2013. Please check that:

    - root@rs17 [/home/thunder/public_html]# zgrep 157.55 /var/log/lfd.log*
    /var/log/lfd.log.1.gz:Jan 29 07:17:29 rs17 lfd[221597]: Incoming IP 157.55.0.0/16 temporary block removed

    - root@rs17 [/home/thunder/public_html]# csf -g 157.55.16
    Chain num pkts bytes target prot opt in out source destination
    No matches found for 157.55.16 in iptables

    So, Bing bots are not blocked by our server firewall rules. We kindly recommend you to contact Bing support once again and provide them with the information from this ticket. It may be useful for them to perform in-depth investigation on their end.

    If you have any questions, don't hesitate to contact us for further assistance (we provide our customers with 24x7 customer and technical support and you can reach us in the Live Chats, tickets and over the phone).

    Sincerely,
    -----------------------------------

    Regards,

    Dave S.,

    Level 2 Technical Team

    WebHostingBuzz

    ----------------------------------

    ----

    I did send the above to Bing but haven't had a response yet. What am I doing wrong? I am losing a bit of traffic and revenue from my site because Google is the only place to find me. I guess if I had a choice, I would rather be on Google, but my content is worth being put out on both search engines.

    Has any other webmaster experienced this? Why would WHB block BingBot to begin with? I get that there are bad bots out there, but blocking it completely isn't cool.

    James

  2. #2
    Tony's Avatar
    Tony is offline Bad Influence
    Join Date
    Apr 2007
    Location
    West Virginia Blue Ridge
    Posts
    654
    Servers
    SS3
    WHB Points this Month
    0.00
    WHB Points
    200.00
    Thanks
    2
    Thanked 10 Times in 10 Posts

    Default

    No clue. On our small site AWStats show "msnbot" instead of "bingbot" but it seems to be doing the job. It shows a small percentage of the hits googlebot does, but a search at Bing shows us right at the top. Of course, that's from my own browser - is there a way to search Bing from an unassociated browser?

    edit: You didn't say much about your site (tl dr?), but could you have an add-on or plugin doing this? Our Google numbers plummeted a couple months back when one of ours started blocking googlebot!
    Last edited by Tony; 02-05-2013 at 02:41 PM.

  3. #3
    CometGSR is offline New Bee
    Join Date
    Dec 2009
    Posts
    7
    WHB Points this Month
    0.00
    WHB Points
    0.00
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    If you are asking about my domain name (TL), its FordThunderbird.NET V8 Ford 1995 1996 2002 2007 1990 1992 Specs Thunderbird Mercury Sable Ranger Probe Taurus Mazda5 Mazda CX9 . My domian register is GoDaddy.com. My Robots.txt is at http://www.fordthunderbird.net/robots.txt . I have tried everything I know to resolve this. It has to be something on WHB side. The site itself is pretty much coded by hand with a couple of freeware add-ins to help because I am lazy. I can't see why Bing would puke on it.. It can't get any simpler than this.

  4. #4
    Tony's Avatar
    Tony is offline Bad Influence
    Join Date
    Apr 2007
    Location
    West Virginia Blue Ridge
    Posts
    654
    Servers
    SS3
    WHB Points this Month
    0.00
    WHB Points
    200.00
    Thanks
    2
    Thanked 10 Times in 10 Posts

    Default

    Actually I think I was more asking if you see bots from MSN or Bing hitting your site at all. In our case, once I started looking for the trouble it was fairly obvious in AWStats as the googlebot was generating errors when it tried to connect.

    Or maybe I'm just misunderstanding the question. I was reading it as an SEO question, but on re-reading it you mention "indexing", not "ranking". I imagine a busy topic like tbirds would require a really busy site to rank in the top 1000's.

    I'm having no trouble reading your robots.txt file so there's no reason Bing shouldn't be able to. I think their answer is misleading.

  5. The Following User Says Thank You to Tony For This Useful Post:

    CometGSR (02-06-2013)

  6. #5
    CometGSR is offline New Bee
    Join Date
    Dec 2009
    Posts
    7
    WHB Points this Month
    0.00
    WHB Points
    0.00
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    I was trying to use their webmaster tools. There is an option that says "Fetch As BingBot" and it says that it can't read the robots.txt file. Its right there, even Bing's customer service sees that. I am not indexed on Bing at all, nothing is being seen on Bing.

    It was interesting that WHB is commenting on my robots.txt file and its creation. I used the creation tool because my hard coded one wasn't working with Bing.

    I will check AWStats to see if I see anything from Bing.

    Does WHB monitor this forum?
    Last edited by CometGSR; 02-06-2013 at 12:05 AM.

  7. #6
    Vitaliy K.'s Avatar
    Vitaliy K. is offline WeeHBie
    Join Date
    Apr 2010
    Posts
    567
    Servers
    localhost
    WHB Points this Month
    0.00
    WHB Points
    60.00
    Thanks
    5
    Thanked 4 Times in 4 Posts

    Default

    Hello James,

    Yes, these forums are monitored by our staff, but we don't provide technical support through this channel. If you need any assistance from our technical staff, please update your ticket at the helpdesk.
    Vitalij K.
    Level 2 Technical Team Leader
    WebHostingBuzz.com

    Help WebHostingBuzz today. Like us on Facebook, follow us on Twitter or Google +1 us.
    Visit http://www.webhostingbuzz.com and look for the social media icons at the top of the page.

  8. #7
    Tony's Avatar
    Tony is offline Bad Influence
    Join Date
    Apr 2007
    Location
    West Virginia Blue Ridge
    Posts
    654
    Servers
    SS3
    WHB Points this Month
    0.00
    WHB Points
    200.00
    Thanks
    2
    Thanked 10 Times in 10 Posts

    Default

    I just sat down and set up Bing Webmaster Tools and checked my own site, and it's fetching normally, robots.txt (chmod 644 - http://www.shannondale.org/robots.txt) and all. Are you sure you set up the site correctly in BWT? I know I'm grasping at straws here but maybe something like a double slash or missing www? Did the site verify okay?

  9. #8
    jayismyson is offline New Bee
    Join Date
    Mar 2013
    Posts
    3
    Servers
    handsome
    WHB Points this Month
    0.00
    WHB Points
    0.00
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    it you mention "indexing", not "ranking". I imagine a busy topic like tbirds would require a really busy site to rank in the top 1000's.

    _________________
    Runescape Gold|Diablo 3 Gold|Rs Gold

  10. #9
    owlowl157 is offline New Bee
    Join Date
    Apr 2017
    Posts
    10
    WHB Points this Month
    5.00
    WHB Points
    5.00
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Excellent post. Thank you very much!!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •