Many of my clients and business friends have asked my expert opinion on some “Facebook related thing” in their website logs. They were worried it might be a spammer, hacker or something with ill intent.
They say something like this in their web logs:
Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
This is the Facebook’s website crawler. It’s safe!
Just like Google has a website crawler that scans your website for content, relavancy, images and all that fun stuff Facebook has the same thing that it uses in-order to make sure the links it posts are safe for users, so that it can get the preview image and the snippet text.
Do not block the Facebook web crawler.
It’s needed if you ever want Facebook to see your website as web safe and friendly.
This is what Facebook’s own documentation says about it:
Why does Facebook appear in my server logs?
Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video. Our system retrieves this information only after a user provides us with a link. You may have found this page because a Facebook user sent a link from your website to other Facebook users. If you have any questions or concerns about any links or content sent by one of our users, please contact us at email@example.com.