Hi everyone,
We keep an IIS8.5 (win-2k12r2) -based web server hosting a website. The website is in hebrew and containing hebrew urls, sample:
IRI (not encoded): http://www.website.com/articles/219-לעשות-רושם-ראשוני
Equal URL (encoded): http://www.website.com/articles/219-%D7%9C%D7%A2%D7%A9%D7%95%D7%AA-%D7%A8%D7%95%D7%A9%D7%9D-%D7%A8%D7%90%D7%A9%D7%95%D7%A0%D7%99
We've noticed that Facebook fails to scrape website pages with BAD REQUEST (400) error. It happens either user shares URL or IRI.
Facebook debugger helps to resolve the problem and sharing works well afterwards, but first share attempt per page performed without fb-debugger always fails.
I've investigated the issue on the server side and that what I saw in the HTTPERR log (C:\Windows\System32\LogFiles\HTTPERR\httperr.log):
2016-05-10 10:11:03 31.13.110.121 30065 [website ip] 80 HTTP/1.1 GET /articles/219-לעשות-רושם-ראשוני - 400 - URL
As far as I know HTTPERR usually logs links exactly as they requested. In this case I see that requested url is not url-encoded, so I assume that for some reason Facebook Scrapper approaches our website by IRI, not by URL. In the simple terms facebook scrapper tries to get page content by the address which is not url-encoded. If I open the logged IRI in web-browser - it opens OK because browser does URL-ENCODING by default.
So the question is actually how I can enable support for IRI (rfc3987) for IIS in order to allow it accepting such requests?
Thanks in advance, Michael
