Some publishers don’t allow to crawl their website and articles and set a restriction in the domains robots.txt – file. If a publisher has declined to crawl his website we’re not allowed to gather any contextual information from this specific domain/URL.
Comments
0 comments
Please sign in to leave a comment.