Wikipeda scrapers aborting from *HTTP Error 403: Forbidden*

[royrada]

The examples for searching Wikipedia are not working for me. Take for instance Cell 1, Chapter 6. My execution is aborted with Http Error 403: Forbidden. When I modify the code only by pointing to my own website rather than Wikipedia, the code works. I experimented on some other Wikipedia examples in the book and had a similar result. Since the book relies heavily on the structure of Wikipedia in many code examples in many chapters, the possibility that Wikipedia has introduced new barriers for the book's Wiki scrapers merits attention. The author has chapters on how to work around barriers from web servers but, to my knowledge, does not address the Wikipedia problem.

[a2z-tech-innovations]

@royrada Wikipedia just blocks requests without headers, request via urlib excludes them by default.

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

url = 'http://en.wikipedia.org/wiki/Kevin_Bacon'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

req = Request(url, headers=headers)
html = urlopen(req)

bs = BeautifulSoup(html, 'lxml')

[royrada]

Thank you very much. I implemented your suggestion of adding a header to pretend to be a web browser and after that Wikipedia executed without error the python program from Ryan Mitchell's book. Ryan has discussed the sophisticated strategies that web servers can employ to stop computer programs from crawling and scraping their websites. She noted that the header facade was so easily implemented that web servers were likely to go to behavioral monitoring. I assumed that Wikipedia was using behavioral monitoring. As I am learning how to scrape the web, your guidance of inserting a browser heading will be followed and each time with a virtual 'thank you' from me to you.