Learning Project

Websites used to scrape data:

• https://www.imdb.com/search/title/?groups=top_250&sort=user_rating
• https://www.worldometers.info/world-population/population-by-country/

configuration:

• imdb dir:
  • Default User-Agent is changed

Setup Environment:

• create a virtual environment;
• Install requirements.txt.;
• place the above two dirs into newly created venv;

Run the project:

• Goto specific dir;
• cmd: scrapy crawl spider_name or scrapy crawl spider_name -o *.ext [ext: json, csv, xml]