{"id":1323,"date":"2020-11-23T10:27:54","date_gmt":"2020-11-23T10:27:54","guid":{"rendered":"https:\/\/thenextweb.com\/?p=1329038"},"modified":"2020-11-23T10:27:54","modified_gmt":"2020-11-23T10:27:54","slug":"a-beginners-guide-to-web-scraping-with-python-and-scrapy","status":"publish","type":"post","link":"https:\/\/www.londonchiropracter.com\/?p=1323","title":{"rendered":"A beginner\u2019s guide to web scraping with Python and Scrapy"},"content":{"rendered":"\n

Since their inception, websites are used to share information. Whether it is a Wikipedia article, YouTube channel, Instagram account, or a Twitter handle. They all are packed with interesting data that is available for everyone with access to the internet and a web browser.<\/p>\n

But, what if we want to get any specific data programmatically?<\/p>\n

There are two ways to do that:<\/p>\n

    \n
  1. Using official API<\/li>\n
  2. Web Scraping<\/li>\n<\/ol>\n

    The concept of API (Application Programming Interface) was introduced to exchange data between different systems in a standard way. But, most of the time, website owners don\u2019t provide any API. In that case, we are only left with the possibility to extract the data using web scraping.<\/p>\n

    Basically, every web page is returned from the server in an HTML format, meaning that our actual data is nicely packed inside HTML elements. It makes the whole process of retrieving specific data very easy and straightforward.<\/p>\n

    This tutorial will be an ultimate guide for you to learn web scraping using Python programming language. At first, I\u2019ll walk you through some basic examples to make you familiar with web scraping. Later on, we\u2019ll use that knowledge to extract data of football matches from Livescore.cz .<\/p>\n

    [Read: Neural\u2019s market outlook for artificial intelligence in 2021 and beyond<\/span><\/a>]<\/em><\/p>\n

    Getting Started<\/h2>\n

    To get us started, you will need to start a new Python3 project with and install Scrapy (a web scraping and web crawling library for Python). I\u2019m using pipenv for this tutorial, but you can use pip and venv, or conda.<\/p>\n

    pipenv install scrapy<\/em><\/p>\n

    At this point, you have Scrapy, but you still need to create a new web scraping project, and for that scrapy provides us with a command line that does the work for us.<\/p>\n

    Let\u2019s now create a new project named web_scraper by using the scrapy cli.<\/p>\n

    If you are using pipenv like me, use:<\/p>\n

    pipenv run scrapy startproject web_scraper .<\/em><\/p>\n

    Otherwise, from your virtual environment, use:<\/p>\n

    scrapy startproject web_scraper .<\/em><\/p>\n

    This will create a basic project in the current directory with the following structure:<\/p>\n

    \n
    <\/figure><\/pre>\n<\/div>\n
    Building our first Spider with XPath queries<\/h2>\n
    We will start our web scraping tutorial with a very simple example. At first, we\u2019ll locate the logo of the <\/span>Live Code Stream<\/a> <\/span>website inside HTML. And as we know, it is just a text and not an image, so we\u2019ll simply extract this text.<\/p>\n
    The code<\/strong><\/p>\n
    To get started we need to create a new spider for this project. We can do that by either creating a new file or using the CLI.<\/p>\n
    Since we know already the code we need we will create a new Python file on this path <\/span>\/web_scraper\/spiders\/live_code_stream.py<\/strong><\/p>\n
    Here are the contents of this file.<\/p>\n
    \n
    <\/figure><\/pre>\n<\/div>\n
    Code explanation:<\/strong><\/p>\n