Web Scraping

In this tutorial, we’ll explore various Python libraries and modules commonly used for web scraping and delve into why Python 3 is the preferred choice for this task. Along with this you will also explore how to use powerful tools like BeautifulSoup, Scrapy, and Selenium to scrape any website.

Requests

The requests library is used for making HTTP requests to a specific URL and returns the response. Python requests provide inbuilt functionalities for managing both the request and response.

$ pip install requests

...\> pip install requests

$ python3
Python 3.13.2 (v3.13.2:4f8bb3947cf, Feb  4 2025, 11:51:10) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://www.teslapython.com/')
>>> print(r)
<Response [200]>
>>> print(r.content)
b'<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <meta charset="utf-8">\n    <meta http-equiv="X-UA-Compatible" content="IE=edge">\n    <meta name="viewport" content="width=device-width, initial-scale=1">\n    <meta name="ROBOTS" content="ALL" />\n    <meta name="MSSmartTagsPreventParsing" content="true" />\n    <meta name="Copyright" content="Django Software Foundation" />\n    <meta name="keywords" content="Python, Django, framework, open-source" />\n    <meta name="description" content="" />\n\n    \n    <!-- Favicons -->\n    <link rel="apple-touch-icon" href="/s/img/icon-touch.b3e2b3183b98.png">\n    <link rel="icon" sizes="192x192" href="/s/img/icon-touch.b3e2b3183b98.png">\n    <link rel="shortcut icon" href="/s/img/favicon.d1d1c1eecf7d.ico">\n

...\> py 
Python 3.13.2 (v3.13.2:4f8bb3947cf, Feb  4 2025, 11:51:10) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://www.teslapython.com/')
>>> print(r)
<Response [200]>
>>> print(r.content)
b'<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <meta charset="utf-8">\n    <meta http-equiv="X-UA-Compatible" content="IE=edge">\n    <meta name="viewport" content="width=device-width, initial-scale=1">\n    <meta name="ROBOTS" content="ALL" />\n    <meta name="MSSmartTagsPreventParsing" content="true" />\n    <meta name="Copyright" content="Django Software Foundation" />\n    <meta name="keywords" content="Python, Django, framework, open-source" />\n    <meta name="description" content="" />\n\n    \n    <!-- Favicons -->\n    <link rel="apple-touch-icon" href="/s/img/icon-touch.b3e2b3183b98.png">\n    <link rel="icon" sizes="192x192" href="/s/img/icon-touch.b3e2b3183b98.png">\n    <link rel="shortcut icon" href="/s/img/favicon.d1d1c1eecf7d.ico">\n

BeautifulSoup

BeautifulSoup provides a few simple methods and Pythonic phrases for guiding, searching, and changing a parse tree: a toolkit for studying a document and removing what you need. It doesn’t take much code to document an application.

$ pip install beautifulsoup4

...\> pip install beautifulsoup4

$ python3
Python 3.13.2 (v3.13.2:4f8bb3947cf, Feb  4 2025, 11:51:10) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from bs4 import BeautifulSoup
>>> r = requests.get('https://www.teslapython.com/')
>>> print(r)
<Response [200]>
>>> soup = BeautifulSoup(r.content, 'html.parser')
>>> print(soup.prettify())
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<meta content="ALL" name="ROBOTS">
    <meta content="true" name="MSSmartTagsPreventParsing">
        <meta content="TeslaPython Software Foundation" name="Copyright">

...\> py 
Python 3.13.2 (v3.13.2:4f8bb3947cf, Feb  4 2025, 11:51:10) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from bs4 import BeautifulSoup
>>> r = requests.get('https://www.teslapython.com/')
>>> print(r)
<Response [200]>
>>> soup = BeautifulSoup(r.content, 'html.parser')
>>> print(soup.prettify())
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<meta content="ALL" name="ROBOTS">
    <meta content="true" name="MSSmartTagsPreventParsing">
        <meta content="TeslaPython Software Foundation" name="Copyright">

Scrapy

Nowadays data is everything and if someone wants to get data from webpages then one way to use an API or implement Web Scraping techniques. In Python, Web scraping can be done easily by using scraping tools like BeautifulSoup. But what if the user is concerned about performance of scraper or need to scrape data efficiently.

« previous | up | next »