Python爬取网页信息

本文将从多个方面对Python爬取网页信息做详细的阐述。

一、爬虫介绍

爬虫是一种自动化程序，可以模拟人对网页进行访问获取信息的行为。通过编写代码，我们可以指定要获取的信息，将其从网页中提取出来，以方便我们进行分析、处理或者存储。

二、Python爬虫的基本框架

Python作为一门简单易学且功能强大的语言，广泛应用于爬虫领域。Python爬取网页数据的基本框架如下：

   import requests
   from bs4 import BeautifulSoup
   
   # 网页url
   url = 'http://www.example.com'
   
   # 向网页发送请求
   r = requests.get(url)
   
   # 通过BeautifulSoup解析网页内容
   soup = BeautifulSoup(r.text, 'html.parser')
   
   # 获取需要的信息并进行处理
   # ...

其中，通过requests库向网页发送请求，获取网页内容；使用BeautifulSoup库解析网页内容，提取需要的信息。

三、使用requests库发送请求

requests是一个常用的Python HTTP库，它可以发送HTTP请求，并处理服务器响应。以下是一个使用requests获取豆瓣电影排行榜前20名的基本代码：

   import requests
   from bs4 import BeautifulSoup
   
   url = 'https://movie.douban.com/top250'
   headers = {
       'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
   r = requests.get(url, headers=headers)
   soup = BeautifulSoup(r.text, 'html.parser')
   
   for movie in soup.find_all('div', class_='item'):
       title = movie.find('span', class_='title').get_text()
       rank = movie.find('em', class_='').get_text()
       print(rank + '：' + title)

通过requests发送GET请求，获取网页内容。设置headers，模拟浏览器对网页的访问。将获取到的网页内容用BeautifulSoup进行解析，并提取需要的信息。

四、使用BeautifulSoup解析网页内容

BeautifulSoup是Python中最流行的HTML解析库之一，它可以将HTML文档解析为树状结构，便于对其中的内容进行提取和处理。以下是一个使用BeautifulSoup解析网页的基本代码：

   from bs4 import BeautifulSoup
   
   html_doc = '''
   
   
   
The Dormouse's story
   
   
Once upon a time there were three little sisters; and their names were   Elsie,   Lacie and   Tillie;   and they lived at the bottom of a well.
   
   
...
   '''
   
   soup = BeautifulSoup(html_doc, 'html.parser')
   
   print(soup.prettify())

使用BeautifulSoup库对HTML文档的标签结构进行解析，以形成树状结构。使用prettify()函数将HTML文档格式化输出。

五、解析网页内容获取所需信息

通过BeautifulSoup解析HTML文档，我们可以通过一些方法获取文档中的各种信息。以下是几种常见的方法：

1. find()

find()方法返回第一个符合条件的标签内容。

   from bs4 import BeautifulSoup
   
   html_doc = '''
   
   
   
The Dormouse's story
   
   
Once upon a time there were three little sisters; and their names were   Elsie,   Lacie and   Tillie;   and they lived at the bottom of a well.
   
   
...
   '''
   
   soup = BeautifulSoup(html_doc, 'html.parser')
   
   print(soup.find('p', class_='title').get_text())

输出结果：

   The Dormouse's story

2. find_all()

find_all()方法返回符合条件的所有标签内容。

   from bs4 import BeautifulSoup
   
   html_doc = '''
   
   
   
The Dormouse's story
   
   
Once upon a time there were three little sisters; and their names were   Elsie,   Lacie and   Tillie;   and they lived at the bottom of a well.
   
   
...
   '''
   
   soup = BeautifulSoup(html_doc, 'html.parser')
   
   for link in soup.find_all('a'):
       print(link.get('href'))

输出结果：

   http://example.com/elsie
   http://example.com/lacie
   http://example.com/tillie

3. get_text()

get_text()方法返回标签内的文本内容。

   from bs4 import BeautifulSoup
   
   html_doc = '''
   
   
   
The Dormouse's story
   
   
Once upon a time there were three little sisters; and their names were   Elsie,   Lacie and   Tillie;   and they lived at the bottom of a well.
   
   
...
   '''
   
   soup = BeautifulSoup(html_doc, 'html.parser')
   
   print(soup.get_text())

输出结果：

   The Dormouse's story
   
   
   The Dormouse's story
   
   
   
   Once upon a time there were three little sisters; and their names were
   
   Elsie,
   
   
   Lacie and
   
   
   Tillie;
   and they lived at the bottom of a well.
   
   
   
   ...

六、总结

Python爬虫是一种获取网页信息的有效方式。本文介绍了Python爬虫的基本框架、使用requests库发送请求、使用BeautifulSoup解析网页内容、解析网页内容获取所需信息等方面的知识。