大家好我是小小明，今天給大家演示如何使用python直接采集百度指數(shù)的數(shù)據(jù)。

百度指數(shù)(Baidu Index) 是以百度海量網(wǎng)民行為數(shù)據(jù)為基礎(chǔ)的數(shù)據(jù)分析平臺(tái)，它能夠能夠告訴用戶(hù)：某個(gè)關(guān)鍵詞在百度的搜索規(guī)模有多大，一段時(shí)間內(nèi)的漲跌態(tài)勢(shì)以及相關(guān)的新聞?shì)浾撟兓P(guān)注這些詞的網(wǎng)民是什么樣的，分布在哪里，同時(shí)還搜了哪些相關(guān)的詞。

百分十先生分享過(guò)如何使用uiautomation采集百度指數(shù)：百度指數(shù) 如何批量獲取？

不過(guò)個(gè)人感覺(jué)這方法好像有點(diǎn)殺雞用牛刀，對(duì)于網(wǎng)頁(yè)使用selenium完全足以，當(dāng)然對(duì)于專(zhuān)門(mén)針對(duì)selenium進(jìn)行反爬檢測(cè)的網(wǎng)頁(yè)就需要特殊修改。

本文不演示如何使用UI自動(dòng)化工具采集百度指數(shù)，為了采集更簡(jiǎn)單將直接讀取并解析接口。

關(guān)于uiautomation，PC端的UI自動(dòng)化可以查看教程：Windows桌面程序自動(dòng)化控制之uiautomation模塊全面講解

打開(kāi)百度指數(shù)發(fā)現(xiàn)查看指數(shù)必須要先登錄，比如我們對(duì)比一個(gè)python和Java最近一周的指數(shù)：

當(dāng)鼠標(biāo)移動(dòng)到每天的坐標(biāo)上時(shí)會(huì)顯示當(dāng)天的數(shù)據(jù)，例如：

如果我們采用UI自動(dòng)化的方式，至少得模擬移動(dòng)到每天的坐標(biāo)。

打開(kāi)開(kāi)發(fā)者工具，重新查詢(xún)發(fā)現(xiàn)獲取數(shù)據(jù)的接口：

實(shí)際的指數(shù)數(shù)據(jù)就存儲(chǔ)在這個(gè)data字段中，但是以某種加密方式加密了。

然后注意第二個(gè)接口的某個(gè)參數(shù)與當(dāng)前接口返回的數(shù)據(jù)某個(gè)值一致。

此時(shí)我全局搜索decrypt，找到了加密函數(shù)：

此時(shí)打上斷點(diǎn)重新搜索，可以看到傳入該函數(shù)的t參數(shù)與ptbk接口返回的值一致：

說(shuō)明我們只需要將這段js翻譯為python來(lái)解密加密數(shù)據(jù)即可。

下面我們總結(jié)一下指數(shù)數(shù)據(jù)獲取的思路：

1. 通過(guò)index接口獲取uniqid和加密后的指數(shù)數(shù)據(jù)userIndexes

2. 通過(guò)ptbk接口傳入uniqid獲取密鑰key

3. 通過(guò)解密函數(shù)根據(jù)密鑰key解密userIndexes

下面我們分別用代碼來(lái)實(shí)現(xiàn)，首先獲取指數(shù)數(shù)據(jù)：

import requestsimport jsonheaders = {    "Connection": "keep-alive",    "Accept": "application/json, text/plain, */*",    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",    "Sec-Fetch-Site": "same-origin",    "Sec-Fetch-Mode": "cors",    "Sec-Fetch-Dest": "empty",    "Referer": "https://index.baidu.com/v2/main/index.html",    "Accept-Language": "zh-CN,zh;q=0.9",    "Cookie": cookie,}words = "[[{"name":"python","wordType":1}],[{"name":"java","wordType":1}]]"start = "2021-11-15"end = "2021-11-21"url = f"http://index.baidu.com/api/SearchApi/index?area=0&word={words}&area=0&startDate={start}&endDate={end}"res = requests.get(url, headers=headers)data = res.json()["data"]data

cookie需要在登錄后復(fù)制粘貼獲取，就是請(qǐng)求中的這段字符串（直接復(fù)制粘貼即可）：

結(jié)果：

{"userIndexes": [{"word": [{"name": "python", "wordType": 1}],   "all": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": "WQ3Q-nWQ.yGnWQ.y3nW3yQsnWW.Q-nysXV3ny.-VG"},   "pc": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": "y3yVXny3yWyny3GWWny3QyVnyQG33nXGsQn-..G"},   "wise": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": "XWVXnXQ-XnX3XWnX-WynX3X3n--XynsQyG"},   "type": "day"},  {"word": [{"name": "java", "wordType": 1}],   "all": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": "-XW.n-ssXnXG3GnXG..nXyyGnVQyWn.QQQ"},   "pc": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": ".VVVn.3Xsn.XX3n.-VWn.sW3nQG-snWVWQ"},   "wise": {"startDate": "2021-11-15",    "endDate": "2021-11-21",    "data": "QW.XnQW-WnQG3VnQyXQnQQ-VnQWW.nWsyG"},   "type": "day"}], "generalRatio": [{"word": [{"name": "python", "wordType": 1}],   "all": {"avg": 21565, "yoy": -24, "qoq": 7},   "pc": {"avg": 12470, "yoy": -32, "qoq": 3},   "wise": {"avg": 9095, "yoy": -10, "qoq": 12}},  {"word": [{"name": "java", "wordType": 1}],   "all": {"avg": 8079, "yoy": -23, "qoq": 11},   "pc": {"avg": 4921, "yoy": -33, "qoq": 6},   "wise": {"avg": 3157, "yoy": "-", "qoq": 18}}], "uniqid": "5f0a123915325e28d9f055409955c9ad"}

這些數(shù)據(jù)中，wise表示移動(dòng)端，all表示pc端+移動(dòng)端。userIndexes是指數(shù)詳情數(shù)據(jù)，generalRatio是概覽數(shù)據(jù)。

下面我們只關(guān)心各個(gè)關(guān)鍵字的整體表現(xiàn)。

下面我們獲取uniqid并獲取ptbk：

uniqid = data["uniqid"]res = requests.get(    f"http://index.baidu.com/Interface/ptbk?uniqid={uniqid}", headers=headers)ptbk = res.json()["data"]ptbk

"LV.7yF-s30WXGQn.65+1-874%2903,"

下面我將下面這段Js代碼翻譯為python：

decrypt: function(t, e) {	if (t) {		for (var n = t.split(""), i = e.split(""), a = {}, r = [], o = 0; o < n.length / 2; o++)			a[n[o]] = n[n.length / 2 + o];		for (var s = 0; s < e.length; s++)			r.push(a[i[s]]);		return r.join("")	}}

python代碼：

def decrypt(ptbk, index_data):    n = len(ptbk)//2    a = dict(zip(ptbk[:n], ptbk[n:]))    return "".join([a[s] for s in index_data])

然后我們遍歷每個(gè)關(guān)鍵字解密出對(duì)應(yīng)的指數(shù)數(shù)據(jù)：

for userIndexe in data["userIndexes"]:    name = userIndexe["word"][0]["name"]    index_data = userIndexe["all"]["data"]    r = decrypt(ptbk, index_data)    print(name, r)

python 23438,23510,23514,24137,22538,17964,15860java 8925,8779,9040,9055,9110,6312,5333

檢查實(shí)際網(wǎng)頁(yè)中的數(shù)據(jù)發(fā)現(xiàn)確實(shí)一致：

那么我們就可以輕松獲取任意指定關(guān)鍵字的指數(shù)數(shù)據(jù)。下面我將其整體封裝一下，完整代碼為：

import requestsimport jsonfrom datetime import date, timedeltaheaders = {    "Connection": "keep-alive",    "Accept": "application/json, text/plain, */*",    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",    "Sec-Fetch-Site": "same-origin",    "Sec-Fetch-Mode": "cors",    "Sec-Fetch-Dest": "empty",    "Referer": "https://index.baidu.com/v2/main/index.html",    "Accept-Language": "zh-CN,zh;q=0.9",    "Cookie": cookie,}def decrypt(ptbk, index_data):    n = len(ptbk)//2    a = dict(zip(ptbk[:n], ptbk[n:]))    return "".join([a[s] for s in index_data])def get_index_data(keys, start=None, end=None):    words = [[{"name": key, "wordType": 1}] for key in keys]    words = str(words).replace(" ", "").replace(""", "/"")    today = date.today()    if start is None:        start = str(today-timedelta(days=8))    if end is None:        end = str(today-timedelta(days=2))    url = f"http://index.baidu.com/api/SearchApi/index?area=0&word={words}&area=0&startDate={start}&endDate={end}"    print(words, start, end)    res = requests.get(url, headers=headers)    data = res.json()["data"]    uniqid = data["uniqid"]    url = f"http://index.baidu.com/Interface/ptbk?uniqid={uniqid}"    res = requests.get(url, headers=headers)    ptbk = res.json()["data"]    result = {}    result["startDate"] = start    result["endDate"] = end    for userIndexe in data["userIndexes"]:        name = userIndexe["word"][0]["name"]        tmp = {}        index_all = userIndexe["all"]["data"]        index_all_data = [int(e) for e in decrypt(ptbk, index_all).split(",")]        tmp["all"] = index_all_data        index_pc = userIndexe["pc"]["data"]        index_pc_data = [int(e) for e in decrypt(ptbk, index_pc).split(",")]        tmp["pc"] = index_pc_data        index_wise = userIndexe["wise"]["data"]        index_wise_data = [int(e)                           for e in decrypt(ptbk, index_wise).split(",")]        tmp["wise"] = index_wise_data        result[name] = tmp    return result

測(cè)試一下：

get_index_data(["python", "java"])

{"startDate": "2021-11-15", "endDate": "2021-11-21", "python": {"all": [23438, 23510, 23514, 24137, 22538, 17964, 15860],  "pc": [14169, 14121, 14022, 14316, 13044, 9073, 8550],  "wise": [9269, 9389, 9492, 9821, 9494, 8891, 7310]}, "java": {"all": [8925, 8779, 9040, 9055, 9110, 6312, 5333],  "pc": [5666, 5497, 5994, 5862, 5724, 3087, 2623],  "wise": [3259, 3282, 3046, 3193, 3386, 3225, 2710]}}

結(jié)果非常不錯(cuò)。