摘要:本篇內(nèi)容為機(jī)器學(xué)習(xí)實(shí)戰(zhàn)第章利用元算法提高分類性能程序清單。將當(dāng)前錯(cuò)誤率與已有的最小錯(cuò)誤率進(jìn)行對(duì)比后,如果當(dāng)前的值較小,那么就在字典中保存該單層決策樹。上述,我們已經(jīng)構(gòu)建了單層決策樹,得到了弱學(xué)習(xí)器。
本篇內(nèi)容為《機(jī)器學(xué)習(xí)實(shí)戰(zhàn)》第 7 章利用 AdaBoost 元算法提高分類性能程序清單。所用代碼為 python3。
AdaBoost
優(yōu)點(diǎn):泛化錯(cuò)誤率低,易編碼,可以應(yīng)用在大部分分類器上,無(wú)參數(shù)調(diào)整。
缺點(diǎn):對(duì)離群點(diǎn)敏感。
適用數(shù)據(jù)類型:數(shù)值型和標(biāo)稱型數(shù)據(jù)。
boosting 方法擁有多個(gè)版本,這里將只關(guān)注其中一個(gè)最流行的版本 AdaBoost。
在構(gòu)造 AdaBoost 的代碼時(shí),我們將首先通過(guò)一個(gè)簡(jiǎn)單數(shù)據(jù)集來(lái)確保在算法實(shí)現(xiàn)上一切就緒。使用如下的數(shù)據(jù)集:
def loadSimpData(): datMat = matrix([[ 1. , 2.1], [ 2. , 1.1], [ 1.3, 1. ], [ 1. , 1. ], [ 2. , 1. ]]) classLabels = [1.0, 1.0, -1.0, -1.0, 1.0] return datMat,classLabels
在 python 提示符下,執(zhí)行代碼加載數(shù)據(jù)集:
>>> import adaboost >>> datMat, classLabels=adaboost.loadSimpData()
我們先給出函數(shù)buildStump()的偽代碼:
""" Created on Sep 20, 2018 @author: yufei Adaboost is short for Adaptive Boosting """ """ 測(cè)試是否有某個(gè)值小于或大于我們正在測(cè)試的閾值 """ def stumpClassify(dataMatrix,dimen,threshVal,threshIneq):#just classify the data retArray = ones((shape(dataMatrix)[0],1)) if threshIneq == "lt": retArray[dataMatrix[:,dimen] <= threshVal] = -1.0 else: retArray[dataMatrix[:,dimen] > threshVal] = -1.0 return retArray """ 在一個(gè)加權(quán)數(shù)據(jù)集中循環(huán) buildStump()將會(huì)遍歷stumpClassify()函數(shù)所有的可能輸入值 并找到具有最低錯(cuò)誤率的單層決策樹 """ def buildStump(dataArr,classLabels,D): dataMatrix = mat(dataArr); labelMat = mat(classLabels).T m,n = shape(dataMatrix) # 變量 numSteps 用于在特征的所有可能值上進(jìn)行遍歷 numSteps = 10.0 # 創(chuàng)建一個(gè)空字典,用于存儲(chǔ)給定權(quán)重向量 D 時(shí)所得到的最佳單層決策樹的相關(guān)信息 bestStump = {}; bestClasEst = mat(zeros((m,1))) # 初始化為正無(wú)窮大,之后用于尋找可能的最小錯(cuò)誤率 minError = inf # 第一層循環(huán)在數(shù)據(jù)集的所有特征上遍歷 for i in range(n):#loop over all dimensions rangeMin = dataMatrix[:,i].min(); rangeMax = dataMatrix[:,i].max(); # 計(jì)算步長(zhǎng) stepSize = (rangeMax-rangeMin)/numSteps # 第二層循環(huán)是了解步長(zhǎng)后再在這些值上遍歷 for j in range(-1,int(numSteps)+1):#loop over all range in current dimension # 第三個(gè)循環(huán)是在大于和小于之間切換不等式 for inequal in ["lt", "gt"]: #go over less than and greater than threshVal = (rangeMin + float(j) * stepSize) # 調(diào)用 stumpClassify() 函數(shù),返回分類預(yù)測(cè)結(jié)果 predictedVals = stumpClassify(dataMatrix,i,threshVal,inequal)#call stump classify with i, j, lessThan errArr = mat(ones((m,1))) errArr[predictedVals == labelMat] = 0 weightedError = D.T*errArr #calc total error multiplied by D # print("split: dim %d, thresh %.2f, thresh ineqal: %s, the weighted error is %.3f" % (i, threshVal, inequal, weightedError)) # 將當(dāng)前錯(cuò)誤率與已有的最小錯(cuò)誤率進(jìn)行比較 if weightedError < minError: minError = weightedError bestClasEst = predictedVals.copy() bestStump["dim"] = i bestStump["thresh"] = threshVal bestStump["ineq"] = inequal return bestStump,minError,bestClasEst
為了解實(shí)際運(yùn)行過(guò)程,在 python 提示符下,執(zhí)行代碼并得到結(jié)果:
>>> D=mat(ones((5,1))/5) >>> adaboost.buildStump(datMat, classLabels, D) split: dim 0, thresh 0.90, thresh ineqal: lt, the weighted error is 0.400 split: dim 0, thresh 0.90, thresh ineqal: gt, the weighted error is 0.600 split: dim 0, thresh 1.00, thresh ineqal: lt, the weighted error is 0.400 split: dim 0, thresh 1.00, thresh ineqal: gt, the weighted error is 0.600 split: dim 0, thresh 1.10, thresh ineqal: lt, the weighted error is 0.400 split: dim 0, thresh 1.10, thresh ineqal: gt, the weighted error is 0.600 split: dim 0, thresh 1.20, thresh ineqal: lt, the weighted error is 0.400 split: dim 0, thresh 1.20, thresh ineqal: gt, the weighted error is 0.600 split: dim 0, thresh 1.30, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.30, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.40, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.40, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.50, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.50, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.60, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.60, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.70, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.70, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.80, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.80, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 1.90, thresh ineqal: lt, the weighted error is 0.200 split: dim 0, thresh 1.90, thresh ineqal: gt, the weighted error is 0.800 split: dim 0, thresh 2.00, thresh ineqal: lt, the weighted error is 0.600 split: dim 0, thresh 2.00, thresh ineqal: gt, the weighted error is 0.400 split: dim 1, thresh 0.89, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 0.89, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.00, thresh ineqal: lt, the weighted error is 0.200 split: dim 1, thresh 1.00, thresh ineqal: gt, the weighted error is 0.800 split: dim 1, thresh 1.11, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.11, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.22, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.22, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.33, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.33, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.44, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.44, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.55, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.55, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.66, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.66, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.77, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.77, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.88, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.88, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 1.99, thresh ineqal: lt, the weighted error is 0.400 split: dim 1, thresh 1.99, thresh ineqal: gt, the weighted error is 0.600 split: dim 1, thresh 2.10, thresh ineqal: lt, the weighted error is 0.600 split: dim 1, thresh 2.10, thresh ineqal: gt, the weighted error is 0.400 ({"dim": 0, "thresh": 1.3, "ineq": "lt"}, matrix([[0.2]]), array([[-1.], [ 1.], [-1.], [-1.], [ 1.]]))
這一行可以注釋掉,這里為了理解函數(shù)的運(yùn)行而打印出來(lái)。
將當(dāng)前錯(cuò)誤率與已有的最小錯(cuò)誤率進(jìn)行對(duì)比后,如果當(dāng)前的值較小,那么就在字典baseStump中保存該單層決策樹。字典、錯(cuò)誤率和類別估計(jì)值都會(huì)返回給 AdaBoost 算法。
上述,我們已經(jīng)構(gòu)建了單層決策樹,得到了弱學(xué)習(xí)器。接下來(lái),我們將使用多個(gè)弱分類器來(lái)構(gòu)建 AdaBoost 代碼。
首先給出整個(gè)實(shí)現(xiàn)的偽代碼如下:
程序清單 7-2 基于單層決策樹的 AdaBoost 訓(xùn)練過(guò)程""" 輸入?yún)?shù):數(shù)據(jù)集、類別標(biāo)簽、迭代次數(shù)(需要用戶指定) """ def adaBoostTrainDS(dataArr,classLabels,numIt=40): weakClassArr = [] m = shape(dataArr)[0] # 向量 D 包含了每個(gè)數(shù)據(jù)點(diǎn)的權(quán)重,初始化為 1/m D = mat(ones((m,1))/m) #init D to all equal # 記錄每個(gè)數(shù)據(jù)點(diǎn)的類別估計(jì)累計(jì)值 aggClassEst = mat(zeros((m,1))) for i in range(numIt): # 調(diào)用 buildStump() 函數(shù)建立一個(gè)單層決策樹 bestStump,error,classEst = buildStump(dataArr,classLabels,D)#build Stump print ("D:",D.T) # 計(jì)算 alpha,本次單層決策樹輸出結(jié)果的權(quán)重 # 確保沒有錯(cuò)誤時(shí)不會(huì)發(fā)生除零溢出 alpha = float(0.5*log((1.0-error)/max(error,1e-16)))#calc alpha, throw in max(error,eps) to account for error=0 bestStump["alpha"] = alpha weakClassArr.append(bestStump) #store Stump Params in Array print("classEst: ",classEst.T) # 為下一次迭代計(jì)算 D expon = multiply(-1*alpha*mat(classLabels).T,classEst) #exponent for D calc, getting messy D = multiply(D,exp(expon)) #Calc New D for next iteration D = D/D.sum() #calc training error of all classifiers, if this is 0 quit for loop early (use break) # 錯(cuò)誤率累加計(jì)算 aggClassEst += alpha*classEst print("aggClassEst: ",aggClassEst.T) # 為了得到二值分類結(jié)果調(diào)用 sign() 函數(shù) aggErrors = multiply(sign(aggClassEst) != mat(classLabels).T,ones((m,1))) errorRate = aggErrors.sum()/m print ("total error: ",errorRate) # 若總錯(cuò)誤率為 0,則中止 for 循環(huán) if errorRate == 0.0: break return weakClassArr,aggClassEst
在 python 提示符下,執(zhí)行代碼并得到結(jié)果:
>>> classifierArray = adaboost.adaBoostTrainDS(datMat, classLabels, 9) D: [[0.2 0.2 0.2 0.2 0.2]] classEst: [[-1. 1. -1. -1. 1.]] aggClassEst: [[-0.69314718 0.69314718 -0.69314718 -0.69314718 0.69314718]] total error: 0.2 D: [[0.5 0.125 0.125 0.125 0.125]] classEst: [[ 1. 1. -1. -1. -1.]] aggClassEst: [[ 0.27980789 1.66610226 -1.66610226 -1.66610226 -0.27980789]] total error: 0.2 D: [[0.28571429 0.07142857 0.07142857 0.07142857 0.5 ]] classEst: [[1. 1. 1. 1. 1.]] aggClassEst: [[ 1.17568763 2.56198199 -0.77022252 -0.77022252 0.61607184]] total error: 0.0
最后,我們來(lái)觀察測(cè)試錯(cuò)誤率。
""" 將弱分類器的訓(xùn)練過(guò)程從程序中抽查來(lái),應(yīng)用到某個(gè)具體的實(shí)例上去。 datToClass: 一個(gè)或多個(gè)待分類樣例 classifierArr: 多個(gè)弱分類器組成的數(shù)組 返回 aggClassEst 符號(hào),大于 0 返回1;小于 0 返回 -1 """ def adaClassify(datToClass,classifierArr): dataMatrix = mat(datToClass)#do stuff similar to last aggClassEst in adaBoostTrainDS m = shape(dataMatrix)[0] aggClassEst = mat(zeros((m,1))) for i in range(len(classifierArr)): classEst = stumpClassify(dataMatrix, classifierArr[0][i]["dim"], classifierArr[0][i]["thresh"], classifierArr[0][i]["ineq"]) aggClassEst += classifierArr[0][i]["alpha"]*classEst print (aggClassEst) return sign(aggClassEst)
在 python 提示符下,執(zhí)行代碼并得到結(jié)果:
>>> datArr, labelArr = adaboost.loadSimpData() >>> classifierArr = adaboost.adaBoostTrainDS(datArr, labelArr, 30) D: [[0.2 0.2 0.2 0.2 0.2]] classEst: [[-1. 1. -1. -1. 1.]] aggClassEst: [[-0.69314718 0.69314718 -0.69314718 -0.69314718 0.69314718]] total error: 0.2 D: [[0.5 0.125 0.125 0.125 0.125]] classEst: [[ 1. 1. -1. -1. -1.]] aggClassEst: [[ 0.27980789 1.66610226 -1.66610226 -1.66610226 -0.27980789]] total error: 0.2 D: [[0.28571429 0.07142857 0.07142857 0.07142857 0.5 ]] classEst: [[1. 1. 1. 1. 1.]] aggClassEst: [[ 1.17568763 2.56198199 -0.77022252 -0.77022252 0.61607184]] total error: 0.0
輸入以下命令進(jìn)行分類:
>>> adaboost.adaClassify([0,0], classifierArr) [[-0.69314718]] [[-1.66610226]] matrix([[-1.]])
隨著迭代的進(jìn)行,數(shù)據(jù)點(diǎn) [0,0] 的分類結(jié)果越來(lái)越強(qiáng)。也可以在其它點(diǎn)上分類:
>>> adaboost.adaClassify([[5,5],[0,0]], classifierArr) [[ 0.69314718] [-0.69314718]] [[ 1.66610226] [-1.66610226]] matrix([[ 1.], [-1.]])
這兩個(gè)點(diǎn)的分類結(jié)果也會(huì)隨著迭代的進(jìn)行而越來(lái)越強(qiáng)。
參考鏈接:
GBDT,ADABOOSTING概念區(qū)分 GBDT與XGBOOST區(qū)別
【機(jī)器學(xué)習(xí)實(shí)戰(zhàn)-python3】Adaboost元算法提高分類性能
$$$$
不足之處,歡迎指正。
文章版權(quán)歸作者所有,未經(jīng)允許請(qǐng)勿轉(zhuǎn)載,若此文章存在違規(guī)行為,您可以聯(lián)系管理員刪除。
轉(zhuǎn)載請(qǐng)注明本文地址:http://specialneedsforspecialkids.com/yun/42479.html
摘要:本篇內(nèi)容為機(jī)器學(xué)習(xí)實(shí)戰(zhàn)第章支持向量機(jī)部分程序清單。支持向量機(jī)優(yōu)點(diǎn)泛化錯(cuò)誤率低,計(jì)算開銷不大,結(jié)果易解釋。注以上給出的僅是簡(jiǎn)化版算法的實(shí)現(xiàn),關(guān)于完整的算法加速優(yōu)化并應(yīng)用核函數(shù),請(qǐng)參照機(jī)器學(xué)習(xí)實(shí)戰(zhàn)第頁(yè)。 本篇內(nèi)容為《機(jī)器學(xué)習(xí)實(shí)戰(zhàn)》第 6 章 支持向量機(jī)部分程序清單。所用代碼為 python3。 支持向量機(jī)優(yōu)點(diǎn):泛化錯(cuò)誤率低,計(jì)算開銷不大,結(jié)果易解釋。 缺點(diǎn):對(duì)參數(shù)調(diào)節(jié)和核函數(shù)的選擇敏感,...
摘要:根據(jù)錯(cuò)誤率決定是否回退到訓(xùn)練階段,通過(guò)改變迭代的次數(shù)和步長(zhǎng)等參數(shù)來(lái)得到更好的回歸系數(shù)。使用回歸方法進(jìn)行分類所需做的是把測(cè)試集上每個(gè)特征向量乘以最優(yōu)化方法得來(lái)的回歸系數(shù),再將該乘積結(jié)果求和,最后輸入到函數(shù)即可。 本篇內(nèi)容為《機(jī)器學(xué)習(xí)實(shí)戰(zhàn)》第 5 章 Logistic 回歸程序清單。 書中所用代碼為 python2,下面給出的程序清單是在 python3 中實(shí)踐改過(guò)的代碼,希望對(duì)你有幫助。...
摘要:本篇內(nèi)容為機(jī)器學(xué)習(xí)實(shí)戰(zhàn)第章決策樹部分程序清單。適用數(shù)據(jù)類型數(shù)值型和標(biāo)稱型在構(gòu)造決策樹時(shí),我們需要解決的第一個(gè)問(wèn)題就是,當(dāng)前數(shù)據(jù)集上哪個(gè)特征在劃分?jǐn)?shù)據(jù)分類時(shí)起決定性作用。下面我們會(huì)介紹如何將上述實(shí)現(xiàn)的函數(shù)功能放在一起,構(gòu)建決策樹。 本篇內(nèi)容為《機(jī)器學(xué)習(xí)實(shí)戰(zhàn)》第 3 章決策樹部分程序清單。所用代碼為 python3。 決策樹優(yōu)點(diǎn):計(jì)算復(fù)雜度不高,輸出結(jié)果易于理解,對(duì)中間值的缺失不敏感,可...
閱讀 2857·2023-04-26 02:49
閱讀 3441·2021-11-25 09:43
閱讀 3367·2021-10-09 09:43
閱讀 2985·2021-09-28 09:44
閱讀 2446·2021-09-22 15:29
閱讀 4506·2021-09-14 18:02
閱讀 2773·2021-09-03 10:48
閱讀 3426·2019-08-30 12:47