摘要：最近一直在看，各類博客論文看得不少但是說實話，這樣做有些疏于實現，一來呢自己的電腦也不是很好，二來呢我目前也沒能力自己去寫一個只是跟著的寫了些已有框架的代碼這部分的代碼見后來發現了一個的的，發現其代碼很簡單，感覺比較適合用來學習算法再一個就

最近一直在看Deep Learning，各類博客、論文看得不少

但是說實話，這樣做有些疏于實現，一來呢自己的電腦也不是很好，二來呢我目前也沒能力自己去寫一個toolbox

只是跟著Andrew Ng的UFLDL tutorial?寫了些已有框架的代碼(這部分的代碼見github)

后來發現了一個matlab的Deep Learning的toolbox，發現其代碼很簡單，感覺比較適合用來學習算法

再一個就是matlab的實現可以省略掉很多數據結構的代碼，使算法思路非常清晰

所以我想在解讀這個toolbox的代碼的同時來鞏固自己學到的，同時也為下一步的實踐打好基礎

(本文只是從代碼的角度解讀算法，具體的算法理論步驟還是需要去看paper的

我會在文中給出一些相關的paper的名字，本文旨在梳理一下算法過程，不會深究算法原理和公式)

==========================================================================================

使用的代碼：DeepLearnToolbox?
，下載地址：點擊打開，感謝該toolbox的作者

==========================================================================================

第一章從分析NN(neural network)開始，因為這是整個deep learning的大框架，參見UFLDL

==========================================================================================

首先看一下 ests est_example_NN.m ，跳過對數據進行normalize的部分，最關鍵的就是：

（為了注釋顯示有顏色，我把matlab代碼中的%都改成了//）

1. nn?=?nnsetup([784?100?10]);??
2. opts.numepochs?=??1;???//??Number?of?full?sweeps?through?data??
3. opts.batchsize?=?100;??//??Take?a?mean?gradient?step?over?this?many?samples??
4. [nn,?L]?=?nntrain(nn,?train_x,?train_y,?opts);??
5. [er,?bad]?=?nntest(nn,?test_x,?test_y);??

那么我們分別來分析著幾個函數，NN nsetup.m

1. function?nn?=?nnsetup(architecture)??
2. //首先從傳入的architecture中獲得這個網絡的整體結構，nn.n表示這個網絡有多少層，可以參照上面的樣例調用nnsetup([784?100?10])加以理解??
3. ??
4. ????nn.size???=?architecture;??
5. ????nn.n??????=?numel(nn.size);??
6. ????//接下來是一大堆的參數，這個我們到具體用的時候再加以說明??
7. ????nn.activation_function??????????????=?"tanh_opt";???//??Activation?functions?of?hidden?layers:?"sigm"?(sigmoid)?or?"tanh_opt"?(optimal?tanh).??
8. ????nn.learningRate?????????????????????=?2;????????????//??learning?rate?Note:?typically?needs?to?be?lower?when?using?"sigm"?activation?function?and?non-normalized?inputs.??
9. ????nn.momentum?????????????????????????=?0.5;??????????//??Momentum??
10. ????nn.scaling_learningRate?????????????=?1;????????????//??Scaling?factor?for?the?learning?rate?(each?epoch)??
11. ????nn.weightPenaltyL2??????????????????=?0;????????????//??L2?regularization??
12. ????nn.nonSparsityPenalty???????????????=?0;????????????//??Non?sparsity?penalty??
13. ????nn.sparsityTarget???????????????????=?0.05;?????????//??Sparsity?target??
14. ????nn.inputZeroMaskedFraction??????????=?0;????????????//??Used?for?Denoising?AutoEncoders??
15. ????nn.dropoutFraction??????????????????=?0;????????????//??Dropout?level?(http://www.cs.toronto.edu/~hinton/absps/dropout.pdf)??
16. ????nn.testing??????????????????????????=?0;????????????//??Internal?variable.?nntest?sets?this?to?one.??
17. ????nn.output???????????????????????????=?"sigm";???????//??output?unit?"sigm"?(=logistic),?"softmax"?and?"linear"??
18. ????//對每一層的網絡結構進行初始化，一共三個參數W,vW，p，其中W是主要的參數????
19. ????//vW是更新參數時的臨時參數，p是所謂的sparsity，(等看到代碼了再細講)??????
20. ???for?i?=?2?:?nn.n?????
21. ????????//?weights?and?weight?momentum??
22. ????????nn.W{i?-?1}?=?(rand(nn.size(i),?nn.size(i?-?1) 1)?-?0.5)?*?2?*?4?*?sqrt(6?/?(nn.size(i)? ?nn.size(i?-?1)));??
23. ????????nn.vW{i?-?1}?=?zeros(size(nn.W{i?-?1}));??
24. ??????????
25. ????????//?average?activations?(for?use?with?sparsity)??
26. ????????nn.p{i}?????=?zeros(1,?nn.size(i));?????
27. ????end??
28. end??

setup大概就這樣一個過程，下面就到了train了，打開NN ntrain.m

我們跳過那些檢驗傳入數據是否正確的代碼，直接到關鍵的部分

denoising 的部分請參考論文：Extracting and Composing Robust Features with Denoising Autoencoders

1. m?=?size(train_x,?1);??
2. //m是訓練樣本的數量??
3. //注意在調用的時候我們設置了opt，batchsize是做batch?gradient時候的大小??
4. batchsize?=?opts.batchsize;?numepochs?=?opts.numepochs;??
5. numbatches?=?m?/?batchsize;??//計算batch的數量??
6. assert(rem(numbatches,?1)?==?0,?"numbatches?must?be?a?integer");??
7. L?=?zeros(numepochs*numbatches,1);??
8. n?=?1;??
9. //numepochs是循環的次數??
10. for?i?=?1?:?numepochs??
11. ????tic;??
12. ????kk?=?randperm(m);??
13. ????//把batches打亂順序進行訓練，randperm(m)生成一個亂序的1到m的數組??
14. ????for?l?=?1?:?numbatches??
15. ????????batch_x?=?train_x(kk((l?-?1)?*?batchsize? ?1?:?l?*?batchsize),?:);??
16. ????????//Add?noise?to?input?(for?use?in?denoising?autoencoder)??
17. ????????//加入noise，這是denoising?autoencoder需要使用到的部分??
18. ????????//這部分請參見《Extracting?and?Composing?Robust?Features?with?Denoising?Autoencoders》這篇論文??
19. ????????//具體加入的方法就是把訓練樣例中的一些數據調整變為0，inputZeroMaskedFraction表示了調整的比例??
20. ????????if(nn.inputZeroMaskedFraction?~=?0)??
21. ????????????batch_x?=?batch_x.*(rand(size(batch_x))>nn.inputZeroMaskedFraction);??
22. ????????end??
23. ????????batch_y?=?train_y(kk((l?-?1)?*?batchsize? ?1?:?l?*?batchsize),?:);??
24. ????????//這三個函數??
25. ????????//nnff是進行前向傳播，nnbp是后向傳播，nnapplygrads是進行梯度下降??
26. ????????//我們在下面分析這些函數的代碼??
27. ????????nn?=?nnff(nn,?batch_x,?batch_y);??
28. ????????nn?=?nnbp(nn);??
29. ????????nn?=?nnapplygrads(nn);??
30. ????????L(n)?=?nn.L;??
31. ????????n?=?n? ?1;??
32. ????end??
33. ??????
34. ????t?=?toc;??
35. ????if?ishandle(fhandle)??
36. ????????if?opts.validation?==?1??
37. ????????????loss?=?nneval(nn,?loss,?train_x,?train_y,?val_x,?val_y);??
38. ????????else??
39. ????????????loss?=?nneval(nn,?loss,?train_x,?train_y);??
40. ????????end??
41. ????????nnupdatefigures(nn,?fhandle,?loss,?opts,?i);??
42. ????end??
43. ??????????
44. ????disp(["epoch?"?num2str(i)?"/"?num2str(opts.numepochs)?".?Took?"?num2str(t)?"?seconds"?".?Mean?squared?error?on?training?set?is?"?num2str(mean(L((n-numbatches):(n-1))))]);??
45. ????nn.learningRate?=?nn.learningRate?*?nn.scaling_learningRate;??
46. end??

nnff就是進行feedforward pass，其實非常簡單，就是整個網絡正向跑一次就可以了

當然其中有dropout和sparsity的計算

具體的參見論文“Improving Neural Networks with Dropout“和Autoencoders and Sparsity

1. function?nn?=?nnff(nn,?x,?y)??
2. //NNFF?performs?a?feedforward?pass??
3. //?nn?=?nnff(nn,?x,?y)?returns?an?neural?network?structure?with?updated??
4. //?layer?activations,?error?and?loss?(nn.a,?nn.e?and?nn.L)??
5. ??
6. ????n?=?nn.n;??
7. ????m?=?size(x,?1);??
8. ??????
9. ????x?=?[ones(m,1)?x];??
10. ????nn.a{1}?=?x;??
11. ??
12. ????//feedforward?pass??
13. ????for?i?=?2?:?n-1??
14. ????????//根據選擇的激活函數不同進行正向傳播計算??
15. ????????//你可以回過頭去看nnsetup里面的第一個參數activation_function??
16. ????????//sigm就是sigmoid函數,tanh_opt就是tanh的函數，這個toolbox好像有一點改變??
17. ????????//tanh_opt是1.7159*tanh(2/3.*A)??
18. ????????switch?nn.activation_function???
19. ????????????case?"sigm"??
20. ????????????????//?Calculate?the?unit"s?outputs?(including?the?bias?term)??
21. ????????????????nn.a{i}?=?sigm(nn.a{i?-?1}?*?nn.W{i?-?1}");??
22. ????????????case?"tanh_opt"??
23. ????????????????nn.a{i}?=?tanh_opt(nn.a{i?-?1}?*?nn.W{i?-?1}");??
24. ????????end??
25. ??????????
26. ????????//dropout的計算部分部分?dropoutFraction?是nnsetup中可以設置的一個參數??
27. ????????if(nn.dropoutFraction?>?0)??
28. ????????????if(nn.testing)??
29. ????????????????nn.a{i}?=?nn.a{i}.*(1?-?nn.dropoutFraction);??
30. ????????????else??
31. ????????????????nn.dropOutMask{i}?=?(rand(size(nn.a{i}))>nn.dropoutFraction);??
32. ????????????????nn.a{i}?=?nn.a{i}.*nn.dropOutMask{i};??
33. ????????????end??
34. ????????end??
35. ????????//計算sparsity，nonSparsityPenalty?是對沒達到sparsitytarget的參數的懲罰系數??
36. ????????//calculate?running?exponential?activations?for?use?with?sparsity??
37. ????????if(nn.nonSparsityPenalty>0)??
38. ????????????nn.p{i}?=?0.99?*?nn.p{i}? ?0.01?*?mean(nn.a{i},?1);??
39. ????????end??
40. ??????????
41. ????????//Add?the?bias?term??
42. ????????nn.a{i}?=?[ones(m,1)?nn.a{i}];??
43. ????end??
44. ????switch?nn.output???
45. ????????case?"sigm"??
46. ????????????nn.a{n}?=?sigm(nn.a{n?-?1}?*?nn.W{n?-?1}");??
47. ????????case?"linear"??
48. ????????????nn.a{n}?=?nn.a{n?-?1}?*?nn.W{n?-?1}";??
49. ????????case?"softmax"??
50. ????????????nn.a{n}?=?nn.a{n?-?1}?*?nn.W{n?-?1}";??
51. ????????????nn.a{n}?=?exp(bsxfun(@minus,?nn.a{n},?max(nn.a{n},[],2)));??
52. ????????????nn.a{n}?=?bsxfun(@rdivide,?nn.a{n},?sum(nn.a{n},?2));???
53. ????end??
54. ????//error?and?loss??
55. ????//計算error??
56. ????nn.e?=?y?-?nn.a{n};??
57. ??????
58. ????switch?nn.output??
59. ????????case?{"sigm",?"linear"}??
60. ????????????nn.L?=?1/2?*?sum(sum(nn.e?.^?2))?/?m;???
61. ????????case?"softmax"??
62. ????????????nn.L?=?-sum(sum(y?.*?log(nn.a{n})))?/?m;??
63. ????end??
64. end??

代碼：NN nbp.m

nnbp呢是進行back propagation的過程，過程還是比較中規中矩，和ufldl中的Neural Network講的基本一致

值得注意的還是dropout和sparsity的部分

1. if(nn.nonSparsityPenalty>0)??
2. ????pi?=?repmat(nn.p{i},?size(nn.a{i},?1),?1);??
3. ????sparsityError?=?[zeros(size(nn.a{i},1),1)?nn.nonSparsityPenalty?*?(-nn.sparsityTarget?./?pi? ?(1?-?nn.sparsityTarget)?./?(1?-?pi))];??
4. end??
5. ??
6. //?Backpropagate?first?derivatives??
7. if?i 1==n?%?in?this?case?in?d{n}?there?is?not?the?bias?term?to?be?removed???????????????
8. ????d{i}?=?(d{i? ?1}?*?nn.W{i}? ?sparsityError)?.*?d_act;?//?Bishop?(5.56)??
9. else?//?in?this?case?in?d{i}?the?bias?term?has?to?be?removed??
10. ????d{i}?=?(d{i? ?1}(:,2:end)?*?nn.W{i}? ?sparsityError)?.*?d_act;??
11. end??
12. ??
13. if(nn.dropoutFraction>0)??
14. ????d{i}?=?d{i}?.*?[ones(size(d{i},1),1)?nn.dropOutMask{i}];??
15. end??

dW{i}基本就是計算的gradient了，只是后面還要加入一些東西，進行一些修改

具體原理參見論文“Improving Neural Networks with Dropout“ 以及?Autoencoders and Sparsity的內容

代碼文件：NN napplygrads.m

1. for?i?=?1?:?(nn.n?-?1)??
2. ????if(nn.weightPenaltyL2>0)??
3. ????????dW?=?nn.dW{i}? ?nn.weightPenaltyL2?*?nn.W{i};??
4. ????else??
5. ????????dW?=?nn.dW{i};??
6. ????end??
7. ??????
8. ????dW?=?nn.learningRate?*?dW;??
9. ??????
10. ????if(nn.momentum>0)??
11. ????????nn.vW{i}?=?nn.momentum*nn.vW{i}? ?dW;??
12. ????????dW?=?nn.vW{i};??
13. ????end??
14. ??????????
15. ????nn.W{i}?=?nn.W{i}?-?dW;??
16. end??

有的話就加入weight Penalty，防止過擬合，然后再根據momentum的大小調整一下，最后改變nn.W{i}即可

nntest再簡單不過了，就是調用一下nnpredict，在和test的集合進行比較

1. function?[er,?bad]?=?nntest(nn,?x,?y)??
2. ????labels?=?nnpredict(nn,?x);??
3. ????[~,?expected]?=?max(y,[],2);??
4. ????bad?=?find(labels?~=?expected);??????
5. ????er?=?numel(bad)?/?size(x,?1);??
6. end??

代碼文件：NN npredict.m

1. function?labels?=?nnpredict(nn,?x)??
2. ????nn.testing?=?1;??
3. ????nn?=?nnff(nn,?x,?zeros(size(x,1),?nn.size(end)));??
4. ????nn.testing?=?0;??
5. ??????
6. ????[~,?i]?=?max(nn.a{end},[],2);??
7. ????labels?=?i;??
8. end??

max(nn.a{end},[],2); 是返回每一行的較大值以及所在的列數，所以labels返回的就是標號啦

(這個test好像是專門用來test 分類問題的，我們知道nnff得到最后的值即可)

? ?總的來說，神經網絡的代碼比較常規易理解，基本上和UFLDL中的內容相差不大

? ?只是加入了dropout的部分和denoising的部分

? ?本文的目的也不奢望講清楚這些東西，只是給出一個路線，可以跟著代碼去學習，加深對算法的理解和應用能力