統計策略搜索強化學習方法及應用

趙婷婷，天津科技大學人工智能學院副教授，主要研究方向為人工智能、機器學習。
中國計算機協會（CCF） 會員、YOCSEF 會員、中國人工智能學會會員、人工智能學會模式識別專委會委員，2017年獲得天津市"131”創新型人才培養工程第二層次人選稱號。

1章 強化學習概述···························································································1 
1.1 機器學習中的強化學習··········································································1 
1.2 智能控制中的強化學習··········································································4 
1.3 強化學習分支··························································································8 
1.4 本書貢獻·······························································································11 
1.5 本書結構·······························································································12 
參考文獻········································································································14 

2章 相關研究及背景知識·············································································19 
2.1 馬爾可夫決策過程················································································19 
2.2 基於值函數的策略學習算法·································································21 
2.2.1 值函數·······················································································21 
2.2.2 策略迭代和值迭代····································································23 
2.2.3 Q-learning ··················································································25 
2.2.4 基於*小二乘法的策略迭代算法·············································27 
2.2.5 基於值函數的深度強化學習方法·············································29 
2.3 策略搜索算法························································································30 
2.3.1 策略搜索算法建模····································································31 
2.3.2 傳統策略梯度算法（REINFORCE算法）······························32 
2.3.3 自然策略梯度方法（Natural Policy Gradient）························33 
2.3.4 期望*大化的策略搜索方法·····················································35 
2.3.5 基於策略的深度強化學習方法·················································37 
2.4 本章小結·······························································································38 
參考文獻········································································································39 

3章 策略梯度估計的分析與改進·································································42 
3.1 研究背景·······························································································42 
3.2 基於參數探索的策略梯度算法（PGPE算法）···································44 
3.3 梯度估計方差分析················································································46 
3.4 基於*優基線的算法改進及分析·························································48 
3.4.1 *優基線的基本思想································································48 
3.4.2 PGPE算法的*優基線······························································49 
3.5 實驗·······································································································51 
3.5.1 示例···························································································51 
3.5.2 倒立擺平衡問題········································································57 
3.6 總結與討論····························································································58 
參考文獻········································································································60 

4章 基於重要性採樣的參數探索策略梯度算法··········································63 
4.1 研究背景·······························································································63 
4.2 異策略場景下的PGPE算法·································································64 
4.2.1 重要性加權PGPE算法·····························································65 
4.2.2 IW-PGPE算法通過基線減法減少方差····································66 
4.3 實驗結果·······························································································68 
4.3.1 示例···························································································69 
4.3.2 山地車任務················································································78 
4.3.3 機器人仿真控制任務································································81 
4.4 總結和討論····························································································88 
參考文獻········································································································89 

5章 方差正則化策略梯度算法·····································································91 
5.1 研究背景·······························································································91 
5.2 正則化策略梯度算法············································································92 
5.2.1 目標函數····················································································92 
5.2.2 梯度計算方法············································································94 
5.3 實驗結果·······························································································95 
5.3.1 數值示例····················································································95 
5.3.2 山地車任務··············································································101 
5.4 總結和討論··························································································102 
參考文獻······································································································103 

6章 基於參數探索的策略梯度算法的採樣技術········································105 
6.1 研究背景·····························································································105 
6.2 基於參數探索的策略梯度算法中的採樣技術····································107 
6.2.1 基線採樣··················································································108 
6.2.2 *優基線採樣··········································································109 
6.2.3 對稱採樣··················································································109 
6.2.4 對稱採樣··············································································111 
6.2.5 多模態對稱採樣··································································116 
6.2.6 SupSymPGPE 的獎勵歸一化··················································117 
6.3 數值示例實驗······················································································119 
6.3.1 平方函數··················································································120 
6.3.2 Rastrigin函數··········································································120 
6.4 本章總結·····························································································124 
參考文獻······································································································125 

7章 基於樣本有效重用的人形機器人的運動技能學習·····························127 
7.1 研究背景：真實環境下的運動技能學習···········································127 
7.2 運動技能學習框架··············································································128 
7.2.1 機器人的運動路徑和回報·······················································128 
7.2.2 策略模型··················································································129 
7.2.3 基於PGPE算法的策略學習方法···········································129 
7.3 有效重用歷史經驗··············································································130 
7.3.1 基於重要性加權的參數探索策略梯度算法 
（IW-PGPE算法）···································································130 
7.3.2 基於IW-PGPE算法的運動技能學習過程·····························131 
7.3.3 遞歸型IW-PGPE算法····························································132 
7.4 虛擬環境中的車桿擺動任務·······························································133 
7.5 籃球擊任務······················································································137 
7.6 討論與結論··························································································140 
參考文獻······································································································142 

8章 基於逆強化學習的藝術風格學習及水墨畫渲染·································145 
8.1 研究背景·····························································································145 
8.1.1 計算機圖形學背景··································································146 
8.1.2 人工智能背景··········································································147 
8.1.3 面向藝術風格化的渲染系統···················································148 
8.2 基於強化學習的筆刷智能體建模·······················································148 
8.2.1 動作的設計··············································································149 
8.2.2 狀態的設計··············································································150 
8.3 離線藝術風格學習階······································································151 
8.3.1 數據採集··················································································152 
8.3.2 基於逆強化學習的獎勵函數學習···········································153 
8.3.3 基於R-PGPE算法的渲染策略學習·······································154 
8.4 A4系統用戶界面················································································155 
8.5 實驗與結果··························································································157 
8.5.1 渲染策略學習結果··································································157 
8.5.2 基於IRL進行筆劃繪製的渲染結果·······································160 
8.6 本章小結·····························································································162 
參考文獻······································································································163