tensorflow41《TensorFlow实战》笔记-08-02 TensorFlow实现深度强化学习-估值网络 code

01强化学习简介

深度强化学习(Deep Reinforcement Learning)
环境状态(Environment State)
行动(Action)
奖励(Reward)
通过连续决策、采用最好的行动，获得最高的奖励

延迟奖励和未来利益

不像无监督学习那样完全没有学习目标，也不像监督学习那样有非常明确的目标。强化学习的目标一般是变化的、不明确的，设置可能不存在绝对正确的目标。

可应用深度神经网络做强化学习的训练模型

02策略网络(Policy Network)和估值网络(Value Network)

Polic-Based(或者 Policy Gradients)和Value-Based(或者Q-Learning)是强化学习中最重要的两类方法
Policy-Based的方法直接预测在某个环境状态下应该采取的Action；Value Based的方法则预测某个环境状态下所有Action的期望价值(Q值)，之后可以通过选择Q值最高的Action执行策略。

这两种方法的出发点和训练方式都有不同，Value Based方法适合仅有少量离散取值的Action的环境，而Policy-Based方法则更通用，适合Action种类非常多或者有连续取值的Action的环境。
结合深度学习后，Polic-Based的方法就成了策略网络(Policy Network)，而Value-Based的方法则成了估值网络(Value Network)。

DQN(Deep Q-Network)，即Value Network
学习Action对应的期望价值(Expected Utility)。Q-Learning学习中的期望价值指从当前的这一步到所有后续步骤，总共可以期望获取的最大价值(即Q值，也可称为Value)

Q-Learning的目标是求解函数 Q(s t ,a t )  ，即根据当前环境状态，估算Action的期望价值。

Q-Learning训练模型的基本思路也非常简单，它以(状态、行为、奖励、下一个状态)构成的元组 (s t ,a t ,r t+1 ,s t+1 )  为样本进行训练，其中 s t   为当前的状态，a为当前状态下的Action，Rt+1为在执行Action后获得的奖励， s t+1   为下一个状态。其中特征是 (s t ,a t )  ，而学习目标(即期望价值)则是 r t+1 +γ∗maxQ(s t+1 ,a)  ，这个学习目标既是当前Action获得的Reward加上下一个可获得的最大期望价值。

学习目标中包含Q-Learning的函数本身，所以这其中使用了递归求解的思想。下一步可获得的最大期望价值被乘以一个 γ  ，即衰减系数discount factor，这个参数决定了未来奖励在学习中的重要性。
Q new (s t ,a t )←(1−α)∗Q old (s t ,a t )+α∗(r t+ +γ∗maxQ a (s t+1 ,a)) 

03训练估值网络(DQN)技巧

3.1 引入卷积层，在神经网络前几层加入卷积让DQN能直接学习原始图像像素
3.2 Experience Replay，存储Agent的Experience(即样本)，并且每次训练时随机抽取一部分样本供给网络学习
3.3 Target DQN，使用第二个DQN网络来辅助计算目标Q值，让Q-Learning训练的目标保持平稳。我们让target DQN进行低频率或者缓慢的学习，这样它输出的目标Q值得波动也会比较小，可以减少对训练过程的影响。
3.4 Double DQN，传统的DQN通常会高估Action的Q值。如果这种高估不是均匀的，可能会导致本来次优的某个Action总被高估而超过了最大Q值得Action，再去获取这个Action在target DQN上的Q值。被选择的Q值，不一定总是最大的Q值，这样就避免了被高估的次优Action总是超过最优的Action。
3.5 Dueling DQN，Dueling DQN将Q值的函数 Q(S t ,a t )  拆分为两部分，一部分是静态的环境状态本身具有价值的 V(S t )  ，称为Value；另一部分是动态的通过选择某个Action额外带来的价值 A(a t )  ，称为Advantage。

04实际使用TensorFlow实现一个Deep-Q Network来学习自动玩游戏

# 《TensorFlow实战》08 TensorFlow实现深度强化学习
# win10 Tensorflow1.0.1 python3.5.3
# CUDA v8.0 cudnn-8.0-windows10-x64-v5.1
# filename:sz08.02.py # TensorFlow实现估值网络

import numpy as np
import random
import itertools
import scipy.misc
import matplotlib.pyplot as plt
import tensorflow as tf
import os
# %matplotlib inline

class gameOb():
    def __init__(self, coordinates, size, intensity, channel, reward, name):
        self.x = coordinates[0]
        self.y = coordinates[1]
        self.size = size
        self.intensity = intensity
        self.channel = channel
        self.reward = reward
        self.name = name

class gameEnv():
    def __init__(self, size):
        self.sizeX = size
        self.sizeY = size
        self.actions = 4
        self.objects = []
        a = self.reset()
        plt.imshow(a, interpolation="nearest")

    def reset(self):
        self.objects = []
        hero = gameOb(self.newPosition(), 1, 1, 2, None, 'hero')
        self.objects.append(hero)
        goal = gameOb(self.newPosition(), 1, 1, 1, 1, 'goal')
        self.objects.append(goal)
        hole = gameOb(self.newPosition(), 1, 1, 0, -1, 'fire')
        self.objects.append(hole)
        goal2 = gameOb(self.newPosition(), 1, 1, 1, 1, 'goal')
        self.objects.append(goal2)
        hole2 = gameOb(self.newPosition(), 1, 1, 0, -1, 'fire')
        self.objects.append(hole2)
        goal3 = gameOb(self.newPosition(), 1, 1, 1, 1, 'goal')
        self.objects.append(goal3)
        goal4 = gameOb(self.newPosition(), 1, 1, 1, 1, 'goal')
        self.objects.append(goal4)
        state = self.renderEnv()
        self.state = state
        return state

    def moveChar(self, direction):
        hero = self.objects[0]
        heroX = hero.x
        heroY = hero.y
        if direction == 0 and hero.y >= 1:
            hero.y -= 1
        if direction == 1 and hero.y <= self.sizeY - 2:
            hero.y += 1
        if direction == 2 and hero.x >= 1:
            hero.x -= 1
        if direction == 3 and hero.x <= self.sizeX - 2:
            hero.x += 1
        self.objects[0] = hero
    def newPosition(self):
        iterables = [range(self.sizeX), range(self.sizeY)]
        points = []
        for t in itertools.product(*iterables):
            points.append(t)
        currentPositions = []
        for objectA in self.objects:
            if (objectA.x, objectA.y) not in currentPositions:
                currentPositions.append((objectA.x, objectA.y))
        for pos in currentPositions:
            points.remove(pos)
        location = np.random.choice(range(len(points)), replace=False)
        return points[location]

    def checkGoal(self):
        others = []
        for obj in self.objects:
            if obj.name == 'hero':
                hero = obj
            else:
                others.append(obj)
        for other in others:
            if hero.x == other.x and hero.y == other.y:
                self.objects.remove(other)
                if other.reward == 1:
                    self.objects.append(gameOb(self.newPosition(), 1, 1, 1, 1, 'goal'))
                else:
                    self.objects.append(gameOb(self.newPosition(), 1, 1, 0, -1, 'fire'))
                return other.reward, False
        return 0.0, False

    def renderEnv(self):
        a = np.ones([self.sizeY + 2, self.sizeX + 2, 3])
        a[1:-1, 1:-1, :] = 0
        hero = None
        for item in self.objects:
            a[item.y+1:item.y+item.size+1,item.x+1:item.x+item.size+1, item.channel] = item.intensity
        b = scipy.misc.imresize(a[:,:,0], [84,84,1],interp='nearest')
        c = scipy.misc.imresize(a[:,:,1], [84,84,1],interp='nearest')
        d = scipy.misc.imresize(a[:,:,2], [84,84,1],interp='nearest')
        a = np.stack([b,c,d], axis=2)
        return a

    def step(self, action):
        self.moveChar(action)
        reward, done = self.checkGoal()
        state = self.renderEnv()
        return state, reward, done

env = gameEnv(size=5)

class Qnetwork():
    def __init__(self, h_size):
        self.scalarInput = tf.placeholder(shape=[None, 21168], dtype=tf.float32)
        self.imageIn = tf.reshape(self.scalarInput, shape=[-1, 84, 84, 3])
        self.conv1 = tf.contrib.layers.convolution2d(
            inputs = self.imageIn, num_outputs = 32,
            kernel_size=[8,8], stride=[4,4],
            padding='VALID', biases_initializer=None)
        self.conv2 = tf.contrib.layers.convolution2d(
            inputs=self.conv1, num_outputs=64, kernel_size=[4,4], stride=[2,2],
            padding='VALID', biases_initializer=None)
        self.conv3 = tf.contrib.layers.convolution2d(
            inputs=self.conv2, num_outputs=64, kernel_size=[3, 3], stride=[1,1],
            padding='VALID', biases_initializer=None)
        self.conv4 = tf.contrib.layers.convolution2d(
            inputs=self.conv3, num_outputs=512,
            kernel_size=[7,7], stride=[1,1],
            padding='VALID', biases_initializer=None)
        self.streamAC, self.streamVC = tf.split(self.conv4, 2, 3)
        self.streamA = tf.contrib.layers.flatten(self.streamAC)
        self.streamV = tf.contrib.layers.flatten(self.streamVC)
        self.AW = tf.Variable(tf.random_normal([h_size//2, env.actions]))
        self.VW = tf.Variable(tf.random_normal([h_size//2, 1]))
        self.Advantage = tf.matmul(self.streamA, self.AW)
        self.Value = tf.matmul(self.streamV, self.VW)
        self.Qout = self.Value + tf.subtract(self.Advantage, tf.reduce_mean(
            self.Advantage, reduction_indices=1, keep_dims=True))
        self.predict = tf.argmax(self.Qout, 1)
        self.targetQ = tf.placeholder(shape=[None], dtype=tf.float32)
        self.actions = tf.placeholder(shape=[None], dtype=tf.int32)
        self.actions_onehot = tf.one_hot(self.actions, env.actions, dtype=tf.float32)
        self.Q = tf.reduce_sum(tf.multiply(self.Qout, self.actions_onehot), reduction_indices=1)
        self.td_error = tf.square(self.targetQ - self.Q)
        self.loss = tf.reduce_mean(self.td_error)
        self.trainer = tf.train.AdamOptimizer(learning_rate=0.0001)
        self.updateModel = self.trainer.minimize(self.loss)

class experience_buffer():
    def __init__(self, buffer_size = 50000):
        self.buffer = []
        self.buffer_size = buffer_size

    def add(self, experience):
        if len(self.buffer) + len(experience) >= self.buffer_size:
            self.buffer[0:(len(experience) + len(self.buffer)) - self.buffer_size] = []
        self.buffer.extend(experience)

    def sample(self, size):
        return np.reshape(np.array(random.sample(self.buffer, size)), [size, 5])

def processState(states):
    return np.reshape(states, [21168])

def updateTargetGraph(tfVars, tau):
    total_vars = len(tfVars)
    op_holder = []
    for idx, var in enumerate(tfVars[0:total_vars//2]):
        op_holder.append(tfVars[idx+total_vars//2].assign((var.value()*tau) +
                                                          ((1-tau)*tfVars[idx+total_vars//2].value())))
    return op_holder

def updateTarget(op_holder, sess):
    for op in op_holder:
        sess.run(op)

batch_size = 32
update_freq = 4
y = .99
startE = 1
endE = 0.1
anneling_steps = 10000.
num_episodes = 10000
pre_train_steps = 10000
max_epLength = 50
load_model = False
path = "./dqn"
h_size=512
tau = 0.001

mainQN = Qnetwork(h_size)
targetQN = Qnetwork(h_size)
init = tf.global_variables_initializer()
trainables = tf.trainable_variables()
targetOps = updateTargetGraph(trainables, tau)
myBuffer = experience_buffer()
e = startE
stepDrop = (startE - endE)/anneling_steps
rList = []
total_steps = 0
saver = tf.train.Saver()
if not os.path.exists(path):
    os.makedirs(path)

with tf.Session() as sess:
    if load_model == True:
        print('Loading Model...')
        ckpt = tf.train.get_checkpoint_state(path)
        saver.restore(sess, ckpt.model_checkpoint_path)
    sess.run(init)
    updateTarget(targetOps, sess)
    for i in (range(num_episodes + 1)):
        episodeBuffer = experience_buffer()
        s = env.reset()
        s = processState(s)
        d = False
        rAll = 0
        j = 0
        while j < max_epLength:
            j += 1
            if np.random.rand(1) < e or total_steps < pre_train_steps:
                a = np.random.randint(0, 4)
            else:
                a = sess.run(mainQN.predict, feed_dict={mainQN.scalarInput:[s]})[0]
            s1,r,d = env.step(a)
            s1 = processState(s1)
            total_steps += 1
            episodeBuffer.add(np.reshape(np.array([s,a,r,s1,d]),[1,5]))

            if total_steps > pre_train_steps:
                if e > endE:
                    e -= stepDrop
                if total_steps % (update_freq) == 0:
                    trainBatch = myBuffer.sample(batch_size)
                    A = sess.run(mainQN.predict, feed_dict={mainQN.scalarInput:np.vstack(trainBatch[:,3])})
                    Q = sess.run(targetQN.Qout, feed_dict={targetQN.scalarInput:np.vstack(trainBatch[:,3])})
                    doubleQ = Q[range(batch_size), A]
                    targetQ = trainBatch[:, 2] + y*doubleQ
                    _ = sess.run(mainQN.updateModel, feed_dict={
                        mainQN.scalarInput:np.vstack(trainBatch[:,0]),
                        mainQN.targetQ:targetQ,
                        mainQN.actions:trainBatch[:,1]
                    })
                    updateTarget(targetOps, sess)

            rAll += r
            s = s1

            if d == True:
                break

            myBuffer.add(episodeBuffer.buffer)
            rList.append(rAll)
            if i > 0 and i % 25 == 0:
                print('episode', i, ', average reward of last 25 episode', np.mean(rList[-25:]))
            if i > 0 and i % 1000 == 0:
                saver.save(sess, path + '/model-' + str(i) + '.cptk')
                print("Saved Model")
    saver.save(sess, path + '/model-' + str(i) + '.cptk')

rMat = np.resize(np.array(rList), [len(rList)//100, 100])
rMean = np.average(rMat, 1)
plt.plot(rMean)
'''
episode 25 , average reward of last 25 episode 1.12
episode 25 , average reward of last 25 episode 1.0
episode 25 , average reward of last 25 episode 0.88
...
episode 10000 , average reward of last 25 episode 6.4
...
episode 10000 , average reward of last 25 episode 14.68
episode 10000 , average reward of last 25 episode 15.12
episode 10000 , average reward of last 25 episode 15.56
'''