Welcome to discuss about this book here!

[MathFoundationRL]

Hi there,

If you have any feedback about the book, you can leave a comment here. Thanks!

[etbox]

There excesses a right parentheses in the equation.

[MathFoundationRL]

Noted and thanks a lot!

[eccstartup]

In slides 9, page 22. I guess this is `exercise`.

[MathFoundationRL]

Yes you are right. Thanks for letting me know.

Shiyu

[L3Y1Q2]

Great book and course! Helped me reorganize some of the points of RL.

In the process of reading this book (ver 2022.8), I met some little confusion, probably clerical errors.

Thanks!

[MathFoundationRL]

Thanks for your detailed feedback.

• page 29: you are right.
• page 64: you are right
• page 91: you are right
• page 110 and more: could you specify more specifically?
• page111: you are right
• page128: you are right
• page133 and more: you are right

Overall, it's my pleasure that you read very carefully. Thanks for your feedback.

Shiyu

[L3Y1Q2]

Page 110:

Page 112:

[MathFoundationRL]

noted and thanks!

[businiaon]

if there should be a1, while a2 represents rightwards and a1 represents upwards.

[MathFoundationRL]

You are right. They should be a1. Thanks for the feedback!

[businiaon]

page 27, from Figure 2.3 if there should be p(r = 0|s1, a3) = 1 and p(r<> 0|s1, a3) = 0, not p(r = −1|s1, a3) = 1 and p(r <> −1|s1, a3) = 0 ?

[MathFoundationRL]

Yes, here r should be or not be equal to 0 (not -1)

[eccstartup]

第6章，page 20，这个好像不收敛的。

w = 0
g = lambda w: w**3 - 5

import random
for i in range(100):
    print(i, w)
    w = w - 1/(i+10) * (g(w) + random.gauss(0, 1))

[eccstartup]

a=1/k总是溢出

[ZiqianXie]

g的梯度是无界的, 收敛性用RM定理没法保证

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[MathFoundationRL]

要想在全局收敛，需要g的梯度有上界。

[ZiqianXie]

Dvoretzky 定理的证明只包括了 $\alpha_k, \beta_k$ 都由 $\mathcal{H}_k$ 决定的情况, 最好额外说明一下.

[MathFoundationRL]

谢谢，我会做相应修改

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[fanshJ]

6.2 Page 107
式子左边的 w_∞ − w_1 是不是应该改为 w_1 - w_∞，或者右边加负号？

[MathFoundationRL]

是的，谢谢

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[ritsuki1227]

Thank you very much for writing such a clear and helpful book!
Could you explain why the inequality shown on the image (bottom of the page 48) implies the contraction property of the Bellman Optimality Equation?
Maybe I don't understand some properties of the sup-norm.

[MathFoundationRL]

This inequality is the same as the definition of a contraction mapping. It shows that the function f(v) is contractive, (not the Bellman optimality equation is contractive).

[ritsuki1227]

Thank you for your comment. My understanding is that the definition states that $f$ is contractive iff

$$||f(v_1) - f(v_2)||\leq \gamma ||v_1 - v_2|| \quad \exists \gamma \in (0, 1) \forall v_1, v_2 \in \mathbb{R}^d,$$

which is not obviously the same as the inequality in the image above for me ($||.||$ denotes vector norm whereas $||.||_\infty$ denotes supremum norm). I guess I'm missing some basic concepts of sup-norm, but is it true that the implication below holds?

$$||f(v_1) - f(v_2)||_\infty \leq \gamma ||v_1 - v_2||_\infty \implies ||f(v_1) - f(v_2)|| \leq \gamma ||v_1 - v_2|| .$$

Also, I've found a minor typo on the section 3.3.3 on page 44. "for any $x_1, x_2 \in \mathbb{R}$" should be "for any $x_1, x_2 \in \mathbb{R}^d$".

[MathFoundationRL]

Now I understand. First, in the definition of contraction mapping, $||\cdot||$ refers to a general vector norm. That is, if there exists any type of vector norm so that this inequality is valid, then the function is contractive. Second, in the proof it is a special $L_\infty$ vector norm (it is still a vector norm). Third, in Theorem 3.2, the norm should also be the $L_\infty$ norm (but I wrote it as a general norm). Hope this answers your question. I will also revise to make it clearer.

You are right about the typo. Thanks!

[ritsuki1227]

Thank you very much. It is clear to me now!

[ZiqianXie]

Some notation inconsistency:

Page 196, the matrix-vector form should have $\nabla_\theta v_\pi\in\mathbb{R}^{m|S|}$ instead of $n|S|$ since $m$ is the dimension of $\theta$.

And later in 9.12, $n$ is being used to denote $|S|$

[MathFoundationRL]

You are right. Thanks!

By the way, I didn't expect but am happy to see that some readers reads the proof details carefully.

[bayncat]

Hello. Is there a missing "αk" before the summation symbol?

[MathFoundationRL]

Yes, you are right. Thanks!

[SakuraToErii]

I think this "not" may need to be deleted.

[SakuraToErii]

Sorry, I forgot to say it is on page 88.

[MathFoundationRL]

Yes, you are right. It will be corrected in the next version. Thanks

[Rino-Li]

In page. 85, the second paragraph of the subsection A comprehensive example: Episode length and sparse reward, "See, for example, Figure 5.3(h)" should be "5.3(a)", because you mentioned that the episode length is 1.

[Rino-Li]

It is a perfect course to study, thanks to teacher Zhao.

[MathFoundationRL]

Thanks. What I meant is actually correct, but it may be confusing. I will revise it in the future.

[SakuraToErii]

On page 214, this "z" may need to be modified as "x".

[MathFoundationRL]

Thanks. This is a new typo that has not been noticed so far. I will correct it.

[SakuraToErii]

On page 157, I think the "column" may need to be modified as "row".

[MathFoundationRL]

Thanks for the feedback. But this statement is correct. The ith row is the probabilities of reaching state i when starting every state.

[SakuraToErii]

Thank you and I have another question: All of this means that [P_π]ij is from state i to state j, in other words, i is the row and j is the column.......

[MathFoundationRL]

Now I understand. It is correct to say "column", but the meaning of the entries in the ith column is the probabilities reaching state i from different starting states. Thanks for point out the problem! You read very carefully!

[SakuraToErii]

Thank you for your patience. I read your book carefully due to its high quality and clear context. I am very grateful that I have been learning with such a cool book in RL.

[MathFoundationRL]

Thank you! I am finalizing the manuscript of the book. The final version will be better than the first draft. I will it will be cooler:) Please stay tuned.

[renjieDLUT]

"the action value of s_t"我理解应该是state value吧

[MathFoundationRL]

Yes, it should be state value. Thanks.

[SakuraToErii]

Hello! I'd love to know if the Truncated Policy Iteration algorithm is not a model-based but a Dynamic programming algorithm.(because you said all introduced algorithms in this book are model-free.) and ----
As I understand it, I think:

• a model-based algorithm computes an approximation of the real state transition matrix by using the collected experiences from the environment. Hence, the environment becomes known, and we can do other things based on it.
• while a model-free algorithm uses the experiences to do policy evaluation and improvement directly.

Am I understanding you correctly?

[MathFoundationRL]

Hi there,

Thanks for using my book for study. Here is my response.

1. I don't think I said all the algorithms are model-free:-) For example, the algorithms in chapter 4 are not model-free.

2. I also use model-based carefully because I know it may be confusing. It can refer to the case where we know the model exactly. But it also refers to the case where we use data to build the model and then use the model for learning.

3. The Truncated policy iteration algorithm still requires the model.

Hope it helps.

Shiyu

[SakuraToErii]

Thanks for your reply!
Let's see this picture, so the right explanation is that

• the algorithms in Chapter 4 are model-based Dynamic programming algorithms but not model-based RL algorithms?
• and the RL algorithms introduced in this book are all model-free algorithms?

[MathFoundationRL]

Since the algos in chapter 4 are called DP algos instead of RL, it is reasonable to say that the RL algorithms introduced in this chapter are model-free. I think I will remove this sentence to avoid confusions

[SakuraToErii]

Thank you very much for solving my problem!

[SakuraToErii]

Hello, I'd love to know how to explain this situation:
When the β is always positive(assuming that all the rewards are positive and taking good action will get the relatively bigger value of the rewards, then all the q-values are also positive, β also will be positive), the probabilities of choosing all (s, a) will be enhanced, but it is impossible.

[SakuraToErii]

But when I was typing those words, I found that the π(a|s,θ) is expressed by a softmax function. So maybe the π in the equation above should be the ${h(s,a,\theta)}$ or other formula?
In the situation I mentioned above, although all the probabilities are increased, through the softmax function, the probability of (s, a) having a relatively higher q-value is increased and that of the relatively lower q-value is decreased.

[MathFoundationRL]

It's not to enhance the probability to enhance all (s,a), but the present (s_t,a_t)

[SakuraToErii]

Yes, but in my opinion, when all the q-values are positive, all the probabilities of (s, a) are enhanced.

[MathFoundationRL]

The value of theta is changed indeed, and the probability of all s,a may be changed (but not enhanced). Only the probability for st,at (some close s,a) is enhanced (this can be analyzed)

[SakuraToErii]

Thanks for your reply, let me think more deeply about that.

[SakuraToErii]

Hello dear Professor Zhao, I think this "+" symbol may should be a "=", this equation has "tortured" me for one month, hahahaha.

[MathFoundationRL]

is incorrect, because s is not a function of theta. We should calculate the derivative of q with respect to theta

[MathFoundationRL]

For the equations I gave about, it is better to calculate \partial f(x,y)/\partial \theta. Then, it is clearer.

[Atsan1006]

[Atsan1006]

[MathFoundationRL]

I already mentioned: "For the equations I gave about, it is better to calculate \partial f(x,y)/\partial \theta. Then, it is clearer." I don't see any inaccuracy in my derivation. Sorry that I cannot reply more.

[AIexanderDicke]

Thanks for the nice write-up!

There is a small (but important) typo in Algorithm 4.1:

While $v_k$ has not converged in the sense that $\|v_k − v_{k-1}\|$ is greater than a predefined
small threshold, for the $k$th iteration, do

should read

While $v_k$ has not converged in the sense that $\|v_k − v_{k-1}\|$ is smaller than a predefined
small threshold, for the $k$th iteration, do

[MathFoundationRL]

Thanks for letting me know. What I am referring to is that "not converged" means "xx is greater than xxx". I understand your meaning. Let me think if there is a better way

[xuxiaolin0926]

163页 $s_{2}$ 与 $s_{3}$ 的typo。

感谢您的书。

[MathFoundationRL]

Yes, s2 should be s3. Thanks.

[Annihillusion]

Thank you so much for writing such a helpful book!
Could you further explain why this equation hold? It may not seen so trivial to me. Thank you in advance!

What's more, in Page 205, it might be a little mistake.

[MathFoundationRL]

Hi there, for the first underlined equation: d^TP=d^T, this is due the property of the stationary distribution d. You can check more about d in the book.

About the second circled equation: \sum d(s)=1. This is why it holds.

[Annihillusion]

Thank you so much for your prompt and patient reply! I understand these two equations now.

[yapeg]

Page 172. Chapter 8. Algorithm 8.1.
'A function $\hat{v}(s,w)$ that is $\cancel{a}$ differentiable in $w$.'

[MathFoundationRL]

Thanks, it is indeed a typo

[HaomingFu]

Page 53, Section 3.3.3, Box 3.1. I think there is a typo.
In the sentence "We prove that the consequence $\{x_k\}_{k=1}^\infty$ with $x_k = f(x_{k=1})$ is convergent", the "consequence" should be "sequence".

[MathFoundationRL]

Thank you. It is indeed a typo.

[wyanyue]

I think there's a symbol \gamma missing here. There's the same typo in slides.

[MathFoundationRL]

Thank you. It is indeed a typo!

[lhz99]

print error
on the right graph, I think it is a “SGD(m=1)" not “SG(m=1)"

[L-lorish]

it is actually Pr(s| s') because in following proof, the author has change s to s' and s' to s, equivalently.

[lhz99]

But why need to change here? I'am confused

[MathFoundationRL]

It is mainly for the notational convenience.

[lhz99]

row 2 and 3 in the derivation of tr[var(X)] is repeated

[MathFoundationRL]

Many thanks for the feedback!

[L-lorish]

I am confused of the result of derivative.if adding the explanation is much better!

[L-lorish]

The derivative formula may be derived or inspired by the result.lol

[MathFoundationRL]

(10.17) is a result of a standard derivative operation. If you have any specific question, you can ask.

[L-lorish]

I have delved further into the matter and discovered that this derivative is indeed correct. However, upon my first look, I found myself puzzled by the derivative of $q_{\mu}$. Later, I came to understand that this confusion arose because initially, $v_{\mu}(s)$ had not been expanded.

$q_{\mu}(s,\mu(s)) = \sum_{r}P(r|s,\mu(s))r + \gamma \sum_{s^{\prime}}P(s^{\prime}|s,\mu(s))v_{\mu}(s^{\prime})$

Combined with $v_{\mu}(s) = q_{\mu}(s,\mu(s))$, it clearly indicates that $q_{\mu}(s)$ is a function of both $q_{\mu}$ and $\mu(s)$

[lhz99]

There is total derivative, right?