Need help to understand the (backward) computational graph

[bssrdf]

Hi all,

I am trying to learn how ggml builds the computational graph, particularly the backward graph for gradient calculation. I am having a hard time understanding the nodes in the backward graph. Please see the below pictures. They were generated by tests/test1.c. The function is a simple y= a*x^2 and dy/dx = 2*a*x

This is the forward graph which I got the ideas.

However, I don't know how node_4 and node_5 are generated in the above backward graph. What do they correspond to mathematically?
Note, there are also node_3 and node_6 which are implicitly embedded in node_0 and node_1 since the former are just grad of the latter.
Thanks,

[ggerganov]

node_5 and node_4 correspond to the 2 ggml_mul ops in the backward pass for b:

ggml/tests/test1.c

Line 23 in 6474641

struct ggml_tensor * b = ggml_mul(ctx0, x, x);

ggml/src/ggml.c

Lines 14929 to 14945 in 6474641

	case GGML_OP_MUL:
	{
	if (src0->grad) {
	src0->grad =
	ggml_add_or_set(ctx,
	src0->grad,
	ggml_mul(ctx, src1, tensor->grad),
	zero_table);
	}
	if (src1->grad) {
	src1->grad =
	ggml_add_or_set(ctx,
	src1->grad,
	ggml_mul(ctx, src0, tensor->grad),
	zero_table);
	}
	} break;

To try to make it more clear, you can apply this patch and rerun the test:

diff --git a/src/ggml.c b/src/ggml.c
index a9e1ea9..363954d 100644
--- a/src/ggml.c
+++ b/src/ggml.c
@@ -14825,7 +14825,7 @@ void ggml_build_backward_gradient_checkpointing(
 // functions to change gradients considering the case that input a might be initial gradient with zero value
 
 static struct ggml_tensor * ggml_add_or_set(struct ggml_context * ctx, struct ggml_tensor * a, struct ggml_tensor * b, struct ggml_hash_set zero_table) {
-    if (ggml_hash_contains(zero_table, a)) {
+    if (false) {
         return b;
     } else {
         return ggml_add_impl(ctx, a, b, false);

The ggml_hash_contains() is an optimization to avoid extra x+y nodes in the accumulation of the gradients when the current value of the gradient is 0. Removing it, you will see a more explicit backward graph where node_3 is summed with a 0 (leaf_2) and stored as the gradient of node_1. In the original graph, this extra summation is avoided thanks to ggml_hash_contains(). Similar case for the gradient of node_0 where we avoid the intermediate summation of (grad) + node_5 -> node_6

[bssrdf]

@ggerganov, thanks for the clear explanation. Now I have a better understanding of ggml's computational graph.