I am new to the TensorFlow framework and I'm trying to understand the fundamental building blocks of a computation graph. When should I use tf.Variable versus tf.constant for my machine learning parameters? I noticed that weights in a neural network are usually variables, but I’m not sure why I can't just update a constant during backpropagation. Can someone clarify this?
3 answers
Simply put: tf.constant is for data that stays the same. tf.Variable is for data that changes, like weights that get updated while your model learns from the dataset.
The primary distinction is mutability. A tf.constant is immutable; once it is created, its value cannot be changed. In contrast, a tf.Variable is designed to be updated via operations like assign_sub. During training, the GradientTape needs to track changes to optimize values. Since weights and biases must evolve to minimize the loss function, they must be Variables. Constants are typically used for fixed values like hyperparameters (learning rate) or data that doesn't change throughout the training loop execution.
Are you looking to implement a custom training loop using GradientTape, or are you mostly sticking to the high-level Keras API for your current model builds?
Christopher, the reason Robert asked is likely because custom loops make this distinction much more obvious. In Keras, layers handle variables automatically. However, if you're writing a manual optimization step, you'll find that trying to apply a gradient to a constant will return a None value. You must wrap any tensor you want to compute gradients for in a tf.Variable to ensure the graph properly tracks its state.
Exactly, Amanda! I'd just add that using constants where possible is actually better for performance because TensorFlow can optimize those parts of the static computation graph more effectively.