[onert] Separate forward and backward configuration of trainable ops #13033
Conversation
This PR separates forward and backward configuration of trainable ops in order to make possible disabling backward propagation of particular ops. ONE-DCO-1.0-Signed-off-by: Mateusz Bencer <m.bencer@partner.samsung.com>
| fn->configure(input_tensor, output_tensor, back_prop_input_tensor, back_prop_output_tensor, | ||
| node.param().alpha, node.param().beta, | ||
| convertElementwiseActivationType(node.param().op_type)); | ||
| auto convertToInferActivationType = [](const ir::operation::ElementwiseActivation::Type &type) { |
There was a problem hiding this comment.
Shouldn't alpha also be considered?
switch (op_type)
{
case ElementwiseActivationType::kReLU:
...
if ((alpha == std::numeric_limits<float>::infinity() || alpha == 6.0f) && beta == 0.f)
{
cpu::ops::ElementwiseActivationLayer::configure(
input, output, alpha, beta, cpu::ops::ElementwiseActivationType::kReLU);
There was a problem hiding this comment.
It's already checked inside ElementwiseActivationLayer::configureBackward (
Do you prefer to have it also here?
There was a problem hiding this comment.
Ah, the alpha is not an issue. Why do you define the convertToInferActivationType function again? Can't use convertElementwiseActivationType function?
There was a problem hiding this comment.
Here we're converting ir::operation::ElementwiseActivation::Type to cpu::ops::ElementwiseActivationType while convertElementwiseActivationType returns train::ops::ElementwiseActivationType. In the previous version this conversion has been made impicitly (
ONE/runtime/onert/backend/train/ops/ElementwiseActivationLayer.cc
Lines 53 to 61 in 8dd9291
There was a problem hiding this comment.
I understand. The train Kernel generator selectively calls the cpu and train configuration functions according to the op.isRequiredForBackward() value. In order to support this, these changes should be implmenet in the KernelGenerator. So, the activation type has to be converted to the cpu and train types depending on the situation.
|
Sorry for being late, I'm reading the code, right now. |
This PR separates forward and backward configuration of trainable ops in order to make possible disabling backward propagation of particular ops.
ONE-DCO-1.0-Signed-off-by: Mateusz Bencer m.bencer@partner.samsung.com
Draft: #12951
Issue: #12386