Abstract
A new non-monotonic piecewise activation function, ULU, and its adaptive variant AULU are introduced, with AULU enabling separate learnable responses for positive and negative inputs and providing a metric to quantify model inductive bias, outperforming ReLU and Mish on image classification and object detection.
We propose ULU, a novel non-monotonic, piecewise activation function defined as {f(x;α_1),x<0; f(x;α_2),x>=0 }, where f(x;α)=0.5x(tanh(αx)+1),α>0. ULU treats positive and negative inputs differently. Extensive experiments demonstrate ULU significantly outperforms ReLU and Mish across image classification and object detection tasks. Its variant Adaptive ULU (AULU) is expressed as {f(x;β_1^2),x<0; f(x;β_2^2),x>=0 }, where β_1 and β_2 are learnable parameters, enabling it to adapt its response separately for positive and negative inputs. Additionally, we introduce the LIB (Like Inductive Bias) metric from AULU to quantitatively measure the inductive bias of the model.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper