Spaces:

konsa15
/

AGLU

Running

App Files Files Community

konsa15 commited on 18 days ago

Commit

3130f5a

verified ·

1 Parent(s): 82b404f

Update index.html

Browse files

Files changed (1) hide show

index.html +61 -43

index.html CHANGED Viewed

@@ -9,74 +9,92 @@
 	<body>
-<div align="center">
 <h1>Adaptive Parametric Activation </h1>
-<a href="https://kostas1515.github.io/">Konstantinos Panagiotis Alexandridis</a><sup>1</sup>,
-<a href="https://jiankangdeng.github.io/">Jiankang Deng</a><sup>1</sup>,
-<a href="https://cgi.csc.liv.ac.uk/~anguyen/">Anh Nguyen</a><sup>2</sup>,
-<a href="https://shanluo.github.io/">Shan Luo</a><sup>3</sup>,
-<sup>1</sup> Huawei Noah&#39;s Ark Lab,
 <sup>2</sup> University of Liverpool,
-<sup>3</sup> King&#39;s College London
 </div>
-<p>This is the official implementation of Adaptive Parametric Activation (APA) for ECCV2024 accepted paper. </p>
-<div align="center">
-<img src="./assets/unified_activations_combined.jpg"
-     alt="APA unifies most activation functions under the same formula."
-     style="float: left; margin-right: 10px; width:50%; height:auto"
-/>
-</div>
  <h3>Abstract</h3>
 <p>The activation function plays a crucial role in model optimisation, yet the optimal choice remains unclear. For example, the Sigmoid activation is the de-facto activation in balanced classification tasks, however, in imbalanced classification, it proves inappropriate due to bias towards frequent classes.  In this work, we delve deeper in this phenomenon by performing a comprehensive statistical analysis in the classification and intermediate layers of both balanced and imbalanced networks and we empirically show that aligning the activation function with the data distribution, enhances the performance in both balanced and imbalanced tasks. To this end, we propose the Adaptive Parametric Activation (APA) function, a novel and versatile activation function that unifies most common activation functions under a single formula. APA can be applied in both intermediate layers and attention layers, significantly outperforming the state-of-the-art on several imbalanced benchmarks such as ImageNet-LT, iNaturalist2018, Places-LT, CIFAR100-LT and LVIS and balanced benchmarks such as ImageNet1K, COCO and V3DET.</p>
 <h3>Definition</h3>
-<p>APA is defined as: $APA(z,λ,κ) = (λ exp(−κz) + 1) ^{\frac{1}{−λ}}$. APA unifies most activation functions under the same formula.</p>
-<p>APA can be used insed the intermediate layers using Adaptive Generalised Linear Unit (AGLU): $AGLU(z,λ,κ) = z APA(z,λ,κ)$. The derivatives of AGLU with respect to κ (top), λ (middle) and z (bottom) are shown below:
-<div align="center">
-  <img src="./assets/derivative_visualisations.jpg"
-     alt="APA unifies most activation functions under the same formula."
-     style="float: left; margin-right: 10px; width:50%; height:auto"/>
-</div>
 <h3> Simple Code implementation </h3>
-<pre><code class="lang-python"><span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Unified</span><span class="hljs-params">(nn.Module)</span>:</span>
     <span class="hljs-string">"""Unified activation function module."""</span>
-    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span><span class="hljs-params">(self, device=None, dtype=None)</span> -&gt; <span class="hljs-keyword">None</span>:</span>
         <span class="hljs-string">"""Initialize the Unified activation function."""</span>
         factory_kwargs = {<span class="hljs-string">"device"</span>: device, <span class="hljs-string">"dtype"</span>: dtype}
-        super().__init__()
         lambda_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs))
         kappa_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs))
-        self.softplus = nn.Softplus(beta=<span class="hljs-number">-1.0</span>)
         self.lambda_param = nn.Parameter(lambda_param)
         self.kappa_param = nn.Parameter(kappa_param)
-    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">forward</span><span class="hljs-params">(self, input: torch.Tensor)</span> -&gt; torch.Tensor:</span>
         <span class="hljs-string">"""Compute the forward pass of the Unified activation function."""</span>
-        l = torch.clamp(self.lambda_param, min=<span class="hljs-number">0.0001</span>)
-        p = torch.exp((<span class="hljs-number">1</span> / l) * self.softplus((self.kappa_param * input) - torch.log(l)))
         <span class="hljs-keyword">return</span> p  <span class="hljs-comment"># for AGLU simply return p*input</span>
 </code></pre>
-<h2 id="bibtex">BibTeX</h2>
-<pre><code class="lang-bibtex">@inproceedings{alexandridis2024adaptive,
-  <span class="hljs-attr">title={Adaptive</span> Parametric Activation},
-  <span class="hljs-attr">author={Alexandridis,</span> Konstantinos Panagiotis <span class="hljs-literal">and</span> Deng, Jiankang <span class="hljs-literal">and</span> Nguyen, Anh <span class="hljs-literal">and</span> Luo, Shan},
-  <span class="hljs-attr">booktitle={European</span> Conference on Computer Vision},
-  <span class="hljs-attr">pages={455--476},</span>
-  <span class="hljs-attr">year={2024},</span>
-  <span class="hljs-attr">organization={Springer}</span>
 }
 </code></pre>
-<h2 id="acknowledgements">Acknowledgements</h2>
-<p>This code uses <a href='https://pytorch.org/'>Pytorch</a> and the <a href='https://github.com/open-mmlab/mmdetection'>mmdet</a> framework. Thank you for your wonderfull work!</p>
 	</body>
 </html>

 	<body>
+<div class="relative min-h-[100px] rounded-b-lg border border-t-0 leading-tight dark:border-gray-800 dark:bg-gray-925">
+						<div class="py-4 px-4 sm:px-6 prose hf-sanitized hf-sanitized-LkzsSYSqbDSFqH1JQZWuD"><div class="not-prose bg-linear-to-t -mx-6 -mt-4 mb-8 max-h-[300px] min-w-full overflow-auto border-b from-gray-50 px-6 pb-5 pt-4 font-mono text-xs transition-all dark:from-gray-900 dark:to-gray-950"><div class="mb-2 inline-block rounded-lg border px-2 py-1 font-mono text-xs leading-none">metadata</div>
+			<pre><!-- HTML_TAG_START --><span class="hljs-attr">title:</span> <span class="hljs-string">AGLU</span>
+<span class="hljs-attr">emoji:</span> <span class="hljs-string">🏆</span>
+<span class="hljs-attr">colorFrom:</span> <span class="hljs-string">green</span>
+<span class="hljs-attr">colorTo:</span> <span class="hljs-string">red</span>
+<span class="hljs-attr">sdk:</span> <span class="hljs-string">static</span>
+<span class="hljs-attr">pinned:</span> <span class="hljs-literal">false</span>
+<!-- HTML_TAG_END --></pre></div>
+	<!-- HTML_TAG_START --><div align="center">
 <h1>Adaptive Parametric Activation </h1>
+<p><a rel="nofollow" href="https://kostas1515.github.io/">Konstantinos Panagiotis Alexandridis</a><sup>1</sup>,
+<a rel="nofollow" href="https://jiankangdeng.github.io/">Jiankang Deng</a><sup>1</sup>,
+<a rel="nofollow" href="https://cgi.csc.liv.ac.uk/~anguyen/">Anh Nguyen</a><sup>2</sup>,
+<a rel="nofollow" href="https://shanluo.github.io/">Shan Luo</a><sup>3</sup>,</p>
+<p><sup>1</sup> Huawei Noah's Ark Lab,
 <sup>2</sup> University of Liverpool,
+<sup>3</sup> King's College London</p>
+<p><a rel="nofollow" href="https://link.springer.com/chapter/10.1007/978-3-031-72949-2_26"><img alt="Static Badge" src="https://img.shields.io/badge/ECCV_2024-APA-blue"></a>
+<a rel="nofollow" href="https://arxiv.org/pdf/2407.08567"><img alt="Static Badge" src="https://img.shields.io/badge/arxiv-2407.08567-blue"></a>
+<a rel="nofollow" href="https://paperswithcode.com/sota/long-tail-learning-on-places-lt?p=adaptive-parametric-activation"><img alt="PWC" src="https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/adaptive-parametric-activation/long-tail-learning-on-places-lt"></a>
+<a rel="nofollow" href="https://paperswithcode.com/sota/instance-segmentation-on-lvis-v1-0-val?p=adaptive-parametric-activation"><img alt="PWC" src="https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/adaptive-parametric-activation/instance-segmentation-on-lvis-v1-0-val"></a>
+<a rel="nofollow" href="https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt?p=adaptive-parametric-activation"><img alt="PWC" src="https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/adaptive-parametric-activation/long-tail-learning-on-imagenet-lt"></a>
+<a rel="nofollow" href="https://paperswithcode.com/sota/long-tail-learning-on-inaturalist-2018?p=adaptive-parametric-activation"><img alt="PWC" src="https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/adaptive-parametric-activation/long-tail-learning-on-inaturalist-2018"></a></p>
 </div>
+<figure class="image text-center">
+  <img alt="APA activation" src="https://huggingface.co/spaces/konsa15/AGLU/resolve/main/assets/unified_activations_combined.jpg">
+  <figcaption> Figure 1: APA unifies most activation functions under the same formula.</figcaption>
+</figure>
  <h3>Abstract</h3>
 <p>The activation function plays a crucial role in model optimisation, yet the optimal choice remains unclear. For example, the Sigmoid activation is the de-facto activation in balanced classification tasks, however, in imbalanced classification, it proves inappropriate due to bias towards frequent classes.  In this work, we delve deeper in this phenomenon by performing a comprehensive statistical analysis in the classification and intermediate layers of both balanced and imbalanced networks and we empirically show that aligning the activation function with the data distribution, enhances the performance in both balanced and imbalanced tasks. To this end, we propose the Adaptive Parametric Activation (APA) function, a novel and versatile activation function that unifies most common activation functions under a single formula. APA can be applied in both intermediate layers and attention layers, significantly outperforming the state-of-the-art on several imbalanced benchmarks such as ImageNet-LT, iNaturalist2018, Places-LT, CIFAR100-LT and LVIS and balanced benchmarks such as ImageNet1K, COCO and V3DET.</p>
 <h3>Definition</h3>
+<p>The Adaptive Parametric Activation APA is defined as: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>A</mi><mi>P</mi><mi>A</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><mi>λ</mi><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><mo>−</mo><mi>κ</mi><mi>z</mi><mo stretchy="false">)</mo><mo>+</mo><mn>1</mn><msup><mo stretchy="false">)</mo><mfrac><mn>1</mn><mrow><mo>−</mo><mi>λ</mi></mrow></mfrac></msup></mrow>APA(z,λ,κ) = (λ exp(−κz) + 1) ^{\frac{1}{−λ}}</math></span><span aria-hidden="true" class="katex-html"><span class="base"><span style="height:1em;vertical-align:-0.25em;" class="strut"></span><span class="mord mathnormal">A</span><span style="margin-right:0.13889em;" class="mord mathnormal">P</span><span class="mord mathnormal">A</span><span class="mopen">(</span><span style="margin-right:0.04398em;" class="mord mathnormal">z</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">λ</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">κ</span><span class="mclose">)</span><span style="margin-right:0.2778em;" class="mspace"></span><span class="mrel">=</span><span style="margin-right:0.2778em;" class="mspace"></span></span><span class="base"><span style="height:1em;vertical-align:-0.25em;" class="strut"></span><span class="mopen">(</span><span class="mord mathnormal">λ</span><span class="mord mathnormal">e</span><span class="mord mathnormal">x</span><span class="mord mathnormal">p</span><span class="mopen">(</span><span class="mord">−</span><span class="mord mathnormal">κ</span><span style="margin-right:0.04398em;" class="mord mathnormal">z</span><span class="mclose">)</span><span style="margin-right:0.2222em;" class="mspace"></span><span class="mbin">+</span><span style="margin-right:0.2222em;" class="mspace"></span></span><span class="base"><span style="height:1.2312em;vertical-align:-0.25em;" class="strut"></span><span class="mord">1</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span style="height:0.9812em;" class="vlist"><span style="top:-3.3902em;margin-right:0.05em;"><span style="height:3em;" class="pstrut"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mopen nulldelimiter sizing reset-size3 size6"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span style="height:0.8443em;" class="vlist"><span style="top:-2.656em;"><span style="height:3em;" class="pstrut"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mathnormal mtight">λ</span></span></span></span><span style="top:-3.2255em;"><span style="height:3em;" class="pstrut"></span><span style="border-bottom-width:0.049em;" class="frac-line mtight"></span></span><span style="top:-3.384em;"><span style="height:3em;" class="pstrut"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span style="height:0.4035em;" class="vlist"><span></span></span></span></span></span><span class="mclose nulldelimiter sizing reset-size3 size6"></span></span></span></span></span></span></span></span></span></span></span></span></span>. APA unifies most activation functions under the same formula as shwon in Figure 1.</p>
+<p>APA can be used insed the intermediate layers using Adaptive Generalised Linear Unit (AGLU): <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>A</mi><mi>G</mi><mi>L</mi><mi>U</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo><mo>=</mo><mi>z</mi><mi>A</mi><mi>P</mi><mi>A</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo></mrow>AGLU(z,λ,κ) = z APA(z,λ,κ)</math></span><span aria-hidden="true" class="katex-html"><span class="base"><span style="height:1em;vertical-align:-0.25em;" class="strut"></span><span class="mord mathnormal">A</span><span class="mord mathnormal">G</span><span style="margin-right:0.10903em;" class="mord mathnormal">LU</span><span class="mopen">(</span><span style="margin-right:0.04398em;" class="mord mathnormal">z</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">λ</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">κ</span><span class="mclose">)</span><span style="margin-right:0.2778em;" class="mspace"></span><span class="mrel">=</span><span style="margin-right:0.2778em;" class="mspace"></span></span><span class="base"><span style="height:1em;vertical-align:-0.25em;" class="strut"></span><span style="margin-right:0.04398em;" class="mord mathnormal">z</span><span class="mord mathnormal">A</span><span style="margin-right:0.13889em;" class="mord mathnormal">P</span><span class="mord mathnormal">A</span><span class="mopen">(</span><span style="margin-right:0.04398em;" class="mord mathnormal">z</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">λ</span><span class="mpunct">,</span><span style="margin-right:0.1667em;" class="mspace"></span><span class="mord mathnormal">κ</span><span class="mclose">)</span></span></span></span>.
+The derivatives of AGLU with respect to κ (top), λ (middle) and z (bottom) are shown in Figure 2:</p>
+<figure class="image text-center">
+  <img alt="AGLU derivatives" src="https://huggingface.co/spaces/konsa15/AGLU/resolve/main/assets/derivative_visualisations.jpg">
+  <figcaption> Figure 2: The derivatives of AGLU with respect to κ (top), λ (middle) and z (bottom).</figcaption>
+</figure>
 <h3> Simple Code implementation </h3>
+<pre><code class="language-python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Unified</span>(nn.Module):
     <span class="hljs-string">"""Unified activation function module."""</span>
+    <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, device=<span class="hljs-literal">None</span>, dtype=<span class="hljs-literal">None</span></span>) -&gt; <span class="hljs-literal">None</span>:
         <span class="hljs-string">"""Initialize the Unified activation function."""</span>
         factory_kwargs = {<span class="hljs-string">"device"</span>: device, <span class="hljs-string">"dtype"</span>: dtype}
+        <span class="hljs-built_in">super</span>().__init__()
         lambda_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs))
         kappa_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs))
+        self.softplus = nn.Softplus(beta=-<span class="hljs-number">1.0</span>)
         self.lambda_param = nn.Parameter(lambda_param)
         self.kappa_param = nn.Parameter(kappa_param)
+    <span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, <span class="hljs-built_in">input</span>: torch.Tensor</span>) -&gt; torch.Tensor:
         <span class="hljs-string">"""Compute the forward pass of the Unified activation function."""</span>
+        l = torch.clamp(self.lambda_param, <span class="hljs-built_in">min</span>=<span class="hljs-number">0.0001</span>)
+        p = torch.exp((<span class="hljs-number">1</span> / l) * self.softplus((self.kappa_param * <span class="hljs-built_in">input</span>) - torch.log(l)))
         <span class="hljs-keyword">return</span> p  <span class="hljs-comment"># for AGLU simply return p*input</span>
 </code></pre>
+<h2 class="relative group flex items-center">
+	<a rel="nofollow" href="#bibtex" class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" id="bibtex">
+		<span class="header-link"><svg viewBox="0 0 256 256" preserveAspectRatio="xMidYMid meet" height="1em" width="1em" role="img" aria-hidden="true" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4"><path fill="currentColor" d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z"></path></svg></span>
+	</a>
+	<span>
+		BibTeX
+	</span>
+</h2>
+<pre><code class="language-bibtex">@inproceedings{alexandridis2024adaptive,
+  title={Adaptive Parametric Activation},
+  author={Alexandridis, Konstantinos Panagiotis and Deng, Jiankang and Nguyen, Anh and Luo, Shan},
+  booktitle={European Conference on Computer Vision},
+  pages={455--476},
+  year={2024},
+  organization={Springer}
 }
 </code></pre>
+<!-- HTML_TAG_END --></div>
+</div>
 	</body>
 </html>