---
tags:
- kernel
---

# Activation

Activation is a python package that contains custom CUDA-based activation kernels, primarily targeting AMD GPUs.

- Currently implemented
  - [PolyNorm](https://arxiv.org/html/2411.03884v1)
  - [RMSNorm](https://docs.pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html)

## Usage

```python
import torch
from kernels import get_kernel

activation = get_kernel("motif-technologies/activation")

torch.set_default_device("cuda")
poly_norm = activation.layers.PolyNorm(eps=1e-6)
x = torch.randn(10, 10)

print(poly_norm(x))
```

## Performance

### PolyNorm

- Test cases are from the Motif LLM
- You can reproduce the results with:

```bash
cd tests
pytest --run-perf --do-plot
```

![PolyNorm Performance](./tests/perf.png)

## Pre-commit Hooks

This project uses [pre-commit](https://pre-commit.com/) to automatically check and format code before commits.

### Setup

1. Install pre-commit:

   ```bash
   pip install pre-commit
   ```

2. Install the git hooks:
  
```bash
   pre-commit install
   ```

Once installed, the configured hooks will run automatically on each commit.

### Included Hooks

The following tools are run via pre-commit:

- **[yapf](https://github.com/google/yapf)** – Python code formatter  
- **[typos](https://github.com/crate-ci/typos)** – Spell checker for common typos  
- **[isort](https://github.com/PyCQA/isort)** – Organizes and sorts Python imports  
- **[clang-format](https://clang.llvm.org/docs/ClangFormat.html)** – Formats C++/CUDA code (`--style=file`)  
- **[pymarkdown](https://github.com/jackdewinter/pymarkdown)** – Lints and auto-fixes Markdown files  
- **[actionlint](https://github.com/rhysd/actionlint)** – Validates GitHub Actions workflows  

### Usage

- Run all checks on the entire codebase:

   ```bash
   pre-commit run --all-files
   ```

- Run a specific hook (example: isort):
  
 ```bash
   pre-commit run isort --all-files
   ```