yb1n/0409_deep_hw

Custom Transformer Encoder

PyTorch로 구현한 Transformer Encoder입니다. PositionWiseFeedForward, EncoderBlock, TransformerEncoder 모듈과 PretrainedConfig를 상속한 CustomConfig를 포함합니다.

Files

• configuration_custom.py: CustomConfig
• modeling_custom.py: PositionWiseFeedForward, EncoderBlock, TransformerEncoder
• config.json: 직렬화된 모델 설정(configuration)

Usage

import torch

from configuration_custom import CustomConfig
from modeling_custom import TransformerEncoder

config = CustomConfig.from_pretrained(".")
model = TransformerEncoder(config)

input_ids = torch.randint(0, config.vocab_size, (2, 16))
pad_mask = torch.ones(2, 16, dtype=torch.long)

outputs = model(input_ids, pad_mask)
print(outputs.shape)