|
|
--- |
|
|
license: gpl-3.0 |
|
|
--- |
|
|
|
|
|
|
|
|
# JiRack GPT-2 Initial Weights |
|
|
|
|
|
This file is strictly intended for saving the **initial weights (checkpoint)** of the JiRack GPT model. |
|
|
The model is **"clean"**: it contains no data and has never undergone any pre-training. |
|
|
- Powered by CMS Manhattan’s cutting-edge Vision-BERT architecture. |
|
|
|
|
|
It is engineered to be a maximally safe and robust base for **training from scratch** for specialized, smaller models, such as: |
|
|
|
|
|
- **SPAM Detection Systems** |
|
|
- **FRAUD Detection Models** |
|
|
- **Background Check (BG Check) Models** |
|
|
|
|
|
_A product of CMS Manhattan._ |
|
|
|
|
|
--- |
|
|
|
|
|
## Tokenizer Choices |
|
|
|
|
|
- For English: **GPT-2 Hugging Face tokenizer** |
|
|
- For multilingual use: **BERT tokenizer** from the Hugging Face library |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Architecture Details |
|
|
|
|
|
### GPT-2 Architecture (Classic, Transformer-like) |
|
|
|
|
|
``` |
|
|
CustomEmbedding |
|
|
FrozenSignatureLayer |
|
|
LearnedPositionalEmbedding |
|
|
[TransformerBlock] |
|
|
├── MultiHeadAttention |
|
|
├── LayerNorm |
|
|
├── LayerNorm |
|
|
├── FFN |
|
|
├── Linear |
|
|
├── Activation: GELU |
|
|
└── Linear |
|
|
LayerNorm |
|
|
Linear |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Checkpoint File Explanations |
|
|
|
|
|
### **12-head Attention Model** |
|
|
|
|
|
**Parameters:** |
|
|
- `VOCAB_SIZE = 50257` |
|
|
- `MODEL_DIM = 768` |
|
|
- `NUM_HEADS = 12` |
|
|
- `NUM_LAYERS = 6` |
|
|
- `MAX_SEQ_LEN = 8192` |
|
|
- `FFN_HIDDEN_DIM = 4 * MODEL_DIM` |
|
|
- `HEAD_DIM = MODEL_DIM // NUM_HEADS` |
|
|
|
|
|
**File:** |
|
|
`JiRack_H12_L6_V50257_D768_MSL8192_FF768x4.pt` |
|
|
|
|
|
--- |
|
|
|
|
|
### **6-head Attention Model** |
|
|
|
|
|
**Parameters:** |
|
|
- `VOCAB_SIZE = 50257` |
|
|
- `MODEL_DIM = 768` |
|
|
- `NUM_HEADS = 6` |
|
|
- `NUM_LAYERS = 6` |
|
|
- `MAX_SEQ_LEN = 8192` |
|
|
- `FFN_HIDDEN_DIM = 4 * MODEL_DIM` |
|
|
- `HEAD_DIM = MODEL_DIM // NUM_HEADS` |
|
|
|
|
|
**File:** |
|
|
`JiRack_H6_L6_V50257_D768_MSL8192_FF768x4.pt` |
|
|
|
|
|
|
|
|
|
|
|
- So About PyTorch script . You can use Pytorch script for AI classification task . |
|
|
- Do not Jit for Chatbot task . Use just state dict PyTorch for GPT (Chatbot) tasks |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
See other models with same patterns for read parameters |
|
|
|
|
|
# install tokenizer before run |
|
|
--- |
|
|
- mkdir -p tokenizer |
|
|
- wget -O tokenizer/tokenizer.json https://huggingface.co/gpt2/resolve/main/tokenizer.json |
|
|
- wget -O tokenizer/vocab.json https://huggingface.co/gpt2/resolve/main/vocab.json |
|
|
- wget -O tokenizer/merges.txt https://huggingface.co/gpt2/resolve/main/merges.txt |
|
|
- wget -O tokenizer/tokenizer_config.json https://huggingface.co/gpt2/resolve/main/tokenizer_config.json |
|
|
|
|
|
--- |
|
|
### JiRack RAG System |
|
|
- It is microservice architecture with API Gateway and Service Discovery |
|
|
- Framework Spring boot and Google embeddings model for JiRack RAG System with Chatbot and JiRach model deployment with docker scipt |
|
|
- video https://www.youtube.com/watch?v=vHClQu76kMc |
|
|
- RAG System https://bitbucket.org/cmsmanhattan/rag/src/main/ |
|
|
|
|
|
|
|
|
# Copyright Office |
|
|
|
|
|
- From: |
|
|
- cop-rc@loc.gov |
|
|
- To: |
|
|
- konstantin.grabko@yahoo.com |
|
|
|
|
|
- Mon, Dec 15 at 7:31 AM |
|
|
|
|
|
- THIS IS AN AUTOMATED EMAIL. PLEASE DO NOT REPLY. |
|
|
|
|
|
- Thank you for submitting your registration claim using the Electronic Copyright Office (ECO) System. |
|
|
|
|
|
- The following files were successfully uploaded for service request 1-15058193231 |
|
|
|
|
|
- File Name :jirack_gpt2_class_pytorch.zip |
|
|
- File Size :2993 KB |
|
|
- Date/Time :12/15/2025 7:27:48 AM |
|
|
|
|
|
- [THREAD ID: 1-6X1C895] |
|
|
|
|
|
- United States Copyright Office |
|
|
|
|
|
--- |
|
|
Welcome to ask to design your corp model over 33B or 70B or more parameters |
|
|
## |
|
|
|
|
|
CMS Manhattan |
|
|
Copyright © 2002–2026 |
