paradox model

README.md

@@ -21,11 +21,11 @@ Phi-4-mini-flash-reasoning is a lightweight open model built upon synthetic data
 The model belongs to the Phi-4 model family and supports 64K token context length. 
  
 📰 [Phi-4-mini-flash-reasoning Blog](https://azure.microsoft.com/en-us/blog/reasoning-reimagined-introducing-phi-4-mini-flash-reasoning/) <br>
-📖 [Phi-4-mini-flash-reasoning Paper](https://arxiv.org/abs/2507.06607) | [HF Paper](https://huggingface.co/papers/2507.06607) <br>
+📖 [Phi-4-mini-flash-reasoning Paper](https://aka.ms/flashreasoning-paper) | [HF Paper](https://huggingface.co/papers/2507.06607) <br>
 📚 [Training Codebase](https://github.com/microsoft/ArchScale) <br>
 👩‍🍳 [Phi Cookbook](https://github.com/microsoft/PhiCookBook) <br>
 🏡 [Phi Portal](https://azure.microsoft.com/en-us/products/phi) <br>
-🚀 vLLM Inference: V0: [PR](https://github.com/vllm-project/vllm/pull/20702) | [Branch](https://github.com/congcongchen123/vllm/tree/congcongchen/phi4-mini-shadow) V1: [PR](https://github.com/vllm-project/vllm/pull/23996) <br>
+🚀 [vLLM Inference](https://github.com/vllm-project/vllm/pull/20702) <br>
 🖥️ Try It [Azure](https://ai.azure.com/explore/models/Phi-4-mini-flash-reasoning/version/1/registry/azureml-phi-prod) [Nvidia NIM](https://build.nvidia.com/microsoft/phi-4-mini-flash-reasoning)<br>
  
 
@@ -236,7 +236,4 @@ Benchmark datasets
 We evaluate the model with three of the most popular math benchmarks where the strongest reasoning models are competing together. Specifically:
 + Math-500: This benchmark consists of 500 challenging math problems designed to test the model's ability to perform complex mathematical reasoning and problem-solving.
 + AIME 2024/AIME 2025: The American Invitational Mathematics Examination (AIME) is a highly regarded math competition that features a series of difficult problems aimed at assessing advanced mathematical skills and logical reasoning. We evaluate the models on the problems from both 2024 and the year 2025 examinations. 
-+ GPQA Diamond: The Graduate-Level Google-Proof Q&A (GPQA) Diamond benchmark focuses on evaluating the model's ability to understand and solve a wide range of mathematical questions, including both straightforward calculations and more intricate problem-solving tasks.
-
-## Data Summary
-https://huggingface.co/microsoft/Phi-4-mini-flash-reasoning/blob/main/data_summary_card.md
++ GPQA Diamond: The Graduate-Level Google-Proof Q&A (GPQA) Diamond benchmark focuses on evaluating the model's ability to understand and solve a wide range of mathematical questions, including both straightforward calculations and more intricate problem-solving tasks.

config.json

@@ -26,6 +26,12 @@
     "num_key_value_heads": 20,
     "resid_pdrop": 0.0,
     "sliding_window": 512,
+    "layer_types": [
+      "full_attention", "sliding_attention", "full_attention", "sliding_attention", "full_attention", "sliding_attention", "full_attention", "sliding_attention",
+      "full_attention", "sliding_attention", "full_attention", "sliding_attention", "full_attention", "sliding_attention", "full_attention", "sliding_attention",
+      "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention",
+      "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention", "full_attention"
+    ],
     "torch_dtype": "bfloat16",
     "tie_word_embeddings": true,
     "transformers_version": "4.46.1",

configuration_phi4flash.py

@@ -112,6 +112,7 @@ class Phi4FlashConfig(PretrainedConfig):
         bos_token_id=1,
         eos_token_id=2,
         sliding_window=2047,
+        layer_types=None,
         mb_per_layer= 2,
         mamba_d_state=16,
         mamba_d_conv=4,
@@ -141,11 +142,16 @@ class Phi4FlashConfig(PretrainedConfig):
         self.use_cache = use_cache
         self.rope_theta = rope_theta
         self.mb_per_layer = mb_per_layer
-        self.sliding_window = [
-            sliding_window if layer_idx < num_hidden_layers // 2 and layer_idx % 2 == 1 else None
-            for layer_idx in range(num_hidden_layers)
-        ]
+        self.sliding_window = sliding_window
+        self.layer_types = layer_types
     
+        if self.layer_types is None:
+            is_sliding = lambda i: i < num_hidden_layers // 2 and i % 2 == 1
+            self.layer_types = [
+                "sliding_attention" if is_sliding(layer_idx) else "full_attention"
+                for layer_idx in range(num_hidden_layers)
+            ]
+
         self.mamba_d_state = mamba_d_state
         self.mamba_d_conv = mamba_d_conv
         self.mamba_expand = mamba_expand
@@ -170,4 +176,4 @@ class Phi4FlashConfig(PretrainedConfig):
             else:
                 layer_block_type = "mamba"
             layer_block_types.append(layer_block_type)
-        return layer_block_types
+        return layer_block_types

data_summary_card.md

@@ -1,149 +0,0 @@
- 
-
- 
-
-# Data Summary for microsoft_Phi-4-mini-reasoning, phi-4-mini-instruct, phi-4-mini-flash-reasoning 
-
- 
-
- 
-
-## 1. General information 
-
-**1.0.1 Version of the Summary:** 1.0 
-
- 
-
-**1.0.2 Last update:** 10-Dec-2025 
-
- 
-
-## 1.1 Model Developer Identification 
-
-**1.1.1 Model Developer name and contact details:** Microsoft Corporation at One Microsoft Way, Redmond, WA 98052. Tel: 425-882-8080 
-
- 
-
-## 1.2 Model Identification 
-
-**1.2.1 Versioned model name(s):** Phi-4-mini-reasoning, Phi-4-mini-instruct, Phi-4-mini-flash-reasoning 
-
- 
-
-**1.2.2 Model release date:** 29-Apr-2025  
-
- 
-
-## 1.3 Overall training data size and characteristics 
-
-### 1.3.1 Size of dataset and characteristics 
-
-**1.3.1.A Text training data size:** 1 billion to 10 trillion tokens 
-
- 
-
-**1.3.1.B Text training data content:** The training data for Phi-4-mini-reasoning consists exclusively of synthetic mathematical content generated by a stronger and more advanced reasoning model, Deepseek-R1. The objective is to distill knowledge from this model. This synthetic dataset comprises over one million diverse math problems spanning multiple levels of difficulty (from middle school to Ph.D. level). For each problem in the synthetic dataset, eight distinct solutions (rollouts) were sampled, and only those verified as correct were retained. 
-
- 
-
-**1.3.1.C Image training data size:** Not applicable. Images are not part of the training data 
-
- 
-
-**1.3.1.D Image training data content:** Not applicable 
-
- 
-
-**1.3.1.E Audio training data size:** Not applicable. Audio is not part of the training data 
-
- 
-
-**1.3.1.F Audio training data content:** Not applicable 
-
- 
-
-**1.3.1.G Video training data size:** Not applicable. Videos are not part of the training data 
-
- 
-
-**1.3.1.H Video training data content:** Not applicable 
-
- 
-
-**1.3.1.I Other training data size:** Not applicable 
-
- 
-
-**1.3.1.J Other training data content:** Not applicable 
-
- 
-
-**1.3.2 Latest date of data acquisition/collection for model training:** February 2025 
-
- 
-
-**1.3.3 Is data collection ongoing to update the model with new data collection after deployment?** No 
-
- 
-
-**1.3.4 Date the training dataset was first used to train the model:** February 2025 
-
- 
-
-**1.3.5 Rationale or purpose of data selection:** Datasets consist of synthetic mathematical problems and verified solutions generated by a stronger reasoning model to distill high-quality reasoning patterns and improve math problem-solving performance across difficulty levels 
-
- 
-
-## 2. List of data sources 
-
-### 2.1 Publicly available datasets 
-
-**2.1.1 Have you used publicly available datasets to train the model?** Yes 
-
- 
-
-## 2.2 Private non-publicly available datasets obtained from third parties 
-
-### 2.2.1 Datasets commercially licensed by rights holders or their representatives 
-
-**2.2.1.A Have you concluded transactional commercial licensing agreement(s) with rights holder(s) or with their representatives?** Not applicable 
-
- 
-
-### 2.2.2 Private datasets obtained from other third-parties 
-
-**2.2.2.A Have you obtained private datasets from third parties that are not licensed as described in Section 2.2.1, such as data obtained from providers of private databases, or data intermediaries?** No 
-
- 
-
-## 2.3 Personal Information 
-
-**2.3.1 Was personal data used to train the model?** Microsoft follows all relevant laws and regulations pertaining to personal information 
-
- 
-
-## 2.4 Synthetic data 
-
-**2.4.1 Was any synthetic AI-generated data used to train the model?** Yes 
-
- 
-
-## 3. Data processing aspects 
-
-### 3.1 Respect of reservation of rights from text and data mining exception or limitation 
-
-**3.1.1 Does this dataset include any data protected by copyright, trademark, or patent?** Microsoft follows all required regulations and laws for processing data protected by copyright, trademark, or patent 
-
- 
-
-## 3.2 Other information 
-
-**3.2.1 Does the dataset include information about consumer groups without revealing individual consumer identities?** Microsoft follows all required regulations and laws for protecting consumer identities 
-
- 
-
-**3.2.2 Was the dataset cleaned or modified before model training?** Yes 
-
- 
-
- 

modeling_phi4flash.py

@@ -129,7 +129,7 @@ def _get_cache(
         cache_to_check = self._cache.self_attention_cache if requires_cross_attention_cache else self._cache
 
     if cache_implementation == "sliding_window":
-        max_cache_len = min(self.config.sliding_window[1], max_cache_len)
+        max_cache_len = min(self.config.sliding_window, max_cache_len)
 
     need_new_cache = (
         not hasattr(self, "_cache")
@@ -243,7 +243,7 @@ class SambaYCache(Cache):
         sliding_cache_shape = (
             self.max_batch_size,
             self.num_key_value_heads,
-            min(config.sliding_window[1], max_cache_len),
+            min(config.sliding_window, max_cache_len),
             self.head_dim,
         )
         conv_cache_shape = (self.max_batch_size, intermediate_size, conv_kernel_size)
@@ -573,7 +573,7 @@ class SambaYFlashAttention2(SambaYAttention):
             key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
             value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
 
-            use_sliding_windows = self.config.sliding_window is not None and self.config.sliding_window[self.layer_idx] is not None
+            use_sliding_windows = self.config.sliding_window is not None and self.config.layer_types[self.layer_idx] == "sliding_attention"
 
             if past_key_value is not None:
 
@@ -710,8 +710,8 @@ class SambaYFlashAttention2(SambaYAttention):
                     softmax_scale=softmax_scale,
                     causal=causal,
                     window_size=(
-                        self.config.sliding_window[self.layer_idx] -1,
-                        self.config.sliding_window[self.layer_idx] -1,
+                        self.config.sliding_window -1,
+                        self.config.sliding_window -1,
                     ),
                 )
 
@@ -735,8 +735,8 @@ class SambaYFlashAttention2(SambaYAttention):
                     softmax_scale=softmax_scale,
                     causal=causal,
                     window_size=(
-                        self.config.sliding_window[self.layer_idx] -1,
-                        self.config.sliding_window[self.layer_idx] -1,
+                        self.config.sliding_window -1,
+                        self.config.sliding_window -1,
                     ),
                 )
 
@@ -1085,9 +1085,9 @@ class SambaYDecoderLayer(nn.Module):
             residual = residual.to(torch.float32) 
             self_attn_weights = None
         else:
-            if self.config.sliding_window is not None and self.config.sliding_window[self.layer_idx] is not None and attention_mask is not None:  # efficient SDPA and no padding
+            if self.config.sliding_window is not None and self.config.layer_types[self.layer_idx] == "sliding_attention" and attention_mask is not None:  # efficient SDPA and no padding
                 if past_key_value is not None and cache_position[0] > 0:  # when decoding
-                    attention_mask = attention_mask[:, -self.config.sliding_window[self.layer_idx]:]
+                    attention_mask = attention_mask[:, -self.config.sliding_window:]
             #hidden_states = self.input_layernorm2(hidden_states.to(dtype=self.input_layernorm2.weight.dtype))
             # Self Attention
             attn_outputs, self_attn_weights, yoco_key_values = self.attn(