docs: 更新说明内容
This commit is contained in:
@@ -1,7 +1,6 @@
|
||||
<div align="center">
|
||||
<!-- <img src="../images/project_logo.png" width="auto" alt="Logo"> -->
|
||||
|
||||
<h1>AstrAI</h1>
|
||||
<img src="../images/logo.png" width="auto" alt="Logo">
|
||||
|
||||
<div>
|
||||
<a href="../../README.md">English</a> •
|
||||
@@ -28,9 +27,8 @@
|
||||
<a href="#chinese">中文</a> •
|
||||
<a href="https://github.com/ViperEkura/AstrAI/issues">问题追踪</a> •
|
||||
<a href="https://github.com/ViperEkura/AstrAI/discussions">讨论区</a> •
|
||||
<a href="https://huggingface.co/ViperEk/AstrAI">HuggingFace</a>
|
||||
<a href="https://huggingface.co/ViperEk">HuggingFace</a>
|
||||
</div>
|
||||
|
||||
<br>
|
||||
|
||||
## 📖 目录
|
||||
@@ -131,7 +129,7 @@ python scripts/demo/generate_ar.py
|
||||
|
||||
- **GitHub Issues**: [问题追踪](https://github.com/ViperEkura/AstrAI/issues)
|
||||
- **Discussions**: [GitHub 讨论区](https://github.com/ViperEkura/AstrAI/discussions)
|
||||
- **HuggingFace**: [模型中心](https://huggingface.co/ViperEk/AstrAI)
|
||||
- **HuggingFace**: [模型中心](https://huggingface.co/ViperEk)
|
||||
|
||||
### 许可证
|
||||
|
||||
|
||||
@@ -1,12 +1,54 @@
|
||||
## Model Introduction
|
||||
|
||||
|
||||
|
||||
### 1. Model Architecture
|
||||
|
||||
This model uses the Transformer architecture with GQA mechanism (q_head=24, kv_head=4), which saves KV cache memory compared to traditional MHA (although KV cache is not currently implemented). The model is built by stacking 24 layers of Transformer blocks, with 1.0 billion parameters. Transformer is an autoregressive model that calculates the relationship between all previous tokens to obtain the probability distribution of the next token.
|
||||
|
||||

|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Layers["Transformer Layers"]
|
||||
direction TB
|
||||
A[Input Embedding] --> B[Transformer Block\nLayer 1]
|
||||
B --> C[Transformer Block\nLayer ...]
|
||||
C --> D[Transformer Block\nLayer 32]
|
||||
D --> E[RMSNorm]
|
||||
E --> F[Linear]
|
||||
F --> G[SoftMax]
|
||||
end
|
||||
|
||||
subgraph TransformerBlock["Transformer Block"]
|
||||
direction TB
|
||||
H[x] --> I[RMSNorm]
|
||||
I --> J[Linear → Q/K/V]
|
||||
J --> K[Q]
|
||||
J --> L[K]
|
||||
J --> M[V]
|
||||
K --> N[RoPE]
|
||||
L --> O[RoPE]
|
||||
N --> P["Q @ K^T / sqrt(d)"]
|
||||
O --> P
|
||||
P --> Q[Masked SoftMax]
|
||||
Q --> R[S @ V]
|
||||
M --> R
|
||||
R --> S[Linear]
|
||||
S --> T[+]
|
||||
H --> T
|
||||
T --> U[RMSNorm]
|
||||
U --> V[Linear]
|
||||
V --> W[SiLU]
|
||||
V --> X[×]
|
||||
W --> X
|
||||
X --> Y[Linear]
|
||||
Y --> Z[+]
|
||||
T --> Z
|
||||
Z --> AA[x']
|
||||
end
|
||||
|
||||
classDef main fill:#e6f3ff,stroke:#0066cc;
|
||||
classDef block fill:#fff2e6,stroke:#cc6600;
|
||||
class Layers main;
|
||||
class TransformerBlock block;
|
||||
```
|
||||
|
||||
What is an autoregressive model? After splitting a sentence into tokens, the model predicts the probability distribution of the next token. This means the model calculates the probability of the next possible token and its corresponding probability based on the given context (the sequence of tokens that have already appeared).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user