This website requires JavaScript.
Explore
Help
Register
Sign In
ViperEkura
/
DataPipeline
Watch
1
Star
0
Fork
0
Code
Issues
Pull Requests
Packages
Projects
Releases
Wiki
Activity
24
Commits
1
Branch
0
Tags
f64a1a15ee12db1d327efb242f47faa6c7ec4f70
T
Code
Clone
HTTPS
Tea CLI
Open with VS Code
Open with VSCodium
Open with Intellij IDEA
Download ZIP
Download TAR.GZ
Download BUNDLE
ViperEkura
f64a1a15ee
feat(utils): 新增 sft_process 函数
2025-07-16 11:35:08 +08:00
.gitignore
feat: 添加中文维基数据集处理脚本
2025-07-11 10:31:10 +08:00
belle_sft.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
chinese-c4.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
chinese-cosmopedia.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
english-fineweb.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
english-wiki.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
pre_train_file.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
sft_file.py
refactor(dataset): 重构数据集处理流程并添加新功能
2025-07-16 09:56:31 +08:00
tokenizer.py
refactor(dataset): 重构数据处理脚本并支持自定义数据集
2025-07-15 12:18:58 +08:00
utils.py
feat(utils): 新增 sft_process 函数
2025-07-16 11:35:08 +08:00
S
Description
No description provided
254
KiB
Languages
Python
100%
