This website requires JavaScript.
Explore
Help
Register
Sign In
ViperEkura
/
DataPipeline
Watch
1
Star
0
Fork
0
Code
Issues
Pull Requests
Packages
Projects
Releases
Wiki
Activity
10
Commits
1
Branch
0
Tags
d8643d11ddd30ef5517fd3d2f153688851d44104
T
Code
Clone
HTTPS
Tea CLI
Open with VS Code
Open with VSCodium
Open with Intellij IDEA
Download ZIP
Download TAR.GZ
Download BUNDLE
ViperEkura
d8643d11dd
refactor(dataset processing): 重构数据集处理流程并添加新功能
2025-07-14 10:57:23 +08:00
.gitignore
feat: 添加中文维基数据集处理脚本
2025-07-11 10:31:10 +08:00
chinese-c4.py
refactor(dataset): 重构数据集处理逻辑
2025-07-14 08:45:01 +08:00
chinese-wiki.py
refactor(dataset): 重构数据集处理逻辑
2025-07-14 08:45:01 +08:00
english-fineweb.py
refactor(dataset processing): 重构数据集处理流程并添加新功能
2025-07-14 10:57:23 +08:00
english-wiki.py
refactor(dataset processing): 重构数据集处理流程并添加新功能
2025-07-14 10:57:23 +08:00
to_ids.py
修复输出文件的bug
2025-06-30 22:37:49 +08:00
tokenizer.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
utils.py
refactor(dataset processing): 重构数据集处理流程并添加新功能
2025-07-14 10:57:23 +08:00
S
Description
No description provided
254
KiB
Languages
Python
100%
