ViperEkura/DataPipeline
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6 Commits 1 Branch 0 Tags
a8f05cccd6528f13171cd91fccb79a6a76b93364
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
ViperEkura a8f05cccd6 修复输出文件的bug
2025-06-30 22:37:49 +08:00
.gitignore
修复输出文件的bug
2025-06-30 22:37:49 +08:00
chinese-c4.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
english-fineweb.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
to_ids.py
修复输出文件的bug
2025-06-30 22:37:49 +08:00
tokenizer.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
S
Description
No description provided
254 KiB
Languages
Python 100%