ViperEkura/DataPipeline
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2 Commits 1 Branch 0 Tags
34dc9632d84ce7b7a3c73881c79107a5429788d0
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
ViperEkura 34dc9632d8 增加文件序列化代码
2025-06-30 21:42:25 +08:00
.gitignore
提交数据下载代码
2025-06-30 12:26:37 +08:00
chinese-c4.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
english-fineweb.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
to_ids.py
增加文件序列化代码
2025-06-30 21:42:25 +08:00
tokenizer.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
S
Description
No description provided
254 KiB
Languages
Python 100%