ViperEkura/DataPipeline
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5 Commits 1 Branch 0 Tags
b508402724667ca89e52c256891886fceda84bb9
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
ViperEkura b508402724 优化处理形式
2025-06-30 22:00:15 +08:00
.gitignore
提交数据下载代码
2025-06-30 12:26:37 +08:00
chinese-c4.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
english-fineweb.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
to_ids.py
优化处理形式
2025-06-30 22:00:15 +08:00
tokenizer.py
提交数据下载代码
2025-06-30 12:26:37 +08:00
S
Description
No description provided
254 KiB
Languages
Python 100%