refactor : 移除 -> None 返回值标注,拆分 FSDP 参数,新增 mmap 数据集存储

- 删除所有 def 函数 -> None 返回值类型标注
- FSDPExecutor 参数从 **kwargs 拆为显式声明,None 值自动过滤
- 新增 MmapStorage (bin) 存储后端,基于 numpy.memmap 零拷贝加载
- 新增 save_bin/load_bin/json_to_bin 工具函数
- detect_format 支持 bin 格式自动检测
This commit is contained in:
2026-05-28 13:57:06 +08:00
parent 2d5dc93b3d
commit cb8dcb97ea
14 changed files with 142 additions and 48 deletions
+8
View File
@@ -8,11 +8,15 @@ from astrai.dataset.storage import (
BaseStorage,
H5Storage,
JSONStorage,
MmapStorage,
MultiSegmentFetcher,
StorageFactory,
detect_format,
json_to_bin,
load_bin,
load_h5,
load_json,
save_bin,
save_h5,
save_json,
)
@@ -25,11 +29,15 @@ __all__ = [
"BaseStorage",
"H5Storage",
"JSONStorage",
"MmapStorage",
"StorageFactory",
"detect_format",
"save_h5",
"load_h5",
"save_json",
"load_json",
"save_bin",
"load_bin",
"json_to_bin",
"ResumableDistributedSampler",
]