refactor : 移除 -> None 返回值标注,拆分 FSDP 参数,新增 mmap 数据集存储
- 删除所有 def 函数 -> None 返回值类型标注 - FSDPExecutor 参数从 **kwargs 拆为显式声明,None 值自动过滤 - 新增 MmapStorage (bin) 存储后端,基于 numpy.memmap 零拷贝加载 - 新增 save_bin/load_bin/json_to_bin 工具函数 - detect_format 支持 bin 格式自动检测
This commit is contained in:
@@ -8,11 +8,15 @@ from astrai.dataset.storage import (
|
||||
BaseStorage,
|
||||
H5Storage,
|
||||
JSONStorage,
|
||||
MmapStorage,
|
||||
MultiSegmentFetcher,
|
||||
StorageFactory,
|
||||
detect_format,
|
||||
json_to_bin,
|
||||
load_bin,
|
||||
load_h5,
|
||||
load_json,
|
||||
save_bin,
|
||||
save_h5,
|
||||
save_json,
|
||||
)
|
||||
@@ -25,11 +29,15 @@ __all__ = [
|
||||
"BaseStorage",
|
||||
"H5Storage",
|
||||
"JSONStorage",
|
||||
"MmapStorage",
|
||||
"StorageFactory",
|
||||
"detect_format",
|
||||
"save_h5",
|
||||
"load_h5",
|
||||
"save_json",
|
||||
"load_json",
|
||||
"save_bin",
|
||||
"load_bin",
|
||||
"json_to_bin",
|
||||
"ResumableDistributedSampler",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user