ViperEkura/AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
811 Commits 1 Branch 14 Tags
432dfec3c2e35e16ec2504fffd6e979fc25d946f
Commit Graph
15 Commits
Author SHA1 Message Date
ViperEkura 1cf7d6c76b perf: fill steady-state decode input ids via d2d copy 
- add InferenceWorkspace.fill_input_ids_from_device copying device tokens straight into the fixed-address input_ids buffer
- cache each decode step's sampled tokens on-device in DecodeSteadyState.last_tokens; when the task signature is unchanged the next step reuses them, replacing the tolist -> python list -> elementwise host fill -> pageable h2d round-trip
- _sample_logits returns (host payload, device tokens); prefill discards the device tensor
- signature change (task join/leave/first decode) still takes the host path; both dispatch paths covered by tests

Benchmark: NVIDIA L20, BF16, 1B model + 0.11B test model (4 layers, hidden 512), contiguous KV cache, CUDA Graph, greedy, prompt 512, generation 256, engine decode via scripts/tools/benchmark.py (alternating A/B, 2-4 paired runs)
- 0.11B batch 32: 21429 -> 24415 tok/s mean (1.14x, +13.9%), 4/4 paired runs faster
- 1B batch 32: 4242 -> 4388 tok/s (1.034x, +3.4%), 7.54 -> 7.29 ms/step
- batch 1: no measurable change (<0.5%)
 2026-08-31 14:24:51 +08:00
ViperEkura 47b3ed4e44 feat: propagate attention backend across scheduler threads 
- InferenceEngine/Scheduler accept an explicit backend
- capture request-level attn_backend context onto Task
- split prefill/decode batches by backend instance
- ASTR_BACKEND env overrides ContextVar as process-wide policy
- report resolved backend and CUDA-graph state in benchmark
 2026-08-09 13:32:40 +08:00
ViperEkura be90dfe2bd fix: isolate continuous batch decode state 
- Match steady-state metadata to the active task IDs
- Rebuild request mappings for cached prefix pages
- Add regressions for batch refill and prefix reuse
 2026-08-09 01:01:41 +08:00
ViperEkura 0c1b7664c1 refactor: split infer core into subpackages by concern 
- Eliminate core/ directory into cache/, runtime/, network/ subpackages plus flat modules
- Split cache.py (647 lines) into cache/{buffer,strategy,pool}.py by layer
- Add explicit ContiguousStrategy, make AllocationStrategy a real ABC
- Move TaskCacheState to cache/strategy.py, drop string forward references
- Rename api/ to network/, server.py to app.py
- Move sample.py into runtime/ alongside executor and graph
- Simplify TaskCacheManager.__init__ to single pool param
- Expose pool.strategy and pool.req_pool as public properties
- Fix KVCache import in attention_backend.py (TYPE_CHECKING guard)
- Fix steady-state decode reading uninitialized position_ids on first step
 2026-08-08 23:43:05 +08:00
ViperEkura 1b1f1a0707 fix: add dtype guard to FlashAttnBackend capability check 
- _backend_supports now rejects fp32 for FlashAttnBackend (flash-attn only supports fp16/bf16), preventing runtime crash on fallback chain
- rename test_default_backend_is_torch_native to reflect multi-backend reality
- scheduler test fixture uses bf16 model (matches production, avoids unnecessary 3-step fallback chain)
 2026-08-07 23:08:22 +08:00
ViperEkura 5c180cfa90 fix : handle zero-token batch generation 
- return empty results without running inference for non-positive limits
- keep scheduler batch outputs aligned with requested max_tokens
- add engine and scheduler regression coverage
 2026-08-06 12:31:09 +08:00
ViperEkura 654e6eb0d1 fix : correct prefill sampling and record alignment 
- sample the first token from prefill logits without duplicating the prompt tail
- reject incomplete multi-output records before preprocessing alignment
- cover cached generation and partial DPO records with regression tests
 2026-08-05 22:20:29 +08:00
ViperEkura a27c8a819d test: prune low-value and duplicate tests 
- Remove tautological test_trainer assertions that never trained
- Drop grpo isfinite-only smokes and merge frozen-model checks via parametrize
- Merge duplicate tool_parser cases (find/streaming/factory) with parametrize
- Collapse duplicate dataset store/detect_format tests
- Remove misleading scheduler/task tests that asserted the opposite of their names
- Merge signal-handler SIGTERM/SIGINT into one parametrized case
- Drop cross-file grpo strategy duplication kept in online_strategy
 2026-08-01 16:01:20 +08:00
ViperEkura 5ba21f4eb3 refactor: eliminate test duplication via shared helpers 
- Add tests/helpers.py with shared config, dataset, tokenizer, executor, and assertion helpers
- Replace 15 copies of device one-liner with session-scoped fixture
- Collapse 5 near-identical Dataset subclasses into RandomTokenDataset
- Remove duplicate _make_config/_make_model/_make_frozen and FakeTokenizer/FakeExecutor definitions
- Make test_callbacks and test_early_stopping use existing train_config_factory
- Replace 6 duplicate meta.json read blocks with load_shard_meta
- Fix mkdtemp leaks in test_lora.py with TemporaryDirectory
 2026-07-27 22:34:53 +08:00
ViperEkura 53c804e233 refactor : merge max_prompt_len into max_seq_len, replace assert with raise 
- Engine/Scheduler/TaskManager: merge max_prompt_len into max_seq_len
- train.py: replace bare assert with ValueError/FileNotFoundError
- server.py: add --max_seq_len CLI option
- engine.py: remove dead page_size param
 2026-07-27 08:05:11 +08:00
ViperEkura 0c86c89af4 refactor : align config field names with Hugging Face 
- dim -> hidden_size, n_layers -> num_hidden_layers
- dim_ffn -> intermediate_size, n_heads -> num_attention_heads
- n_kv_heads -> num_key_value_heads, max_len -> max_position_embeddings
- norm_eps -> rms_norm_eps, tie_weight -> tie_word_embeddings
- update model, inference, training, scripts, tests, docs
 2026-07-20 22:05:31 +08:00
ViperEkura 95c43368ae refactor: unify rollout onto inference engine KV-cache path 
- RolloutGenerator now delegates prefill/decode to InferenceScheduler.run_batch (sync API, no background thread), sharing one KV-cache code path with the inference server and eliminating O(n^2) recompute in rollout
- Add sample(return_logprobs=) and Executor.execute_decode(return_logprobs=) to expose behaviour-policy log-probs through the engine; Task gains output_logprobs
- RolloutResult now subclasses RawRollout (adds rewards only), removing duplicated fields
- RolloutRunner.__call__ returns (result, is_fresh) instead of relying on object identity, removing the fragile refresh-detection contract
- Remove O(n^2) generate_responses helper and dead code (_tokenize_prompts, unused old_model arg)
- train_context.py wires InferenceScheduler directly instead of hand-rolling SamplingPipeline
- Tests: +11 covering return_logprobs, run_batch, and KV-cache-backed rollout semantics; 404 pass
 2026-07-20 12:52:20 +08:00
ViperEkura 94d6e713e9 test: 补充推理协议层单测覆盖 
- StopChecker、GenContext、StopInfo 单测
- OpenAIResponseBuilder / AnthropicResponseBuilder 全部方法
- Anthropic 停止序列裁剪逻辑（含 unyielded 边界）
- GenerationRequest 参数校验含负值边界
- Scheduler prefill 短路验证
 2026-05-26 00:21:52 +08:00
ViperEkura 2196c34c52 refactor: 重构 inference 模块架构，引入设计模式并分组文件 
- 新增 protocol.py 协议层，Template Method 模式消除流/非流分支 45% 重复
- SSEBuilder 统一 SSE 构造，StopChecker 独立 stop_sequence 检测
- AnthropicHandler 追踪已产出文本，修复 stop 时重复 delta
- server.py 路由从约 100 行缩减至 3 行
- 拆分为 core/（cache/executor/scheduler/task）和 api/（protocol/server）
- 外部保持二级导入路径（from astrai.inference import Name）
- 删除所有分隔线注释，代码按语义自然分组
 2026-05-14 17:42:37 +08:00
ViperEkura 7440e9c809 style: 重命名 test_scheduler_concurrency 为 test_scheduler 2026-05-12 12:24:36 +08:00