JoshWoo2003 7cb1b88ec4 Add ZenFlow code for Stage 3 (#7516) ...

This PR completes the ZenFlow integration for DeepSpeed ZeRO Stage 3. 

Highlights:

- ZenFlowSelectiveAdamW_stage3: Optimizer with importance-aware
selective parameter updates for ZeRO Stage 3.
- ZenFlowZeroOptimizer_Stage3: Full Stage 3 optimizer integration with
partitioned parameters and CPU offload.
- Configurable via ZenFlowConfig, fully integrated with
DeepSpeedZeroConfig for Stage 3.
- Unit tests for Stage 3 cases ensuring correctness and compatibility.

Note: Intergration with ZeRO Stage 1&2 was introduced in #7391

---------

Signed-off-by: Yusen Wu <xrn4ub@virginia.edu>
Co-authored-by: Ma, Guokai <guokai.ma@intel.com>
Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Co-authored-by: Tingfeng Lan <erc8gx@virginia.edu>

2025-10-13 12:19:18 -04:00

..

accelerator

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

benchmarks

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

hybrid_engine

DeepSpeed Chat (#3186)

2023-04-11 11:53:38 -07:00

lightning

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

model

Sanitize inputs to eval() (#6745)

2024-11-13 09:04:56 -08:00

onebit

Add Compressedbackend for Onebit optimizers (#5473)

2024-06-05 20:28:46 +00:00

perf

CPUAdam fp16 and bf16 support (#5409)

2024-05-20 12:50:20 +00:00

small_model_debugging

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

torch_compile

[compile] Show breakdown of graph break (#6601)

2024-10-14 17:31:34 +00:00

unit

Add ZenFlow code for Stage 3 (#7516)

2025-10-13 12:19:18 -04:00

.coveragerc

Reduce Unit Test Times (Part 3) (#3850)

2023-07-12 00:35:49 +00:00

conftest.py

tests/conftest.py: automatically add local deepspeed repo when running tests (#7317)

2025-05-28 23:32:49 +00:00

pytest.ini

Inference V2 Human Eval (#4804)

2024-02-22 22:55:40 +00:00