Nir Sonnenschein 407708cdb6 add support for tensor learning rate (vs scalar) (#7633) ...

This change is intended to help enable support for using a tensor
learning rate value vs a scalar ones.
We found this helpful in cases where the optimizer is torch.compiled (in
such cases changing the scalar LR value could cause recompilation
degrading the performance).
The implementation allows the model script to determine the type of LR
value used by setting the initial value.

Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>

2025-10-20 05:32:29 +00:00

..

activation_checkpointing

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

checkpoint_engine

undo the revert (#7536)

2025-09-02 14:24:48 -04:00

comm

Update GH org references (#6998)

2025-02-05 00:56:50 +00:00

compression

Use new dlpack api; Formatting fixes (#7101)

2025-03-03 17:50:24 +00:00

data_pipeline

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

domino

Cross layer overlapping for domino (#7178)

2025-03-28 17:59:36 +00:00

fp16

Enable non-ZeRO mode (#7515)

2025-08-27 14:07:29 -04:00

model_checkpointing

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

pipe

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

sequence_parallel

[TiledMLP] moe support (#7622)

2025-10-07 13:33:34 +00:00

superoffload

Take **kwargs in __init__ of DeepSpeedZeroOptimizer subclasses (#7634)

2025-10-19 22:07:48 -07:00

swap_tensor

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

tensor_parallel

async tp allreduce (#7115)

2025-03-28 22:48:17 +00:00

zenflow

Take **kwargs in __init__ of DeepSpeedZeroOptimizer subclasses (#7634)

2025-10-19 22:07:48 -07:00

zero

Take **kwargs in __init__ of DeepSpeedZeroOptimizer subclasses (#7634)

2025-10-19 22:07:48 -07:00

__init__.py

Improve universal checkpoint (#5289)

2024-03-28 08:12:57 +00:00

base_optimizer.py

Fix scaling and allgather with torch.autocast (#7534)

2025-09-03 01:22:19 +00:00

bf16_optimizer.py

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

compiler.py

Reduce performance impact of compiler.enable decorator (#7498)

2025-08-18 22:04:10 +00:00

config_utils.py

Fix assert when 'pp_int' object has no attribute 'custom_print_str' (#7507)

2025-08-25 10:57:08 -04:00

config.py

Enabling Muon Optimizer in DeepSpeed (#7509)

2025-08-26 18:34:35 -07:00

constants.py

Enable non-ZeRO mode (#7515)

2025-08-27 14:07:29 -04:00

dataloader.py

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

eigenvalue.py

Fix invalid f-strings (#7457)

2025-08-16 18:22:19 +00:00

engine.py

Add ZenFlow code for Stage 3 (#7516)

2025-10-13 12:19:18 -04:00

hybrid_engine.py

Improve overflow handling in ZeRO (#6976)

2025-06-09 17:30:51 +00:00

lr_schedules.py

add support for tensor learning rate (vs scalar) (#7633)

2025-10-20 05:32:29 +00:00

precision_config.py

Improve overflow handling in ZeRO (#6976)

2025-06-09 17:30:51 +00:00

progressive_layer_drop.py

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

quantize.py

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

sparse_tensor.py

fix issues raised by Coverity scans (#7431)

2025-08-02 12:16:10 -04:00

state_dict_factory.py

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00

torch_autocast.py

Relax restrictions of torch.autocast integration (#7543)

2025-09-03 12:15:10 -07:00

utils.py

Change current_device() to current_device_name() (#7600)

2025-09-28 10:19:49 -07:00

weight_quantizer.py

Update DeepSpeed copyright license to Apache 2.0 (#3111)

2023-03-30 17:14:38 -07:00