16 Commits

Author SHA1 Message Date
177c25c9d7 Add venv to .gitignore (#7605)
Since `make format` will generate `venv` directory, we should add it to
`.gitignore`.

Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
2025-09-30 20:44:07 +00:00
25a02047ae Add HIP conversion file outputs to .gitignore (#5111)
This PR adds the following HIP output files to `.gitignore`:
``` 
*_hip.cpp
*_hip.h
*.hip
*.cuh
*hip_layers.h
```

---------

Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
2024-02-09 14:43:54 -08:00
3418b869f0 Re-organize .gitignore file to be parsed properly (#4740)
The file was re-organized in da652d0[^1] with better documentation, but
that unfortunately made the syntax invalid. According to the gitignore
documentation:[^2]

> A line starting with # serves as a comment. Put a backslash ("`\`") in
> front of the first hash for patterns that begin with a hash.

I needed to move the comments for Git to parse the file properly. To
maintain readability, I added an extra `#` to the comments at the top of
each section.

[^1]:
da652d0e0b
[^2]: https://git-scm.com/docs/gitignore/2.43.0#_pattern_format

Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
2023-11-28 21:05:30 +00:00
da652d0e0b Update .gitignore [Adding comments , Improved documentation] (#4631)
Refine and clarify the .gitignore file by adding more descriptive
comments to explain the purpose of each rule. This improves the
readability and maintainability of the file.

- Added comments to explain each ignore rule
- Improved documentation for better understanding

---------

Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
2023-11-09 20:25:56 +00:00
56c5223868 bf16+pipeline parallelism (#1801)
* bf16 updates

* Got bf16 working

* fp32 reduction; flattened tensors

* bf16+zero_stage_1 first cut

* finish zero_stage 1 sharding

* Matching fp16 with debugging codes

* Matching loss with fp16

* Fix gradient clipping

* bf16 gradient clipping fix
bf16 checkpoint save/load

* Unscale grad norm

* Fix grad norm scaling

* Enable loading fp16_zero_1 into bf16_zero_1 engine and vice versa

* Fix clip_grad key error

* Reduce tied weight gradients

* Fix grad norm for moe

* Reduce specified gradients

* Use O(n) instead of O(n^2)

* Remove optimizer restriction for bf16

* Link bf16 & fp32 params

* Clip gradients of last stage tied weights

* Simplify tied weights reduction logic

* Also clip all tp rank parameters

* lp to hp mapping

* Link lp/hp/optim state; Refresh links after checkpoint load

* Remove debug print

* Remove debug print

* Simplify zero_grad logic

* fp32 accessors

* Fix update bug

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
2022-04-19 10:15:59 -07:00
31f46feee2 DeepSpeed JIT op + PyPI support (#496)
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
2020-11-12 11:51:38 -08:00
5812e84544 readthedocs yaml configuration (#410)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
2020-09-16 18:57:43 -07:00
c82756cd15 readthedocs upgrade (#402) 2020-09-10 15:44:47 -07:00
e5bbc2e559 Sparse attn + ops/runtime refactor + v0.3.0 (#343)
* Sparse attn + ops/runtime refactor + v0.3.0

Co-authored-by: Arash Ashari <arashari@microsoft.com>

Co-authored-by: Arash Ashari <arashari@microsoft.com>
2020-09-01 18:06:15 -07:00
e8dd47df26 Update .gitignore 2020-08-31 21:15:50 -07:00
f2ac7eafd5 ZeRO-2 (#217)
Updates for ZeRO stage 2 + ZeRO stage 1 w. RS

Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
2020-05-19 01:00:53 -07:00
dd166ee6b6 README and RTD improvements. (#198) 2020-04-21 22:18:47 -07:00
a76572dc7c Adding static loss scaling for ZeRO. (#166) 2020-03-25 09:34:27 -07:00
5042dc0085 drafting Jekyll webpage (#143) 2020-03-17 13:49:48 -07:00
010f6dc0cf Updating .gitignore (#55)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
2020-02-10 07:07:37 -08:00
e63b6b0113 add gitignore 2020-01-31 16:06:44 -08:00