19 Commits

Author SHA1 Message Date
0db21a6b23 Remove most rockset references (#139922)
Remove most references to rockset:
* replace comments and docs with a generic "backend database"
* Delete `upload_to_rockset`, so we no longer need to install the package.
* Do not upload perf stats to rockset as well (we should be completely on DynamoDB now right @huydhn?)

According to VSCode, it went from 41 -> 7 instances of "rockset" in the repo
Pull Request resolved: https://github.com/pytorch/pytorch/pull/139922
Approved by: https://github.com/huydhn, https://github.com/ZainRizvi
2024-11-12 21:17:43 +00:00
235f7e06f4 [CI] upload_metrics function to upload to s3 instead of dynamo (#136799)
* Upload_metrics function to upload to ossci-raw-job-status bucket instead of dynamo
* Moves all added metrics to a field called "info" so ingesting into database table with a strict schema is easier
* Removes the dynamo_key field since it is no longer needed
* Removes the concept of reserved metrics, since they cannot be overwritten by user added metrics anymore
* Moves s3 resource initialization behind a function so import is faster
---
Tested by emitting a metric during run_test and seeing that documents got added to s3
Pull Request resolved: https://github.com/pytorch/pytorch/pull/136799
Approved by: https://github.com/ZainRizvi
2024-10-02 23:19:28 +00:00
1266be21f4 deprecated datetime.utcnow() fix and _RendezvousJoinOp module initiation bug fix (#136141)
Fix to #136140

Pull Request resolved: https://github.com/pytorch/pytorch/pull/136141
Approved by: https://github.com/kwen2501
2024-09-24 07:26:10 +00:00
8a67daf283 [BE][Easy] enable postponed annotations in tools (#129375)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/129375
Approved by: https://github.com/malfet
2024-06-29 09:23:35 +00:00
a32ce5ce34 Revert "[BE][Easy] enable postponed annotations in tools (#129375)"
This reverts commit 59eb2897f1745f513edb6c63065ffad481c4c8d0.

Reverted https://github.com/pytorch/pytorch/pull/129375 on behalf of https://github.com/huydhn due to Sorry for reverting your change but I need to revert to cleanly revert https://github.com/pytorch/pytorch/pull/129374, please do a rebase and reland this ([comment](https://github.com/pytorch/pytorch/pull/129375#issuecomment-2197800541))
2024-06-29 00:44:25 +00:00
59eb2897f1 [BE][Easy] enable postponed annotations in tools (#129375)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/129375
Approved by: https://github.com/malfet
2024-06-28 15:37:54 +00:00
fb6d052e9c Specify the exact table we upload metrics to (#124321)
Part of https://github.com/pytorch/ci-infra/issues/113

Since this table is only located in one AWS account, but the ARC account
also needs to access it, explicitly specify the account name for the
table
2024-04-23 10:55:52 -05:00
06b52dd103 TD outside of test job (#118250)
Give TD it's own job so that each shard can get the results from this one job artifact and they will always be in sync with each other/no longer need to worry about consistently issues

* Move test discovery to its own file that is not dependent on torch so it can be run without building torch
  * Cannot do cpp test discovery before building pytorch
* Move TD calculation to own file that will create a json file with the final results
* TD is now job/build env agnostic
* TD will rank all tests, including those that test jobs may not want to run (ex it will rank distributed tests along with default tests, even though these tests are never run on the same machine together)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/118250
Approved by: https://github.com/huydhn
2024-03-01 23:08:10 +00:00
9a9232956f Include job name in the emitted metrics (#113884)
What it says in the title
Pull Request resolved: https://github.com/pytorch/pytorch/pull/113884
Approved by: https://github.com/clee2000
2023-11-16 21:26:49 +00:00
6e73ae2022 [ci][ez] Add job_id to emit_metrics (#113099)
As in title.

Also print the job id in the step since I'm struggling to find it
Pull Request resolved: https://github.com/pytorch/pytorch/pull/113099
Approved by: https://github.com/seemethere
2023-11-08 10:32:41 +00:00
0c448526a4 [experiment][TD] Rating number system (#112676)
Emit excessive amount of heuristic info emitted, but that just means I can do more with it later?
Pull Request resolved: https://github.com/pytorch/pytorch/pull/112676
Approved by: https://github.com/ZainRizvi
2023-11-07 19:40:11 +00:00
1277d0e834 [BE] Add sharding data by default to metrics (#110035)
Extend metric library to allow setting global metrics on a process level which will always be emitted.

Current use case for them is to include shard information every time a metric is emitted by run_test.py

<!--
copilot:poem
-->
### <samp>🤖 Generated by Copilot at 0cae92c</samp>

> _`run_test` refactored_
> _Sharding metrics in Rockset_
> _Autumn of testing_
Pull Request resolved: https://github.com/pytorch/pytorch/pull/110035
Approved by: https://github.com/clee2000
2023-09-26 17:06:49 +00:00
d6cc3ac8b2 Add PR number to metrics when available (#109406)
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 780bfa6</samp>

Add a new metric for pull request number in `tools/stats/upload_metrics.py`. This allows tracking the CI performance of pull requests.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/109406
Approved by: https://github.com/kit1980, https://github.com/malfet, https://github.com/clee2000
2023-09-25 19:57:34 +00:00
b1f1b39feb Revert "Add PR number to metrics when available (#109406)"
This reverts commit 5e19216a6e0e6ee322b7416f9a793a51b1ff8c82.

Reverted https://github.com/pytorch/pytorch/pull/109406 on behalf of https://github.com/atalman due to breaks lint ([comment](https://github.com/pytorch/pytorch/pull/109406#issuecomment-1730049340))
2023-09-21 17:59:12 +00:00
5e19216a6e Add PR number to metrics when available (#109406)
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 780bfa6</samp>

Add a new metric for pull request number in `tools/stats/upload_metrics.py`. This allows tracking the CI performance of pull requests.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/109406
Approved by: https://github.com/kit1980, https://github.com/malfet, https://github.com/clee2000
2023-09-21 16:47:05 +00:00
a399f839ac Revert "Add PR number to metrics when available (#109406)"
This reverts commit f0fb4b3897e9cac3b99ee5b9b2ecab255e9e2da3.

Reverted https://github.com/pytorch/pytorch/pull/109406 on behalf of https://github.com/ZainRizvi due to breaks trunk ([comment](https://github.com/pytorch/pytorch/pull/109406#issuecomment-1724061024))
2023-09-18 17:35:37 +00:00
f0fb4b3897 Add PR number to metrics when available (#109406)
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 780bfa6</samp>

Add a new metric for pull request number in `tools/stats/upload_metrics.py`. This allows tracking the CI performance of pull requests.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/109406
Approved by: https://github.com/kit1980, https://github.com/malfet, https://github.com/clee2000
2023-09-18 03:17:54 +00:00
5727b07ac6 TD: logging bugfix (#108288)
Fix bug where logging metrics don't get emitted unless the 'keep-going' label is specified on the PR

Also adds some extra logging to make debugging easier
Pull Request resolved: https://github.com/pytorch/pytorch/pull/108288
Approved by: https://github.com/Skylion007
2023-08-31 16:51:49 +00:00
5ddb8ef827 Make emit_metrics importable without having boto3 installed (#107070)
Make it so that scripts can import and run the `emit_metrics` function even if they don't have boto3 installed, in which case it will still validate the inputs but skip the actual metric emission part.

It's purely a refactor without any real logic changes

Motivation: So that run_test.py and the target determination code can use this library easily without worrying about if it was imported or if it's dependencies are installed.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/107070
Approved by: https://github.com/huydhn
2023-08-21 21:13:01 +00:00