frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 21:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

only fail regressions>10% on pr_time benchmarks (#158577)

Browse Source 
Moving to a new framework, maintaitning the pr_time benchmark test right now is hard and often breaking.
1. only fail PRs >10% regressions.
2. post monitor with pr_time benchmarks dashboard (oncall), and update expected results (frequently or on big changes)
(supposed to already be doing https://www.internalfb.com/unidash/dashboard/pt2_diff_time_metrics)
3. setting up some one detections  detectors warnings that would be triggered at regressions and notify internally post land
https://www.internalfb.com/monitoring/detector/1140915271179237

Pull Request resolved: https://github.com/pytorch/pytorch/pull/158577
Approved by: https://github.com/xmfan, https://github.com/janeyx99
This commit is contained in:
Laith Sakka
2025-07-18 15:06:02 -07:00
committed by PyTorch MergeBot
parent ab557421a4
commit 64dabb2cf5
1 changed files with 23 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
benchmarks/dynamo/pr_time_benchmarks/expected_results.csv
View File

@ -1,89 +1,89 @@
add_loop_eager,compile_time_instruction_count,3017000000,0.015
add_loop_eager,compile_time_instruction_count,3070000000,0.10
add_loop_eager_dynamic,compile_time_instruction_count,4352000000,0.025
add_loop_eager_dynamic,compile_time_instruction_count,4432000000,0.10
add_loop_inductor,compile_time_instruction_count,29490000000,0.015
add_loop_inductor,compile_time_instruction_count,30280000000,0.10
add_loop_inductor_dynamic_gpu,compile_time_instruction_count,38760000000,0.025
add_loop_inductor_dynamic_gpu,compile_time_instruction_count,39910000000,0.10
add_loop_inductor_gpu,compile_time_instruction_count,26000000000,0.015
add_loop_inductor_gpu,compile_time_instruction_count,26800000000,0.10
basic_modules_ListOfLinears_eager,compile_time_instruction_count,947600000,0.015
basic_modules_ListOfLinears_eager,compile_time_instruction_count,969100000,0.10
basic_modules_ListOfLinears_inductor,compile_time_instruction_count,18490000000,0.015
basic_modules_ListOfLinears_inductor,compile_time_instruction_count,18030000000,0.10
basic_modules_ListOfLinears_inductor_gpu_force_shape_pad,compile_time_instruction_count,17020000000,0.015
basic_modules_ListOfLinears_inductor_gpu_force_shape_pad,compile_time_instruction_count,17020000000,0.10
basic_modules_ListOfLinears_inductor_gpu,compile_time_instruction_count,10297683772,0.2
basic_modules_ListOfLinears_inductor_gpu,compile_time_instruction_count,11090000000,0.2
update_hint_regression,compile_time_instruction_count,1673000000,0.02
update_hint_regression,compile_time_instruction_count,1719000000,0.10
sum_floordiv_regression,compile_time_instruction_count,986800000,0.015
sum_floordiv_regression,compile_time_instruction_count,966100000,0.10
symint_sum,compile_time_instruction_count,3166000000,0.015
symint_sum,compile_time_instruction_count,3237000000,0.10
symint_sum_loop,compile_time_instruction_count,4202000000,0.015
symint_sum_loop,compile_time_instruction_count,4299000000,0.10
aotdispatcher_inference_nosubclass_cpu,compile_time_instruction_count,2103000000,0.015
aotdispatcher_inference_nosubclass_cpu,compile_time_instruction_count,2151000000,0.10
aotdispatcher_inference_subclass_cpu,compile_time_instruction_count,6004000000,0.015
aotdispatcher_inference_subclass_cpu,compile_time_instruction_count,6124000000,0.10
aotdispatcher_partitioner_cpu,compile_time_instruction_count,8783000000,0.015
aotdispatcher_partitioner_cpu,compile_time_instruction_count,9005000000,0.10
aotdispatcher_partitioner_cpu2,compile_time_instruction_count,1940000000,0.015
aotdispatcher_partitioner_cpu2,compile_time_instruction_count,1989000000,0.10
aotdispatcher_training_nosubclass_cpu,compile_time_instruction_count,3885000000,0.015
aotdispatcher_training_nosubclass_cpu,compile_time_instruction_count,3959000000,0.10
aotdispatcher_training_subclass_cpu,compile_time_instruction_count,10470000000,0.015
aotdispatcher_training_subclass_cpu,compile_time_instruction_count,10650000000,0.10
mm_loop_inductor_gpu,compile_time_instruction_count,4324000000,0.015
mm_loop_inductor_gpu,compile_time_instruction_count,4461000000,0.10
mm_loop_inductor_dynamic_gpu,compile_time_instruction_count,8116000000,0.015
mm_loop_inductor_dynamic_gpu,compile_time_instruction_count,8417000000,0.10
basic_NestedModule_eager,compile_time_instruction_count,8152524390,0.015
basic_NestedModule_eager,compile_time_instruction_count,8348000000,0.10
basic_InlineMod_eager,compile_time_instruction_count,7255000000,0.015
basic_InlineMod_eager,compile_time_instruction_count,7464000000,0.10

1 add_loop_eager compile_time_instruction_count 3017000000 3070000000 0.015 0.10
2 add_loop_eager_dynamic compile_time_instruction_count 4352000000 4432000000 0.025 0.10
3 add_loop_inductor compile_time_instruction_count 29490000000 30280000000 0.015 0.10
4 add_loop_inductor_dynamic_gpu compile_time_instruction_count 38760000000 39910000000 0.025 0.10
5 add_loop_inductor_gpu compile_time_instruction_count 26000000000 26800000000 0.015 0.10
6 basic_modules_ListOfLinears_eager compile_time_instruction_count 947600000 969100000 0.015 0.10
7 basic_modules_ListOfLinears_inductor compile_time_instruction_count 18490000000 18030000000 0.015 0.10
8 basic_modules_ListOfLinears_inductor_gpu_force_shape_pad compile_time_instruction_count 17020000000 17020000000 0.015 0.10
9 basic_modules_ListOfLinears_inductor_gpu compile_time_instruction_count 10297683772 11090000000 0.2 0.2
10 update_hint_regression compile_time_instruction_count 1673000000 1719000000 0.02 0.10
11 sum_floordiv_regression compile_time_instruction_count 986800000 966100000 0.015 0.10
12 symint_sum compile_time_instruction_count 3166000000 3237000000 0.015 0.10
13 symint_sum_loop compile_time_instruction_count 4202000000 4299000000 0.015 0.10
14 aotdispatcher_inference_nosubclass_cpu compile_time_instruction_count 2103000000 2151000000 0.015 0.10
15 aotdispatcher_inference_subclass_cpu compile_time_instruction_count 6004000000 6124000000 0.015 0.10
16 aotdispatcher_partitioner_cpu compile_time_instruction_count 8783000000 9005000000 0.015 0.10
17 aotdispatcher_partitioner_cpu2 compile_time_instruction_count 1940000000 1989000000 0.015 0.10
18 aotdispatcher_training_nosubclass_cpu compile_time_instruction_count 3885000000 3959000000 0.015 0.10
19 aotdispatcher_training_subclass_cpu compile_time_instruction_count 10470000000 10650000000 0.015 0.10
20 mm_loop_inductor_gpu compile_time_instruction_count 4324000000 4461000000 0.015 0.10
21 mm_loop_inductor_dynamic_gpu compile_time_instruction_count 8116000000 8417000000 0.015 0.10
22 basic_NestedModule_eager compile_time_instruction_count 8152524390 8348000000 0.015 0.10
23 basic_InlineMod_eager compile_time_instruction_count 7255000000 7464000000 0.015 0.10
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89