frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 12:54:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated PyTorch OSS benchmark infra (markdown)

Huy Do
2025-06-13 12:49:22 -07:00
parent 9991f7c54a
commit 43fe0984b0
1 changed files with 44 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
PyTorch-OSS-benchmark-infra.md

@ -234,8 +234,50 @@ jobs:
## Query benchmark results
### Query API
TODO
### [Experimental] Query API
An experimental query API has been setup at `https://queries.clickhouse.cloud/run/84649f4e-52c4-4cf9-bd6e-0a105ea145c8` to query benchmark results from the database
```
import os
import json
import requests
username = os.environ.get("CLICKHOUSE_API_USERNAME")
password = os.environ.get("CLICKHOUSE_API_PASSWORD")
params = {
"format": "JSONEachRow",
"queryVariables": {
# REQUIRED: The repo name in org/repo format
"repo": "pytorch/pytorch",
# REQUIRED: The name of the benchmark
"benchmark": "TorchInductor",
# REQUIRED: YYYT-MM-DDThh:mm:ss
"startTime": "2025-06-06T00:00:00",
# REQUIRED: YYYT-MM-DDThh:mm:ss
"stopTime": "2025-06-13T00:00:00",
# OPTIONAL: Only query benchmark results for these models. Leaving this as an empty array [] will fetch all of them
"models": ["BERT_pytorch"],
# OPTIONAL: Only fetch these metrics. Leaving this as an empty array [] will fetch all of them
"metrics": ["speedup"]
# OPTIONAL: Filter the benchmark results by device, i.e. cuda, and arch, i.e. H100
"device": "",
"arch": "",
# OPTIONAL: Use this when you only care about the benchmark results from a specific branch and commit
"branch": "",
"commit": "",
}
}
api_url = "https://queries.clickhouse.cloud/run/84649f4e-52c4-4cf9-bd6e-0a105ea145c8"
r = requests.get(api_url, params=params, auth=(username, password))
with ("benchmark_results.txt", "w") as f:
print(r.text, file=f)
```
### Benchmark database
The benchmark database on ClickHouse Cloud is currently accessible to all Metamates. We also provide a [ClickHouse MCP server](https://github.com/izaitsevfb/clickhouse-mcp) that you can install to access the database via an AI agent like Claude Code.