9 Commits

Author	SHA1	Message	Date
Rémi Ouazan	f7c33abab3	Small changes to benchmarking script (#41662)	2025-10-16 17:25:49 +02:00
Yuanyuan Chen	9e99198e5e	Use | for Optional and Union typing (#41646) ... Signed-off-by: Yuanyuan Chen <cyyever@outlook.com>	2025-10-16 14:29:54 +00:00
Rémi Ouazan	94df0e6560	Benchmark overhaul (#41408) ... * Big refactor, still classes to move around and script to re-complexify * Move to streamer, isolate benches, propagate num tokens * Some refacto * Added compile mode to name * Re-order * Move to dt_tokens * Better format * Fix and disable use_cache by default * Fixed compile and SDPA backend default * Refactor results format * Added default compile mode * Always use cache * Fixed cache and added flex * Plan for missing modules * Experiments: no cg and shuffle * Disable compile for FA * Remove wall time, add sweep mode, get git commit * Review compliance, start * Apply suggestions from code review Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Update benchmark_v2/framework/benchmark_runner.py Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Disable workflow * Pretty print * Added some pretty names to have pretty logs * Review n2 compliance (end?) * Style and end of PR --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>	2025-10-14 21:41:43 +02:00
Ryan Mullins	71717ce91c	docs: Fix Tool Use links and remove dead RAG links (#41104) ... docs: Fix tool use links. Remove dead RAG links. Fix style	2025-09-23 09:18:49 -07:00
Ákos Hadnagy	62ce6fcb60	Fix argument name in benchmarking script (#41086) ... * Fix argument name in benchmarking script * Adjust vars	2025-09-23 13:05:27 +02:00
Ákos Hadnagy	b9d337b6f3	Add write token for uploading benchmark results to the Hub (#41047) ... * Separate write token for Hub upload * Address review comments * Address review comments	2025-09-22 14:13:46 +00:00
Ákos Hadnagy	61eff450d3	Benchmarking v2 GH workflows (#40716) ... * WIP benchmark v2 workflow * Container was missing * Change to sandbox branch name * Wrong place for image name * Variable declarations * Remove references to file logging * Remove unnecessary step * Fix deps install * Syntax * Add workdir * Add upload feature * typo * No need for hf_transfer * Pass in runner * Runner config * Runner config * Runner config * Runner config * Runner config * mi325 caller * Name workflow runs properly * Copy-paste error * Add final repo IDs and schedule * Review comments * Remove wf params * Remove parametrization from worfkflow files * Fix callers * Change push trigger to pull_request + label * Add back schedule event * Push to the same dataset * Simplify parameter description	2025-09-19 08:54:49 +00:00
Yuanyuan Chen	a5ecd94a3f	Enable ruff on benchmark and scripts (#40634) ... * Enable ruff on benchmark and scripts Signed-off-by: cyy <cyyever@outlook.com> * Cover benchmark_v2 Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> * correct * style * style --------- Signed-off-by: cyy <cyyever@outlook.com> Signed-off-by: Yuanyuan Chen <cyyever@outlook.com> Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>	2025-09-10 11:38:06 +02:00
Ákos Hadnagy	f22ec7f174	Benchmarking V2: framework impl (#40486) ... * Start revamping benchmarking * Start refactoring benchmarking * Use Pandas for CSV * import fix * Remove benchmark files * Remove sample data * Address review comments * Benchmarking v2 * Fix llama bench parameters * Working checkpoint * Readme touchups * Remove unnecessary test * Massage the framework a bit * Small cleanup * Remove unnecessary flushes * Remove references to mock benchmark * Take commit ID from CLI * Address review comments * Use Events for thread comms * Tiny renaming	2025-09-03 22:26:32 +02:00