🔍 Read the full analysis: What Makes Fine-Tuning Slower Than The FLOPs Suggest? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the results come from one production GPU pool and do not show whether improved predictions save time or money in a live scheduler.
Hugging Face reports that a CatBoost model predicted the runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation, compared with 12.41 hours for the best tested FLOP-based estimate. The result could help operators schedule scarce GPUs, but it comes from one corporate production pool and has not yet been tested in a live scheduling trial.
The runs accounted for about 7,382 GPU-hours on shared NVIDIA H200 machines. Job durations ranged from 1.6 to 172.4 hours and covered models up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and allocations of one, two or four GPUs. The pool contained between eight and 32 H200s across one to four nodes, with other training and inference workloads running at the same time.
The target was wall-clock time from GPU allocation to job completion; queue time was excluded. The predictor used 24 selected features, including submitted configuration and model metadata available before dispatch. Those inputs included evaluation cadence, checkpoint settings, per-GPU token volume and fine-tuning regime. A FLOP-based duration estimate was also included as one feature.
In stratified five-fold cross-validation, the CatBoost model scored an R² of 0.98 and a root mean squared error of 3.52 hours. Its median absolute error was 0.51 hours. The FLOP estimate performed best at an assumed 30% Model FLOPs Utilization, or MFU, with R² of 0.31 and a 12.41-hour mean absolute error. Its error rose to 13.01 hours at 45% MFU and 14.62 hours at 60%; the preview says testing MFU values from 25% to 65% did not resolve the gap.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
More Reliable GPU Job Scheduling
Runtime estimates affect when cluster operators dispatch work and how they assign shared accelerators. If a job finishes sooner than expected, GPUs may sit idle until another job is ready; if it runs long, it can delay other jobs and shift delivery plans. A predictor that better reflects the full job lifecycle could give operators a more useful basis for planning than a compute estimate alone.
The reported error difference is large within this dataset, but it is not a measured reduction in idle GPU time, cost or delays. The study evaluates prediction accuracy, not the operational effects of deploying the model. Its practical value will depend on whether similar accuracy holds across other clusters and whether schedulers can act on the predictions effectively.
Products Worth Considering

Amazon Kindle Scribe (16GB) with Premium Pen, Tungsten

Bose QuietComfort Ultra Wireless Noise Cancelling Earbuds, 2nd Gen, with 20W 2-Port Wall Charger

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery
As an affiliate, we earn on qualifying purchases.
Why FLOP Counts Miss Runtime
A common training-time estimate approximates dense Transformer compute as six times the parameter count and training-token count, then divides by GPU count, peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice; the preview describes typical values as ranging roughly from 30% to 60%.
That calculation estimates training compute, while Hugging Face’s target covers time after GPUs are allocated until the job finishes. Model setup and loading, tokenization, scheduled evaluation, checkpoint uploads and final storage can all add time. The preview says these tasks vary by configuration and can compete for network and storage resources on multi-tenant clusters. It also says researchers included evaluation tokens in the baseline comparison and tested a range of MFU assumptions.
“Everything is known before dispatch.”
— Hugging Face, in its study preview
How Far the Results Extend
The results come from 272 jobs in one corporate production environment. Cross-validation tests performance on held-out portions of that dataset; it does not establish accuracy on independently collected clusters, GPU types, software stacks or workload mixes. The preview also does not provide a live scheduler trial or report whether using the predictions reduced GPU idle periods, delays or costs.
The full paper has not yet been posted, and the preview gives no publication date. Further details on the dataset and evaluation design may appear there. It is also unclear how frequently the model would need retraining as workloads and shared-cluster conditions change, or whether performance would hold on runs collected in a later period.
Full Paper and External Testing
Hugging Face says a full version will be posted on arXiv, but has not specified when. That paper may give readers more information about feature selection, validation and the underlying production data.
To establish whether the approach generalizes, it would need testing on jobs from different time periods or environments, followed by a trial in an operational scheduler. Such results could show whether the low cross-validation error persists beyond the original GPU pool and whether more accurate estimates lead to fewer scheduling mismatches.
Key Questions
What did the Hugging Face model predict?
It predicted wall-clock runtime for supervised fine-tuning jobs, measured from GPU allocation until completion. Queue time was not included.
How did CatBoost compare with the FLOP estimate?
Across five-fold cross-validation on the reported dataset, CatBoost had a 1.71-hour mean absolute error. The best tested FLOP-based setting, at 30% MFU, had a 12.41-hour error.
Does this show that GPU costs or delays were reduced?
No. The preview reports prediction errors, not results from a live scheduling deployment or measured savings in GPU time, cost or delays.
Can the results be expected on other clusters?
That is not established. The evaluation used one production pool of shared NVIDIA H200 GPUs, so independent testing on other hardware and workloads is needed.
Primary source: Hugging Face · via ThorstenMeyerAI.com
