📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large-scale LLM from scratch with significant investment but achieved poor results on academic benchmarks. This challenges assumptions about native-language investment needs for effective country-specific models.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, a result that challenges assumptions about the importance of native-language training at current scales.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, using Italy’s national supercomputing resources. The project aimed to create a highly capable Italian language model through extensive pre-training on a large, native dataset, and published its weights and data openly. Despite outperforming comparable multilingual models on Italian benchmarks, Minerva-3B’s performance on the INVALSI exam was near chance, indicating a significant gap between technical capability and real-world language understanding.
Researchers noted that while dataset composition and size are important, the overall scale—parameters and data volume—may still be insufficient for complex language tasks. The empirical results suggest that the investment in native-language data and parameters, even at these large scales, may not be enough to produce models with deep country-specific knowledge. This finding complicates the narrative that larger, native-language-focused models automatically yield better performance.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
large language model training kit
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI model training dataset
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI research supercomputing hardware
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
open source NLP model weights
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Models
The results from Minerva demonstrate that simply scaling up native-language training data and parameters may not guarantee effective performance on complex, real-world tasks. This raises questions about the effectiveness of native-language models and the strategic investments needed by European countries in developing sovereign language models, suggesting that the current scale may be insufficient for achieving deep country-specific language understanding. The findings suggest a need for re-evaluating the scale and approach of national AI initiatives in Europe.
Background on European Sovereign-Language AI Projects
Italy’s Minerva project emerged as a major effort to build a European sovereign LLM from scratch, involving 128 GPUs on the Leonardo supercomputer and a large, openly available dataset. It was part of Italy’s broader national AI strategy, funded through the PNRR, and aimed to produce a model capable of outperforming multilingual models in Italian tasks. For more on European AI initiatives, see this overview of European sovereign-language projects. Prior efforts, such as Portugal’s AMÁLIA, took different approaches, focusing on continuation training of multilingual models with smaller native datasets. Minerva’s results highlight the challenges faced by these strategies, especially when scaling native-language datasets and parameters.
“While dataset composition is important, the overall size of the dataset and the number of parameters are more crucial for handling complex language tasks.”
— Research team
Unresolved Questions About Scale and Effectiveness
It is not yet clear what specific scale of data and parameters is necessary to produce models capable of handling complex language understanding in a native context. Researchers continue to explore whether larger models, different training methodologies, or additional native data will close the performance gap observed in Minerva.
Next Steps for European Native-Language Model Development
Researchers plan to continue iterating on Minerva’s architecture and training data, including ongoing experiments with increased scale and different training strategies. The focus will be on determining the threshold at which native-language models can reliably perform complex tasks, and how European projects can integrate these insights into future AI initiatives.
Key Questions
Why did Minerva-3B perform poorly on the Italian school exams?
The evaluation suggests that despite large-scale native-language training, the model lacks sufficient depth of country-specific knowledge, likely due to scale limitations at current parameter and data levels.
Does this mean native-language models are not effective?
Not necessarily. It indicates that current scale and investment levels may be insufficient for complex tasks, and further research is needed to identify the appropriate scale and methodology.
What does this mean for European AI strategies?
It suggests that European efforts may need to significantly increase scale or adopt new approaches to achieve models with deep country-specific language understanding.
Is Minerva an ongoing project?
Yes, the team continues to iterate and improve Minerva, with upcoming experiments aimed at addressing current limitations.
Source: ThorstenMeyerAI.com
