{"id":16538,"date":"2025-06-30T06:00:02","date_gmt":"2025-06-30T06:00:02","guid":{"rendered":"http:\/\/TheNextWeb=1414045"},"modified":"2025-06-30T06:00:02","modified_gmt":"2025-06-30T06:00:02","slug":"the-race-to-make-ai-as-multilingual-as-europe","status":"publish","type":"post","link":"https:\/\/www.londonchiropracter.com\/?p=16538","title":{"rendered":"The race to make AI as multilingual as Europe"},"content":{"rendered":"\n

The European Union has <\/span>24 official languages<\/span><\/a> and dozens more unofficial ones spoken across the continent. If you add in the European countries outside the union, then that brings at least a dozen more into the mix. Add dialects, <\/span>endangered languages<\/span><\/a>, and languages brought by migrants to Europe, and you end up with hundreds of languages.<\/span><\/p>\n

One thing many of us in technology could agree on is that the US dominates \u2014 and that extends to online languages. There are many reasons for this, mostly due to American institutions, standards bodies, and companies defining how computers, their operating systems, and the software they run work in their nascent days. This is changing, but for the short term at least, it remains the norm. This has also led to the majority of the web being in English. An astounding <\/span>50% of websites are in English<\/span><\/a>, despite it being the native tongue of only about 6% of the world\u2019s population, with Spanish, German, and Japanese next, but a long way behind, each only between 5-6% of the web.<\/span><\/p>\n

As we delve deeper into the new wave of AI-powered applications and services, many are driven by data in large language models (LLMs). As much of the data in these LLMs is scraped (controversially in many cases) from the web, <\/span>LLMs predominantly understand and respond in English.<\/span><\/a> As we find ourselves at the start of or in the midst of a shift in technological paradigm caused by the rapid growth of AI tools, this is a problem, and we\u2019re bringing that problem into a new age.<\/span><\/p>\n

Europe already boasts several high-profile AI companies and projects, such as <\/span>Mistral<\/span> and <\/span>Hugging Face<\/span>. <\/span>Google DeepMind<\/span> also originated as a European company. The continent has research projects that develop language models to enhance how AI tools comprehend less commonly spoken languages.<\/span><\/p>\n

This article explores some of these initiatives, questions their effectiveness, and asks whether their efforts are worthwhile or if many users default to using English versions of tools. As Europe seeks to build its independence in AI and ML, does the continent have the companies and skills necessary to achieve its goals?<\/span><\/p>\n

\n
<\/p>\n

TNW Conference 2025 – That’s a wrap!<\/p>\n

Check out the highlights!<\/p>\n

<\/a><\/div>\n<\/div>\n

Terminology and technology primer<\/h2>\n

To make sense of what follows, you don\u2019t need to understand how models are created, trained, or function. But it\u2019s helpful to understand a couple of basics about models and their human language support.<\/span><\/p>\n

Unless model documentation explicitly mentions it is <\/span>multilingual or cross-lingual, prompting it or requesting a response in an unsupported language may cause it to translate back and forth or respond in a language it <\/span>does<\/span><\/i> understand. Both strategies can produce unreliable and inconsistent results \u2014 especially in low-resource languages.<\/span><\/p>\n

While <\/span>high-resource languages, such as English, benefit from abundant training data<\/a>. <\/span>Low-resource languages, such as Gaelic or Galician, have far less, which often leads to inferior performance<\/span><\/p>\n

The harder concept to explain regarding models is \u201copen,\u201d which is unusual, as software in general has had a fairly clear definition of \u201copen source\u201d for a while. I don\u2019t want to delve too deeply into this topic as the exact definition is still in flux and controversial. The summary is that even when a model might call itself \u201copen\u201d and is referenced as \u201copen,\u201d the meaning of \u201copen\u201d isn\u2019t always the same.<\/span><\/p>\n

Here are two other useful terms to know:<\/span><\/p>\n

Training<\/b> teaches a model to make predictions or decisions based on input data.<\/span><\/p>\n

Parameters<\/b> are variables learned during model training that define how the model maps inputs to outputs. In other words, how it understands and responds to your questions. The larger the number of parameters, the more complex the model is.<\/span><\/p>\n

With that brief explanation done, how are European AI companies and projects working to enhance these processes to improve European language support?<\/span><\/p>\n

Hugging Face<\/h2>\n

When someone wants to share code, they typically provide a link to their GitHub repository. When someone wants to share a model, they typically provide a Hugging Face link. Founded in 2016 by French entrepreneurs in New York City, the company is an active participant in creating communities and a strong proponent of open models. In 2024, it started an AI accelerator for European startups and partnered with Meta to develop translation tools based on <\/span>Meta\u2019s <\/span>\u201cNo Language Left Behind\u201d model<\/span><\/a>. They are also one of the driving forces behind the <\/span>BLOOM model<\/span><\/a>, a groundbreaking multilingual model that set new standards for international collaboration, openness, and training methodologies.<\/span><\/p>\n

Hugging Face is a useful tool for getting a rough idea of the language support in models. At the time of writing, <\/span>Hugging Face lists <\/span>1,743,136 models<\/span><\/a> and <\/span>298,927 datasets<\/span><\/a>. Look at its <\/span>leaderboard<\/span><\/a> for monolingual models and datasets<\/span>, and you see the following ranking for models and datasets that developers tag (add metadata) as supporting European languages at the time of writing:<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n
Language<\/strong><\/th>\nLanguage code<\/strong><\/th>\nDatasets<\/strong><\/th>\nModels<\/strong><\/th>\n<\/tr>\n
English English<\/span><\/th>\nen<\/span><\/a><\/th>\n27,702<\/span><\/a><\/th>\n205,459<\/span><\/a><\/th>\n<\/tr>\n<\/thead>\n
English<\/span><\/td>\neng<\/span><\/a><\/td>\n1,370<\/span><\/a><\/td>\n1,070<\/span><\/a><\/td>\n<\/tr>\n
French<\/span><\/td>\nfra<\/span><\/a><\/td>\n1,933<\/span><\/a><\/td>\n850<\/span><\/a><\/td>\n<\/tr>\n
Spanish Espa\u00f1ol<\/span><\/td>\nes<\/span><\/a><\/td>\n1,745<\/span><\/a><\/td>\n10,028<\/span><\/a><\/td>\n<\/tr>\n
German Deutsch<\/span><\/td>\nde<\/span><\/a><\/td>\n1,442<\/span><\/a><\/td>\n9,714<\/span><\/a><\/td>\n<\/tr>\n
English<\/span><\/td>\neng<\/span><\/a><\/td>\n1,370<\/span><\/a><\/td>\n1,070<\/span><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

You can already see some issues here. These aren\u2019t tags set in stone. The community can add values freely. While you can see that they follow them for the most part, there is some duplication.<\/span><\/p>\n

As you can see, the models are dominated by English. A similar issue applies to the datasets on Hugging Face, which lack non-English data.<\/span><\/p>\n

What does this mean?<\/span><\/p>\n

Lucie-Aim\u00e9e Kaffee, EU Policy Lead at Hugging Face, said that the tags indicate that a model has been trained to understand and process this language or that the dataset contains materials in that language. She added that the confusion between language support often comes during training.\u201cWhen training a large model, it\u2019s common for other languages to accidentally get caught in training because there were some artefacts of it in that dataset,\u201d she said. \u201cThe language a model is tagged with is usually what the developers intended the model to understand.\u201d<\/span><\/p>\n

As one of the main and busiest destinations for model developers and researchers, Hugging Face not only hosts much of their work, but also lets them create outward-facing communities to tell people how to use them.<\/span><\/p>\n

\"Thomas
Thomas Wolf, co-founder of Hugging Face, described Bloom as \u201cthe world\u2019s largest open multilingual language model.\u201d Credit: Shauna Clinton\/Web Summit via Sportsfile<\/a><\/figcaption>