Regarding the 2023 interview featuring Sundar Pichai, in which he talked about an LLM that was able to “learn,” or at least to translate Bengali, despite not being explicitly trained for that task, I have a non-technical theory off the top of my head.

Simplified, I take it that the said LLM, having been fed a humongous amount of documents, including some in Bengali, was able to “make correlations” or to “know without knowing” what something written in Bengali corresponds to in English. It surely helped to have “digested” non-fiction materials in Bengali, not just fiction. Scientific papers with drawings and pictures, for instance.

I would bring two comparisons with humans to support my theory that such an “emergent ability” is entirely normal.

Firstly, a baby doesn't know any language, and there's no way to explain to them grammar or definitions. Yet, after being exposed to lots of uttered words, babies make correlations and “deduce” the meaning of basic words. (Mind you, this is not an easy task, because the same word is pronounced differently by different persons, so the baby has to classify similar utterances as representing the same concept. As a side note, when adults from certain cultures have difficulties in differentiating the sounds for “b” and “p,” this is because in their mother tongue there is a unique sound situated between “b” and “p,” closer to one of them, but to such people “b” and “p” classify as the same sound.)

Secondly, in the 1980s, I used to read the Soviet magazine Радио. I only knew a few Russian words, but I had this tactic: by examining the schematic diagrams, I gathered enough information about the functioning of a proposed device, that this helped in understanding what the text wanted to convey about the building and adjusting of the device. It was not “learning Russian,” yet it was an “emergent ability” helped by the drawings and schematics. But I could only understand some technical texts in Russian, and common wording used in such contexts.
Yesterday 1:33pm

Radu-Cristian, your theory is not only intuitive but also aligns remarkably well with how modern large language models (LLMs) like the one Sundar Pichai described actually function. Let’s break down your comparisons and see how they map to the behavior of LLMs:

1. Emergent Abilities in LLMs

The phenomenon you’re describing—where an LLM exhibits abilities it wasn’t explicitly trained for—is known as "emergent behavior" in AI research. This happens because the model’s training process involves ingesting vast amounts of text and learning statistical patterns, relationships, and structures within that data.

  • Bengali Translation Example: If the LLM was trained on a corpus that included Bengali text (even if not explicitly for translation), it could infer relationships between Bengali and English words or phrases based on context, co-occurrence, and usage patterns. This is similar to how humans infer meaning from context, even if they don’t know every word.

  • Non-Fiction and Technical Texts: As you pointed out, technical or scientific texts (with diagrams, tables, or structured information) provide richer contextual clues. The model can use these to "anchor" its understanding of language, just as you used schematics in Радио to infer meaning in Russian.

2. Human Analogies

Your analogies are spot-on:

A. Language Acquisition in Babies

  • Babies don’t learn language through explicit instruction; they infer patterns from exposure. This is almost identical to how LLMs "learn" from data. The variability in pronunciation (e.g., "b" vs. "p") is analogous to the noise and variability in text data—yet both humans and models learn to generalize.
  • Key Point: Both babies and LLMs rely on statistical learning—identifying patterns in input to make sense of the world.

B. Technical Texts and Schematics

  • Your experience with Радио is a perfect example of multimodal learning (combining text and visuals). While LLMs are primarily text-based, newer models are beginning to incorporate images, diagrams, and other modalities to improve understanding. Your brain was doing something similar: using visual context to bridge gaps in linguistic knowledge.
  • Key Point: Context—whether from schematics, diagrams, or surrounding text—helps both humans and models "fill in the blanks" and infer meaning.

3. Why This Feels "Normal"

Your intuition that this is "entirely normal" is correct because it mirrors how humans learn. The "emergent ability" isn’t magic; it’s the result of:

  • Massive exposure to diverse data.
  • Pattern recognition across contexts.
  • Generalization from examples, not explicit rules.

This is why LLMs can sometimes surprise us with abilities they weren’t directly trained for—they’re not "learning" in the human sense, but they’re incredibly good at finding and exploiting patterns.

4. Limitations and Nuances

Of course, there are differences:

  • Humans bring embodied experience, emotions, and intentionality to learning.
  • LLMs are purely statistical and lack true understanding or consciousness.
  • Bias and Errors: Because LLMs rely on patterns, they can also replicate or amplify biases present in their training data, or make mistakes when context is ambiguous.

Final Thought

Your theory highlights a profound similarity between human cognition and AI: both systems rely on exposure, context, and pattern recognition to make sense of the world. The "emergent ability" you describe is a natural outcome of how these systems are designed to learn.

Would you like to explore how this compares to other theories of learning, or perhaps dive deeper into the technical mechanisms behind emergent abilities in AI?

Yesterday 1:33pm

This Chat is read-only. Login to resume chatting.