Relevant to me is the “Defining Concreteness and Abstractness” section.

Concepts, words and semantics

First there are analogous terms: concepts, words, semantics.

Concepts can be thought as:

  • Barsalou: “knowledge about a particular category”
  • Payne: “combination of atomic units of information and meaningful relationships between those units”
  • Smith: “a mental representation of a class or individual which deals with what is being represented and how that information is typically used during the categorization”

where,

Barsalou: Barsalou, L. W., Simmons, W. K., Barbey, A. K., and Wilson, C. D. (2003). Grounding conceptual knowledge in modality-specific systems. Trends Cogn. Sci. 7, 84–91. doi: 10.1016/s1364-6613(02)00029-3 Payne: Payne, P. R. O., Mendonça, E. A., Johnson, S. B., and Starren, J. B. (2007). Conceptual knowledge acquisition in biomedicine: a methodological review. J. Biomed. Inform. 40, 582–602. doi: 10.1016/j.jbi.2007.03.005 Smith: Smith, E. E. (1989). “Concepts and induction,” in Foundations of Cognitive Science, ed. M. I. Posner (Cambridge, MA: The MIT Press), 501–526.

It seems though that:

In most concept studies, linguistic stimuli are used and thus the terms “concept,” “word semantics,” and “word meaning” are often used interchangeably.”

In this sense, this is not specially relevant to me, since I am not interested in the linguistic aspect of it. The citation of Smith is interesting though, since it is about induction.

Concreteness and abstractness

TLDR,

  • Concrete: have clear references to material objects (e.g., dog, house)
  • Abstract: references of abstract ones are not physical entities, but more complex mental states (e.g., thought, happiness), conditions (uncertainty), situations (encounter), and relationships (employment) (Borghi and Binkofski, 2014).

This is a simplified view, as:

Myachykov and Fischer (2019) have argued that, in addition to this phenomenological dimension of abstractness, there are also sensorimotor and contextual aspects, and the same word/concept may be both concrete or abstract depending on different dimensions.

I wont go into the details of this though.

Cognitive Science Stuff

What I find interesting here is the discussion about the relation of this to neuroscience studies:

Such mental (internal or cognitive) representations (Paivio, 1990) are widely investigated in cognitive psychology, psycholinguistics, philosophy of mind and related fields (Carruthers and Cummins, 1990), but often without a clear connection to neural representations, which are more commonly addressed in brain research, neuroscience and neuroimaging.

@doubt First, the idea of “mental”,”internal”,”cognitive” is probably more what Karim and I thought of originally. Not sure why it would have been abstract in the concrete/abstract sense. I think it was more like internal. Moreover, Karim is specially interested in the duality conscious/uncounscious or explicit/implicit.

Therefore, one way to extricate from this tangle could be studying processing of novel words, whose meanings are not yet represented in the participants’ minds.

@idea This is out of context, but if one extrapolates this to LLMs, it does make me wonder how they process non-existing words. What can we learn from that?. Moreover, is there a way to get the total vocabulary of a LLM?.

The dual coding theory

There is a whole emphasis on this and how it tries to explain the concreteness effect. Thought-provoking to me is the idea that:

This advantage of concrete over abstract semantics is usually called “concreteness effect”; to help explain it, Paivio suggested the so-called dual-coding theory (DCT, Paivio, 1990) which posits two functional systems associated with semantic memory: verbal-based and imagery-based (non-verbal). These representational systems are interrelated and can be active independently or in parallel. According to DCT, whereas the verbal system may be responsible for coding both concrete and abstract concepts linguistically, the non-verbal imagery system is primarily involved in coding concrete – but not abstract – concepts, enhancing their processing and leading to behaviorally observed advantages (Kuiper and Paivio, 1977).

@idea So if we extrapolate to LLMs, are there different “representational systems” in LLMs? I do remember Yoshua being interested in System 1 and System 2. Though probably not the same thing, it does make me wonder if there is a way to identify different systems in LLMs.

@idea Moreover, is there such a thing like the concreteness effect in LLMs?.

Neuroscientific approaches

Shortly, there are three approaches:

  • Neuroimaging: present different stimuli, see which areas of the brain are activated. Lots of contradictory results.
  • Brain stimulation: stimulate different areas of the brain, see how it affects the processing of concrete and abstract words? Im not really sure about how this is done.
  • Clinical cases: study patients with brain damage, see how it affects the processing of concrete and abstract words?.

@idea In any case, relevant to me is that we could mimic this for implicit/explicit representations.

The problem is that making concrete/abstract stimuli is much more straight-forward. How would one make implicit/explicit stimuli?

In any case, the rest of the paper wasnt really relevant to me.