Elsevier

Acta Psychologica

Volume 115, Issue 1, January 2004, Pages 69-96
Acta Psychologica

Object concepts and action: Extracting affordances from objects parts

https://doi.org/10.1016/j.actpsy.2003.11.004Get rights and content

Abstract

Two experiments with a part-generation task show that rated salience and production order of parts in artifacts are first predicted by their relevance for canonical actions, but also that they vary, depending on the current situation.

In three further experiments participants read sentences describing actions (e.g., ‘The woman shares the orange’) followed by objects’ parts from which it was easy or not to extract affordances (e.g., ‘slice’ vs. ‘pulp’). They had to perform a part verification task or to evaluate whether or not the combination made sense. Parts from which it was easy to derive affordances were processed earlier and the combination was evaluated as the one which made more sense.

Overall, results support the view that sensory-motor simulations underlie conceptualization and that concepts are action-based.

Introduction

Amodal views of conceptual knowledge assume that concepts, i.e., our knowledge units about categories, are represented through propositional symbols. The relationship between these symbols and their referents is arbitrary. Amodal views are usually based on the premise that perceptual and motor experience is translated into amodal and abstract symbols (Fodor, 1975; Smith & Medin, 1981; for a recent powerful formulation of this view see Landauer & Dumais, 1997). So, for example, the concept ‘cup’ would be represented through propositional features such as ‘has a handle,’ ‘you drink from it,’ and so on. There is no relationship between the features we associate with the cup and the sensory-motor experience we have when we see a cup or drink from it.

A logical consequence of this vision of concepts is the assumption that the semantic system, where knowledge is “contained”, is clearly differentiated from the other modular systems of perception and action (Rumiati & Humphreys, 1998; Tulving, 1972).

As far as research programs are concerned, the adoption of amodal views of conceptual knowledge has often led researchers to focus on the stable rather than on the flexible aspects of conceptual organization (see for example the critiques of Smith, 1995). This does not mean that symbolic amodal theories cannot account for or predict variability in conceptual organization (Landauer & Dumais, 1997). Simply, the source of this variability is ascribed to semantic relatedness between concepts and to frequency and is not attributed to the re-enactment of a sensory-motor experience.

Recently, in different fields, a different view of the relationship between cognition, perception and action has begun to gain credit (Thelen & Smith, 1994). In particular, it has been proposed that cognition is embodied, i.e., that it depends on the kind of experiences produced as a result of our having a body with a particular sensory-motor system. This view of cognition is clearly in opposition with the classical cognitivist view, according to which the mind is a device for manipulating arbitrary symbols. In line with this perspective, the view held by Gibson (1979) has been given new resonance. Many theories in various fields, from those regarding perception to those regarding attention, to language (Berthoz, 1997; Humphreys & Riddoch, 2001; Liberman & Whalen, 2000; Rizzolatti & Arbib, 1998; Rizzolatti, Riggio, Dascola, & Umiltà, 1987), share the idea that perception, action and cognitive systems cannot be considered as separated, and defend the claim that cognition is deeply grounded in sensory-motor processes. For example, in the field of vision studies, O’Regan and Noë (2001) have recently proposed that seeing is a way of acting, i.e., that it is a way of exploring the environment. Accordingly, the experience of seeing occurs when the organism masters the governing laws of sensorimotor contingency.

In the field of categorization, as well, a different view of conceptual knowledge has begun to gain credit in recent years. It has been argued that the existence of a translation process––from sensory experience into amodal symbols––is not necessary, nor is it plausible from an evolutionary point of view. In this perspective, concepts are conceived of as re-enhancement of neural activation patterns, directly referring to sensory-motor experiences (Barsalou, 1999; Barsalou, Simmons, Barbey, & Wilson, 2003).

Much neural evidence convergent with this view has been provided: in particular, proponents of the sensory-motor theory, on the basis of functional neuroimaging data, argue that conceptual knowledge does not constitute information which is physically distinct from modality specific input and output representation. On the contrary, “the features that define an object are stored close to the primary sensory and motor areas that were active when information about that object was acquired” (Martin, Ungerleider, & Haxby, 2001, p. 1023; for a review adopting a different position, see Mahon & Caramazza, in press).

An implication of this view is that, because knowledge is anchored in experience, it cannot be separated from perception and action. In antithesis with the amodal view, recent studies show that perception, action and cognition are deeply related. This idea was anticipated by Gibson’s (1979) theory, according to which perception is not a channel of information flow made of different processing stages (Sternberg, 1969) but it is deeply influenced by action and movement. Much neurophysiological evidence is consistent with the view of the reciprocal interaction of action intention and perceptual systems (Jeannerod, Arbib, Rizzolatti, & Sakata, 1995; Knoblich & Flach, 2001; Prinz, 1997; Ward, 1999). On the behavioral side, recent evidence indicates that the relationships between perception and action may be reciprocal. Tucker and Ellis, 1998, Tucker and Ellis, 2001 have shown that the vision of an object may directly elicit action patterns independent of the intentions of the subject: for example, the vision of a cup elicits (affords) the action of grasping it, and the vision of objects different in size elicits different kinds of grasping (precision vs. power grasp). Bekkering and Neggers (2002) have shown that the intention to perform an action modulates visual processing by favoring the perceptual features that are related to action. They measured the accuracy of saccades in grasping and pointing to target objects that could have a different orientation (45° vs. 135°) and color (green vs. orange). They found that the first eye movement was more accurate in selecting a target object with a given orientation located in the midst of distractors when the object had to be grasped afterwards than when it had to be pointed to. There was no difference in errors between conditions when participants had to select an object with a predetermined color. Given that orientation is relevant for grasping but not for pointing, the results indicate that action planning influences visual processing. The fact that action intention (e.g., grasp vs. point) leads to different ways of focusing on visual properties can deeply influence theories on concepts. In this line, recent proposals argue that conceptualization has its basis in both perception and action, and that it has the adaptive role of preparing for situated action (Barsalou, 2002). Thus, concepts, i.e., our knowledge units, can be conceived of as the coding of possible interaction patterns with the world surrounding us (Glenberg, 1997).

A further implication of this view is that, because knowledge is grounded in bodily and situational experience, conceptual variability is highly stressed. In fact, depending on our kind of body and on the situation we are experiencing, different conceptual aspects are activated. Accordingly, concepts are conceived of as situated and embodied, because they vary depending on the situation and on the relations between their referents and our body (Barsalou, 1987, Barsalou, 1999; Smith & Samuelson, 1997).

A central notion for the view that “knowledge is for action”(Wilson, 2002) is that of affordance (Gibson, 1979), for two reasons: (1) because it demonstrates the close connection between perception and action, and (2) because it provides an understanding of the importance of variability and situationatedness. Affordances are ways in which a perceiver can interact with an object. Thus the notion of affordance is not an absolute one. Depending on the constraints of our body, on the perceptual characteristics of objects and on the situation at hand, different objects or different parts of objects may afford actions. When we are driving a car, its steering wheel may become particularly salient for guiding our actions, while when we are repairing a car, its motor may become more salient. However, regardless of the current situation, both the car’s steering wheel and motor are probably more salient for the concept ‘car’ than other parts typically less salient for acting, such as the roof.

If the claim that “knowledge is for action” is true, an important function of concepts may reside in the role they play when one is preparing for situated actions (Barsalou, 2002). The advantage of preserving perceptual and motor characteristics of conceptual referents may reside in facilitating our interaction with objects.

The aim of this paper is to verify whether objects are represented as patterns of potential actions by focusing on their parts. But what is a part? Different proposals have been advanced, in order to define parts. For example, Biederman (1987) proposed that it is possible to segment objects into a set of 3-D volumetric primitives called geons. In a different view, Hoffman and Richards (1984) have proposed that individuals are more likely to identify a particular patch of shape as a part because it lies between two points of extreme negative curvatures, rather than assuming that objects are parsed into primitive shapes. Here, parts are defined in a very broad sense, as any fragment or component of an object-stimulus. Given that the studies proposed involve three tasks which imply a linguistic mediation––a part production, a part verification and a sensibility-evaluation on parts task––parts with an easily expressible name will have an advantage over components of objects without a name.

Parts are particularly important because actions are generally directed towards them. There is much evidence showing that objects are represented componentially (Biederman, 1987), and that parts play a special role for object concepts, especially for basic level ones (Murphy, 1991; Rakinson & Butterworth, 1998; Schyns & Murphy, 1994; Tversky, 1989; Tversky & Hemenway, 1984). In particular, as the action intention selects perceptually relevant properties in perception, in conceptualization different parts should also be activated depending on the activated action and situation.

If concepts are not represented as patterns of potential actions,

  • 1a.

    the salience of a part should be independent of the role parts play for guiding acting;

  • 1b.

    the salience of a part should not vary depending on the currently activated action.


If object concepts are represented as embodied and situated entities (Barsalou, 1999; Pecher, Zeelenberg, & Barsalou, 2003),

  • 2a.

    the salience of a part should depend on the role parts play for canonical actions directed towards an object, i.e., the most important parts in an object concept should be the ones affording the more frequent actions performed with it;

  • 2b.

    the salience of a part should vary depending on the currently activated action.


Hypotheses 2a and 2b are not conflicting; in fact, it is plausible that not all possible affordances are necessarily activated during a simulation, only affordances elicited by canonical actions as well as affordances relevant for the current goals (Kaschak & Glenberg, 2000; Zwaan, Stanfield, & Yaxley, 2002). The term “simulation” refers to the fact that we may simulate an object, for example a car, in its absence (Barsalou, 1999), because we have integrated the properties of the object in a coherent and organized system. Thus we might simulate cars’ parts by re-enhancing the sensory-motor experience we have had with them. The same is true for event sequences. For example, the simulation of the event sequence of moving a table focuses on the actions involved in lifting and pushing it, but not on the experience of eating on it.

To test these hypotheses in Experiments 1 and 2 a feature generation task was used, which is widely assumed to assess the way concepts are represented (Tversky & Hemenway, 1984; Wu & Barsalou, submitted for publication).

Consider that modal views naturally predict that concepts, similarly to percepts, have perspectives, i.e., that activated features are best predicted by action and the current context. It would be reductive, however, to argue that amodal theories cannot explain these effects. Proponents of such theories might account for such results by arguing, for example, that certain concept parts have stronger semantic associations with certain situations than with others.

In particular, proponents of the amodal view might claim that feature production tasks are not sufficiently informative, as their results might be explained by the fact that there are simply stronger lexical associations, say, among words denoting parts within a given perspective than across different perspectives. For this reason in Experiments 3 and 4 a part verification task was used, and in Experiment 5 a sensibility-rating task of words and sentences was used. These methods allow an easier control of the association degree between words denoting parts and words and sentences denoting situations, and make it possible to rule out the hypotheses that the results might be due to semantic associations rather than to the creation of mental simulations of the objects.

Experiment 3 tests with a part verification task the hypothesis that different parts are activated depending on the action expressed by a sentence. If it is true that processing a sentence like ‘He grasped the knife’ activates a mental simulation of the scene, and if it is true that objects are represented componentially (Biederman, 1987), the part ‘handle’ following the sentence should be verified faster than the part ‘blade’, due to the fact that the part ‘handle’ better affords the action of grasping a knife than the part ‘blade’. Consider, however, a possible objection to this experiment. The demonstration that different actions activate different parts may simply show that conceptual organization is variable, but not that concepts are represented through simulations preserving their perceptual and motor characteristics. Simply, such a result could be explained by a semantic network account: the verb ‘grasp’ may be more semantically associated with ‘handle’ than with ‘blade’, while the verb ‘cut’ may be more semantically associated with ‘blade’ than with ‘handle’.

In order to rule out a semantic network account, in Experiments 4 and 5 parts were selected which were not semantically associated to a given action, but which, due to their perceptual features, might or might not afford a particular action (see Glenberg & Robertson, 2000). Parts from which it is easy to extract affordances will be called affording parts. For example, the neck of a bottle can more easily afford the action of putting it down than its cork, even though neither the word ‘neck’ nor the word ‘cork’ are strongly semantically associated with the sentence expressing the action of putting something down. Experiment 4 tests whether affording parts are processed quicker than non-affording parts in a part verification task; Experiment 5 tests whether affording parts are considered more sensible in the context of sentences compared with non-affording parts. If affording parts are processed earlier and evaluated as being more plausible than non-affording parts, this advantage cannot be due to the semantic relatedness between words denoting parts and sentences. Thus, the results will favor the view according to which concepts are simulations preserving perceptual and motor characteristics of objects. A possible advantage of preserving perceptual characteristics of objects is to prepare for situated action (Glenberg, 1997), thus facilitating interaction with the object.

Section snippets

Experiment 1

In this experiment the type of simulated interaction with objects was manipulated: three groups of participants were required to imagine using/acting, building, or seeing objects. For the critical objects they were also required to produce parts. If hypothesis 1 is true, i.e., if concepts are represented through amodal and arbitrary symbols and are stable across situations, then (a) parts relevant for acting should not be produced more frequently across conditions and earlier than other parts;

Experiment 2

If object concepts are represented as action-based, then in a neutral condition the parts relevant for actions should be those which are rated as most important and produced earlier. Experiment 2 represents a control experiment of Experiment 1, consisting of a simple part-generation task, performed without asking participants to simulate a particular situation.

Comparison between Experiments 1 and 2

If hypothesis 2a is true, i.e., object concepts are patterns of potential actions and the canonical perspective in which objects are represented is action/use dependent, then the results obtained in Experiment 2, i.e., in a part-generation task performed in a neutral condition, should resemble those obtained in the action/use condition of Experiment 1 and differ from those obtained in the building and vision situations of Experiment 1 (see Fig. 1).

Experiment 3

The aim of Experiment 3 is to show with a part verification task that, depending on the actions expressed by a sentence, different parts of an object are activated. For example, the part ‘slice’, due to its perceptual features, better affords the action of dividing an orange than the part ‘pulp’.

Both artifacts and natural kind concepts were used in order to see whether the hypothesis according to which concepts are patterns of potential actions holds for both kinds of concepts or only for

Experiment 4

The aim of Experiment 4 was to verify, using the same procedure, that the results of Experiment 3 were not simply due to word association but that they were the result of a meshing process between the action to accomplish and the objects’ parts. A new list of trials was construed: the material was controlled so that in each sentence the verb was not more semantically associated to the affording than to the non-affording part.

Experiment 5

The aim of Experiment 5 is to confirm the results of Experiment 4 with a different paradigm, i.e., to demonstrate that the results of Experiment 3 are not simply due to word association but to the sentences expressing actions which selected affording parts. The same material of Experiment 4 was used. The task consisted in rating on a seven-point scale how much a given part made sense in combination with the sentence.

The paradigm used strongly resembles the one used by Glenberg and Robertson

General discussion

The results have a number of theoretical implications.

First, they indicate that parts relevant for actions are activated more than other parts across situations; this suggests that objects are conceived of in terms of the potential actions we may perform with them (hypothesis 2a). The fact that default affordances are dictated by their importance for more frequent actions (Palmer, Rosch, & Chase, 1981) is very plausible from an evolutionary point of view. There is much evidence converging on

Acknowledgements

I would like to thank the students of a seminar in cognitive processes (in alphabetical order) who participated in the preparation of the material, data collection, and the ideation of experiment 3: Federica Calanchi, Diego Cofone, Maurizio Giudice, Francesco Migliorini, Edoardo Osculati, Andrea Rubbini. Thanks to Ann Gagliardi for help with English. I would also like to thank Art Glenberg for useful suggestions during planning of the last experiments and for discussion of these themes and to

References (73)

  • A.M Liberman et al.

    On the relation of speech to language

    Trends in Cognitive Sciences

    (2000)
  • G Rizzolatti et al.

    Language within our grasp

    Trends in Neurosciences

    (1998)
  • G Rizzolatti et al.

    Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention

    Neuropsychologia

    (1987)
  • P.G Schyns et al.

    The ontogeny of part representation in object concepts

  • L.W Barsalou

    Context-independent and context-dependent information in concepts

    Memory and Cognition

    (1982)
  • L.W Barsalou

    The instability of graded structure: Implications for the nature of concepts

  • L.W Barsalou

    Perceptual symbol systems

    Behavioral and Brain Sciences

    (1999)
  • L.W Barsalou

    Being there conceptually: Simulating categories in preparation for situated action

  • L.W Barsalou et al.

    Perceptual simulation in conceptual tasks

  • H Bekkering et al.

    Visual search is modulated by action intentions

    Psychological Science

    (2002)
  • A Berthoz

    Le sens du mouvement

    (1997)
  • I Biederman

    Recognition by components: A theory of human image understanding

    Psychological Review

    (1987)
  • Borghi, A. M. (submitted for publication). Object concepts and action. In D. Pecher, & R. Zwaan (Eds.). Grounding...
  • Borghi, A. M., & Barsalou, L. W. (2001). Perspectives in the conceptualization of categories. In Abstracts of the 42th...
  • Borghi, A. M., Glenberg, A. M., & Kaschak, M. P. (submitted for publication). Putting words in...
  • L.J Buxbaum et al.

    The role of semantic memory in object use

    Cognitive Neuropsychology

    (1997)
  • L.J Buxbaum et al.

    Function and manipulation tool knowledge in apraxia: Knowing ‘what for’ but not ‘how’

    Neurocase

    (2000)
  • E Capitani et al.

    What are the facts of semantic category-specific deficits? A critical review of the clinical evidence

    Cognitive Neuropsychology

    (2003)
  • L.A Carlson

    Object use and object location: The effect of function on spatial relations

  • Chaigneau, S. E., & Barsalou, L. W. (in press). The role of function in categories. Theoria et Historia...
  • K Duncker

    A qualitative (experimental and theoretical) study of productive thinking (solving of comprehensible problems)

    Journal of Genetic Psychology

    (1926)
  • J Fodor

    The language of thought

    (1975)
  • C Gerlach et al.

    When action turns into words. Activation of motor-based knowledge during categorization of manipulable objects

    Journal of Cognitive Neuroscience

    (2002)
  • J.J Gibson

    The ecological approach to visual perception

    (1979)
  • A.M Glenberg

    What memory is for

    Behavioral and Brain Sciences

    (1997)
  • A.M Glenberg et al.

    Grounding language in action

    Psychonomic Bulletin & Review

    (2002)
  • Cited by (72)

    • On the neurocognitive origins of human tool use: A critical review of neuroimaging data

      2016, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      Moreover, it encodes egocentric relationships, namely, relationships between the user (and particularly his/her hand) and a tool. The manipulation knowledge hypothesis has found resonance in recent years, with the growing interest in the embodied cognition approach, suggesting that knowledge is constituted by information represented within the motor and sensory systems (Barsalou, 2008; Borghi, 2004; Borghi et al., 2013; Pezzulo et al., 2013a, 2013b; Mizelle and Wheaton, 2010; Thill et al., 2013). As mentioned just above, manipulation knowledge encodes egocentric relationships (i.e., user-tool).

    • Visual objects speak louder than words: Motor planning and weight in tool use and object transport

      2015, Acta Psychologica
      Citation Excerpt :

      Theories of embodied cognition posit that cognitive representations are grounded in the brain's motor and sensory systems (Barsalou, 2008; Semin & Smith, 2008). In this view, reading a word activates sensorimotor representations in a similar manner to seeing the physical object the word represents (Borghi, 2004; Glenberg, 1997; Pecher & Zwaan, 2005). Evidence with response time studies (Borghi & Riggio, 2009; Buccino et al., 2005), kinematic measures (Glover & Dixon, 2002; Nazir et al., 2008) or brain-imaging studies (Grafton, Fadiga, Arbib, & Rizzolatti, 1997; Kellenbach, Brett, & Patterson, 2003; Pulvermüller, 2003) supports this view by showing that words evoke perceptual and motor information regarding their referents.

    View all citing articles on Scopus
    View full text