Open Access

Can a kitchen teach languages? Linking theory and practice in the design of context-aware language learning environments

  • Anne Preston1Email author,
  • Madeline Balaam1,
  • Paul Seedhouse1,
  • Salla Kurhila2,
  • Lari Kotilainen2,
  • Ashur Rafiev1,
  • Daniel Jackson1 and
  • Patrick Olivier1
Smart Learning Environments20152:9

https://doi.org/10.1186/s40561-015-0016-9

Received: 29 August 2014

Accepted: 7 May 2015

Published: 15 May 2015

Abstract

Smart learning environments offer rich opportunities for language learners. In particular, context-aware systems which allow learners’ progress to be sensed within and across an activity, enable instructed language learning to move beyond the traditional confines of the classroom walls. In this paper we present the European Kitchen, a real-world task-based environment for cooking and language learning. In doing so, we demonstrate how specific design decisions, in the development of this longer-term iterative design project, conjoin Human Computer Interaction practice and learning theory for situated language learning. We also show how this approach is combined with Conversation Analysis, which is used as a tool to measure the impact of these decisions on the interactions taking place in and with the kitchen. Our work reveals that in order to design for and evaluate effective and meaningful language learning, there should be more balance between technologically-driven theory and theory driven research which has a strong pedagogical foundation. Our work has implications for a transferable, interdisciplinary model of task-based, situated learning which can be applied and adapted to different skill and knowledge sets.

Keywords

Task-based language learning Technology-enhanced learning Language learning pedagogy Situated learning Information processing theory Attentional manipulation

Introduction

Smart language learning environments, such as those with the capacity to provide timely and situated scaffolding, enable individualized and autonomous learning to occur in real-world tasks. Learners are able to experience what they can do using their language skills rather than merely having corrected what they can’t. In turn, this helps builds motivation, confidence and language knowledge. Designing for meaningful and effective smart language learning is not straightforward. One core issue is the lack of theoretically grounded frameworks and guidance for their development. In a recent paper on the role of learning theory, Antle and Wise (2013) argued that theory is used “at a very broad level” (p. 5) without specific links between the affordances of technological interfaces and how they mediate and affect learning processes. They also posited that there are still very few descriptive examples of interdisciplinary approaches which draw on theories of learning to analyse learning behaviour and formulate frameworks for subsequent use (2013). This view is echoed in the Human Computer Interaction (HCI) field by Rogers (2012) who framed HCI future as the “convergence of concepts and methods from different research areas” including “using a combination of strategies, design methods and theories” (p. 86-87). With specific reference to smart learning, Hwang (2014) called on researchers to propose new thoughts about the pedagogy of smart learning and to develop more in-depth analyses of learning patterns. Both calls suggest a similar merging and mixing from the point of view of established learning theory and approaches to evaluation.

This paper addresses the convergence of theory and practice in designing for smart learning by explicating the interdisciplinary process undertaken in the design and evaluation of the European Kitchen (EK) project. The EK is a novel context-aware environment designed for pairs of language learners to cook dishes linked to different cultures and countries in a foreign language. In particular, we show how we intertwined language learning theory and HCI practice in its development and coupled this with methods aimed at examining the moment to moment development of learning processes to drive the evaluation.

Background: related work

Smart environments are attractive to language learners and teachers alike as they support situated learning practices which promote enjoyment, motivation and confidence, as well as autonomous and authentic learning experiences which are seen to greatly enhance learning outcomes (Bax 2003; Chambers and Bax 2006). Such learner-centred approaches hold much potential for the development of new pedagogies where technology is used as a means rather than end for learning. These technologies are designed for learning to take place outside the classroom (with minimal intervention or without the presence of a teacher) but with the appropriate scaffolding to support learners’ management of these environments, to achieve the same outcomes which may be found in a regular classroom. What these systems have in common and importantly, what makes them smart, is their capacity to support what Hutchins (1993) described as the “heart of intelligent performance […] the properties of interaction between individual minds and artefacts in the world” (p. 62).

The Ambient Wood (Rogers et al. 2004) and Hunter and Snark projects (Price et al. 2003; Harris et al. 2004) are well-cited interdisciplinary examples of learning applications in context-aware environments beyond the classroom. These projects developed novel interaction environments for enquiry-based situated learning experiences. In the Ambient Wood project, mobile devices were used to present information to children engaged in scientific enquiry and explore biological ideas and triggered by the immediate environment of a woodland area. As pairs of children interacted with the woodland area, they captured data about the wood and used these later on as a reflection tool to form and test hypotheses about the various woodland habitats. Games developed through the Hunter and Snark enabled interactions for children to create and discover more about characters in stories through audio-visual feedback received via Radio Frequency Identification (RFID) tags embedded inside coloured clay, mobile devices and ultra-sonic sensing.

The use of physical devices to collect data demonstrates how context-aware technologies can be integrated into existing educational practices as well as having the potential to enhance learning (Price et al. 2003; Rogers et al. 2005). In this previous work, theory had a specific role to play as part of an integrated design process which was enhanced and constrained by learning aims and technological affordances. More recent work (Price and Falcão 2011; Price and Jewitt 2013a, 2013b) showed how developments of these novel learning environments enabled new forms of engagement and interaction which exploit and build on learning theory. For example, those rooted in socio-constructivist approaches through specific and systematic investigation have led to new frameworks, rich descriptions and classifications for the design and evaluation of learning technologies (Price and Jewitt 2013a). The EK builds on such innovative research by promoting the learning of traditional language curriculum content outside the classroom. The EK takes language learning into the real-world setting of an everyday and familiar setting where there is close relationship between the knowledge, skills and competences being acquired and how they will be ultimately applied beyond experimental and educational settings.

In the language learning field more specifically, a recent example of a context-aware system which focusses on discovery is iSpy (Lee and Doh 2013), a language learning toy for preschool children augmented with capabilities based on RFID tag mapping. The toy reads RFID tags placed on objects in a home such as a sofa or book case and provides customized audio content such as storytelling, fairy tales, sounds and chants in a second language on a smart device. The system also stores and processes information linked to play in order to provide updates and feedback to parents via a smart device about what the child played with or was interested in.

Beyond the context of play, a system using RFID technology for language learning was the TANGO (Tag Added learNinG Objects) project developed by Ogata et al. (2004). The TANGO system tasked language learners with adding RFID tags to physical objects in a room to teach Japanese vocabulary. Housed in a hand-held device, the system aurally asked learners questions linked to a specific object which they had to answer by selecting the correct objects via tagging. The system detected learners’ actions in terms of whether or not they had selected the correct object. It also offered access to additional information if required, such as a repetition or hint. Learners could also add a personal comment or annotation via a user interface.

Ogata et al. claimed that their system specifically engaged with “right time and right place learning”. Their user feedback via self-reports showed that learners found it a helpful tool and gained a sense of achievement through the game. In other work on language learning, Ogata et al. (2010) used wireless digital sensors to support the learning of Japanese mimetic words and onomatopoeia (JAMIOLAS2). JAMIOLAS2 used context-aware technology to get data such as temperature and light from different environments in order to recommend “the right place for the right person to learn the right” Japanese mimetic words and onomatopoeias.

A more specific focus on linguistic knowledge is found in the work of (Dearman and Truong 2012). Vocabulary Wallpaper is an interactive application for a mobile phone which incorporates vocabulary learning as a live wallpaper using the user’s location to display relevant vocabulary (for example, nouns and verbs), linked to where the user is in their day to day activities (for example, a coffee shop or a supermarket). Dearman and Truong stated that the Vocabulary Wallpaper application was based on the notion of microlearning, which occurs in the brief pervasive learning opportunity windows, for example, when users are waiting in a queue in a coffee shop or in a lift at a hotel. The EK develops on the valuable work of Lee, Doh, Ogata, Dearman and Truong, which showed that context-aware environments can support object-specific vocabulary training. We propose that as well as more traditional exercises, everyday real world tasks can be augmented by technology in an almost invisible way to engage users in learning by doing which goes beyond the word level.

Methodologically, the role of theory and its relationship to learning design, as well as how learning is subsequently evaluated in these language learning systems, is not as prevalent as the earlier work of Price, Rogers and Harris et al. For example, although Dearman and Truong presented an explanation of vocabulary learning via the Vocabulary Wallpaper (implicit learning through repeated exposure to the second language through interactions with the mobile phone application) they made no reference to theories of language learning or vocabulary learning processes. The notion of implicit learning (unconscious learning) and explicit (conscious learning) has long been an issue of debate (see Hulstijn 2005, for an overview) within the second language learning sciences literature. The work of (Lee and Doh 2013) and Ogata et al. (2004, 2010) did not draw on any specific theory of language learning or systematic analysis of vocabulary learning outcomes. This is noticeable given the focus on a specific language dimension (vocabulary) and the role of interactional processes with and around the systems.

How interactions with and around the technologies are measured, beyond user feedback, is becoming of increasing importance in evaluating the impact of technology on learning and particularly with reference to novel learning environments (Hooper et al. 2012; Price and Falcão 2011; Verdines, 2012; Price and Jewitt. 2013a, 2013b). As designers of smart learning environments, we can do more to show how our systems are designed and evaluated according to appropriate learning theories and their related methodologies. The work reviewed here on language learning would also suggest that we aneed a tighter focus on a dedicated design approach to supporting novel and creative approaches to learning outside the classroom, one which integrates more balance between technologically-driven theory and theory driven research.

Theory in the language learning sciences

In the language learning and technology field more specifically (also known as Computer Assisted Language Learning, CALL), Chapelle (2003) has long argued that the use of theory is vital for making links beyond the specifics of particular study and to show the relevance between research and practice in teaching and learning. In the fields of Second Language Learning and Acquisition (SLL/A), the role given to learning theory in development practices are particularly relevant to technologies for language learning. Language learning is seen as a highly complex process and similarly to complex systems research, its order is seen to arise out of the interaction of its components (Seedhouse 2010). The SLL/A fields are not lacking in theoretical work and are dominated by a wide range of theories which seek to explain the processes involved in learning another language when a person already knows another (see Mitchell et al. 2013 for a summary and Myles 2014). Whilst there is no definitive model, out of the variety of perspectives that exist, psycholinguistic and cognitive approaches have had the most impact on research into understanding the nature of SLL/A and have generated the most interest when applied to the study of second language learning and teaching (Mitchell et al. 2013; Gass and Selinker 2008).

As opposed to generative perspectives, which generally seek to account for language learning as linked to some innate capacity (Chomsky 1968; White 1989), cognitive approaches focus on the representation and control of language knowledge in language performance (Gass and Selinker 2008). An Information Processing (IP) account for deep learning sits within these cognitive approaches and focuses on the automaticity and restructuring of language knowledge to explain the process of learning (Maclaughlin et al. 1983). IP has also been proposed as a sound basis for to underpin instructional design in the context of language learning tasks (Skehan 2003).

IP theory forms part of a well-known body of theoretical work in HCI based on cognitive modelling and has inspired the design of many systems. Such mental models attempt to “characterise the knowledge that people are assumed to have when interacting with a system” (Rogers 2012, p. 27). Rogers (2012) has suggested however that the future of HCI practice is increasingly pointing towards “pluralistic” approaches where different theoretical positions “can sit side-by-side and be synthesized in novel ways” (Rogers 2012, p. 86). Recent work around the theoretical basis of design for learning specifically posited that IP perspectives in interaction design can provide “a theoretical lens with which to consider individuals’ internal cognition, the process of how learners manage and organize information from the world to acquire memory structures to represent it” (Antle and Wise 2013, p. 5).

Second language learning IP approaches seek to describe and explain learning as a result of the psychological mechanisms involved. Here more specifically, learning has been described in terms of psycholinguistic processing, involving:

“the transfer of information to longterm memory and is regulated by controlled processes […] it is controlled processes that regulate the flow of information from working to long-term memory […] thus, controlled processes can be said to lay down the “stepping stones” for automatic processing as the learner moves to more and more difficult levels” (Maclaughlin et al. 1983, p. 139-140).

What and how attention is given by the learner to formal properties of language in a given situation is seen to determine the extent to which information is transferred to the long term memory and therefore the learning that takes place (Maclaughlin et al. 1983). Maclaughlin et al’s classic work characterised this relationship in terms of 4 cases which were distinguished according to where the learner’s attention to information can differ:
  1. 1)

    controlled performance based on formal rule learning;

     
  2. 2)

    automatic performance in a test situation;

     
  3. 3)

    controlled performance based on implicit (unconscious) learning

     
  4. 4)

    automatic performance in a communicative situation.

     

An IP approach helps to provide a model for conceptualising what happens inside the head of a language learner and the relationship between the internal representations of second language knowledge and different external situations. How IP theory and explanations of attention-giving translate into a basis for HCI practice in design is another challenge. This challenge underpins the system presented and evaluated in this paper.

In the following section, we describe the context-aware EK kitchen designed to motivate and enable language learning. We begin with some background to our longer-term iterative design project which helps to situate why we ended up with a particular focus on the domain of a kitchen. We then show, in relation to specific presentation of the context-aware system, how we have integrated IP theory into both the interaction design of the system and the task-based language learning framework which was used to organize learning processes. Our aim is to be explicit about the relationship between the theory and HCI practice informing our decision making. A demonstration of our evaluation method is then presented, drawing on the analytical scope of Conversation Analysis (CA). Finally, we make links between the CA and our design then provide some implications for further research.

The European kitchen: system design and implementation

The European Kitchen (EK) was designed for pairs of language learners of all entry levels to cook dishes linked to seven European cultures and countries: Catalan, English, Spanish, French, German, Italian and Finnish. Learning another language, to any proficiency, offers rich rewards including improved cultural understanding, communication abilities and job prospects and is inextricably linked to human, cultural and linguistic rights (Piri 2002). The domain of an everyday kitchen provides a familiar environment for introducing learners to language and arguably has the potential to reach a far greater variety of people than the classroom alone.

The EK used wireless digital sensors which are integrated into the handles of cooking utensils, incorporated into containers that hold ingredients and directly attached to kitchen appliances (e.g., oven knob, weighing scales). This augmented set up, which can be integrated into any existing normal kitchen, provided the learning environment with sensing capabilities that allowed for the tracking of learners’ progress in a cooking task and provide situated and language-related feedback in a real-world context for authentic language use. The EK was designed to lead pairs of learners step by step through the practical activity of the preparation of a dish linked to the chosen target-language culture (Fig. 1).
Fig. 1

Augmenting a learning environment for language learning and cooking

The EK grew out of three previously published kitchens (the Ambient Kitchen, Olivier et al. 2009; the Assistive Kitchen, Kranz et al. 2007 and the French Kitchen. Hooper et al. 2012). The move to the EK (and this second iteration of the language learning kitchen), came about through the opportunity to extend the number of languages available through the kitchen and develop them in a European collaborative project (five partners). The data used in this paper are drawn from a study of 250 learners from school-age, vocational, migrant and adult learning contexts who had a range of European and non-European language skills. 125 video-recorded cooking sessions involving pairs of cooks took place based on recipes in English, Spanish, Catalan, Finnish, Italian and German.

Linking theory, pedagogy and HCI

Augmenting an authentic task with a clear goal and end product lends itself well to Task Based Language Learning (TBLL). TBLL is gaining global appeal in the language teaching world and is an approach which interests both practitioners and researchers. The notion of a task for language learning is specific in that it is seen as “an activity which requires learners to use language, with emphasis on meaning, to attain an objective” (Prabhu 1987). TBLL is thus an approach which seeks to develop language learning by prompting learners to achieve a goal or complete a task. Like real-world tasks, such as asking for directions, TBLL seeks to develop students’ language through providing a task and then using language to solve it. TBLL stands in contrast to more traditional approaches where the learning takes place via explicit grammar explanations and drill exercises. TBLL task design incorporates two key features: 1) a focus on supporting task progression (for example, by providing adequate instructions to the learner so that they can achieve the goal of the task) and 2) a language learning specific focus. A language learning specific focus can be facilitated in the way that the task is designed to provide feedback to the learner in terms of their comprehension whilst they progress through the task.

The EK drew on a TBLL framework based on Skehan (1998, 2003), which divides real-world language learning activity in the completion of a task into 3 phases: pretask, during-task and post-task. These 3 distinct phases move the learner through: a) planning to use language; b) using language to complete a specific meaningful task and c) consolidating and reflecting about language. There are synergies here with HCI practice and interaction design, in that the notion of a framework is seen as a “set of core concepts, questions or principles to consider when designing for a user experience” (Rogers 2012).

Information processing theory applied to TBLL

The TBLL framework informed the overall structure of the cooking session but did not explicitly provide any basis for the theoretical underpinnings which could ultimately account for learning. The notion of a learning framework is therefore different from a learning theory in a formal sense. When designing for interaction with the system within the three phase TBLL framework of the EK we drew on IP theory.

An important factor informed by IP for the interaction design in the EK was how the system presented the information (in this case another language) and how this was subsequently manipulated by the learner so as to promote deep learning. The information had to support task progression (adequate step by step cooking instructions to prepare the dish) and provide feedback designed in such a way as to allow learners embedded in the task activities to notice (consciously or unconsciously) the formal properties of the language. In terms of IP, as learners’ progressed though the task, it was hypothesized that they would be operating between the working and long term memory systems.

Drawing on IP in language learning more specifically, we aimed to maximize the means through which opportunities for attentional manipulation (Skehan 1998) could take place. Manipulation is more specifically linked to the notion of ‘noticing’ (Schmidt 1990), which posits that the more frequent and salient a language form, the more likely it is to be processed and picked up by the working memory.

We designed our system in the different stages of the task to strike a balance in the interaction design between case 3 (performance based on implicit learning) and case 4 (performance in communicative situations) in the IP perspective proposed by MacLaughlin et al (1983). For case 3, our design decisions were formulated to support language learning at an interface between facilitating learners’ real-time processing of and attention to what was being asked of them (the cooking instruction) and how it was being asked (the language contained in the cooking instruction). For case 4, our priority was to not distract learners from the real-world and authentic communicative context. Striking this balance also reflected the dual nature of the task itself, one of cooking and language learning. In doing so, we expected the attentional focus to go back and forth as the process unfolded to the learner. The following section outlines how the three phase TBLL framework is underpinned by IP information theory and a focus on attentional manipulation.

Pre-task: presentation and preparation

From a practical perspective, the pre-task was essential to the overall task design to provide a dual focus on cooking and language. It was divided into presentation and preparation of language and cooking knowledge.

The pre-task involved situated prompting whereby learners collected the individual utensils and ingredients needed to prepare a dish. In undertaking this section of the task, not only would learners start to plan to use the target language throughout the cooking task, but they also gain experience of the underlying technological infrastructure of the kitchen: the ways by which the kitchen is able to make sense of their activity in the kitchen.

In addition, learners in this phase could explore other ways of interacting with the EK:
  1. 1)

    User controlled repeats and ‘back’ and ‘forward’ functions to let learners move back or forward a step

     
  2. 2)

    Timely automated help available indicators prompted by timeouts (for example, help prompts such as “do you need any help?” in all languages)

     
  3. 3)

    one user-controlled help visual prompt made up of a photo of the utensil or ingredient item accompanied by both textual and audio prompts.

     
The decisions embedded in the pre-task linked to the following underlying principles:
  1. a)

    the activation of existing language knowledge;

     
  2. b)

    the introduction of new language knowledge;

     
  3. c)

    the making salient of language forms presented in the overall activity;

     
  4. d)

    the practical preparation of learners for the cooking activity (to ease the burden of cognitive load linked to cooking procedure).

     

From a learner-centred perspective, the pre-task provided the opportunity for the learner to decide for themselves (to ‘notice’) what was important for them in terms of attention to language knowledge.

Interaction tools

The addition of two non-utensil based tangible interaction tools within the kitchen, a green and red tool, was driven by the desire to offer user control over the cooking process and soliciting help. More specifically, rather than the alternative of operating a Graphical User Interface (GUI), these tools were designed to keep learners focussed on the task, on the physicality of the kitchen and the cooking task. The green tool enabled learners to alert the kitchen of their cooking step completion where it would have otherwise not been possible to sense (for example, kneading dough or folding a pastry sheet). The red tool also offered an alternative to using a GUI when the learners wanted to request help to be made available (Fig. 2).
Fig. 2

Red and green interaction tools

During task: facilitating attention in language learning

For the ‘during task’, the core cooking activity, the system was designed to allow learners to move step by step through a set of cooking actions. After the provision of an initial instruction in the target language, a range of supports were available to guide them in their completion of the step, should they need them. The EK was designed to offer learners the choice as to whether they would first like any help before it was actually given. This feature was therefore designed to enable to learners to focus attention on resolving any language trouble themselves. The supports were tailored in the level of support provided to the learner, from an initial audio repetition via automated audio instructions with emphasis, to a photo accompanied by the instruction through to a video of the step being carried out, again with the audio-instruction. The escalation meant that the kitchen adapted to the real-world situation of the learners at a specific moment. If only a first level of escalation was required then only an automated audio instruction with emphasis was provided and the system would move on to the next prompt relevant to the learners’ position in the task.

For the learner, these supports were designed to help them understand the kitchen’s instructions and therefore what they needed to do next to complete the cooking task. It was essential given the TBLL context of the EK that the audio-visuals both assisted the learner when they came across previously unknown language and/or required support with a cooking action. Such a requirement reflected the balance between controlled (more explicit focus on language) and automatic (focus on the task meaning) performance. From the point of view of theory and design, the escalation of support therefore offered different levels of control in performance and therefore over their attention to language. The audio-visuals were designed to act as multimedia glosses (Bowles 2004; see also Mohsen and Balakumar 2011 for an extensive recent review). For the language learner, these glosses provided a substitution for a dictionary and provided different modalities (textual, visual and auditory) and modes (video, picture and text) through which information could be accessed. In the EK, their specific role was designed to facilitate attentional manipulation in terms of how information about the cooking step was provided (and thus language input) and how learners accessed it (i.e. via the GUI or through the use of the interactive tools). The glosses can be viewed as a type of “adaptive learning technique” for language learning in real-world tasks more specifically (Hwang 2014).

The use of multimedia glosses can initiate the processing of information through dual systems such as ‘text and sounds’ and ‘pictures or objects’. Rather than increasing cognitive load, information accessed via these two systems activates the other and therefore eases the burden of processing (Mohsen and Balakumar 2011).

The automated and user-controlled situated support were made available in three different ways:
  1. 1)

    user-controlled ‘back’ and ‘forward’ functions allowing learners to navigate the stages in the cooking task if required, as well as user-controlled repeats of an instruction (both accessed via the GUI)

     
  2. 2)

    Timely automated help available indicators (“do you need any help?”) prompted by timeouts within the EK system

     
  3. 3)

    After the prompting of the ‘help-is available’, user-controlled help prompts were made available in varying degrees (repetition with stress, image plus instruction, video plus instruction) to decrease the linguistic complexity (accessed by either the GUI or the red tool).

     

A final mechanism provided language input beyond the main cooking instructions: the automated provision of tips. These were phrases about cooking technique intended to provide advanced learners with more complex language while at the same time conveying cooking relevant information.

Post-task: consolidation and reflection

The post-task focused on consolidation and reflection of learning and attention towards a more explicit and controlled focus on linguistic knowledge (case 1, Maclaughlin et al.). This task was carried out away from the EK through a series of face-to-face activities.

A demonstration of the EK as classroom

This section presents our excerpts based on findings from our qualitative analysis which shaped the evaluation of the EK. We drew on a qualitative evaluation approach to enable the systematic study of user behavioural phenomena in the EK (Patton 2014; 2015). Our evaluation methodology aimed to explore the extent to which the theoretical grounded design decisions led to interactions within the EK that facilitated language learning.

In this demonstration, we draw on data from the English and Finnish learning kitchens, which were part of a corpus of 125 video-recorded cooking sessions. The corpus included cooking sessions from five European countries and seven different languages. Full ethical considerations were applied in the collection of the data and participants were offered different alternatives as to how data featuring them could be subsequently shared. Footage of different learners using the full range of language learning kitchens can be found at www.europeandigitalkitchen.com.

The evaluation work undertaken of the EK addressed the kinds of issues raised by (Rogers 2012), Hwang (2014) and Price and Jewitt (2013a; 2013b): that learning environments could benefit from a more holistic view of learning which considered how learners’ orientation ‘in the moment’ to examine the significance of the underlying design. Lueg’s work on analysing context awareness is also relevant which warned against “isolating” specific aspects of context and promoted a closer focus on “understanding how and to what extent different aspects of environments influence cognition and actions when used as a resources” (Lueg 2001).

Conversation analysis as an evaluation tool

We used Conversation Analysis (CA) (a classic introduction to CA can be found in Schegloff et al. 1977; Sacks et al. 1974) as a methodological tool for the analysis of moment-to-moment language learning processes. CA, as a tool, helps to “identify ways in which participants themselves orient to, display and make sense of each other’s cognitive states” (Drew 1995) but does not make inferences about what a learner ‘is thinking’. With its roots in sociology, CA for HCI has been used since the early 1990’s to investigate interaction around computers (for example, Norman and Thomas 1991; Luff et al. 1990; Woodruff et al. 2002). More recently, CA has formed part of a shift in HCI to allow a closer focus on embodiment, interaction and learning in order to acknowledge the impact and influence of the complexities of the total context in learning environment (Hooper et al. 2012; Verdines 2012; Price and Jewitt 2013a, 2013b).

CA treats verbal and non-verbal interaction as based on general principles which are part of the structural ‘toolkit’ of conversation (turn taking, adjacency pairs, repair). This toolkit is open to a range of uses (referred to as organisations) according to the local activity in which one is interacting (Sacks et al. 1974). By examining the moment to moment development of interaction with and around the EK therefore, the interactions of the learners are analysable in terms of demonstrating their orientations to the local activity.

In terms of the qualitative analytical results presented here, the examples focus on the way in which the conversational ‘tools’ of repair are used in the interactions. Repair (Schegloff et al. 1977) concerns the treatment of trouble in everyday talk but also plays an important part in the study of learning discourse as the initiation of solutions to breakdowns in communication provide learning opportunities. Repair is therefore a pedagogical concept as well as a feature of general interaction in everyday settings (Seedhouse 2010). Repair practices are analyzed in terms of who initiates: the interactant/s (self-initiation) or someone else (other-initiation) and who repairs the breakdown (in the context of the cooking activity, the completion of the step demonstrates repair and a solution to the trouble). The occurrence of breakdowns and how they get resolved is rooted in the localised ways in which interactants’ organise their talk as it happens.

The examples below are from two cooking sessions chosen for this demonstration because of their role in the pre-task and during-task and because they are from two different language learning kitchens. Due to limitations of space and our focus on the application of CA as an analytical tool to further our evaluation work, we focus on only two representative cases taken from our collection of analysed cases in the five language kitchens. A purely CA-based study would conventionally consist of an elaborated discussion of numerous transcribed episodes representative of behavioural phenomena in a data set (more information about methodological considerations in CA can be found in the classic work of Ten Have (1990).

In line with the CA approach taken to analyse the organization of the interactions, the transcription conventions adopted represent the EK as one of the participants in the interaction. Its verbal turns are represented as plain speech (including pauses, emphasis etc.), and its non-verbal actions (such as the help or success sounds, or the display of pictures) are encoded in double brackets. Other relevant symbols are in Table 1.
Table 1

Transcription guide

EK:, L1, L2

European kitchen, Learner 1, Learner 2

[

Overlapping speech

-

An abrupt cut off

((talk))

Non-verbal actions produced by kitchen or learners

<>

talk produced slowly

(.)

a very short pause

//help available//

Sound to indicate help is available made by the kitchen

HT

Help Tool

°talk °

Talk is quieter than the surrounding talk

?

Rising intonation

Te:xt:

extended sound

text

text in bold indicates a translation into English but not talk in English produced by the speakers

English: the case of the pastry brush in the pre-task

The interaction opens with an audio prompt from the EK which communicates a next step: collect the pastry brush. The learners take their time to select what they deem to be the appropriate item, engaging in some negotiation about what the pastry brush could be from the physical objects in front of them (as shown in Fig. 3) and finally picking up the sieve in line 7. In line 8, we see the initiation of a repair sequence in which the kitchen indicates to the learners that there is a problem with their choice of object through use of the timely correction prompt: Table 2.
Fig. 3

“I think it’s this one”

Table 2

Lines 1 to 8

1 EK

pastry brush

2 L1

huh

3 L2

huh

4 L1

((moves to other side of kitchen counter)) I think it is this one ((pointing))

5 L2

yeah yeah

7 L1

yes ((picks up sieve, moves towards other side of kitchen counter with sieve in hand))

8 EK

no

Lines 9 and 10 show the learners orient to the other-initiated of repair with an explicit acknowledgement demonstrating their active listenership. A series of relevant next actions, represented in lines 11 to 14, demonstrate how the duo accept this repair initiation and respond accordingly: L2 attempts to resolve the problem in the next turn, seeking to immediately ‘self-repair’ and L1 chooses to ‘ask for help’, demonstrating a preference for more information in order to conduct a ‘self-repair’-something L1 indicates to her partner explicitly in line 13 as shown in Fig. 4: Table 3.
Fig. 4

“I asked for help”

Table 3

Lines 9 to 14

9 L2

no

10 L1

no?

11 L2

((moves towards the other side of the kitchen counter)) do you think it is this maybe

12 EK

do you need any help?

13 L1

((selects help via GUI)) I asked for help ((pointing at the touchscreen))

14 L2

oh okay ((moves towards the GUI))

Lines 15 to 22 show the shared orientation to the ‘other-repair initiation’ which has served a solution to their shared lack of knowledge of item, pastry brush. This is shown in the acknowledgement by L2 and both learners’ full and partial repetition of the trouble source. The sequence is extended by L1 (lines 19 to 22) who seeks further clarification about the item and this time self-initiates other repair by L2: “what is pastry”. This demonstrates L1’s specific trouble with item which is the unknown word ‘pastry’. L2 attempts to repair this trouble by explaining the meaning of the word pastry linking it to food items (pie and bread). In this extended sequence, the kitchen is omnipresent during the interaction between the learners, seen in the positive feedback provided in line 21: Table 4.
Table 4

Lines 15 to 22

15 EK

pastry brush ((help 1 pre-task))

16 L2

pastry brush

17 L1

pastry

18 L2

oh: ((looking closely at pastry brush))

19 L1

what does pastry mean?

20 L2

pastry mean like the pie pie or like the bread

21 EK

((plays positive feedback sound))

22 L1

oh:

This episode shows how the environment and more specifically the EK, enters in and out of focus at relevant and timely points in the interaction between the learners. The kitchen is oriented to as an interactant and part of the organization of the talk by the learners who initiates the occurrence of repair based on their actions and aids in the resolution to interactional trouble made relevant by them. The notion that interactions with the kitchen are based around co-ordinated but different interactional troubles for each pair is also evident. Both learners orient to the occurrence of trouble concerning the original instruction, with L1 demonstrating an explicit problem with the word ‘pastry’. These troubles are resolved by the learners through mutually agreed actions and collaborations with the kitchen and each other.

Finnish: the case of the fish soup in the during-task

The interaction opens with an audio prompt from the Finnish Kitchen which communicates a next step: Huuhtele perunat ja porkkanat (rinse the potatoes and the carrots). In line 2, we see the initiation of a repair sequence, where L2 indicates a trouble source in the prompt given by the kitchen. The initial nature of the trouble is not clear until the following turn, during which L1 offers a candidate repair which makes relevant the trouble as linked to the verb huuhtele (rinse), which she identifies as ‘stir’. In line 4, this is the subject of a further repair initiation by L2 who repeats L1’s repair. L1 then offers a further candidate repair, this time offering an additional gesture and reformulation (as shown in Fig. 5) of what she orients to as the repair of the original trouble source Table 5.
Fig. 5

‘sekoittaa?’

Table 5

Lines 1 to 5

1 EK

huuhtele perunat ja porkkanat rinse the potatoes and the carrots

2 L2

Mitä- what

3 L1

huuhtele ämm, ((swirls hand, takes sieve)) rinse ehm

4 L2

huuhtele rinse

5 L1

amm, ((swirls hand)) om, (.) sekoittaa? ehm to stir

In line 6, L2 displays L1’s further attempts to repair the trouble, appearing at this point to accept the solution that the verb huuhtele means to stir, she takes the spoon and stirs the carrots and potatoes in the sieve. L1’s confirmation in line 7 demonstrate the pair’s mutual orientation to finding a solution: Table 6.
Table 6

Lines 6 to 7

6 L2

Aa. (.) se tämä. ((takes spoon and stirs)) oh it this

7 L1

juu. yes

The solution, however, is short-lived. In line 8, L2 appears to re-initiate the repair in a partial turn and in line 9, the kitchen indicates that help is available. The prompt demonstrates a 5th repair initiation sequence where help availability is activated by the kitchen as the system has detected non-activity on the relevant object assigned to that step (the tap). Both L1 and L2 orient to the prompt as an ‘other repair initiation’ linked to their ongoing activity (and therefore indirectly to their linguistic understanding of the action ‘huuhtele’), then L2 accepts the initiation of repair by activating (shaking) the help tool. The availability prompt from the kitchen is designed to be only a partial initiation of repair, offering a choice over the initiation of a full repair. In this case, the learners choose the full repair initiation, leading to the activation of Help 1: Table 7.
Table 7

Lines 8 to 12

8 L2

Heh eh miksi sek- heh ehm why sti-

9 EK

//help available// ((L1 and L2 freeze))

10 L1

e-i? he no

11 L2

ei? hehh ((moves HT)) no?

12 EK

<Huuhtele perunat (.) ja (.) porkkanat. > ((help 1 during task)) rinse the potatoes and the carrots

Lines 13 and 14 demonstrate that despite the help prompt, the pair are unable to repair the trouble. In line 15, the kitchen indicates again that help is available (the kitchen detects nonactivity). In line 16, L1 uses a further partial ‘other repair initiation’ which makes relevant that something further is required to enable a solution. L1 then accepts a full repair initiation once again, leading to the activation of Help 2 (as shown in Fig. 6). Then the picture of the action is shown and the duo laugh. Subsequently, repair is carried out: the carrots and potatoes are rinsed Table 8.
Fig. 6

Help 2

Table 8

Lines 13 to 19

13 L2

Heh ((L2 moves HT, looks at GUI))

14 L1

Hehh °me haluamme kuunnella- kuvasta.° ((also looks at GUI)) heh we want hear– a picture

15 EK

//help available//

16 L1

no? (.) uudelleen? ((moves HT)) well? again?

17 EK

((help 2))

18 L2

[aaa heh heh

19 L1

[aaa he

Demonstrating language learning though design for attentional manipulation

Using CA, the previous section demonstrated the moment to moment development of interactions in two different cooking sessions where progress is halted due to unknown words ‘huuhtele’ (’rinse’) and ‘pastry brush’. This section discusses the CA-based findings to evaluate the balance we sought to establish in the interaction design between promoting learning through “performance in communicative situations” (case 4) and “performance based on implicit learning” (case 3) (MacLaughlin et al. 1983).

First, in terms of promoting learning through performance in communicative situations, the excerpts clearly show the power of the TBLL framework in facilitating the immersion of the learners’ in a real life activity whilst at the same time; they have great motivation to find out what the unknown words are. In the English kitchen, trouble was solved in the process of working out the meaning of the word ‘pastry brush’ and the learners discussed the nature of pastry and what it is used for. In the Finnish kitchen, the learners demonstrated the goal-oriented nature of the task and the need to complete the current step (through solving the trouble) in order to move on to the next stage. These communicative acts show the situated use of spontaneous and authentic language facilitated in the environment. As the presence of the system moves in and out of focus the EK becomes both collaborator and a smart tutor.

In terms of promoting performance based on implicit learning, the repair practices showed learners’ attention to language. The interaction design, and more specifically its attentional manipulation which controls the flow of information given to the learner, afforded mutual negotiation about unknown words: Accessing the help was a co-constructed experience involving the kitchens and learners in a shared consideration of what needed to be done to resolve the problems.

In both kitchens, the learners demonstrated that they were well aware of what kind of affordances they would need. In the Finnish kitchen, Help 1 (slow audio prompt) was not sufficient. When confronted with a linguistic problem (‘huuhtele’ rinse), neither the learners nor the EK would let the problem pass. Instead, the learners managed the resources available to them in the interaction space, which included the help available and each other.

The excerpts showed how the system supported multimodal interaction between the learners as well as giving timely and appropriate prompts (two different kinds of help which were audio and visual) to maximize attentional manipulation in resolving language problems. Using the affordances of the context-awareness, the intervention from the EK came at the right moment to scaffold both learners to restructure their language knowledge and thus had a specific learning purpose beyond merely coordinating the interaction. By becoming the focus of attention within a controlled process, the form was made salient, noticed by the learners, thus becoming more likely to be processed and picked up by the working memory (Schmidt, 1990).

Can a kitchen teach languages?

Can a kitchen teach languages? Our answer is yes, but in order for such smart learning environments to do so, our system design and evaluation methods have shown that there needs to be closer focus on the relationship between learning theory, context-aware technology as well as what happens in the ‘interaction space’ when these systems are used. There should be more balance between technologically-driven practice and theory driven research approaches.

We presented the European Kitchen and explicated the specific focus on decision making underpinning the interaction design for the environment. Central to that description was showing the conjoining of Human Computer Interaction practice, TBLL and IP theory. Using CA, we considered the moment to moment analysis of interactions in the English and Finnish kitchens, part of a larger corpus data used in our qualitative evaluation. Our discussion of the CA showed how we have evaluated the balance between real-world communicative learner performance in the environment and the negotiation of attentional manipulation for implicit learning performance.

We consider our work on the EK to be at an interim point in terms of its development cycle. We have therefore now established a much stronger theoretical basis for learning in the environment, enhanced by our micro-level evaluation work using the explanatory power of CA. In future iterations of the kitchen, we are adopting an increasingly mixed methods approach to complement our CA which includes a language testing, stimulated recall and log data from the system. Our next steps are to take our design and evaluation to the next level: to probe learning in the EK further through a focus on ‘motivation to learn’ as a dynamic process in the environment, where we seek, of course, to build on our existing interdisciplinary model.

Consent

Written informed consent was obtained from the participants for the publication of this report and any accompanying images.

Declarations

Acknowledgements

The European Kitchen is part of an EU project funded with the support of the Lifelong Learning Programme (Key Activity 2) of the European Union.

Authors’ Affiliations

(1)
University of Newcastle
(2)
University of Helsinki

References

  1. AN Antle, AF Wise, Getting down to details: Using theories of cognition and learning to inform tangible user interface design. Interacting with Computers 25, 1 (2013)Google Scholar
  2. S Bax, CALL: Past, Present and Future. System. An Int J Educ Technol App Ling 31, 1 (2003)Google Scholar
  3. MA Bowles, L2 glossing: To CALL or not to CALL. Hispania 87, 3 (2004)View ArticleGoogle Scholar
  4. A Chambers, S Bax, Making CALL work: Towards normalisation System. An Int J Educ Technol App Ling 34, 4 (2006)Google Scholar
  5. CA Chapelle, English Language Learning and Technology (John Benjamins Publishing Company, Philadelphia, 2003)View ArticleGoogle Scholar
  6. N Chomsky, Language and Mind (Hartcourt Brace Jovanovich, New York, 1968)View ArticleGoogle Scholar
  7. D Dearman, K Truong, CHI 2012, in SIGCHI Conference on Human Factors in Computing Systems. Evaluating the implicit acquisition of second language vocabulary using a live wallpaper (ACM, New York, 2012), p. 1391. doi:10.1145/2207676.2208598Google Scholar
  8. P Drew, in Rethinking Methods in Psychology, JA Smith, R Harre, LU Langenhove (Sage Publications Ltd, London, 1995), p. 65Google Scholar
  9. S Gass, L Selinker, Second Language Acquisition: An Introductory Course (Routledge, New York, 2008)Google Scholar
  10. E Harris, G Fitzpatrick, Y Rogers, S Price, T Phelps, C Randell, From snark to park: lessons learnt moving pervasive experiences from indoors to outdoors in Proceedings of the fifth conference on Australasian user interface (Darlinghurst, Australia, Australia, 2004), pp. 39–48Google Scholar
  11. CJ Hooper, A Preston, M Balaam, P Seedhouse, D Jackson, C Pham, C Ladha, K Ladha, T Plotz, P Olivier, UBICOMP 2012, in Conference on Ubiquitous Computing. The French kitchen: task-based learning in an instrumented kitchen (ACM, New York, 2012), p. 193. doi:10.1145/2370216.2370246Google Scholar
  12. JH Hulstijn, Theoretical and empirical issues in the study of implicit and explicit secondlanguage learning. Studies Sec Language Acquisition 27(129–140), 129–140 (2005). doi:10.1017/S0272263105050084Google Scholar
  13. E Hutchins, in Understanding Practice: Perspectives on Activity and Context, ed. by S Chaiklin, J Lave (Cambridge University Press, Cambridge, 1993), p. 35Google Scholar
  14. G Hwang, Definition, framework and research issues of smart learning environments – a context-aware ubiquitous learning perspective. Smart Learn Environ 1, 4 (2014)View ArticleGoogle Scholar
  15. M Kranz, A Schmidt, A Maldonado, RB Rusu, M Beetz, B Hörnler, G Rigoll in TEI '07. 1st International Conference on Tangible and Embedded Interaction, Louisiana, February 2007. Context aware kitchen utilities, (ACM, New York, 2007), p. 213. doi:10.1145/1226969.1227013Google Scholar
  16. S Lee, YY Doh, in CHI 2013. SIGCHI Conference on Human Factors in Computing Systems, in iSpy: RFID-driven language learning toy integrating living environment (ACM Press, New York, 2013), p. 697. doi:10.1145/2468356.2468479Google Scholar
  17. C Lueg, CHINZ 2001, in Symposium on Computer Human Interaction, place, month, 2001. Pitfalls and perspectives in context-awareness, ed. by E Kemp, C Phillips, J Yanes (ACM Press, New York, 2001), p. 37. doi:10.1145/2331812.2331820Google Scholar
  18. P Luff, N Gilbert, D Frolich, Computers and Conversation (Academic Press, New York, 1990)Google Scholar
  19. B Maclaughlin, T Rossman, B McLeod, Second Language Learning: An Information Processing Perspective. Lang Learn 33, 135–158 (1983). doi:10.1111/j.1467-1770.1983.tb00532.xView ArticleGoogle Scholar
  20. R Mitchell, F Myles, E Marsden, Second Language Learning Theories (Routledge, New York/London, 2013)Google Scholar
  21. MA Mohsen, M Balakumar, A review of multimedia glosses and their effects on L2 vocabulary acquisition in CALL literature. ReCALL 23(2), 135–159 (2011)View ArticleGoogle Scholar
  22. F Myles, Second language acquisition (SLA) research: its significance for learning and teaching issues, in Subject Centre for Languages, Linguistics and Area Studies Guide to Good Practice, 2014. https://www.llas.ac.uk/resources/gpg/421. Retrieved on December 1Google Scholar
  23. MA Norman, PJ Thomas, Informing HCI design through conversation analysis. Int J Man Mach Stud 35(2), 235–250 (1991)View ArticleGoogle Scholar
  24. H Ogata, C Yin, M El-Bishouty, Y Yano, Computer supported ubiquitous learning environment for vocabulary learning. Int J Learn Techno 5(1), 5 (2004)View ArticleGoogle Scholar
  25. H Ogata, M Miyata, H Bin, Y Yano, JAMIOLAS2: supporting Japanese mimetic words and onomatopoeia learning with wireless sensor networks for overseas students. Int J Mobile Learn Org 4(4), 333–345 (2010)View ArticleGoogle Scholar
  26. P Olivier, X Guangyou, A Monk, J Hoey, The 2nd International Conference on PErvasive Technologies Related to Assistive Environments, in Ambient Kitchen: Designing Situated Services using a High Fidelity Prototyping Environment (ACM Press, New York, 2009), p. 1. doi:10.1145/1579114.1579161Google Scholar
  27. MQ Patton, in Qualitative inquiry in evaluation: From theory to practice, ed. by L Goodyear, J Jewiss, J Usinger, E Barela (JosseyBass, SanFrancisco, 2014), p. 25Google Scholar
  28. MQ Patton, Qualitative Research and Evaluation methods, 4th edn. (Thousand Oaks; London (Sage Publications, New Delhi, 2015)Google Scholar
  29. R Piri, Teaching and learning less widely spoken languages in other countries (Council of Europe, Strasbourg, 2002)Google Scholar
  30. NS Prabhu, Second language pedagogy (Oxford University Press, Oxford, 1987)Google Scholar
  31. S Price, TP Falcão, Where the attention is: Discovery learning in novel tangible environments. Interact Computers 23(5), 499–512 (2011)View ArticleGoogle Scholar
  32. S Price, C Jewitt, TEI 2013. 7th International Conference on Tangible, Embedded and Embodied Interaction. A multimodal approach to examining 'embodiment' in tangible learning environments (ACM Press, New York, 2013a), p. 43. doi:10.1145/2460625.2460632Google Scholar
  33. S Price, C Jewitt, Interview approaches to researching embodiment, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13) (ACM Press, New York, 2013b), pp. 2907–2910. doi:10.1145/2470654.2481402View ArticleGoogle Scholar
  34. S Price, Y Rogers, M Scaife, D Stanton, H Neale, Using ‘tangibles’ to promote novel forms of playful learning. Interact Computers 15(2), 169–185 (2003)View ArticleGoogle Scholar
  35. Y Rogers, S Price, G Fitzpatrick, R Fleck, E Harris, H Smith, C Randell, H Muller, C O’Malley, D Stanton, M Thompson, M Weal, Ambient Wood: Designing New Forms of Digital Augmentation for Learning Outdoors, in Proceedings of 2004 Interaction Design for Children (IDC 2004) (ACM Press, New York, 2004), pp. 3–10Google Scholar
  36. Y Rogers, S Price, C Randell, DS Fraser, M Weal, G Fitzpatrick, Ubi-learning integrating indoor and outdoor learning experiences. Commun ACM 48(1), 55–59 (2005)View ArticleGoogle Scholar
  37. Y Rogers, HCI Theory: Classical, Modern and Contemporary (Morgan Claypool, 2012) doi:10.2200/S00418ED1V01Y201205HCI014Google Scholar
  38. H Sacks, EA Schegloff, G Jefferson, A Simplest, Systematics for the Organization of Turn-Taking for Conversation. Language 50(4), 696–735 (1974)View ArticleGoogle Scholar
  39. E Schegloff, G Jefferson, H Sacks, The preference for self-correction in the organisation of repair in conversation. Language 53, 361–382 (1977)View ArticleGoogle Scholar
  40. RW Schmidt, The role of consciousness in second language learning. Applied Linguistics 11(2), 17–46 (1990)View ArticleGoogle Scholar
  41. P Seedhouse, in Conceptualising ‘Learning’ in Applied Linguistics, ed. by P Seedhouse, S Walsh, J C’ (Palgrave Macmillan, Basingstoke, 2010), p. 240View ArticleGoogle Scholar
  42. P Skehan, A cognitive approach to language learning (Oxford University Press, Oxford, 1998)Google Scholar
  43. P Skehan, Task-based instruction. Language Teach 36, 1–14 (2003)View ArticleGoogle Scholar
  44. P Ten Have, Methodological issues in conversation analysis. Bull Am Meteorol Soc 27(June), 23–51 (1990)Google Scholar
  45. P Verdines, Conversation Analysis: Design and Evaluation of Computer-Supported Cooperative Learning Environments (AV Akademikerverlag, 2012Google Scholar
  46. L White, Universal Grammar and Second Language Acquisition (John Benjamins, Amsterdam, 1989)View ArticleGoogle Scholar
  47. A Woodruff, MH Szymanski, RE Grinter, PM Aoki, DIS 2002. 4th conference on Designing interactive systems: processes, practices, methods, and techniques, in Practical strategies for integrating a conversation analyst in an iterative design process (ACM Press, New York, 2002), p. 255. doi:10.1145/778712.778748Google Scholar

Copyright

© Preston et al. 2015

Advertisement