FeedbackGive The subject shows they have perceived the message and are willing to maintain contact and go on in the communication. In MUMIN 3.3, the two subtypes CPU and CP were distinguished. FeedbackElicit The subject elicits willingness to maintain contact and go on in the communication, or signs that the interlocutor has perceived the message. In MUMIN 3.3, the two subtypes CPU and CP were distinguished. Understand The subject gives or elicits signs that the message is being understood. NonUnderstand The subject gives or elicits signs that the message is not being understood. Accept The subject gives or elicits signs of agreement. NonAccept The subject gives or elicits signs of lack of agreement. Happy Sad Surprised Disgusted Angry Frightened Certain Uncertain Interested Uninterested Disappointed Satisfied Other TurnTake The speaker takes a turn that wasn't offered, possibly by interrupting. TurnAccept The speaker accepts a turn that is being offered. TurnYield The speaker releases the turn under pressure. TurnElicit The speaker offers the turn to the interlocutor. TurnComplete The speaker completes the turn as well as the conversation. The turn is not passed on. TurnHold The speaker wishes to keep the turn (usually done by rotating the head and the gaze away from the listener). SeqOpen Opening uence indicates that a new speech sequence is starting, e.g. a gesture occurring together with the phrase "by the way...". SeqResume Resume sequence indicates that a preceding speech sequence is resumed after an interruption or a false start. SeqContinue Continue sequence indicates that the current speech sequence is going on, e.g. a gesture occurring together with enumerative phrases such as "the first... the second... the third...". SeqClose Closing sequence indicates that the current speech sequence is closed, e.g. a gesture occurring together with phrases such as "that's it, that's all". InfoStructure IndexDeictic Indexical gestures express a relation of cause-effect between the sign (the gesture) and its meaning. In particular, Indexical Deictics locate aspects of the discourse in the physical space (e.g. by pointing). They can also be used to index the addressee. IndexNon-deictic Indexical gestures express a relation of cause-effect between the sign (the gesture) and its meaning. In Indexical Non-deictic the indexical relation is between the gesture and the effect it establishes. Batonic or beat gestures fall into this category. Iconic Iconic gestures express a semantic feature by similarity or homomorphism. Examples are gestures that express size (length, height, etc.) of an object mentioned in the discourse. Included in this category are also gestures that are sometimes called metaphoric. Symbolic Symbolic gestures (emblems) are gestures in which the relation between form and content is based on social convention (e.g. the okay gesture). They are culture-specific. IconicandIndexNon-deictic Iconic and IndexNon-deictic. New type wrt. MUMIN 3.3. SymbolicandIndexNon-deictic Simbolic and IndexNon-deictic. New type wrt. MUMIN 3.3. Smile The facial expression shows pleasure, favour, amusement, or sometimes derision and scorn. Smile is characterized by an upturning of the corners of the mouth and usually accompanied by a brightening of the face and eyes. Laughter The facial expression shows merriment, amusement, or derision, or nervousness and it is accompanied by an audible vocal expulsion of air from the lungs that can range from a loud burst of sound to a series of chuckles. Scowl An angry expression, like a frown. The term "frown" is reserved for the attribute Eyebrows. FaceOther To be used if none of the other values are appropriate. A comment *must* be added. Frown The eyebrows contract and move towards the nose. Raise The eyebrows are lifted. BrowsOther To be used if none of the other values are appropriate. A comment *must* be added. X-open Exaggerated Opening: the eyes are wide open as in the case of surprise. CloseBoth The eyes are both closed, e.g. to underline focusing, but not in case of a biological blinking. CloseOne One eye winks, that is opens and closes quickly. CloseRepeat Both eyes wink, that is open and close quickly. EyesOther To be used if none of the other values are appropriate. A comment *must* be added. GazeForward The subject is looking forward wrt. own body. GazeBackward The subject is looking backward wrt. own body (implies turning the head and naybe part of the torso). GazeUp The subject is looking up wrt. own body. GazeDown The subject is lookng down wrt. own body (the head may be bent or not). GazeSide The subject is looking to the side of own body. GazeDirectionOther To be used if none of the other values are appropriate, for instance for complex movements. A comment *must* be added. ToInterlocutor The subject is looking at the interlocutor. AwayFromInterlocutor The subject is looking at the interlocutor. OpenMouth The mouth is open or opens as in the case of surprise (that is not just to talk). CloseMouth The mouth is close (relevant in combination with other values, e.g. lips retracted). CornersUp Corners up as in a smile. CornersDown Corners down as in a scowl, sulk or sad expression. Protruded The lips are rounded and protruded. Retracted The lips are sucked-in, retracted in the mouth. LipsOther To be used if none of the other values are appropriate. A comment *must* be added. Nod A head movement down-up. Jerk A quick head movement backwards up. HeadBackward A movement of the head backward (up), this can either be a movement of the head only or can be movement of the whole trunk. This movement occurs often as a turn accepting signal. HeadForward A movement of the head forward (down), this can either be a movement of the head only or can be a movement of the whole trunk. This movement occurs often as a turn elicit signal. Tilt A movement of the head leaning on one side. SideTurn A rotation of the head towards one side. Waggle A movement of the head back and forth, side to side, like a mixture of shake and move backward or forward. Usually produced to show uncertainty, doubtfulness. HeadOther To be used if none of the other values are appropriate. A comment *must* be added. Single The movement is not repeated. Repeated The movement is repeated. BothHands Both hands are involved SingleHand Either right or left hand are involved alone PalmOpen The palm is open. PalmClosed The palm is closed. PalmOther To be used if none of the other values are appropriate. A comment *must* be added. PalmUp The palm is facing upwards. PalmDown The palm is facing down. PalmSide The palm is facing sideways. PalmPosOther To be used if none of the other values are appropriate. A comment *must* be added. IndexExtended The index is extended while the rest of the fingers are not. ThumbExtended The thumb is extended while the rest of the fingers are not. AllFingersExtended All fingers are extended. FingersOther To be used if none of the other values are appropriate. A comment *must* be added. Centre The centre refers to a region roughly corresponding to the subject's torso. See MacNeill, D. (1992). Periphery The periphery refers to a region outside of the subject's torso, that above neck and shoulders, to the side of the torso or beneath the stomach. This means that the arm is somewhat extended. See MacNeill, D. (1992). AmplitudeOther AmplitudeOther: To be used if none of the other values are appropriate. A comment *must* be added. RightHandForward RightHandBackward RightHandSide RightHandUp RightHandDown RightHandComplex RightHandOther LeftHandForward LeftHandBackward LeftHandSide LeftHandUp LeftHandDown LeftHandComplex LeftHandOther Single Repeated BodyForward The subject is leaning forward. BodyBackward The subject is leaning backward. BodyUp The subject is arching or lifting the torso up. BodyDown The subject is bending the torso down. BodySide The subject is leaning to the side. BodyDirectionOther To be used if none of the other values are appropriate, for instance for complex movements. A comment *must* be added.

Anvil implementation of the MUMIN 4 coding scheme, created for the CLARIN annotations, June 2008, Copenhagen and revised in January 2009.

Authors: Costanza Navarretta, Patrizia Paggio

Based on the MUMIN 3.3. coding scheme created by Jens Allwood, Loredana Cerrato, Laila Dybkær, Kristiina Jokinen, Costanza Navarretta and Patrizia Paggio. See original coding scheme.

GENERAL PURPOSE

The purpose of this version of the MUMIN coding scheme is to allow functional annotation of multimodal expressions, as well as the way in which expressions belonging to different modalities are combined. The functions defined are:

Feedback, turn management and sequencing are formalised similarly to what done in MUMIN 3.3. However, the current feature taxonomy is not totally equivalent. In the old scheme, understanding and acceptance were subtypes of FeedbackGive and FeedbackElicit. Now, they are independent features that can be combined with a basic FeedbackGive or FeedbackElicit feature. We no longer distinguish between CPU and CP under Basic Feedback.

In this version of the coding scheme, the modalities taken into consideration in addition to speech are facial displays, hand gestures and body posture.

TRACKS

Tracks are specified to code speech and gestures of two dialogue participants, here called Speaker A and Speaker B. The names can of course be changed, and more tracks may be added.

Speaker A: Speech track Speaker A: Face track Face attributes refer to general face expressions. Eyebrows attributes refer to eyebrows movements. Eyes refer to movements of the eyelids and not to gaze. Gaze direction is defined with respect to the subject's own body axis. Gaze and head movement are related. For instance, if head is up, gaze is also up if the subject does not move the pupils. GazeInterlocutor defines gaze direction with respect to the interlocutor. This feature is new wrt. MUMIN 3.3. GazeInterlocutor and GazeDirection are of course related. Which is the default value of GazeDirection when looking at the interlocutor depends of course on how the dialogue participants are positioned wrt. each other. In case of multi-party conversations, one may want to distinguish among several interlocutors. Mouth openness can be combined with mouth lips values. Mouth lips can be combined with mouth openness values. Head movements are defined similarly to gaze wrt. the subject's own body. Gaze and head movement are related. For instance, if head is up, gaze is also up if the subject does not move the pupils. The values are slightly different from MUMIN 3.3 in that repetition is now a separate feature. It indicates whether the movement is repeated or not. A repeated gesture is a sequence of similar gestures in rapid succession. See also HandRepetition. New feature wrt. MUMIN 3.3. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added. Speaker A: Hand gesture track It refers to whether one or two hands are involved. It refers to whether the palm is open or closed. New wrt. MUMIN 3.3. It refers to the position of the palm in space. New wrt. MUMIN 3.3. It accounts for finger extension. New wrt. MUMIN 3.3. It accounts for how broad the gesture is. New wrt. MUMIN 3.3. Similar values to those used for the direction of gaze, head and body. The feature also indicates the trajectory of the fingers if extended. The value 'complex' is a combination of several of the more basic ones. 'Other' should be used if none of the other values can be applied, and a comment *must* then be added. In MUMIN 3.3. there is a general trajectory feature that does not distinguish between left and righ hands. The same values as for Trajectory Right Hand. The feature also indicates the trajectory of the fingers if extended. The value 'complex' is a combination of several of the more basic ones. 'Other' should be used if none of the other values can be applied, and a comment *must* then be added. In MUMIN 3.3. there is a general trajectory feature that does not distinguish between left and righ hands. It indicates whether the movement is repeated or not. A repeated gesture is a sequence of similar gestures in rapid succession. See also HeadRepetition. New feature wrt. MUMIN 3.3. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added. Speaker A: Body posture track Body Posture is defined with respect to the subject's own body axis. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added. Speaker B: Speech track Speaker B: Face track. All attributes are the same as for Speaker A. Face attributes refer to general face expressions. Eyebrows attributes refer to eyebrows movements. Eyes refer to movements of the eyelids and not to gaze. Gaze direction is defined with respect to the subject's own body axis. Gaze and head movement are related. For instance, if head is up, gaze is also up if the subject does not move the pupils. GazeInterlocutor defines gaze direction with respect to the interlocutor. This feature is new wrt. MUMIN 3.3. GazeInterlocutor and GazeDirection are of course related. Which is the default value of GazeDirection when looking at the interlocutor depends of course on how the dialogue participants are positioned wrt. each other. In case of multi-party conversations, one may want to distinguish among several interlocutors. Mouth openness can be combined with mouth lips values. Mouth lips can be combined with mouth openness values. Head movements are defined similarly to gaze wrt. the subject's own body. Gaze and head movement are related. For instance, if head is up, gaze is also up if the subject does not move the pupils. The values are slightly different from MUMIN 3.3 in that repetition is now a separate feature. It indicates whether the movement is repeated or not. A repeated gesture is a sequence of similar gestures in rapid succession. See also HandRepetition. New feature wrt. MUMIN 3.3. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added. Speaker B: Hand gesture track. All attributes are the same as for Speaker A. It refers to whether one or two hands are involved. It refers to whether the palm is open or closed. New wrt. MUMIN 3.3. It refers to the position of the palm in space. New wrt. MUMIN 3.3. It accounts for finger extension. New wrt. MUMIN 3.3. It accounts for how broad the gesture is. New wrt. MUMIN 3.3. Similar values to those used for the direction of gaze, head and body. The feature also indicates the trajectory of the fingers if extended. The value 'complex' is a combination of several of the more basic ones. 'Other' should be used if none of the other values can be applied, and a comment *must* then be added. In MUMIN 3.3. there is a general trajectory feature that does not distinguish between left and righ hands. The same values as for Trajectory Right Hand. The feature also indicates the trajectory of the fingers if extended. The value 'complex' is a combination of several of the more basic ones. 'Other' should be used if none of the other values can be applied, and a comment *must* then be added. In MUMIN 3.3. there is a general trajectory feature that does not distinguish between left and righ hands. It indicates whether the movement is repeated or not. A repeated gesture is a sequence of similar gestures in rapid succession. See also HeadRepetition. New feature wrt. MUMIN 3.3. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added. Speaker B: Body posture track. All attributes are the same as for Speaker A. Body Posture is defined with respect to the subject's own body axis. The basic feedback feature is used to distinguish between feedback giving and eliciting. Feedback understanding is used to specify whether basic feedback is accompanied by signs of understanding or non-understanding. Basic feedback can also be accompanied by signs of agreeing or not agreeing. Emotions and attitudes can co-occur with any of the communicative features. They include i.a. Ekman's six basic emotions: happiness, surprise, anger, sadness, fear, disgust/contempt. Turn features refer to the way in which one of the dialogue participants gains, ends or holds the turn, and reflect whether this is done in agreement or not. In a more coarse-grained scheme, one could do with the three values TurnGain, TurnEnd and TurnHold (see MUMIN 3.3). Sequencing is concerned with speech act organisation and structuring into sub-dialogues. It should not be confused with turn-management, that models turn shifts among dialogue participants. The gesture has a function relating to information structure, i.e. the organisation of the information within a sentence or utterance. It could be underlining the topic, the focus or creating a contrast. Based on Peirce's theory, three types are defined: indexical, iconic and symbolic. Combinations are also allowed, which is new wrt. MUMIN 3.3. The gesture adds information to the corresponding speech segment, e.g. by making a concept more specific. See Poggi and Magno Caldognetto (1996). The feature was subsumed under Compatible in MUMIN 3.3. The gesture has the same meaning as the corresponding speech segment. See Poggi and Magno Caldognetto (1996). It was subsumed under Compatible in MUMIN 3.3. The gesture occurs in absence of words. See Poggi and Magno Caldognetto (1996). The gesture contradicts the corresponding speech segment e.g. to denote sarcasm or irony. See Poggi and Magno Caldognetto (1996). It corresponds to Incompatible in MUMIN 3.3. To be used if none of the other values are appropriate. A comment *must* be added.