To Gesture or Not?

# To Gesture or Not? An Analysis of Terminology in NIME Proceedings 2001–2013

*Alexander Refsum Jensenius*    
*fourMs group, Department of Musicology, University of Oslo*

???

C = two screens    
p = presentation mode    
cmd+f = fullscreen   
    
Good afternoon everyone, my name is ARJ...
    
This will be a presentation on terminology, a meta-analysis, if you will

---

# Gesture?

???

My starting point is the term "gesture", which I have been struggling with for some time.

It is not that gesture in itself is such a bad term.

Rather: it seems like we in the NIME community are using it somewhat differently from the rest of the world.

That may not be a problem in itself, if we understand each other, BUT:

1. we may be confused within NIME

2. we may get into trouble with people outside our community

Gesture can be used for many things...

---

---

---

---

---

---

---

---

---
    
# ? 
--

1. How much is ‘gesture’ used at NIME?

2. How much is ‘gesture’ used in related fields?

3. How is ‘gesture’ used, and with what meaning(s)?

???
        Three main points
    
---

# Gesture definitions

---

# Oxford dictionary

**Gesture:**

*"a movement of part of the body,*

*especially a hand or the head,*

*to express an idea or meaning"*

???

Similar to MacMillan and Dictionary

---

# Oxford dictionary

**Gesture:**

*"a **movement** of part of the body,*

*especially a **hand** or the **head**,*

*to **express** an **idea** or **meaning**"*

???

Similar to MacMillan and Dictionary

---

# MacMillan dictionary

**Gesture:**

*"a movement that communicates a feeling or instruction [...]:*

- *hand movement that you use to control something such as a smartphone or tablet [...]*

- *the use of movement to communicate, especially in dance"*

---

# MacMillan dictionary

**Gesture:**

*"a **movement** that communicates a **feeling** or **instruction** [...]:*

- *hand movement that you use to **control something** such as a smartphone or tablet [...]*

- *the use of **movement to communicate**, especially in dance"*

---

# Research definitions of gesture

*(Cadoz & Wanderley, 2000) – (Jensenius et al., 2010):*

**Communication**:    
convey meaning in social interaction (linguistics, psychology)

**Control**:    
interact with a computer-based system (HCI, computer music)

**Metaphor**:    
project movement and sound (and other phenomena) to cultural topics (cognitive science, psychology, musicology)

???

All in all, in the sub-communities around NIME, gesture is used very differently.

Now...

---

# Gesture@NIME?

???
     Using the NIME proceedings as source material

---

# Method
--

1. Download PDFs from nime.org

2. Consistency check in Acrobat

3. Replace/remove PDF files

4. Define search terms:    
‘music’,    
‘gesture’, ‘action’, ‘motion’, ‘movement’,    
‘emotion’, ‘expressive’,    
‘motion capture’, ‘accelerometer’, ‘wii’, ‘kinect’, ‘leap motion’

5. OSX shell script: ```mdfind -count```

6. Manual check of some files

7. Calculate basic statistics

???

Result: 1 108 files!

---

# NIME

| % | Total | music | gest- ure | act- ion | mot- ion | move- ment | emot- ion | expre- ssive | motion capture | acceler- ometer | wii | kinect | leap motion | 
| :-:| -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | 
| 2001 | 14 | 100 | 64 | 57 | 50 | 64 | 14 | 57 | 29 | 29 | 0 | 0 | 0 |
| 2002 | 48 | 100 | 65 | 52 | 58 | 65 | 17 | 60 | 23 | 15 | 0 | 0 | 0 |
| 2003 | 48 | 100 | 71 | 35 | 40 | 48 | 19 | 50 | 15 | 17 | 0 | 0 | 0 |
| 2004 | 54 | 100 | 56 | 37 | 39 | 54 | 22 | 43 | 22 | 20 | 0 | 0 | 0 |
| 2005 | 75 | 100 | 63 | 48 | 45 | 56 | 23 | 48 | 23 | 24 | 0 | 0 | 0 |
| 2006 | 81 | 100 | 64 | 41 | 36 | 52 | 7 | 41 | 23 | 16 | 0 | 0 | 0 |
| 2007 | 103 | 100 | 55 | 38 | 40 | 57 | 20 | 50 | 18 | 17 | 4 | 0 | 0 |
| 2008 | 87 | 100 | 60 | 52 | 52 | 59 | 14 | 45 | 25 | 22 | 16 | 0 | 0 |
| 2009 | 110 | 90 | 54 | 36 | 37 | 51 | 12 | 35 | 13 | 26 | 12 | 0 | 0 |
| 2010 | 111 | 100 | 66 | 50 | 44 | 55 | 23 | 45 | 26 | 26 | 14 | 0 | 0 |
| 2011 | 130 | 100 | 67 | 60 | 45 | 59 | 16 | 51 | 25 | 26 | 13 | 5 | 0 |
| 2012 | 129 | 99 | 63 | 61 | 44 | 57 | 16 | 53 | 26 | 33 | 10 | 12 | 0 |
| 2013 | 118 | 99 | 65 | 51 | 49 | 64 | 25 | 56 | 30 | 26 | 11 | 22 | 1 |
| ||||||||||||| | 
| **Mean** | 85 | **99** | **62** | 48 | 45 | 57 | 18 | 49 | 23 | 23 | 6 | 3 | 0 |
| **Stdev** | 36 | 3 | 5 | 9 | 6 | 5 | 5 | 7 | 5 | 6 | 6 | 7 | 0 |

???

Music: almost 100%

Gesture: 62% -- remarkably stable!

Action, motion, movement: around 50%

Emotion: low

Expressive: high (removed expression)

Mocap + accelerometer: 20% (fairly low)

Technologies as they are introduced to the market

---

# SMC

| % | Total | music | gest- ure | act- ion | mot- ion | move- ment | emot- ion | expre- ssive | motion capture | acceler- ometer | wii | kinect | leap motion | 
| :-:| -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | 
| 2004 | 47 | 100 | 30 | 38 | 15 | 36 | 17 | 40 | 9 | 2 | 0 | 0 | 0 | 
| 2005 | 32 | 100 | 28 | 47 | 28 | 56 | 25 | 31 | 13 | 0 | 0 | 0 | 0 | 
| 2006 | 26 | 104 | 27 | 50 | 27 | 42 | 19 | 31 | 23 | 0 | 0 | 0 | 0 | 
| 2007 | 61 | 100 | 38 | 44 | 41 | 54 | 25 | 41 | 13 | 7 | 2 | 0 | 0 | 
| 2008 | 34 | 100 | 32 | 32 | 26 | 56 | 9 | 24 | 9 | 0 | 0 | 0 | 3 | 
| 2009 | 62 | 100 | 26 | 39 | 26 | 47 | 16 | 35 | 10 | 13 | 3 | 0 | 0 | 
| 2010 | 74 | 99 | 27 | 39 | 34 | 34 | 18 | 36 | 11 | 4 | 1 | 0 | 3 | 
| 2011 | 79 | 99 | 38 | 44 | 35 | 54 | 22 | 41 | 22 | 14 | 5 | 4 | 0 | 
| 2012 | 75 | 99 | 40 | 52 | 31 | 39 | 19 | 23 | 19 | 11 | 4 | 7 | 1 | 
| 2013 | 111 | 100 | 38 | 38 | 33 | 47 | 22 | 30 | 18 | 11 | 5 | 5 | 1 | 
| ||||||||||||| | 
| **Mean** | 60 | 100 | 32 | 42 | 30 | 47 | 19 | 33 | 14 | 6 | 2 | 2 | 1 | 
| **Stdev** | 26 | 1 | 6 | 6 | 7 | 9 | 5 | 7 | 5 | 6 | 2 | 3 | 1 |

???

Same analysis for SMC

Gesture: 32%

action, motion, movement: higher than gesture

emotion, expressive, lower

---

# Comparison

| % 					| Total 		| music | gest- ure | act- ion | mot- ion | move- ment | emo- tion | expres- sive | motion capture | acceler- ometer | wii | kinect |  
| :- 					| -: 			| -: 	| -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | -: | 
| NIME 					| 1,108 			| 99 	| 62 | 48 | 44 | 57 | 18 | 48 | 23 | 24 | 8 | 4 |
| ICMC 					| 3,687 			| 100 	| 17 | 17 | 20 | 24 | 4 | 20 | 2 | 3 | 1 | 0.2 |
| SMC 					| 601 			| 100 	| 34 | 42 | 31 | 46 | 19 | 33 | 15 | 8 | 3 | 2 |
| ACM + Aff. 			| 399,664 		| 4 	| 3 | 22 | 8 | 10 | 2 | 4 | 3 | 1 | 0.4 | 0.4 |
| ACM Guide 			| 2,193,894 		| 2 	| 1 | 11 | 5 | 5 | 1 | 2 | 1 | 0.4 | 0.1 | 0.1 |
| J. bio- mechanics 	| 18,193 		| 0.3 	| 0.2 | 38 | 51 | 43 | 0.1 | 0.03 | 6 | 4 | 0.1 | 0.02 |

???

NIME + SMC = similar analysis

The others: search in databases

ICMC: small percentage, but similar absolute numbers as NIME

---

# Gesture collocation @NIME
--

- "gesture **recognition**" (R1)

- "**sound** gesture" (L1)

- "gesture and **sound**" (R2)

---

# Collocation method
--

1. PDF -> txt (CasualText)

2. Clean up (TextMate)

3. Collocation analysis (CasualConc)

4. Remove common words

---

# Collocation (total)

| Word			| LR total 		
| :- 			| -: 			
| sound			| 402			
| recognition	| 378			
| control		| 243			
| musical		| 214			
| data			| 212			
| mapping		| 210			
| based			| 207			
| music			| 189			
| analysis		| 173			
| system		| 149

---

# Collocation (left)

| Word		 	| L total
| :--- 			| ---: |  
| expressive 	| 115
| musical		| 110
| sound			| 87
| mapping		| 86
| time			| 85
| hand			| 78
| physical		| 68
| control		| 65
| instrumental	| 58
| instrument	| 52

---

# Collocation (right)

| Word			| R total
| :- 			| -: |
| recognition	| 344
| sound			| 315
| control		| 178
| data			| 165
| based			| 158
| music			| 142
| analysis		| 128
| mapping		| 124
| musical		| 104
| system		| 97

---

# Conclusions

NIME proceedings = interesting meta-research material!

Gesture@NIME: 
--

- in 689 papers (62%)

- used 4,211 times (6 times per paper)

- used more widely than at SMC (34%) and ICMC (17%)

- many different meanings: motion, technology, metaphor...

---

# Conclusions

So...

- Define your use of the term

- Use other words when appropriate

---

# www.arj.no