Cognitive Processing

Emoji as Visual Metaphors in Cognitive and Liguistic Processes

Our everyday virtual communication underwent a shift in recent years, when the Unicode Standard introduced Emoji, the set of more than one thousand pictograms, which became a standard in most of our online messaging services. Now Emoji are a substantial part of our virtual communication with more and more words becoming substituted by Emoji. Companies emerge around the Emoji trend such as Emogi.com, which states in its Emoji Report 2015 that about 92% of the online population uses Emoji. Emogi.com uses Machine Learning, NLP and biometric studies to analyse how people communicate with Emoji. All big social networks such as Facebook, Twitter and Instagram are using sentiment analysis on their Emoji usage statistics.

Iconic signs and natural language both can construct complex sentences but pictograms - such as Emoji - can hardly express narration, conversation or argumentation in longer sequences. Emoji show polysemy and only few are assumed to be universally understood. Theoretically Emoji, as much as hieroglyphs, denote and connote ideas visually and those ideas might be primitive semantic building blocks of complex meaning.

Emoji are semiotic objects (Danesi, 2016) as they demonstrate a multitude of linguistic semiotic layers and can be interpreted as signs, metaphors, analogies or symbols. Nevertheless, there is no effective human language solely based on pictograms. Even the Egyptian hieroglyphs are not pictographic in nature, they are a phonetic, sound-based system.

The idea of semantic primes has been introduced in AI by Y. Wilks (Wilks, 1975) and R. Schank (Schank, 1972). In this theory, any complex idea can be expressed as the product of two or more simpler ideas, and the irreducibly atomic ideas are the semantic primes. As such, they can be used to derive any complex sentence in any language (Wierzbicka, 1996). Given this theory, Emoji should prove useful in the expression of complex ideas as sequences of atomic ideas.

Hence, a story telling system should be able to articulate its stories entirely in Emoji, if the Emoji as semantic primes hypothesis has any validity. We choose the twitterbot @MetaphorMagnet, which provides a large set of plot verbs (800+) and we translate them into Emoji sequences using a set of distinctive methods. Analysing the understanding of the translations (with crowdsoucing) allows us to investigate the individual methods. This thesis provides an empirical investigation of the techniques used to compose the translation sequences and a manual on how to apply them. The results suggest with strong signi cance that the Emoji sequences prove to translate comprehensibly. Moreover, the results suggest that Emoji can tell stories and thus allow considering Emoji to be able to combine a set of fundamental ideas. The scheme is a novel approach in parsing and translating Emoji, that could prove highly bene cial given the aforementioned interest in Emoji.