Multimodal Semiosis In Mass Media: Several Remarks On Methodology

Abstract

The paper discusses several methodological challenges and perspectives for the semiotic discourse analysis of media practices considered in its multimodality. Despite the somewhat tautological character of the expression multimodal communication , there is a strong epistemic potential of it. The use of the term multimodal corresponds to the analysis of texts as coherent, both verbal and non-verbal unities. The author argues that the more realistic typology of media modes and semiotic means would base on the differentiation of perception channels, which includes auditory (oral-verbal, prosodic and acoustic-musical) and visual (figural-visual, oculesic, kinesic, proxemic and written-verbal) perception modes. Since they have different semiotic and communicational potential and they function differently in expressing and transmission of meaning structures, it is reasonable to itemize the sorts of interaction of them in discursive practices. The main semiotic interaction processes comprise: (inter)repetition, (inter)addition and convergence of modes and semiotic elements. The other methodological challenge is the search for the tool to evaluate the meaning migrations in media texts. The author distinguishes two processes: intrasemiotic and intersemiotic transduction, depending on the identity or difference of semiotic codes and means used in the meaning migration processes in media texts. The paper discusses also the perspectives of multimodal analysis for the specification of professional media discourses.

Keywords: Multimodalitysemiosiscommunicational modesemiotic codemedia communicationmass media

Introduction

The contemporary cross-disciplinary studies of language and communication have been focusing on multimodal communicational phenomena for already more than 15 years (Kress & van Leeuwen, 2001; Baldry & Thibault, 2005; Kress, 2010; O’Halloran & Smith, 2012; Wildfeuer & Bateman, 2016; Bateman, Wildfeuer, & Hiippala, 2017; Jovanovic & Van Leeuwen, 2018). It is rather commonplace today to consider verbal means not to be the crucial element of mass media semiosis (Sindoni, Wildfeuer, & O'Halloran, 2016; Danesi, 2018; Djonov & Van Leeuwen, 2018; Hiippala, 2018). It means that observing them as central in media communication would hardly contribute to constructing the manifold and realistic view of media discursive processes.

The current media studies tend to analyse the media discourse as the multimodal phenomenon (Kress & van Leeuwen, 2001; Machin & Mayr, 2012). The semiotic construction and translation of meaning structures, or semiosis, enacts both verbal and non-verbal – auditory, visual and proxemic means of communication. Besides, the multimodal nature of the mass media semiosis appears in the wide usage of various semiotic means for translating the message (news, facts, opinions, etc.). Messages can be communicated in the mode of verbal text, infographics, video message or animation, while the audience tends to prefer visual means of communication.

The technological background of media communications as well as the intentions of media communicators to set the story or content in the focus of audience's attention contribute to the intensive and expansive visualization of the data originally created in the verbal mode. In mass media communications, there are now various means to stimulate the involvement and cooperation of addressees in the discursive media practices. They are, for example, inquiries, quizzes, integrated educational content, gamification forms, etc. Media communicators use these means extensively to promote the engagement of audience. The pragmatic dispositions and intentions of such “visual turn” of media communications are obvious today, since they aim at increasing the audience outreach, controlling the audience attention and supporting the customers’ loyalty.

However, the contexts and effects of such processes are much further than marketing efficiency. Though the practices mentioned above are often focusing on rather marketing goals, their prevalence in media communication certainly changes the semiotic aspects of media communications, audience perception dispositions and communicators’ data processing. These impacts and effects correlate to the phenomenon of multimodality .

Kibrik (2010) shows that the term multimodal “bases on the notion of modality accepted in psychology, neurophysiology and information sciences: modality is a type of the external stimulus perceived by one of the human receptors, mainly by sight and hearing” (p. 135).

Problem Statement

There is a widespread claim that any discourse and any communication is multimodal by its nature. As Kibrik (2010) states, “the attempts to divide crucially language from communication, thinking and behavior are of little efficiency, and artificial, and are determined only by the logics of science development but not by the nature of things” (p. 135). Moreover, “there is no principal difference between verbally transmitted information and the one sent prosodically or by gestures. The only approach which has the chance to build the realistic linguistics of future is the one accepting the multimodality of communication” (p. 148).

It is important to discuss two crucial methodological aspects referring to the Kibrik’s statement mentioned above. Researchers in the field of social semiotics (Kress, 2016) and critical discourse analysis (Machin & Mayr, 2012) provide largely the key arguments for this position. The mentioned research fields implement the cross-disciplinary approach to discursive phenomena and semiotic means of communication

This view towards the multimodal analysis of communication and discourse seems to be reasonable. However, if one assumes the communicational equality of all the semiotic means used in the media communication, there would be no need for discussions about the differentiation of effects and impacts of media of different modes (radio, printed media, TV, Internet, etc.). Yet, there is a difference in how they affect audience, while mode types determine obviously this difference. Thus, it is rather important to reveal the differentiation in multimodal semiotic processes occurring in mass media.

Research Questions

The paper aims at discussing two principle methodological questions that appear to be central for the multimodal analysis of media semiosis and media discursive practices:

a) If one assumes that any communicational situation is multimodal by its nature, is there a need for the term ‘multimodal’ in appliance to media communications?

b) What types of semiotic processes can be distinguished in media discursive practices according to the multimodal nature of media?

Purpose of the Study

To achieve the clear view on the methodological basis for multimodal analysis of media texts, we have to:

a) Set the terminological ‘area’ of the notion multimodal in comparison to synonymous and allied notions,

b) Classify and specify various kinds of semiotic changes that occur in multimodal discursive practices.

Research Methods

To achieve the purpose stated above we used:

a) The method of terminological comparative analysis to evaluate the semantic potential and the character of usage of terms multimedial , cross-medial , trans-medial and multimodal ,

b) The typological method to itemize and specify the types of communicational channels engaged into the multimodal media practices, the types of semiotic changes and migration of meaning structures in multimodal communication.

All the findings base on the results of the observation of the most quoted current works in the field of media discourse analysis and social semiotics.

Findings

There are three aspects of practices of intensive semiotic closely related to the ways of informational representation in mass media: multimediality , cross-mediality and trans-mediality .

Multimediality refers to the transmission of the same content with different semiotic means but via the certain channel (Suciu & Mocofan, 2015). Cross-mediality is the way of content transmission or storytelling with the certain semiotic means and via different channels (Hasebrink & Hepp, 2017). Trans-mediality represents the construction of the “big narrative” with different semiotic systems and via different communicational channels (Zeiser, 2015). We appreciate this differentiation in terms of analysis of storytelling technologies, media marketing and communicating with different segments and chronotopes of the audience. However, to define the migration of meaning structures in the process of semiotic transformations, it would be more efficient to focus on the category of multimodality and sematic operations of the “meaning translation”. The latter if often marked as transduction (Kress, 2010).

The notion multimodal indicates the communicational characteristics of use of various modes of perception of information or – communicational channels. Relatively, the authors often prefer to use the synonymous term multichannel (Fedorova & Kibrik, 2018). In this field, at the first glance the usage of terms multimodal ( multichannel ), polycode , hybrid , mixed , creolized , multimedial seem to be polyphonic. However, in most cases these terms are used as context (e.g., terms multimodal , hybrid , creolized and polycode or absolute (e.g., terms multimodal and multimedial ) synonyms (Omelyanenko & Remchukova, 2018).

Despite the variety of term usage, the researchers express solidarity in defining the central specifying features of multimodal texts and discourses (Bateman et al., 2017; Fedorova & Kibrik, 2018; Danesi, 2018). These are semiotic heterogeneity , simultaneous multichannel transmission , complexity of message perception , communicational combinativity . As Egorchenkova (2014) remarks, "the interactive behavior of communicants, based on (…) the simultaneous ‘union’ of heterogenic components, is characterized with the multimodality” (p. 26). Multimodality represents the use of different sensory capacities of recipients to perceive the message. Egorchenkova’s (2014) research results show that multimodal interaction contributes to the development of communicational interaction, while the dominant usage of verbal elements of communication can often lead to the interruption of communication.

We tend to share the view towards the multimodal analysis of communication and discourse. The mentioned research fields implement the cross-disciplinary approach to discursive phenomena and semiotic means of communication.

Yet, it is important to discuss two crucial methodological aspects referring to the Kibrik’s statement quoted above.

First, if one assumes the ontological status of multimodality of the discursive, and wider – communicational, practices, one should face the tautology of the use of the term multimodality . The reasonable question is: what is the use of the term multimodal communication is any communication is multimodal by its nature?

We consider that there is no semantic exaggeration in such terms as multimodal text , multimodal communication and multimodal discourse , since the word multimodal indicates the semiotic complexity, coherence and integrativity of texts that base on different semiotic systems and refer to addressees’ different receptors. When defining the epistemic phenomenon, the use of the term multimodal corresponds to the analysis of coherent, both verbal and non-verbal, texts.

Secondly, the Kibrik’s quotation above suggests the interchange of semiotic means for informational interaction (“there is no principal difference between verbally transmitted information and the one sent prosodically or by gestures”), which makes state the methodological issue of relevance of semiotic means: what are the conditions and limits for coherence (integration) of multimodal ways to use semiotic means in discursive practices? It seems obvious that they interact in many different ways apart from interchanging. Poymanova (1997) distinguishes such ways of interaction of verbal and non-verbal means in mixed texts as repetitive (one semiotic means repeats semantically the other), additive (one means adds semantically to other), emphasizing (one means emphasizes a semantic aspect of the other), oppositive (one means opposites semantically the other), integrative (all means cooperate semantically in the semiosis).

It is worth remarking that the opposition “one means – other means” does not necessarily comprise the juxtaposition of verbal and non-verbal means. We claim that in various discursive practices the valuable volume of information can be transmitted without any verbal elements. As one can observe it in advertising, music or cinema, the main semiotic interactions occur between audial and visual means. We consider the more realistic typology of modes / modalities and semiotic means to base on the differentiation of perception channels. What follows is the attempt to elaborate such typology.

Multimodal discourse is accomplished via a) auditory (hearing or acoustic) and b) visual (viewing) channels. In its turn, the auditory channel includes the oral-verbal , prosodic and acoustic-musical modes, since the visual comprises the figural-visual , oculesic, kinesic, proxemic and written-verbal modes; In terms of intensive development of VR, AR and neuro-technologies, the tacesic mode of media discursive practices also becomes of higher importance.

There can be the following types of interaction between modes and relevant semiotic elements:

(inter)repetition : for instance, the infographic statistics is denotatively identical to the data represented in the written-verbal mode, and the ticker in the commercial is denotatively identical to the oral-verbal text;

(inter)addition: for example, gestures-indicators can be used while the visual demonstration of a product or to emphasize several fragments of the oral speech;

convergence : for instance, color, logotype and visualized product are integrally used in commercials, as well as the photograph, verbal comment and animation may go along at the news website.

Thus, it is rather important for discourse analysts to have the methodological tools for evaluation the discursive potential of semiotic resources that are available to addressers.

It is considerable that not any semiotic tool fits any of the intermodal interaction type mentioned above. For example, communicants can hardly use the prosodic means to reproduce or repeat the information expressed before in the written-verbal mode. Onу would fail in communicating a press release with pauses and sighs, as well as it is almost impossible to transmit the TV series plot with colors and gestures. However, many “semiotically centered” means, such as verbal elements, do not principally go autonomously or separately from other semiotic means. The word is always implemented in extralinguistic parameters, such as visual (type, color) and auditory (volume, tone) ones.

This fact provides the notion of transduction in/of media texts. A text is transductive if there is the potentiality for its semiotic transformation, i.e. recoding in other semiotic system while preserving the original meaning structure – whether denotative as minimum or connotative one as maximum.

Kress (2010) observed in detail the phenomena of semiotic transduction and transmodal semiosis in mass media. According to Kress (2016),

while in linguistics the starting point for thinking or working is language which is understood as speech or writing, “material” resources of language in multimodality come much beyond speech and writing. The word material is used here in the meaning referring to the phenomena available for receptors, for (human) sensorium. (p. 78)

The fact that media communicators use extralinguistic means to communicate the meaning attests to the multimodal “turn” of their profession, determined by the technical progress and material development. However, there is a risk in exaggerating these processes. As it always happened in the history of media systems, the appearance of new communicational means lead to enlarging of representational capacities of journalism, advertising, public relations and other media communicational practices. Thus, the appearance of photography contributed to the growth of objectivity of news printed texts, as well as technological capacities of cinema stimulated the development of documentaries. However, the scale and tension of multimodality are immense nowadays, and its effects are hardly predictable.

Thus, two main semiotic processes linked to the migration of meaning structures in contemporary media texts are intra - and intersemiotic transduction . Following Kress (2010), media researchers should distinguish the processes of transduction , i.e. transmission of the original meaning with other semiotic means, and transformation , i.e. changing the original meaning with the same semiotic means.

We define intrasemiotic transduction as the reproduction of the meaning structure with other means of the same semiotic system, in terms of which the original message was made. The proper examples are communicating a press release in one of journalistic news genres, editorial rewriting of a text when preparing it for publishing or translation of a foreign language journalistic text for a national media. This type of transduction embraces two kinds of semiotic changes that Jacobson marked as intralinguistic and interlinguistic types of translation (Kozin, 2018). In the semiotic perspective, the changes occur in the same semiotic system – the natural verbal language but provided by different semiotic means (PR and journalistic genres, styles and text types).

However, it is not the only mode of semiotic changes. The meaning structures can be transmitted with semiotic means belonging to other codes and modes, e.g. audiovisual, graphical, actional (interactive), etc. Thus, it is rational to take into account the intersemiotic transduction.

We define intersemiotic transduction as the reproduction of the meaning structure with means of the semiotic system that differs from the one of the original messages. Media communicators widely use infographics to communicate an originally printed text, or podcasts to transmit the originally video messages.

The transduction can be both reversible and irreversible . For instance, the content of the infographic text is the example of reversible intersemiotic transduction , since one is able to recover the complex of meanings with the original, verbal, semiotic system. The example of irreversible intersemiotic transduction is the screen version of Conan-Doyle’s books in the series about Sherlock Holmes, since the resources of the original semiotic code are insufficient for reproducing the meaning structure of the original printed fiction text.

Another important methodological issue is the motivation of semiotic changes. The motivation can be voluntary and involuntary . In many technological terms, the very capacity of transductive processes determines them. However, it is often that one can observe voluntary, engaged or even ideologically biased semiotic changes in media discourse.

Conclusion

Thus, there is an actual task traditionally set in social semiotics and critical discourse analysis. i.e. the task of revealing the ideological basis of discursive processes. Social semiotics and critical discourse analysis treat multimodality not only as the immanent ability of texts to express the reality with various semiotic means, but also as the “advanced” way to construct the reality.

As Machin and Mayr (2012) state,

visual communication, as well as language, both shapes and is shaped by the society. MCDA [multimodal critical discourse analysis – E.K.] therefore is not so much interested in the visual semiotic choices in themselves, but also in the way that they play a part in the communication of power relations. (p. 10)

Let us add that this fact surely refers to a wider range of processes than only visual communication, i.e. to all possible modes of communication, and different semiotic resources and codes. Hence, the possible objects of social semiotics and critical discourse analysis can also be, for instance, the prosodical means of domination and subordination in radio communication, proxemic construction of inequality in Instagram, or kinesic means of representation of “us” and “them” on TV, etc.

At last, the perspectives of multimodal analysis of media discourse correlate with the study of discursive practices in different professional fields. The issue of intersemiotic and interdiscursive coherence and intersemiotic transduction is highly important in cross-medial and trans-medial communication. For example, the choice of strategies and techniques of storytelling on different media platforms, the search for appropriate infographic means for representation of complex data and adopting the content to the semiotic expectations of the target audience – all these practices presume the special awareness in intersemiotic multimodal processes. It is rather obvious that these research initiatives are cross-disciplinary and they encircle the whole range of sciences, such as semiotics, linguistics, IT studies, social anthropology, cultural studies and many other research fields (O’Halloran, Tan, Pham, Bateman, & Vande Moere, 2018) contributing to the construction of the full coherent view on the multimodal discursive practices.

Acknowledgments

I thank my colleagues from the faculty of journalism, Belgorod National Research University, who provided insights, discussions and comments, and who encouraged the research.

References

1. Baldry, A., & Thibault, P. J. (2005). Multimodal transcription and text analysis. London: Equinox.
2. Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research and analysis – a problem-oriented introduction. N.Y.: Walter de Gruyter GmbH & Co KG.
3. Danesi, M. (2018). Understanding media semiotics. L., N.Y.: Bloomsbury Academic.
4. Djonov, E., & Van Leeuwen, T. (2018). Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice. Social Semiotics, 28(5), 641-664.
5. Egorchenkova, N. B. (2014). Strategicheskiy potentsial multimodalnoy interaktsii v media-politicheskom diskurse [Strategic potential of multimodal interaction in media political discourse]. Bulletin of Tomsk State University, Volume: Philology, 5(31), 24-37.
6. Fedorova, O. V., & Kibrik, A. A. (2018). Obschee, individualnoye i kontekst v multikanalnoy kommunikatsii [General, individual and context in multimodal communication]. Cognitive Studies of Langiuage, 33, 637-645.
7. Hasebrink, U., & Hepp, A. (2017). How to research cross-media practices? Investigating media repertoires and media ensembles. Convergence, 23(4), 362-377.
8. Hiippala, T. (2018). Structure of multimodal documents: An Empirical Approach. L.: Routledge.
9. Jovanovic, D., & Van Leeuwen, T. (2018). Multimodal dialogue on social media. Social Semiotics, 28(5), 683-699.
10. Kibrik, A. A. (2010). Multimodalnaya lingvistika [Multimodal linguistics]. In Cognitive Studies – IV (pp. 134-152). Moscow: IP RAN.
11. Kozin, A. V. (2018). Consecutive Interpreting: From Language to Communication. In Consecutive Interpreting (pp. 37-66). Cham: Palgrave Macmillan.
12. Kress, G., & van Leeuwen T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.
13. Kress, G. (2010). Multimodality. A social semiotic approach to contemporary communication. London: Routledge.
14. Kress, G. (2016). Sotsialnaya semiotika i vyzovy multimodalnosti [Social semiotics and challenges of multimodality]. Political Science, 3, 77-100.
15. Machin, D., & Mayr, A. (2012). How to Do Critical Discourse Analysis: A Multimodal Introduction. London, N.Y.: Sage.
16. O’Halloran, K. L., Tan, S., Pham, D. S., Bateman, J., & Vande Moere, A. (2018). A digital mixed methods research design: Integrating multimodal analysis with data mining and information visualization for big data analytics. Journal of Mixed Methods Research, 12(1), 11-30.
17. O'Halloran, K., & Smith, B. A. (Eds.). (2012). Multimodal studies: Exploring issues and domains (Vol. 2). London: Routledge.
18. Omelyanenko, E. A., & Remchukova, E. N. (2018). Polikodovyye teksty v aspekte teorii multimodalnosti [Polycode texts in terms of the theory of multimodality]. Communication Studies, 3(17), 66-78.
19. Poymanova, O. V. (1997). Semanticheskoye prostranstvo videoverbalnogo teksta [Semantic field of videoverbal text]. Moscow: MGU.
20. Sindoni, M. G., Wildfeuer, J., & O'Halloran, K. (Eds.) (2016). Mapping Multimodal Performance Studies. N.Y.: Taylor & Francis.
21. Suciu, L., & Mocofan, M. (2015). Multimodality and multimediality in communication. Professional Communication and Translation Studies, 26, 33.
22. Wildfeuer, J., & Bateman, J. A. (Eds.). (2016). Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning. N.Y.: Taylor & Francis.
23. Zeiser, A. (2015). Transmedia marketing: From film and TV to games and digital media. London: Routledge.