The paper discusses several methodological challenges and perspectives for the semiotic discourse analysis of media practices considered in its multimodality. Despite the somewhat tautological character of the expression
Keywords: Multimodalitysemiosiscommunicational modesemiotic codemedia communicationmass media
The contemporary cross-disciplinary studies of language and communication have been focusing on multimodal communicational phenomena for already more than 15 years (Kress & van Leeuwen, 2001; Baldry & Thibault, 2005; Kress, 2010; O’Halloran & Smith, 2012; Wildfeuer & Bateman, 2016; Bateman, Wildfeuer, & Hiippala, 2017; Jovanovic & Van Leeuwen, 2018). It is rather commonplace today to consider verbal means not to be the crucial element of mass media semiosis (Sindoni, Wildfeuer, & O'Halloran, 2016; Danesi, 2018; Djonov & Van Leeuwen, 2018; Hiippala, 2018). It means that observing them as central in media communication would hardly contribute to constructing the manifold and realistic view of media discursive processes.
The current media studies tend to analyse the media discourse as the multimodal phenomenon (Kress & van Leeuwen, 2001; Machin & Mayr, 2012). The semiotic construction and translation of meaning structures, or semiosis, enacts both verbal and non-verbal – auditory, visual and proxemic means of communication. Besides, the multimodal nature of the mass media semiosis appears in the wide usage of various semiotic means for translating the message (news, facts, opinions, etc.). Messages can be communicated in the mode of verbal text, infographics, video message or animation, while the audience tends to prefer visual means of communication.
The technological background of media communications as well as the intentions of media communicators to set the story or content in the focus of audience's attention contribute to the intensive and expansive visualization of the data originally created in the verbal mode. In mass media communications, there are now various means to stimulate the involvement and cooperation of addressees in the discursive media practices. They are, for example, inquiries, quizzes, integrated educational content, gamification forms, etc. Media communicators use these means extensively to promote the engagement of audience. The pragmatic dispositions and intentions of such “visual turn” of media communications are obvious today, since they aim at increasing the audience outreach, controlling the audience attention and supporting the customers’ loyalty.
However, the contexts and effects of such processes are much further than marketing efficiency. Though the practices mentioned above are often focusing on rather marketing goals, their prevalence in media communication certainly changes the semiotic aspects of media communications, audience perception dispositions and communicators’ data processing. These impacts and effects correlate to the phenomenon of
Kibrik (2010) shows that the term
There is a widespread claim that any discourse and any communication is multimodal by its nature. As Kibrik (2010) states, “the attempts to divide crucially language from communication, thinking and behavior are of little efficiency, and artificial, and are determined only by the logics of science development but not by the nature of things” (p. 135). Moreover, “there is no principal difference between verbally transmitted information and the one sent prosodically or by gestures. The only approach which has the chance to build the realistic linguistics of future is the one accepting the multimodality of communication” (p. 148).
It is important to discuss two crucial methodological aspects referring to the Kibrik’s statement mentioned above. Researchers in the field of social semiotics (Kress, 2016) and critical discourse analysis (Machin & Mayr, 2012) provide largely the key arguments for this position. The mentioned research fields implement the cross-disciplinary approach to discursive phenomena and semiotic means of communication
This view towards the multimodal analysis of communication and discourse seems to be reasonable. However, if one assumes the communicational equality of all the semiotic means used in the media communication, there would be no need for discussions about the differentiation of effects and impacts of media of different modes (radio, printed media, TV, Internet, etc.). Yet, there is a difference in how they affect audience, while mode types determine obviously this difference. Thus, it is rather important to reveal the differentiation in multimodal semiotic processes occurring in mass media.
The paper aims at discussing two principle methodological questions that appear to be central for the multimodal analysis of media semiosis and media discursive practices:
a) If one assumes that any communicational situation is multimodal by its nature, is there a need for the term ‘multimodal’ in appliance to media communications?
b) What types of semiotic processes can be distinguished in media discursive practices according to the multimodal nature of media?
Purpose of the Study
To achieve the clear view on the methodological basis for multimodal analysis of media texts, we have to:
a) Set the terminological ‘area’ of the notion multimodal in comparison to synonymous and allied notions,
b) Classify and specify various kinds of semiotic changes that occur in multimodal discursive practices.
To achieve the purpose stated above we used:
a) The method of terminological comparative analysis to evaluate the semantic potential and the character of usage of terms
b) The typological method to itemize and specify the types of communicational channels engaged into the multimodal media practices, the types of semiotic changes and migration of meaning structures in multimodal communication.
All the findings base on the results of the observation of the most quoted current works in the field of media discourse analysis and social semiotics.
There are three aspects of practices of intensive semiotic closely related to the ways of informational representation in mass media:
Despite the variety of term usage, the researchers express solidarity in defining the central specifying features of multimodal texts and discourses (Bateman et al., 2017; Fedorova & Kibrik, 2018; Danesi, 2018). These are semiotic
We tend to share the view towards the multimodal analysis of communication and discourse. The mentioned research fields implement the cross-disciplinary approach to discursive phenomena and semiotic means of communication.
Yet, it is important to discuss two crucial methodological aspects referring to the Kibrik’s statement quoted above.
First, if one assumes the ontological status of multimodality of the discursive, and wider – communicational, practices, one should face the tautology of the use of the term
We consider that there is no semantic exaggeration in such terms as
Secondly, the Kibrik’s quotation above suggests the interchange of semiotic means for informational interaction (“there is no principal difference between verbally transmitted information and the one sent prosodically or by gestures”), which makes state the methodological issue of relevance of semiotic means: what are the conditions and limits for coherence (integration) of multimodal ways to use semiotic means in discursive practices? It seems obvious that they interact in many different ways apart from interchanging. Poymanova (1997) distinguishes such ways of interaction of verbal and non-verbal means in mixed texts as
It is worth remarking that the opposition “one means – other means” does not necessarily comprise the juxtaposition of verbal and non-verbal means. We claim that in various discursive practices the valuable volume of information can be transmitted without any verbal elements. As one can observe it in advertising, music or cinema, the main semiotic interactions occur between audial and visual means. We consider the more realistic typology of modes / modalities and semiotic means to base on the differentiation of perception channels. What follows is the attempt to elaborate such typology.
Multimodal discourse is accomplished via a)
There can be the following types of interaction between modes and relevant semiotic elements:
Thus, it is rather important for discourse analysts to have the methodological tools for evaluation the discursive potential of semiotic resources that are available to addressers.
It is considerable that not any semiotic tool fits any of the intermodal interaction type mentioned above. For example, communicants can hardly use the prosodic means to reproduce or repeat the information expressed before in the written-verbal mode. Onу would fail in communicating a press release with pauses and sighs, as well as it is almost impossible to transmit the TV series plot with colors and gestures. However, many “semiotically centered” means, such as verbal elements, do not principally go autonomously or separately from other semiotic means. The word is always implemented in extralinguistic parameters, such as visual (type, color) and auditory (volume, tone) ones.
This fact provides the notion of transduction in/of media texts. A text is transductive if there is the potentiality for its semiotic transformation, i.e. recoding in other semiotic system while preserving the original meaning structure – whether denotative as minimum or connotative one as maximum.
Kress (2010) observed in detail the phenomena of semiotic transduction and transmodal semiosis in mass media. According to Kress (2016),
while in linguistics the starting point for thinking or working is language which is understood as speech or writing, “material” resources of language in multimodality come much beyond speech and writing. The word material is used here in the meaning referring to the phenomena available for receptors, for (human) sensorium. (p. 78)
The fact that media communicators use extralinguistic means to communicate the meaning attests to the multimodal “turn” of their profession, determined by the technical progress and material development. However, there is a risk in exaggerating these processes. As it always happened in the history of media systems, the appearance of new communicational means lead to enlarging of representational capacities of journalism, advertising, public relations and other media communicational practices. Thus, the appearance of photography contributed to the growth of objectivity of news printed texts, as well as technological capacities of cinema stimulated the development of documentaries. However, the scale and tension of multimodality are immense nowadays, and its effects are hardly predictable.
Thus, two main semiotic processes linked to the migration of meaning structures in contemporary media texts are
However, it is not the only mode of semiotic changes. The meaning structures can be transmitted with semiotic means belonging to other codes and modes, e.g. audiovisual, graphical, actional (interactive), etc. Thus, it is rational to take into account the intersemiotic transduction.
We define intersemiotic transduction as the reproduction of the meaning structure with means of the semiotic system that differs from the one of the original messages. Media communicators widely use infographics to communicate an originally printed text, or podcasts to transmit the originally video messages.
The transduction can be both
Another important methodological issue is the motivation of semiotic changes. The motivation can be
Thus, there is an actual task traditionally set in social semiotics and critical discourse analysis. i.e. the task of revealing the ideological basis of discursive processes. Social semiotics and critical discourse analysis treat multimodality not only as the immanent ability of texts to express the reality with various semiotic means, but also as the “advanced” way to construct the reality.
As Machin and Mayr (2012) state,
visual communication, as well as language, both shapes and is shaped by the society. MCDA [multimodal critical discourse analysis – E.K.] therefore is not so much interested in the visual semiotic choices in themselves, but also in the way that they play a part in the communication of power relations. (p. 10)
Let us add that this fact surely refers to a wider range of processes than only visual communication, i.e. to all possible modes of communication, and different semiotic resources and codes. Hence, the possible objects of social semiotics and critical discourse analysis can also be, for instance, the prosodical means of domination and subordination in radio communication, proxemic construction of inequality in Instagram, or kinesic means of representation of “us” and “them” on TV, etc.
At last, the perspectives of multimodal analysis of media discourse correlate with the study of discursive practices in different professional fields. The issue of intersemiotic and interdiscursive coherence and intersemiotic transduction is highly important in cross-medial and trans-medial communication. For example, the choice of strategies and techniques of storytelling on different media platforms, the search for appropriate infographic means for representation of complex data and adopting the content to the semiotic expectations of the target audience – all these practices presume the special awareness in intersemiotic multimodal processes. It is rather obvious that these research initiatives are cross-disciplinary and they encircle the whole range of sciences, such as semiotics, linguistics, IT studies, social anthropology, cultural studies and many other research fields (O’Halloran, Tan, Pham, Bateman, & Vande Moere, 2018) contributing to the construction of the full coherent view on the multimodal discursive practices.
I thank my colleagues from the faculty of journalism, Belgorod National Research University, who provided insights, discussions and comments, and who encouraged the research.
- Baldry, A., & Thibault, P. J. (2005). Multimodal transcription and text analysis. London: Equinox.
- Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research and analysis – a problem-oriented introduction. N.Y.: Walter de Gruyter GmbH & Co KG.
- Danesi, M. (2018). Understanding media semiotics. L., N.Y.: Bloomsbury Academic.
- Djonov, E., & Van Leeuwen, T. (2018). Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice. Social Semiotics, 28(5), 641-664.
- Egorchenkova, N. B. (2014). Strategicheskiy potentsial multimodalnoy interaktsii v media-politicheskom diskurse [Strategic potential of multimodal interaction in media political discourse]. Bulletin of Tomsk State University, Volume: Philology, 5(31), 24-37.
- Fedorova, O. V., & Kibrik, A. A. (2018). Obschee, individualnoye i kontekst v multikanalnoy kommunikatsii [General, individual and context in multimodal communication]. Cognitive Studies of Langiuage, 33, 637-645.
- Hasebrink, U., & Hepp, A. (2017). How to research cross-media practices? Investigating media repertoires and media ensembles. Convergence, 23(4), 362-377.
- Hiippala, T. (2018). Structure of multimodal documents: An Empirical Approach. L.: Routledge.
- Jovanovic, D., & Van Leeuwen, T. (2018). Multimodal dialogue on social media. Social Semiotics, 28(5), 683-699.
- Kibrik, A. A. (2010). Multimodalnaya lingvistika [Multimodal linguistics]. In Cognitive Studies – IV (pp. 134-152). Moscow: IP RAN.
- Kozin, A. V. (2018). Consecutive Interpreting: From Language to Communication. In Consecutive Interpreting (pp. 37-66). Cham: Palgrave Macmillan.
- Kress, G., & van Leeuwen T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.
- Kress, G. (2010). Multimodality. A social semiotic approach to contemporary communication. London: Routledge.
- Kress, G. (2016). Sotsialnaya semiotika i vyzovy multimodalnosti [Social semiotics and challenges of multimodality]. Political Science, 3, 77-100.
- Machin, D., & Mayr, A. (2012). How to Do Critical Discourse Analysis: A Multimodal Introduction. London, N.Y.: Sage.
- O’Halloran, K. L., Tan, S., Pham, D. S., Bateman, J., & Vande Moere, A. (2018). A digital mixed methods research design: Integrating multimodal analysis with data mining and information visualization for big data analytics. Journal of Mixed Methods Research, 12(1), 11-30.
- O'Halloran, K., & Smith, B. A. (Eds.). (2012). Multimodal studies: Exploring issues and domains (Vol. 2). London: Routledge.
- Omelyanenko, E. A., & Remchukova, E. N. (2018). Polikodovyye teksty v aspekte teorii multimodalnosti [Polycode texts in terms of the theory of multimodality]. Communication Studies, 3(17), 66-78.
- Poymanova, O. V. (1997). Semanticheskoye prostranstvo videoverbalnogo teksta [Semantic field of videoverbal text]. Moscow: MGU.
- Sindoni, M. G., Wildfeuer, J., & O'Halloran, K. (Eds.) (2016). Mapping Multimodal Performance Studies. N.Y.: Taylor & Francis.
- Suciu, L., & Mocofan, M. (2015). Multimodality and multimediality in communication. Professional Communication and Translation Studies, 26, 33.
- Wildfeuer, J., & Bateman, J. A. (Eds.). (2016). Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning. N.Y.: Taylor & Francis.
- Zeiser, A. (2015). Transmedia marketing: From film and TV to games and digital media. London: Routledge.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
About this article
07 August 2019
Print ISBN (optional)
Communication studies, press, journalism, science, technology, society
Cite this article as:
Kozhemyakin*, E. (2019). Multimodal Semiosis In Mass Media: Several Remarks On Methodology. In Z. Marina Viktorovna (Ed.), Journalistic Text in a New Technological Environment: Achievements and Problems, vol 66. European Proceedings of Social and Behavioural Sciences (pp. 18-25). Future Academy. https://doi.org/10.15405/epsbs.2019.08.02.3