Cross-Cultural Research Of Strategies And Efficiency In Visual Semantic Search  

Abstract

The influence of linguistic background on cognitive task performance was identified as one of prospective research areas in the early 80s. New technologies evoke a new interest to investigation in this area. However, most of the studies on linguistic experience and cognitive skills have been conducted on an English-language sample. It becomes obvious that the identification of similarities and differences in language processing should be studied in systems with different language organization, including different writing systems: hieroglyphic, syllabic and alphabetic writing. The current study involved participants of three language groups (Russian, Chinese and Japanese, n=64) who took part in an eye-tracking experiment on semantic visual search. The participants were asked to look for meaningful English words in 18 different letter matrices (15*15), search success and oculomotor parameters were registered with the SMI RED system. The results indicate significant differences in most of the considered characteristics. Data show that different aspects of linguistic experience (native language, the level of linguistic competence, current proportion of language use) influence word search strategies. Task performance seems to be associated with specific oculomotor patterns, indicating a high level of cognitive processing with longer fixations on the relevant information and faster glides along irrelevant noise. The findings not only clarify the influence of linguistic experience on the cognitive architecture of semantic search but also sets directions for technology implementation in linguistic competence assessment.

Keywords: Cross-cultural comparisonefficiency predictorseye-movementslinguistic experiencescanning patternsvisual semantic search

Introduction

Culture-specific manifestations of behaviour and mental functioning have been profoundly studied across subfields of psychology. Several comparative studies of the ‘theories of mind’ demonstrated that even their neural correlates may begin to vary depending upon cultural and linguistic background both in children (Kobayashi et al., 2007) and in adults (Toyama et al., 2015). Contrast cultures and various social practices not only produce differences in mental representations but also influence the architecture of cognitive processes involved. There are several solid reasons to believe social and cognitive practices are mutually supportive (Nisbett et al., 2001).

An interesting approach was suggested by Nisbett (2003), who considered the differences in cognitive processing systems that are inherent in a particular culture. His theory postulates that representatives of Western culture (primarily European countries and the USA) use analytical methods of cognitive processing, concentrating on objects and their characteristics, while the inhabitants of the East (Chinese, Japanese and Koreans) process information more holistically, concentrating on both the object and its context. Holistic practices tend to regard the environment in the complexity of interconnected factors. The observer takes into account the full spectrum of characteristics of the field to make a decision (Kitayama et al., 2003).

Language and writing system are definitely among the most important and spread social practices that support cognitive differences between their users (Hong et al., 2000). Attempts were made to prove that mental processing depended greatly upon these systems. Preferences in cognitive patterns are associated with the language systems mastered previously. The grammar and structure of Indo-European languages contribute to the idea of a world built of atomic blocks while Asian language systems tend to focus on the continuous and holistic nature of the world. East Asian languages (One of the obvious characteristics of the Chinese writing system is its visual complexity. Chinese characters represent specific meaningful forms of the Chinese language. Hieroglyphs include wider units of meaning than letters or syllables (Yu & Reichle, 2017). In Japanese, three types of characters can be used in a text: kanji, hiragana, and katakana. They differ at the writing system, orthography, and script levels and have different linguistic roles. Japanese kanji characters are logograms historically borrowed from Chinese characters, which represent phonological and semantic information. In a text, they are primarily used for the stems of content words such as verbs, nouns, and adjectives. At the writing system level, Japanese kanji characters represent a syllable or syllables and are therefore similar to Chinese characters (Kajii et al., 2001).) are characterized by high contextuality, whereas English-speakers teach their child to understand speech without context (Heath & Mangiola, 1991).

In a number of studies, it was shown that, when performing tests on the perception of the ‘part-whole’ or the ‘figure-ground’, the residents of Southeast Asia are more likely to pay attention to the entire perceptual field and notice elements of the context. The inhabitants of North America, in contrast, focus on the details of an object, abstract from its context (Na & Chan, 2015). However, the underlying mechanisms for the apparent differences in cognitive processing strategies have not been known.

Eye movement indicators have been used to identify the cognitive architecture of information processing and word recognition in people with different linguistic competence (Blinnikova & Izmalkova, 2016; Leinenger & Rayner, 2016). Most previous research into eye-movement patterns has been carried out with the material of alphabetic languages. Although nonalphabetic languages such as Chinese (e.g., Yen et al., 2012; Zhang et al., 2019) and Japanese (e.g., White et al., 2012) have attracted an increasing number of eye movement researchers recently, substantial studies on the material of hieroglyphic languages are still limited. However, the investigation into the characteristics of texts using nonalphabetic writing may add unique contributions to understanding the mechanisms of oculomotor control in reading (Schotter & Rayner, 2015) and other verbal visual tasks (Blinnikova et al., 2019). It has already been shown that the writing system (alphabetic or hieroglyphic) and reading direction influence eye movement strategies (Reilly & Radach, 2012).

Problem Statement

In the present research, we examined how cultural factors influence cognition. Thus, three groups of subjects took part in our study – Russian, Chinese and Japanese students belonging to three different cultures and having different experiences in mastering writing systems. Writing systems are important social practices associated with the development of specific features in cognitive activity. Logographic scripts differ from alphabetic scripts in many dimensions (Yu & Reichle, 2017; Kajii et al., 2001). Although most Chinese and Japanese words include 2 or 3 hieroglyphs, the latter are not similar to letters. First, there are much more hieroglyphs than there are letters in the alphabet, so the information density in each character is much higher than in a letter. Second, hieroglyphs are pictographic symbols which are perceived in a rather holistic manner as compared to the processing of letter strings, composing the words of alphabetic scripts. Third, word length is shorter in logographic scripts. Forth, there are no spaces in Chinese and Japanese texts to separate words (Li et al., 2011). It is worth noting that hierographic writing sets less requirements for the direction of reading. These differences should influence cognitive processing, though, as previous studies show, this may not be apparent in all sorts of tasks.

We used a visual semantic search task that implies detecting a target object on a screen filled with distracters (Blinnikova et al., 2015). Participants start the search, being unaware of which word to look for; the goal is to find any correct word in a stimulus letter matrix. This task combines the reading task and the lexical decision task. Just like in reading, the subject is to scan the space filled with letters. However, unlike reading, eye movements are unguided and less organized. Just like in a lexical decision task, the subject constantly analyzes whether the letter string is a word. Whereas, in contrast to the lexical decision task, it is the subject who highlights a chain of letters to consider (Blinnikova et al., 2015). We decided that all the subjects should receive an equal task, so the task of word search in the alphabetical matrix was in English. It can be assumed that representatives of the Asian cultures will show lower performance in this task, since they will not be able to take advantage of the semantic context and holistic processing. The opposite assumption can also be made that Chinese and Japanese subjects, on the contrary, will benefit, since they are used to dealing with unspaced texts. There is also reason to believe that subjects mastering hieroglyphic writing since childhood will be less dependent on the orientation of the words in matrices. Therefore, we included vertically and horizontally arranged words in the matrix.

We expect to find cross-cultural differences in both semantic search efficiency and eye-movement parameters. Such eye-movement measures as fixation duration, fixation count, saccadic amplitude, duration and velocity are taken into account. It has been previously shown that some aspects of eye movement control in reading (Li et al., 2011) and looking at scenes (Chua et al., 2005) appear to be script-specific. However, existing data are of conflicting nature (Tsang & Chen, 2012) that require investigation and explanation.

Research Questions

In this study, we compared performance and eye-movements of Russian, Chinese and Japanese students in a visual semantic search task. We address the main question of to what extent the information processing is governed by the properties of specific languages and writing systems. Identifying the differences in performance between the representatives of different cultures might reveal important universal and culture-specific perceptual patterns. The presentation of stimulus materials in English language – mastered by all the participants as a foreign – raised the question of the influence of language competence. Since this question has been widely addressed in previous studies, we focused on the interaction of foreign language competence with the experience of mastering the written system of the native language.

Purpose of the Study

The main goal is to reveal the effect of linguistic background. We assume that the users of hieroglyphic writing systems will demonstrate holistic patterns of semantic stimuli perception and show better results in reading vertically-oriented targets in a visual semantic search task. On the contrary, those who have acquired only the alphabetic system of writing will follow the analytical strategy of step-by-step search and show the best results in the search and identification of horizontally arranged words. We also tested the hypotheses that the level of language competence affects the efficiency and strategies of word search and recognition.

Research Methods

Findings

Eye-movement parameters in word search

Analysing eye movement indicators, we identify the distinctive features of the word search and recognition process in Russian, Japanese and Chinese language groups. The linguistic cognitive strategies used by representatives of each language group are of particular interest.

A total of 1152 trials were recorded. Further analysis was performed on trials. The analysis of the obtained data showed that the three groups of participants (Chinese, Japanese and Russian students) differ significantly in a set of eye-movement parameters. These include fixation count, total and average fixation duration, total and average saccade amplitude, average saccade duration, average saccade velocity (see Table 02 ). However, in order to clarify the nature of the differences, the means of the three groups were compared separately.

Comparing the results of Russian respondents to those of Japanese ones, we found that the Russian sample demonstrates a larger total and average duration of fixations. At the same time a shorter average duration of saccades (31,19 ms vs 60,78 ms, F (1,719) = 71,64; p < 0,01), shorter overall and average amplitudes, and lower speed of saccades are observed.

See Full Size >

The patterns of eye movements in Japanese and Chinese samples are also significantly different. Compared to Chinese group, Japanese subjects demonstrate a smaller number of fixations, their shorter duration, and at the same time, almost double the values in saccade duration (60,78 ms vs 31,63 ms; F(1,746) = 76,83; p < 0,01) and total amplitude. They also demonstrate a high saccadic speed, while Chinese subjects demonstrate a higher average saccade amplitude. Japanese subjects demonstrate a pattern with short fixations but long saccades with greater speed and overall amplitude.

Comparing the Chinese and Russian samples resulted in significant differences in saccade parameters (see Table 02 ). Chinese subjects demonstrate not only a higher saccade speed, but also greater saccadic amplitude (4,95° vs 4,08°; F(1,763) =8,00; p < 0,01), in comparison with Russian subjects. As This corresponds with the findings of several investigations which demonstrate that script-specific aspects of eye movements primarily relate to the perceptual span (see Tsang, & Chen, 2012). Chinese has hieroglyphic writing, the distance between the hieroglyphs exceeds the distance between the letters in the alphabetic letters. The hieroglyphs are analysed mainly separately, while in the process of reading letters we focus on the selective components of the word.

Thus, the obtained results characterize the oculomotor patterns of word search in the three language groups. Russian subjects demonstrate the highest rates in parametres, associated with fixations: a greater number of fixations, a greater duration of fixations. At the same time, their results take an intermediate position between the results of the other groups in such saccadic parameters as count and frequency, while demonstrating the lowest scores in saccadic amplitude, duration and speed. This pattern might be connected to the alphabetic writing, which requires shorter saccadic movements due to small distance between the letters in the words and more fixations on a single word for its semantic processing (Although, with the developed reading skills of native speakers, practicing alphabetical writing, visually presented words can be perceived integrally (like hieroglyphs) and read out in one single fixation (Rayner, 1998). However, judging from our results, the experience of letter-by-letter gathering of words persists. The participants can use it perfectly well in difficult conditions.) as compared to the process of retrieving the meaning of a hieroglyph.

Japanese subjects demonstrate the lowest rates associated with fixations duration; however, they overcome the results of the other groups in saccade parameters almost twice: they have the greatest overall saccade amplitude, the greatest saccadic durations and the highest speed. This pattern of oculomotor activity with an emphasis on the breadth of saccadic movements is associated with syllabic and hieroglyphic writing and reading systems, which suggest a greater distance between characters than letter systems do. However, in addition to the language system, both vertical and horizontal reading systems practiced in Japanese, can influence the characteristics of saccades.

It was expected that the Chinese subjects would demonstrate similar patterns with the Japanese sample, however, the data obtained indicate a greater proximity of the Chinese sample to the Russian one. Despite the similarity between the Russian and Chinese samples in terms of eye movements, the Chinese subjects did not achieve a high level of results in the detection and identification of English words. Such a result remained inexplicable until we began to examine the scanning patterns of the subjects (typical examples of scanning patterns for each group of the subjects are shown in Figure 01 ).

See Full Size >

The comparison of patterns revealed different strategies of looking for meaningful units in a noisy context. Russian students mainly used a reading strategy, while their eyes moved along the rows of the matrix from left to right and from top to bottom. Japanese students also apparently used a reading strategy, trying to combine vertical and horizontal reading directions. Chinese students demonstrated a strategy significantly different from the first two. Their search was fixed on the sides of the square where the matrix was enclosed. It was evident that they needed external contextual support for the search. Such a strategy limited the ability of Chinese students to detect words, as they narrowed the search field. This resonates with the findings of Chua et al. (2005).

Conclusion

The study demonstrated the influence of such aspects of linguistic experience as qualitative features of the native language and level of linguistic competence on the strategies used by the subjects. An analysis of performance indicators showed that groups of different native languages were not equally successful in the word search task: Russian subjects found more words than Chinese and Japanese subjects. Japanese students found a greater number of vertical words in comparison with two other groups.

An analysis of language competence, measured with the Word Associates Test, confirmed the interconnection between language knowledge and the success of word search. An increase in the level of language competence leads to an increase in the total number of found words and the number of horizontally - oriented words found. This effect persists in each language group.

In Russian-speaking students, practicing an alphabetic writing system seems to require shorter saccadic movements due to small distance between letters in words. It also requires more fixations on a single word for its semantic processing than is required for fixing the meaning of a hieroglyph.

For Japanese respondents, the pattern of oculomotor activity with an emphasis on the breadth of saccadic movements is associated with syllabic and hieroglyphic writing systems. This suggests a greater distance between characters than in letter systems. Saccade parametres can also be influenced by the presence of both vertical, and horizontal reading systems in Japanese.

For Chinese students, the fixation and saccade parametres were in the middle between representatives of Russian and Japanese samples. The duration of fixations was longer than in Japanese sample but shorter than in Russian one. The amplitude, duration, and speed of saccades were lower than in Japanese group but higher than in Russian sample. At the same time, significant differences in the parameters of oculomotor activity in Chinese students were more vivid in comparison with Japanese students than with Russian ones. However, the lowest task performance in visual semantic search was shown. We found an explanation for such a strange pattern in the general strategy of visual behavior of Chinese subjects. Their eye movements were fixed close to the boundaries of the matrix. We suggest that the pronounced contextuality of perceptual processing in Chinese subjects is manifested this way. When processing visual verbal information, they are more in need of including it in some external context. Of course, this assumption requires further analysis and more rigorous confirmation. However, this confirmation will indicate the influence not only of the writing system itself but also of a wider cultural context.

The core hypotheses of the study were confirmed, suggesting further research into the influence of language system on cognitive processing. Cross-cultural differences were established for both the productivity of visual semantic search and eye-movement parameters. The experience of mastering the writing system of the mother tongue constitutes a set of cognitive skills, which determine the strategy used for the visual semantic search in complex contexts. We also identified the influence of language proficiency. It was also established that the experience of mastering the writing system of mother tongue is not directly connected to the characteristics of foreign language competence.

Acknowledgments

This research is supported by the Russian Foundation of Basic Research; Grant # 20-013-00674a.

References

1. Blinnikova I., Izmalkova A., & Rotova N. (2015) Eye movements in the visual semantic search. Journal of eye movement research, 8(4).
2. Blinnikova, I., & Izmalkova, A. (2016). Eye movement evidence of cognitive strategies in sl vocabulary learning. Intelligent Decision Technologies: Smart Innovation, Systems and Technologies, 57, 311–323. https://doi.org/10.1007/978-3-319-39627-9_27
3. Blinnikova, I., Rabeson, M., & Izmalkova, A. (2019). Eye movements and word recognition during visual semantic search: differences between expert and novice language. Psychology in Russia: State of the Art, 12(1), 129–146. https://doi.org/10.11621/pir.2019.0110
4. Chua, H. F., Boland, J. E., & Nisbett, R. E. (2005). Cultural variation in eye movements during scene perception. Proceedings of the National Academy of Sciences, 102(35), 12629-12633.
5. Heath, S. B., & Mangiola, L. (1991). Children of promise: Literate activity in linguistically and culturally diverse classrooms. National Education Association.
6. Hong, Y., Morris, M. W., Chiu, C.-Y., & Benet Martinez, V. (2000). Multicultural minds: A dynamic constructivist approach to culture and cognition. American Psychologist, 55, 709–720.
7. Kajii, N., Nazir, T., & Osaka, N. (2001). Eye movement control in reading unspaced text: The case of the Japanese script. Vision research, 41, 2503-10. https://doi.org/10.1016/S0042-6989(01)00132-8
8. Kitayama, S., Duffy, S., Kawamura, T., & Larsen, J. (2003). Perceiving an object and its context in different cultures: A Cultural Look at New Look. Psychological science, 14, 201-206.
9. Kobayashi, C., Glover, G. H., & Temple, E. (2007). Cultural and linguistic effects on neural bases of ‘Theory of Mind’ in American and Japanese children. Brain research, 1164, 95-107.
10. Leinenger, M., & Rayner, K. (2016). What we know about skilled, beginning, and older readers from monitoring their eye movements: Implications for teaching reading. In J. A. León, & I. Escudero (Eds.), Reading Comprehension in Educational Settings (pp. 1-27). John Benjamins.
11. Li, X., Liu, P., & Rayner, K. (2011). Eye movement guidance in Chinese reading: is there a preferred viewing location? Vision Research, 51(10),1146-1156. https://doi.org/10.1016/j.visres.2011.03.004
12. Na, J., & Chan, M.Y. (2015). Culture, Cognition, and Intercultural Relations. In J. Warnick & D. Landis (Eds.), Neuroscience in Intercultural Contexts (pp. 49-71). Springer Science+Business Media. 10.1007/978-1-4939-2260-4_3
13. Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: holistic versus analytic cognition. Psychological Review, 108(2), 291-310. https://doi.org/10.1037/0033-295x.108.2.291
14. Nisbett, R. E. (2003). The Geography of Thought: How Asians and Westerners Think Differently, and Why. Free Press.
15. Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3), 372. https://doi.org/10.1037/0033-2909.124.3.372
16. Read, J. (2013). Validating a test to measure depth of vocabulary knowledge. In Validation in language assessment (pp. 55-74). Routledge.
17. Reilly, R., & Radach, R. (2012). The dynamics of reading in non-Roman writing systems: a Reading and Writing Special Issue. Reading and Writing, 25(5), 935-950.
18. Schotter, E. R., & Rayner, K. (2015). The work of the eyes during reading. In A. Pollatsek & R. Treiman (Eds.), Oxford Handbook of Reading (pp. 44-62). Oxford University Press.
19. Toyama, K. Imuta, K., Slaughter, V., Kitazaki, M., & Itakura, S. (2015). Comparison of Japanese and Australian adults in theory of mind: Investigating the use of subject and mental state language. IEICE, 114(440), 103-107.
20. Tsang, Y-K., & Chen, H. ‐Ch. (2012). Eye movement control in reading: Logographic Chinese versus alphabetic scripts. PsyCh Journal, 1(2), 128-142. https://doi.org/10.1002/pchj.10
21. White, S., Hirotani, M., & Liversedge, S. (2011). Eye movement behaviour during reading of Japanese sentences: Effects of word length and visual complexity. Reading and Writing, 25, 1-26. https://doi.org/10.1007/s11145-010-9289-0
22. Yen, M.-H., & Radach, R., & Tzeng, O., & Tsai, J.-L. (2012). Usage of statistical cues for word boundary in reading Chinese sentences. Reading and Writing, 25, 1007-1029. https://doi.org/10.1007/s11145-011-9321-z
23. Yu, L, & Reichle, E. D. (2017) Chinese versus English: Insights on Cognition during Reading. Trends in Cognitive Sciences, 21(10), 721-724. https://doi.org/10.1016/j.tics.2017.06.004
24. Zhang, M., Liversedge, S. P., Bai, X., Yan, G., & Zang, C. (2019). The influence of foveal lexical processing load on parafoveal preview and saccadic targeting during Chinese reading. Journal of Experimental psychology. Human Perception and Performance, 45(6), 812-825. https://doi.org/10.1037/xhp0000644