Principles Of Task Development For Training Post-Editing Of Machine Translation

Developing PEMT tasks and preparing materials

First of all, we prepared a lecture based on theoretical studies and European programs listed above. It covers differences between translation and PEMT, types of MT systems and types of PEMT, areas of PEMT applicability and PEMT strategies.

Next, we developed tasks with the focus on three important aspects of the post-editing process: determining the type of PEMT (corresponds to criteria 4), determining PEMT applicability (criteria 1), and evaluating the results of the MT system (criteria 3) (Dorozhkina & Ivleva, 2020).

And finally, we chose a MT system as we wanted students to get the same MT output consistently. We recommended using “Yandex Translator” because it is neural MT system, which provides coherent text for a wide range of text registers. According to the TA TranExpress study, Yandex system demonstrates the best of quality for the Russian language (TranExpress, 2020).

Working with group 1 and group 2

To formulate task-developing principles, we planned and conducted an experimental class for 2 groups of third-year students who major in linguistics. Each group was given both theoretical information and practical tasks. All of them were asked to use “Yandex Translator” (Mitchell, 2012). The action research was conducted with the group 1 of 14 students with subsequent statistical analysis. Processing the analysis results, we made necessary changes and repeated the updated PEMT training with group 2 of 13 students. The results of the two groups were compared as well.

First task: types of PEMT

In some tasks here we include not all content from tasks as they are quite long.

.

The human brain is made of densely packed neurons. Some estimates place the number at more than 100 billion (about the number of stars in the Milky Way), each of which is capable of receiving and passing neural impulses to sometimes thousands of other neurons and is more complex than any other system known to be in existence (Solso, 2014).

Головной мозг человека содержит огромное количество нейронов. По некоторым оценкам, их число составляет более 100 миллиардов (примерно столько звезд в Млечном Пути). Каждый из нейронов способен принимать нервные импульсы и и передавать…

: The study of sensation generally deals with the structure and processes of the sensory mechanism and the stimuli that affect those mechanisms. Perception, on the other hand, involves higher-order cognition in the interpretation of the sensory information. Basically, sensation refers to the initial detection of stimuli; perception to an interpretation of the things we sense (Solso, 2014).

: Изучение ощущений обычно связано с устройством и работой органов чувств и со стимулами, которые воздействуют на эти органы. С другой стороны, восприятие подразумевает участие высших когнитивных механизмов в интерпретации сенсорной информации...

The results of both groups are presented in the Table 1.

Obviously, group 1 did not always manage to determine Full PEMT in the last text. The post-editor made minimal changes to the text, so the students were not able to recognize Full PEMT. Consequently, it is important to note in the rubric that the type of PEMT is determined not by the number of changes made, but by the final result. That is why we reformulated the rubric for the group 2 and the result was better:

Determine the type of PEMT (light or full). When completing the task, you should pay attention not to the number of changes made, but to the quality of the PEMT output.

Second task: PEMT applicability

Evaluate the applicability of the PEMT for each text using pre-translation analysis. Determine the target audience, the communication task, the text register and its main type of information and make a conclusion about the applicability of the PEMT.

The first step in devising efficient and effective methods for teaching women equitation, is to make certain that both riders and their instructors know the differences between male and female anatomy, and understand how such important parts as the hip sockets, lower back and seat bones of each sex function in the saddle (Bennet, 2008, p. 1).

Join some of your “Frozen” favorites—Anna, Elsa, Olaf, and Kristoff—around the campfire as they tell legends exploring the world of “Frozen 2.” You can hear these stories on Google Assistant-enabled Android and iOS phones, smart speakers and Smart Displays. To get started, just say, “Hey Google, tell me a ‘Frozen’ story” and you can pick which character you’d like to narrate (Byer, 2020).

To better handle that kind of nuance, a group led by researchers at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) has developed a machine learning model that can look at an X-ray to quantify how severe the edema is, on a four-level scale ranging from 0 (healthy) to 3 (very, very bad). Results are presented in Table 2:

For the second text, only slightly more than half of group 1 chose the correct answer. This text is an advertisement for the Google Assistant feature. Notably, 83.3% of students who gave the wrong answer identified the type of text as an instruction. According to Alekseeva (2004), the advertising text and the instructions have a common feature: the predominance of operational information. It is often expressed by a large number of imperatives, which can be seen in text 2. The difference is that the advertising text combines operational and emotional types of information, while the instructions contain operational and cognitive.

In addition, the students of group 1 did not pay attention to critical meaning errors in MT output. The output may seem logical to a post-editor who has no experience in the IT field, and affect the assessment of PEMT applicability accordingly. Text 4 is also an advertisement (KitKat advertisement), and a larger number of students gave the correct answer. This is due to the fact that the topic of chocolate was more understandable to the participants of group 1. So for group 2, text 2 was replaced with text on a familiar topic – the advertisement of sport app without IT-specific descriptions – and the result was better.

Third task: MT output errors

Specify the type of MT in each case. Use the LISA classification. Note that there may be several errors in a single fragment. The categories of errors can also be duplicated (for example, if you see that a sentence has both a typo and a punctuation error, put LA twice).

1. There is another side to the sensory and perceptual process that is supported by studies… (Solso, 2014) Есть другая сторона к сенсорному и перцепционный процессу, которая поддержан исследованиями...

2. The pelvic construction of men and women is different (Bennet, 2008, p. 2). Тазовое строительство мужчин и женщин отличается.

3. How can Britons simultaneously be both self-controlled and prone to rip our clothes off in a drunken haze? Как британцы могут одновременно быть самостоятельным и склонны рвать одежду в пьяном тумане?

4. Heaven forfend–remember the Monkey’s Paw! Небеса защитят – помните Лапу Обезьяны!

5. Many health issues are tied to excess fluid in the lungs. A new algorithm can detect the severity by looking at a single X-ray (Conner Simons, 2020). Многие проблемы со здоровьем связаны с избытком жидкости в легких. Новый алгоритм может определить степень тяжести, посмотрев на один рентгеновский снимок.

Results are shown in Table 3:

. The semantic error was revealed by a smaller number of group 1 students. There is ambiguity in the sentence: it is not clear whether studies support the side or the process. It is recommended to draw the attention of students to the fact that ambiguity is also a meaning error. We did so with group 2 and the result was better.

. A stylistic error (“отличается – различается”) was noticed by a little more than half of the students of both groups. This issue should be elucidated within the existing course. We should explain the significance of connotative meanings of words. Importantly, 3 out of 8 students who answered correctly, indicated that they were not sure whether the error was connected with style or meaning. In this connection, a survey of 11 professional editors working in different organizations was conducted, and 9 out of 11 (81,8 %) specialists identified the categories of edits as stylistic, arguing that the wrong choice of prefix leads to a distortion of the text logic, while the reader is more likely to understand the meaning.

The meaning error (“self-controlled – самостоятельным”) was noticed by a little more than half of the students of both groups. The following tendency is revealed: if several errors are made in the text, the meaning errors are harder to see. It is highly recommended to add more tasks on searching for meaning errors made by the MT system and to focus on them in complex tasks.

Notably, some of the remaining students who did not identify the stylistic error indicated it as a meaning one (4 people from group 1 and 3 people from group 2). The meaning of the set expression(“Боже упаси”) is indeed distorted by the machine translation system, but the error is still stylistic, since it appeared because of missing cultural adaptation and literal translation. Approaches to these error categories differ and it is worth analyzing the differences. It is also recommended to focus students' attention on the fact that only one category can be assigned to an error. In our case, if the meaning is distorted due to the word-to-word translation of an original set expression, the error is stylistic.

Participants of group 1 did not identify ambiguity: it is not clear whether the refers to health problems or to the filling of the lungs with fluid. As for group 2, putting ambiguity in a separate subcategory affects the result.