Communication With Low Performer Makes Project Success In Project-Based Learning
Abstract
In this study, we investigated for group exercises for the third year students of the information science department to improve PBL in the software development process. We set up two Research Questions. One is to find "a correlation between individual performance and team performance in software development PBL." The other is to analyze "what factors affect the team's performance in software development PBL." In order to verify the Research Questions, this study evaluates the performance of individuals at first. Second, assess the team's performance. Third, examine the correlation between individual performance and team performance. We measured competencies as an individual performance by peer review. We found in high performers' analysis that teammates believe that they have communicated with high performers. However, high performers do not think that they have communicated with teammates as much as teammates believe. Also, we found in low performers' analysis that low performers believe that they have communicated with teammates. However, teammates do not think that they have communicated with low performers as much as low performers think. On top of that, we found a correlation between communication to low performers and team performance. The results of this study imply that communication with low performers is also crucial for project success even in commercial projects since PBL is a microcosm of society.
Keywords: Software developmentPBL (Project-Based Learning)competencypeer reviewlow performer
Introduction
In the field of education, diverse researchers and practitioners have been implementing approaches to encourage students to learn independently through group exercises. The authors also saw how the students skilled up through group exercises. The students seemed to develop not only technical skills such as environment construction and programming but also the skills of group work. As an example of a group exercise, Yuasa, Oshima and Oshima (2010) regarded PBL (Project-Based Learning) (These definitions are made in comparing Problem-based learning and Project-based learning.) as "the idea of constructivism that knowledge is what learners construct by themselves." In this context, the group activities mean leaners manage to learn by themselves, and teachers support learners as facilitators.” Also, they insisted the learning process should be "dedicated to individual practice."
Many efforts on PBL have been reported. For example, Matsuura (2007) implemented a software development PBL with intending to create the goal of creating product quality (functionality, usability, and maintainability) and to improve the knowledge of students (development technology and communication ability). He used a questionnaire to measure the effectiveness of the implemented PBL. Some studies used tools for evaluation instead of such a questionnaire. For example, Inoue and Kaneda (2008) stated that "As a result of evaluating PBL students by the problem-solving ability test PSI, students and graduate students who played an active role as core members showed a high index of problem-solving ability." There were some discussions about the items to be evaluated. In IPA (2010), the type of companies was classified into problem-solving and value creation, and surveyed about the human resource quality required for both problem-solving and value creation. It can be seen that problem solving and value creation are required in society and also required by human resources. IPA (2010) also defined knowledge, skill, and competency as keywords related to human resource evaluation, as shown in Table
.
Problem Statement
In order to evaluate competency, it is necessary to clarify what and how to evaluate. In the group exercise context, we usually adopt evaluation for group outcomes and schedule management. However, the performance of individual members is also essential. Here we face difficulty in evaluation for the contribution of individuals to their groups. Therefore, the primal motivation of this research is to measure competency in group exercises and to evaluate individual contributions to the groups.
One of the methods of PBL evaluation is peer review among learners. A review paper described by Fujihara, Ōnishi and Katō (2007) cited previous studies by Arnold, Willoughby, Calkins, Gammon and Eberhart (1981), Orpen (1982), and Falchikov (1986) on the peer review between learners, and made an assertion "the evaluation performed by university students is no big difference with or more appropriate than the evaluation performed by teachers."
In this research, we focus on the evaluation of individuals in PBL. In addition to teacher evaluation, consequently, peer review among learners was implemented. In the world of business, peer review evaluation is often performed, which may solve the difficulties in individual evaluation. For example, Asahi and Ohsawa (2005) researched how to use artificial intelligence to find a personnel evaluation scale from a mixture of free-answer and optional-response in 360-degree evaluation (often called multi-faceted evaluation).
Also, we regard PBL as a microcosm of our society. For example, Nakamura et al. (2010) reported a PBL specialized for high school that "the problems found in practice" are described as "everything is the similar as the problems observed in universities and graduate schools." Besides, Makihara et al. (2017) measured that "students are often errors due to the dependency of elements between files as well as practitioners" and described the importance of preventing errors to "run the build frequently locally and check the results so that it is possible to remotely." Such tips are also useful for system engineers in real companies. In summary, this study tries to measure learners' competencies in PBL as non-technical skills, especially to apply peer review to assess the contribution of individuals to groups.”
On the other hand, there are researches conducted by business persons in real companies that can be helpful to students. Edmondson (1999) surveyed 51 project teams from a manufacturer and showed that team psychological safety was related to learning behavior. Team psychological safety is defined as “shared belief held by members of a team that the team is safe for interpersonal risk taking.” The questionnaire includes questions such as "It is safe to take a risk on this team," and "No one on this team would deliberately act in a way that would undermine my efforts." This is also useful for student PBL.
Research Questions
In order to achieve the purpose of this study, we set up the following research questions.
Is there a correlation between individual performance and team performance in software development PBL?
What factors affect the team's performance in software development PBL?
This chapter describes the hypotheses set up for these research questions.
Is there a correlation between individual performance and team performance in software development PBL?
The following hypotheses are set for the first research question. These hypotheses correspond to the items of the questionnaire in Table
[Hypotheses] When the team performance is high, its members
communicate with each other well (Q 1, Q 2)
know each other well what they are doing (Q 3)
share schedule with team members (Q 4)
make enough contribution to the team (Q 5)
are trusted by teammates (Q 6)
We will verify these hypotheses by compiling an average of team members’ performance and analyzing the correlation between the team's average and the team's performance.
What factors affect the team's performance in software development PBL?
The following hypotheses are set for the second research question. “When the team performance is high, its members communicate with low performers.” At the time of team formation, we did not create a team only for high-performed students due to educational considerations. It means every team had at least one low-performed student. When observing students in the first semester, there were some teams whose members are communicating with the low-performed students and others not communicating with low-performed students. It seemed that the performance of the team communicating well with low-performed students was excellent. Therefore, in the second half, we tried to analyze and focus on team performance and communication with low performers. We will verify this hypothesis by analyzing communication with low-performed students and correlation of team performance.
Purpose of the Study
In order to verify the Research Questions described in Chapter 3, we adopt competency to measure non-technical skills in group exercises and to assess contributions to the groups. In this study, we first evaluate the performance of individuals through software development PBL, then, assess the team's performance, and finally, examine the correlation between individual performance and team performance. This chapter describes how to define and evaluate individual performance. Section
Competency
One of the goals of this study is to assess the performance of individuals in software development PBL. In this study, competencies are measured as an evaluation of individual performance with peer review. Here, competency means "distinctive behavioral characteristics that generate specific activities using knowledge and skills," as shown in Table
Some studies tried to measure competency in PBL. Komaya (2009) used a commercially available skill diagnostic tool to measure the competency before and after PBL and showed that PBL increased the competency. Hasegawa, Sakurai and Yuura (2014) proposed students had six roles (leader, practical expert, coordinator, worker, free rider, and the others.) and six types of remarks (verifier, intuitionist, critic, optimist, thinker, and organizer) in PBL. They classified and measured the individual competency according to the role, and the six types of remarks from student records and team competencies measured from deliverables. Yoshikawa, Sakurai and Yuura (2015) used the six role concepts of Hasegawa et al. (2014) to measure competency development trends by the role. They measured students’ competencies not by peer review but from student records.
Peer Review
Another goal of this study is to measure the correlation between individual and team performance. In Fujihara et al. (2007), peer review was defined as "a method of involving students to evaluate classmates with each other in class groups, etc." In addition, they pointed out that it was necessary to analyze the differences between the evaluation criteria of teachers and learners after describing the fundamental matters and issues of peer review. Fujihara addressed this issue through the practice of peer review evaluation.
Ozawa, Mochizuki, Egi and Kunifuji (2005) was an empirical study of peer review and showed that "inter-group evaluation can affect research activities more than feedback by evaluation by assistants" by evaluating each other between groups. Moreover, Mochizuki, Kato, Yaegashi, Nishimori and Fujita (2007) claimed that peer review raised “the sense of learning community and promotes the progress of tasks shared by themselves while being aware of the work of others” in the evaluation experiment of mobile phone software for visualizing division of labor in project learning. The relevance of peer review has been shown in various studies. Our past studies have also shown that peer review is a stable standard in Susumago and Hasegawa (2018, 2019). However, these studies did not measure the correlation between individual and team performance.
This study is characterized by aiming to evaluate students' peer review in software development PBL and to measure the correlation between individual and team performance. Then, we measure the correlation with the performance of the team.
Research Methods
In this study, we evaluated the individual's performance by peer review and the team's performance. This chapter describes the methods of both assessments
PBL targeted for this study
In this study, we conducted group exercises for the third year students of the information science department for half a year. Students took courses in object-oriented programming in Java in their first and second years and object-oriented design with UML in their second year.
We gathered 35 participants for the group exercises in fiscal 2018. In this group exercise, the teacher first presented a couple of themes. Then, students selected their preferred themes. Some teams set the theme by themselves. In addition, the teacher corrected the team members in consideration of the variation in student skills. Finally, it became nine teams of three to six students. Table
Group exercises were conducted in a procedure that was conscious of real software development in business. The leader of each team submits the schedule to the teacher in a predetermined format weekly. The teacher calls each team in turn and asks questions about the schedule. This is called a progress meeting. In principle, all team members will attend this progress meeting. At the progress meeting, the teacher talks with the leader mainly but may check with other members. Besides, each team requests the teacher for a review according to their development stage. For the review request, the students write the team name in the schedule section of the whiteboard. The teacher conducted the review according to the schedule written. In the review, students explain the design document to the teacher and receive comments from the teacher. One of the students fills in a review which manages the issues pointed out in the review and their actions and responses. The team with many unfinished tasks managed the issues by creating a task management table to report at the progress meeting.
Individual's Performance by Peer Review
Table
In the PBL process, students seemed to divide into those who teach to others and those who are taught by others. So, we set up Q 1 and Q 2 about communication to see whether they are the teaching student or the taught student. Q 3 asks if they were aware of the teammate's task. Q 4 asks whether they are aware of the schedule. Q 5 and Q 6 were items for the other teammates as the peer review. High performers and low performers can be defined numerically by these two questions.
The high performer score in each team was almost the highest rating of 4.0. Therefore, we defined students who received the highest evaluation from their teammates as high performers. Regarding the low performer's score, there were variations by the team. Some teams gave low evaluations to low performers, but some teams gave not such bad evaluations to low performers. Therefore, we defined low performers as the lowest scored members of the team.
Team Performance
We evaluated 8 teams, excluding 1 team that could not produce deliverables. The team performance was evaluated by the teacher's and students’ peer review. The teacher's evaluation was a qualitative evaluation by schedule management, progress, and deliverables, because each team had a different theme, and it was difficult to quantify their deliverables in this PBL. Schedule management consisted of the planned schedule and achieved progress. The planed schedule was either a detailed schedule or a rough schedule. Achieved progress showed the extent of delay since every team was late from the planned schedule. Concerning implementation, a qualitative evaluation was also conducted. Since no team was able to implement all design, we evaluated Partial Function Implementation or Design Only (a static HTML demo). In addition, the team with the theme of image recognition could only give a demonstration with a low recognition rate. Therefore, it was evaluated as Less Accurate Demo. For total evaluation, the teacher made a comprehensive judgment based on the evaluation of Schedule, Progress, and Implementation.
Students made a presentation in front of classmates in the final class and evaluated each team in five-scale. Table
Findings
Correlation between Questions of Individual Performance
Table
Table
Students were NOT divided into teaching students and taught students. They taught each other;
Students thought that teammates who had contributed to the team were possible to be trusted.
In Table
The students did not think that they could trust the person who talked about the schedule;
The other students who talked about the schedule did not think that they contributed to the team.
Correlation between Individual Performance and Team Performance
Table
Q 1, 2, and 4 show almost no correlation.
There is a weak correlation for Q 3.
There is a moderate correlation between Q 5 and 6.
The result is that the performance of the team with many students contributing to the team is high, and the performance of the team with many students trusted by the teammates is high. Therefore, we can conclude the hypotheses set in section
Factors Affected to the Team's Performance
High Performer Analysis
Table
All the number of the bottom raw are negative. It means the high performers gave lower ratings than the peer review ratings in all the questions. Especially paying attention to Q1 and 2, the followings can be said:
Teammates believed that they had communicated with high performers;
However, high performers did not think that they had communicated with teammates as much as they thought.
Low Performer Analysis
Table
Low performers believed that they had communicated with teammates;
However, teammates did not think that they had communicated with low performers as much as low performers thought.
From these findings, we can claim that there were recognition gaps among high performers, low performers, and the other teammates in PBL.
Correlation between Low Performer Communication and Team Performance Analysis
Here, we analyze the correlation between communication with the low performers and the team performance from two perspectives. First one is self-recognition that the low performers communicated with their teammates about "teammate deliverables." We regard the average of Q1 in the low performers as this type of self-recognition. Another one is self-awareness that the low performers communicated with their teammates about "their own deliverables." This self-awareness is measured by the average of Q2 in the low performers. Table
We can see a significant correlation between communication about "teammate deliverables" in low performers and team performance. On the other hand, there is no correlation between communication about "their own deliverables" and team performance.
Why is the difference between both perspectives? In Table
Conclusion
In this study, we conducted the group exercises for 35 third-year students in the information science department to investigate the correlation between individual performance and team performance, and which factors affect the team's performance in software development PBL. We set up two Research Questions as we describe in Chapter 3. As stated in 3.1, five hypotheses were set up for the first research question. These hypotheses correspond to the questions in the questionnaire described in 5.1. As the results of the analysis for correlation among the questionnaire items by the peer review related to the team performance, as stated in 6.1, three hypotheses were denied, and two hypotheses were supported. The positive hypotheses are the following; One is "When the team performance is high, its members make enough contribution to the team." The other is "When the team performance is high, its members are trusted by teammates." Both results are natural but not so excited that confirming factors affected the team's performance in software development PBL.
As stated in 3.2, one hypothesis was set up as the second research question. In order to verify this hypothesis, as described in 6.3.3, we considered two definitions as communication to low performers. Based on the analysis of the team performance, we found a strong correlation at the one definition, “the low performers’ self-recognition that they have communicated with teammates.” Therefore, there is a correlation between communication to low performers and team performance. The results of this study imply that communication to low performers is also a key factor for project success, even in real projects since PBL is a microcosm of society.
This PBL was implemented in the second semester for third-year students. Since all teams started at the same time in the first semester, it was possible to evaluate the schedule, implementation and so on quantitatively. In the second semester, some teams worked on a different theme from the first semester, and teams that continued to work on the same theme from the first semester. Therefore quantitative evaluation based on the same standard was difficult. These students will work on similar PBL next year as fourth-year students of the information science department. We want to re-investigate at PBL, where team evaluation is precise.
References
- Arnold, L., Willoughby, L., Calkins, V., Gammon, L., & Eberhart, G. (1981). Use of peer evaluation in the assessment of medical students. Academic Medicine, 56(1), 35-42.
- Asahi, H., & Ohsawa, Y. (2005). Discovery of the evaluation measure from mixed data 360-degree feedback in personnel evaluation. Journal of Japan Society of Artificial Intelligence, 20(3), 167-176.
- Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative science quarterly, 44(2), 350-383.
- Falchikov, N. (1986). Product comparisons and process benefits of collaborative peer group and self assessments. Assessment and Evaluation in Higher Education, 11(2), 146-166.
- Fujihara, Y., Ōnishi, J., & Katō, H. (2007). Review of Research on Peer Evaluation. Journal of Multimedia Aided Education Research, 4(1), 77-85.
- Hasegawa, K., Sakurai, Y., & Yuura, K. (2014). The assessment of improvement of competency of students in practical IT seminar. IPSJ Transactions on Computers in Education, 124, 1-10.
- Inoue, A., & Kaneda, S. (2008). A PBL Approach using Real World Application Development between University and Local Government. IPSJ Journal, 49(2), 930-943.
- IPA (Information-technology Promotion Agency, Japan) (2010). Retrieved from https://www.ipa.go.jp/files/000004543.pdf
- Komaya, S. (2009). Teaching practice of design and development of Enterprises system by real PBL. IPSJ Transactions on Information Systems,107, 177-184.
- Makihara , E., Igaki, H., Yoshida, N., Fujiwara, K., Kawashima, N., & Lida, H. (2017). Investigation of Build Error on Software Development PBL. IPSJ Journal, 58(4), 871-884.
- Matsuura, S. (2007). Software Engineering Education Based on Practical Software Development Experiments. IPSJ Journal, 48(8), 2578-2595.
- Mochizuki, T., Kato, H., Yaegashi, N., Nishimori, T., & Fujita, S. (2007). Portable Development and Evaluation of Cellular Phone Software to visualize the Status of Division of Labour in Project-Based Learning. Japan Society for Educational Technology, 31(2), 199-209.
- Nakamura, S., Lingxiong, B., Hieu, T. T., Hosozawa, A., Yokoyama, W., Yamamoto, H., … Suzuki, N. (2010). The Implementation of PBL in After-School Activities in a Specialized High School. IPSJ Transactions on Computers in Education, 104, 1-6.
- Orpen, C. (1982). Student versus lecturer assessment of learning: a research note. Higher Education, 11, 567-572.
- Ozawa, S., Mochizuki, T., Egi, H., & Kunifuji, S. (2005). Facilitating Reflection in Collaborative Leaming Using Formative Peer Evaluation among Groups. Japan Society for Educational Technology, 28(4), 281-294.
- Susumago, T., & Hasegawa, S. (2018). Multifaceted evaluation of competency in software development PBL. IPSJ Transactions on Computers in Education, 147(15), 1-8.
- Susumago, T., & Hasegawa, S. (2019). Relationship between the character by KT personality test and social skills of students and performance of individuals and teams in software development PBL. IPSJ Transactions on Computers in Education, 149(10), 1-9.
- Yoshikawa, A., Sakurai, Y., & Yuura, K. (2015). Relations between the improvement of competencies and individual roles in group practices. IPSJ Transactions on Computers in Education, 128(4), 1-11.
- Yuasa, K., Oshima, J., & Oshima, R. (2010). PBL dezain no tokuchô to sono kôka no kenshô [Characteristics and effectiveness of PBLs (problem-based learning & Project-Based learning]. Studies in information, Shizuoka University, 16, 15-22.
Copyright information
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
About this article
Publication Date
07 November 2019
Article Doi
eBook ISBN
978-1-80296-071-6
Publisher
Future Academy
Volume
72
Print ISBN (optional)
-
Edition Number
1st Edition
Pages
1-794
Subjects
Psychology, educational psychology, counseling psychology
Cite this article as:
Susumago, T., & Hasegawa, S. (2019). Communication With Low Performer Makes Project Success In Project-Based Learning. In P. Besedová, N. Heinrichová, & J. Ondráková (Eds.), ICEEPSY 2019: Education and Educational Psychology, vol 72. European Proceedings of Social and Behavioural Sciences (pp. 514-525). Future Academy. https://doi.org/10.15405/epsbs.2019.11.61