Biology Problem-Solving: The High Achiever Students

Abstract

Problem-solving has been acknowledged as one of the compulsory skills needed to face and overcome challenges of the modern world in either learning or everyday life. However, there is limited information regarding the level of problem-solving skills demonstrated by school students in learning biology subject compared to physics and mathematics. This study aims to identify the problem-solving level of 16-year old high achievers in selected boarding school in the Southern and Central Regions of Malaysia. The problem-solving skills of 70 students were measured using a validated open-ended test, UKPM, which consists of general and topic-specific problem-solving questions for biology. These questions focus on the different steps in the problem-solving processes. High achievers from boarding schools were chosen to ensure the homogenous background of the participants. The data were descriptively analysed and the overall score was used to determine the students’ problem-solving level based on the classification in Programme of International Students Assessment (PISA). The result showed that the majority of the participants are low (35%) and intermediate (64%) problem solvers and they showed incompetence in manipulating information and making justifications. They possess high tendency to find the absolute answer, but lack the reflecting ability when answering the test. The criteria and limitations showed that the participants are prone to practise a converged thinking pattern. In this, educators should introduce innovative alternative teaching and learning approach need to enhance the students’ problem-solving skills.

Keywords: Problem-solving SkillsProblem-solving ProcessesBiologySchool

Introduction

Problem-solving (PS) skills refer to a person’s ability to make critical judgment and decision based on the appropriate justification of the problem’s situation and its surrounding (Kivunja, 2014). Solving a problem requires an individual to explore the root cause of a problem and create potential solutions pragmatically by using logic, lateral, and creative thinking (Ismail & Atan, 2011). This approach is parallel with the 21st-century learning that emphasised on the construction of new knowledge, a shift from focusing solely on rote memorisation and classroom knowledge transfer in schools that have become habitual over the years. Problem-solving (PS) PS is not an innate skill (Bal & Esen, 2016) therefore, providing the students with the chance to solve the problem is actually an effective way to develop this skill (Shute, Ventura, & Ke, 2015; Shute & Wang, 2013). Learning instruction that emphasises on the understanding of core concept helps in developing students’ PS as this skill is best learnt through the use of domain-specific problem-solving activities that are challenging for students to learn (Prevost & Lemons, 2016). During the process, increasing students’ understanding of the topic will help them to create and relate to the new knowledge. In the aspect of learning and education, the repetition cycle of the PS process through practices will equip them with PS skills that can be applied in different problems regardless of the context, discipline, or situation (Yang, 2012). In this light, PS skills can have long-term benefits and subsequently, help the students to take charge of their profession, personal encounter and everyday hurdles (Bal & Esen, 2016; Syafii & Yasin, 2013). Moreover, biologists agreed that students should acquire PS skills in order to learn biology better (Hoskinson, Caballero, & Knight, 2013).

The numerous proposed models on PS stipulate that the basic component of the PS process is to identify problems, to suggest solutions, to apply solutions, and to reflect at the end of the process. One may interpret that the problem-solving process is the sequential steps in a linear process. However, in reality, most individuals demonstrate flexible and inventive approaches based on the different circumstance and they do not adhere to an perpetual linear PS process (Yu, Fan, & Lin, 2014). PS skills are taught through the integration with the teaching of different subjects, fields, domains, or contents, the steps, process, or stages remained unchanged, therefore, understanding the meanings and function of each PS step is crucial for the success of problem-solving. PS step can be used as a guideline on what to observe and measure in evaluating the proficiency of PS. Besides that, teachers will know the types of learning support or scaffolding that have to be given to the students during the teaching and learning process.

PS requires a variety of mental skills, including interpreting information, planning, trying alternative strategies, reflecting and decision-making. However, studies have found that students are not aware of the processes taking place in problem solving (Yu et al., 2014). Early PS studies demonstrated that students have difficulties in PS steps, especially when tackling the orientation stage, which is to identify the problem. In this regard, the first step of PS is vital and students should be taught for a better understanding of this component. Hence, this component should be set as the rubric or benchmark during PS measurement and assessment. Studies done in both general and domain specific PS affirmed that the students’ incapability in solving a problem does not stemmed from their lack of knowledge or skills specific domain, rather, it is due to the failure of properly identifying the source of the problem and its details. This can be seen in PS studies that compared the differences between novice and expert problem solvers where PS performance is highly influenced by individual ability to understand the problem, as well as analysing the potential answer or solution (Prevost & Lemons, 2016). Nevertheless, enhancing students’ PS skills is one of the prior goals of all educational institutions therefore, developing PS skills is necessary in order to improve students’ ability in scientific thinking especially in science subjects, such as biology (Ulusoy, Turan, Tanriverdi, & Kolayis, 2012; Yenice, Ozden, & Evren, 2012). In other words, PS skills should be developed early as in the students’ schooling years.

Previous studies have shown that Malaysia students face difficulty in problem-solving (Abd Razak, Mohd Johar, Andriani, & Yong, 2014; Johnny, Abdullah, Abu, Mokhtar, & Atan, 2017; Kaus, Phang, Ali, Abu Samah, & Ismail, 2017). On the other hand, the term ‘problem-solving’ is commonly synonymous with obvious calculation and this resulted in the lack of study on problem-solving associated with Biology. In this regard, despite the differences in problems’ structure and contents between science subjects, the instructional purpose, which is to elucidate the patterns and processes in the natural world and systems, align comparatively with each other. Research noted that solving biology problem requires the engagement of the same skill practised by physicists and biologists (Hoskinson et al., 2013). Nevertheless, compared to mathematics or physics there is still a prominent gap in the research of PS skills in biology for the past three decades(Kim, Prevost, & Lemons, 2015).

Problem Statement

The implementation of PS in pedagogical activities has led to the measurement of PS skills among the students. Studies have shown that there is a significant positive relationship between academic achievement, career success, and certain habits of mind or behaviour with skills competencies (OECD, 2014; Stecher & Hamilton, 2014; Wüstenberg, Greiff, & Funke, 2012). In this light, it is more challenging to measure the competency level of PS skills compared interpersonal skills, therefore, there are guidelines that can be used in developing the instrument or selecting the rubrics to measure PS skills. On the other hand, these higher-order PS skills are arduous to be measured and the discrepancies on relevant and credible measurement scales are still debatable among the researchers (McCoy, Braun-Monegan, Bettesworth, & Tindal, 2015; Stecher & Hamilton, 2014). By observing and measuring these PS processes, this study will obtain valuable information related to the cognitive habit in ones’ mind when solving a task. Observing and measuring these processes during intervention study will provide formative information as well as evidence of the student’s development of PS skills. There is a lack of information about problem-solving skills for biology and how students solve biology problems, among school students is still in its formative stages.

Research Questions

The research questions that lead this study are as follows:

• What is the students’ problem-solving competency level for Biology?

• How is the students’ performance in non-routine Biology questions in terms of the problem-solving steps in problem-solving process?

Purpose of the Study

For the purposes of this article, domain-specific problem-solving refers to topic Cell Division of secondary school biology syllabus investigate the PS level of the students. Therefore, this study aims to identify the students’ abilities regarding the steps in PS as well as their problem-solving competency level for Biology.

Research Methods

This study was participated by 70 science stream students (39 females and 31 males) who are 16 years old from three high-achieving fully residential schools located in the Central and Southern Regions of Malaysia. The students were chosen to ensure the homogenous background of the participants. Their PS skills for the biology subject were measured using the UKPM, which is a validated open-ended test with 20 topic-specific questions in Section A and 20 general questions in Section B. The topic-specific PS questions are related to cell division, while the general PS questions are related to biology or science as well as questions adapted from the problem-solving domain in Programme for International Student Assessment (PISA). This study referred to the Ge & Land PS Model that comprises four problem-solving steps, which are identifying problems (PS1), giving suggestions and options to solve the problem (PS2), making justification (PS3), and reflecting the action (PS4) (Bixler, 2007). A total of 10 questions were allocated for each PS step and each question focuses on the different steps in the PS process. The maximum score for the UKPM is 120 and the data were analysed descriptively to identify the participants’ performance for each step in the PS process. The assessment rubric was adapted from previous research (Bixler, 2007) while the PS classification of the competency level was done by referring to the OECD or ‘Organisation for Economic Cooperation and Development’ classification that was used in PISA (OECD, 2014). The UKPM was validated prior to the research by experts in the PS domain, as well as against the biology syllabus for the Malaysian secondary school.

Findings

Table 2 summarises the participants’ scores. The results show that neither the female nor male participants score excellently in the UKPM test. The mean score for female participants was 42.92, and the difference is not distinct with the male participants with the mean score of 42.42. The overall achievement did not reach 50% of the overall score with only 42.70. The dispersion of the score patterns for all groups is almost similar, with the average of 8. The highest score is 62/120, while the lowest is 26/120. Both extreme scores were obtained by female participants.

Each individual score was compared and classified according to the six PS competency level as presented by Organisation for Economic Co-operation and Development (2014). Diagram 1 shows the percentage of the number of participants in each competency level; only 1% of the participants could be classified as possessing level 4 competencies. The majority of the participants (64%) could be categorised as level 3 problem solvers, while 35% could be classified as having level 2 competencies.

See Full Size >

The level of problem-skill for each gender was compared against and there are only minor differences. Diagram 2 shows that 2% of the female participants possess level 4 competence. Moreover, there is a 4% difference between the number of females (36%) and males (32%) who demonstrated level 2 competence. The same difference was observed in level 3 and there is only a 6% difference between both groups.

See Full Size >

Table 3 describes the score of each PS step in details. Out of the four steps in the PS process, making reflection (PS4) has the lowest mean score of 9.33 ± 3.90 for females and 9.39 ± 3.35 for males. The mean scores show that most participants are capable to obtain at least 10 out of the total 40 marks allocated for PS4 in UKPM. Although the other three steps have higher mean scores, they are still considered to be in the low range as none of the PS steps are able to reach at least 50% of the mean score compared to the allocated marks. The minimum and maximum scores for each PS step are in the lower range as the highest score is 21 (PS4) and the lowest score is 3 (PS2). Diagram 3 summarises the findings related to the PS steps. In this light, there are no major differences in the overall achievement each PS steps between each gender.

See Full Size >

The results provide the insights on students’ behaviour when solving problems during the biology subject. The biology subject is different from physics and mathematics; this is because, the calculations only play minimal roles compared to reading and understanding the fact. It was discovered that the participants from both genders have poor knowledge and capabilities in all the PS steps, and consequently, they obtained poor results in the PS domains based on the Programme International Student Assessment (PISA).

It was discovered that the participants did not plan well and did not evaluate the situation in the questions. In one of the PS1 questions, the participants were asked to list all the barriers and factors that they should consider before choosing the most appropriate option and only a small percentage of students managed to list the appropriate answers beyond the question given, while the rest only listed down a few factors that could be found in the question. In another PS1 question, the participants were required to propose an arrangement plan regarding the number of people to be placed in eight rooms. For this question, the participants should consider the criteria given when proposing the arrangement. The researcher expected the participants to perform some calculations, however, the majority of them presented wrong answers even though a draft table was provided to assist them in planning and evaluating the problem by providing specific directions for the key stages. They only provided answers that they are familiar despite the expectation that they would be able to find the solution when they delve deeper into the question. This shows excellent achievement in public school examination will not ensure good competency in non-routine PS as most school examination revolves around routine problem (Abd Razak et al., 2014)

In the meantime, the planning process is seldom practised in answering open-ended problems even though it is commonly used in routine and algorithmic problems. Hence, it should be considered in developing the skills to solve open-ended problems (Reid & Yang, 2002). In this light, most past PS studies only focused on the earlier PS steps, which are identifying the root cause of the problem and planning the solutions and actual success in PS is actually determined based on the capability to determine what is needed to be solved and how to do it effectively (Ulusoy et al., 2012).

Identifying the root cause of the problem and planning the solutions are categorised as knowledge acquisition according to the PS framework in PISA (OECD, 2014) or rule identification and application (Schweizer, Wüstenberg, & Greiff, 2013; Wüstenberg et al., 2012). It is assumed that these particular steps are more utilised in higher-order thinking skills (HOTS) compared to the later steps in PS model. Meanwhile, reflections and monitoring require judgement and deep thinking and reflections can be done in most of the PS steps. Due to the huge influence in PS stages, some studies have divided the ‘Identify Problem’ and ‘Solution Planning’ steps into smaller sub-steps (E.g., ‘Gather Info’). Some studies also added other indicators that are relative to these steps (E.g., ‘Avoiding Problem’ and ‘Flexibility’) with a specific checklist of criteria that has to be observed in the study. These additions were based on the researchers’ perspectives and research needs.

The results revealed that the students are confused and facing difficulties in linking the function of spindle fibre with mitosis or meiosis failure (Figure 4 ). The participants’ lack of understanding of the key concepts has contributed to the poor results for the PS steps. As an example, the question related to the concepts of meiosis and its functions in producing haploid gamete cell which are different in terms of numbers of chromosomes, genetic content due to random desegregation, and crossing over processes. In this light, the students were unable to answer the question even though it is just slightly different from the examination format questions (Figure 5 ). This shows that the students were confused and the students had come out with varied segments of inaccurate response and totally incorrect answers. On the other hand, the success rate was improved when the same question was modified with additional explicit hints or organised to be similar to the pattern of the examination format questions. Without this explicit linkage, the participants had difficulties in linking what they had learnt on the idea of cell division with examination questions. It seems that solving previous exam questions and drill practices are common in a biology lesson so that the students could ace their examination. In this light, despite their impressive results, the students tend to have limitations in terms of their level of thinking skills. These students tend to answer the questions through memorisation, and the drilling practices create a mind model and schema that will be stored in their minds, rather than creating understanding of the principles. In other words, the students memorise the content and they face difficulties when presented with questions in a new context or structure.

See Full Size >

See Full Size >

Step four in the PS process is making reflection. In UKPM, questions in PS4 prompt the participants to present their agreement on the topic (Figure 6 ). For example, the students are provided with a formula as a guide for their answers and to answer PS4 questions, students are expected to review and identify the formula and they have to suggest the correct formula when giving their justifications. Unfortunately, there were not many participants who were able to present a sound reflection. Most of them only provide their answers by referring to the given calculation without reflecting and they also provided incorrect answers. As a result, they scored very low for PS4 which affected their overall score. This shows that the learners have low abilities and face difficulties in creating a link between skills and knowledge (Reid & Yang, 2002). Moreover, this study also observed the habit and pattern related to how the participants answer the test. Besides that, at the end of the UKPM test, the researcher had obtained verbal feedbacks from students who seemed to not prefer lengthy questions as they only glanced through the instruction and provide answers without any description, explanation, or justification. It was found that these students are more familiar with routine questions, which only require one right answer and they looked uncomfortable when asked to answer non-routine abstract questions that require giving opinions and justifications and consequently, gave opinions that did not reflect the lesson that they had learnt.

See Full Size >

In the meantime, a good problem solver has three characteristics, which are having a good conceptual understanding of the domain involved, including domain-specific skills and being able to adjust wisely to the use of automated skills. This is because PS requires two types of knowledge namely declarative and procedural knowledge that are interdependent during the PS activity (Yu et al., 2014). Expert problem solvers are more mature when performing an integrated mental representation of the problem, as well as demonstrating a better understanding of the core concepts, nature, and form of the problem (Prevost & Lemons, 2016). They need more time to define and understand the issues compared to novice problem solvers who prefer to complete the task impatiently and often ignore the PS steps, including the first and most important step, which is problem identification (Yu et al., 2014). The same pattern could also be observed among primary, secondary, and college students. Novice problem solvers usually seek the solution without the definite understanding of the problem and they lacked the ability to reflect their own performances. They tend to overlook the analysis and reflection process, even though they knew that they were stuck with an inappropriate solution during the process.

The flexible, linear, and sequential PS processes can be practised differently according to the problem solvers’ creativity and needs, as well as the situation. However, the students’ lack of understanding on these PS processes will affect their perceptions on the processes’ progression and nature. Furthermore, research (Yu et al., 2014) reported a similar pattern in the early phase of their study. They found that students lacked the flexibility and creativity as they opted for the linear mode of an incomplete PS process. Therefore, it is important to incorporate effective teaching strategies to enhance the students' understanding on the meaning and function of each PS step, and the students can develop individual skills when solving problems. A good problem solver that practises effective monitoring step would consistently reflect the chosen strategy to ensure that they are on the right track, as well as checking for other solutions. The students should be able to monitor and steer the direction of their own progress, to ask questions among themselves that could help to maximise the effective strategies and to prevent themselves from constantly using the unproductive approach in generating solutions (Jamari, Mohamed, Abdullah, Mohd Zaid, & Aris, 2017b). Choosing an effective strategy without making revisions or having self-correcting mechanisms to monitor the progress of PS is comparable to those who fail to choose good method and strategy right from the start. This issue can contribute to the failure in PS. Therefore, students must be encouraged to make verification by monitoring and reflecting their choice that could increase their HOTS and PS level.

Previous studies suggested that instructional scaffolding is necessary in aiding the students’ problem-solving processes (Jamari, Mohd Zaid, Abdullah, Mohamed, & Aris, 2017a; Kim et al., 2015). In this light, it is important to focus on content specific scaffolding, which is also known as conceptual scaffolding in school. This is because mastering the content of the lesson is the ultimate goal of having learning assistance, either with or without instructional materials or the presence of a teacher. Therefore, the teacher is responsible to help students in understanding the function of each step involved in the PS in class regardless of subject or domain (Yenice et al., 2012). The action of mentioning these processes during the teaching and learning process without giving the students with the opportunity to perform activities that require them to think, learn and practise each step will not enhance the students’ PS skills (Yu et al., 2014). The similarities between ill-structured task and common everyday problems make it worthwhile to inculcate and develop the students’ PS skills. PS skills helps to cater the needs of solving multiple tasks in a short term which refers to schooling and learning and at the same time shaping an individual to be a capable problem solver later in life as a long term goal. Since an ill-structured task usually has complex structure and may have numerous potential solutions, this type of task requires more cognitive activity to process all the problems’ information in the attempt to find the best solution.

Promising instructional strategies to enhance HOTS and PS have been widely studied and including the inquiry learning approach and focused on STEM education (Jamari et al., 2017b). However, there are still not much studies being done on these approaches for the Malaysian context although there are plenty of studies done in other countries. Examples of teaching strategies that emphasised on authentic ill-structured problem that can be applied by teachers include case-based learning (CBL) and problem-based learning (PBL). The problem or task does not stem from the textbook, but from the everyday problems which require the application of the similar concepts or principles. CBL and PBL are categorised under inquiry and they are suitable to be used in the environment of science learning due to their potential to attract the interest of students, to spark inquiry, and to encourage them to continue exploring the task (Herreid, Schiller, & Herreid, 2012; Pai et al., 2010). Teacher’s face to face or online involvement provides the suitable guidance to help them interpret and accelerate active information transfer processes by providing a learning environment that can develop HOTS and PS skills (Kivunja, 2014; McCoy et al., 2015).

Conclusion

Although time is a factor that can affect the development of PS in an individual, it is important to expose the students to PS steps and processes so that they can learn and practise these skills to become a competent citizen. Therefore, teaching approaches and strategies that emphasise on authentic problem and active learning such as Inquiry Learning and STEM (Science, Technology, Engineering and Mathematics) education should be combined with appropriate instructional scaffolding that focuses on the students’ ability to master the lesson, as well as nurturing and developing their PS skills (Bybee, 2010; Moore, Johnson, Peters-Burton, & Guzey, 2016; Tseng, Chang, Lai, & Chen, 2013). In the meantime, this study has several limitations. One of the limitations is the small number of participants as the study was focused on high achievers. Hence, the sample for this study may not represent the whole population. Nevertheless, the sample provides insights on how high achieving students conduct PS. This information will add to the body of knowledge on problem solving in the context of Malaysian school students. It is assumed that other students are facing the same PS problems as shown by the high achievers, therefore, future research can be implemented on students from different categories and backgrounds. A PS research with more focus on the biology subject can be conducted by replicating this research to other topics in the biology syllabus..

References

1. Abd Razak, N. N. F., Mohd Johar, A., Andriani, D., & Yong, C. Y. (2014). Mathematical Problem Solving Ability Among Form Two Students. Jurnal Pendidikan Matematik, 2(2), 1-13.
2. Belgin Bal, İ., & Esen, E. (2016). Problem Solving Skills of Secondary School Students. China-USA Business Review, 15(6). doi:10.17265/1537-1514/2016.06.002
3. Bixler, B. A. (2007). The Effects of Scaffolding Students' Problem-Solving Pocess Via Question Prompts on Problem Solving and Intrinsic Motivation In An Online Learning Environment. (Doctor of Philosophy phd thesis), Pennsylvania State University,
4. Bybee, R. W. (2010). Advancing STEM Education A 2020 Vision. Technology And Engineering Teacher, 70(1), 30-35.
5. Herreid, C. F., Schiller, N. A., & Herreid, K. F. (2012). Science Stories: Using Case Studies to Teach Critical Thinking. VA, USA: National Science Teachers Association.
6. Hoskinson, A. M., Caballero, M. D., & Knight, J. K. (2013). How can we improve problem solving in undergraduate biology? Applying lessons from 30 years of physics education research. CBE Life Sci Educ, 12(2), 153-161. doi:10.1187/cbe.12-09-0149
7. Ismail, S., & Atan, A. (2011). Aplikasi Pendekatan Penyelesaian Masalah Dalam pengajaran Mata Pelajaran Teknikal dan Vokasional di Fakulti Pendidikan UTM. Journal of Educational Psychology and Counseling, 2, 113-144.
8. Jamari, D., Mohamed, H., Abdullah, Z., Mohd Zaid, N., & Aris, B. (2017b). Fostering Higher Order Thinking And Problem Solving Skills Through Social Media. Man In India, 97(12), 1-10.
9. Jamari, D., Mohd Zaid, N., Abdullah, Z., Mohamed, H., & Aris, B. (2017a). Instructional Scaffolding To Support Ill-Structured Problem Solving A Review. Sains Humanika, 9(1-4), 33-39.
10. Johnny, J., Abdullah, A. H., Abu, M. S., Mokhtar, M., & Atan, N. A. (2017). Difficulties In Reasoning Among High Achievers When Doing Problem Solving In Mathematics. Man In India, 97(12), 61-70.
11. Kaus, M. A., Phang, F. A., Ali, M. B., Abu Samah, N., & Ismail, A. K. (2017). Problem Solving And Social Supports: The Roles of Parents. Man In India, 97(12), 279-287.
12. Kim, H. S., Prevost, L., & Lemons, P. P. (2015). Students' usability evaluation of a Web-based tutorial program for college biology problem solving. Journal of Computer Assisted Learning, 31(4), 362-377. doi:10.1111/jcal.12102
13. Kivunja, C. (2014). Do You Want Your Students to Be Job-Ready with 21st Century Skills? Change Pedagogies: A Pedagogical Paradigm Shift from Vygotskyian Social Constructivism to Critical Thinking, Problem Solving and Siemens’ Digital Connectivism. International Journal of Higher Education, 3(3). doi:10.5430/ijhe.v3n3p81
14. McCoy, J. D., Braun-Monegan, J., Bettesworth, L., & Tindal, G. (2015). Do Scaffolded Supports between Aspects of Problem Solving Enhance Assessment Usability? Journal of Education and Practice, 6(36), 175-185.
15. Moore, T., Johnson, C. C., Peters-Burton, E. E., & Guzey, S. S. (2016). The need for a STEM road map: A framework for integrated STEM education. In (pp. 33 -12). NY: Routledge Taylor & Francis Froup.
16. OECD. (2014). PISA 2012 Results: Creative Problem Solving Students’ skills in tackling real-life problems Volume V. Retrieved from La rue André-Pascal, PARIS
17. Pai, A., Benning, T., Woods, N., McGinnis, G., Chu, J., Netherton, J., & Bauerle, C. (2010). The Effectiveness of a Case Study-Based First-Year Biology Class at a Black Women's College. Journal of College Science Teaching, 40(2), 32.
18. Prevost, L. B., & Lemons, P. P. (2016). Step by Step: Biology Undergraduates' Problem-Solving Procedures during Multiple-Choice Assessment. CBE Life Sci Educ, 15(4). doi:10.1187/cbe.15-12-0255
19. Reid, N., & Yang, M.-J. (2002). Open-ended problem solving in school chemistry: A preliminary investigation. International Journal of Science Education, 24(12), 1313-1332. doi:10.1080/09500690210163189
20. Schweizer, F., Wüstenberg, S., & Greiff, S. (2013). Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity. Learning and Individual Differences, 24, 42-52. doi:10.1016/j.lindif.2012.12.011
21. Shute, V. J., Ventura, M., & Ke, F. (2015). The power of play: The effects of Portal 2 and Lumosity on cognitive and noncognitive skills. Computers & Education, 80, 58-67. doi:10.1016/j.compedu.2014.08.013
22. Shute, V. J., & Wang, L. (2013). Measuring Peoblem Solving Skills In Portal 2. In IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013) (pp. 33-39). Fort Worth, Texas, USA: International Assn for Development of the Information Society (IADIS).
23. Stecher, B. M., & Hamilton, L. S. (2014). Measuring Hard-to-Measure Student Competencies A Research and Development Plan (13 978-0-8330-8806-2). Retrieved from Santa Monica, California:
24. Syafii, W., & Yasin, R. M. (2013). Problem Solving Skills and Learning Achievements through Problem-Based Module in teaching and learning Biology in High School. Asian Social Science, 9(12). doi:10.5539/ass.v9n12p220
25. Tseng, K. H., Chang, C. H., Lai, S. J., & Chen, W. P. (2013). Attitudes towards science, technology, engineering and mathematics (STEM) in a project-based learning (PjBL) environment. International Journal of Technology Design Education, 23, 87-102.
26. Ulusoy, Y. O., Turan, H., Tanriverdi, B., & Kolayis, H. (2012). Comparison of Perceived Problem Solving Skills of Trainee Students Graduated from Different. Procedia - Social and Behavioral Sciences, 46, 2099-2103. doi:10.1016/j.sbspro.2012.05.435
27. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving — More than reasoning? Intelligence, 40(1), 1-14. doi:10.1016/j.intell.2011.11.003
28. Yang, Y. T. C. (2012). Building virtual cities, inspiring intelligent citizens: Digital games for developing students’ problem solving and learning motivation. Computers & Education, 59(2), 365-377. doi:10.1016/j.compedu.2012.01.012
29. Yenice, N., Ozden, B., & Evren, B. (2012). Examining of Problem Solving Skills According to Different Variables for Science Teachers Candidates. Procedia - Social and Behavioral Sciences, 46, 3880-3884. doi:10.1016/j.sbspro.2012.06.165
30. Yu, K.-C., Fan, S.-C., & Lin, K.-Y. (2014). Enhancing Students’ Problem-Solving Skills through Context-Based Learning. International Journal of Science and Mathematics Education, 13(6), 1377-1401. doi:10.1007/s10763-014-9567-4