Approaches To The Implementation Of The Out-Comes Control At The Digital University

Abstract

The purpose of the study is to investigate how to apply the world-leading experience of knowledge testing to the development of an independent examination center in Plekhanov Russian University of Economics. In the article the authors consider the task of development of the information system inter-departmental (independent) examination (Certification) center of the University, which opens new opportunities for the use of the forms and methods of evaluation of educational achievements of students corresponding to the tendencies of modern information society development. The principal advantage of Prometric is much broader list of the vendors (more than 80) providing the examinations, however there is a number of tests which can be handed over only in VUE (such as Corel, Macromedia, Linux Professional Institute, etc.). Information system of IEC PRUE support monitoring of information base and examination processes from the side of divisions. In addition to statistical methods of data processing attraction the Data Mining methods are using. Information system IEC PRUE has developed analytical part with great variety of the following opportunities. The task has an innovative character in the field of higher education. The result is a knowledge base. This knowledge base can be considered as intellectual property of the University.

Keywords: Distant form of educationfunctional modelinterfaculty independent examination centerknowledge baseNeural Information Systemtraditional forms of education

Introduction

An improvement of work of the interfaculty examination center is a planned action in the current area of work of improvement of quality of teaching and rating positions of Plekhanov Russian University of Economics in national and foreign educational space by means of expansion of range of opportunities for estimation of knowledge, improvement of examination procedures, giving them bigger objectivity and increasing trust in results of the current, midterm and intermediate knowledge control.

The aim of this study was to investigate how to apply the experience of leading world systems of knowledge testing in the development of an independent examination center in Plekhanov Russian University of Economics. Further improvements of the interfaculty examination center are undertaken in a planned manner and as a part of overall improvements of teaching quality and advancement of Plekhanov Russian University of Economics in international and domestic ratings. The examination center improvements include broadening options for knowledge assessment, improvement of examination procedures, with aim for greater objectivity, and increase of results reliability of current, midterm and intermediate tests.

The Certification Materials (CM) database represents one or several linked repositories (with the possibility of replications) that consists of units – the minimal semantically-completed and logically-isolated CMs: a question, a task, an essay subject, etc. A unit can be provided to the trainee in a form of text, document, “picture”, html-page, sound file, script, etc. For convenience, during the preparation of test problems, it is a possible to filter CM units by meta data. Meta data is used for classification and filtering by subject domains, departments, authors, dates of development or updating, and when exporting CM into the LMS systems

The received results remove uncertainty from many questions and provide several alternatives of the solution of the denoted problem, which is modern in the sphere of the university education.

Problem Statement

Now in the Center of development of electronic teaching of Plekhanov Russian University of Economics colleagues conducted work on creation of independent tools of a scoring of knowledge of students based on testing in the 1C environment. However, testing as an examination method, is suitable for intermediate control, but not for final examination of discipline at which we have to make the complex evaluation of mastering all volume of theoretical knowledge, skills, and techniques, and also decision-making and their rationalization (Gerasimova, Jenova, Melamud, Romanova, & Sorokina, 2018). Especially it concerns disciplines of information cycle where it is very important not just to solve a problem, but also to choose the correct tool (Gerasimova & Sorokina, 2016; Dyakonova & Romanova, 2014) and technology for its decision.

Research Questions

The main questions of the research are next:

How to adapt the existing testing system for the purposes of final control of students' competences?

The possibility of implementation in the information system solution of information problems?

Purpose of the Study

The goal of this study was to investigate how to apply the experience of leading world systems of testing of knowledge in the development of an independent examination center in Plekhanov Russian University of Economics

Research Methods

In this article we consider an experience of holding certified examinations in two leading international examination centers Pearson VUE and Thomson Prometric (Warner, 2017; About Prometric, 2017, Computer-Based Test, 2017) for the following implementation of relevant ways in work of the interfaculty (independent) examination center of university (IEC PRUE).

Findings

Services, which are provided by both testing organizations for candidates, are absolutely equal. The principal advantage of Prometric is much broader list of the vendors (more than 80) providing the examinations, however there is a number of tests which can be handed over only in VUE (such as Corel, Macromedia, Linux Professional Institute, etc.). In total, there are now more than 7000 test centers of Prometric and VUE in 140 countries of the world, and their number is permanently growing. The worldwide network of the authorized centers provides to a candidate an opportunity to pass test practically in any country. Testing is available for any interested person without any limits like age, the place of residence, education level or the degree of training.

Today the international system of certification functions as follows:

• The vendor company (Microsoft, Novell, Cisco, etc.) defines professional requirements in the form of tests for their own program of certification;

• Prometric and VUE provide preparation and delivery of personal tests in to the authorized centers of testing to candidates, execution of tests on the computer and reception of results of testing immediately after it is finished;

• The vendor company sends the certificates to the candidates who successfully passed test, enters them in the databases and provide them support.

Examinations (tests) are differentiated in several types (Muzychkin & Romanova, 2015):

• The main exams are conducted for obtaining the status according to the existing program of certification, everyone can hand take them.

• Transitional examinations suitable for the experts who already have a certain certification when updating programs of certification.

• Regular examinations for confirmation of certification – either new exams, or special exams for confirmation of knowledge and experience.

• Trial (beta-) examinations are offered to the certified experts for verification of new tests. These examinations contain more questions. Unlike other types of examinations, the result of testing is reported only after the company processes all results and will choose passing score.

By the principle of construction, tests can be adaptive and not adaptive:

• Not adaptive test consists of the sequence of questions, which get out of base at random. Questions are untied among themselves. The candidate has an opportunity to come back to the previous questions and to change his answers at any time.

• In an adaptive examination, on the other hand, each following question depends on the answer to previous. If the candidate has answered correctly, his next question is more difficult and if his answer was wrong, difficulty of the next question is lower

In addition, examinations can be open and closed. During an open examination, candidates are allowed to use dictionary, handbook, reference book, manual, guide, etc. At a closed examination, it is forbidden to use any materials. All tests have English-language versions, most of them in addition are available in the main European languages: German, French, Spanish, Italian and also Russian.

The procedure of passing an examination is identical in all centers of testing. It consists of the following stages: registration of the written application for passing of the test; invoicing and payment by the center of testing; ordering of the test in PROMETRIC or VUE which is formed and sent by communication channels in encrypted form and is launched by specially allocated secured computer. After finishing of the test, the personalized report is formed. This report is the only document that attests passing of the test and it is automatically transferred to the producer. If the candidate has passed the necessary number of tests, the manufacturing company sends him the certificate by mail confirming his qualification (Gerasimova, Romanova, & Zhenova, 2018).

According to available to authors information (Gladilina, Polovova, Sergeeva, Antipova, & Umanetc, 2016; Bruskin, 2016), we have considered an order of preparation and holding examination, a technique of formation of tests (exams), and also the final reporting of both service providers. We have compared these procedures with the existing functionality of IEC PRUE, and also offered new approaches to promotion of this service in information space of university, with the following local goals (Zhenova, 2014; Kovaltchuk, Dedusenko, Blinova, & Miloradov, 2016):

• an increase of objectivity of examinations and tests;

• creation of equal conditions of examination for all participants;

• providing the full and authentic accounting of results of examinations;

• minimization of “a human factor” at all stages of examination, including reduction of subjectivity in rating of level of preparedness of students;

• increasing of transparency and openness of results of examination procedures;

• strengthening of the quality control of teaching disciplines by means of the analysis of results of tests and examinations;

• creation of conditions for development of the new models of an quality assessment of education based on new standard and legal, technological and organizational bases;

• formation of the individual rating of teachers on the base of a share of students who have successfully passed tests and got certificate;

• expansion of electronic ways of control of knowledge, along with traditional forms of examinations;

• increasing of randomization of contents of examination cards;

• possibility of development of examination cards and tests, equivalent on difficulty;

• creation of bigger independence of examination procedures from a subjective factor due to use of uniform base of questions;

• creation of conditions for formation of the examination tasks based on use of innovative methods of representation of control materials: text, numerical, graphic, audio, video, simulation (results of computer modeling) and interaction of the examinee with the testing system;

• speeding of processing results of examinations in Atomized Information System (AIS) of management of educational process;

• development of recommendations about increasing integration of AIS with the Mark and Rating System (MRS) of an academic performance of students and providing to administration full and reliable information about the individual ratings of students;

• an exception of arithmetic mistakes at recalculation of points of BRS;

• simplification of carrying out audit of results of examinations and tests due to usage of electronic resources of the examination center;

• granting technical capabilities for carrying out the analysis of results, including usage of modern analytical methods.

Regulations: silence and video filming in audience; placement of computers by the principle “I don't see someone else's screen”; it is forbidden to use gadgets, etc., special safe for storage of personal belongings; consumables of the center; it is forbidden to take out the sheet of paper on which the examinee made notes during the examination.

During the preparation for examination a short movie with possible types of questions and their quantity is demonstrated. It is made for convenience of examined.

Unlike the considered international certified centers in IEC PRUE we form our own certification materials (CM) independently, and we will consider for it the “to be” model in which we will define elements of “road map”.

Only authorized user of relevant department represents the database of questions. He makes preparation of CM to each discipline which need to held an examination according to curriculum.

Possibilities of holding examinations and offsets are in the following forms.

1) Traditional - written or oral examination with the tickets approved at department with rating a result by the teacher (commission) and automated inputting of the results of an examination in to the AIS.

2) Electronic monitoring of knowledge on a computer by the given scenario with different types of the tasks displayed the monitor. Methods to input the answers can be different (a choice; coupling; calculating; filling forms of answers with numerical and text information; recognition of graphic objects; an audio recording, etc.), including usage of innovative interactive forms. Results are automatically inputted as well.

3) Complex monitoring – a combination of written, oral and computer forms of checking of level of knowledge of trainees. We consider this like the most perspective form of monitoring.

The technique of preparation of tickets based on the database of questions and developed scenarios allows to organize certification for any discipline of the curriculum interactively – with usage of student's barcodes on an entrance and the digital signature of student.

For automated formation of examination cards, the database of questions and tasks with the greatest possible abundance and frequent updating is used. Algorithms of formation of tickets must be intellectual with randomization elements. Otherwise, the electronic database of tickets will appear very quickly on social networks.

The CM database represents one or several connected repositories (with the support of replications), consisting of units - the minimum semantically finished and logically isolated CM units: a question, a task, an essay subject and so on. A unit can be provided to the trainee in a form of text, document, “picture”, html-page, sound file, script, etc. For convenience during forming of tasks, it is a possible to filter CM units by meta data.

Meta data is used for classification and filtration by subject domains, departments, authors, terms of development and updating, and when exporting CM into the LMS systems (Gerasimova, Melamud, Tutaeva, Romanova, & Zhenova, 2018) supporting the SCORM standard.

In the special master of formation of tickets we specify: the discipline, the module, the group, the algorithm of formation of tickets and other metadata, like difficulty of tickets, duration of exam, an algorithm of calculation of points depending on number of the correct and wrong answers, etc. Scenarios remain in special “library” for multiple use. It is allowed to connect a script for an opportunity to do any algorithms of formation of tickets. However, the main mode has to be the mode in which programming is not required.

Except search by metadata, the full text search has to be supported. Blocks of CM have to have life cycle. They are created, verified by experts, used, updated, temporarily blocked, logically removed and physically destroyed.

Security is supported by enciphering the data in a repository with usage infrastructure of open keys (PKI – Public Key Infrastructure) of our university and permanent logging of all operations with blocks.

The file of ticket can be printed, copied, sent by mail or shown on the computer screen in the in interactive mode of examination. For the interactive mode of the examination the LMS system, a test cover or a special operating mode is used.

It is important to check (control) examination works by several subjects: 1) the teacher of a subject, 2) experts, 3) the group consisting of the teacher and experts, 4) completely automated way.

Digital form of storage of responses let us organize distant cooperation of the authorized teachers and experts, which is important when you work with far branches.

Information system of IEC PRUE support monitoring of information base and examination processes from the side of divisions (departments, deans, departments of educational and methodical management, departments of control and audit, etc.). In addition to statistical methods of data processing attraction the Data Mining methods are using. The modern techniques of Data Mining allow to estimate quality of teaching, quality of preparation of teaching staff and quality of functioning of educational process in University in general (Romanova, Muzychkin, Koval, & Lesnichaya, 2018).

The final report about results of examination in IEC PRUE satisfy requirement of the main international approaches. Nevertheless, it also has its own features of rules of information processing: personalized reports by student and by group, reports by requests of departments, unregimented reports that are displayed on a screen and also in the popular formats docx, PDF, xlsx.

Information system IEC PRUE has developed analytical part with great variety of the following opportunities (Nedelkin, Titov, Tikhomirova, & Romanova, 2016): viewing and printing of the list of disciplines of department according to faculties, semester or modules; creation of the modular timetable of examinations of department; formation of reports of results of examinations according to faculties, student's groups, teachers; formation of summary reports by disciplines, teachers, faculties, topics, competences; summary reports by the criteria which is voluntary set in the interactive mode; openness for creation of other procedural and not procedural reports dictated by logic of the developing system; developed possibilities of export of information to Excel, Word, XML, etc., for future analytical processing, etc.

Conclusion

According to the tendency of developing individual ways of training, especially in distant education it is important to provide a possibility of formation of individual examination cards, examination records, etc.

The received results remove uncertainty from many questions and provide several alternatives of the solution of the denoted problem, which is modern in the sphere of the digital university education.

One of the results is a knowledge base. This knowledge base can be considered as intellectual property of the University.

In conclusion we found much in common between examination procedures of the international centers of testing and IEC PRUE. However, an implementation of them in work of the University, which has to satisfy additional requirement necessitate to make an introduction of innovative approaches.

References

1. About Prometric (2017). The Prometric Advantage. Retrieved August 15, 2018 from: http://www.prometric.com/en-us/Pages/index.html.
2. Bruskin, S. (2016). Methods and tools of advanced business analytics for corporate information analytical systems in the digital transformation era. CEUR Workshop Proceedings, 1761, 449-454.
3. Computer-Based Test (2017). Development and Delivery: Pearson VUE. Retrieved August 17, 2018 from: http://home.pearsonvue.com/.
4. Dyakonova, L. P., & Romanova, U. D. (2014). Tasks of teaching of professional disciplines in higher education with the aim of integration Russia into the international educational space. Scientific Works of the Free Economic Society of Russia, vol. 186, 553-558.
5. Gerasimova V. G., Jenova N. A., Melamud M. R., Romanova U. D., & Sorokina M.U. (2018). Davis Method to assess the attractiveness of implementing e-learning in the Plekhanov Russian University of economics. Transport business in Russia, 4 (137), 102-105.
6. Gerasimova, V. G., & Sorokina, M. Y. (2016). Some issues in training in e-learning environment. Slavic Forum, 1(11), 61-67.
7. Gerasimova, V.G., Melamud, M.R., Tutaeva, D.R., Romanova, Y.D., & Zhenova, N.A. (2018). The Adoption of E-Learning Technology at the Faculty of Distance Learning of Plekhanov Russian University of Economics. Journal of Social Studies Education Research, 9(2), 172-188. Retrieved August 11, 2018 from: http://www.jsser.org/article/viewFile/5000215686/5000184010.
8. Gerasimova, V.G., Romanova, Y.D., & Zhenova, N.A. (2018). Russian market of LMS for higher education. Astra Salvensis, 6, 757-767.
9. Gladilina, I.P., Polovova, T.A., Sergeeva, S.A., Antipova, J.O., & Umanetc, D.O. (2016). Professional competences of specialists in purchasing as a factor of purchasing activity quality. Journal of Internet Banking and Commerce, 21, 007.
10. Kovaltchuk, A., Dedusenko, E., Blinova, E., & Miloradov, K. (2016). Concept and Procedures of Crisis Management in Russian Hotel Enterprises. Journal of Environmental Management and Tourism, 7(3), 473-480.
11. Muzychkin, P A., & Romanova, U. D. (2015). Information system of the examination center of Russian Plekhanov University of Economics: setting goals. Scientific Bulletin of PRUE, 1(7), 135-178.
12. Nedelkin, A. A., Titov, V. A., Tikhomirova, E. I., & Romanova, Y. D. (2016). The processing's automation of digital documents for hypertext scientific library. ARPN Journal of Engineering and Applied Sciences, 11(7), 4681-4684.
13. Romanova, U. D., Muzychkin P. A., Koval P. E., & Lesnichaya I. G. (2018). Control of knowledgebased on digital technologies. Transport business in Russia, 4 (137), 58-60.
14. Warner, T. L. (2017). VUE and Prometric: What's the Difference? Retrieved May 25, 2017 from: http://www.informit.com/blogs/blog.aspx?uk=VUE-and-Prometric-Whats-the-Difference.
15. Zhenova, N. A. (2014). Integral indicators of the effectiveness of investment projects. In Information Society: state, problems, perspectives 2014 Conference (pp. 49-54). Russia: PRUE.