Efficiency Assessment Of Scientific Organizations In The Region

Abstract

The paper presents the efficiency assessment of regional scientific organizations based on quantitative indices of their activity. The authors made an attempt to connect human and resource potentials of scientific activity, current performance, as well as to consider regional development priorities. The need for complex assessment of the activity of scientific organizations, including regional development objectives, is caused by the requirements of the science management system to increase the contribution to social and economic development of the region. The relation of science monitoring with regional development will make it possible to coordinate research directions thus creating the research agenda relevant for the region. To create the reasonable assessment system, the study defines the requirements to indicators and highlights the levels of assessment of scientific activity: region-scientific organization-division-scientist; and forms the structure of performance assessment by aggregation levels. The study was based on 30 assessment indicators, their standard values, scaling methods and aggregation in the uniform assessment system were defined. The aggregation of lower assessment indicators is made through additive and multiplicative convolution using the importance coefficients of each indicator. The importance coefficients of indicators are determined via the pair-wise comparison of indicators according to their contribution to the improvement of the higher indicator. The suggested methodical approach will allow performing the system assessment of efficiency of regional scientific organizations by authorities and other stakeholders.

Keywords: Efficiency assessmentscientific organizationregional development

Introduction

The relevance of the assessment system of scientific organizations are connected, inter alia, with numerous attempts to increase the efficiency of scientific institutes. (Androsova, & Fadeyev, 2014; Grinstein, 2013; Dosuzheva, & Lyamzin, 2012; Larin, 2015) A clear statement on the R&D potential of any scientific organization and the result achieved by the scientific organization using this scientific potential forms the basis for the assessment.

The specified potential is formed during a long period of time and includes research staff, equipment and areas for research, cooperation agreements, management systems, including organizational structure with divisions dealing with commercialization of obtained results. The bigger accumulated scientific potential shall result in great scientific success.

Problem Statement

The performance result of a scientific organization is expressed by a set of recognized scientific achievements, including training of research staff, financial performance of R&D results and their contribution to the real sector of economy, publishing activity, patents, certificates and fulfillment of target indicators within state research programs.

The following requirements shall be taken into account within the assessment system (Pronichkin, & Tikhonov, 2013; Sinitsyn, Nikiforov, & Andreyev, 2014):

1) Integrity of the system. Coherence with performance assessment of a scientist, a division of scientific organization and other estimates.

2) Measurability of assessment result. The system shall ensure the numerical value of performance assessment of scientific organizations.

3) Unambiguity of performance assessment of scientific organizations.

4) Integrated approach to performance assessment of scientific organizations.

5) Assistance in development of scientific organizations (focus not on the control over scientific organizations but on the assessment of development and management dynamics of scientific organizations).

6) Objectivity of assessment. Comprehensive accounting of factors influencing the performance of scientific organizations.

The scientific organization acts as one of the key elements of the system of scientific activity. The activity of scientific organizations is interrelated; however, certain research fellows or research groups create the majority of performance results.

Research Questions

The management of regional scientific institutes includes a variety of issues: choice of priority research fields relevant to the global research agenda, search of the best organizational forms, methods and tools to implement the research projects and others. However, within the given study the authors focused on the assessment system of scientific organizations.

The complexity of assessment is determined by the levels of scientific potential of the region as the assembly of potentials of scientific organizations, certain divisions and scientists. Considering the above, there are the following levels of scientific performance assessment in the region (Sharova, & Kotov, 2009):

• level of the region in general;

• level of scientific organizations;

• level of laboratories, sectors and other divisions of a scientific organization;

• level of a scientist.

Each level of assessment is characterized by a set of performance indicators. The higher level of assessment includes the indicators of the lower level and adds general indicators to the assessment, which can only be formed at this level.

The scientific result created by certain scientists shall be considered in the sum of results of a division within the scientific organization, as well as in the scientific organization as such. The joint work aimed at scientific result shall indicate its accessory thus accurately defining the part of result to be considered in a certain organization. This will avoid double accounting.

Purpose of the Study

The purpose of the study on performance efficiency of scientific organizations is to form the basis for management decisions based on the following:

• enhancing the efficiency of using scientific and technical potential;

• increasing the contribution to the Russian and world science and ensuring priority of regional scientific organizations in R&D areas;

• increasing the prestige of science in the society and its influence on public opinion;

• optimizing and structuring the network of scientific organizations to develop a complete and effective system of scientific organizations;

• developing the infrastructure of scientific, research and technical activity and ensuring its efficient use;

• developing scientific organizations and improving the quality of their management;

• improving the development and approval of state tasks and development programs of scientific organizations, including for effective use of budgetary funds.

In terms of the efficiency of scientific organizations in the region, the suggested methodical approach defines indicators and the performance assessment system of scientific organizations performing research and R&D having commercial value approved by the Decree of the Government of the Russian Federation No. 312 of 8 April 2009 (Resolution of the Government, 2009; Resolution of the Government, 2013).

Research Methods

According to the scientific approach to performance assessment of scientific organizations, two groups of indicators are suggested as the assessment principles and requirements to the system of indicators. The first – to assess the potential of a scientific organization (Table 01 ) and the second – to assess the achieved result (Table 02 ).

The corresponding symbols given in both tables are introduced to analyze the indicators. For potential indicators generalizing the assessment fields these include the following:

P_is – staff potential of i -assessed organization as of January 01 of the year under assessment;

P_ir – resource potential of i -assessed organization as of January 01 of the year under assessment.

For simplicity the symbol “ i ” defining the code of the corresponding organization is not further specified. For indicators generalizing the results of scientific organization assessment the following symbols are used:

R_ir – research result of i -assessed organization of the year under assessment;

R_ip – publishing result of i -assessed organization of the year under assessment;

R_is – staff result of i -assessed organization of the year under assessment;

R_if – result of attraction of financial resources or financial result of i -assessed organization of the year under assessment.

For simplicity the symbol “ i ” defining the code of the corresponding organization is not further specified.

The accumulation system of indicators’ based on the division theory of a set of alternatives into several predetermined ordered groups (classes of solutions or indicators) is developed. The main objective of the convolution is to reduce the size of the attribute space to simplify the solution of problems related to assessment and comparison. The study proposes the approach to comparison and classification of scientific organizations in terms of their properties expressed by numerical values of suggested indicators, where the large number of initial indicators is consistently aggregated in a small number of criteria having small rating scales. The procedure of features convolution uses various methods of verbal analysis of solutions and has a block character thus reducing the labor input for decision procedures and the possibility of explaining the obtained result. This also allows using the convolution system for its implementation in information systems (Ivanov & Khripunova, 2012; Koroleva, Vasilyev, & Torzhkov, 2014).

Figure 02 shows the scheme of criteria and rating scales. The key indicator (zero level) allowing assessing the efficiency of a scientific organization – Eso is a dimensionless value measured in points. The calculation is based on P and R values characterizing the potential of a scientific organization at the beginning of the reporting year and the result of its performance over the reporting year (first level). The convolution of indicators (according to assessment fields) of the second level is made for the calculation of the first level indicators, which are calculated through the convolution of initial indicators of the third level.

See Full Size >

As a result of additive convolution five values of the second level indicators were obtained. The idea of this method is that the generalized criterion looks as follows:

$$f\left(X\right)={\sum }_{i=\mathrm{1}}^m {\lambda }_i{F}_i\left(X\right)$$

where λ_i ≥ 0 – weight coefficients that set the preference of i-criterion in comparison with other criteria. λi defines the importance of i -private criterion. At the same time, bigger weight is attributed to more important criterion, and the general importance of all criteria is accepted equal 1.

The reliability of the calculated value for each indicator of the second level is reached by the introduction reasonable weight coefficients λ_i. The method of pair-wise comparison is recognized as sufficient and admissible to define the weight coefficients for the assessment of indicators in the course of aggregation. At the same time, some private indicators of the third level in each assessment field are compared among themselves. The obtained relations make sense >, ~, <, i.e. it is more important (equals 2), is equivalent (equals 1) or is less important (equals 0). These relations are identified based on the analysis of performance assessment of scientific institutions taking into account the assessment trends of scientific efficiency determined by the development of science and technology in the Russian Federation.

The indicators having various dimension require aggregation for their comparison. For this purpose, the scales of comparability measured in points were introduced for each indicator. Considering the need for aggregation of indicators the scales were selected for the first and second level according to the order of numerical values of indicators of personnel potential of a scientific organization. Table 03 shows an example of the scale of transfer of resource potential indicators to uniform size.

The total point is determined for each indicator of assessment by assigning points for the corresponding numerical value of initial indicators. Further calculations consider the values of each indicator in points. The calculated values shall be rounded to thousands for scaling. The weight coefficients are defined for additive convolution of indicators and calculation of P_r (indicator of the second level). Indicators of research result of the scientific organization are measured in units. One research work (RW) is recognized as a result of the scientific organization and is considered in the assessment in case the obtained result is used in the current governmental activity of the Republic of Bashkortostan. (Kotov, 2016; Kotov et al, 2016))

The second level of indicators represents two massifs, one of which has three indicators, and the other only two. As a result of convolution of two massifs two values of indicators of the first level shall be obtained. The method of multiplicative convolution is justified at this level. The approach to the multiplicative method is similar, only the criterion function looks as follows:

$f\left(X\right)={\prod }_{k=\mathrm{1}}^K f_k^{{\sigma }_i}\left(X\right)$, where ${\sum }_{k=\mathrm{1}}^K {\alpha }_k=\mathrm{1},\mathrm{}{\alpha }_k\ge \mathrm{0}$.

This method will allow receiving the indicators of potential and result of a scientific organization comparable in size.

It is necessary to prove the choice of α_k coefficients for this method. The method of pair-wise comparison reduces the reliability of quantitative assessment at a small number of compared indicators. However, the arising mistakes can be considered insignificant in the unified method for further comparison of scientific organizations.

The potential of a scientific organization characterizes its possibility to achieve certain indicators in the future. The value and duration of staff potential development of the scientific organization is more complex. At the same time, the accumulation of resource potential can take 10 times less. The contribution to the achievement of a scientific result is first of all caused by the activity of a researcher, but the existence of special equipment, materials, software products is still important. The coefficients for convolution (degrees) can be accepted as 0.7 – for staff potential of a scientific organization and 0.3 – for resource potential of a scientific organization (Varshavsky, 2015).

The formula for the calculation of the potential P (П) of a scientific organization is as follows:

$P=*{\left(P_s\right)}^{\mathrm{0,7}}*{\left(P_r\right)}^{\mathrm{0,3}}$.

The performance assessment of a scientific organization is carried out through the calculation of an indicator – Eso (zero level). The indicator is defined by the comparison of the first level indicators. The following formula is used to assess the result of using the potential of a scientific organization:

$E_{so}=\frac{R}{P}·100$.

The obtained value, in items, characterizes the efficiency of a scientific organization. The bigger value shows higher efficiency and vice versa, smaller – the smaller.

Findings

To make the conclusion on sufficient or insufficient efficiency of a scientific organization there is a need to set the criterial (standard) values of such indicators as ΔЕso, ΔP, ΔR, etc.

Within the pilot study it is suggested to accept the standard values for Еso only, considering the assessment of variable indicators within data from performance reports of scientific organizations for 2017, Table 04 .

Conclusion

The study reflects the results on the system of complex performance assessment of scientific organizations in the region that makes it possible to monitor scientific organizations, analyze the structure of resulting indicators and reveal the surpassing and lagging behind indicators. Unlike earlier proposed assessment systems, the indicators considering the importance of scientific activity in social and economic development of the region were suggested for the first time.

References

1. Androsova, S. I., Fadeyev, D. A. (2014). Methodical approaches to the assessment of cost efficiency of scientific organizations. Scientific and technical materials of the St. Petersburg State Polytechnic University. Economic sciences, 1 (187), 126-132.
2. Dosuzheva, E. E., Lyamzin, O. L. (2012). Methods of assessment of efficiency of research and production integration of organizations in the 21st century. Modern problems of science and education, 3, 247.
3. Grinstein, V. E. (2013). Analysis of existing techniques on efficiency assessment of scientific organizations. Materials of the All-Russian scientific and practical conference. Legal protection, economy and management of intellectual property, (pp. 40-43).
4. Ivanov, P. E., Khripunova, A. S. (2012). Assessment of effective management of innovative activity of research organizations. University bulletin (State University of Management), No. 16, 140-150.
5. Koroleva, T. S., Vasilyev, I. A., Torzhkov, I. O. (2014). Criteria for efficiency evaluation of activity of scientific institutions. Works of the St. Petersburg Research Institute of Forestry, 2, 94-111.
6. Kotov, D. V. (2016). The system of monitoring and control of social and economic development strategy of the Republic of Bashkortostan until 2030. Strategic development of the Republic of Bashkortostan in the 2015-2030s of the 21st century (pp. 165-170). Ufa.
7. Kotov, D. V., Gamilova, D. A., Burenina, I. V., Kovshov, V. A., Lavrenyuk, N. (2016). Formation of priority directions of social and economic development of the Republic of Bashkortostan. Strategic development of the Republic of Bashkortostan in the 2015-2030s of the 21st century (pp. 74-101). Ufa.
8. Larin, S. N. (2015). Scientific and technical programs: approaches to the organization of monitoring and assessment of efficiency. National interests: priorities and safety, 11, No. 8 (293), 24-32.
9. Pronichkin, S. V., Tikhonov, I. P. (2013). Development of criteria and methodical approaches to expert assessment of efficiency of scientific organizations. National interests: priorities and safety, Vol. 9, No. 37 (226), 13-18.
10. Resolution of the Government of the Russian Federation. (2009). Russian Federation Code, No. 15, Article 1841. Retrieved from No. 312 of 8 April 2009
11. Resolution of the Government of the Russian Federation. (2013). Russian Federation Code, No. 45, Article 5815.
12. Sinitsyn, A. A., Nikiforov, O. Yu., Andreyev, M. A. (2014). Concept and structure of information and analytical system of analysis of staff printing activity within scientific and educational organization. Fundamental research, No. 11-6, 1276-1280.
13. Sharova, O. O., Kotov, D. V. (2009). New mechanisms of development and management of innovative activity of non-profit organizations. Innovations and investments, No. 4, 69-73.
14. Varshavsky, A. E. (2015). Main problems of effectiveness and efficiency of scientific organizations. Concepts, 1 (33), 3-8.