Methodology And Tools For Evaluating State Regional Development Programs

Abstract

The success of the modern national concept of socio-economic development and increasing the quality of life is determined by the efficiency of state administration bodies in the region. In the modern model of public administration, the main tool for managing the socio-economic development of the region is state programs. The experience of implementing the program-target approach in the practice of regional development management actualizes the problem of evaluating the efficiency of programs. In the context of the formulated problem, the authors suggest a methodology for evaluating state programs. The novelty of the author's approach to the evaluation of state programs lies in the integrated approach, which includes the evaluation of the program document as a tool, and the analysis of its efficiency. Evaluation of the program as a document is important from the point of view of improving the methodology of the program-target approach. As practice shows, the failure of many state programs is associated with the inconsistency of program documents with the methodology of the program-target approach. When evaluating the program as a document, special attention is paid to determining the financial sources of the program implementation. The article proposes a methodology aimed at analyzing extra-budgetary sources of funding. The novelty of the author's approach to assessing the effectiveness of state programs is to determine different levels of program results, depending on the stability of cause-and-effect relationships between program activities and the results achieved.

Keywords: Efficiencysocio-economic development of the regionstate programpublic administration

Introduction

The development and implementation of state programs of the socio-economic development of the territory is based on program-target and project approaches. The program and project-oriented strategy is the leading one in the modern state economic policy of Russia. The program-target approach in research is considered as a management technology aimed at achieving a specific result in solving a certain problem in a predetermined time frame, through the implementation of a system of measures (Malitskaya, 2016).

The project-oriented approach is the subject of management impact on the project. A project is understood as an innovation designed by the initiator, the purpose of which is to create, modernize, or maintain material or spiritual value in a changed environment, which has space-time and resource boundaries (Lukov, 2016). The project approach defines the subject of management as the processes of activity limited in time and resources for solving tasks local in time and determined by the subject of management. The resources available set their time limit to the manager.

The role and place of the program-target approach in the management of socio-economic development of the region is clearly defined and obvious. Within the framework of the program-target approach, targets, results, indicators of results achievement, volumes and sources of resources necessary for the implementation of program targets are determined. Projects are part of the program and reflect specific actions, activities, to implement the program's targets, the product of which are quite specific material or information objects and services. Based on this conclusion about the logical and structural balance between the program and the project, the article will focus on the program-target approach, which includes the project approach.

The place of program-target and project approaches in the modern model of public administration is considered in connection with the search and justification of tools for attracting extra-budgetary investments in the industry of the regions (Jia et al., 2020), in connection with a comprehensive solution to the issues of activating internal sources of socio-economic development of the territory. In particular, modern research focuses on assessing the impact of the activity of residents of the region, subjective value orientations of economic entities on the involvement of resources in projects and programs for the development of territories (Mityagin et al., 2017). In this connection, the problem of developing effective tools for assessing the role of projects in the implementation of state programs in terms of achieving planned results and efficient use of resources is being updated. The international practice of implementing the program-target approach in public administration is represented in a significant number of studies, among which there are works dedicated to evaluating the effectiveness of programs and projects in the field of social development (Rolfe, 2019), the development of socio-economic infrastructure, transport (Mavraki et al., 2020). A separate role is assigned to program-target and project approaches in the management of innovation processes in the regions (Falck et al., 2019).

For all the importance of target-oriented and project-based approaches, the researchers note limitations for their use in public administration. Pischik (2015) believes that increase of a project-oriented approach in the management of social systems is the main reason for the dominance of the ideology of consumption. In addition, human society is too complex system for designing compared to technical objects. Kleiner (2019) reads that the excess of project strategies in the management of economic systems, for all their importance in terms of the development of the economic system, overcoming stagnation, leads to overheating of the economy. We add to this that the project approach in managing public systems is applicable to solving local problems with the least number of participants, operations and limited time intervals. According to Pischik (2015), the project-oriented approach, which has been used in engineering for thousands of years and is transferred to "non-engineering areas", has a right to exist if society is considered as a "megamachine". Then the entire existing and future methodology and technology of project management is justified, and with them the expanding "world of projects". Thus, in addition to evaluating the efficiency of programs implementation, there is a need to evaluate the program as a document from the point of view of the feasibility of using a program-target approach to solve a specific problem

Modern approaches to evaluating the efficiency of programs and projects in foreign countries were formulated as a result of the activities of the USA city institute (Rohacek & Isaacs, 2016) and received the general name "outcome - focused performance measurement systems", which were used to evaluate the implementation of programs and activities of government agencies. Also, the specialists of the city Institute have developed methods for attracting customers, beneficiaries of the products and results of the project to the procedures for evaluating its effectiveness.

Problem Statement

Full-scale implementation of the program-target principle of organizing the activities of executive authorities and, accordingly, program budgets at all levels of management began in 2012 and was associated with the announced in June 2011. Budget message of the President of the Russian Federation to the Federal Assembly of the Russian Federation, according to which the main state programs were approved, which is reflected in the Budget Code of the Russian Federation. In accordance with the federal state programs, each region has developed state programs of the subject of the Russian Federation and municipal programs. Federal Law No. 172-FZ "On strategic planning in the Russian Federation" the state program got the status of a strategic document as a tool for implementing the strategy of socio-economic development. Since 2018, state programs, as a document reflecting the target settings and resources for their implementation, have become filled with federal projects, structured by content and resources into national projects. Thus, from a methodological point of view, there was a combination of program-target and project approaches. This has given the state programs greater consistency and coherence of the targets, results and projects of the programs. At the same time, the inclusion of national and federal projects in state programs has highlighted the problem of developing a new system for evaluating their efficiency.

In the modern practice of state authorities, the program-target approach is one of the leading management technologies, so the efficiency of target programs determines the quality of the entire public administration system. In this regard, the current problem of managing state and municipal programs is the adequacy of the system for evaluating the results obtained as a result of the project implementation. This problem is particularly important, both from the point of view of the methodology of the program-target approach aimed at efficiency, and in the context of evaluating the effectiveness of spending state budget funds on the implementation of project activities.

For each state program of the Russian Federation, the state program of the subject of the Russian Federation, an annual assessment of the effectiveness of its implementation is carried out. The procedure for conducting this assessment and its criteria are established respectively by the government of the Russian Federation, the supreme executive body of state power of the subject of the Russian Federation. But at the same time, official documents do not contain the entire list of requirements for programs that allow them to be evaluated from the point of view of implementing all the principles of the program-target approach, which calls into question the rationality and expediency of using this technology.

Research Questions

The study of the problem of the lack of adequate evaluation of state programs from the point of view of the program-target approach assumed answers to the following questions.

1.What are the current methodological approaches to the development of a system for evaluating state programs in the world?

2.How should the system of evaluation of state programs reflect the prevailing concepts and models of public administration in the Russian Federation?

3.What are the requirements for state programs presented in state regulatory documents?

4.What methodological guidelines of the program-target and project approaches are fundamental for the development of programs?

5.To what extent do modern state programs meet the requirements of the program-target approach?

6.What are the shortcomings of state programs in terms of implementing the principles of the program-target approach?

Purpose of the Study

The purpose of the study is to develop a methodology for evaluating state programs in accordance with the requirements of the program-target and project approaches.

1.The methodology should meet the needs of public authorities in improving the efficiency of state management of socio-economic development.

2.The methodology should include indicators that evaluate the program as a document in the context of implementing the principles of program-target and project approaches.

3.The developed system of indicators should include indicators that assess the volume and sources of resources attracted for the implementation of planned activities.

4.The methodology should include a tool that assesses the feasibility of solving the problem of socio-economic development of the region by program methods.

5.Performance indicators of the program should not go beyond the powers of public authorities in managing the socio – economic development of the region.

6.The process of developing and implementing government programs should be evaluated from the perspective of all stakeholders, including local communities.

Research Methods

The system of evaluation of state programs of socio-economic development of the region proposed in the article is developed on the basis of system, process approaches, participatory management principles and methodology of program-target and project approaches.

1. A systematic approach involves the development of a system for evaluating state programs, including the subjects of evaluation, evaluation indicators, and criteria values of indicators, methods and tools for evaluation. Activities, results, and responses are defined as the subject of evaluation.

2. Process approach. Based on the process approach, the attached methodology for evaluating state programs provides for a clear separation of activities and results as subjects of evaluation. Activity as a subject of evaluation characterizes the program development process and the program as a document (Karlina & Ustina, 2018). Evaluation of the development process allows to prevent deviations of the final result from the planned goals, guide the process in the necessary direction, ensure the sustainability of the results of the process and the reproduction of program activities. Performance indicators can be expressed in numerical values that characterize the course of the process itself and its costs: time, financial, resource, human. In an integrated form, this indicator reflects the number, cost and quality of work of civil servants spent on working with the program. In general, the evaluation of the program as a document makes it possible to determine the feasibility of using the program-target approach as a tool for managing socio-economic development, as well as compliance with the methodology of the program-target and project approaches.

The system of evaluation of program performance proposed in this article was developed based on the methodology Hartree, who proposed a model of the system results, consisting of four levels, the logical sequence of which involves climbing from the results obtained by direct consumers to the results affecting the whole society (Hartree, 2005).

Level 1- results that reflect the level of solving specific problems related to meeting the needs of the population.

Level 2 - results that reflect the creation of mechanisms for solving the problem and give sustainability to the results of the program after its funding is completed.

Level 3 - results that reflect the potential for development of the population.

Level 4 - results that reflect the mission of public authorities, the ultimate goal, and the area of their responsibility.

3. Participatory principles of public administration involve the development of mechanisms for public participation in the implementation of public administration. In the system of evaluation of state programs, participatory principles are reflected in the assessment of the role of the population and all stakeholders in the development of the program and the formation of responses to its results.

Responses as a subject of evaluation are the reaction of the environment to the action on the work of state bodies during the implementation of programs. Response indicators are numerical values that characterize the response of all stakeholders to the products and results of the process of developing and implementing government programs.

4. Methodology of program-target and project management. The basic principles of this methodological approach are:

The principle of expediency of developing a program to solve the problem of socio-economic development. The practice of developing and implementing state programs in the Russian Federation has shown that the most effective software tool works in the following cases:

- to solve a socially significant problem in full, the authorities do not have enough powers and resources;

- the problem cannot be solved by using market mechanisms;

- solving the problem requires the introduction of innovative organizational, managerial and technological solutions;

- the implementation of the program requires the establishment of horizontal cross-functional links between authorities and organizational structures.

If the problem can be successfully solved within the resources and powers of a separate authority, and does not require the involvement of other sources of funding, the use of a program-oriented approach is impractical. The development of the program, the organization of its implementation and evaluation require additional serious organizational resources, which are justified if the problem cannot be solved within the framework of a standard functional system of public administration.

The principle of resources that are planned and determined in accordance with the goals and activities to achieve them. Given the limited budgetary resources of state authorities, this principle can be implemented only if extra-budgetary sources of funding for the program are attracted.

The principle of project management. The implementation of the program requires the construction of a special management system based on project teams and working groups, the main task of which is to maintain horizontal cross-functional interaction of authorities.

Findings

In accordance with the above methodological grounds, a system for evaluating state programs has been developed. To assess the process of developing the program as a document, a system of indicators has been developed in accordance with the sections of the program passport, which is presented in Table 1 .

To assess the effectiveness of the program, a system of indicators has been developed, put into 5 groups, depending on the level of results achieved. The 1st level of results is the results of one cycle of the process: the number of developed and implemented state programs, the number of implemented project activities. In accordance with the participatory principles of state program management, an indicator is added - the number of social projects and initiatives of the population included in state programs.

The 2nd level of results characterizes the effectiveness of financial and participatory mechanisms for mobilizing the capabilities of local communities. Financial performance indicators of the program are evaluated by measuring the ratio of budgetary and extra-budgetary resources attracted for the implementation of the program. The volume of activated opportunities of local communities as a numerical indicator is calculated by adding up the financial, material, personnel, and organizational capabilities of social projects. This indicator has both an absolute value as an assessment of the degree of strengthening of the budget to address local issues, and in relation to total budget expenditures as an indicator of the initiative activity of citizens and their role in the management of state programs.

The 3rd level of results describes the creation during the development and implementation of potential programs for sustainability of program results for example the number of professionals, increased the qualification, the number of volunteers involved in the activities of the project; the number of newly established non-profit organizations and other institutions.

The 4th level of results is the results associated with the capitalization of the city's population, achieved as a result of the implementation of the program's projects. For example, the number of citizens engaged in physical education and sports; the number of small entrepreneurs, the level of employment of the population, the level of wages.

The 5th level of results is the final results determined by the primary results and repeated cycles of the program development and implementation process, which cause qualitative changes in the environment and ensure the achievement of strategic indicators of socio-economic development. For example, this group may include indicators such as life expectancy, birth rate, and level of satisfaction with the quality of life.

In order to draw a conclusion about the degree of their connection with the activities of the program, quantitative methods are not enough. Here it is possible to use the method of expert assessments, the use of indicators of the level of satisfaction with the quality of life of the population, etc. In this case, the degree of influence of the program's activities on the achievement of final results that affect the quality of life indicators should be subject to expert evaluation.

To develop a program evaluation system, the following conditions must be taken into account:

1.The initial situation should be described with all possible specific indicators, arranged in a hierarchy from immediate to final.

2.For each indicator, depending on the degree of connection with the activities of the program, a different tool for evaluating performance is developed.

3.The selected indicators should reflect the positive dynamics that have emerged as a result of the project. You can track the dynamics with the help of special sample studies; tracking the internal documentation of the organization or relevant statistics of state bodies; attracting independent experts.

To assess responses to the state program, a system of indicators has been developed that reflect the responses of the following stakeholders from various stakeholders. Stakeholders interested in the project development and implementation process include:

- internal stakeholders: responsible executors are co-executors and participants of the process that shape the response to the development process of the program implementation from the point of view of rationality of the organization of this process, optimal time and organizational costs of participants in the process;

- external stakeholders: developers and participants in the implementation of social projects included in state programs, whose responses are aimed both at evaluating the process of developing and implementing the program, and at the results achieved;

- beneficiaries of the program's products and results - groups of the population directly targeted by the program's project activities;

- society as a whole.

There are several levels of responses that depend on the degree of activity of subjects in relation to the program. The first level is passive responses that reflect the emotional state, namely the level of satisfaction with the process and its results. The second level of responses characterizes the desire /unwillingness in relation to the temporal and spatial expansion of the scope of the program's activities. The third level of feedback reflects the degree of willingness of stakeholders to participate in the multiplication of the results of the program with their own resources.

Conclusion

The developed methodology has received practical testing in the evaluation of 20 regional state and municipal programs in the areas of health care and improvement and housing and communal services. As a result of the study conducted using this methodology, the following conclusions were made: In 85% of state programs, there is no or poorly justified need to solve the problem by program methods. In more than 70% of state programs, there is no justification for the implementation tools of the program. In more than 85% of state programs, extra-budgetary sources of project financing are absent or poorly justified. Among the indicators of state programs, the indicators of the first group, related to the solution of specific needs of the population and which are directly related to the activities of the programs, prevail. Significantly, less attention is paid to the indicators of the fourth group, which reflect changes in the quality of life of the population. The indicators of the third group, which should show the mechanisms of sustainability of programs in the future, are poorly developed and relate mainly to professional development of specialists. All indicators of achievement are uniformly evaluated, regardless of their level. At the same time, the indicators of the third and fourth levels are only indirect consequences of the program and depend on the action of many factors, sometimes not related to the program. Participation of the population and public groups in the evaluation of the program is organized only by publishing the program and the report on its implementation on the official website of the authority. There are no channels for receiving reverse connection. In general, it suggests that the state programs implemented in practice do not meet the requirements of the program-target approach and modern principles of public administration in many respects. Adjusting programs in accordance with the methodology proposed in the article will increase their efficiency. Restrictions in the use of the proposed methodology for evaluating state programs are due to the fact that public authorities in their practice of developing and implementing programs are guided by the norms of legislation. Although the requirements for programs in the proposed methodology do not contradict the normative provisions, but its use in practice requires additional time and organizational resources of the authorities.

Experience in the development and implementation of state programs in the Russian Federation from 2010 to 2019 showed that the lack of attention of developers to the principles of program-target and project approaches, the lack of mechanisms for the participation of the population and public groups, the emphasis on financing activities only from their budget sources significantly reduce the results of the program implementation and its effectiveness. Together with the above, this actualizes the need to introduce the methodology of program evaluation attached in the article into the practice of public administration.

References

1. Budget code of the Russian Federation of 31.07.1998 N 145-FZ. Retrieved from: http://base.garant.ru/12112604/#ixzz6fm2W41v8. Accessed: 21.11.2020.
2. Budget message of the President of the Russian Federation to the Federal Assembly of 29.06.2011 " On budget policy in 2012-2014". http://www.consultant.ru/document/cons_doc_LAW_115713/
3. Falck, O., Koenen, J., & Lohse, T. (2019). Evaluating a place-based innovation policy: Evidence from the innovative regional growth cores program in East Germany. Regional Science and Urban Economics, 79, 103480.
4. Federal law No. 172-FZ of 28.06.2014 "On strategic planning in the Russian Federation". http://base.garant.ru/70684666/#ixzz6fm2spruj
5. Hartree, G. P. (2005). Monitoring performance in the public sector. Institute of City Economics Foundation.
6. Jia, J., Ma, G., Qin, C., & Wang, L. (2020). Place-based policies, state-led industrialisation, and regional development: Evidence from China's great western development programme. European Economic Review, 123, 103398.
7. Karlina, A. A., & Ustina, N. A. (2018). Evaluating the effectiveness and efficiency of municipal programs as a strategic planning tool. Bulletin of the Samara Municipal Institute of Management, 2, 7-16.
8. Kleiner, G. B. (2019) The economics of ecosystems: Step into the future. Economic Revival of Russia, 1(59), 40-45.
9. Lukov, V. A. (2016). Social engineering. Flint.
10. Malitskaya, E. A. (2016) Program-target approach in the management of integrated territory development projects. Bulletin of the Institute of Economics of RAS, 2, 56-68.
11. Mavraki, С., Arabatzis, G., Kantartzis, A., & Malesios, C. (2020). Fostering regional development in eastern Macedonia and Thrace, Greece, through road transport projects. Economic Analysis and Policy, 65, 56-67.
12. Mityagin, S. A., Tikhonova, O. B., & Repkin, A. I. (2017). Methodology of estimation of achieving regional goals of sustainable development on the basis of program and goal oriented approach. Procedia Computer Science, 108, 2038-2048.
13. Pischik, A. M. (2015). New architectonics of society. Wschodnioeuropejskie Czasopismo Naukowe, 1(1), 67-70.
14. Rohacek, М. H., & Isaacs, J. B. (2016). PFS + ECE: Outcomes measurement and pricing. https://www.urban.org/research/publication/pfs-ece-outcomes-measurement-and-pricing
15. Rolfe, J. (2019). Simple economic frameworks to evaluate public investments in sporting events in regional Australia. Economic Analysis and Policy, 63, 35-43.