Findings Of The National Study For The Preschool Education Quality In Russia

The problem of preschool education quality assessment in Russia

The Federal Law of Education in Russia, which was adopted in December 2012, and the Federal State Educational Standard for Preschools, which was legalised since September 2014, provide the idea of preschool education quality within three aspects: educational outcomes, the structure of the program and conditions for the program implementation. But children’s outcomes assessment is prohibited for program management purpose. The reason is flexibility of early childhood and unstable frames of age orientations (Zaporozhets, 1978). There are common practices in Russia which use the results of program structure assessment or parental satisfaction as the objective basis for quality management. But this approach doesn’t give the information about process quality which is about 40% of the quality concept, as Tietze discovered (2003). So, there is another – more fruitful, as we consider – way to assess preschool education quality. It is an educational environment evaluation. It helps to find out what kinds of opportunities each child and the whole preschool group get for emotional comfort, health and safety, learning and socialization.

ECERS-R as the main tool for ECE quality evaluation in Russia

The “Early Childhood Environment Rating Scale - revised edition” (ECERS-R) (Harms et al., 2005) was proved as an appropriate instrument for quality assessment in Russia. In 2014-2015 there was an approbation which discovered the distinctive ability of ECERS-R in preschools with objectively high and low quality (Shiyan et al., 2016). ECERS-R became the main tool in the study in Moscow which highlighted the advantages and risk areas of the common approach to preschool education (Remorenko et al., 2017).

Sample

423 preschools from 40 regions of Russia took part in the study in 2016; and in 2017 – 1301 institutions from 74 regions (approximately 2.6% of all Russian preschools, 87% of territories). 367 preschools participated in both years. The selection of preschools was executed by regional education ministries and expert teams under the project guidelines. 1/3 of the sample was composed of preschools that appeared to be municipal rating leaders (this cluster was nominally named “Best”). The remaining 2/3 of the sample consisted of randomly selected preschools (nominally categorized as “Random”).

Procedure

An observation of educational environment was performed by 155 trained experts. In the “newcomer” preschools one randomly selected class was observed. In the preschools which participated for the second time an expert observed the same class that in 2016. In both cases “children-adults” ratio was additionally examined through the observation. Also, sociometric and professional data about the staff was collected with the help of a questionnaire, such as the teacher’s age, professional experience, the type of the basic professional training program, the involvement in different in-service training programs during the last 3 years, the involvement in collaboration for constructing the curriculum, the age of children the teacher works with, the teacher’s awareness of evaluation criteria and the results of the study in 2016.

The 2017 score of preschools that participated in the study for the first time (Group 1) and for the second time (Group 2) was compared within “Best” and “Random” clusters. It also was compared with the 2016 scores in both clusters (in 2016 all participants belonged to Group 1 – “newcomers”).

The dynamics of each second-time participant was tracking. The preschools which had more than 0,18 score gap were considered to have dynamics, equal or less than 0,18 – no dynamics. This score gap was established according to the precision of comparison which was calculated based on the probable error of expert work that equalled 5%. That amounted to 0.35 from possible 7 scores, i.e. +/- 0.175 scores two-way and rounded to 0.18.

Ethical Considerations

The research was carried out in accordance with Code of Ethics of the Russian Psychological Society.

Comparison within and between clusters

The 2017 mean of general quality index for education environment amounted to the following: “Best” – 4.05 which is 0.17 points higher than in 2016 (sd = 1.78 with the confidence interval of <X>min=4.00, <X>max=4.09 (p ≤ 0.01)), and “Random”– 3.74 which is 0.37 points higher than in 2016 (sd = 1.68 with the confidence interval of <X>min=3.72, <X>max=3.76 (p≤ 0.01)).

According to the study findings presented in table 01 , in cluster “Best” the difference between means of Group 1 and 2 amounted to 0.41 in the favour of the latter. Meanwhile, in “Random” it equalled 0.19, also in the favour of Group 2. In both clusters the means of “newcomers” (Group 1) rose in 2017 if we compare with 2016. But the gap in cluster “Random” is wider than in cluster “Best” – 0.51 and 0.45 in accordance. The mean of cluster “Best” in 2016 equalled the mean of second-year participants of cluster “Random” in 2017.

According to the figures in table 02 , in both clusters the ranges of confidence intervals of Group 1 and 2 for 2017 did not overlap which suggests these groups are different. But the confidence interval of “Best” newcomers (2017 Group 1) partly overlaps with the confidence interval of “Random” second-time participants (2017 Group 2), that discovers possible similarity of these groups.

As table 02 displays, the range of confidence interval of Group 1 for “Random” in 2016 does not overlap with the range of Group 1 in 2017 (both Groups are preschools that participated for the first time). But the confidence interval of Group 1 for “Best” in 2016 partly overlaps with the confidence interval of Group 1 for the same cluster in 2017, that indicates possible similarity of these groups.

The hypotheses about the similarity of the Group 1 in 2016 and in 2017 within clusters was tested with Welch Two Sample t-test for predictive confidence intervals. The results are presented in table 03 . The amount of data is sufficient for the cluster “Random” to consider that the differences between Group 1 in 2016 and Group 1 in 2017 are statistically significant. As for the cluster “Best” the amount of data is not sufficient for rejecting the 0-hypotheses about the similarity of these groups.

As the data in table 04 presents, in all subscales there are significant differences in the results of 2016 (“Best” & “Random”, Group 1) and 2017 (“Best” & “Random”, Group 2) with p ≤ 0.01.

The worse score belongs to the subscale “Activities” in both years. In 2016 the mean was on the inadequate quality level, in 2017 it rose up to the minimum level and still is the worse among all subscales. The best score belongs to the subscale “Interaction”. In 2017 it rose up close to the good quality level.

Figure 1 shows the quality profiles of clusters “Best” and “Random” in 2017. As we see, there are some common strong features and deficiencies for both clusters. The strong points are “Furniture for routine care, play and learning”, “Greeting/departing”, “Staff-child interaction”, “Interaction among children”, “Provisions for parents”, “Staff interaction and cooperation”, “Opportunities for professional growth”. Among deficiencies there are “Nap/rest”, “Blocks”, “Music/movement”, “Use TV, video, and/or computers”, “Promoting acceptance of diversity”, “Provisions for personal needs of staff”.

See Full Size >

There is one more common particularity of these two clusters. The analysis of standard deviation (table 05 ) indicates that there are the same items with high deviation, such as “Toileting/diapering”, “Provisions for children with disabilities”, “Provisions for professional needs of staff”. And these are the same items with low deviation, such as “Nap/rest”, “Provisions for personal needs of staff”, “Dramatic play”.

“Adult-child” ratio

The “adult / child” ratio indicates that 2 or more adults are simultaneously engaged with children in the cluster “Best” in 41.2% of the total time of educational activity, in the cluster “Random” – in 41.8%. It makes sense to pay attention to this fact because the number of children in preschool groups is rather high: in the cluster “Best” average mean for children which present during an observation is 17.7, in “Random” – 16.5. But the maximum number a centre allows in a group is average 28.1 in the cluster “Best” and 26.6 in the cluster “Random”. Although 59.5% of preschools have more than 25 children in a group, 34.4% have 30-39 children, and 3% - 40 children or more (even up to 51 in “Random” and 55 in “Best”).

But the correlation analysis indicates no significant correlations between the general quality index and the involvement of 2 or more adults: in the cluster “Best” R=0.072, in the cluster “Random” R=0.052.

Dynamics of the 2016 and the 2017 scores in each preschool

56.60% of the preschools improved their general quality index with more than 0.18 points. The index of 23.40% of preschools did not change, and in 20% it decreased. Some items were identified as the most often improved, such as “Math/numbers”, “Discipline”, “Child-related display”, “Art” and so on. There were some items which changed very seldom, such us “Provisions for children with disabilities”, “Provisions for professional needs of staff”, “Promoting acceptance of diversity”, “Provisions for personal needs of staff”, “Supervision and evaluation of staff”, “Nap/rest”, “Indoor space” and so on.

The correlation analysis of factors which might affect the dynamics of the quality indicates that there are no significant correlations with the teacher’s age, professional experience, the type of the basic professional training program, the involvement in different in-service training programs, the involvement in collaboration for constructing the centre curriculum, the age of children the teacher works with and even the teacher’s awareness of evaluation criteria. But rather stable correlations between positive dynamics of the majority of subscales and the intervention with teachers were discovered (Rmin=0.189; Rmax=0.315). As we can see on figure 02 , if the teacher was involved in discussion of the study results on workshops (CEM) it correlates with the dynamics for the subscales “Space and furnishings” (D_ECERS_1), “Language-Reasoning” (D_ECERS_3), “Activities” (D_ECERS_4) and “Program structure” (D_ECERS_6).

See Full Size >

The intervention with teachers differs much. In some preschools the teachers whose classes were observed in 2016 didn’t know the evaluation scores. In some preschools the staff took part in workshops after the study but not the teacher who participated in the assessment. In some preschools the teacher took part in one workshop where the evaluation scores were discussed. And in the rest, there were several workshops with discussion of evaluation scores in which the teacher took part.

Thus, the dynamics of preschool groups in which the teacher took part in discussion of 2016 evaluation scores is better than in groups with minimum intervention or without it. E.g. the subscale “Activity”: no intervention – dynamics equalled -0.4 – 0.23, intervention without the teacher – dynamics equalled -0.27 – 0.47, intervention with score discussion once – dynamics equalled 0.31 – 0.83, intervention with score discussion several times – dynamics equalled 0.38 – 0.82.

Comparison within and between clusters

Preschools from both clusters that participated for the second time performed in 2017 better than the “newcomers” which provides the assumption about positive influence of such participation itself on ECE quality. The cluster “Random” increased the mean of general quality index for education environment more than the cluster “Best”. It indicates that the leaders of local ratings are less open to new experience or that they are already close to the maximum level of quality they can get in current situation.

The 2016 confidence intervals range for preschools in the cluster “Random” did not overlap with the 2017 range for the “newcomers” in the same cluster. It supports the assumption of a tendency towards improvement of preschool education quality irrespective of participation in the study.

The 2017 ranges of confidence intervals of “newcomers” and second-time participants did not overlap which allows regarding these groups as different

An analysis of the results on subscales discovers a general trend towards improving the quality of preschool education.

An analysis of the results on items in comparison of the two clusters presents some similarity of the quality profiles.

On the basis of the data obtained, it can be assumed that in the system of preschool education there is a general tendency to improve the quality of the educational environment, to make the social situation of development more child-oriented and personalized.

The fact of participation in the study proved to be a factor which stimulates the improvement of the quality. It means that the low scores on items are useful for the development of quality and, presumably, influence the formation of reflectiveness of the teachers.

A group of items with a low standard deviation indicates that the preschools in the sample turned out to be little different from each other. In this case, it means either the established norms of professional consciousness (the item “Provisions for children with disabilities”, etc.), or regulatory documents that do not allow the environment to be more flexible and child-oriented (e.g. “Nap/rest”). The study makes available for the authorities and principals the important data that allows them to carry out appropriate strategies in different areas of improving the educational environment.

“Adult-child” ratio

“Adult-child” ratio is not a significant factor which affects the quality of ECE in general, but it needs to be studied in every type of educational activities. It presumably can give different correlations.

Dynamics of the 2016 and the 2017 scores in each preschool

The teachers’ awareness of the assessment outcomes plays a significant role for positive dynamics, while the number of workshops did not influence it.