Independent Effectiveness Study

The Mathseeds Effect: An Independent Effectiveness Study of Mathseeds on Kindergarteners (2016)

Purpose and Background

During the 2015-2016 school year, SEG Measurement and MarketingWorks conducted a study of the effectiveness of Mathseeds use with students in kindergarten. The primary goal of this study was to determine the extent to which students who used Mathseeds learn more math skills than comparable students who did not use Mathseeds.

The study investigated the following question: Do students in classes that use Mathseeds show more improvement in math skills than comparable students who do not use Mathseeds?

About Mathseeds

Mathseeds is an online math instructional product developed by Blake eLearning that teaches kids aged 3-8 the core math and problem solving skills needed to be successful at school with fun, highly interactive and rewarding lessons. Mathseeds combines highly structured lessons with fun motivational elements that keep children engaged and keen to learn.

The key features of Mathseeds are:

• The program features a wide variety of lessons and activities that have been created by educational experts. Each lesson is carefully structured to build early mathematical skills.
• Once a lesson is complete, it can be repeated as many times as needed.
• Mathseeds provides parents with a simple “dashboard” to show each child’s mathematical progress.
• Children can progress at their own level.
• Each child can create their own avatar/online character.
• Golden Acorns rewards can be used to add items to the child’s online treehouse or avatar.
• Children love collecting the Pets – a new pet hatches at the end of every lesson.
• Children receive a certificate after successfully completing the quiz at the end of every map.

The Study

Design

Both quantitative and qualitative methods were used to understand the impact of Mathseeds use on student learning. The research team used a quasi-experimental study comparing students in kindergarten classes using Mathseeds (Treatment Group) and students in comparable classes without access to Mathseeds (Control Group).

In August and September of 2015, all of the participating students took a pretest of math skills. In May of 2016, all of the students took a posttest of math skills and were evaluated by their teachers using an observational assessment that covered number sense, operations, measurement, and geometry. Posttest performance on the math assessment was compared between the Treatment and Control Groups while controlling for any differences in initial math ability at the beginning of the school year. Teachers using Mathseeds were surveyed throughout the study to collect information regarding their use of Mathseeds and their perceptions of its effectiveness.

Data Collection

Student mathematics skills were measured using the Stanford Achievement Test Series published by Pearson. Specifically, kindergarteners took the SESAT 1 Mathematics assessment for both the pre and post measure. In August/September 2015, we conducted the pretest. Teachers in the Treatment and Control Group classes were provided with the pretests and instruction for administration. Teachers were asked to administer the pretests during a class period and to encourage the students to do their best. The teachers were provided with materials and instructions to return the materials to SEG Measurement for processing. SEG Measurement entered and validated the student assessment data collected from the pretest. During the same timeframe as the pretest, the teachers completed an online survey prior to starting their instruction.

In May 2016, the posttest were conducted. The same procedures were followed for the posttest as for the pretest administration. In addition to the administration of the Stanford Assessments, teachers were asked to complete an observational assessment of each student. The observational assessment asked the teachers to evaluate their students’ skills in number sense, operations, measurement, and geometry. Each of those domains included 1–3 grade level-specific skills that were rated on a scale of 1–5, with 1 indicating that the student was struggling with the skill and a 5 indicating that the student was performing well beyond the level of the skill. After the conclusion of their math instruction, the teachers in the Treatment Group completed an online post survey regarding their instruction, their use of Mathseeds, and their evaluation of the effectiveness of Mathseeds.

Fidelity of use of Mathseeds was monitored through usage data collected in the program.

Participants

A total of 18 teachers and over 300 students across six states participated in at least part of this study of Mathseeds effectiveness. Seven classes used Mathseeds (Treatment Group) and eleven classes did not use Mathseeds (Control Group). Students who took both the pretest in the Fall and the posttests in the Spring were included in the study. Classes who used Mathseeds with an average of at least 14 completed Mathseeds per student were included in the analyses as part of the Treatment Group. Classes that were assigned to the Control Group and reported not using Mathseeds were included in the Control Group for analyses.

Three hundred seventeen kindergarten students had pre and post data and met the qualifications for their respective Treatment or Control Group. The Treatment Group consisted of 125 students and the Control Group consisted of 192 students.

The Results

Quantitative Results

When comparing outcomes, it is critical to establish initial comparability of the groups being compared. The starting math ability (as measured by the Stanford 10 SESAT 1 in August/September 2015) of the Treatment (Mean=455) and Control (Mean=453) classes was confirmed to be comparable (F=.088, p=.767). These means were similar to the publisher’s reported mathematics mean score for kindergarteners in the fall (Mean=460).

Kindergarten students using Mathseeds showed greater improvement in a composite of mathematics skills than the comparable group of students who did not use Mathseeds, when comparing student pretest performance to the end-of-year assessment of student math skills.

The comparison was first conducted for a composite of the SESAT posttest score and all the math domains measured in the observational assessment using MANCOVA. There was a statistically significant difference found between Treatment and Control (Pillai’s trace, F=13.02, p<.01). Further examination of the individual domains found that the difference in Treatment and Control Group performance was found in every score included in the composite.

“

Students in kindergarten who used Mathseeds outperformed the comparable classes that did not use Mathseeds on every measure. ”

The effect sizes for these differences found in performance between the Treatment Group and Control Group range from .42 to .66 standard deviations and the associated percentile gains range from 16 to 25.

Qualitative Results

During October 2015, 10 Kindergarten Treatment Group teachers participating in the Mathseeds efficacy research were individually interviewed by telephone about their use of Mathseeds and their opinions about it. In November, these same teachers completed an online survey about their use of Mathseeds; and in May 2016 they completed a concluding survey about their yearlong experiences.

Usage The Treatment teachers reported consistency in their usage of Mathseeds across the entire school year in the online surveys they completed in November and May. Eight teachers reported using the program either once (4) or twice a week (4), with two reporting usage more than three times a week. Most (7 out of 10) used it for 21–30 minutes per student session. Almost all teachers reported that students used Mathseeds on their own (either entirely or after a whole-class session). About one-quarter sometimes used Mathseeds with just the teacher leading a whole class or with small groups of students working together. Only three or four teachers said they were assigning Mathseeds as homework consistently throughout the year.

Satisfaction with Mathseeds All teachers said they would be extremely likely (6 teachers) or very likely (4 teachers) to recommend Mathseeds to other teachers in their school or other schools who teach Kindergarten. And all said they would be extremely likely (9 teachers) or very likely (1 teacher) to use Mathseeds next year if they are teaching Kindergarten. In the end-of-year survey, instructors reported that satisfaction among students was exceptionally high, with Kindergarten teachers rating students’ satisfaction with Mathseeds an average of 9.2 on a ten-point scale.

All teachers felt that Mathseeds complemented their regular math lessons either extremely well (4 teachers) or very well (6 teachers). Teachers commented on the fact that Mathseeds went in-depth and gave students more opportunities to practice and that the program was useful as a supplement to state-adopted programs as it follows most of the important skills. The type of content teachers named as fitting particularly well with the curriculum was place value, shapes, money and number.

Meeting Student Needs All ten Kindergarten teachers said that Mathseeds’ content met the needs of their average students either extremely well (3 teachers) or very well (7 teachers). Teachers felt similarly about how well Mathseeds met the needs of their other students, with all saying it met the needs of their more able students either extremely well (7 teachers) or very well (3 teachers). There was some disagreement about how well Mathseeds met the needs of less able students, with two saying extremely well, five saying very well, and three saying somewhat well.

Overall, teachers agreed strongly that Mathseeds was easy for students to use, engaging, helpful in teaching students, motivating, and contributing to students’ confidence and to their attitude about math. Satisfaction with Mathseeds Components and Features Teachers said that Mathseeds was highly effective in teaching the four domains of math that it covers: Number Sense, Operations, Measurement, and Geometry. The effectiveness of each domain was rated from 1 (highly ineffective) to 10 (high effective), with average ratings ranging from 9.2 to 9.7.

Conclusion

During the 2015–2016 school year, over 300 students and 18 teachers participated in an effectiveness study of Mathseeds in kindergarten. The primary research question was to investigate whether students who used Mathseeds showed more math skills growth than comparable students who did not use Mathseeds. Student growth in math skills was compared using Multivariate Analysis of Covariance with the posttests as the outcome and the pretest as the covariate to control for differences in the groups.

The quantitative study results showed that Kindergarten students who used Mathseeds showed significantly higher math growth than comparable students that did not use Mathseeds. These findings were consistent for both the SESAT 1 assessment and the teacher observational assessment. The effect size for the difference on the SESAT 1 mathematics assessment was .48 and the effect sizes for the math domains on the observational assessment ranged from .42 to .66.

In addition to the quantitative study, surveys and interviews were conducted throughout the school year. The purpose of this qualitative study was to learn more about the specific implementations of Mathseeds and to methodically collect information regarding the teachers’ evaluation of product effectiveness. The qualitative study results showed that the teachers found Mathseeds to be effective in helping students learn math skills. The overall satisfaction rating for teachers was 8.9 out of 10 and for students was 9.2 out of 10.

Teachers agreed strongly that Mathseeds was easy for students to use, engaging, helpful in teaching students, motivating, and contributing to students’ confidence and to their attitude about math. All teachers said they’d be extremely or very likely to recommend Mathseeds to others teaching Kindergarten and to use Mathseeds next year if they were teaching that grade themselves.

Download the Report