June 2014 - June 2015 Program Report

What learning outcomes have Code.org students achieved?

Code Studio, our online learning platform, is designed to host our interactive CS courses. Students are using Code Studio, on their own and in classroom settings, to demonstrate the use of a number of computer programming concepts at the scale of tens of thousands of classrooms.

We saw phenomenal growth in the 2014-15 school year for both student and teacher account creation.

Table: Code Studio User Growth, June 1 2014 - June 30 2015

Source: Code Studio database

The diversity of the audience we are reaching is also interesting to note. Across all student accounts that self-reported demographic data, 43% self-identified as female and 37% were identified as African-American or Hispanic by their teacher.

By comparison, the demographic breakdown of students taking the most recent AP Computer Science exam in high school was 20% female students and 17% African American, Hispanic, or other historically underrepresented students.

CS Fundamentals Course Completion and Concepts Learned

Code.org offers a series of 20-hour long courses we refer to as "Computer Science Fundamentals." These interactive tutorials are hosted on Code Studio and complemented by lesson plans teachers can deliver in the classroom. The courses focus on helping students develop skills with basic CS concepts including sequencing, debugging, loops, conditionals, nested loops, event-handlers and functions.

Table: Number of Students Demonstrating Skill in CS Concepts*

Source: Code Studio database

*"Skill demonstration" is defined as having solved 80% of the puzzles in at least one stage that teaches that concept. Note: If the same student did CS Fundamentals Courses 1 and 2, they are double counted for a concept. On the other hand, this does not account for learning the concept more deeply by repeating it in different contexts, or at more depth.

In looking at our completion data for any of our three 20-hour CS Fundamentals courses versus our shorter hour long tutorials, we see unsurprisingly a higher completion rate for shorter tutorials than we do for courses which require more time and expertise to complete.

Teachers Matter

Students working on Code Studio courses 2-4 with a teacher’s guidance are more likely to complete the course than students working on their own, outside of a classroom setting. This shows how students who complete a course with adult guidance have an advantage, in terms of their overall persistence in learning CS. Teachers also benefit because Code.org focuses on teacher support and professional development.

Students who are learning with the support of their teacher in a classroom setting complete courses more than those learning on their own, supporting Code.org’s focus on a mix of teacher development and self-guided material.

Measuring Learning Outcomes for ECS

During the 2014-15 school year, Code.org partnered with Exploring Computer Science (ECS) to deliver professional development for the program at scale. In order to gauge the impact of that work, Code.org conducted a pilot evaluation of student learning outcomes in Exploring Computer Science (ECS) using a concepts assessment developed with Outlier as well as a programming test developed independently on Code Studio. Results are promising, but more rigorous design is needed to validate and measure mastery learning and transfer.

A programming challenge was designed by Code.org and piloted to about 1,000 ECS students via Code Studio to see if student programming skills would transfer to a different environment. The data below shows the percentage of students who successfully completed a puzzle challenge related to a particular concept/task (untimed, no hints, unlimited attempts). Further work is required to validate and revise the questions.

Table: Programming challenge pilot results (delivered through Code Studio, N=~1000).

Source: Code Studio database

Further work is required to validate and revise the questions.

Back to 2014-15 Summary