Prepared for Success
Groups. Maybe you belonged to a few in your school years? Chess club. Drama club. Debate club. Student government, the club that organizes clubs.
All groups have one aspect in common: Participation. Today, we'll look at how to measure and set goals for participation. In particular, our report card should measure the availability of services. You can't participate in something that doesn't exist.
Let's start with a favorite group - teenagers! Are they Prepared for Success? We'll cover Gap Closing, gifted services, and K-3 Literacy in later posts.
Does Success Equal Showing Up?
Of course, we all know that success is not the same as just showing up. That gold-toned plastic little league participation trophy is clearly different from an Olympic gold medal.
The same is true for report card metrics. A key design principle is to state participation goals and achievement goals separately. Participation goals tell if a service is available. Achievement goals tell if students are getting value from a service.
Prepared for Success consists of two sections. The first is standard preparation, measured by ACT and SAT scores. The second is "bonus work" (Advanced Placement, International Baccalaureate, and Industry Credentials). Let's dive into standard preparation first.
Participation is No Substitute for Missing Data
There are many situations where you'd like to compute a metric for a group, but not everyone participated. How should we deal with non-participants? We could ignore them, and just report the achievement of participants, but that's not really fair. Our results might be skewed if only high-performers participated.
Why are people not participating? In other words, is there a participation rate goal? We'll see the answer can be yes or no, and introduce strategies for both.
Prepared for Success currently measures who "passes" the ACT and SAT, which means achieving a score above some threshold (remediation-free). Then, a score is computed:
Prepared for Success score =
Number of students "passing" test / Total students
in other words:
Prepared for Success score = Percentage of students "passing" test
Is this really what we want? The score formula is kind-of-assuming that everyone participates. There's no adjustment for the participation rate, though this is shown on a nearby graph. We could introduce the participation rate into the score calculation. But would that solve our problem? Is there a participation rate goal?
Who participates in the ACT and SAT? Generally, college-bound seniors. Who might not participate? Students with other plans after graduation, students who doubt they would pass, and students who like keeping their $58.50.
And... that's OK!
Our only concern is to fairly represent the population, which means estimating a score for students who doubt they would pass the ACT or SAT.
What we really have is a missing-data problem. This can't be solved via a participation rate. Forcing everyone to fork over cash and participate would "solve" it, sort of, but we sure won't make friends that way.
A better way is to compute a proxy score for non-participants. For example, we could use previous years' state standardized test scores. In fact, the current metric attempts to introduce estimated scores for some people by allowing an honors diploma or industry credential to substitute for a passing score. We're just taking this a step further.
If we can compute proxy scores, our metric accurately reflects the entire student body:
Prepared for Success score =
Number of students "passing" test or proxy / Total students
​
A bullet chart nicely shows achievement compared to a goal. Bullet charts are even more amazing when used for group comparisons. Here, we could show the percentage of students achieving an acceptable score (ACT, SAT or proxy score). The goal (80% passing) is indicated by the black line.
Participation Can Be a Proxy for Availability
Now let's look at the Prepared for Success bonus section, covering Advanced Placement credit, International Baccalaureate credit, and industry credentials.
The current metric for AP achievement is:
Prepared for Success Bonus =
(Number of students "passing" AP test * 0.3 weight) / Total students
What's going on here? Advanced Placement courses are optional classes taken by college-bound students to gain college credit while in high school. Offering AP courses is good for students who want to get a leg up on college studies (and possibly save some cash compared to college-priced credits).
As a "bonus" score calculation, the current metric is appropriate. Participation here is an indicator of availability of AP courses. The more courses available, the more students will enroll. The Achievement Level factor tracks if students benefit from the courses, as an AP exam of 3 or higher is required for college credit. It's a similar story for International Baccalaureate.
Going further, we'd like to show which particular AP courses are available. 38 AP courses exist, so let's group them by topic, and draw a shaded block for each course offered. It's then easy to see which courses are offered while getting a sense of how many more remain.
One final note - the number of AP courses available will depend on the school's size. A larger school will be able to offer more courses. To avoid unfairly penalizing smaller schools, the availability graph could be adjusted for the size of the school, by adding a classification (high/medium/low availability given the size) or by shrinking the width of each non-offered block for smaller schools.
Further, this fits neatly with our other achievement metrics. Essentially, we're computing the achievement of the high school group. We can place the chart there. We'll cover report card reorganization in a later post.
Recommendations
1. Compute substitute scores for students not taking ACT or SAT.
2. Use a bullet chart for Prepared for Success.
3. (Optional) Show AP Course availability.
