Behind the SAT: The Art and Science of Making Math Questions

As part of our coverage of the 100th anniversary of the SAT, we’re interviewing some of the people behind the test. These are the question writers, psychometricians, accessibility specialists, and others who do the exacting and immersive work that makes the SAT so trusted and influential.  

In this post, we’re talking with Dona Carling, executive director of Math Assessment. Carling joined College Board in 2016 and helped build—and now leads—the team that writes the math questions for the SAT. It’s a “dream job,” she says, for a former math teacher who had strong opinions about assessment and the power of math to open doors—even if her two teenage grandsons have some doubts about whose side she’s on.  

Question: Can you tell us a bit about your background? How does one grow up to direct the team writing the questions for the SAT Math section? 

Dona Carling: I always describe myself as a recovering high school teacher. Early in my career, I taught math in high schools in the Salt Lake City area for 12 years. At one point, I wrote a letter to the district about what a bad match a particular state assessment was to our curriculum—a nicely worded letter. Later, when my district was asked if they had someone who could work on revising the state tests, they said, "Oh, that Carling woman." That's how I got interested in assessment. I then worked for an assessment company in New England for many years. They had a contract with College Board, and I had taught AP courses, so I knew who College Board was from that angle. When I found myself job hunting again, I begged College Board to take me on, and here I am. 

Honestly, if I were designing a job for myself, this is what it would be. I'm very passionate about high school mathematics, and about the fact that it's such a gateway for students—if you do well in math, your options broaden. I really care about the quality of instruction. I think the SAT is a top-notch assessment program. I love the opportunity it gives students to broaden how they think about their future, and I think it's a very fair assessment that does a nice job recognizing the accomplishments of students from various backgrounds. 

Question: How many people are on the Math Assessment team?

Dona Carling: There are 18 of us. I work with 17 absolutely fabulous, passionate mathematicians. As a group, we have a real affinity for high-school-age learners—it's such a pivotal point in their lives when they're making big decisions. Most of us are former teachers; maybe only three people on my team didn't teach in a traditional high school setting but have other kinds of teaching experience. We find that people who've worked with high school students are better at writing test questions. 

Question: What makes a good SAT math question? 

Dona Carling: It's a genuinely complicated process to establish that a math question is a good one. It needs to represent the content students are actually learning in the classroom—a fair representation of what they're expected to know and do. A question is drafted, then reviewed multiple times to refine it until we think it will measure the skill it was assigned to measure. Then it goes through a pretesting process, and we get statistics that help us further evaluate whether it's useful. There's a statistic in test development called "discrimination." In the rest of my life, I try not to discriminate, but professionally you want a question that discriminates among test takers: Kids who don't know the material don't do well—kids who do know it do well, across the whole continuum of student achievement. We write questions ranging from very easy to easy, medium, medium-hard, and hard so we can measure skills fairly and reliably across that whole range. 

A good test question also measures a skill consistently—accurately across lots of students, not just one. We also write questions that are set in a real-world context, as well as questions devoid of context. It's important that they can demonstrate raw skill but also apply that skill in a setting where math is traditionally needed. One of the arts my team has perfected is writing a context with enough information that the student can sort out what math they're actually being asked to do. A good number of questions are set in science contexts, referencing material students would likely encounter in their education, or everyday science that's easier to grasp. We also write some questions in a financial literacy context—discounts, costs, wholesale versus retail—and other contexts as well. 

Question: How do you prevent bias in math questions to ensure they don’t favor certain students over others? 

Dona Carling: It’s important that the people who write the questions come from backgrounds working with students—most of us have worked with students from a variety of backgrounds and have a good sense of what's accessible to kids from a broad range of backgrounds. Our questions also go through a couple of rounds of external review, where specific bias-and-sensitivity experts review any that are set in a context, to make sure that context is accessible broadly. We also have content experts—current classroom practitioners — who look at our questions and give feedback. 

Question: How many humans, would you say, review a single SAT math question from initial drafting to it appearing on the test? 

Dona Carling: Internally, it's at least 7 people. External reviews vary, but generally when we send a question out, at least 4 people review it, sometimes as many as 8, and there are two separate rounds of external review. So, at a minimum, it's 15 to 17. 

Question: How did allowing students to use calculators on the SAT change the question writing process? 

Dona Carling: When I joined College Board, the test already had a calculator-active section and a no-calculator section. One of the arts we've perfected over time is writing questions that are impervious to calculator-only solutions. We want to measure a student's knowledge and skill, not their ability to push buttons on a calculator. Sometimes that means asking the question in a more conceptual way, or in a way where the student might choose to use the calculator as a tool to help find an answer. We don't ask trivial questions that would only require calculator use and no real understanding. 

Question: On the Reading and Writing side, there's a goal to make the test engaging to students. Sounds like you have a similar objective for the Math section? 

Dona Carling: We're careful to make sure there's no repetition of context across a student's test form — you don't want three questions on the same test all involving a bicycle, for example. We watch for that to ensure context varies across a student's experience. The test is also designed so easy questions come first, and difficulty increases gradually as the student progresses—research shows that's a better experience, and students do better when difficulty isn't mixed up. That steady progression is intentional.  

We also ensure every test form has the widest possible variety of content, because any standardized test is necessarily a sampling of everything students may have learned. We write test questions aligned to over 100 different standards, and within each standard there might be 2 or 3 skills measured. Imagine there are roughly 350 possible skills that could be measured on the SAT, and a student's test covers 40 of those. We want to make sure those 40 span the breadth of content as evenly as possible. 

Question: Did you have any experience with the SAT when you were a student?  

I took the PSAT as a sophomore in high school, and it really changed how I thought about my own academic possibilities. I scored very well and got a lot of attention from teachers and administrators. I went to a medium-size high school in a medium-size town in the middle of Washington State. They didn't offer AP at the time. But seeing how my performance compared nationally was very eye-opening. I suddenly saw myself as college-bound, ready to take on the world. I double-majored in math and computer science. There were a couple of other women in the math program, but I was the only woman in the computer science program at the time. I wouldn't have known that path was possible for me if I hadn't taken that test and seen that my math skill compared favorably to others across the country. 

Question: And now you have a grandson who’s preparing for the SAT? 

Dona Carling: Yes, my older grandson is a college freshman this year, and his younger brother is a junior in high school. First, they say, "Grammy, how could you do that for a job?" And when I tell them it's the best job I've ever had, "Oh, Grammy, you poor thing." But they become more appreciative when I talk with them more about the importance of being well-prepared for an exam and for college once you get there.