Behind the SAT: A Test Writer’s Quest

This year marks the 100^th anniversary of the SAT. It’s a significant milestone. But what explains the test’s staying power and influence as a signal of achievement and readiness? A big part of the answer lies in the care, expertise, and research that go into creating it. In this series, we’re going behind the scenes with talk with some of the professionals who do that work at College Board—the real humans, including question writers, psychometricians, and accessibility specialists, who make the SAT what it is.  

Our first insider is Moses Utomi, a senior assessment specialist in ELA/literacy who’s been with College Board for nine years. He’s part of the team that does the exacting and immersive work of writing the questions for the Reading and Writing section of the SAT. Prior to taking this unique position, Moses was—appropriately enough—an English teacher. He’s also a published author of fantasy novels and short fiction.  

Question: What’s the main goal when you’re designing an assessment item? 

Moses Utomi: The most important thing is to assess a skill we want to assess. If we want to see how skilled a student is with comma rules and punctuation, we want an item that tests specifically that. We want an item that doesn’t bring in other skills, like a student’s understanding of sentence structure unrelated to commas or their familiarity with the content itself. We really want to home in on a discrete skill so the student can show that skill by answering the question. That way, the information we get back─who answered it correctly, how long it took them─is actually useful because we know we’re assessing the thing we intended to assess.  

The other piece, especially on the design side, is that we want to give good, interesting examples of reading and writing. We want students to read the question and be engaged. So we spend a lot of time finding engaging topics and, increasingly, a diversity of figures from different cultures and backgrounds. We want it to feel like if you sit through an SAT reading or writing section, you’ll learn something about the world, about history, science, art—things that are interesting in a way that actually stick with the student. 

Question: Walk us through the process. How do you go about creating a test item?  

Moses Utomi: First, our team leads do the math on what our needs for questions are. We promise a unique test form to every student. Depending on the number of test administrations, team leads figure out, among other things, how many items we need, how many should be easy, medium, or hard, and how many should be history, science, and social studies. We get those specs, and then as item writers, we start by selecting the ones we want to work on. I might decide I need to write a humanities item testing rhetorical synthesis, appropriate for the 9th-to-10-grade reading band, at very high difficulty. 

Once I have that spec, the process starts with article research: finding the source material we can shape to the specs. All of us have slightly different backgrounds and interests. I was a psychology student who’s also a writer, so I lean that way. Others have more background in, say, Italian history or 17th-century British monarchy. We draw on our own knowledge to find a topic that fits the spec. Once we have a topic—a person or a specific article that covers a lot of the ground—it becomes the foundation, and we build out our knowledge base from there.  

The goal with topics is that they need to be curricular but not on-curriculum. You don’t want a student who got more exposure to particular works in English class to have an advantage on an item just because of that. We’re still testing reading or writing ability—the discrete skill. We look for things that might be touched on in a school curriculum but aren’t answerable on the basis of prior familiarity with that specific subject.  

Once we get to actually writing the item, we spend a lot of time talking through what the spec means. On the writing side, we have three main specs: rhetorical synthesis, transitions, and English conventions. Within conventions alone, there are things like commas, verb tense, colon usage, clausal connections, and clause coordination. We think through which subcategory could support a very hard item or, regardless of subcategory, what makes an item very hard versus medium versus easy. 

After initial drafting come the reviews. Every item goes through many rounds of expert critique and refinement before being pretested. After pretesting comes statistical analysis and potentially more rounds of revision and re-pretesting. Only after an item has cleared dozens of content and statistical checks will it count toward a student's score on an SAT. 

Question: Do you have your own personal “wall of fame” of items? 

Moses Utomi: Honestly, at some point you make and review so many that you stop remembering which ones are yours. This happens a lot in meetings. Someone will say, “We found this item with great stats. Whose is it?” We’ll all go quiet for 30 seconds trying to remember. But some items do perform exceptionally well statistically. It’s genuinely hard to make very hard items. Differentiating the top 2% of students from the top 5% is tough. When an item is both statistically strong and good content-wise, it feels like a little badge of pride. I think we all have at least one item we’re proud of.  

Question: Can you tell us about yours? 

Moses Utomi: I can, though it’s probably in the current pool, so I’d rather not detail it exactly. I made a rhetorical synthesis item—a category that’s newer to the SAT and didn’t exist in this form on previous versions. The student gets a set of notes and has to combine them to fulfill a specific rhetorical purpose. It was exceedingly hard to make very difficult versions of this item type. We ended up with a lot of medium-difficulty ones. I managed to make one that was very challenging, but it still felt fair. Challenge for its own sake isn’t the goal, but this one felt both fair and genuinely complex.  

Question: It seems like you’re not unlike a writer for Jeopardy! Have you talked to any Jeopardy writers?  

Moses Utomi: I haven’t had any firsthand conversations with them, but I’d suspect our item-writing processing might be more involved than even theirs. In a way, the stakes might be higher for us. There's no cash prize like on a game show, but millions of students around the world depend on us—and of course there’s a lot on the line, like scholarships.  

Question: Tell us how you got into this work—and your own SAT experience or experience with other College Board programs as a student.  

Moses Utomi: This is a little embarrassing, but I was a standardized-test nerd as a kid. I actually enjoyed taking them growing up. I remember the tests we took in 4th grade and held onto the results. In high school I wasn’t the greatest student. I wasn’t particularly attentive or interested in school. But when the SAT came around, I thought, “Oh, a test. I like tests.” I took it, and while I don’t know that I scored exceptionally, I did well compared to my peers. That helped me get into a better college than my GPA and lack of extracurriculars probably would have on their own.  

Fast forward. I became a writer, taught English, moved to New York, and was job hunting. Like a lot of people, I didn’t realize the SAT was made by actual people. It felt like it was just handed down, like a document that existed. When I saw this job, I thought it sounded amazing, exactly the kind of thing I was interested in. The interview process reinforced that.  

Question: So when we talk about the SAT as a “searchlight for undiscovered talent,” you’re basically describing your own story?  

Moses Utomi: Yeah. That message really resonated with me when I got here: the idea that a student’s SAT score can give them another chance to show what they’re capable of, beyond what their transcript says. I think that’s essential.