The following article is part 1 in a series.

Some time ago, I wrote about a flawed question, in The Official SAT Study Guide from the College Board. In trying to understand the thinking of the question writer, I reviewed the official explanations to the test that are available on the College Board's website through its online course.

The patent inadequacy of the explanation irritated me so much that I began a review of the explanatory answers for the writing sections of all the tests available through the online course, supplemented by other publicly available information. My purposes were to determine what sort of grammatical framework the College Board assumes for the writing test, how accurately that framework describes the English language, and how adequate the College Board's explanations are in accounting for the acceptability or lack of acceptability of particular constructions.

I found this review disturbing. The SAT is the most prominent high-stakes test in the United States, taken by over two million students a year. The test functions as a gatekeeper for many elite American universities. Although a high score certainly doesn't guarantee admission to the highly competitive schools, a low one can doom an applicant's chances. Given the stakes, the test's authors and sponsors have a high obligation to ensure that test is based on the best expert understanding of the domain tested, that the scope of topics covered is specifically delineated and publicly disclosed, and that test developers have sufficient knowledge of the subject to apply the specifications consistently.

The available information on the SAT Writing test suggests that none of these standards have been met. Although the College Board has provided some public information about the test's scope, it lacks adequate specificity, leaving many questions unresolved.[1] Moreover, the explanatory answers available in the online course demonstrate a flawed understanding of grammar. They contain numerous mistakes, ranging from those that are merely irritating to those severe enough to result in flawed test questions. Some explanations are inconsistent with others, suggesting that different writers of those explanations may well be using different grammatical frameworks. More worrying still, some explanations betray a failure to understand key syntactic concepts.

At the very least, these explanations do not provide adequate value to those students who have paid the College Board to receive them. To the extent that these explanations reflect the grammatical reasoning of the test authors, they raise concerns that go beyond problems with a mere study aid and question the validity of the test itself.

In the critique that follows, I will have frequent occasion to cite the tests and explanations provided by College Board. The tests include eight tests available in The Official SAT Study Guide and seven from the online course.[2] Since the explanations exist only in an online format, I adopt the following notation to refer to specific items: "SG-" followed by a number indicates the practice test of that number found in The Official SAT Study Guide. Thus "SG-1" refers to practice test 1, and so on. "OC-" followed by a number indicates practice tests available in the online course. "OC-p" refers to the online course pretest. Specific questions are referenced by section and question number. For example, "OC-5.10.12" refers to the online course, test 5, section 10, question 12. Other references to printed material use conventional page reference.

Vague Standards

The Standards for Educational and Psychological Testing require that "the construct that the test is intended to assess should be clearly described." (Standard 1.2). In a superficial way, the College Board has met this obligation.

Nonetheless, to anyone familiar with issues surrounding English usage, there are numerous unresolved questions, most prominently what standards for "correct" usage are assumed. The following are the instructions given in the tests themselves:

In making your selection, follow the requirements of standard written English; that is, pay attention to grammar, choice of words, sentence construction, and punctuation. Your selection should result in the most effective sentence—clear and precise, without awkwardness or ambiguity. (College Board, 2005, p. 153)

What, however, do the test developers consider "standard written English" to comprise? Since there is no single authority that dictates what constitutes standard English, these instructions gloss over significant variations among different groups and different usage guides.

One obvious point of variation occurs with the different national varieties of English. American and British English do not follow an identical standard, and those differences extend beyond the obvious issues of spelling and punctuation at the end of quotation marks, which aren't tested anyway.

Does the lack of qualification mean that only points of bedrock usage, those that transcend national boundaries and that are consistent across the English-speaking world appear on the test? No. In practice the test makers consistently follow American usage, and answering some questions correctly depends on that knowledge. For example, the SAT appears to treat collective nouns like government or committee as always singular for the purposes of subject-verb and pronoun-antecedent agreement. Yet British usage allows such words to be plural. British English certainly counts as a variety of standard English, but students who follow British conventions are at a high risk of answering questions on this point incorrectly. This is particularly a problem for international test takers, since British English is the variety taught to EFL students in many countries, and since nonnative speakers are more likely to be unfamiliar with the nuances of regional variation in English.

The instructions within a test, of course, are necessarily compressed, but the assumption of American standards is nowhere made explicit.

More specific list of the standards that are assumed for the purposes of the writing test appear in several different places. One list appears in College Board (2005, pp. 101-102). A slightly more detailed one appears in Kobrin and Kimmel (2005):[3]

• Avoiding faulty predication in sentences
• Avoiding dangling modifiers
• Using comparative modifiers appropriately
• Using appropriate idiomatic words, phrases, or structures
• Avoiding weak, passive constructions
• Using connectives appropriately
• Avoiding illogical comparisons
• Subordinating and coordinating ideas in sentences
• Avoiding pronoun shift
• Combining sentences appropriately
• Maintaining parallel structure in sentences
• Using appropriate verb forms
• Avoiding wordiness
• Controlling errors in subject–verb agreement
• Avoiding errors in pronoun agreement, case, and reference
• Maintaining tense sequences
• Making acceptable word choices
• Avoiding run-on sentences
• Avoiding sentence fragments
• Avoiding comma splices

Again, the appearance of specificity masks real uncertainty about what is included.

For example, one item mentions errors in pronoun case. But there are many varieties of case error (and putative error). The example given in the list in College Board (2005) involves case in a coordinated structure ("between you and I"). But what about the predicate nominative ("It was she who gained our trust.")? What about the case of relative pronouns (who vs. whom)? What about the case of pronouns in comparative structures ("No one is more dedicated than he")?

Other items in the list appear on the surface to be clear, for example "maintaining tense sequence." And yet, as an analysis of the explanatory answers will make clear, the test makers do not have the same notion of tense sequence as linguists. As a result, a number of problems involving verb phrases other than tense appear on the test that the test makers mis-classify as involving tense sequence.

The end result is that there are more varieties of flawed sentences on the test than are acknowledged by the public specification of test content, yet the only way that anyone can know that such is the case is to conduct an exhaustive analysis of test questions. Test takers should not be required to reverse engineer the test specifications in order to know what content they are responsible for.

In Part 2, I will analyze the College Board's explanations to its own questions.

----
Notes

[1] One source of information that I did not have available to consult was the technical manual for the test. All tests that adhere to the standards for educational and psychological testing have such a manual, and if an adequate account of the specific grammatical framework that the test assumes is specified anywhere, it should be here. The manual for the SAT does not appear to be publicly available. Efforts to contact the College Board about the manual received no response. Since one purpose of a technical manual is to avoid such problems as I catalog here, though, whatever material is in that manual would appear to be inadequate.

[2] The tests in the online course and explanations for those tests and the ones in the printed study guide were retrieved from the College Board web site between June 10 and June 15, 2007. These explanations are only available to those who have paid for the online course. Url: http://www.collegeboard.com/student/testing/sat/prep_one/prep_one.html

[3] Most differences between the lists stem from differences in terminology and are not significant. Kobrin and Kimmel (2005) contains a few items not covered in College Board (2005): appropriate connectives , combining sentences, and appropriate verb forms. One item, faulty predication, is partly covered in College Board (2005), under the heading "noun-number agreement." One item, adjective/adverb confusion, does not appear in Kobrin and Kimmel.