Guest post by Paul Nichols, ACT

This is one of a series of blog posts from chapter authors from the new Handbook of Cognition and Assessment.  See Beginning of a Series: Cognition and Assessment Handbook for more details.

In our chapter in the Handbook, we present and illustrate criteria for evaluating the extent to which theories of learning and cognition, and the associated research, when used within a principled assessment design (PAD) approach, support explicitly connecting and coordinating the three elements comprising the assessment triangle: a theory or set of beliefs about how students think and develop competence in a domain (cognition), the content used to elicit evidence about those aspects of learning and cognition (observation), and the methods used to analyze and make inferences from the evidence (interpretation). Some writers have cautioned that the three elements comprising the assessment triangle must be explicitly connected and coordinated during assessment design and development or the validity of the inferences drawn from the assessment results will be compromised.

We claimed that a criterion for evaluating the fitness of theories of learning and cognition to inform assessment design and development was the extent to which theories facilitate the identification of content features that accurately and consistently elicit the targeted knowledge and skills at the targeted levels of complexity. PAD approaches attempt to engineer intended interpretations and uses of assessment results through the explicit manipulation of the features of content that tend to effectively elicit the targeted knowledge and skills at the targets complexity levels. From the perspective of PAD, theories of learning and cognition, along with the empirical research associated with the theories, should inform the identification of those important content features.

The claim from a PAD perspective is that training item writers to intentionally manipulate characteristic and variable content features enables item writers to systematically manipulate these features when creating items and tasks. Subsequently, items and tasks with these different features will elicit the kind of thinking and problem solving at the levels of complexity intended by the item assignment. But I have no scientific evidence supporting this claim. I have only rationale arguments, e.g., if item writer understand the critical content features then they will use them, and anecdotes, e.g., item writers told me they found the training helpful, to support this claim.

An approach that might help me and other researchers gather evidence with regard to such claims is called design science. Design science is fundamentally a problem solving paradigm. Design science is the scientific study and creation of artefacts as they are developed and used by people with the goal of solving problems and improving practices in peoples’ lives. In contrast to natural entities, artefacts are objects, such as tests, conceptual entities, such as growth models, scoring algorithms or PAD, or processes, such as standard setting methods, created by people to solve practical problems. The goal of design science is to generate and test hypotheses about generic artefacts as solutions to practical problems. Design science research deals with planning, demonstrating and empirically evaluating those generic artefacts. A main purpose of design science is to support designers and researchers in making knowledge of artefact creation explicit, thereby moving design from craft to science.

ACT is hosting a conference on educational and psychological assessment and design science this summer at their Iowa City, IA, headquarters. A small group of innovators in assessment are coming together to consider the potential of design science to aid assessment designers in designing and developing the next generation of assessments. Look for the findings from that conference at AERA or NCME in 2017.