Assessment System: Continuous Quality Improvement of Assessment Process

Rationale

Continuous quality improvement is needed throughout the assessment processes in the Foundational Phase of the curriculum (Blocks 1-4). This procedure will outline a process to improve assessments and improve the vertical and horizontal alignment of content within assessments and provide central oversight by the Executive Curriculum Committee.

Scope

This procedure applies to multiple-choice assessment items within the Foundational Phase of the Curriculum (during Blocks 1-4). This policy outlines responsibilities of Office of Medical Education staff, administrators, course directors.

Definitions

Point-biserial correlation: a correlation coefficient that discriminates between high scoring and low scoring learners. Correlation’s coefficients range from -1.0 (perfect negative discrimination) to 1.0 (perfect positive discrimination).

CQI: continuous quality improvement
Item difficulty: the percent of overall test takers that selected the correct response for a given item. Item difficulty is measured by a p-value.

Reviewable Items Exam: assessment created in testing software that includes all items that have been identified for review. This exam allows for Course Directors to amend questions for future iterations.

Multiple-choice question (MCQ): for the purposes of this procedure, MCQ items must meet the standards outlined by the BSOM Testing Procedure as approved by the ECC, including:

• Utilizing single best response format
• Avoiding use of true/false (C-type, K-type, and X-type) questions
• Avoiding negatively phrased A type questions- for example, all of the following are correct except or which of the following statements is not correct
• Writing options that are grammatically consistent and logically compatible with the stem.
• Avoiding use of imprecise terms (usually, frequently, often, commonly, most of the time, almost never)

Reviewable test items: any multiple-choice question that returns a p-value less than or equal to 0.40, a point-biserial correlation below 0.00, or assessment adjustment(s).

p-value: measure of item difficulty where lower values indicate more difficult items while higher values indicate easier items (range: 0-1). For this procedure, easier items are defined as having a p- value greater than or equal to 0.90; more difficult items are defined as having a p-value of less than or equal to 0.40.

Procedures for Implementation and Review

Pre-exam period for review of test item quality

Prior to the administration of a Foundational Phase examination, the following processes occur:

• Course Directors are responsible for vetting of all assessment questions.
• Course Directors must follow the assessment blueprint (e.g., number of allowed questions).
• Testing Administrator creates the assessment shell in ExamSoft for internally created assessments.
• Course Directors are responsible for populating assessment questions into the correct shell and for reviewing all questions on the blended examination for grammatical and content related accuracy.

Post-exam period for review of test item quality

Following the administration of a Foundational Phase examination, several processes occur:

• Secure exam review – specific time dedicated for learners to review exam questions, reflect on rationales and reasons for correct answers, and provide an educational opportunity to gain insight into tested concepts. Secure exam reviews should be part of the block calendar approved by the ECC.
• Question clarification process – during the secure review learners can ask for student clarification forms for internally created exams only. The purpose of the student clarification form is to identify learners’ critical thinking/processing of questions and answer choices and to help provide clarifications to learners’ misperceptions. Student clarification forms can only be submitted to the proctor during the secure review and not after. The Testing Administrator is responsible for returning completed student clarification forms to the Course Directors after the completion of each secure review.
• Assessment quality assurance – Course Directors review item analysis report and student clarification forms to determine if credit for items needs to be amended.
• Course Directors must submit the Assessment Performance Report to the Testing Administrator no later than two weeks after the completion of an examination.

Reporting of CQI results

OME collates the following data and reports to the ECC and annual foundational sciences retreat:

• p-value less than or equal to 0.40
• point-biserial less than 0.00
• assessment adjustment(s)

• This data will be collected for each of the participating courses and aggregated for the block and phase.

Responsibilities

Multiple people contribute to the continuous improvement of test items.

Course Director

Course Directors are responsible for populating assessment questions into the correct shell and for reviewing all questions on the blended examination for grammatical and content related accuracy. Course Directors are responsible for vetting of all assessment questions. Following each examination, the course director should review any items that are identified via item analysis or question clarification process. They should seek to correct/amend/archive questions based on identified weaknesses in item design. Each course director must review an item analysis report after each examination and improve items for future iterations. The Course Director is responsible for submitting the Assessment Performance report to the Testing Administrator two weeks after all learners complete each examination.

Testing Administrator

The Testing Administrator creates the assessment shell in ExamSoft for internally created assessments. The Testing Administrator is responsible for collecting the Assessment Performance Report from each course director two weeks after the completion of each examination. The Testing Administrator is also responsible for returning completed student clarification forms to the Course Directors after the completion of each secure review.

Assistant Dean of Assessment and Outcomes

The Assistant Dean is responsible for reviewing the item analysis produced by the testing administrator after each examination. The Assistant Dean is also responsible for providing oversight of the CQI process related to testing and reporting progress to the ECC and annual curriculum retreats.

Associate Dean of Curricular Innovation in Medical Education

The Associate Dean is responsible for reviewing the item analysis produced by the testing administrator after each examination. The Associate Dean is also responsible for supporting the Assistant Dean in providing oversight of the CQI process related to testing and reporting progress to the ECC and annual curriculum retreats.

Schedule for Review

This procedure is reviewed and revised by the Executive Curriculum Committee as necessary, but at least every three years.

Related Policies

SOP 9.4A.1- Assessment System: Creation
SOP 9.4B.1 – Assessment System: Administration
SOP 9.4 – Assessment System – Formative Assessment and Feedback
SOP 9.5 – Assessment System – Narrative Assessment

Applicable Laws, Regulations & Standards

LCME 8.3 Curricular Design, Review, and Revision/Content Monitoring
LCME 9.0 Teaching, Supervision, Assessment, and Student and Patient Safety
LCME 9.4 Assessment System