Assessment System: Assessment Creation

Rationale

Assessment within the MD program at the Brody School of Medicine will occur in the context of programmatic curricular alignment such that the assessments are aligned with institutional and course/clerkship level learning objectives and learning activities. This includes formative and summative assessments, and will utilize written examinations, (MCQ, SAQ, Essay), performance assessments (OSCE and simulation), assignments (reports, projects, self-reflection), and oral examinations among the assessment tools.

Internally created, blended assessments administered at Brody School of Medicine follow the procedures in this document to provide a secure and reliable examination environment, consistent with what is required by the National Board of Medical Examiners. This allows consistent and accurate assessment of student knowledge fund.

Scope

The Standard Operating Procedure applies to all medical learner assessments at BSOM. It outlines the procedures for creating assessments to learners enrolled in BSOM courses and clerkships and contains recommendations to enable the creation of a highly reliable process. The SOP also communicates expectations to key stakeholders for creating assessments and affects learners, staff, faculty members, and course/clerkship directors of the Brody School of Medicine who design, create, and administer assessments.

Definitions

Accommodation: A change or adjustment from the normal curriculum or equipment format that allows an individual with a disability to access content or complete tasks to pursue a regular course of study.

ADA: The Americans with Disabilities Act, which prevents discrimination against people with disabilities

Assessment Architecture: a planning document including the key characteristics of an assessment, such as the date, time, duration, type of assessments, identity of courses participating, number of items per course, and total number of items.

BSOM: Brody School of Medicine

CD: Course or clerkship director

ECC: Executive Curriculum Committee

Exam, examination: a high-stakes assessment that contributes greater than or equal to 5% of a course grade.

Extended Quiz: a low stakes MCQ assessment that offers flexibility to be completed over a specific time frame.

Learner Identity: a module or application that works in conjunction with the assessment platform that verifies the learner’s identity.

Learning Management System: The technology used to house, plan, implement, assess, and evaluate the learning process

MCQ: multiple-choice questions, written in the single best answer format used by the NBME.

NBME: National Board of Medical Examiners

Quiz: a low stakes MCQ assessment that contributes to 10-20% of the final grade to course in the foundational phase.

Secure Review: an assessment related event that allows students to review the questions and answers on an assessment either immediately or at a fixed time after completing the assessment

General Sequence of Events for Blocks 1-4

Scheduling of Dates and Times for Assessments, Faculty Review and Student Review of the Assessments.
Each course director, with approval by the ECC, will designate the type of assessment that will be administered to learners to appropriately measure mastery of the course/clerkship learning objectives. During the calendar planning for the academic year, the year specific curriculum committee chair and course directors identify the types of assessments to be utilized, dates and times for assessments, faculty review of the assessment, and post-administration learner review. For room scheduling, the estimated duration of the examinations is 30 minutes longer than anticipated to allow for delays in starting and completing the examinations.

Because the timing and duration of the MCQ and laboratory portions of the examination for learners with ADA testing accommodations (see BSOM ADA Accommodations SOP for full details) do not match the schedule of learners without testing accommodations, additional planning is needed to create schedules that support the learners’ needs. Accommodation scheduling information is communicated to the impacted learners, Course Directors, and Deans for Medical Education and Student Affairs by the Testing Administrator. Logistics for all assessment related items will be associated to individual learning events in the the learning management system calendar.

Once the draft of the assessment plan is determined, the year specific curriculum committee chair, along with the calendar committee, reviews this plan to identify and resolve any scheduling conflicts within and across the foundational sciences curriculum. After approval by the year-specific curriculum committee and the ECC, the assessment schedule is incorporated into the block calendar and published on the OME webpage.

Determination of the Assessment Blueprint

Before the beginning of each block, each quiz and examination require an Assessment Blueprint identifying the details of each assessment. Course Directors use the number of examination items indicated in the assessment architecture to determine the distribution of questions and topics for each examination and quiz.

Generally, each hour of lecture or labs should have 1-5 assessment questions. The syllabus must contain a description of the distribution of questions and how it contributes to the course grade. The questions should be equitably distributed across the content to reflect the course objectives. Each question is linked to the course content using a consistent nomenclature. The testing administrator will create a shell for each quiz and examination at the beginning of the block to allow for Course Directors to add questions to their assessments.

Identification of an Assessment (Course) Roster

At the beginning of each academic year, the testing administrator populates each learner into the assessment platform. The testing administrator confirms the identity and provisions of learners granted testing accommodations that are currently enrolled in the undergraduate medical education curriculum. Additionally, the testing administrator confirms with the course directors the identity and provisions of any graduate learners enrolled in the medical courses who participate in the MCQ assessments. The Office of Graduate Studies is responsible for notifying the Testing Administrator of any graduate learners with testing accommodations. This information is used to determine the number of versions of an assessment that must be duplicated by the testing administrator. Additionally, the Testing Administrator generates and posts seating charts in the testing center for each in-person assessment.

Guidelines for Developing MCQ Items

Assessments not employing essay or free text should only use single best answer formats that require test-takers to select the single best response. Avoid use of true/false (C-type, K-type, and X-type) questions.

Best Practices for Item Construction: Refer to Appendix A

Course directors must communicate expectations regarding question development, timeline for submission, and rationale requirements to the instructors teaching in the course.

Questions written by individual instructors must be reviewed and vetted at the course/clerkship director or departmental level prior to being added to the assessment shell. The Course Director is required to tag questions to the USMLE content outline and course/lecture objectives. The Course Director is responsible for the quality, relevance, and oversight of questions submitted. The Dean of Office of Medical Education will report on Assessment Performance at the Foundational Curriculum Committee Meeting that follows each exam.

Uploading of Examination Questions and preparation for assessment administration

Course directors utilizing the assessment platform must ensure that the questions are uploaded into the correct shell. All assessment questions should be reviewed two business days prior to assessment administration. The testing administrator will communicate a proctoring plan one week prior to assessment administration.

Faculty Review of the Assessment

The shell is available at any time to review once populated. Course Directors must review the assessment and submit modifications no later than two business days prior to the assessment. Following this review, the testing administrator posts the assessment and any additional variations needed using the time frames identified in the Assessment Blueprint.

General Sequence of Events for Clerkships

The clerkship director proposes an assessment blueprint that is at least 60% objective evaluations including OSCE, NBME, oral exams, presentations, and/or projects and no more than 40% clinical evaluations.

The assessment blueprint is reviewed and approved by the Clinical Curriculum Committee and Executive Curriculum Committee.

Once the assessment blueprint is approved by the ECC, the clerkship director will work with the Testing Administrator, Director of Clinical Skills, and other appropriate individuals to schedule the assessments. Each clerkship is responsible for providing 1 proctor for NBME exam administration.

Learners must be available until 8 PM on the days of any in-person assessment regardless of the scheduled time of administration.

Responsibilities

There are several key stakeholders involved in the assessment creation process. Listed below are each key stakeholder, along with their expected responsibilities and time frame for completion.

Associate Dean of Medical Education

• Ensures compliance with test development timeline.
• Leads item writing development sessions.
• Oversees all testing operations at BSOM and reports on irregularities to the Executive Curriculum Committee.
• Ensures that the Testing Administrator is trained and operating according to established procedures of BSOM.
• Monitors any failures in courses & clerkships and oversees remediation plans.
• Communicates with the Division of Academic Affairs to request non-faculty proctors.

Executive Curriculum Committee

Responsible for oversight of assessment blueprint and assessment creation process.

Curriculum Committee Chair

Block Planning

Set/Schedule Calendar for block two months prior to the start of the block. Identify assessment days. Establish assessment blueprint, including types of assessments, exam/quiz dates and durations, dedicated makeup dates, question allotment, and secure review dates as appropriate.

Course Director

• Block Planning
  • Reviews assessment blueprint (working with Curriculum Committee Chairs), exam durations, question allotment, and secure review dates.
  • Responsible for communicating the assessment blueprint to all teaching faculty.
  • Ensures that the timing for questions (90 seconds per question) is applied to blended assessments and clearly communicated to learners in the syllabus.
  • Beginning of Block – by the end of the first week of each block: Send a list of graduate learners and confirm off-cycle learners enrolled in the course to the testing administrator.
  • Meet with the Testing Administrator for testing/proctor protocol training.
  • Faculty and Course Directors are advised to only proctor if all proctoring opportunities have been exhausted.
  • Two business days prior to assessment:
    • Questions are reviewed for content and clarity at the course director and departmental level before being uploaded to the testing software.
    • Course directors identify questions to include in the assessment and submit them to the appropriate Assessment Shell.
    • Questions are tagged to USMLE content outline and course/lecture objectives.
• Course Director or a designee should attend all Secure Review sessions to answer learner questions about the assessment. Faculty must rotate through all testing rooms.
• Course Directors should address all clarification forms sent to them by the testing administrator post secure review.
• No changes can be made to the approved calendar or assessment blueprint without approval of the curriculum committees. These must be communicated to the testing administrator immediately.
• Item analysis should be performed after each exam. The following analyses should be performed and reviewed with instructors in the context of the entire exam:
  • Analysis of item difficulty
  • Analysis of item discrimination
  • Analysis of item options (point biserial)
• Assessment statistics should not be shared until all learners have completed the assessment.

Clerkship Director

Responsible for recommending assessment blueprint to the CCC and ECC for approval that meets the objectives of the clerkship.

Instructor(s)

• Throughout the Block
  • Generates questions for assessments using best practices (Appendix A).
  • Uses item analysis and/or comments from previous administrations to improve quality and clarity questions prior to submission.
  • Instructor generated questions should be completed in time to allow departmental and course director review of questions prior to the submission deadline of two business days before an assessment.
• After the Assessment
  • Instructors submitting questions should attend the Secure Review sessions so that learners’ questions can be answered.
  • Reviews the item analysis to revise questions for subsequent use.

Testing Administrator

• Training
  • Testing Administrator must annually review the NBME Chief Proctor video tutorial and the NBME Facility Readiness Checklist.
  • Testing Administrator is responsible for developing and providing BSOM proctor training to all proctors.
  • Prepares and communicates to all proctors how to handle crises, assessment emergencies, and assessment troubleshooting.
  • Provides all proctors with read aloud instructions, proctors report form, seating chart, and a blank sign in/out sheet.
  • Trains proctors on confidential information protected by FERPA guidelines.
• Identifies learners with ADA Accommodations, confirms their profiles in the assessment platform, and communicates accommodations rosters to course/clerkship directors
• Posts SOP and assessment calendar in the learning management system at the beginning of each academic year.
• Environment:

  Routinely(weekly) checks all testing rooms and ensures all equipment (computers, iPads, cameras) is functioning properly, is ready for examinations, and is secure. If a computer, partitions, lighting, HVAC, etc. is not operating properly, the testing administrator contacts the proper department to resolve the issue.

• Beginning of Block or Cohort
  • Orders all NBME exams for M1-M4 courses and clerkships.
  • Orders and communicates availability of self-assessment vouchers for NBME assessments to clerkship coordinators for distribution to their learners.
  • Uploads rosters to NBME testing page
• Block Planning

  Schedule Exams:

  • Works with year-specific Committee Chair to schedule Secure Reviews as a component of block planning
  • Uses 25 Live to reserve all rooms for in-person assessments and in-person secure review.
  • Updates and manages OME Assessment Calendar in Outlook so all involved faculty/staff have access to assessment schedules and locations.
  • Coordinates and communicates schedules for learners with testing accommodations in the learning management system.
  • Oversees creation of exam seating charts prior to every assessment including a late arrival station.
    • Develops and maintains standardized log of learner breaks.
    • Identifies and schedules proctors on the OME Assessment Calendar in Outlook and confirms that proctors are available 1 week prior to the assessment.
    • NBME requires one proctor per 25 learners per room.
    • OME provides proctors for testing of learners with ADA approved testing accommodations.
  • Builds assessment shells for each examination and quiz in courses at the beginning of each block.
  • Communicates with Course Directors to ensure assessment questions have been submitted and are ready to be posted two business days prior to the assessment start time.
  • Communicates with course director, department chair, and Associate Dean of OME if questions are not submitted within two business days prior to the assessment.
  • Includes logistics and pertinent information regarding all assessments in the learning event in the learning management system.
  • Assembles and posts the assessment with approved questions from each course participating.

   

• Prior to the Day of Assessment:
  • For extended assessments, limited individual study spaces are available for learners to reserve at Laupus Library. Spaces should be reserved in advance.
  • Ensures that a representative from OSA is available, in the testing center, prior to each assessment seat time and 20 minutes after each assessment seat time.
• After Completion of Assessment:

  Notifies course directors of absent learners to allow grading and timely posting of scores

Schedule for Review

This procedure is reviewed and approved by the Curriculum Committees, including the Executive Curriculum Committee, Foundational Curriculum Committee, and the Clinical Curriculum Committee every three years.

The Assessment Creation Policy is posted on the OME Website by the Office of Medical Education to allow learners and teaching faculty/administration to reference at any time.

Related Policies

None

Applicable Laws, Regulations & Standards

LCME Standards for Accreditation of Medical Education Programs Leading to the MD Degree: Published March 2022; Standard 8, Element 2; Standard 8, Element 3; Standard 8, Element 7; Standard 9, Element 4, and Standard 9, Element 8

Appendices

Appendix A

Recommendations from the Ad-hoc Testing Committee on Question Construction

Content adapted from the NBME Item Writing Guide. Additional information can be accessed.

Why should learners be tested?

• Communicate material that is important
• Motivate learning
• Identify learning needs
• Assess obtainment of learning objectives
• Determine grades
• Identify areas where instruction can be improved

What should be tested?

• Exam content should match course/clerkship and session objectives
• Important topics should be emphasized
• The testing time devoted to each topic should reflect the relative importance
• The sample of items should be representative of the instructional goals.

Recommendation #1

The committee recommends that assessments not employing essay or free text should only use single best answer formats that require test-takers to select the single best response. Avoid use of true/false (C-type, K-type, and X-type) questions.

Background: One Best Answer (A-type) questions consist of a stem (e.g. a clinical case presentation) and a lead in question, followed by a series of choices, typically one correct answer and three to four distractors. When written correctly, incorrect options do not have to be entirely wrong, but less correct than the keyed option.

Why Single Best Answer?

• Prevents test-taker from having to guess author’s intent
• More efficient/easier to write as incorrect options do not have to be entirely wrong
• Same stem can be paired with different lead-ins to create item sets (diagnosis and management)

Best Practices for Item Construction

• Items should focus on an important concept
• Items should assess application of knowledge, not recall of an isolated fact, ideally by using clinical vignettes or analysis of data.
• Question should be able to be answered without seeing the options.
• The options should all be homogenous and plausible.
• Do not use true-false questions.
• Avoid negatively phrased A type questions- for example, all of the following are correct except or which of the following statements is not correct.
• Include as much of the item as possible in the stem; the stems should be long and the options short.
• Avoid superfluous information.
• Avoid tricky or overly complex items.
• Write options that are grammatically consistent and logically compatible with the stem.
• Distractors should have the same relative length as the answer.
• Avoid use of imprecise terms- usually, frequently, often, commonly, most of the time, almost never.

Recommendation #2

Item analysis should be performed after each exam. The following analyses should be performed and reviewed in the context of the entire exam:

• Analysis of item difficulty
• Analysis of item discrimination
• Analysis of item options

Avoid flaws related to testwiseness

• Grammatical clues- one or more distractors don’t follow grammatically from the stem
• Logical cues- a subset of the options is collectively exhaustive
• Absolute terms- terms such as always or never are in some options
• Long correct answer- correct answer is longer, more specific, or more complete than other options
• Word repeats- a word or phrase is included in the stem and in the correct answer
• Convergence strategy- the correct answer includes the most elements in common with the other options

Avoid flaws related to irrelevant difficulty:

• Options are long, complicated, or double
• Numeric data are not stated consistently
• Terms in the options are vague (rarely, usually)
• Language in the options is not parallel
• Options are in a nonlogical order
• None of the above is used as an option
• Stems are tricky or unnecessarily complicated
• The answer to an item in hinged to the answer of a related item.

Guidelines for Writing Foundational and Clinical Science Items

• An assessment blueprint should be developed for each assessment to keep item writers focused on important topics and key content.
• Test application of knowledge using clinical vignettes or experimental vignettes to provide context to the question being asked.
• Focus items on common or potentially catastrophic problems; Avoid “zebras.”
• Pose decision-making tasks appropriate for the level of training.
• Questions should focus on specific tasks that students must be able to undertake at the next stage of training: What is the most likely diagnosis, additional labs needed, formulate the next step in management.
• Focus on areas in which clinical reasoning mistakes are commonly made.
• Use of vignettes ensures that the student not only knows the information but can apply it to hypothetical situations.

Writing one-best-answer items

• Whenever possible, items should be written with a clinical stem.
• Clinical vignettes should begin with presenting problem of a patient, followed by the history including duration, physical findings, results, initial treatment, subsequent findings.
• Vignette may only include a portion of this information but should be supplied in that order.
• Stem lead-in should pose a clear question that the examinees should be able to answer without looking at the options.
• Vignettes should avoid red herrings and lying patients.
  Examples of types of foundational and clinical lead-ins:
• Which of the following is the most like cause/mechanism of this effect?
• Which of the following is the most likely causal infectious agent?
• This patient most likely has a defect in which of the following?
• This patient most likely has a defect in which of the following enzymes?
• Which of the following cytokines is the most likely cause of this condition?
• Which of the following structures is at greatest risk for damage during this procedure?
• The most appropriate medication for this patient will have which of the following mechanisms of action?
• Which of the following factors in the patient’s history most increased his/her risk for developing this condition?
• Which of the following is the most likely explanation for these findings?
• Which of the following is the most likely location of the patient’s lesion?
• Which of the following is the most likely pathogen?
• Which of the following findings is most likely to be increased/decreased?
• Which of the following is the most appropriate intervention?
• For which of the following conditions is the patient at greatest risk?
• Which of the following is most likely to have prevented this condition?
• Which of the following is the most appropriate next step in management to prevent morbidity/mortality/disability?
• Which of the following is the most likely diagnosis?
• Which of the following is the most appropriate next step in diagnosis?
• Which of the following is most likely to confirm the diagnosis?
• Which of the following is the most appropriate initial or next step in patient care?
• Which of the following is the most effective management?
• Which of the following is the most appropriate pharmacotherapy?
• Which of the following is the first priority in caring for this patient?

Example of grammatical cues

A 60-year-old man is brought to the emergency department by the police who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of

1. Examination of CSF
2. Glucose with vitamin B1 (thiamine)
3. CT scan of the head
4. Phenytoin
5. Diazepam

Options A and C do not follow logically from the stem.

Logical Cues

Crime is

1. Equally distributed among the social classes
2. Overrepresented among the poor
3. Overrepresented among the middle class and rich
4. Primarily an indication of psychosexual maladjustment
5. Reaching a plateau of tolerability for the nation

Options A, B, and C include all possibilities so the student knows that one must be correct.

Absolute terms

In patients with advanced dementia, Alzheimer’s type, the memory defect

1. Can be treated adequately with phosphatidylcholine
2. Could be a sequel of early parkinsonism
3. Is never seen in patients with neurofibrillary tangles at autopsy
4. Is never severe
5. Possibly involves the cholinergic system

Options C and D can be eliminated based on the absolute terms.

Word Repeats

A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalizations is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called

1. Depersonalization
2. Derailment
3. Derealization
4. Focal memory deficit
5. Signal anxiety

Unreal appears in the stem and in the options.