Assessment System: Assessment Administration

Rationale

Assessment within the MD program at the Brody School of Medicine will occur in the context of programmatic curricular alignment such that the assessments are aligned with institutional and course/clerkship level learning objectives and learning activities. This includes formative and summative assessments, and will utilize written examinations, (MCQ, SAQ, Essay), performance assessments (OSCE and simulation), assignments (reports, projects, self-reflection), and oral examinations among the assessment tools.

Internally created, blended assessments administered at Brody School of Medicine follow the procedures in this document to provide a secure and reliable examination environment, consistent with what is required by the National Board of Medical Examiners. This allows consistent and accurate assessment of student knowledge fund.

Scope

The Standard Operating Procedure applies to all medical learner assessments at BSOM. It outlines the procedures for administering assessments to learners enrolled in BSOM courses and clerkships. The SOP also communicates expectations to key stakeholders for administering assessments and affects students, staff, faculty members, and course/clerkship directors of the Brody School of Medicine who design, create, and administer assessments.

Definitions

Accommodation: A change or adjustment from the normal curriculum or equipment format that allows an individual with a disability to access content or complete tasks to pursue a regular course of study.

ADA: The Americans with Disabilities Act, which prevents discrimination against people with disabilities

Assessment Architecture: a planning document including the key characteristics of an assessment, such as the date, time, duration, type of assessments, identity of courses participating, number of items per course, and total number of items.

BSOM: Brody School of Medicine

CD: Course or clerkship director

ECC: Executive Curriculum Committee

Exam, examination: a high-stakes assessment that contributes greater than or equal to 5% of a course grade.

MCQ: multiple-choice questions, written in the single best answer format used by the NBME

NBME: National Board of Medical Examiners

Quiz: a low stakes MCQ assessment that contributes to 10-20% of the final grade to course in the foundational phase.

Secure Review: a component of the assessment platform allowing learners to review the questions and answers on an assessment either immediately or at a fixed time after completing the assessment.

Testing Blueprint: a planning document used by the course director to ensure the balanced content representation of the MCQs used on an assessment.

The assessment platform: the testing platform used for the development and administration of multiple-choice quizzes and examinations at BSOM.

Administration of Internally Created Assessments

The Brody School of Medicine administers timed, internally created assessments throughout the foundational phase of the curriculum. All multiple-choice assessment items for standard testing are allotted 90 seconds per question for answering. Assessment timing is cumulative and allows students to self-pace. Assessments are delivered utilizing the following parameters.

Remote Extended Quizzes

Quiz Structure:

• All foundational science quizzes will be administered using the assessment platform given the following format:
  • Remotely administered
  • Timed quizzes open after the last scheduled full class session on Fridays and remain open until Sunday at 6:00PM (local time).
  • The assessment platform lockdown browser will be applied.
  • Exam ID will be used for verification of test taker ID.
  • Learners will only receive 1 download per quiz unless technical issues arise.
  • Learners are allowed 1 sheet of blank scratch paper.
  • Secure review will be offered for 20 minutes immediately following the quiz and is included in the allotted upload deadline.
  • During secure review, learners will only have access to incorrect questions with rationales.
  • Learners must adhere to honor code with the additional following restrictions:
    • All quizzes must be completed independently.
    • Learners must not copy or share quiz content with any learners.
    • Learners are not allowed to take photos/videos.
    • Learners are not allowed to write down quiz questions or share them electronically.
    • Learners are not allowed to discuss the quiz questions with other learners.
  • Quiz grades must not be dropped in the foundational phase.
  • Quizzes must be weighted 10-20% of the final grade for courses in the foundational phase of the curriculum.
  • Each hour of lecture or laboratory session covered on a quiz should have between 1-5 questions.
  • For remote extended quizzes, the upload times will be monitored. If a learner fails to upload quiz during the required window for an unexcused reason, the time will be recorded and shared with the Promotion and Review Committee/Community Standards Committee.
  • If technical issues arise, learners should review the troubleshooting guide. If the issue persists, learners must contact the assessment platform technical support first.
  • If the assessment platform has not resolved the issue, the OME assessment team will follow up the morning of the next operational business day.
  • Laupus library has loaner devices and/or study rooms available if needed.
  • If a learner is granted an excused absence for a quiz, they must first complete the REDCap survey to notify the Office of Medical Education. All make-up quizzes must be taken during the predetermined make-up session identified on the Block calendar. Quizzes must not be made up during class sessions.
  • Quiz flex days will not be offered.

In-Person Assessments

Prior to the assessment, the testing administrator routinely checks the testing rooms to ensure that the testing stations are functioning properly.

Prior to Assessment

The following are done in advance:

• Scans the room for any prohibited items.
• Ensures a dry erase noteboard and a dry erase pen is placed at each testing station.
• Verifies that earplugs are placed at the proctoring station.
• Verifies camera system functionality.

Day of Assessment

• The assessment team arrives 45 minutes early to ensure that the rooms are open and seating chart is posted outside the exam rooms 30 minutes before the beginning of the assessment.
• The assessment team ropes off testing area and places disposable cups at the water fountain.
• Proctors arrive at least 30 minutes before the assessment.
• Proctors ensure the room does not have prohibited items.
• Proctors and testing administrator ensure each station has the appropriate items required for testing: usually noteboards and pens.
• Throughout this process, the testing rooms in the testing center remain locked until they are opened by a proctor. During this time, proctors remain in the hallway outside of 2L-07 and 2L-55 to answer learner questions and ensure learners are not walking through the secured areas prior to the assessment beginning.
  • The assessment team checks Pirate IDs and reminds learners about not having any unauthorized items or devices.
  • Proctors assist learners with finding their seat locations and with any potential log in issues in the testing platform.
• All learners are expected to be in their assigned seats no later than the posted seat time in the learning management system.
• Check-in for assessments begins 9 minutes before the assessment begins.
• A learner will be considered late if they are not in their assigned seat by the posted seat time in the learning management system.
• Any late arriving learner must complete the late arrival slip with the Office of Student Affairs prior to beginning the assessment.
• Learners with testing accommodations follow a schedule matching their accommodation listed in the learning management system.
• Learners should proceed to their assigned seat, log on to the testing platform, and ensure the correct information is displayed.
• Learners having trouble logging on or starting an assessment should alert a proctor for assistance.
• Once all learners have successfully opened the exam platform, the proctor will begin reading the pre-assessment instructions, which covers a full list of unauthorized items, other housekeeping items such as restroom breaks, exam procedures, and a brief description of the assistance proctors can and cannot provide to learners.
• After pre-assessment instructions have been read, the assessment password is released.
• There will be an honor code statement for learners to acknowledge once logged into the computer, prior to beginning all assessments.
• Prior to leaving the testing center, learners should ensure a successful assessment completion and log off completely.
• All learners completing in-person assessments are expected to be available until 8 pm on their assessment day.
• Secure review for internally created assessments:
  • All learners are expected to be in their assigned seats no later than the posted seat time in the learning management system for Secure Reviews. Late arrivals will not be admitted.
  • Occurs immediately after a quiz completion.
  • Occurs within a few days after all learners have completed the examination, at the date and time indicated in the learning management system and block calendars.
  • Proctors in each testing room will have a student clarification form for learners to ask for further clarification from the course director.
  • Learners that are unable to attend secure review, but desired to do so, must attain an approved absence from OSA to schedule a makeup secure review at a later date.
  • Makeup secure reviews must be scheduled within two calendar weeks of the initial secure review date.
  • Attendance will be documented for each secure review.

Administration of NBME Exams

• Learners are expected to be available until 8 PM on in-person assessment days.
• Administration of the NBME subject exams will follow guidelines described by the NBME.
• NBME exams that are required and contribute towards a course or clerkship grade can follow the accommodations procedure provided by Student Disability Support Services.
• Learners are not allowed to make notes on their dry erase noteboards prior to the official start of the first question, this includes the tutorial. Once the exam begins, learners are permitted to make calculations or notes on the dry erase noteboard provided. In the event a new dry erase noteboard is needed, the learner can raise her/his hand and request to exchange the original dry erase noteboard for a new one.
• NBME Testing Irregularities –
  • All testing rooms should be reserved until 8 PM on testing day to accommodate learners.
  • Technical difficulties will be addressed through instructions in the NBME Chief Proctor’s Manual.
  • If exam cannot be immediately restarted, the Testing Administrator will contact the NBME and attempt to restart the exam within 30 minutes.
  • If unsuccessful, the Associate Dean of Medical Education, in conjunction with other appropriate parties, will make the decision whether or not to exit the exam with specific keystrokes, found in the proctor’s manual in each testing room, to keep the first exam from being scored.
  • If the exam is unable to be administered on the scheduled morning, learners will return at 1pm for exam administration.
  • In the event that a major technical difficulty is experienced and is unable to be resolved by the final day of the clerkship, the exam will be administered on the next day of classes. If the technical difficulty occurs at a time that is followed by a significant break (greater than 1 week), every attempt will be made to allow learners the opportunity to take the exam on the Monday of the week of break.

Responsibilities

There are several key stakeholders involved in the testing process. Listed below are each key stakeholder, along with their expected responsibilities and time frame for completion.

Associate Dean of Medical Education

• Makes final decisions regarding any testing irregularities that may require postponement of any assessment.
• Provides back-up to the assessment team as needed.
• Collects assessment adjustment data from Course Directors through the Assessment Performance Report.

Testing Administrator/ Assessment Team Member

• Oversees all day-to-day operations of assessments.
• Communicates with Student Affairs regarding any learner absences and tardies.
• Assists with check-in procedures.
• Day of Assessment:
  • Arrives 45 minutes before the start of the assessment. Oversees the preparation of the rooms for the assessment (lighting, noise machines, cameras), checks testing rooms and restrooms for any prohibited items, posts seating charts outside of rooms, ropes off secure testing area, distributes noteboards, dry erase pens and ear plugs.
  • Provides read aloud instructions for each assessment to proctors along with proctor report template and sign in/out sheet.
  • Responsible for distribution of assessment password to proctors and records start and end times of each assessment.
  • Reviews testing platform functionality post-assessment to identify any irregularities or absences. Reports any absences to OME and OSA stakeholders.
  • Monitors all rooms for technical difficulties, irregularities, and other concerns throughout the assessment.
• Post Assessment:
  • Ensures all noteboards and dry erase pens are collected.
  • Verifies all learners have successfully completed the assessment. Contacts learners if upload issues arise.
  • Reports all testing irregularities on assessments to the Deans in the Office of Medical Education, Course Directors, ECC and, if appropriate, to the NBME.
  • Maintains files of seating arrangements, testing schedules, proctor assignments, test irregularities, and learner breaks.
  • Reports compliance with test development timeline and any irregularities to the Deans in the Office of Medical Education and ECC.
  • Responsible for scheduling all make-up exams in coordination with the Office of Student Affairs.
  • Uploads individual scores and performance profile from NBME to the learning management system for learners to review.
  • Communicates any failures to learners to facilitate remediation plans.
  • Prepares a monthly report of outstanding assessments and coordinates scheduling learners for make-up assessments.

Proctors

• Day of Assessment:
  • Arrives 30 minutes prior to the assessment seat time.
  • Inspects the room for prohibited items and contacts the Test Administrator with any concerns (temperature, lighting, etc.).
  • Ensures the room is set up with all required resources (such as dry erase boards, iPads, and pens) and are appropriately distributed.
  • As Learners Arrive:
    • Instructs learners entering exam room that only their one card and car keys are allowed in the secure testing area. All other personal belongings should be left in a secure area such as lockers or their personal vehicle.
    • Ensures that learners are reminded of all prohibited items, including smart watches, cell phones, etc.
    • Ensures that all learners head and ears are not covered. All headwear is prohibited unless for religious purposes. No unauthorized items are allowed in the secure testing area. Such items include but are not limited to: hats, purses, briefcases, notebooks, notes, resources, pagers, watches, cell phones, recording devices, and photographic equipment.
    • Learners will be required to push back all sleeves and turn out any pockets prior to entering the secure testing area.
    • Verifies that foam earplugs are available to learners. If headphones are preferred, then these must be provided by the learner and inspected by the proctor prior to use. Headphones with Bluetooth (or any sending/receiving capability) will not be allowed.
    • Food and drinks are prohibited in the testing rooms.
    • If a learner wants to bring a snack into the secure testing area to consume during a self-paced break, the food must be in a clear sealable bag or inspected by a proctor at check in.
    • Single use cups will be provided for use at the water fountain outside of the restrooms in the secure testing area.
    • If a learner wants to bring their own beverage, it must be in a label free container and inspected by a proctor at check in.
    • All food and beverages must be placed on the high-top tables directly adjacent to the bathroom facilities in the secure testing area.
    • Learners are not permitted to access any unauthorized items during exam administration.
    • Learners are not allowed to make notes on their dry erase noteboards prior to the official start of the first question. Once the exam begins, learners are permitted to make calculations or notes on the dry erase noteboard provided. In the event a new dry erase noteboard is needed, the learner can raise her/his hand and request to exchange the original dry erase noteboard for a new one.
  • At the Beginning of the Assessment:
    • Verifies all learners are taking the assessment under their correct pirate ID.
    • Reads aloud assessment instructions to learners prior to start of all assessments.
    • Provides assessment password and start time.
    • Records the start and end time on the proctor report.
    • Obtains student headcount.
  • During the Assessment:
    • Proctors are not authorized to answer questions from learners regarding assessment content/format/questions.
    • Monitors and communicates with the Testing Administrator any testing irregularities.
    • Monitors and addresses distractions during assessments.
    • Maintains a standardized log of learner breaks, records the time of departure and return.
    • Ensures that only one learner is allowed in each restroom (male/female) at a given time.
    • Ensures that learners do not leave the secure testing area at any time during the administration unless accompanied by a proctor.
  • After the Assessment:
    • Collects noteboards, dry erase pens, and cleans room.
    • Ensures that all dry erase noteboards are accounted for by seat assignment and returns to the Testing Administrator.
    • Returns completed proctor report and sign in/out sheet to the Testing Administrator.
    • Ensures that all learners have logged off all testing stations.
    • Ensures that testing room doors are closed prior to leaving the testing center.

Student Affairs

For each in-person assessment:

Ensures that a representative from OSA is available, in the testing center, prior to each assessment seat time and 20 minutes after.

Medical Students (Learners)

• General Procedures:
  • Learners must adhere to the Brody School of Medicine Code of Professional Conduct and the Code of Student Conduct as they are always in effect. Any testing irregularities will be brought to the attention of the Deans of OME and OSA and processed accordingly. Any testing irregularity on a NBME exam must be reported to the NBME as outlined in their policy: NBME Security Policy.
  • Learners are expected to be available until 8 PM on all in-person assessment days.
  • During an assessment, learners are only allowed to communicate with staff/faculty.
  • Learners are not permitted to ask proctors, or other personnel, questions regarding assessment content, format, or other questions.
  • Learners must notify proctors if there are any technical issues.
  • Learners are not allowed to erase any noteboards prior to turning them in to the proctor. If a clean noteboard is needed, learners should raise their hand to signify a proctor.
  • Learners are not allowed to leave the testing room with any notes, noteboards, and/or dry erase pens.
  • Unauthorized personal items are prohibited in the secure testing areas. Such items include but are not limited to hats, purses, briefcases, notebooks, notes, resources, pagers, watches, cell phones, recording devices, and photographic equipment. Foam earplugs are allowed. OME will supply disposable earplugs for learners. If headphones are preferred, these must be provided by the learner and inspected by the proctor prior to use. Headphones with Bluetooth capabilities or other sending/receiving capabilities are not allowed.
  • Outerwear must be removed and inspected prior to entry into the secure testing area.
  • Food and drinks are prohibited in the testing rooms.
  • If a learner wants to bring a snack into the secure testing area to consume during a self-paced break, the food must be in a clear sealable bag or inspected by a proctor at check in.
  • Single use cups will be provided for use at the water fountain outside of the restrooms in the secure testing area.
  • If a learner wants to bring their own beverage, it must be in a label free container and inspected by a proctor at check in.
  • All food and beverages must be placed on the high-top tables directly adjacent to the bathroom facilities in the secure testing area.
  • Learners are not allowed to access any unauthorized items during breaks.
  • Learners are required to sign in/out with a proctor prior to taking a break.
  • Learners must not leave the secure testing area at any time during the assessment unless accompanied by a proctor.
  • Along with appropriate attire, shoes are expected to be worn in the testing center at all times.
  • Restrooms in the secure testing area are restricted to only learners actively taking assessments. Only one learner is allowed in each restroom (male/female) at a given time.
• Day of Assessment:
  • A learner will be considered late if they are not in their assigned seat by the posted seat time in the learning management system.
  • For in-person assessments, if a learner arrives late for an unexcused reason, they will need to acquire a late arrival slip from Student Affairs to obtain permission to be admitted into the testing room. The learner’s name, signature, date, and time of arrival will be recorded. This information will be shared with the Promotion and Review Committee/Community Standards Committee. The late arriving learner will be strategically assigned a new seat upon arrival to mitigate distraction for other learners.
  • The late arriving learner will not be given extra time to complete the assessment and will have to forfeit the time they were late. For example, if a learner arrived 30 minutes late for an assessment that was four hours in length, they would only receive 3.5 hours to complete the exam. The testing administrator will be responsible for monitoring start and stop times for late arriving learners.
  • If a learner is granted an excused absence for an exam, they must first complete the REDCap survey (Makeup Request – Foundational Science) to notify the Office of Medical Education. All make-up exams must be taken during the predetermined make-up session identified on the Block calendar.
  • Learners must adhere to the instructions provided by the proctors administering the assessment.
  • Learners are responsible for ensuring that they are taking the correct assessment.
  • Learners must acknowledge the Honor Code prior to starting the assessment.
  • Learners are not allowed to make notes on their noteboard prior to starting their first question. Once learners have started their first question, they are permitted to make calculations or notes only on the noteboard provided. In the event a new noteboard is needed, the learner can raise her/his hand and request to exchange the noteboard for a new one.
  • Learners must ensure that the assessment has been successfully completed prior to leaving the testing center.
  • Learners are required to log off their assigned station prior to leaving the testing center.
• Secure Review:
  • Learners can ask for student clarification forms during the secure review for internally created exams only. The purpose of the student clarification form is to identify learners’ critical thinking/processing of questions and answer choices and to help provide clarifications to learners’ misconceptions.
  • Student clarification forms can only be submitted to the proctor during the secure review and not after.
  • No references, other than those allowed in the examination room, may be utilized for the purpose of substantiating a clarification.
  • Forms that are illegible may not be considered by the course director.
  • All course directors will review item analysis and make determinations for any changes to grade such as dropping questions, accepting multiple answers, wrong answer coded, etc. based on item performance.

Schedule for Review

This procedure is reviewed and approved by the Curriculum Committees, including the Executive Curriculum Committee, Foundational Curriculum Committee, and the Clinical Curriculum Committee every three years.

The Assessment Administration policy is posted to the OME website to allow learners and teaching faculty/administration to reference at any time.

Related Policies

Assessment Creation Policy, Absence Policy

Applicable Laws, Regulations & Standards

LCME Standards for Accreditation of Medical Education Programs Leading to the MD Degree: Published March 2022; Standard 8, Element 2; Standard 8, Element 3; Standard 9, Element 4, and Standard 9, Element 8