Let us look at what PASS can do to assess a student's mark on a written test. PASS produces Student Performance Assessment Graphs (SPAG) containing detailed information about the student's performance on a test. This information helps to identify specific areas where the student is having difficulty.
In order to produce the SPAG for a specific test, you will need to download the Excel grid and follow these basic steps:
- Read the How-to documentation provided to install it.
- Label your test questions according to the categories and variables we provide that you want to measure.
- Transfer this information into the PASS Data Entry Form.
- Input student results in the Entry Form.
- Use the PASS Results Analyzer to treat student results according to categories and variables and generate SPAG.
In this section, we will provide you with information about your test questions. This includes:
- Outlining the categories of the student's performance that can be measured using PASS.
- Describing how to label test questions according to the categories being measured.
What do we mean by a category?
Categories help us to name the elements of the student’s performance that we want to measure. For example, in the category of Class Content, we could measure how the student did on every topic being tested.
What do we mean by a variable?
Each category has a fixed number of variables. It is the variables that allow you to analyze your student's performance in each category. In the category of Class Content, the variables are used to identify each class topic - for example, Pneumonia, Postpartum Hemorrhage.
Categories and Variables
| Categories | Variables |
|---|---|
| Fields of Nursing | Medicine and Surgery Geriatric Medicine Perinatal Care Pediatrics Mental Health |
| Class Content | Fixed number of variables as determined by the topics being tested on each test |
| Professional Component (OIIQ Mosaic) | Assessment Intervention Continuity of Care Not Applicable* |
| Functional Component (OIIQ Mosaic) | Scientific Communication Moral and Professional Ethics Legal Organizational Operational |
| Bloom's Taxonomy | Remember Understand Apply Analyze Evaluate Create |
| Other (to be identified by the teacher) |
* We have included a Not Applicable variable to cover questions that do not fit into any of the Professional Component dimensions.
Let us show you what we've done. Here is an example of a well-labeled question according to the categories and variables that we have chosen to assess.
EXAMPLE
Clinical Situation
Mr. Flanagan, 65 years old, was admitted to your medical unit two days ago. He was diagnosed with COPD (chronic obstructive pulmonary disease).
Today at 08:00, on initial assessment you notice the following: productive cough, dyspnea on exertion, bilateral wheezing throughout the lung fields, and restlessness. His vital signs are: BP 140/90, P 90, RR 28 regular and shallow, O2 saturation 87% on O2 2L/min, T 37C PO.
Medical orders include:
- O2 via nasal prongs to maintain O2 saturation ≥ 92%
- Capillary blood gases qAM
Learning Outcome
Generate actual and potential nursing diagnoses that can arise in persons with COPD.
Categories and Variables
- Field of Nursing: Medicine & Surgery
- Class Content: Chronic Obstructive Pulmonary Disease
- Professional Component: Assessment
- Functional Component: Scientific
- Bloom’s Taxonomy: Apply
Question
Identify the nursing problem.
Answer
Ineffective Gas Exchange or low oxygen saturation or hypoxia
Reference
Lewis, S., Bucher, L., Heitkemper, M., Harding, M., Barry, M., Lok, J., Tyerman, J., & Goldsworthy, S. (2019). Medical-Surgical Nursing in Canada, 4th ed. Toronto: Elsevier, p. 681.
Statistics
| A'18 | A'19 | A'20 | A'21 | |
| Difficulty Index | .63 | .72 | .66 | |
| Discrimination Index | .09 | .21 | .20 |
Now, let us show you how to label your test questions.
Step-by-Step Procedure for Labelling your Test Questions
Step 1 – Decide which categories and related variables you want to assess
It is up to you and/or your team to determine the categories that are most applicable to the test you are preparing.
The more categories you choose, the more comprehensive the analysis you generate for the student. However, the depth of the analysis should be adjusted to the course level. For example, a first-year nursing student may be overwhelmed if given too much data about their performance.
If you are just beginning to label your questions, start with a small number of categories
Below are the categories and related variables that the PASS Excel grid is designed to assess.
Category
Fields of Nursing
Variables
Label the question with the Field of Nursing being tested, using the following variables:
- Medicine & Surgery
- Geriatric Medicine
- Perinatal Care
- Pediatrics
- Mental Health
This category can be used when there is more than one field of nursing included on a test – for example, a nursing course that includes Geriatric Medicine and Mental Health. The clinical course is composed of two rotations: a rotation in a geriatric setting and a rotation in a mental health setting. Students rotate – some complete their geriatric rotation first while others complete their mental health rotation first. However, the first test includes theory from both fields: geriatrics and mental health. By including this category in your PASS analysis, you will be able to determine if there is a relationship between the student’s clinical placement and their performance on theory in that Field of Nursing.
The Fields of Nursing category is also useful for a comprehensive test, when all Fields of Nursing are being tested. We do not assess this category when only one Field of Nursing is being tested – for example, in a Medicine & Surgery course.
Category
Class Content
Variables
Label the question with the name of the class or the topic that is being tested. Here are some examples of topics in each of the fields:
- Medicine & Surgery: diabetes, postoperative care
- Geriatric Medicine: delirium, falls assessment
- Perinatal Care: pregnancy-induced hypertension, newborn assessment
- Pediatrics: childhood asthma, vaccinations
- Mental Health: schizophrenia, depression
The number of variables will be determined by the topics you are testing.
Category
Professional Component (OIIQ Mosaic, 2009, pp. 18–23)
Variable
Label the question using the dimensions of the Professional Component listed below:
- Assessment
- Intervention
- Continuity of Care
- Not Applicable*
The Professional Component category is specific to Nursing and provides information on how the student performed in relation to different nursing activities. The OIIQ Mosaic describes three separate groups of professional dimensions: assessment, intervention, and continuity of care. If you need a refresher on how to label a question in terms of Professional Component dimensions, refer to the Glossary or to the OIIQ Mosaic.
*In addition, we have included a “Not Applicable” variable, to cover questions that do not fit into any of the Professional Component dimensions. We use the Not Applicable variable for questions that do not relate directly to a nursing activity. Examples include testing knowledge of pathophysiology and testing application of protocols/principles.
Category
Functional Component (OIIQ Mosaic, 2009, pp. 12–17)
Variable
Label the question using the Fields of Knowledge of the Functional Component listed below:
- Scientific
- Communication
- Moral and Professional Ethics
- Legal
- Organizational
- Operational
We consider this to be the most difficult category to assign the variables. As questions become more complex, it is possible that more than one Field of Knowledge is being tested in the same question. An example would be testing a student’s ability to describe the purpose of a new medication to the client. This question encompasses both the Scientific Field of Knowledge and the Communication Field of Knowledge. We suggest labelling the question according to the predominant Field of Knowledge.
We suggest incorporating the Functional Component only when you’ve become comfortable with using the other categories.
For help on how to label a question in terms of the Functional Component Fields of Knowledge, refer to the Glossary or to the OIIQ Mosaic.
Category
Bloom’s Taxonomy
Variable
Label the level of question according to Bloom’s Taxonomy listed below:
- Remember
- Understand
- Apply
- Analyze
- Evaluate
- Create
Bloom’s Taxonomy Framework
We have chosen the revised Bloom’s Taxonomy. This revised terminology reflects the view that thinking is an action. It provides a structure for the rating of thinking skills from simple to complex (Armstrong, 2019).
When testing new content, we do use some lower-level questions (Remember and Understand). As our students move towards completion of the program, we try to increase the percentage of higher-level questions with each semester. One of our goals as nursing teachers is to prepare students for the OIIQ professional exam, which "assesses the integration of knowledge, skills, attitudes and judgment required to make clinical decisions and intervene appropriately in clinical situations" (OIIQ, 2018, p.12).
As teachers, we sometimes have difficulty distinguishing between a Remember-level and an Understand-level question. While we strive to be as accurate as possible, we have found that the strategies to help the student are very similar for the two levels of thinking. Therefore, we are not overly concerned if a question is occasionally mislabelled at the Remember or Understand level. In Nursing, we are more concerned with the teacher’s ability to recognize the difference between Apply-level and Analyze-level questions, as these measure more discrete functions.
By being able to identify the level of questions that the student has difficulty answering, we will be able to guide the student in the kinds of strategies that could help. We will discuss strategies in Strategies for Success.
Refer to the Glossary for definitions of the levels of Bloom’s Taxonomy.
Category
Other
The PASS Excel grid allows you to customize the list of categories by including one or more additional categories that you might want to assess.
Variable
The list of variables will be determined by what you choose to assess in these categories.
The greater the consensus on how questions are labelled, the more accurate the analysis, the more useful the feedback to the student.
Step 2 – Prepare your test questions
Here is the format that we recommend for preparing test questions.
Clinical Situation (if applicable)
The OIIQ suggests that all questions be derived from a clinical situation.
Learning Outcome
Select the learning outcome being tested.
Variables
Include the variables you chose in Step 1
- For Fields of Nursing, choose a field
- For Class Content, choose a topic and assign an abbreviation
- For Professional Component, assign a dimension
- For Functional Component, assign a field of knowledge
- For Bloom’s Taxonomy, assign a level of question
Question
Draft the item in either a question or a statement format.
Answer
For short-answer format, draft the essential elements of the answer. For multiple-choice format, draft the answer and the distractors.
Reference
Cite the reference for your answer based on the current nursing literature that is available to students.
Statistics (optional)
After each test, record the statistics for each question. We use Difficulty Index and Discrimination Index (Tarrant & Ware, 2012).
- Difficulty Index: the proportion of examinees who answered the item correctly
- Discrimination Index: the difference in the number of high-achieving and low-achieving examinees who answered the question correctly
Options
Now that we have shown you how to label and prepare your test questions, you have two options:
- Review more examples of labeled questions:
- Using Excel to Help Your Students Excel, where we will demonstrate how to use the PASS Excel grid.