Creating patient prioritization tools

Use 1000minds to create a consensus-based tool or system for prioritizing patients for access to health care, e.g. elective surgery.

The process outlined below shows how you can use 1000minds to create a consensus-based tool or system for prioritizing patients for access to health care, e.g. elective surgery. It makes use of three types of 1000minds survey: “ranking”, “categorization” and “preferences”. A voting option is also available to supplement a preferences survey.

The steps in the process can be adapted to create your own patient prioritization tool, depending on your needs. For example, if you’re in a hurry, steps 1-5 could be dispensed with – assuming you already know you need a new tool and the criteria you want to use – but do you really?

1. Show the need for a new tool

  1. 1.1 Collate 10-15 patient vignettes (case studies) and run a ranking survey – a “noise audit” – to capture individual clinicians’ intuitive rankings.
  2. 1.2 Show them how variable they are (e.g. see “Noisy” expert judgments), and that, therefore, a new prioritization tool is needed!

2. Elicit prioritization criteria

  1. 2.1 As the group of clinicians discusses the ranking survey results – i.e. which patients should be prioritized and why – start specifying the prioritization criteria.
  2. 2.2 Run the same ranking survey but this time by consensus to get a pseudo-gold standard for step #8 later.

3. Refine the criteria (and their levels)

  1. 3.1 Refine the criteria and levels – e.g. based on the literature and available evidence/experience.
  2. 3.2 Get the number of levels and the wording right!

4. Test the criteria / levels

  1. 4.1 Run a categorization survey for the clinicians to rate the patient vignettes on the criteria.
  2. 4.2 Show them how consistent they are (inter-rater reliability).
  3. 4.3 Further refine the criteria (and their levels), and finalise them.

5. Average / consensus rating

Resolve disagreements in the ratings of the patients on the criteria using the rating-averaging feature in 1000minds or by reaching consensus in preparation for step #8 later.

6. Individual judgments

  1. 6.1 Run a preferences survey for clinicians to experience making trade-offs between hypothetical patients rated on the criteria.
  2. 6.2 Show them similarities and differences in their weights.

7. Consensus weighting

  1. 7.1 Weight the criteria / levels by group consensus – using 1000minds’ voting option.
  2. 7.2 Alternatively, use the mean of their individual weights (previous step).

8. Validate the tool

  1. 8.1 Compare the ranking of the patient vignettes from the initial consensus (the pseudo-gold standard) at step #2 with the ranking produced by the tool.
  2. 8.2 Perform other tests of (face) validity and reliability.

9. Implement the tool

  1. 9.1 Implement the tool in information systems or a prioritization web service (e.g. see NZ’s Ministry of Health).
  2. 9.2 Socialize the tool with clinicians who will use it.

The process summarized above is illustrated in this article: