Evaluation

Measuring the impact of changes 

After implementing a change to practice or processes you will need to establish whether the intended effect has been achieved. Generally, this would be in comparison to no change or usual care. It is also important that you identify any unintended consequences of the changes. Evaluation will help you to do this. 

Evaluation is a formal systematic process to assess the worth and merit of what has been done. [1]

Using a framework suited to health care services and programs provides structure and facilitates the evaluation process in clinical practice. [2] This supports reporting of processes and outcomes.

In their review of evaluation frameworks, Bradford 2018 noted that evaluation frameworks:

• Simplify a complex process
• Promote meaningful evaluation
• Identify mechanisms driving or inhibiting change

Determining the impact and effectiveness of intervention or activity

The purpose of knowledge translation activities is to support the use of research evidence in practice. To encourage this, many projects now look at communication and implementation strategies. To determine whether implementation activities have been successful, it is important to be able to measure changes. Tracking changes can also enable plans to be modified or adapted if circumstances change or if elements are not working as intended.

There are many measures that can be used as part of the evaluation of a knowledge translation activity. Measures can look at how the process is being implemented and delivered. They can focus on whether people’s knowledge or attitudes change. Measures can also be used to monitor whether behaviour, such as prescribing habits, change. these change measures can be applied to individual clinicians or to an organisational group such as a multidisciplinary team or a health service. [3]

Evaluation plans

When planning how to implement a change, it is also worthwhile to consider an evaluation strategy. Evaluation involves collecting and analysing information and data to measure the impact and effectiveness of the intervention or activity against the purpose of the activity. It helps to focus what you are expecting to happen and how you can determine the results of what you do by looking at appropriate measures.

Evaluation generally reflects a series of familiar questions and then looks at how to find the answers:

1. Did we do what we said we would do? (Accountability reporting)
2. As we go along, are there things we could change to make it more useful? (Formative evaluation)
3. What happened as a result of what we did? (Summative evaluation. Can also include impact and outcome assessment)
4. What would we do differently next time? (Feedback loops in quality improvement)
5. How can these findings be used to further develop what we are trying to achieve? (Ongoing). [4]

The type of evaluation approach and measures used will be determined by the specific project or activity.

Data sources

When planning the measures to show changes, it can be useful to identify what data are collected as part of normal practices. For example, many services are involved with the Palliative Care Outcomes Collaboration (PCOC). The PCOC Data Set includes information about, the person receiving care, administrative episode details about the period of care, and clinical characteristics. Many hospitals collect data as part of quality assurance processes. These could be quite relevant in measuring change in a project to reduce evidence-practice gaps.