Aratohu: Te arotake hanganga, panoni hoanga hoki Guidance: Evaluating structural system architecture and design changes

Te Kawa Mataaho Public Service Commission would like to acknowledge the work of Isabelle Collins, Andrea Knox and Viv Smith of the Social Policy Evaluation and Research Unit (Families Commission) who originally developed this material for us in 2017.

Evaluation helps us to learn what works well, and under what circumstances, and it should be an integral part of any major change. Changes can be costly and disruptive, and so it is vital to understand whether they have achieved their objectives, and to identify any lessons for the future.

In relation to system architecture and design, evaluation can look at:

• what contribution a structural change or joint initiative made to target outcomes and unintended outcomes
• the value for money, cost-effectiveness, or cost-benefit that has been realised from the structural change or joint initiative
• how, and how well the structural change or joint initiative has been implemented.

With this knowledge, we are better able to identify and prioritise next steps, and to make further improvements.

Evaluation should be designed into the structural change at the outset.

It can be tempting for agencies to delay evaluation in favour of getting on with implementation, especially if the new initiative is urgent, but this can be a false economy. What you learn from the evaluation’s set-up and early findings will help to improve the initiative’s design, and to identify and address problems that arise. An early start to the evaluation will also show what data should be gathered as the initiative progresses, to show effectiveness.

There are some principles of evaluation quality and ethics.

• The Evaluation Standards for Aotearoa New Zealand describe principles for conducting evaluation with integrity (Superu & ANZEA, 2016).
• The Australasian Evaluation Society Guidelines for the Ethical Conduct of Evaluations cover ethical behaviour and decision-making in evaluation (Australasian Evaluation Society, 2013).

When beginning an evaluation, you should investigate the ethics requirements at your agency. Evaluations that will collect information about individual people will probably need ethics approval. Even if approval is not required, you should still think about how to manage any ethical issues that are likely to arise. There are a number of committees responsible for oversight of these issues, such as the Health and Disability Ethics Committees.

Home — Health and Disability Ethics Committees

This glossary uses a mix of legal definition, conventional interpretation and additional description to help you understand system design and architecture terms.