Skip to main content Skip to navigation
Back to top anchor
Home Department of the Prime Minister and Cabinet (DPMC)
Page url
  • Events
  • Media
  • Contact us
  • Home
Open search Close search
  • Contact us
    • How to make an Official Information Act request
      • Tips for requesting information
      • Guidance on how we will respond
  • About DPMC
    • Who we are
      • Our purpose
      • Our strategic intentions
      • How we deliver
    • Our work
      • Supporting the Prime Minister and Cabinet
      • Supporting the Governor-General
      • Other roles and responsibilities
    • Our history
      • Business units and programmes
    • Careers
  • Our business units
    • Cabinet Office
      • Roles and responsibilities
        • Statutory and formal responsibilities
      • Supporting the work of the Cabinet
        • Cabinet Committees
        • Elections
        • Cabinet Manual
          • Publication information
          • Foreword
          • Preface
          • Introduction
          • 1. Sovereign, Governor-General, and Executive Council
            • Introduction
            • Sovereign of New Zealand
            • Governor-General
            • Executive Council
            • Clerk of the Executive Council
            • Meetings of Executive Council
            • Gazetting and entry into force
            • Announcement
            • New Zealand Royal Honours System
            • Heraldry
            • Order of Precedence
            • Related Information
          • 2. Ministers of the Crown: Appointment, Role, and Conduct
            • Introduction
            • Prime Minister
            • Deputy Prime Minister
            • Ministers
            • Parliamentary Under-Secretaries
            • Parliamentary Private Secretaries
            • Conduct, public duty, and personal interests
            • Interactions with representatives from non-government or commercial organisations
            • Personal email accounts and phone numbers
            • Gifts and awards
            • Speaking engagements, endorsements, and non-ministerial activities
            • Government advertising and publicity guidelines
            • Ministerial travel
            • Related Information
          • 3. Ministers of the Crown and the Public Sector
            • Introduction
            • The public service and the public sector
            • Ministers and the public service
            • Ministers and Crown entities
            • Ministers and companies in the public sector
            • Integrity and conduct throughout the public sector
            • Related Information
          • 4. Ministers, the Law, and Inquiries
            • Introduction
            • Attorney-General
            • Comment by Ministers on judicial decisions
            • Crown legal business
            • Litigation involving Ministers
            • Legal advice and legal professional privilege
            • Inquiries
            • Related Information
          • 5. Cabinet Decision-making
            • Introduction
            • Cabinet
            • Cabinet committees
            • Principles of Cabinet decision-making
            • Cabinet and Cabinet committee procedures
            • Secretary of the Cabinet and the Cabinet Office
            • Related Information
          • 6. Elections, Transitions, and Government Formation
            • Introduction
            • The electoral cycle
            • Transitions following an election
            • Caretaker convention
            • Government formation
            • Mid-term transitions
            • Early election
            • Provision of information by the state sector during transitions
            • Related Information
          • 7. The Executive, Legislation, and the House
            • Introduction
            • Speech from the Throne
            • Prime Minister's statement
            • Government legislation programme
            • Revision Bills
            • Law Commission work programme
            • Development and approval of bills
            • Secondary legislation
            • Ministers and the House
            • Ministers and select committees
            • Parliamentary treaty examination
            • Crown's financial veto
            • Citizens initiated referenda
            • Declarations of inconsistency
            • Related information
            • Related information
          • 8. Official Information and Public Records
            • Introduction
            • Information held by government
            • Official Information Act 1982
            • Privacy Act 1993
            • Ombudsmen Act 1975
            • Providing information to select committees
            • Production or discovery of official documents
            • Requests for parliamentary information
            • Ministers and Public Records
            • Disclosure and use of official information by former Ministers
            • Convention on access to Cabinet records of a previous administration
            • Related Information
          • Appendix A: The Treaty of Waitangi/Te Tiriti o Waitangi
          • Appendix B: Letters Patent Constituting the Office of Governor-General of New Zealand 1983
          • Index
        • History of the Cabinet Manual
      • Ministers and their portfolios
        • Ministerial List
        • Ministerial portfolios
          • Briefings to incoming ministers (BIMs)
          • Directory of Ministerial portfolios
        • Register of Assigned Legislation
        • Delegations
        • Ministers' interests
          • Ministers' interests 2020-2021
          • Ministers' Interests 2019-2020
          • Ministers' Interests 2018-2019
          • Ministers' interests 2017-2018
          • Ministers' interests 2016-2017
          • Ministers' interests 2015-2016
          • Ministers' interests 2014-2015
          • Ministers' interests 2013-2014
          • Ministers' interests 2012-2013
      • Executive Council
      • The Honours Unit
    • National Security Group
      • National Security Systems
      • National Assessments Bureau
        • Working for the National Assessments Bureau
      • National Security Policy
        • National Cyber Policy Office
      • National Intelligence and Risk Coordination
      • National Security Workforce
    • Policy Advisory Group
      • Roles and responsibilities
    • Implementation Unit
      • Latest news and updates
    • Strategy, Governance and Engagement Group
    • Child Wellbeing and Poverty Reduction Group
    • COVID-19 Group
    • Government House
  • Departmental agency
    • NEMA
      • Hurunui/Kaikōura Earthquakes Recovery Act 2016
      • Ministerial Review: Better Responses to Natural Disasters and Other Emergencies in New Zealand
  • Our programmes
    • New Zealand Royal Honours
      • Make a nomination
        • Nominations for Honours
        • New Zealand Royal Honours nomination guidelines
        • Nomination Form and Guide to making Nominations
        • Bravery Award Nominations
        • The New Zealand Antarctic Medal: Nomination Guidelines
      • Honours lists and recipients
        • Honours lists
          • Current Members of the Order of New Zealand
          • Knights and Dames of the New Zealand Order of Merit
          • Knights and Dames of the Orders of Chivalry
          • The New Zealand Gallantry Awards
          • The New Zealand Bravery Awards
          • The New Zealand Antarctic Medal
          • The Privy Council
        • Titles and styles of knights and dames
        • Wearing of insignia
          • Order of wear: orders, decorations and medals in New Zealand
          • Method of mounting insignia
      • The New Zealand Royal Honours system
        • Overview of the New Zealand Royal Honours system
        • Forfeiture of Honours
        • History
        • Components of the New Zealand Royal Honours system
          • The Order of New Zealand
          • The New Zealand Order of Merit
          • The Queen's Service Order
          • Gallantry and Bravery Awards
            • New Zealand Gallantry Awards
            • New Zealand Bravery Awards
          • The New Zealand Antarctic Medal
          • The New Zealand Distinguished Service Decoration
          • Other distinctive New Zealand Honours
          • "The Right Honourable"
            • The Right Honourable in New Zealand
          • "The Honourable"
            • Roll of The Honourables
          • Privy Council
        • Order of Wear
        • Rules relating to the acceptance and wearing of Foreign Honours
        • New Zealand Herald of Arms Extraordinary
      • New Zealand Royal Honours Image Gallery
      • New Zealand Royal Honours FAQs
    • The Policy Project
      • About the Policy Project
      • How the Policy Project can help you
      • Policy improvement frameworks
        • Policy quality
        • Policy skills
          • Development Pathways
            • Use the tool
            • How we developed the Tool
        • Policy capability
      • Policy Methods Toolbox
        • Start Right
          • Commissioning Conversations
          • Exploration
          • Green Light
        • Design thinking
          • Journey mapping
          • Role play
          • Prototyping
          • Experience interviews
          • Personas
        • Behavioural insights
          • Behavioural Change Models
        • Community engagement
          • Citizen juries
        • Futures thinking
          • Mythbusters
          • Horizon Scanning
          • Assumption Testing
          • Futures Wheel
          • Scenarios
          • Wind Tunnelling
          • Backcasting
        • Treaty of Waitangi analysis
      • Long-term Insights Briefings
      • Policy advice themes
        • Commissioning a policy project
        • Communicating policy advice
        • Engagement
        • Evidence and evaluation
        • Free and frank advice
        • Innovation
        • Policy and law
        • Stewardship
      • Case studies
        • Inland Revenue Case Study
        • Ministry of Transport Case Study
        • Land Information New Zealand Case Study
      • Policy community
        • Policy system leadership
          • Head of the Policy Profession
          • The Policy Profession Board
          • Tier 2 Policy Leaders' Network
          • Policy capability leads
        • Policy managers
        • Policy practitioners
        • Ways to connect to the policy community
          • The New Zealand community
          • The international community
        • Open Government Partnership
      • Progress and performance
      • Publications
      • Policy community events
      • Blog
      • Navigator
    • Greater Christchurch recovery and regeneration
      • Greater Christchurch Group
        • Roles and responsibilities
          • Disestablishment of CERA
        • External websites
      • The Greater Christchurch Regeneration Act 2016
      • Our regeneration partners
      • Global Settlement Agreement with Christchurch City Council
      • Recovery and regeneration plans
        • Cranford Regeneration Plan
        • Ōtākaro Avon River Corridor Regeneration Plan
        • Lyttelton Port Recovery Plan
        • Residential Red Zone Offer Recovery Plan
        • Land Use Recovery Plan
        • Christchurch Central Recovery Plan
      • Section 71 proposals
        • Commercial film or video production facilities in Christchurch – Section 71 Proposal
        • Hagley Oval
        • Lyttelton Commercial Zone Parking
        • Yaldhurst Recreation and Sports Facility
        • Residential Unit Overlay District Plan Changes
        • Redcliffs School relocation
      • Christ Church Cathedral Reinstatement Project
      • Past Orders in Council
    • National security
      • New Zealand’s national security system
        • National security objectives
        • National security principles
        • Coordination across government
        • DPMC’s role
        • Role of central government
        • National Security Systems Handbook
        • National security governance structure
          • ODESC governance boards
          • Working Groups and committees
          • Lead and support agencies
        • National security system during a crisis
          • ODESC
          • Watch Groups
          • Inter-agency Working Groups and Specialist Groups
      • National Risk Approach
        • New Zealand's Nationally Significant Risks
      • National Security Intelligence Priorities
      • Counter-terrorism
        • He Whenua Taurikura
          • He Whenua Taurikura Governance Board
          • He Whenua Taurikura Hui
            • He Whenua Taurikura Hui 2021
            • He Whenua Taurikura Hui 2022
          • Master’s Scholarship
            • Master's Scholarship 2021/2022
            • Master's Scholarship FAQs
          • He Whenua Taurikura Co-Directors
        • New Zealand's Countering Terrorism and Violent Extremism Strategy
        • Preventing and Countering Violent Extremism Strategic Framework
        • Preventing and Countering Violent Extremism Fund
        • Terrorist and National Security Event Media Protocols
      • Countering foreign interference
      • Cyber Security Strategy
        • Cyber ransom advice
        • United Nations Cybercrime Treaty
          • We asked, you said, we did
      • Intelligence and Security Act 2017
        • Intelligence and Security Act Review
        • Why did we need new legislation?
        • What is intelligence?
        • Defining national security
        • Bringing the agencies further into the public service
        • Powers and warrants
          • Activities allowed under intelligence warrants
          • The authorisation framework
          • Protections for New Zealanders
        • Strengthening oversight of NZSIS and GCSB
        • Whistleblowing and protecting classified information
        • Cover and immunities
          • Cover and assumed identity arrangements
          • Immunities from legal liabilities
        • Sharing information
          • Domestic information sharing
          • Arrangements with foreign partners
        • The National Assessments Bureau
        • Case studies
          • Counter-terrorism
          • Investigating a suspected terrorist
          • Counter-espionage
          • Cyber security in action
        • Resources
      • Strengthening resilience to disinformation
      • Aotearoa's National Security Strategy
      • National Security Long-term Insights Briefing
        • LTIB Topic Consultation Summary
        • 2022 National Security Survey Report
      • Royal Commission of Inquiry into the terrorist attack on Christchurch masjidain
        • The Government’s Response
        • Working with Communities - get involved
          • Kāpuia - Ministerial Advisory Group Nominations
        • Response Steering Group
        • Response Progress
        • Measuring Success
        • Improving Community Engagement
        • Key Documents
        • Get in touch
      • Five Country Ministerial
    • Child and youth wellbeing strategy
      • Publications and resources
    • Reducing child poverty
      • Formal gazetting of targets for reducing child poverty
    • Special programmes
      • Kāpuia – Ministerial Advisory Group
        • Advice from Kāpuia
        • Kāpuia Pānui
      • Health and Disability Review Transition Unit
      • COVID-19 Proactive Releases
      • Prime Minister's Chief Science Advisor archives
        • Archive
          • Professor Sir Peter Gluckman 2009-2011
          • Professor Sir Peter Gluckman 2011-2013
          • Professor Sir Peter Gluckman 2013-2015
          • Professor Sir Peter Gluckman 2015-2018
          • Professor Dame Juliet Gerrard 2018-2021
      • Historical Programmes
        • Inquiry into the EQC
        • Connect Smart
        • Canterbury Earthquake Recovery Learning and Legacy Programme
        • Ministerial Committee on Poverty
        • Flag consideration project
        • Christ Church Cathedral Working Group
        • Tackling methamphetamine
  • Publications
  • Browse all sections
  • Publications
  • News and notices
  • Annual reports
  • CabGuide
  • Cabinet Manual
  • Contact us
    • How to make an Official Information Act request
      • Tips for requesting information
      • Guidance on how we will respond
  • About DPMC
    • Who we are
      • Our purpose
      • Our strategic intentions
      • How we deliver
    • Our work
      • Supporting the Prime Minister and Cabinet
      • Supporting the Governor-General
      • Other roles and responsibilities
    • Our history
      • Business units and programmes
    • Careers
  • Our business units
    • Cabinet Office
      • Roles and responsibilities
        • Statutory and formal responsibilities
      • Supporting the work of the Cabinet
        • Cabinet Committees
        • Elections
        • Cabinet Manual
        • History of the Cabinet Manual
      • Ministers and their portfolios
        • Ministerial List
        • Ministerial portfolios
        • Register of Assigned Legislation
        • Delegations
        • Ministers' interests
      • Executive Council
      • The Honours Unit
    • National Security Group
      • National Security Systems
      • National Assessments Bureau
        • Working for the National Assessments Bureau
      • National Security Policy
        • National Cyber Policy Office
      • National Intelligence and Risk Coordination
      • National Security Workforce
    • Policy Advisory Group
      • Roles and responsibilities
    • Implementation Unit
      • Latest news and updates
    • Strategy, Governance and Engagement Group
    • Child Wellbeing and Poverty Reduction Group
    • COVID-19 Group
    • Government House
  • Departmental agency
    • NEMA
      • Hurunui/Kaikōura Earthquakes Recovery Act 2016
      • Ministerial Review: Better Responses to Natural Disasters and Other Emergencies in New Zealand
  • Our programmes
    • New Zealand Royal Honours
      • Make a nomination
        • Nominations for Honours
        • New Zealand Royal Honours nomination guidelines
        • Nomination Form and Guide to making Nominations
        • Bravery Award Nominations
        • The New Zealand Antarctic Medal: Nomination Guidelines
      • Honours lists and recipients
        • Honours lists
        • Titles and styles of knights and dames
        • Wearing of insignia
      • The New Zealand Royal Honours system
        • Overview of the New Zealand Royal Honours system
        • Forfeiture of Honours
        • History
        • Components of the New Zealand Royal Honours system
        • Order of Wear
        • Rules relating to the acceptance and wearing of Foreign Honours
        • New Zealand Herald of Arms Extraordinary
      • New Zealand Royal Honours Image Gallery
      • New Zealand Royal Honours FAQs
    • The Policy Project
      • About the Policy Project
      • How the Policy Project can help you
      • Policy improvement frameworks
        • Policy quality
        • Policy skills
        • Policy capability
      • Policy Methods Toolbox
        • Start Right
        • Design thinking
        • Behavioural insights
        • Community engagement
        • Futures thinking
        • Treaty of Waitangi analysis
      • Long-term Insights Briefings
      • Policy advice themes
        • Commissioning a policy project
        • Communicating policy advice
        • Engagement
        • Evidence and evaluation
        • Free and frank advice
        • Innovation
        • Policy and law
        • Stewardship
      • Case studies
        • Inland Revenue Case Study
        • Ministry of Transport Case Study
        • Land Information New Zealand Case Study
      • Policy community
        • Policy system leadership
        • Policy managers
        • Policy practitioners
        • Ways to connect to the policy community
        • Open Government Partnership
      • Progress and performance
      • Publications
      • Policy community events
      • Blog
      • Navigator
    • Greater Christchurch recovery and regeneration
      • Greater Christchurch Group
        • Roles and responsibilities
        • External websites
      • The Greater Christchurch Regeneration Act 2016
      • Our regeneration partners
      • Global Settlement Agreement with Christchurch City Council
      • Recovery and regeneration plans
        • Cranford Regeneration Plan
        • Ōtākaro Avon River Corridor Regeneration Plan
        • Lyttelton Port Recovery Plan
        • Residential Red Zone Offer Recovery Plan
        • Land Use Recovery Plan
        • Christchurch Central Recovery Plan
      • Section 71 proposals
        • Commercial film or video production facilities in Christchurch – Section 71 Proposal
        • Hagley Oval
        • Lyttelton Commercial Zone Parking
        • Yaldhurst Recreation and Sports Facility
        • Residential Unit Overlay District Plan Changes
        • Redcliffs School relocation
      • Christ Church Cathedral Reinstatement Project
      • Past Orders in Council
    • National security
      • New Zealand’s national security system
        • National security objectives
        • National security principles
        • Coordination across government
        • DPMC’s role
        • Role of central government
        • National Security Systems Handbook
        • National security governance structure
        • National security system during a crisis
      • National Risk Approach
        • New Zealand's Nationally Significant Risks
      • National Security Intelligence Priorities
      • Counter-terrorism
        • He Whenua Taurikura
        • New Zealand's Countering Terrorism and Violent Extremism Strategy
        • Preventing and Countering Violent Extremism Strategic Framework
        • Preventing and Countering Violent Extremism Fund
        • Terrorist and National Security Event Media Protocols
      • Countering foreign interference
      • Cyber Security Strategy
        • Cyber ransom advice
        • United Nations Cybercrime Treaty
      • Intelligence and Security Act 2017
        • Intelligence and Security Act Review
        • Why did we need new legislation?
        • What is intelligence?
        • Defining national security
        • Bringing the agencies further into the public service
        • Powers and warrants
        • Strengthening oversight of NZSIS and GCSB
        • Whistleblowing and protecting classified information
        • Cover and immunities
        • Sharing information
        • The National Assessments Bureau
        • Case studies
        • Resources
      • Strengthening resilience to disinformation
      • Aotearoa's National Security Strategy
      • National Security Long-term Insights Briefing
        • LTIB Topic Consultation Summary
        • 2022 National Security Survey Report
      • Royal Commission of Inquiry into the terrorist attack on Christchurch masjidain
        • The Government’s Response
        • Working with Communities - get involved
        • Response Steering Group
        • Response Progress
        • Measuring Success
        • Improving Community Engagement
        • Key Documents
        • Get in touch
      • Five Country Ministerial
    • Child and youth wellbeing strategy
      • Publications and resources
    • Reducing child poverty
      • Formal gazetting of targets for reducing child poverty
    • Special programmes
      • Kāpuia – Ministerial Advisory Group
        • Advice from Kāpuia
        • Kāpuia Pānui
      • Health and Disability Review Transition Unit
      • COVID-19 Proactive Releases
      • Prime Minister's Chief Science Advisor archives
        • Archive
      • Historical Programmes
        • Inquiry into the EQC
        • Connect Smart
        • Canterbury Earthquake Recovery Learning and Legacy Programme
        • Ministerial Committee on Poverty
        • Flag consideration project
        • Christ Church Cathedral Working Group
        • Tackling methamphetamine
  • Publications
  • Browse all sections
  1. Home
  2. Publications
  3. Policy Quality Framework – a guide on panels and processes for assessing policy advice papers
Resources

Policy Quality Framework – a guide on panels and processes for assessing policy advice papers

Publication type:
Resources
Published by:
Policy Project
Part of:
Policy Project
Issue status:
Current
Issue date:
Tuesday, 23 July 2019
Additional details
Publication category:
Resources
Last updated:
Monday, 30 August 2021
Copyright:
© Crown Copyright, Creative Commons Attribution 4.0 International

This guide contains advice on how to run a panel-based assessment of policy and other advice papers after their delivery (an ex-post assessment process) using the Policy Quality Framework. It also explains how to score advice papers using the Policy Quality Framework's paper-scoring template. The template sets out the Policy Quality Framework's standards in a format that helps in assessing the quality of an agency's written advice, and whether it is fit for purpose.

This document is an editable PDF. The PDF can be completed electronically and then printed, or you can print it out and write on the form manually. Please download the PDF and open it from your computer before using and printing, as it contains pages in both A4 and A3 that may not print correctly from the web browser.

The HTML is provided for accessibility purposes. Please do not print the HTML or use it as the primary source for this information. The PDF is the primary product and should be used for distribution and printing.

Formats

  • Policy Quality Framework: A guide on panels and processes for assessing policy advice papers
    PDF
    6.25 MB

About this guide#

This guide contains advice on how to run a panel-based assessment of policy and other advice papers after their delivery (an ex-post assessment process) using the Policy Quality Framework. It also explains how to score advice papers using the Policy Quality Framework’s paper scoring template. The template sets out the Policy Quality Framework’s standards in a format that helps in assessing the quality of an agency’s written advice, and whether it is fit for purpose. This guide can be found on the Policy Quality Framework webpage. Its purpose is to support government agencies in understanding, reporting and improving their policy advice performance.

Policy Quality Framework: standards for quality policy advice#

The framework sets out seventeen elements of quality policy advice, organised under four standards. These standards can be applied by panels to assess papers and decide whether they are fit for purpose. See Appendix 1 for the full version of the Policy Quality Framework, which provides more detail on each of the elements below.

The Policy Quality Framework sets out seventeen elements of quality policy advice, organised under four standards.

 

Why assessing policy advice is important#

Estimates and annual reporting#

From 2019/20, all government agencies are required to report in their Estimates and annual reports on the numeric score for the quality of their policy advice, using the Policy Quality Framework. This requires an agency to assess the quality of the advice it delivers during the year. Agencies choose to do this by either establishing their own assessment panels or contracting out the assessment process. 

Assess to improve#

Beyond holding agencies to account for their performance, using the Policy Quality Framework to assess papers after delivery is also vital in improving performance – and thereby supporting better government decision making.

At the individual agency level, the assessment process provides an opportunity to:

  • identify how policy advice performance in a given period compares with performance in the past, and with the performance of other agencies
  • understand what the agency’s strengths are, within the standards that make up the Policy Quality Framework, what the areas for improvement are, and provide a basis for looking at how to improve
  • develop and implement action plans to improve future performance.

At a system level, tracking agency performance results on a consistent basis will enable reporting on the state of the policy system, and identify opportunities for improvement.

Who assesses papers and when?#

All agencies with policy appropriations need to determine who assesses the quality of their policy advice after delivery, on the basis of what works best in their circumstances. Assessments may be undertaken by an internal panel or by independent external assessors (or a combination, such as a panel with some external membership). Smaller agencies who prefer a panel approach may find it helpful to match themselves up with bigger agencies or opt into a cross-agency panel. The Policy Project can be approached for contacts for quality assurance panels. Shared panels offer the chance to share best practice and learning.

While the results of quality of advice assessments are reported externally annually, the assessments may be undertaken once a year, or more regularly (such as quarterly or six monthly). Assessing quality of advice performance more regularly provides an opportunity to assess how well new initiatives and practices are being used, and what difference they’re making to the quality of advice. Assessing advice papers several times throughout the year also helps distribute the work to avoid the busiest times.

How to set up a quality of advice assessment panel#

Panel size and membership#

For an internal or cross-agency panel, three to five assessors will usually be adequate. Larger panels can be costly in terms of time, and difficult to coordinate.

Having a mixture of internal and external panel members can be beneficial. While external members aren’t required, your agency might see benefit in bringing in an external chair or subject matter expert. External panel members are likely to provide a fresh set of eyes and bring other elements of good practice to the panel process. Panel members could also include representatives from other agencies in the same sector, or with similar functions.

  • There’s benefit in having some continuity in panel membership or chairing. Distributing panel expertise is also important, so you may want to rotate some panel members.
  • Internal panel members have a key role in promoting quality advice, performance improvement and best practice. Panels may also choose to invite less experienced policy staff to attend as observers, as this is likely to provide them with a good learning opportunity.
  • Ensure all members of your panel have appropriate security clearances if they’ll be asked to review any classified material.

Panel roles and expertise#

The roles and required expertise of the panel chair, panel members, and panel support people are as follows:

Chair

  • Needs to have credibility and considerable experience in policy advice.
  • Reviews and scores papers as a panel member.
  • As the panel moderator, ensures that the assessments are consistent and fair across all papers, and must be able to make decisions when there are conflicting opinions between panel members on papers and what they should be scored.
  • May take on the task of writing up results, or may delegate this to an administrative support role.

Panel members

  • Must be skilled in giving concrete, constructive feedback on policy papers.
  • Must be able to commit time to the panel and be supported by their manager to participate.
  • Individually review papers, share views on papers’ strengths and weaknesses relative to the Policy Quality Framework standards, and collectively score papers.

Panel support

  • Providing support to the panel can be a good development opportunity for less experienced policy staff members. Consider involving an analyst or advisor to take notes, or assist in writing up reports and feedback summaries.
  • Support will be required to help select the sample of papers, distribute papers, organise meetings, and seek the background and context on papers if needed.
  • A checklist to help with setting up the assessment process is provided in Appendix 2.

Before the panel meeting#

Provide this guide to panel members#

When the Chair and panel members are appointed to their roles, provide them with this guide and take them through it, so they can become familiar with all aspects of it – especially the full version of the Policy Quality Framework in Appendix 1, and the paper-scoring template in Appendix 3 (also available as a separate download). The paper-scoring template provides a structured way for panel members to record the results of their assessments. New panel members will also benefit from being provided copies of previous reviews.

Decide which papers are suitable for assessment#

Assessment should be undertaken of a randomly selected sample from the population of papers delivered to a decision maker by an agency (or agencies jointly) during the relevant period. The sample needs to be stratified so that it covers a mix of different types of papers from different policy teams. Papers could, for example, include:

  • ministerial briefings on significant policy projects
  • Regulatory Impact Statements
  • Cabinet papers
  • meeting or event briefs for ministers
  • process briefs (e.g. board position nominations, funding decisions)
  • A3 or PowerPoint slide pack reports
  • decision papers prepared for senior leadership teams
  • aide memoires for decision makers (e.g. advice for a Cabinet or ministerial meeting).

Outside the sample, you may also like to add other papers you’d like to be reviewed (e.g. a specific suite of papers associated with a particular policy issue). The additional papers wouldn’t be included in the sample scores.

Agencies might want to consider how they can document papers eligible for assessment throughout the year, as they’re delivered. This would make it easier to identify the population of papers from which the sample for assessment could be drawn.

Try to make papers available at least two to three weeks before the panel convenes. Give assessors adequate time to read all the papers and undertake a preliminary assessment of each using the paper-scoring template in this guide.

Determine the sample size#

The appropriate sample size will depend on the population of papers an agency is drawing from. For this reason, there is no strict guidance on the number of papers that should be assessed. Agencies will need to choose a big enough sample size of their policy papers to provide a meaningful measure for each ministerial portfolio. For larger agencies, we suggest a sample size of at least 40 papers, for medium agencies 20 to 40 papers, and for smaller agencies at least 5 to 10 papers.

Learn about the context of papers#

It may be helpful for the assessors to hear about the context of the paper – verbally or in writing. This information can be included with the pack of papers for assessors – based on some quick feedback from the manager responsible for the paper. In some cases, panels may want the author or their manager to be part of the assessment discussion. If so, the appropriate lead contact for each paper should be identified.

Context information could include whether the paper was part of preliminary, mid-stream or final advice, or whether important information was provided in other written advice (e.g. the problem may have been diagnosed in an earlier paper). This will help assessors judge which quality standards and characteristics are relevant to the paper.

It may also be helpful to discuss constraints to quality, such as:

  • time and resources
  • the experience of the primary author
  • familiarity with the subject matter
  • available evidence
  • perceived constraints to providing free and frank advice.

Understanding such constraints can be helpful in deciding the best way to frame panel feedback on the paper, but shouldn’t affect a paper’s score (see the section Reaching a collective score).

How individual panel members should use the paper-scoring template before the panel meets#

Before a panel meeting, individual panel members should use the template attached to this guide as Appendix 3, to record their preliminary assessment of each paper they read.
The template provides:

  1. space to work through each of the four standards – Context, Analysis, Advice, and Action – and the more detailed elements that sit within each standard (you can tick yes, no, or N/A as applicable)
  2. space for comments on key aspects of the paper overall (strengths and weaknesses, and what could have been done differently to improve the paper)
  3. space for an overall rating of the paper, out of five.

Depending on the type of paper, not all of the elements of the standards will be relevant. You may want to mark these as ‘not applicable’ when completing the template for a specific paper.

As you work your way through a paper, assessing it against the four standards of quality advice in the assessment template, remember to write brief comments about what specifically was done that met a standard well or represented best practice, or what specifically could have been done differently to better meet the standard. Review the results to identify which aspects of the paper you think are its greatest strengths and weaknesses. These are the matters you’ll later raise in discussion with others who have assessed the same paper. Individual panel members’ comments will provide useful content for the narrative element of the panel’s report on each paper. The panel discussion will, in turn, help the panel to later reach a collective judgement on the overall numeric score to award the paper out of 5.

Panel meetings#

Purpose of panel meeting#

The purpose of the panel meetings is to undertake a collective assessment process that:

  • reaches agreement about the overall numeric score for each individual paper in a way that reflects the strengths and weaknesses in the quality of advice they provided
  • identifies overall patterns of strengths and weaknesses in the quality of advice provided by this sample of papers, and specific areas for the organisation (or organisations within a cross-agency panel) to target for improvement in future.

We recommend that if a panel member has been actively involved in the development of a paper, they should abstain from scoring the paper and from the panel discussion. They may even want to leave the room, so as not to constrain other panel members in their discussion.

Scale for scoring the quality of advice#

The following table outlines the scoring scale to be used in awarding a paper a score out of 5. Half points can be awarded where a paper falls between two points on the scale.

1 = Unacceptable
Does not meet the relevant quality standards in fundamental ways

  • Lacks basic information and analysis
  • Creates serious risk of poor decision making
  • Should not have been signed out
  • Needed fundamental rework

2 = Poor
Does not meet the relevant quality standards in material ways

  • Explains the basic issue but seriously lacking in several important areas
  • Creates risk of poor decision making
  • Should not have been signed out
  • Needed substantial improvement in important areas

3 = Acceptable
Meets the relevant quality standards overall, but with some shortfalls

  • Provides most of the analysis and information needed
  • Could be used for decision making
  • Was sufficiently fit-for-purpose for sign-out
  • Could have been improved in several areas

4 = Good
Meets all the relevant quality standards

  • Represents good practice
  • Provides a solid basis for decision making
  • Could have been signed out with confidence
  • Minor changes would have added polish

5 = Meets all the relevant quality standards and adds something extra

  • Represents exemplary practice
  • First-rate advice that provides a sound basis for confident decision making
  • Could have been signed out with great confidence
  • A polished product

Reaching a collective score for a paper#

At panel meetings, you should discuss individual papers by working through the following steps:

  1. Consider how the paper performs against each of the Policy Quality Framework’s four high-level standards (and all of the seventeen more specific elements that apply).
  2. Compare notes on what individual panel members consider to be the strengths and weaknesses of the paper.
  3. Identify what the author could have done that would have improved the ratings.
  4. Collectively agree an overall score out of 5, applying the scoring scale above.

When scoring advice papers, don’t adjust scoring based on constraints to quality. You will want to note these, but they should not affect the score. Assessors may want to reflect on which constraints most affect the quality of papers overall and any recurring themes.

Moderating paper scores, relative to one another#

After the panel or team of assessors have arrived at a preliminary numeric score out of 5 for each of the papers being assessed, it is important to undertake a moderation process before finalising the numeric scores for each paper. Taking the following steps will help ensure that papers of a similar quality are scored the same, while papers of markedly different levels of quality are appropriately scored differently:

  1. List the names and scores of all the papers assessed where everyone can see them (e.g. on a white board, flip chart or screen).
  2. For each sub-group of papers with the same score, discuss whether they really merit the same score, or whether one or more was markedly better or worse than the others and hence should be scored differently.
  3. Discuss whether the outliers (papers with very high or very low scores) are really so much better or worse than the other papers that they really merit such high or low scores.
  4. Where assessors have also taken part in a previous assessment round, discuss whether those papers awarded a given score in this round are of comparable quality to those awarded that score in the previous round.
  5. In light of any discrepancies in the initial numeric scores revealed during the above moderation discussions, revise individual paper scores so there’s internal consistency – both within this assessment round and if possible between assessment rounds.

After the assessment panel meeting#

Panel commentaries on individual papers and overall themes#

The role of the assessment panel is to drive policy improvement. In addition to scoring each paper, the panel should provide a brief written commentary on each paper, and on themes for improvement across the papers. Panel assessments are also a valuable opportunity to identify exemplars of what was done well and what to avoid.

The panel’s report should also track policy performance by producing a tabular summary of statistics. The panel can also graph the distribution of quantitative scores for a given year and compare them with previous years, if they are available.

Optional external review#

After you’ve run an internal panel assessment, you may also choose to undertake an external review to validate and benchmark your internal processes for reviewing papers. An external reviewer may also have useful ideas for improvement.

An external reviewer would usually assess and score a random sample of the papers that your internal panel has previously reviewed. Moderation of your papers’ final scores would then take place, with the panel chair taking into account the external reviewer’s scores and their relativity to the internal panel’s scores.

Feedback to authors#

Consider holding individual feedback sessions with authors, managers, and peer reviewers on your findings about the quality of advice. Look for opportunities to talk through the panel’s assessment of the paper, main strengths and weaknesses, and how it could have been improved. The most helpful feedback sessions for authors focus on discussing the overall paper before moving on to scores, with an emphasis on the positives and any concrete suggestions for improvements. Remember the author will have put a lot of care into their paper.

This learning opportunity is worth incorporating into your annual planning schedule for panels and assessments.

Reporting agency performance#

While all agencies have to score the quality of their policy advice papers, until now, there has been no public service-wide performance target or reporting requirement.

As part of having a universal Policy Quality Framework that all public services agencies must use in assessing and improving the quality of their policy advice, we recommend there be a consistent approach to measuring performance.

We recommend that agencies use the following two targets to report on overall performance:

  1. An average score. For example, this could be that the average score for papers that are assessed is at least 3.5 out of 5.
  2. A distribution of scores to show the percentage of papers that exceed, meet, or don’t meet the performance target that has been set. For example, this could be that 70% of assessed papers score 3 or higher, 30% score 4 or higher, and no more than 10% score 2.5 or less.

Reporting on both an average target for policy quality and distribution targets will provide a better reflection of an agency’s performance. Relying on one or the other may not give a true indication of how an agency has performed over time.

Average scores seem easy to understand but they may be misleading. They draw attention away from the outliers. For example, if an agency receives two high scores, two medium scores and two low scores, an average of these scores (the medium score) does not reflect the distribution of quality. If, however, all the papers received a medium score then it does. Low scores on a few papers can drag down the average, which will not resonate with an agency if it knows most of its papers were good. The distribution of scores in addition to an average score reveals the fuller picture.

Getting a complete picture of agency policy performance#

Policy leaders may want to consider the quality of advice assessment panel’s results along with other information on the performance of the policy function for a complete picture (e.g. the results from the Ministerial Policy Satisfaction Survey, an assessment of the policy function’s capability using the Policy Capability Framework, and the skills of the policy team using the Policy Skills Framework).

Policy improvement across the policy system#

Our intention is that the information collected from agencies’ performance reporting will enable the Policy Project to reach a system-wide view of policy performance. This will show how agencies are performing relative to one another and the overall system.

Communicating lessons learned for the agency#

Agencies should think about how they can share quality of advice findings with policy teams across the agency. You may want to publicise and celebrate good practice and achievements. Key messages about areas to improve with links to resources can be helpful. Consider if there are recurring constraints to quality, and if so, what your agency can do to mitigate them.

If you are staggering assessment panels across the year, consider tracking trends. For example, if the same weaknesses are recurring every quarter, stronger actions may be needed to improve in these areas. You can also track whether new initiatives or practices to improve policy quality are actually making a difference.

If you identify negative trends or deficits that put the agency’s reputation at risk, these should be escalated to senior leaders and organisational development teams.

Appendices#

  1. The full Policy Quality Framework
  2. Checklist for panel administration
  3. Policy Quality Framework paper-scoring template

 

  • Previous
  • Next
Sidebar anchor
View all on one page
Show content paged
Print this publication

Table of Contents

  1. About this guide
    1. Policy Quality Framework: standards for quality policy advice
  2. Why assessing policy advice is important
    1. Estimates and annual reporting
    2. Assess to improve
    3. Who assesses papers and when?
  3. How to set up a quality of advice assessment panel
    1. Panel size and membership
    2. Panel roles and expertise
  4. Before the panel meeting
    1. Provide this guide to panel members
    2. Decide which papers are suitable for assessment
    3. Determine the sample size
    4. Learn about the context of papers
    5. How individual panel members should use the paper-scoring template before the panel meets
  5. Panel meetings
    1. Purpose of panel meeting
    2. Scale for scoring the quality of advice
    3. Reaching a collective score for a paper
    4. Moderating paper scores, relative to one another
  6. After the assessment panel meeting
    1. Panel commentaries on individual papers and overall themes
    2. Optional external review
    3. Feedback to authors
    4. Reporting agency performance
    5. Getting a complete picture of agency policy performance
    6. Policy improvement across the policy system
    7. Communicating lessons learned for the agency
  7. Appendices

Related publications

Making Laws That Work: How laws fail and how we can do better

Resources
8 Dec 2022

Legislation Checklists

Resources
8 Dec 2022

Strengthening policy capability: New Zealand's Policy Project

Resources
31 May 2022

Making sense of evidence: A guide to using evidence in policy

Resources
31 May 2022

Free and Frank Advice & Policy Stewardship

Resources
31 May 2022

Kapasa – The Pacific Policy Analysis Tool

Resources
27 May 2022
View all publications

Content wrapper

Help us improve DPMC

Your feedback is very important in helping us improve the DPMC website.


More menu anchor
Home Department of the Prime Minister and Cabinet (DPMC)
Page url

Back to top

About DPMC
  • Who we are
  • Our work
  • Our history
  • Careers
Our business units
  • Cabinet Office
  • National Security Group
  • Policy Advisory Group
  • Implementation Unit
  • Strategy, Governance and Engagement Group
  • Child Wellbeing and Poverty Reduction Group
  • COVID-19 Group
  • Government House
Departmental agency
  • NEMA
Our programmes
  • New Zealand Royal Honours
  • The Policy Project
  • Greater Christchurch recovery and regeneration
  • National security
  • Child and youth wellbeing strategy
  • Reducing child poverty
  • Special programmes
Publications
  • Publications
  • News and notices
  • Annual reports
  • CabGuide
  • Cabinet Manual
  • Contact us
  • Copyright and licensing
  • Privacy
  • About this site
  • Disclaimer
  • Site map
Visit the Homepage | New Zealand Government Visit the Homepage | New Zealand Government

© Copyright 2023 Department of the Prime Minister and Cabinet.

New Zealand Government