Policy Quality Framework – a guide on panels and processes for assessing policy advice papers

About this guide#

This guide contains advice on how to run a panel-based assessment of policy and other advice papers after their delivery (an ex-post assessment process) using the Policy Quality Framework. It also explains how to score advice papers using the Policy Quality Framework’s paper scoring template. The template sets out the Policy Quality Framework’s standards in a format that helps in assessing the quality of an agency’s written advice, and whether it is fit for purpose. This guide can be found on the Policy Quality Framework webpage. Its purpose is to support government agencies in understanding, reporting and improving their policy advice performance.

Policy Quality Framework: standards for quality policy advice#

The framework sets out seventeen elements of quality policy advice, organised under four standards. These standards can be applied by panels to assess papers and decide whether they are fit for purpose. See Appendix 1 for the full version of the Policy Quality Framework, which provides more detail on each of the elements below.

Why assessing policy advice is important#

Estimates and annual reporting#

From 2019/20, all government agencies are required to report in their Estimates and annual reports on the numeric score for the quality of their policy advice, using the Policy Quality Framework. This requires an agency to assess the quality of the advice it delivers during the year. Agencies choose to do this by either establishing their own assessment panels or contracting out the assessment process. 

Assess to improve#

Beyond holding agencies to account for their performance, using the Policy Quality Framework to assess papers after delivery is also vital in improving performance – and thereby supporting better government decision making.

At the individual agency level, the assessment process provides an opportunity to:

• identify how policy advice performance in a given period compares with performance in the past, and with the performance of other agencies
• understand what the agency’s strengths are, within the standards that make up the Policy Quality Framework, what the areas for improvement are, and provide a basis for looking at how to improve
• develop and implement action plans to improve future performance.

At a system level, tracking agency performance results on a consistent basis will enable reporting on the state of the policy system, and identify opportunities for improvement.

Who assesses papers and when?#

All agencies with policy appropriations need to determine who assesses the quality of their policy advice after delivery, on the basis of what works best in their circumstances. Assessments may be undertaken by an internal panel or by independent external assessors (or a combination, such as a panel with some external membership). Smaller agencies who prefer a panel approach may find it helpful to match themselves up with bigger agencies or opt into a cross-agency panel. The Policy Project can be approached for contacts for quality assurance panels. Shared panels offer the chance to share best practice and learning.

While the results of quality of advice assessments are reported externally annually, the assessments may be undertaken once a year, or more regularly (such as quarterly or six monthly). Assessing quality of advice performance more regularly provides an opportunity to assess how well new initiatives and practices are being used, and what difference they’re making to the quality of advice. Assessing advice papers several times throughout the year also helps distribute the work to avoid the busiest times.

How to set up a quality of advice assessment panel#

Panel size and membership#

For an internal or cross-agency panel, three to five assessors will usually be adequate. Larger panels can be costly in terms of time, and difficult to coordinate.

Having a mixture of internal and external panel members can be beneficial. While external members aren’t required, your agency might see benefit in bringing in an external chair or subject matter expert. External panel members are likely to provide a fresh set of eyes and bring other elements of good practice to the panel process. Panel members could also include representatives from other agencies in the same sector, or with similar functions.

• There’s benefit in having some continuity in panel membership or chairing. Distributing panel expertise is also important, so you may want to rotate some panel members.
• Internal panel members have a key role in promoting quality advice, performance improvement and best practice. Panels may also choose to invite less experienced policy staff to attend as observers, as this is likely to provide them with a good learning opportunity.
• Ensure all members of your panel have appropriate security clearances if they’ll be asked to review any classified material.

Panel roles and expertise#

The roles and required expertise of the panel chair, panel members, and panel support people are as follows:

Chair

• Needs to have credibility and considerable experience in policy advice.
• Reviews and scores papers as a panel member.
• As the panel moderator, ensures that the assessments are consistent and fair across all papers, and must be able to make decisions when there are conflicting opinions between panel members on papers and what they should be scored.
• May take on the task of writing up results, or may delegate this to an administrative support role.

Panel members

• Must be skilled in giving concrete, constructive feedback on policy papers.
• Must be able to commit time to the panel and be supported by their manager to participate.
• Individually review papers, share views on papers’ strengths and weaknesses relative to the Policy Quality Framework standards, and collectively score papers.

Panel support

• Providing support to the panel can be a good development opportunity for less experienced policy staff members. Consider involving an analyst or advisor to take notes, or assist in writing up reports and feedback summaries.
• Support will be required to help select the sample of papers, distribute papers, organise meetings, and seek the background and context on papers if needed.
• A checklist to help with setting up the assessment process is provided in Appendix 2.

Before the panel meeting#

Provide this guide to panel members#

When the Chair and panel members are appointed to their roles, provide them with this guide and take them through it, so they can become familiar with all aspects of it – especially the full version of the Policy Quality Framework in Appendix 1, and the paper-scoring template in Appendix 3 (also available as a separate download). The paper-scoring template provides a structured way for panel members to record the results of their assessments. New panel members will also benefit from being provided copies of previous reviews.

Decide which papers are suitable for assessment#

Assessment should be undertaken of a randomly selected sample from the population of papers delivered to a decision maker by an agency (or agencies jointly) during the relevant period. The sample needs to be stratified so that it covers a mix of different types of papers from different policy teams. Papers could, for example, include:

• ministerial briefings on significant policy projects
• Regulatory Impact Statements
• Cabinet papers
• meeting or event briefs for ministers
• process briefs (e.g. board position nominations, funding decisions)
• A3 or PowerPoint slide pack reports
• decision papers prepared for senior leadership teams
• aide memoires for decision makers (e.g. advice for a Cabinet or ministerial meeting).

Outside the sample, you may also like to add other papers you’d like to be reviewed (e.g. a specific suite of papers associated with a particular policy issue). The additional papers wouldn’t be included in the sample scores.

Agencies might want to consider how they can document papers eligible for assessment throughout the year, as they’re delivered. This would make it easier to identify the population of papers from which the sample for assessment could be drawn.

Try to make papers available at least two to three weeks before the panel convenes. Give assessors adequate time to read all the papers and undertake a preliminary assessment of each using the paper-scoring template in this guide.

Determine the sample size#

The appropriate sample size will depend on the population of papers an agency is drawing from. For this reason, there is no strict guidance on the number of papers that should be assessed. Agencies will need to choose a big enough sample size of their policy papers to provide a meaningful measure for each ministerial portfolio. For larger agencies, we suggest a sample size of at least 40 papers, for medium agencies 20 to 40 papers, and for smaller agencies at least 5 to 10 papers.

Learn about the context of papers#

It may be helpful for the assessors to hear about the context of the paper – verbally or in writing. This information can be included with the pack of papers for assessors – based on some quick feedback from the manager responsible for the paper. In some cases, panels may want the author or their manager to be part of the assessment discussion. If so, the appropriate lead contact for each paper should be identified.

Context information could include whether the paper was part of preliminary, mid-stream or final advice, or whether important information was provided in other written advice (e.g. the problem may have been diagnosed in an earlier paper). This will help assessors judge which quality standards and characteristics are relevant to the paper.

It may also be helpful to discuss constraints to quality, such as:

• time and resources
• the experience of the primary author
• familiarity with the subject matter
• available evidence
• perceived constraints to providing free and frank advice.

Understanding such constraints can be helpful in deciding the best way to frame panel feedback on the paper, but shouldn’t affect a paper’s score (see the section Reaching a collective score).

How individual panel members should use the paper-scoring template before the panel meets#

Before a panel meeting, individual panel members should use the template attached to this guide as Appendix 3, to record their preliminary assessment of each paper they read.
The template provides:

1. space to work through each of the four standards – Context, Analysis, Advice, and Action – and the more detailed elements that sit within each standard (you can tick yes, no, or N/A as applicable)
2. space for comments on key aspects of the paper overall (strengths and weaknesses, and what could have been done differently to improve the paper)
3. space for an overall rating of the paper, out of five.

Depending on the type of paper, not all of the elements of the standards will be relevant. You may want to mark these as ‘not applicable’ when completing the template for a specific paper.

As you work your way through a paper, assessing it against the four standards of quality advice in the assessment template, remember to write brief comments about what specifically was done that met a standard well or represented best practice, or what specifically could have been done differently to better meet the standard. Review the results to identify which aspects of the paper you think are its greatest strengths and weaknesses. These are the matters you’ll later raise in discussion with others who have assessed the same paper. Individual panel members’ comments will provide useful content for the narrative element of the panel’s report on each paper. The panel discussion will, in turn, help the panel to later reach a collective judgement on the overall numeric score to award the paper out of 5.

Panel meetings#

Purpose of panel meeting#

The purpose of the panel meetings is to undertake a collective assessment process that:

• reaches agreement about the overall numeric score for each individual paper in a way that reflects the strengths and weaknesses in the quality of advice they provided
• identifies overall patterns of strengths and weaknesses in the quality of advice provided by this sample of papers, and specific areas for the organisation (or organisations within a cross-agency panel) to target for improvement in future.

We recommend that if a panel member has been actively involved in the development of a paper, they should abstain from scoring the paper and from the panel discussion. They may even want to leave the room, so as not to constrain other panel members in their discussion.

Scale for scoring the quality of advice#

The following table outlines the scoring scale to be used in awarding a paper a score out of 5. Half points can be awarded where a paper falls between two points on the scale.

1 = Unacceptable
Does not meet the relevant quality standards in fundamental ways

• Lacks basic information and analysis
• Creates serious risk of poor decision making
• Should not have been signed out
• Needed fundamental rework

2 = Poor
Does not meet the relevant quality standards in material ways

• Explains the basic issue but seriously lacking in several important areas
• Creates risk of poor decision making
• Should not have been signed out
• Needed substantial improvement in important areas

3 = Acceptable
Meets the relevant quality standards overall, but with some shortfalls

• Provides most of the analysis and information needed
• Could be used for decision making
• Was sufficiently fit-for-purpose for sign-out
• Could have been improved in several areas

4 = Good
Meets all the relevant quality standards

• Represents good practice
• Provides a solid basis for decision making
• Could have been signed out with confidence
• Minor changes would have added polish

5 = Meets all the relevant quality standards and adds something extra

• Represents exemplary practice
• First-rate advice that provides a sound basis for confident decision making
• Could have been signed out with great confidence
• A polished product

Reaching a collective score for a paper#

At panel meetings, you should discuss individual papers by working through the following steps:

1. Consider how the paper performs against each of the Policy Quality Framework’s four high-level standards (and all of the seventeen more specific elements that apply).
2. Compare notes on what individual panel members consider to be the strengths and weaknesses of the paper.
3. Identify what the author could have done that would have improved the ratings.
4. Collectively agree an overall score out of 5, applying the scoring scale above.

When scoring advice papers, don’t adjust scoring based on constraints to quality. You will want to note these, but they should not affect the score. Assessors may want to reflect on which constraints most affect the quality of papers overall and any recurring themes.

Moderating paper scores, relative to one another#

After the panel or team of assessors have arrived at a preliminary numeric score out of 5 for each of the papers being assessed, it is important to undertake a moderation process before finalising the numeric scores for each paper. Taking the following steps will help ensure that papers of a similar quality are scored the same, while papers of markedly different levels of quality are appropriately scored differently:

1. List the names and scores of all the papers assessed where everyone can see them (e.g. on a white board, flip chart or screen).
2. For each sub-group of papers with the same score, discuss whether they really merit the same score, or whether one or more was markedly better or worse than the others and hence should be scored differently.
3. Discuss whether the outliers (papers with very high or very low scores) are really so much better or worse than the other papers that they really merit such high or low scores.
4. Where assessors have also taken part in a previous assessment round, discuss whether those papers awarded a given score in this round are of comparable quality to those awarded that score in the previous round.
5. In light of any discrepancies in the initial numeric scores revealed during the above moderation discussions, revise individual paper scores so there’s internal consistency – both within this assessment round and if possible between assessment rounds.

After the assessment panel meeting#

Panel commentaries on individual papers and overall themes#

The role of the assessment panel is to drive policy improvement. In addition to scoring each paper, the panel should provide a brief written commentary on each paper, and on themes for improvement across the papers. Panel assessments are also a valuable opportunity to identify exemplars of what was done well and what to avoid.

The panel’s report should also track policy performance by producing a tabular summary of statistics. The panel can also graph the distribution of quantitative scores for a given year and compare them with previous years, if they are available.

Optional external review#

After you’ve run an internal panel assessment, you may also choose to undertake an external review to validate and benchmark your internal processes for reviewing papers. An external reviewer may also have useful ideas for improvement.

An external reviewer would usually assess and score a random sample of the papers that your internal panel has previously reviewed. Moderation of your papers’ final scores would then take place, with the panel chair taking into account the external reviewer’s scores and their relativity to the internal panel’s scores.

Feedback to authors#

Consider holding individual feedback sessions with authors, managers, and peer reviewers on your findings about the quality of advice. Look for opportunities to talk through the panel’s assessment of the paper, main strengths and weaknesses, and how it could have been improved. The most helpful feedback sessions for authors focus on discussing the overall paper before moving on to scores, with an emphasis on the positives and any concrete suggestions for improvements. Remember the author will have put a lot of care into their paper.

This learning opportunity is worth incorporating into your annual planning schedule for panels and assessments.

Reporting agency performance#

While all agencies have to score the quality of their policy advice papers, until now, there has been no public service-wide performance target or reporting requirement.

As part of having a universal Policy Quality Framework that all public services agencies must use in assessing and improving the quality of their policy advice, we recommend there be a consistent approach to measuring performance.

We recommend that agencies use the following two targets to report on overall performance:

1. An average score. For example, this could be that the average score for papers that are assessed is at least 3.5 out of 5.
2. A distribution of scores to show the percentage of papers that exceed, meet, or don’t meet the performance target that has been set. For example, this could be that 70% of assessed papers score 3 or higher, 30% score 4 or higher, and no more than 10% score 2.5 or less.

Reporting on both an average target for policy quality and distribution targets will provide a better reflection of an agency’s performance. Relying on one or the other may not give a true indication of how an agency has performed over time.

Average scores seem easy to understand but they may be misleading. They draw attention away from the outliers. For example, if an agency receives two high scores, two medium scores and two low scores, an average of these scores (the medium score) does not reflect the distribution of quality. If, however, all the papers received a medium score then it does. Low scores on a few papers can drag down the average, which will not resonate with an agency if it knows most of its papers were good. The distribution of scores in addition to an average score reveals the fuller picture.

Getting a complete picture of agency policy performance#

Policy leaders may want to consider the quality of advice assessment panel’s results along with other information on the performance of the policy function for a complete picture (e.g. the results from the Ministerial Policy Satisfaction Survey, an assessment of the policy function’s capability using the Policy Capability Framework, and the skills of the policy team using the Policy Skills Framework).

Policy improvement across the policy system#

Our intention is that the information collected from agencies’ performance reporting will enable the Policy Project to reach a system-wide view of policy performance. This will show how agencies are performing relative to one another and the overall system.

Communicating lessons learned for the agency#

Agencies should think about how they can share quality of advice findings with policy teams across the agency. You may want to publicise and celebrate good practice and achievements. Key messages about areas to improve with links to resources can be helpful. Consider if there are recurring constraints to quality, and if so, what your agency can do to mitigate them.

If you are staggering assessment panels across the year, consider tracking trends. For example, if the same weaknesses are recurring every quarter, stronger actions may be needed to improve in these areas. You can also track whether new initiatives or practices to improve policy quality are actually making a difference.

If you identify negative trends or deficits that put the agency’s reputation at risk, these should be escalated to senior leaders and organisational development teams.

Appendices#

1. The full Policy Quality Framework
2. Checklist for panel administration
3. Policy Quality Framework paper-scoring template