Citizen Participation has been a growing theme in democratic practice in recent years, to the extent that it has now become an area of scrutiny, as well as a growing area of practice.

In February 2022, the Citizen Participation and Public Petitions Committee agreed to launch an inquiry into Public Participation in the work of the Scottish Parliament. As part of this, it asked a group of 19 randomly selected people from across Scotland, who were broadly representative of Scotland's population, to come together as a Citizens' Panel to answer the question "How can the Scottish Parliament ensure that diverse voices and communities from all parts of Scotland influence our work?". The Scottish Parliament Information Centre (SPICe) acknowledged at the time that an inward-looking scrutiny focus was novel in the Scottish Parliament.

The panel came up with 17 recommendations in December 2022, and the Committee took further evidence around these recommendations, which SPICe supported by looking at examples of practice similar to the Panel's suggestions, and deliberative processes in other legislatures.

The purpose of this report is to respond to one of the recommendations from the Panel to 'Build a strong evidence base for deliberative democracy to determine its effectiveness and develop a framework for measuring impact'.

This report starts in Section 2 by providing a brief overview of deliberative democracy. It then focuses on the background of the public participation activity happening in the Scottish Parliament in Section 3. Section 4 introduces the key principles for deliberative practice, and Section 5 sets out a framework to apply when evaluating practice against core principles. Section 6 provides a toolkit of evaluation measurements, providing detail on when and how these methods can be used.

Section 7 signposts further resources including overviews of practices, case studies, facilitation methods and evaluation tools in and around the Scottish Parliament. It does so for two reasons, first for openness and transparency, in a willingness from the Scottish Parliamentary service to evaluate its own work. And second to develop a toolkit for good practice and produce a handbook to support organisers hoping to utilise participatory and deliberative practice to improve their own evaluation

Finally, the last section sets out recommendations for applying the evaluation framework and highlights area for future work.

Deliberative democracy is a form of democracy that emphasises communication, in particular the use of reasoned dialogue and deliberation as the foundation for informed policy and decision-making. Deliberative democrats argue that the process of decision-making is just as important as the outcome (see Warren 2002¹Williamson, A., & Barrat, J. (2022). Mapping Deliberative Democracy in Council of Europe Member States. Prepared for the Division of Elections and Participatory Democracy, Department of Democracy and Governance, Council of Europe. Retrieved from https://rm.coe.int/-mapping-deliberate-democracy/1680a87f84 [accessed 24 February 2024], Dryzek 2010²Smith, G. (2009). Democratic Innovations: Designing Institutions for Citizen Participation. Cambridge: Cambridge University Press.). Ideally, the core principles for embedding deliberative approaches should reflect the founding principles of the parliament - openness, accountability, the sharing of power and equal opportunities (Elstub et al. 2021³Elstub, S. (2014). Deliberative Pragmatic Equilibrium Review: A Framework for Comparing Institutional Devices their Enactment of Deliberative Democracy in the UK. The British Journal of Politics and International Relations, 16(3), 386-409., Davidson et al. 2011⁴Lightbody, R. (2023). ‘Advice to inform the Open Government public participation strategy’. A report commissioned by the Scottish Government. Retrieved from https://www.gov.scot/publications/advice-inform-scotlands-open-government-public-participation-strategy/documents/ [accessed 24 February 2024]). According to Bussu (et al. 2022⁵Boswell, J., Dean, R., & Smith, G. (2023). Integrating citizen deliberation into climate governance: Lessons on robust design from six climate assemblies. Public Administration, 101(1), 182-200.), embeddedness is the capacity for a process to be flexible, to work with other processes, but to be embedded it must have ‘rootedness’, it must be established, and an accepted means of participating by the public and their representatives.

For deliberative and participatory democracy to be impactful, it is important to examine how these principles are being operationalised throughout the public policy decision-making process.  As Williamson and Barrat (2022) say:

Deliberative methods must become normative tools within the policy cycle and deliberative initiatives properly resourced and planned. Such initiatives must be transparent, auditable and accountable to ensure that participant selection is appropriate, evidence is not biased and outcomes not dictated or predetermined. It is vital, too, that the cycle of participation is always closed by providing feedback on what actions have resulted from the recommendations and by building public engagement into the process.

For citizens, these deliberative initiatives are a learning experience and their design must reflect not only appropriate onboarding but also space for "just-in-time" learning. Good facilitation is vital to steer debate and ensure that all voices are heard, and recruitment must ensure that minority voices are present, listened to and respected. It is vital at this relatively early stage of deliberative democracy to increase opportunities for learning and evaluation – both within organisations and as a shared body of knowledge for all.” Mapping deliberative democracy in Council of Europe

Williamson, A., & Barrat, J. (2022). Mapping Deliberative Democracy in Council of Europe Member States. Prepared for the Division of Elections and Participatory Democracy, Department of Democracy and Governance, Council of Europe. Retrieved from https://rm.coe.int/-mapping-deliberate-democracy/1680a87f84 [accessed 24 February 2024]

As highlighted in the OECD's Deliberative Democracy Database (launched in 2023) the "deliberative wave" has been building since 1979, which accelerated in 2010. 

Engagement has been a long-standing feature of the Scottish Parliamentary service, but in recent years this has expanded to include more participatory and deliberative practice. It has also become a more formalised process, which is expected to continue and grow in the coming years. Aiming to build a practice and governance framework requires some reflection on the history of participation in the Parliament, and the current structures in place.

The Commission on Parliamentary Reform, in 2017, highlighted the importance of engaging the public in parliamentary processes and recommended that "The Parliament should establish a dedicated team whose main purpose is to support (and challenge) committees to undertake more innovative and meaningful engagement".

The Committee Engagement Unit was formed soon afterwards and became the Participation and Communities Team at the start of Session 6 of the Scottish Parliament in 2021, which coincided with the name and remit of the Public Petitions Committee being expanded to include Citizen Participation.

In parallel, the Scottish Parliament Corporate Body's Strategic Plan for Session 6 includes a commitment to "embedding deliberative democracy in the work of the Parliament".

Since the development of specialised services to support participation in the Scottish Parliament, several high-profile engagement and deliberative activities have taken place. This has included citizens' panels on primary care (2019) and COVID-19 (2021), and a citizens' jury on land management (2019). Further detail and more recent practice is set out in Section 3:2: Case studies.

It's useful, in looking forward, to also look back at the past practice which the Parliament can learn from. This helps to frame more recent work, but also helps to demonstrate the long-term expertise that the Parliamentary service has in the field.

Since its inception, the Scottish Parliament has always had services to support:

• Public engagement and outreach, which includes both inward and outward education visits, public exhibitions and tours of the Parliament, and a public information service.

• Committee consultation and evidence-taking, both formal (through ‘calls for views’ on inquiries and in committee meetings), and informal, through committee visits and stakeholder engagement such as focus groups.

• Events in the Parliament, both sponsored by Members of the Scottish Parliament (MSPs) and Committees, and those held to celebrate milestone occasions such as the session opening and Parliamentary anniversary events. There have also been visiting exhibitions and the annual Festival of Politics, all of which give people an opportunity to spend time in the Parliament building.

• Separate to activities driven by Parliamentary business, MSPs connect with constituents in their role, and often use the Parliament building for this. Cross-party groups also meet in the Parliament building

In Session 4 of the Scottish Parliament (2011-2016), the Parliament Day initiative focused on "taking the Parliament out of Edinburgh and into communities across Scotland.". Fourteen Parliament Days took place between November 2012 and December 2015.

At this point, the capacity within the outreach staffing resource was increased to support this work, and outreach officers were more formally integrated into the wider committee support teams (which includes clerks, SPICe researchers and media/communications specialists).

Additional outreach support for activities outside of Parliament Days facilitated committee factfinding and engagement trips, such as the Equal Opportunities Committee's visit to Islay in 2015 as part of its inquiry into Age and Social Isolation, or the Rural Economy and Connectivity Committee's visits to Orkney, Mull, Lewis and Harris as part of the Islands (Scotland) Bill scrutiny process in 2017.

Means for engaging digitally, such as using online surveys, were supported by the Parliament's internal research service. An early example would be the public survey onthe Marriage and Civil Partnership (Scotland) Bill in 2013. In 2017, a cross-organisation working group was established to explore the use of digital engagement tools, which led to the adoption of the consultation platform Citizen Space (by Delib) for use in all committee calls for views in 2021.

The majority of these services are now primarily delivered and supported by the Participation and Communities Team, in collaboration with colleagues from clerking, research, communications and public information services. With the expansion in services, there has been an increase in capacity to deliver deliberative activities, which are typically more involved and resource-intense than engagement activities like tours and visits, and participation activities like consultations, surveys and focus groups.

The Citizen Participation and Public Petitions Committee agreed to launch an inquiry into Public Participation in the work of the Scottish Parliament in February 2022. This included a representative sample of 19 people, randomly selected from across Scotland, to come together as a Citizens' Panel to answer the question "How can the Scottish Parliament ensure that diverse voices and communities from all parts of Scotland influence our work?". An inward-looking scrutiny focus was novel in the Scottish Parliament and in many ways trailblazing innovative practice for others to build on.

The panel came up with 17 recommendations in December 2022, and the Committee took further evidence around these recommendations, which SPICe supported by looking at examples of practice similar to the Panel's suggestions, and deliberative processes in other legislatures.

The Committee set out its vision for embedding deliberative democracy in the work of the Scottish Parliament in its final report in September 2023, agreeing to the majority of the panel's recommendations, at least in principle. Most significantly, the Committee concluded that the Parliament should use Citizens' Panels more regularly to help committees with scrutiny work, and made several recommendations for pilot and preparatory work, with certain guiding principles (see Section 4). The expectation is that, by the end of Session 6 (May 2026), the Committee will recommend a model that the Parliament can use after the 2026 election.

The literature is clear that deliberative processes should embody some aspects, if not all, of the following democratic norms: inclusivity, transparency, popular control and considered judgement, and the institutional goods of efficiency and transferability (Smith 2009¹Smith, G. (2009). Democratic Innovations: Designing Institutions for Citizen Participation. Cambridge: Cambridge University Press.).  These norms arguably align with the Scottish Parliament's founding principles. Deliberative democrats have long acknowledged that not all democratic institutions can do all these things but, by combining various mechanisms, deliberative democrats believe that each will fulfil a different but equally vital role in the democratic process (Elstub 2014²Elstub, S. (2014). Deliberative Pragmatic Equilibrium Review: A Framework for Comparing Institutional Devices their Enactment of Deliberative Democracy in the UK. The British Journal of Politics and International Relations, 16(3), 386-409.). In adopting a framework to guide the core principles, the central issues of equality, inclusivity and diversity must be considered so that these processes are accessible, representative of Scotland's rich and diverse population, and built into the political culture in Scotland (Lightbody 2023³Lightbody, R. (2023). ‘Advice to inform the Open Government public participation strategy’. A report commissioned by the Scottish Government. Retrieved from https://www.gov.scot/publications/advice-inform-scotlands-open-government-public-participation-strategy/documents/ [accessed 24 February 2024]). Most literature and practice-based research highlights that the impact of deliberative processes should be measured through multiple means (see Boswell et al. 2023⁴Boswell, J., Dean, R., & Smith, G. (2023). Integrating citizen deliberation into climate governance: Lessons on robust design from six climate assemblies. Public Administration, 101(1), 182-200., Demski and Capstick 2022⁵Demski, C., & Capstick, S. (2022). ‘Impact evaluation framework for climate assemblies’, KNOCA. Retrieved from https://knoca.eu/app/uploads/2022/11/Impact-evaluation-framework-for-climate-assemblies-1.pdf [accessed 24 February 2024], Elstub et al. 2021⁶Elstub, S., Carrick, J., & Khoban , Z. (2021). Democratic Innovation in the Scottish Parliament: An Evaluation of Committee Mini-Publics. Scottish Affairs, 30(4), 493-521., Deligiaouri and Suiter 2021⁷Deligiaouri, A., & Suiter, J. (2021). A policy impact tool: Measuring the policy impact of public participation in deliberative e-rulemaking. Policy Internet, 13, 349-365., Bächtiger and Parkinson 2019⁸Bächtiger, A., & Parkinson, J. (2019). Mapping and Measuring Deliberation: Towards a New Deliberative Quality. Retrieved from https://doi.org/10.1093/oso/9780199672196.001.0001 [accessed 24 February 2024], Michels and Binnema 2019⁹Michels, A., & Binnema, H. (2019). Assessing the Impact of Deliberative Democratic Initiatives at the Local Level: A Framework for Analysis. Administration & Society 2019, 51(5), 749-769. doi: 10.1177/0095399718760588), for example by exploring:

• How such processes shape policy

• How they impact on civil society

• How transparent they are

• How they empower the public

• How transparent they make the political system

• If they lead to power-sharing, power transfer and/or partnerships between various actors

• How well they into link with other political processes (parliamentary committees, referendums)

• How they contribute to the deliberative quality of other processes

The figure below, adapted from Demski and Capstick 2022⁵Demski, C., & Capstick, S. (2022). ‘Impact evaluation framework for climate assemblies’, KNOCA. Retrieved from https://knoca.eu/app/uploads/2022/11/Impact-evaluation-framework-for-climate-assemblies-1.pdf [accessed 24 February 2024], shows how the impact of deliberative processes can be measured over time. This can be seen in more detail in the table in Annex 1.

There is a cacophony of 'deliberative' guides, toolboxes, case studies and research publicly available now which has been a result of the so-called deliberative turn (OECD 2021¹¹OECD. (2021). Evaluation Guidelines for Representative Deliberative Processes. Retrieved from https://www.oecd-ilibrary.org/sites/10ccbfcb-en/index.html?itemId=/content/publication/10ccbfcb-en [accessed 24 February 2024]). And yet, what is less common is specific research on Parliamentary application of deliberative methods, and how to effectively evaluate and understand the impact of this work. 

One aim of this research project is to support the Citizen Participation and Public Petitions Committee in its aim of suggesting a set of practice principles and a governance and accountability framework which can be applied across all of these activities. Although there are currently a set of fixed processes, these are not part of a wider delivery structure, and there are no static evaluation and review processes in place.

As noted, the Committee Engagement Unit in the Scottish Parliament service was formed following the recommendation of the Commission on Parliamentary Reform in 2017, and has since expanded to become the Scottish Parliament's Participation and Communities Team (PACT).

This section focuses on the types of work done to date by PACT by taking examples from the practice history and sets out the current process for the use of deliberative practice. Note that because of the scope of this project, the focus is solely on deliberative practice, but this only forms part of the role and responsibilities of this service.

Deliberative activities have typically taken place to support the work of committees. The Scottish Parliament currently has 15 committees, most of whom focus on specific areas of policy. These committees will be responsible for the scrutiny of legislation and policy, which typically takes the form of an inquiry process. Some of this work will be allocated to committees, i.e., in the form of a draft Bill, others will be determined by Scottish Government timetables (such as scrutiny of the Budget), and other inquiries will be proactively pursued by committees.

Each committee has a 'Participation plan' created by the Community/Participation Specialist working in the wider committee support team. Design of the participation and who will lead on activities varies from Committee to Committee. Key functions include:  

• Community Participation - Working with partners, including third sector interfaces regionally, and other local and national voluntary and charitable organisations who work with the communities PACT and the committees want to reach.   

• Relationship development – PACT has its own Customer Relationship Management CRM system to send out requests, send regular newsletters on upcoming opportunities to participate and highlight key work completed. PACT has held two annual conferences inviting these partners to the Parliament to share practice, network and develop relationships. These conferences have included input and support from other offices, including SPICe, clerking and communications. 

• Specialised team – The team currently has five community participation specialists, all with experience of working with and for a range of communities across Scotland, bringing their own expertise and networks.  In addition, two participation specialists focus on the development of digital and deliberative methods, as well as ongoing support for any other Parliamentary business where participation is requested. 

Committee inquiries will typically begin with a scoping stage. At this point, the committee clerks will work with researchers in SPICe, participation specialists and communications officers to present options to the committee of how an inquiry could be focused, and how it might take place. This might also include consultation with external stakeholders and experts and will include consideration of the Committee's anticipated work programme.

Part of this process will include considering whether participation would support the scrutiny process, if it is appropriate, and what form it might take. Proposals and the design of participation and deliberative activities will then be taken to the committee for discussion and agreement.

The process from there will vary depending on the anticipated costs and competing bids for activities to take place. In most instances, the approval process will fall to Committee Office leadership. On occasion, where a panel is expected to feed into strategic aims, approval from the Conveners Group (the strategic group made up of all committee conveners and chaired by the Presiding Officer). Should committees wish to meet formally outside of Holyrood, or travel abroad as part of factfinding, Convenors Group approval must be sought. To support the decision-making process, a payment for participation policy was agreed by the Scottish Parliamentary Corporate Body on 22 Feb 2024 and will come into force on 1 April 2024, and a commissioning form for all PACT work is in development.

With Committee and, where needed, financial agreement, delivery will be driven by the Participation and Communities Team, but typically with regular and sustained input from colleagues in other teams. The model of delivery, and evaluation, will vary depending on the process being used and the situation.

Within these processes, there a suite of techniques used, including the involvement of external partners and stakeholders. These are explored in Section 7 and Annex 3 of this research briefing. The following section gives examples that show the wider context of how deliberative techniques have been applied.

The following sections outline examples of deliberative processes used to date in the Scottish Parliament.

• In March 2019 the first ever citizens' jury to be hosted by the Scottish Parliament considered the question of how funding and advice for land management should be designed to help improve Scotland's natural environment. Over one weekend, the 21 members of the jury heard from a range of experts about the topic and worked together to come up with a set of principles that the Environment, Climate Change and Land Reform (ECCLR) Committee should consider when exploring the issue of funding for land management and environmental impact. The jury also came to a consensus on preferred aspects for a new funding model. As is the case for later citizens' jury/panel activities, the Sortition Foundation was commissioned to recruit a random stratified sample broadly representative of the Scottish population (as at the 2011 Census), and an expert steering group was used to agree the final question being posed to the jury and ensure that the process was fair, credible and transparent.

• In early-2019, the use of three public panels covering the west, north and east of Scotland and formed the first half of the Health and Sport Committee's inquiry into primary care. The Sortition Foundation recruited a randomly selected and stratified sample for each panel, and each panel went through a two-day learning and deliberative process on a citizens' panel model. Each panel came up with suggested themes and priorities for the future of primary care. The panel process was independently evaluated by Dr Stephen Elstub, and alongside a public survey and submission form the Scottish Youth Parliament, the findings went on to inform the focus of the second half the Committee's inquiry.

• In 2021, a fully virtual citizens' panel on COVID-19 was held over four Saturdays. 18 panel members, recruited with the help of the Sortition Foundation, took part across all four sessions. The findings were used to support the COVID-19 Recovery Committee's work to scrutinise the Scottish Government's response to the COVID-19 pandemic at a time when the health protection measures put in place to respond to COVID-19 continued to have a considerable impact on all people in Scotland.

• In Autumn 2022, as referenced elsewhere in this paper, a citizens' panel took place over two weekends to answer the question "How can the Scottish Parliament ensure that diverse voices and communities from all parts of Scotland influence our work?". The panel's 19 recommendations were then explored with further evidence-taking before being (for the most part) endorsed by the Citizen Participation and Public Petitions Committee, who agreed with the panel that the Parliament should aim to institutionalise deliberative democracy.

• In February and March 2024 a People's Panel was formed to carry out post-legislative scrutiny of the Climate Change (Scotland) Act 2009. Participants produced a set of recommendations, which, at the time of this report being published, were soon to be incorporated in a report and then presented to the Net-Zero, Energy and Transport Committee. It is intended that the recommendations will feed directly into the Committee's scrutiny of the Climate Change (Scotland) Act. This panel is in part a pilot supporting the Citizen Participation and Public Petitions Committee's aspirations, in which it requested that two people's panels take place, one on post-legislative scrutiny and another on a topic of current interest. The latter of these panels will take place in late-2024.

• In 2019-20, the digital platform Your Priorities was used to inform the Local Government and Communities Committee's scoping of work into community wellbeing. Over 600 people contributed to the discussion and prioritised ideas, and, combined with the input of 65 people across five community-based sessions, this gave the Committee a clear steer on where to focus scrutiny. The outcome was the Committee's decision to carry out post-legislative scrutiny on the Community Empowerment (Scotland) Act 2015.

• Your Priorities was used again in early-2023 to prioritise and gather further evidence on the recommendations made by the Citizens' Panel on Public Participation.

• The platform has also been used to support committees to scope inquiries with the public; crowdsource questions to be put to the Scottish Government in Committee meetings; enable the public to comment on and discuss legislation; and provide feedback on government implementation of policy.

• The use of Citizen Space for written submissions has supported an increase in submissions and has allowed Committees to vary their approach to receiving written evidence.

• As part of its pre-budget scrutiny for the 2024-25 budget, the Equalities, Human Rights and Civil Justice Committee commissioned 12 members of an existing external citizens' panel to explore the human rights implications of the Budget from an intersectional and ethnic minority perspective. Following half-day education and discussion sessions, the panel came up with six questions for the Scottish Government. In October 2023, members of the panel gave evidence to the Committee, then the Committee asked the panel's questions verbatim to the Minister for Equalities, Migration and Refugees. The outcomes supported the Committee's exploration into the use of participation and deliberation in budget scrutiny and demonstrated the impact of smaller scale participation on the scrutiny process.

• The Health, Social Care and Sport Committee ran a series of lived experience panels with groups of stake holders to support post-legislative scrutiny of the Social Care (Self-directed Support) (Scotland) Act 2013. Participants from five informal engagement workstreams were asked to develop a set of recommendations for the Health, Social Care and Sport Committee on what it should focus on in phase two of its scrutiny, and presented these to the Committee in February 2024.

• From 2017-2020 the Young Women Lead Committee, a committee of 30 young women between ages 14 and 30 who live in Scotland, was run in partnership with the Scottish Parliament. The Committee is part of Young Women Lead (YWL), a leadership programme which focuses on increasing political participation. In each year, the YWL Committee held its own inquiry in the Parliament, and relevant Parliamentary subject committees have followed this up. In 2017-18, the Committee explored sexual harassment in schools. In 2018-19, the focus was on issues affecting the participation of young women in sport and physical activity. And in 2019-20, the Committee looked into the transition from education to employment for young women in Scotland from ethnic minorities. Consideration is currently being given to how to reintroduce Young Women Lead in the Parliament.

• From January 2024 onwards the Local Government, Housing and Planning Committee is running a series of lived experience housing panels in order to inform its scrutiny of legislative proposals concerning the private rented housing sector.

In response to recommendation 8 from the Citizens' Panel on Public Participation to Build a strong evidence base for deliberative democracy to determine its effectiveness and develop a framework for measuring impact, the next section is designed to support this strategic path towards institutionalising deliberative democracy by first considering what the core underlying principles of a deliberative democracy approach in the Scottish Parliament might be; and how these principles might be reflected in both a governance process (with or without legislation), and in a monitoring and evaluation framework which uses clear and measurable outcomes to understand impact.  

Here, it is important to recognise that the future shape of the Parliament's deliberative democracy model will continue to evolve, but it is possible at this stage to commit to the aims below which underpin the principles and guide the process of developing an evaluation model. Based on the Panel's recommendations and its own subsequent factfinding, the Citizen Participation and Public Petitions Committee recommended:

• That deliberative democracy should complement the existing model of representative democracy and be used to support the scrutiny process. 

• That the way in which deliberative methods are used, from recruitment through to reporting and feedback, should be transparent and subject to a governance and accountability framework.  

• That the deliberative methods used should be proportionate and relevant to the topic, and the scrutiny context. 

• That participants in deliberative democracy should be supported, empowered and given feedback on how their recommendations are used.

The core principles (figure 2, below) have been identified with the above recommendations in mind and developed through a scoping review of the deliberative material, both practice based and theory based, as well as from reviewing the documents from the Citizens' Panels.  They have also been informed by the expertise and knowledge of PACT and SPICe Senior Researchers, and through academic application.

Vitally though, while democratic norms and key principles have been set out in Section 2 from the deliberative democracy literature, they have been adapted here specifically for use as part of an internal parliamentary scrutiny process and not an external deliberative institution. This distinction is important, as most existing examples of deliberative practice are driven from an executive level, rather than from a scrutiny perspective.

For this reason, some principles have been subsumed under different headings in order to ensure it is conducive to achieving the recommendation of the Citizens' Panel but with the recognition that it must work within the representative democratic framework of the Scottish Parliament. Within internal deliberative scrutiny processes, partnerships are likely to be built through political buy-in and public input rather than power-sharing and public led initiatives (although the public may still have the opportunity to set agenda and highlight important issues).

Furthermore, the quality of deliberation, while important, it is not deemed more important than reaching decisions, therefore principles like quality of discourse and considered judgement would be included under effective scrutiny and citizen empowerment. 

The key principles and interactions with evaluation areas are explained below: 

Political buy in. This principle refers to the support and commitment needed from the policy actors for the adoption of deliberative practice in policy design, not in theory but in practice.  This includes a willingness to engage with the processes connected with their policy areas, to accept the resultant recommendations and outcomes, or to offer explanations as to why they cannot accept the recommendations. Processes which are explicitly supported or initiated by politicians are more likely to be 'successful' i.e. have political impact because it has political buy-in. In addition, politicians witnessing the scope of public input and their capabilities will improve relations and assure politicians that public input is valuable (see Michels and Binnema 2019¹Michels, A., & Binnema, H. (2019). Assessing the Impact of Deliberative Democratic Initiatives at the Local Level: A Framework for Analysis. Administration & Society 2019, 51(5), 749-769. doi: 10.1177/0095399718760588, Caluwaerts & Reuchamps 2016²Caluwaerts, D., & Reuchamps, M. (2016). Generating democratic legitimacy through deliberative innovations: The role of embeddedness and disruptiveness. Representation, 52, 13-27.).

Public input. Public input is the ability of citizens to operate alongside government officials and public authorities. Citizens are also more likely to accept decisions, even when they go against their own views, when those affected by them are involved in decision-making. This links with the willingness of the public to be involved, to know what is going on and to trust in the process. To witness decisions doesn't necessarily need to come from participation, and the Parliament and media have a role in making proceedings transparent and open (see Esaiasson et al. 2019³Esaiasson, P., & et al, . (2019). Reconsidering the role of procedures for decision acceptance. British Journal of Political Science, 49, 291-314.; de Fine Licht et al. 2014⁴De Fine Licht, J., & et al, . (2014). When does transparency generate legitimacy? Experimenting on a context-bound relationship. Governance, 13, 349-365.; Clayton et al. 2019⁵Clayton, A., O'Brien, D.Z., & Piscopo, J.M. (2019). All male panels? Representation and democratic legitimacy. American Journal of Political Science , 63(1), 113-29.; Rasmussen and Reher 2022⁶Rasmussen, R., & Reher, S. (2022). (Inequality in) Interest Group Involvement and the Legitimacy of Policy-Making. British Journal of Political Science.). 

Scrutiny. Scrutiny of proceedings or evidence can have myriad of meanings. It can include scrutiny of the effectiveness of policy design (looking for strengths, weaknesses, good or bad practice, or gaps), implementation (impact assessment of policy), cost effectiveness of policy, and/or scrutinising from a particular standpoint – i.e. applying a gender analysis and mainstreaming the policy. Evidence can be presented orally, written, and/or digitally etc. Scrutiny can be undertaken by fact checking and evidence formation, asking questions and deliberating over possible outcomes and impacts, and being supported by experts to do this. The importance of capacity building and up-skilling participants ensures they are prepared and supported to be critical participants, able to apply considered judgement to proceeding, which can only be determined through the understanding of technical details by the relevant participating citizens. Drawing on lived experience and local knowledge can offer a valuable degree of scrutiny too. The chair or facilitator will be key to the independence and effectiveness of the scrutiny that takes place by guiding the agenda and ensuring evidence is robust (see Roberts et al. 2020⁷Roberts, J.J., Gooding, L., Ford, R., & Dickie, J. (2023). Moving From “Doing to” to “Doing With”: Community Participation in Geoenergy Solutions for Net Zero—The Case of Minewater Geothermal. Earth Sci. Syst. Soc. , 3(10071). doi: 10.3389/esss.2023.10071, Geddes 2019⁸Geddes, M. (2019). The effectiveness and influence of the select committee system, House of Commons Liaison Committee. Retrieved from https://www.gov.scot/publications/advice-inform-scotlands-open-government-public-participation-strategy/documents/ [accessed 24 February 2024]). The use of an independent expert stewarding board in the design of activities also supports this aim.

Inclusiveness. It is vital that a process considers limits to inclusion and areas of exclusivity. How accessible proceedings are and how equality, diversity and inclusion (EDI) considerations are implemented are key to the quality of process. External and internal inclusion enable us to consider where the key areas need to be focused. External inclusion refers to the ability to access a process, whereas internal inclusion refers to the quality of interaction and communication once the process is underway (see Young 2000⁹Young, I.M. (2000). Inclusion and Democracy. Oxford: Oxford University Press., Lightbody 2017¹⁰Lightbody, R. (2017). Hard to reach’ or ‘easy to ignore’? Promoting equality in community engagement’. Centre for Families and Relationships, What Works Scotland Evidence Bank. Retrieved from http://whatworksscotland.ac.uk/publications/hard-to-reach-or-easy-to-ignore-promoting-equality-in-community-engagement-evidence-review/ [accessed 24 February 2024]).

Transparency and openness. The public understands how the issue being considered has been selected, who is organising the proceeding, how decisions about the proceeding are being made and finally, how the outcome of proceedings will affect political decisions. Bringing the Parliament and its work closer to the public, opening up the scrutiny process and engaging them in policy scrutiny, supporting groups of people to engage with the policy process in a longer process, and supporting their understanding of policy through upskilling (O'Hagan 2017¹¹O'Hagan, A. (2017). Gender budgeting in Scotland: A work in progress. Administration, 65(3), 17-39. doi: 10.1515/admin-2017-0022) will accommodate openness and trust. People will have the opportunity to become more aware of what is happening and how the Parliament and policy process works. 

Legitimacy: The legitimacy of any democratic process is key to ensuring that the process is respected, and that the outcomes are seen as valid and democratic, which ensures that all actors are prepared to comply with those rules. While the deliberative processes will not be able to draw legitimacy from elections, they can draw legitimacy by following procedure, including factual and robust evidence, reasoned and informed deliberation, being open and transparent, and undergoing a robust evaluation process, but further to this, they must be perceived as legitimate by both the public and the political actors (Sandover et al. 2021¹²Sandover, R., Moseley, A., & Devine-Wright, P. (2021). Contrasting Views of Citizens’ Assemblies: Stakeholder Perceptions of Public Deliberation on Climate Change. Politics and Governance, 9(2), 76-86., Moulds 2019¹³Moulds, S. (2019). From Disruption to Deliberation: Improving the Quality and Impact of Community Engagement with Parliamentary Law Making, Paper for Presentation at ASPG Annual Conference Parliaments navigating disruption. Retrieved from https://www.aspg.org.au/wp-content/uploads/2019/10/20190915-Moulds-APSG-Paper-.pdf [accessed 24 February 2024]).

Policy impact: Most research indicates that policy change is a vital test of impact. Policy response can include changes to policy and legislation and resulting action and/or amendments in the policy area. Therefore, policy response may mean clear change in policy design and implementation.  It can also include changes to political debate/positions on issues as well. But, importantly, it can also mean no change, if the deliberative process indicates this is necessary. Therefore, measuring policy impact may include evaluating the capacity to shape and improve policy through policy responsiveness (see figure 1, Demski and Capstick (2022¹⁴Demski, C., & Capstick, S. (2022). ‘Impact evaluation framework for climate assemblies’, KNOCA. Retrieved from https://knoca.eu/app/uploads/2022/11/Impact-evaluation-framework-for-climate-assemblies-1.pdf [accessed 24 February 2024]). Within a scrutiny context, it is more challenging to aaffect policy change and assure participants that their aspirations will be met, as the Scottish Parliament and its Committees are not the enactors of policy. Rather, impact might be visible within the content, tone and strength of the committees’ recommendations for the Scottish Government, their follow-up of these recommendations, and the changing tone of debate. Paramount in these instances is a clear demonstration that citizens' voices have been heard and reflected in the committees’ own decision-making process. One area where impacts may be more visible is during the legislative process, where committees and the Parliament as a whole play a greater role in the design and implementation approach to policy.

Citizen empowerment: This explores the longer-term public input on the proceedings of parliamentary committee work, on the way laws are debated in the Parliament, the link between law makers and people of Scotland, the creation of a culture of scrutiny by an engaged citizenry.

Deciding on the type of impact framework that would work for Scotland's geopolitical sphere, which can capture all the various working parts requires an understanding of all the different types of assessments and how they work currently. Therefore, this is not an exhaustive list but an important start in developing a framework which can be adapted and applied flexibly. It is also important to recognise that the internal deliberative practices would be aligned, coupled and linked with other democratic processes, so they would not be expected to achieve all principles at the same time and to the same extent.

The following section suggests questions and areas to consider when evaluating whether the core principles are evident. Not all these questions will be applicable in all processes but can offer a useful starting point. 

Political buy-in

• Have politicians and civil servants been willing to enter into these conversations and commit to the process?

• How do policy makers use this evidence?

• How do they engage with the process?

• How supportive are they?

• How do petitions, consultations, deliberative processes etc. impact on or feature in parliamentary comments, motions, bills etc.

Increased public input

• Is there a willingness to engage with the process?

• Is there anyway of including the media?

• Do evaluations and consultations indicate that trust is increasing? 

• Are participants supported throughout and after the process?

• Does the public remain engaged with Parliament following the process?

• Is there an increased interest or engagement with the Scottish Parliament?

Scrutiny of evidence/proceedings

• How is the quality of evidence been assured? (has it been fact checked? etc)

• Does the process support members to question and hold policy makers and the policy process to account?

• Is there support to develop participant's skills to scrutinise evidence?

• How is the evidence presented? (consider the accessibility, readability or if orally, time for questions/reflections?

• Who are the experts and who recruited/invited them? (links with policy change/impact)

Inclusiveness

• What checks have been put in place to assure process is inclusive?

• What accessibility issues were identified and how were they dealt with/overcome?

• Who identified accessibility issues? Were participants able to highlight accessibility issues themselves?

• How is it linked with other processes to widen participation?

Transparency

• Does the process make the parliament more transparent? (perception)

• Does it clarify roles and responsibilities for policy issue action?

• Is the wider public aware that the process is taking place?

• Is how the outcome/outputs will be used known to the participants, wider public and political actors?

• How might wider public, political actors, groups feed in if required?

Legitimacy

• Does it follow rules, advised best practice?

• How legitimate is it viewed to be? (by public, by political actors)

• Has it increased trust in the policy process? (by public, by political actors)

• (how) is it being externally evaluated?

Policy impact

• Has it improved the policies and policy making process?

• Who listens and responds to output?

• Have there been changes to (or new) political coalitions, networks, or cross-party collaborations engaging with processes?

• If there has been no change, are there reasons for this and are the participants satisfied with these reasons?

Citizen empowerment

• How do participants feel following the process? (reflection) how do they view the fairness, impact, legitimacy etc.)?

• How does wider public feed into these processes?

• Is there an increased uptake in participation and engagement with parliament?

• Is there an increased awareness and trust in processes?

• How does the process influence, shape or feed into civil society, how do civil society and media/other groups make use of outputs/recommendations?

Evaluating a deliberative process fulfils several purposes.

Evaluation can:

• Strengthen key actors' (such as citizens and politicians) value and trust in the process.

• Ensure quality and neutrality of process is adhered to – committing to a transparent process.

• Improve outputs and outcomes by increasing legitimacy and understanding.

• Help organisers to learn and develop best practice.

• Give public trust in parliament (OECD 2021¹OECD. (2021). Evaluation Guidelines for Representative Deliberative Processes. Retrieved from https://www.oecd-ilibrary.org/sites/10ccbfcb-en/index.html?itemId=/content/publication/10ccbfcb-en [accessed 24 February 2024]).

The OECD recommends that evaluation is undertaken by independent bodies. They consider this to be salient for longer events (over four days) or more complex/politically sensitive processes. Evaluation undertaken by trained evaluators, who are experts in process, can be objective and fair. Actors and groups are considered independent if they have not been involved in the process or are politically conflicted.

If this is not possible then self-reporting can be necessary and is particularly suitable for shorter processes. This includes feedback from the members involved – usually anonymously – and can also include organisers, facilitators and expert contributors. These actors can offer solutions or recommendations in moving forward. Members of Parliament, parliamentary staff and individuals can be trained relatively easily to do this. Another option is opening the process up for the wider public, peers and political actors to observe and evaluate the process.

It is key to remember that there are many methods to evaluate deliberative processes and many aspects of the deliberative process that require evaluating, including the quality of deliberation, transformation of preferences, consensus, equal participation and quality of output which have not necessarily been included under these principles.

The following table concentrates on tools for evaluating deliberative practice within Parliament. Where external stand-alone processes may prioritise deliberative capacity and have wider scope or impact, here we prioritise the ability to evaluate the core principles set out in figure 2. The exact metrics will depend on the scale and duration of the process, and its purpose. It may be appropriate to discuss evaluation priorities and metrics with the participants, evidence-givers and facilitators or even the wider public to assure that the evaluation is evaluating what they would like it to i.e. what do they feel to be important? The following evaluation methods are recommended by many key researchers and organisations (including but not limited to Warburton et al. 2007²Warburton, D., Wilson, R., & Rainbow, E. (2007). Making a Difference: A guide to evaluating public participation in central government. Retrieved from https://www.involve.org.uk/sites/default/files/uploads/docuemnt/Making-a-Difference-.pdf [accessed 24 February 2024], Michels and Binnema 2019³Michels, A., & Binnema, H. (2019). Assessing the Impact of Deliberative Democratic Initiatives at the Local Level: A Framework for Analysis. Administration & Society 2019, 51(5), 749-769. doi: 10.1177/0095399718760588, OECD 2021¹OECD. (2021). Evaluation Guidelines for Representative Deliberative Processes. Retrieved from https://www.oecd-ilibrary.org/sites/10ccbfcb-en/index.html?itemId=/content/publication/10ccbfcb-en [accessed 24 February 2024], Demski and Capstick 2022⁵Demski, C., & Capstick, S. (2022). ‘Impact evaluation framework for climate assemblies’, KNOCA. Retrieved from https://knoca.eu/app/uploads/2022/11/Impact-evaluation-framework-for-climate-assemblies-1.pdf [accessed 24 February 2024], Curato et al. 2023⁶Curato, N., Chalaye, P., Conway-Lamb, W., De Pryck, K., Elstub, S., Morán, A., … Werner, H. (2023). Global Assembly on the Climate and Ecological Crisis Evaluation Report. Retrieved from https://researchprofiles.canberra.edu.au/en/publications/global-assembly-on-the-climate-and-ecological-crisis-evaluation-r [accessed 24 February 2024]) and collated and adapted here for Parliamentary proceedings.

While this report looks at the deliberative practices in the Scottish Parliament and sets out a suggested framework for evaluating the principles in practice, it is key to connect that with the wider work of the Parliament. More repositories and databases are popping up (see OECD, Participedia, LATINNO and on a smaller scale the Scottish Poverty and Inequality Research Unit's Local Directory) and it would be wise for the Scottish Parliament to follow suit, and for it to lead the charge for Parliamentary transparency and development of databases. Building a repository of cases studies will provide a long-standing and useful resource on knowledge and good practice, and highlight areas to grow. It will facilitate a national conversation and excite international interest from policy makers, practitioners, academics, and the public.

In Annex 3 we start to build such a resource. Using Roberts et al. (2023)¹Roberts, J.J., Gooding, L., Ford, R., & Dickie, J. (2023). Moving From “Doing to” to “Doing With”: Community Participation in Geoenergy Solutions for Net Zero—The Case of Minewater Geothermal. Earth Sci. Syst. Soc. , 3(10071). doi: 10.3389/esss.2023.10071 schematic ladder of participation (in Annex 2) we map different participatory processes which have taken place in or with parliamentary members. Here, PACT has begun work on detailing the different type of work happening, highlighting where processes work 'with', 'for' and 'to' citizens. For different aims, issues and outcomes, different processes will work and while it is not always appropriate to power-share or for processes to be citizen driven, sometimes it is. PACT shows examples of where there are opportunities to work 'with' citizens (i.e. People's Panels), but also to work 'for' them (Inquiries and Committees) and sometimes to be doing outreach 'to' the public (one-way information campaigns and educative sessions). 

Combining these practices and mapping good practice is fundamental to achieving the principles set out in this report and delivering wider civic engagement and longer deliberative practice through up-skilling. Ultimately participatory and deliberative processes aim to develop a critical body of citizenry capable of recognising credible sources, unpacking government information, empowering citizens to scrutinise the policy process and the work of the Parliament.

Work is evolving and ongoing in this area and we should not assume this to be an end point. The principles for this type of deliberative practice will not necessarily be the same for deliberation happening elsewhere, such as large and national Citizens' Assemblies or activities at a community level. However, a series of recommendations can be made from the evidence to date, and here we set out a summary of the main recommendations and future work.

1. Political buy-in is complex to measure. Political actors (MSPs) must show willingness to engage and show evidence of using recommendations moving forward. Tokenistic gestures of engagement will only succeed in breaking down relations between the public and Parliament.

2. Barriers to engaging with these sorts of processes exist, which impacts on public input and inclusiveness. Work must be ongoing to ensure activities are inclusive with proactive efforts to support wide ranging engagement. This requires a recognition that many individuals need support to engage. Barriers are multifaceted, and practitioners should learn from past recruitment and let participants feedback about what has helped them to participate. Crucially, they should also give opportunity to those that cannot participate to feedback about the challenges they faced in participating. (see Public participation in the Parliament report Key Message 8)

3. Openness and transparency are key to developing public trust and public engagement so making processes transparent, and feeding back in a timely manner to participants is essential. The Citizen's Panel's recommendation 12 recommends nine months after an activity as a default time to feedback and update participants, but there is also merit in ensuring the wider public is abreast of activities and can stay informed if they wish to.

4. Deliberative processes will not always result in change, but the policy scrutiny it affords is designed to improve policy, by making scrutiny more transparent, and decisions more robust and sustainable. Expectations of participants and political actors should be managed.

There are key areas to remember when putting this framework into practice, which the next section explores.

1. Evaluation is paramount for gauging impact, experience and improving practice. Robust evaluation contributes to legitimacy and accountability. Furthermore, evaluation ensures the deliberative practice is doing what it is designed to do.

2. Common sense must be used for resourcing evaluation. The amount of resources (time, money etc.) should be proportionate to the process. Previous external evaluation contracts for citizens’ panels have run to £5000-6000 per panel, not including the internal resource required to recruit and manage an evaluation contract, the use of this approach should be proportional, and the requirement for use clear. A small, day long process could be evaluated internally, depending on the policy area, and a longer event (four days) would require more time and a more robust evaluation process using external actors, such as practitioners and critical friends.

3. It is vital to consider the timing of the deliberative activity. Timing is paramount for the success of a deliberative initiative and an ill-timed process (i.e. too late in the wider scrutiny process) can be a futile exercise. The timing of the activity in terms of accessibility for participants should also be considered. For instance, consider using evening and weekends to support wider participation. The overall delivery of a deliberative process is key in order to manage expectations and feasibility. This should include allowing enough time to hold steering groups, secure participation of experts, prepare materials, and for development of any panel-commissioned sessions between activity days. Therefore, when considering the use of a deliberative process, timing is key.

4. The deliberative process must be proportionate and relevant to the topic. PACT has drawn together a useful toolkit of different deliberative methods and facilitation methods which can help the user identify what is right for their needs.

5. Any deliberative process should ensure that its members are supported and empowered. This can be achieved by adhering to the principles and ensuring the members know what the outcomes might be and how they will be used. In addition, facilitators and any experts who are brought in to contribute to the process must also receive support and transparency from organisers.

6. The core principles are not exhaustive, and their application must be proportionate to the deliberative process. It may not be possible to achieve all principles all in one process but adhering as closely as possible will ensure that the purpose of the process is met, and participants are valued.

Based on the findings of the research the following sets out recommendations for future work:

The tables in this section show PACT activity to date, mapped out against Roberts et al's Ladder of Community Participation. This is not an exhaustive list of activity, or a fixed format, rather it aims to demonstrate how PACT work might be set out in a repository of practice as per the repost section 'Evidence, case studies and repositories' and recommendations on 'Evaluating deliberative practice'.

Each table represents a different step on the ladder - doing 'to', doing 'for', and doing 'with'