UNIVERSITY OF OKLAHOMA GRADUATE COLLEGE ENGAGING WITH STAKEHOLDERS TO PRODUCE ACTIONABLE SCIENCE: MOVING FROM THEORY TO PRACTICE A DISSERTATION SUBMITTED TO THE GRADUATE FACULTY in partial fulfillment of the requirements for the Degree of DOCTOR OF PHILOSOPHY By APARNA BAMZAI-DODSON Norman, Oklahoma 2022 ENGAGING WITH STAKEHOLDERS TO PRODUCE ACTIONABLE SCIENCE: MOVING FROM THEORY TO PRACTICE A DISSERTATION APPROVED FOR THE DEPARTMENT OF GEOGRAPHY AND ENVIRONMENTAL SUSTAINABILITY BY THE COMMITTEE CONSISTING OF Dr. Renee A. McPherson, Chair Dr. Amanda E. Cravens Dr. Travis Gliedt Dr. Jennifer A. Koch Dr. Joseph T. Ripberger © Copyright by Aparna Bamzai-Dodson 2022 All Rights Reserved. Acknowledgements My journey to this dissertation has taken about fifteen years, with many ups and downs along the way. I changed institutions, advisors, and research topics before I figured out what sparked my joy, but now that I am here, this is truly a joyous moment for me. My dissertation is a labor of love that builds personally on my cultural inheritance from my ancestors and intellectually on the work of countless scholars and practitioners who came before me. Perhaps it is trite to quote Isaac Newton, but I cannot imagine a truer statement: "If I have seen further it is by standing on the shoulders of giants." I have so many to thank and am certain there are errors of omission in these acknowledgments – just know that I am eternally grateful to each and every one of you who has lent me your shoulder to stand on over the years. First and foremost, I’d like to thank my committee – Drs. Renee McPherson, Amanda Cravens, Travis Gliedt, Jennifer Koch, and Joe Ripberger. You all set the bar for open and collegial conversation that pushed my academic boundaries and helped me grow into a researcher in my own right. I feel so fortunate to be able to say that you made my general exam fun (!!!), even while still being a grueling experience. To Renee, thank you for your unwavering commitment to seeing me to the end of this journey. You mentored me, shared your insights with me, opened doors for me, and kept me going even when my own commitment wavered. To Amanda, thank you for showing me a new world of possibility and helping me find my place as a social scientist. And to Drs. Gabriele Hegerl, Inez Fung, and Kirsten de Beurs, I appreciate the parts of this journey that we walked together and how they helped me figure out who I am. Second, I’d like to thank the staff, students, investigators, and partners of the U.S. Geological Survey (USGS) National and Regional Climate Adaptation Science Centers (CASC). They have given so generously of their time to help me develop and pretest survey and interview protocols, collect data, and interpret results. My understanding of how actionable science is done and societal impact is achieved has been profoundly shaped by the people and programs of the CASC network. Special thanks to my supervisors, Mr. Robin O’Malley and Dr. Nicole DeCrappeo, for continuing to encourage and support my research, even when it meant pausing other initiatives. Third, I’d like to thank the broader research community with whom I’ve engaged in intellectual discourse and collaboration, particularly about climate impacts and adaptation science. I have had the privilege to interact with individuals and groups spanning, but not limited to, the University of Oklahoma Department of Geography and Environmental Sustainability, the CASC Evaluation Working Group, the USGS Human Dimensions Community of Practice, the Science of Actionable Knowledge Working Group, the American Geophysical Union Science and Society Section, the National Oceanic and Atmospheric Administration Regional Integrated Sciences and Assessments Program, the U.S. Department of Agriculture Regional Climate Hubs, the National Park Service Climate Change Response Program, and the Fish and Wildlife Service Science Applications Program. I look forward to many more future conversations and collaborations. To Dr. Jamie Clark and the USGS “social writing while social distancing” crew, thank you for being my virtual writing partners through the pandemic. Those sessions held me accountable for getting words on the page even if they needed significant editing later, and this dissertation wouldn’t exist without you. Fourth, I’d like to thank my family. To my Mom and Dad, Anjuli and Nalin, thank you for your patience as I wandered in life and for trusting that I was always moving forward. To my brother, Aditya, and his wonderful family, Divya, Shaila, and Rohan, thank you for an endless supply of stress-relieving stories about dinosaurs, horses, and legos. To my sister-in-law, Tiffany, and her boys, Addison and Caden, thank you for the memes, text games, and late night laughs. And to my extended family members around the world, thank you for reminding me where I came from and showing me where I can go. We are like that only. Last but not least, I have to thank my champions – my husband, James, and our dog, Chip. Every day, you carry the load when I am tired and remind me that I am more than my work. There are no words to convey just how much you mean to me. 2 vii Table of Contents Acknowledgements iv Table of Contents vii CH. 1 - Introduction 1 1.0 Abstract 1 1.1 Situating the organization 2 1.2 Situating the individual 5 CH. 2 - Engaging with stakeholders to produce actionable science 10 2.0 Abstract 10 2.1 Introduction 11 2.2. Engagement for actionable science in the real world 16 2.2.1 Inform 17 2.2.2 Consult 20 2.2.3 Participate 24 2.2.4 Empower 29 2.3. Guidance for actionable science engagement approach selection 34 2.4. Conclusions 37 2.5 Acknowledgments 39 Figure 2.1 - Adapted wheel of participation 40 Table 2.1 - Summary description of each approach from the adapted wheel 41 Table 2.2 - Guiding thought questions 42 CH. 3 - The road to actionable science is paved with good intentions 43 3.0 Abstract 43 3.1 Introduction 44 3.1.1 Background 44 3.1.2 Dialogues from geography that inform stakeholder engagement 48 3.1.3 Research ethics related to studies of human subjects 50 3.2 Methods 52 3.3 Results and discussion 55 3.3.1 Belmont principle #1: respect for persons 55 3.3.2 Belmont principle #2: beneficence 60 3.3.3 Belmont principle #3: justice 64 3.4 Conclusions 67 3.5 Acknowledgments 71 Table 3.1 - All emergent codes 72 Table 3.2 - Summary definitions and codes used for analysis 73 CH. 4 - When does science achieve societal impact? 74 4.0 Abstract 74 4.1 Introduction 75 4.1.1 Background 75 4.1.2 Evaluation theory and practice 76 4.1.3 Success and evaluation for actionable science 79 4.2 Case study 81 4.3 Methods 83 4.4 Results 85 4.4.1 Process: engagement in the process of knowledge production 85 4.4.2 Outputs and outcomes: production and use of outputs 86 4.4.3 Impacts: building of relationships and trust 89 4.5 Discussion 90 4.6 Conclusion 93 4.7 Acknowledgments 94 Table 4.1 - Response rate information for each region 95 Table 4.2 - Demographics of respondents 95 Figure 4.1 - Word cloud of data provided by projects 96 Figure 4.2 - Word cloud of reasons why respondents would collaborate again 96 CH. 5 - Conclusions 97 5.1 Summary 97 5.2 Recommendations 99 5.3 Next steps 100 References 102 Appendix A - Survey data public summary 139 2 2 CH. 1 - Introduction 1.0 Abstract Over time, the idea of the public value of federally funded science has slowly transitioned from basic science that helps fight disease and maintain national security (Bush 1945) to use-inspired science that directly informs decisions about the most urgent issues facing society, such as climate change (Lubchenco 1998, 2017). Natural and cultural resources across the world are already experiencing demonstrable impacts due to changes in our climate system, and stewards of these resources are turning to the scientific community for actionable science – information and tools that can be directly applied to decisions about how best to adapt to these future conditions (IPCC 2022). Such societal impact is more likely to be met when decision makers are engaged in the process of knowledge production (Ferguson et al. 2022). In response, public funders of climate science are changing how they design solicitations, review proposals, and make other programmatic decisions to encourage research to meet decision making needs (Arnott et al. 2020a). However, when the knowledge created does not fit decision contexts or is not used appropriately, vulnerability or contributions to climate change can instead increase and maladaptation can occur (Barnett and O’Neill 2010). Ensuring that engagement of decision makers in actionable science is carried out in a thoughtful and reflexive way is critical to achieving the desired societal impact. In this dissertation, I examine three key questions regarding the engagement of stakeholders in the production of actionable climate adaptation science. First, how should researchers align their stakeholder engagement processes with their desired goals for societal impact and actionability? In Chapter 2, I propose a framework of engagement approaches that describes the wide variety of tools that are available for including stakeholders in the creation of actionable science, and I provide guidance on how researchers might consider tradeoffs among those approaches and tools. Second, how should researchers conduct engagement for societal impact in a way that maximizes benefits and minimizes harms to stakeholders? In Chapter 3, I analyze interviews (n=15) with stakeholders who were highly engaged in actionable climate adaptation science projects to examine their perspectives on the benefits and harms that they experienced, and I argue that researchers must proactively consider the ethical implications of engagement when developing their project idea. Third, how should researchers define successful societal impact and evaluate against such standards? In Chapter 4, I draw on the discipline of evaluation to develop a survey tool to examine the process, outputs, and outcomes of actionable science based on the perspectives of stakeholders engaged in those projects, and I analyze survey responses (n=49) in a case-study deployment of the tool. In Chapter 5, I synthesize the findings from Chapters 2-4 and summarize key takeaways. Overall, I recommend that researchers thoughtfully consider stakeholder engagement goals and benefits as early as possible to best meet expectations of societal impact and actionability, ideally at or prior to the proposal development stage. Doing so in a robust manner often involves skills in which many biological and physical researchers may not be trained, requiring additional resources and expertise to be included in project plans. 1.1 Situating the organization This dissertation centers around a case study of the U.S. Geological Survey’s National Climate Change and Wildlife Science Center, created in 2008 by Congress to “help managers of the country’s fish and wildlife resources respond to climate change.” This role was expanded by the U.S. Secretary of the Interior to include oversight of eight newly established regional Climate Science Centers (Salazar 2009) to fund and conduct climate impacts and adaptation research. In 2018, these centers were renamed to the National and Regional Climate Adaptation Science Centers (CASCs), and in 2021, a ninth regional center was created in the Midwest by subdividing the Northeast region. The Federal Advisory Committee on Climate Change and Natural Resource Science (ACCCNRS) was created in 2012 to provide guidance about the CASCs to the Secretary of the Interior. Three years later, ACCCNRS delivered a report recognizing the achievements of the CASCs and providing recommendations on how to improve their operations (ACCCNRS 2015). The report emphasized the need for continued provision of actionable science -- research that “provides data, analyses, projections, or tools that can support decisions regarding the management of the risks and impacts of climate change” (ibid). Since their creation, the National and Regional CASCs have established a portfolio of hundreds of projects that examine climate impacts on natural and cultural resources and inform decision makers on adaptation strategies. Researchers funded by programs such as the CASCs produce information that is considered actionable for a variety of uses by decision makers, ranging from being generally better informed about a subject to implementing a specific action (VanderMolen et al. 2020). Tools and techniques for examining the societal impact of science have been developed for use by other federal funding programs, and understanding previous lessons learned can help lay the groundwork for applying them to science produced by the CASCs. For example, within the National Oceanic and Atmospheric Administration’s Regional Integrated Sciences and Assessments program, the Western Water Assessment has conducted archival research and semi-structured interviews with selected researchers and stakeholders to examine the overall impact of their program (McNie 2011). Additionally, the Great Lakes Integrated Sciences and Assessment Center has examined how boundary organizations can serve as partnership networks to provide climate services to stakeholders and improve the usability of climate information (Lemos et al. 2014). The interagency Joint Fire Science Program has developed an Evaluation Resource Guide (Singletary et al. 2015) that outlines how an individual exchange network can use a survey instrument to measure and report the overall educational impact of their activities on their regional fire management community. Even with these resources available, there is a lack of clarity around what methods are most effective for achieving societal impact and in what contexts those methods should be deployed. As an employee of the CASC network (Deputy Director, North Central CASC) and a doctoral candidate at the University of Oklahoma, I was uniquely positioned to utilize the CASCs as a case study to examine questions around the societal impact of science. In particular, the challenge of defining success and conducting evaluation was an emerging priority in the CASC network. Using skills and expertise gained within both roles, I was able to examine my three key motivating questions: How should researchers align their stakeholder engagement processes with their desired goals for societal impact and actionability? (Chapter 2) How should researchers conduct engagement for societal impact in a way that maximizes benefits and minimizes harms to stakeholders? (Chapter 3) How should researchers define successful societal impact and evaluate against such standards? (Chapter 4) 1.2 Situating the individual Traditional approaches to physical science (i.e., positivism, Khagram et al. 2010) encourage pursuit of an unbiased and objective truth, removing the scientists from the science (Webb 1992), for example by discouraging the use of first-person pronouns in academic writing. Actionable science acknowledges that individual world views and value systems may shape one’s truths (i.e., constructivism, Khagram et al. 2010). Who someone is and how they see the world influences how they make decisions about and carry out their research, which is often referred to as researcher positionality (Holmes 2020). People have multiple salient aspects of positionality or have multi-faceted positionalities that shape how they act in the world (ibid), and I am no exception. By acknowledging and reflecting upon my positionality, I hope to better document my biases, take greater care in my research, and engender confidence in my findings. Here, I consider my positionality across three dimensions: identity, journey, and opportunities (Hampton et al. 2021), and I describe how these considerations influence my approach to conducting research. This positionality statement is an imperfect offering, and I will continue to iteratively reflect on my positionality across these dimensions over my research career. The elements of identity are characteristics that define and categorize individuals into social groups, such as race, gender expression, and class (Hampton et al. 2021), As a researcher conducting actionable science, it is important to recognize these characteristics and understand how they influence power dynamics and perceptions in working relationships. I am a cisgender, heterosexual woman of color. I grew up in a family with a high degree of scientific fluency and professional accomplishments. Although I am a first generation immigrant and a naturalized U.S. citizen, English is my first language. I have lived and worked in urban areas on both the East and West American coasts and in semi-rural areas in the central U.S. I am married to a cisgender white man who grew up working on a multi-generational farm in the rural Appalachian foothills. I have faced adversity as a result of my gender, ethnicity, and immigration status, but I have also had the privilege of financial stability, higher education, and fluency of language. To navigate these varied identities, I practice cultural and linguistic code-switching, which entails modifying my interactions with others to best fit contextual norms (Molinsky 2007). A researcher’s journey is the events that brought them to and influenced them to pursue a given topic (Hampton et al. 2021). The journey of an actionable science researcher is integral to understanding their motivations and what experiences and expertise they bring with them to a project. I hold a professional master’s in environmental management, and I completed three years of coursework in a doctoral program in earth and planetary science. For my master’s project, I examined global precipitation trends over time in two distinct satellite-derived data sets and found that in regions where precipitation totals had an increasing trend, the larger totals appeared to come from heavier precipitation events instead of from a greater number of rain days (Bamzai 2007). My unfinished doctoral research focused on spatial and temporal interactions between the biological and physical components of the hydrologic cycle based on observational analysis of field sites in Northern California and numerical modeling. While having a robust understanding of physical climate science is integral to my ability to participate in the dialogue around climate impacts and adaptation, I realized it was not the area where I wanted to focus my own research, so I decided to withdraw from my doctoral program prior to degree completion. I am personally motivated to contribute to the conversation on societal response to climate change and need to feel a connection between the science that I am doing and society. My family are Kashmiri Pandits, and our story is one of exodus and exile (Bamzai 1994). While there is a long and complicated history of conflict in the Kashmir Valley, its modern geopolitical classification as a disputed territory (administered in pieces by India, Pakistan, and China) is a direct consequence of the post-colonial Partition of India in 1947 (ibid). As a result, I have a complex relationship with the legacy of colonialism on people and places. My role as the colonized, while not directly experienced, has been infused into the very fabric of my being, and as an immigrant, I could be perceived as a colonizer (or an agent of colonization). Recognizing this tension within myself, I try to foreground considerations of decolonization, equity, and justice in my work (Trisos et al. 2021). Opportunities are unique abilities or characteristics of a researcher that provide them with access to information or communities to which others may not have access (Hampton et al. 2021). Opportunities may result in novel research design or data collection that might otherwise be impossible (ibid). I work for a network of regional centers that fund and produce climate impacts and adaptation science, information, and tools to support resource management plans and decisions (described in Section 1.1). I spent four and a half years working for the South Central CASC, and I am now in my fifth year of working for the the North Central CASC, which serves managers across the states of Montana, Wyoming, Colorado, North Dakota, South Dakota, Nebraska, and Kansas with science for on-the-ground climate adaptation (Averyt et al. 2017). Both of these positions opened many doors for me, including being able to enroll in my current doctoral program part time while concurrently working full time. I have connections and access to scientists and practitioners working at research institutions and in resource management agencies, as well as insight into the program from which their projects received funding. So what does this mean for how I approach my own research? Foundationally, I honor and respect the connection that Tribal and Indigenous peoples have as caretakers of the lands that comprise the United States of America, both as a result of historical ties to their homelands and as a result of settler and colonial policies. I also honor and respect the connection of African peoples and their descendants as caretakers of these lands as a result of over 400 years of forced enslavement and subjugation. I carry the burden of my own internal biases and must actively strive for my work to be feminist and anti-racist and to counter colonial and parachute science (Trisos et al. 2021). How I carry out my work is shaped by my desire to understand the “why” and “how” of things (Khagram et al. 2010) and learning about how different parts of a system interact and work together. I have enough technical coursework to understand the physical science of climate change and climate modeling. I also can bring in my client-oriented consulting background and skills to think about science use. I value working collaboratively and building consensus across the boundary between research and management. I read literature from a range of academic journals and practitioner information sources (e.g., newsletters) that use their own set of specialized language, which allows me to code switch (Molinsky 2007) across cultural and disciplinary barriers. I am particularly drawn to geography as a discipline because of its consideration of both the physical and social elements of knowledge production and its tradition of critique (e.g., Lave 2014, Lane et al. 2018). My academic interest lies in the nexus between science and society – why and how do we create knowledge and what do people do with that knowledge once it has been created? My identity and journey led me to the broad scientific and societal challenge of adapting to climate change and specifically to questions around power, equity, and reflexivity. My opportunities gave me unique access to projects, scientists, and stakeholders to explore these questions. All three dimensions come together in this dissertation to inform my selection of research questions and methods, collection and analysis of data, and interpretation of results. The information reflected upon in this positionality statement was used to take greater care in how selection, collection, and analysis were conducted and is meant to provide transparency to the reader regarding any personal biases that may influence interpretation. CH. 2 - Engaging with stakeholders to produce actionable science Authors: Aparna Bamzai-Dodson1,2; abamzai@usgs.gov [corresponding author] Amanda E. Cravens3; aecravens@usgs.gov Alisa Wade1; awade@usgs.gov Renee A. McPherson2,4; renee@ou.edu 1U.S. Geological Survey North Central Climate Adaptation Science Center 2University of Oklahoma Department of Geography and Environmental Sustainability 3U.S. Geological Survey Fort Collins Science Center 4U.S. Geological Survey South Central Climate Adaptation Science Center Submitted to Weather, Climate, and Society 11 Mar 2021 Published online: 25 Oct 2021 Print publication: 01 Oct 2021 DOI: https://doi.org/10.1175/WCAS-D-21-0046.1 2.0 Abstract Natural and cultural resource managers are increasingly working with the scientific community to create information on how best to adapt to the current and projected impacts of climate change. Engaging with these managers is a strategy that researchers can use to ensure that scientific outputs and findings are actionable (or useful and usable). In this article, the authors adapt Davidson’s wheel of participation to characterize and describe common stakeholder engagement strategies across the spectrum of Inform, Consult, Participate, and Empower. This adapted framework provides researchers with a standardized vocabulary for describing their engagement approach, guidance on how to select an approach, methods for implementing engagement, and potential barriers to overcome. While there is often no one “best” approach to engaging with stakeholders, researchers can use the objectives of their project and the decision context in which their stakeholders operate to guide their selection. Researchers can also revisit this framework over time as their project objectives shift and their stakeholder relationships evolve. 2.1 Introduction Climate change is impacting natural and cultural resources throughout the US and globally (IPCC 2018). Resource managers and decision makers are more frequently in need of science-informed tools and guidance on how best to adapt to current and projected future conditions (Filho 2015). However, adaptation to climate change is a wicked problem (Rittel and Webber 1973) with many complex tradeoffs and potential barriers (Bierbaum et al. 2013). The creation of climate adaptation science cannot be separated from the application of such science, and engagement between researchers and science end-users is critical to the successful integration of science into actions and policy (Lubchenco 2017; McNie 2007; Sarewitz and Pielke Jr 2007). This engagement also needs to include the integration of local communities and knowledge because people and places have unique characteristics, and scientific products must respond to those unique qualities to be perceived as credible and to be accepted (Laursen et al. 2018). The term stakeholder is often used to mean an ambiguous and amorphous group of all interested parties (Sharfstein 2016). In this article, we define stakeholders as the end-users of the outputs and findings of a scientific process. Specifically, we focus on resource managers (e.g., state wildlife agency), decision makers (e.g., conservation land trust), and members of the public use project outputs to take action (e.g., private landowners). End-users may also include Indigenous peoples and communities, whose rights to sovereignty and self-governance are governed by a complex landscape of federal, state, and local treaties and laws (e.g., 43 U.S.C. § 1971). We recognize the colonial legacy of the term stakeholder (Barry and Thompson-Fawcett 2020) and following Sarkki et al. (2021), suggest the term rights-holder in these contexts instead. For brevity, we continue to use stakeholder throughout this article as an umbrella term for all end-user communities (rights- and non-rights- holders). Engaging these various stakeholders is one important means to ensuring that scientific outputs and findings are actionable – defined generally as useful and usable – with a growing body of literature on how to do this effectively (Arnott et al. 2020b). At its core, information and data must be seen as salient (relevant to the decision choices), credible (scientifically plausible and technically accurate), and legitimate (created through a fair and unbiased process) to be accepted as actionable (Cash et al. 2002). Although scientists may perceive their products as authoritative, stakeholders tend to approach new information and tools from a skeptical point of view, resulting in a need for salience, credibility, and legitimacy to be negotiated through extended dialogue and trust building (Cravens and Ardoin 2016; White et al. 2010). For example, the effective provision of climate information to stakeholders has been shown to require an understanding of the decision context and information usability (Dilling et al. 2015), an incorporation of managers’ perceptions of risk (Kirchhoff et al. 2013), and an assessment of research impact tied to understanding the audience and their need for evidence (Fisher et al. 2020). Engaging stakeholders in science creation can result in products tailored to appropriate spatial scales with focus on variables relevant to their decision contexts, a matter of key importance to resource managers. Increases in globally averaged temperature do not begin to scratch the surface of the climate change story, and managers instead seek information specific to the resources under their charge (Clifford et al. 2020). Additionally, the successful creation of usable science needs institutional buy-in and support for interactions on behalf of science users, producers, and analysts (Dilling and Lemos 2011). Although there is a long history of calls for actionable science in other fields of public policy (e.g., Bush, 1945; National Research Council, 1978, 2012), similar calls relevant to climate adaptation have increased only recently (e.g., Lemos et al., 2012; National Research Council, 2009). In a Federal Advisory Committee report on natural resource and climate adaptation science submitted to the U.S. Secretary of the Interior in 2015, a recommendation emphasized the need for continued provision of research that “provides data, analyses, projections, or tools that can support decisions regarding the management of the risks and impacts of climate change” (ACCCNRS 2015; Beier et al. 2017). Implementation of this recommendation has included a renewed commitment to the use of regional climate collaboratives working to support management action (Averyt et al. 2017), the development of specific informal partnerships to support the creation of actionable science (Bisbal 2019), and the use of formal stakeholder advisory committees and science advisory panels (DeCrappeo et al. 2017). Specific calls for more landscape-scale science for the management of federal and working lands also emphasize the need for engagement with managers. One oft-cited approach to achieve actionability is the use of co-production, which generally refers to researchers and stakeholders working on the same team to produce science useful for decision making (Carter et al. 2020; Naugle et al. 2020). Co-production is an imprecise term, however, used in the literature to describe a range of different approaches to working with stakeholders (Bremer and Meisch 2017; Mach et al. 2020; Norström et al. 2020; Wyborn et al. 2019). Lack of clarity around the language used by researchers and practitioners obscures the full range of approaches available for stakeholder engagement and limits our ability to inter-compare implementation of engagement methods and to identify lessons learned, effective practices, and pitfalls to avoid. Conceptual frameworks can be a useful way to organize complex concepts with a high degree of explanatory power, provide a mental model of how a diverse set of literature works together (Collins and Stockton 2018; Imenda 2014), and be used as a mechanism to distill theoretical concepts for practical application (Magliocca et al. 2018; Pulver et al. 2018). In this article, we adapt an existing conceptual framework to better characterize common stakeholder engagement approaches used to conduct actionable climate adaptation research. The framework presents a broader spectrum of engagement, clarifying the position of co-production and other ambiguous terminology. The foundational basis for the chosen framework relies on Davidson (1998) , selected because it is grounded in stakeholder engagement for community planning and prioritizes important concepts for climate adaptation research (e.g., role of stakeholder). While Davidson (1998) is a continuous spectrum of engagement with distinct endpoints, its non-hierarchical wheel structure promotes a deliberate, thoughtful choice of an approach to engage stakeholders instead of simply prioritizing the highest intensity of engagement (unlike linear hierarchical structures, e.g., a ladder in Arnstein 1969). Other similar frameworks exist but are not highlighted here because they prioritize concepts that are less relevant for scientific research or beyond the scope of this article; for example, citizen participation and empowerment (Arnstein 1969), the quality of resulting stakeholder decisions (Beierle 2002), or the direction and type of flows of information between researcher and stakeholder (Rowe and Frewer 2005). Additionally, while other frameworks describe engagement from an academic viewpoint, they often do not prioritize the practical application for researchers charged with conducting engagement or are focused on a narrow form of engagement (e.g., communications methods in Bojovic et al., 2021). Our adapted framework (Figure 2.1) helps researchers navigate and select the approach that best fits their objectives for a given project, thereby providing a standardized vocabulary for describing stakeholder engagement and filling a critical gap in the literature regarding practical application. The framework consists of four segments: Inform, Consult, Participate, and Empower. The Inform segment is characterized by minimal interaction with stakeholders, focused on one-way communication of quality information, such as press releases or newsletters, from researcher to stakeholder. The Consult segment includes collecting targeted input from stakeholders through avenues such as surveys or public meetings, while the Participate segment is distinguished by active partnership with stakeholders that allows them limited decision making capabilities (e.g., through a formal or informal advisory committee). The Empower segment seeks to delegate or entrust stakeholders with significant decision making power such that they become co-equal team members. The most important part of utilizing our adapted framework is understanding the context of a specific project and ensuring that investments are made in the most appropriate kind of engagement to meet the stakeholders’ science needs (Lemos et al. 2018). In this article, we outline our adapted framework, its four approaches, and examples of their appropriate use in section 2.2. In section 2.3, we provide guidance to researchers, in particular, on how they can select the most appropriate way to work with stakeholders based on their goals. We conclude in section 2.4 by urging scientists to work with end-users of information and products to ensure current knowledge is implemented, from national to local decisions. 2.2. Engagement for actionable science in the real world As climate adaptation researchers have aimed to engage stakeholders, time and financial stresses tend to pressure researchers into adopting previously used engagement strategies without deep thought about what strategy might be best suited to a particular outcome. Yet, building a habit of assessing which strategy to use on the front end can result in better outcomes. For each of the four approaches in this framework, we discuss associated science objectives, engagement methods, and barriers to implementation. Each engagement method has strengths, weaknesses, and resource constraints that influence what might be the best fit for a project. While many of these methods might span across approaches, we have tried to seat each method within the approach in which those activities commonly occur. We also include a few examples to illustrate successful uses of each approach. Table 2.1 provides a summary of this information. A researcher’s choice of approach might be dictated by resource constraints or other barriers (Rose et al. 2018) and require balancing the transaction costs of greater stakeholder interaction against expected gains in the quality and quantity of outputs and outcomes (Lemos et al. 2019). Although the level of technical complexity or degree of interdisciplinarity are important elements of the project context, they do not tell the whole story and may in fact prove to be misleading if they are the only elements considered. For example, a project focused on running a complex physical model may need extensive stakeholder engagement to determine the appropriate parametrization for key variables (e.g., Morisette et al., 2017). On the other hand, for a different type of complex study, such as a social-ecological systems study, limited stakeholder engagement might be possible when the end-user community and decision context can be clearly and narrowly defined (e.g., Beeton et al., 2019). Additionally, a researcher may need to revisit this framework over the lifetime of a project as existing stakeholder relationships are strengthened or new audiences are engaged (Klenk et al. 2017). 2.2.1 Inform As outlined in Figure 2.1, Inform is characterized by the one-way communication of decision-relevant project results. It differs from Consult in the direction of information exchange; Inform transfers information from researcher to stakeholder whereas Consult moves it from stakeholder to researcher. Inform is the most traditional and common approach of providing scientific information from researchers to users. Researchers in the basic science community often select Inform as their approach by default because there has been a tendency to consider any interactions with the users of the research as a potential source of bias. Some have criticized this approach as insufficient, terming it as the “loading dock approach;” Cash et al. (2006) describe it as “You take it out there, and you leave it on the loading dock and you say, there it is. And then you walk away and go back inside.” However, we argue that Inform can be a viable and proper approach for creating actionable science when chosen strategically. Appropriate uses of the Inform approach intend to transfer knowledge rapidly and result in some science-based action. These uses either create and disseminate datasets applicable across a broad geographic scale (Arguez et al. 2012), summarize a large body of complex, novel, or synthetic science (Guido et al. 2013), or require scientific legitimacy that stands above the political or policy context (Pielke Jr 2007). The International Network for Government Science Advice, an affiliated body of the International Science Council, studies issues of credibility and legitimacy of evidence used to inform policy conversations around topics that are highly contested, such as climate change mitigation, genetically modified crops, and biodiversity conservation, and has found that such research must be seen as unimpeachable and not unduly influenced by any one side of a policy debate (Gluckman and Wilsdon 2016). An effective use of Inform is exemplified by “Tornado Politics,” as described by Pielke Jr (2007), whereby an information provider does not have time to iterate on scientific products prior to one-way delivery to an audience (e.g., meteorologists issuing a tornado warning to the public) to meet a desired outcome (e.g., saving lives). In some cases, Inform may be selected as a training approach when there is a need for clear one-way information transfer to participants, such as teaching storm spotters how to recognize tornado development (Doswell et al. 1999). Another appropriate use of Inform is when the potential user base is so large and diverse that stakeholder interactions would be prohibitively complex, costly, and time-consuming. Examples of actionable scientific outputs for broad audiences include the data and graphics published for researchers, managers, and the public at large for the Coupled Model Intercomparison Project (CMIP; Eyring et al., 2016; Taylor et al., 2012) or the synthesis and interpretation of climate information in La Niña Drought Tracker during the drought of 2010-11 in Arizona and New Mexico (Guido et al. 2013). In these cases, the initial knowledge delivery by the scientific entity was followed with engagement with other scientists (e.g., broadcast meteorologists) using other approaches to increase saliency, yet the original transfer of science for public action was a one-way knowledge transfer. Examples of methods for the Inform approach include scientific publications, as well as webinars, seminars, presentations, white papers, briefings, brochures, and fact sheets. In the research community, Inform is dominated by peer-reviewed journal articles, which energize their scholarly audience (i.e., their stakeholders) to consider new theories, data, models, or methods in their own research (Fyfe et al. 2017). Science published in peer-reviewed journals or other jargon-rich or highly technical formats is not intended to be and, as a result, is often not necessarily useful for most decision makers (Hassol 2008); hence, many government agencies and independent organizations summarize specific subject matter using language appropriate for broader audiences. In all of these Inform examples, information that is intended to be actionable is translated without extensive stakeholder engagement; the information is intended or assumed to be useful without the need for two-way engagement. Barriers to Informing stakeholders without consultation, participation, or empowerment tend to preclude the effective use of Inform except in specific circumstances. For example, scientific articles that appear in professional journals, magazines, or blogs may be hidden behind paywalls, limiting their audience (e.g., Archie et al., 2014). Even with open-access publications or other more targeted information, such as fact sheets, stakeholders may not know where to find the information or have limited time to read it. For example, Doemeland & Trevino (2014) found that less than 15 percent of the World Bank's policy reports were downloaded at least 250 times, yet over twice that many were never downloaded and over 85 percent were never cited. In addition, without intervention in the process of delivering science to stakeholders, there generally is a mismatch between the climate information product and the user’s decision content that requires some negotiation and engagement (e.g., Briley et al., 2015). 2.2.2 Consult As opposed to Inform, Consult reverses the flow of one-way communication –– from stakeholder to researcher. Consult seeks to access stakeholder input, on pre-identified options, at discrete waypoints in the research process when it is not necessary to deeply understand the decision making context of the science nor experiences of those being consulted, such as required for Participate. Consult is appropriate for engagement when input is only needed on certain specifics of the broader research project, when addressing a topic already identified as a priority by stakeholders, or to ensure transparency and input when stakeholder numbers are large and individual stakeholder engagement is not tractable. Consulting with stakeholders often is conducted during one of three distinct phases of a research project: needs assessment, design and implementation, and products and outputs. Traditional climate services rely on these multiple, discrete stakeholder engagements to identify priority topics and delivery mechanisms for climate information provision in response to user needs (World Meteorological Organization 2014). Needs assessments seek to identify key science or support that would most benefit the end-user. There is evidence that organizations are more apt to use information that they requested (McNie 2013). This phase may be the most open-ended for consultation, whereby stakeholders may inform practice-relevant topics for research (e.g., Land et al., 2017). In this regard, Consult can blur into Participate, as a diverse stakeholder group can fundamentally shift the research agenda. Many examples of needs assessments focus on asking stakeholders to rank a pre-existing list of “knowledge gaps” assembled from the literature (e.g., Human & Davies, 2010). However, often these knowledge gaps themselves may have been identified through prior engagement efforts. For example, Crausbay et al. (2020) incorporated stakeholder input in a modified horizon-scanning exercise to identify gaps in the state of knowledge about ecological drought that could improve management response to events if filled. Related to a needs assessment, but more specific, is stakeholder engagement for problem framing. Phillipson et al. (2012) found that this type of engagement resulted in the greatest shift in research relevance to the stakeholders themselves. Consult can be applied during the research design and implementation phase to guide specific decisions on research approaches and methods or to provide specific information inputs. Data collection or provision is a form of Consult conducted early in research implementation. For example, crowdsourcing data (a subset of citizen science) can promote topical understanding and speed the dissemination and management uptake of findings, while also providing relatively inexpensive data (Lee et al. 2020; McKinley et al. 2017). Stakeholders also can provide information in the form of their own expertise. For example, in science syntheses, stakeholders may contribute to or review sections, such as in the National Climate Assessment (Cloyd et al. 2016). Expert elicitation processes effectively serve as opportunities to Consult stakeholders when gathering data, such as the Delphi Method (Naskar et al. 2018). Empirical social science also treats stakeholders as data sources, to better understand current preferences or inform how research findings can best inform management options (e.g., Klemm & McPherson, 2018; Wilkins et al., 2019). Another common form of Consult is relying on stakeholder expertise to parameterize models, such as the use of Bayesian Belief Networks (Richards et al. 2013) or other elicitation methods (e.g., see engagement for individual-based models in Samson et al., 2017). At the end of a research project, Consult can use stakeholder input to validate a project or estimate uncertainty in the findings (e.g., Johnson & Gillingham, 2004). Alternatively, engagement can identify which outputs are most useful and how to disseminate them. For example, decision support tool development can greatly benefit from Consult to test the usability of the tool (Oakley and Daudert 2016; Wong-Parodi et al. 2020). End-of-project Consults can help ensure that research products are useful, such as helping to identify clear visualizations or review accessible outputs such as fact sheets and online story maps. A recent project used eye-tracking data from website visitors to better understand how natural resource managers look for information and how layouts can make research findings most accessible (Maudlin et al. 2020). Consult includes a number of approaches for discrete interactions. Examples that seek breadth of input include surveys and opinion polls or voting on pre-established options. For more in-depth elicitation, Consult applies focus groups, town halls, interviews, or expert elicitation. Workshops or “colliders,” where researchers meet intensively with resource managers about a given modeling effort, can also be considered. Typically, citizen science projects (or other means of engaging stakeholders in data provision) define the frequency, thresholds, or occurrences for individuals to gather and send data to researchers. O’Haire et al. (2011) provide a useful summary of the strengths and limitations of a number of these engagement methods. A key barrier to the Consult process follows from the disconnect between a stakeholder’s input and the research results. Consult is founded on an assumed “promise” that stakeholder input will be used, and given the often one-off nature of Consult, stakeholders may feel exploited if there is no clear pathway between their time and input and the culmination of –– and self-benefit from –– the research project (Friesen et al. 2017). Researchers must resist what Arnstein (1969) calls “the empty ritual of participation,” where input is collected and claimed to be considered but the status quo persists, particularly in cases where power imbalances exist across groups of stakeholders. This problem may be compounded as stakeholder engagement requests increase, with a concomitant decline in survey responses in natural resources social sciences (Stedman et al. 2019) and growing reports of “stakeholder fatigue” (Bracken et al. 2015). To resolve this issue, it is imperative that researchers follow-up with stakeholders, reporting back in detail how their input was used. Another common barrier is a mismatch in the timing of the stakeholder engagement and the research input need. Consult elicits input that represents a single moment in time and that may not match either the researcher’s process nor the immediacy of the stakeholder’s need for the information and/or outputs produced by the project. Further, in the case of needs assessments, it may be difficult to elicit useful information from busy stakeholders, who often are too burdened with projects to think long-term and proactively about their needs for research outputs that would not be available for multiple years. Seeking to elicit upcoming key management decisions or problems rather than explicitly asking “what science would benefit your work?” may assist researchers in overcoming this obstacle (Beier et al. 2017). A shortcoming to Consult is that stakeholder input does not necessarily result in clear direction for the researcher. Seeking input from a large array of diverse stakeholders is recommended, but these differences can result in divergent input. Certain approaches, such as the Delphi Method, seek to bring a smaller group to consensus, but other forms of engagement, such as surveys, offer no such promise. Further, the input obtained can vary greatly depending on who or what is asked. For example, Archie et al. (2014) found that the level of knowledge about climate change greatly affected survey respondents’ reporting of climate adaptation efforts. In general, there is substantial evidence that how an issue is framed (e.g., presented in a positive or negative light) significantly affects how stakeholders respond (Scheufele and Iyengar 2017), although there is also research suggesting ways forward to better elicit input in response to complex issues (e.g., Jansen et al., 2019). 2.2.3 Participate Participate encompasses a range of engagement methods whereby scientists and stakeholders share decision making. Unlike Inform and Consult, these approaches represent sustained two-way interaction (though the length of time and type of interaction varies). Stakeholders do not play the leadership role of co-equal project partner or team member, however, in the manner of Empower. Participate supports science projects that seek to provide place-based, contextualized, or customized information. Participate generates information needed to support planning, policymaking, or other kinds of agency decision making where there is a need to understand stakeholders’ perceptions or experiences in greater depth than might be captured by a Consult approach. Another common objective for Participate is when developing a product such as a computerized decision support tool or a guidance document that needs sustained input from stakeholders. Participate encompasses approaches that engage deeply enough with diverse stakeholders to gain significant understanding of their point of view, while not fully integrating those stakeholders into the decision making process. One way to enact Participate approaches is through iterative engagement with (either the same or different groups of) stakeholders at defined points in a project’s life cycle. Particularly common in the development of tangible products such as decision support tools, this approach plans multiple touch points during a project to engage with stakeholders to ensure that concerns and aspirations are mutually understood. For instance, users might provide input in the scoping phase, then provide input to a series of prototype versions as the tool is developed, have an opportunity to test the final design to improve the usability of a tool, and finally engage during training sessions to learn how to use the final product (Leitch et al. 2019). Despite the substantial sustained engagement over what might be a period of months or years, the stakeholders are not fully integrated into the project team and do not participate in the work that happens between these discrete touch points. For instance, they are not included in project management or planning, the evaluation of alternatives, identifying preferred options, or the technical aspects of tool building. This iterative process of engaging with users multiple times over the course of the project described in Leitch et al. (2019) generally follows a framework variously called human- or user-centered design (Boy 2017; Gasson 2003; Rouse 2007; Wright and McCarthy 2010) or design thinking (Plattner et al. 2011). Such a design framework represents a structured methodology for developing products or experiences and focuses on developing a deep understanding of the intended user’s needs and point of view from the start of a project. Christel et al. (2018) described a climate services project that used a design-based approach to create a product to visualize seasonal wind speed predictions for decision makers in wind energy. The team found that a human-centered design framework helped sustain cross-disciplinary collaboration between the intended users in the wind energy industry, the research team, and the visualization experts. Others have similarly argued that user-centered design improved the development of drought indicators by providing a systematic method to understand and incorporate the needs of specific types of drought-information users (Purdy et al. 2019). Other Participate approaches emphasize the value of incorporating diverse partnerships and points of view into a project. Translational ecology is an emerging strategy arising from a recognition among ecologists of the need for actionable ecological knowledge (Schlesinger 2010). It is defined as “intentional processes in which ecologists, stakeholders, and decision makers work collaboratively to develop ecological research via joint consideration of the sociological, ecological, and political contexts of an environmental problem that ideally results in improved environment-related decision making” (Enquist et al. 2017). Beaury et al. (2020) used translational ecology as their strategy for engaging with invasive species managers to understand their needs and barriers and for developing an agenda for continued interactions between managers and researchers. This iterative, two-way engagement strategy will inform future research directions and ensure that scientific findings and outputs are designed to be directly incorporated into on-the-ground invasive species management. Boundary spanning is another broad participatory strategy that recognizes that many partners might be needed to address issues across the science-policy interface and that provides guidance for the deliberate engagement of individuals and organizations (Bednarek et al. 2018; Goodrich et al. 2020). One “boundary organization” can work as a translator with both science producers and users, or two or more boundary organizations can join together in a “boundary chain” to collaborate, share costs, and pool resources (Kalafatis et al. 2015; Kirchhoff et al. 2015); different types of goals or projects will be easier for organizations at certain points in the chain to accomplish. We have placed boundary spanning within Participate as the place where this activity is most likely to occur but recognize that boundary individuals and organizations often utilize approaches and methods across the entire spectrum of the wheel. An example of a boundary organization is the National Integrated Drought Information System (NIDIS). In response to a call from the Western Governors’ Association for integrated physical, hydrological, environmental, and socio-economic decision-ready drought data and tools (WGA 2004), the National Oceanic and Atmospheric Administration (NOAA) created NIDIS. Recognizing the existing expertise and breadth of ongoing drought science, NIDIS brings together federal, state, local, and Tribal agencies and taps into existing data and information networks to identify management needs, synthesize science, and create decision support tools. NIDIS also leverages partnerships with other boundary organizations, such as the National Drought Mitigation Center and the NOAA Regional Integrated Sciences and Assessments Teams, to facilitate and synthesize dialogue from national and regional working groups at a variety of scales (NIDIS 2016). Spanning a broad pool of science producers, users, and other boundary individuals and organizations allows NIDIS to efficiently and dynamically call on expertise, capabilities, and resources to address questions and challenges as they arise. Engagement methods for Participate overlap with Consult and Empower but are distinguished from the first by the depth of the engagement and from the second by the way that the research team retains control of decision making throughout the engagement process. Methods include surveys and semi-structured interviews, though with more interaction than Consult. For instance, the same survey might be administered at multiple points in time, or a research team might do a set of interviews and then have a follow-up workshop to share results and get feedback about how the team is interpreting the findings. Other engagement methods include repeated focus groups or listening sessions, whose potential for social learning can support the need for diverse perspectives of Participate methods (Gerlak et al. 2020), or advisory boards, which provide a formal means for stakeholders to continually provide input to a project. For example, during the State of California’s process of siting new marine protected areas under the auspices of the Marine Life Protected Act Initiative, the Science Advisory Team provided ongoing guidance about how to evaluate the scientific merit of proposals (Saarman et al. 2013). Projects using human-centered design approaches also might draw inspiration for engagement methods from the field of design; for instance, designers make extensive use of participant observation (IDEO.org 2015). Simulations, scenarios, and role-playing games can educate stakeholders to understand how to use scientific data and information, build collaborative decision making capacity around wicked problems, and create local or regional communities of practice for sustained social learning (Rosendahl et al. 2019; Rumore et al. 2016). For instance, drought simulations have aided local, state, and national drought preparedness and planning efforts (Bathke et al. 2019). The U.S. Geological Survey’s Science Application for Risk Reduction (SAFRR) Project runs large-scale scenarios (i.e., 200+ person) to support communities in preparing for natural disasters (Porter et al. 2011; Ross et al. 2013). Demonstrating the complex consequences of hazards, such as earthquakes, SAFRR scenarios allow stakeholders to participate in hazard response and recovery exercises designed to connect scientific information and community needs around natural disasters. There are three common barriers to using Participate approaches. One, this approach requires the time, resources, and expertise for researchers to run such an engagement effort because it relies on deep stakeholder engagement sustained over the course of a project (Kemp et al. 2015). Two, it requires that the chosen stakeholders similarly have the time, capacity, and motivation to (or expected benefit from) participation (Bracken et al. 2015). Three, at the beginning of a project, it may not be possible to precisely define when, how, or with whom engagement will happen; hence, sustained engagement, especially across multiple points in time, requires flexibility and adaptability (LaChapelle et al. 2003). 2.2.4 Empower As in Participate, approaches within Empower require sustained, two-way interaction between stakeholders and researchers. In Empower approaches, however, stakeholders are included as co-equal team members through every step of the project, including but not limited to defining the problem statement, designing research questions, selecting methods, collecting and interpreting data, and developing output products. Stakeholders are entrusted to make project decisions; not just to participate in or be consulted on project decisions. To build and sustain the partnerships needed for Empower approaches, researchers and stakeholders must dedicate significant resources (e.g., time, money) within the project towards engagement. When viewed from outside of the project team, Participate approaches with highly equitable stakeholder inclusion may appear to be indistinguishable from Empower approaches. Empower approaches can be particularly effective when projects take place within contexts where the voices of particular stakeholders or rights-holders have been muted or disenfranchised in decision making processes (Ardener 2005; Orbe 1998). Empowerment of stakeholders can result in fundamental shifts in power and governance by placing equal importance on information from diverse knowledge systems (Wyborn et al. 2019) and by allowing for holistic representations of human-nature connectedness informed by the ethics of care (West et al. 2020). As a result, Empower can support the creation of place-based science where local knowledge is critical and the implementation of inclusive science-informed plans, decisions, or actions. Co-production is the term often used to describe the process of engaging with and empowering stakeholders in the process of creating actionable science (Meadow et al. 2015). Hegger et al. (2012, 2014) instead suggest calling this approach joint knowledge production, as the term co-production is used by Science and Technology Studies scholars to refer to mutually shaping interaction between the production of knowledge and the production of social order (Jasanoff 2004). The foundational principles of joint knowledge production include that the scientific objectives must be grounded in the decision context, that iterative discussions between all members throughout the project are essential, that management risk and scientific uncertainty must both be clearly defined and communicated, and that the process of engagement is just as important as the outputs and must be evaluated (Beier et al. 2017). Scientists and non-scientists work together to include and give equal weight to local knowledge and qualitative information in the sum of the evidence considered by the project team (e.g., Diver, 2017). Joint knowledge production that ignores power differentials can reinforce power imbalances and impede wider societal progress (Turnhout et al. 2020), and the removal of local knowledge from the context of its production can result in outputs and outcomes that are ineffective for local governance and practices (Klenk et al. 2017). We have situated joint knowledge production within Empower but recognize that, in practice, researchers often utilize joint knowledge production principles in forms that span other segments of the wheel (e.g., contractual, consultative, and collaborative in Meadow et al., 2015). Approaches similar to joint knowledge production include consensus building, action research, and participatory design. Consensus building entails iterative, usually facilitated, conversations with stakeholders to determine outcomes that are at least acceptable to all involved. Consensus building not only considers the range of stakeholder views on an issue but explicitly recognizes power differentials between stakeholders and focuses on distributing power in a just and equitable manner (Prior 2013). Action research acknowledges that blending theory and practice to create solutions to real-world problems requires a deep understanding of the values of the individuals and communities involved. Action research also recognizes that researchers and stakeholders are continually impacted by their interactions and can learn and evolve over the course of the project (Bradbury et al. 2019; Brydon-Miller et al. 2003). Participatory design emerged from the production of informational technologies as a process by which users and designers can experience mutual learning and co-design products (Simonsen and Robertson 2013). Users and designers participate in frequent, iterative conversations such that re-design elements can be continually and critically examined for usefulness and usability (Kruk et al. 2018). Important to all these approaches is active stakeholder participation characterized by inclusion and empowerment (Few et al. 2007). Specific tools support empowering stakeholder voices by shifting the balance of power and purposefully including end-user knowledge and perspectives in the production of scientific research. Counter-mapping or participatory mapping allows local stakeholders to develop visual representations of the spatial relationships between physical locations and boundaries and can be effective for establishing Indigenous rights and sovereignty (Rundstrom 2009). Discourse-based valuation diverges from conventional valuation of public goods by examining convergent values across a group of stakeholders through a well-ordered deliberative forum as opposed to aggregating individual preferences. This method provides a non-market based valuation for public goods that can enhance social equity (Wilson and Howarth 2002). Scenario planning provides a structured format through which researchers and managers can explore quantitative and qualitative “what if” scenarios for the future together. Participants are able to consider multiple, layered dimensions of uncertainty and risk and are encouraged to consider innovative or outside-the-box resource management next steps (Symstad et al. 2017). In addition to this sample of high engagement methods and techniques, Empower projects can utilize those from the Inform, Consult, and Participate approaches at various points in the engagement process, in combination with each other, and at increased frequency. Projects have flexibility in selecting which of these tools to use in which circumstances and thus need to carefully consider their individual strengths and weaknesses. While there is often no single right engagement method to use, selecting the wrong method can widen the gulf between researchers and stakeholders (Lynam et al. 2007). Barriers for Empower approaches include many of those described for Inform, Consult, and Participate. Additionally, Empower can be particularly hampered by the unique cultural and institutional barriers faced by both stakeholders and researchers (Jarvis et al. 2020). For example, Archie (2014) found that elected officials were less likely than career officials to consider climate change a priority for planning in Rocky Mountain communities. Without effective leadership, shifts in political will or agency priorities can redirect the allocation of resources and leave stakeholders with limited scales of influence (Moser and Ekstrom 2010). Formal implementation of environmental legislation also can impede adaptation planning if it is perceived as prescriptive (e.g., Endangered Species Act, can prevent action) and not process-oriented (e.g., National Environmental Policy Act, informs but does not prevent action) (Jantarasami et al. 2010). Researchers also face a number of barriers to engaging in Empower approaches. Unpublished research by the second author of this article (Cravens) noted that funding models, data access, job descriptions that omitted stakeholder engagement, and similar constraints prevented producers of drought information from engaging with stakeholders in ways they wished, even though the majority recognized the importance of stakeholder engagement to effective tool development. Academic faculty, especially those who are pre-tenure, may not be evaluated favorably unless they produce peer-reviewed publications. In such cases, researchers also must consider the costs of investing in relationship building and stakeholder engagement with no guarantee of scientific outputs or even management impact (Oliver et al. 2019). In these situations, the process of Empower itself can sometimes be considered as an important outcome. Interactions that are part of a project whose goals are not fully realized can be essential to strengthening relationships and provide a foundation for future success (Jagannathan et al. 2020b). 2.3. Guidance for actionable science engagement approach selection A project team may elect to Inform stakeholders when providing rapid response information in a crisis situation, working with a homogenous or well-studied community, or responding to needs that already have been clearly articulated. Consulting stakeholders may be appropriate when the stakeholder community is large and diverse, whereas using Participatory approaches may work best when the stakeholders and their needs are able to be well defined. Empowering the stakeholders may be an option when a project is addressing a complex, non-time sensitive issue or in decision contexts where a shift in the balance of power between groups of stakeholders or between end-users and researchers needs to occur. The wheel of participation represents a continuous spectrum, such that individual engagement approaches may span across segment boundaries depending on implementation. It is important to note that there is no individual “best” approach to engagement and that the use of multiple approaches may be needed to engage with different stakeholders on the same project. We suggest some guiding thought questions in Table 2.2 that researchers can use when selecting an approach. We provide sample end point answers to the Inform, Consult, Participate, Empower spectrum that researchers can use to gauge where their answer to each thought question falls. Multiple answers clustered under a single approach might provide an indicator of what kind of engagement a researcher can start exploring for their project. Conversely, answers that span approaches might indicate the need to use a range of engagement approaches or methods to achieve project objectives. We strongly encourage researchers to keep in mind cultural elements when selecting an approach and to default to the use of Empower approaches when working with muted groups. For example, when engaging with Indigenous communities, it is important to start with a justice-forward mindset, moving past only addressing historical wrongdoing to actively leading the removal of institutional and bureaucratic obstructions in order for partner communities to flourish (Whyte 2014). While there is a demonstrated need for training to inform how non-Indigenous researchers ethically engage with Indigenous communities without causing additional harm (Kirby et al. 2019), all researchers can keep in mind that there are tipping points around the areas of consent, trust, accountability, and responsibility which may result in permanent and irreversible relational damage if crossed (Whyte 2020). Demands for stakeholder input and participation may also be particularly burdensome to these communities when projects do not provide them with adequate support for the capacity and resources necessary to respond. Although the research community has taken some steps to move from an extractive model of Indigenous knowledge (Matsui 2015) to an integrative model of bringing together Indigenous and Western ways of knowing (e.g., Two-eyed seeing in Bartlett et al., 2012), we must now encourage research leadership that is foundationally grounded in Indigenous scholarship and that empowers Indigenous ways of knowing, sovereignty, and self-determination (Kimmerer 2014; Latulippe and Klenk 2020). Additionally, a lack of good communication between scientists and decision makers is a major barrier to evidence-informed conservation policy (Rose et al. 2018), and we thus encourage researchers to explicitly address communications when selecting an engagement approach. Social processes that aid in knowledge exchange within a group include cultivating an atmosphere of psychological safety and trust from the beginning, using laughter and humor throughout the conversation to regulate the collective mood, and regularly reflecting group dynamics back to the group to promote communal ownership of process and outcomes (Morisette et al. 2017). Fundamental to this exchange are good communications practices for improving information relevance and uptake: encouraging critical thinking, valuing multiple types and lines of evidence over opinions, avoiding excessive jargon, and eliminating pointless meetings (McCarthy et al. 2020). To sustain strong relationships over the long-term, good communication must be augmented by supportive organizational behavior centered on trust, commitment, and openness (Ledingham 2003; Ledingham and Bruning 1998). While the focus of this article is to provide the research community with a framework and guidance for engaging with stakeholders, such information may also be valuable to practitioners working at the nexus of knowledge creation, synthesis, translation, and application. Individuals and organizations in boundary spanning roles can be integral initiators and facilitators of the dialogue between researchers and stakeholders, often investing in long-term relationship building across diverse communities. Such boundary spanners can use this framework to help researchers and stakeholders negotiate an appropriate role within a project that best suits their application needs and available resources. Program administrators and funders of actionable science projects also have a role to play; by asking researchers to articulate how stakeholder engagement strategies support a given project’s scientific objectives, administrators and funders can encourage thoughtful approach selection. Finally, resource managers, or any stakeholders, can benefit from this framework by engaging in projects that have a well-considered articulation of how their role and the selected engagement approach is likely to result in outputs and outcomes that meet their decision or planning needs. 2.4. Conclusions In this article, we have adapted Davidson’s wheel of participation to create a standardized vocabulary for actionable science researchers and the wider scientific community to use when determining their stakeholder engagement strategy. Our adapted framework puts different approaches to stakeholder engagement on a continuous spectrum (Inform, Consult, Participate, Empower) and demonstrates that there are multiple paths to the same end goal of actionable science. We emphasize the need to find the right fit for each specific project and stakeholder community, and we provide high-level direction on how selection of an approach might be influenced by both the characteristics of the research context (e.g., science objectives; Table 2.1) and the decision context (e.g., power and stakeholder dynamics; Table 2.2). Our suggested guiding thought questions give researchers factors that they might want to consider in designing their stakeholder engagement strategy. Researchers can also use our adapted framework when describing their projects to funding agencies, senior leadership, or evaluators. Public science funders are increasingly requiring stakeholder engagement as part of their funding criteria to increase the relevance of research outputs to society (Arnott et al. 2020a). These requirements likely will expand over time, as evidence demonstrates that well-designed stakeholder engagement can lead to scientific uptake in the policy process (e.g., Nguyen et al., 2019). By providing researchers with the language and context for selecting a particular approach, this framework can help provide justification for why a project might be required to invest substantial time and resources into engagement and why institutions need to consider this work as an integral part of the scientific process (i.e., a research and creative activity rather than as professional service or outreach). A constraint of the framework, however, is that it does not provide definitions for what successful engagement and actionability look like and how those characteristics might be measured and evaluated. Wall et al. (2017) can serve as a starting point for researchers interested in defining and evaluating success, but deeper consideration was outside the scope of this article. We have presented our framework for stakeholder engagement using the example of climate change, which shares many qualities in common with other wicked problems in environmental and sustainability science. These challenges are ill-defined and multi-layered, have no clear single optimal solution, require value judgments to resolve, and may continue on in the form of a new wicked problem once action has been taken (Rittel and Webber 1973). Science to address wicked problems in general requires an actionable approach and at least some degree of engagement with users of the science. Thus, our framework could potentially be generalized for use by researchers across these fields and is not necessarily constrained solely to the topic of actionable climate adaptation science. 2.5 Acknowledgments This work was funded by the U.S. Geological Survey South Central and North Central Climate Adaptation Science Centers. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. government. We thank author Bamzai-Dodson’s dissertation committee members and the members of the Climate Adaptation Science Center Evaluation Working Group for insightful discussions around what constitutes actionable science, what success might look like, and how it could be measured. We thank our peer reviewers for their thoughtful feedback, which honed and improved the concepts in this article. Special thanks are given to Kristen Donahue for her design work on Figure 2.1. We hope that our research supports future continued conservation and protection efforts. Figure 2.1 - Adapted wheel of participation Outer text organizes strategies for the creation of actionable science described in this article by their corresponding approach from the wheel of participation; inner text is from Davidson (1998) and demonstrates the continuous spectrum of community engagement across each approach. Inform is characterized by minimal interaction with stakeholders, focused on one-way communication of quality information. Consult includes collecting targeted input from stakeholders, whereas participate is distinguished by active partnership with stakeholders that allows them limited decision making capabilities. The empower segment seeks to delegate or entrust stakeholders with significant decision making power such that they become coequal team members. Table 2.1 - Summary description of each approach from the adapted wheel Summarized by intended project objective(s), potential engagement methods, common barriers, and case examples. Inform Consult Participate Empower Project objective(s) Create and disseminate datasets applicable across a broad geographic scale; Summarize a large body of science; Require scientific legitimacy that stands above the political or policy context Need stakeholder input on certain specifics of the broader research project, e.g., needs assessment, design and implementation, or products and outputs Provide place-based, contextualized, or customized information to support planning, policymaking, or other kinds of agency decision making Consider stakeholder input and decisions to be co-equal to the rest of the research team;  Fundamental shift in power and governance Engagement methods Traditional scientific publications, webinars, seminars, presentations, white papers, briefings, brochures, and fact sheets Surveys, focus groups, town halls, and other modes of expert elicitation Sustained and iterative two-way interaction with individuals via repeated Consult methods, advisory boards, or simulations Tools to shift the balance of power such as counter-mapping, discourse-based valuation, or scenario planning Common barriers Limited access to information behind paywalls; Communicated in a way that’s not obviously relevant to the end-user Implicit expectations that stakeholder input will be utilized; Mismatch in timing between research progress and decision context; Lack of clarity in stakeholder response  Significant resources must be dedicated to engagement by both the researchers and the stakeholders; Requires adaptability and flexibility to changing needs Cultural and institutional barriers such as shifts in agency priorities, implementation of legislation, and the lack of recognition for engagement work Example Stakeholders learn about project results via a public webinar Stakeholders identify desired model output and late how to summarize it usefully via two surveys Stakeholders help refine research questions and provide input at regular points during research process Stakeholders are an integral part of science team and help define research questions, approach, and products Table 2.2 - Guiding thought questions Questions inform the selection of an engagement approach across the inform, consult, participate, and empower spectrum. Each question provides example end points, which researchers can use to situate themselves along the spectrum. A clear answer may emerge if multiple responses cluster under a single approach. Question Inform Consult Participate Empower 1. To what degree do the products need scientific legitimacy above the political or policy context? Legitimacy comes from complete scientific independence ←  →  Legitimacy comes from engagement process 2. Who makes decisions about managing this resource? What do the power and governance structures look like? Decision roles are well defined and balanced, no stakeholders are excluded ←  →  Key stakeholders are somehow marginalized, excluded, muted, or disenfranchised  3. Considering the size and diversity of your stakeholder community, from what kind of insights would your project most benefit? Broad and diverse insights, covering many stakeholders ←  →  In-depth, focused insights, covering key stakeholders 4. How well understood are the stakeholder needs and the decision context? What information already exists from previous formal or informal needs or vulnerability assessments? Stakeholder needs are well understood, large volume of pre-existing information ←  →  Stakeholder needs are poorly understood, no pre-existing information 5. What resources (time, money, staff, etc.) are available for engaging with stakeholders? Resources for engagement are low or non-existent ←  →  Resources for engagement are plentiful 6. What kind of leadership or institutional support exists for this kind of work? Engagement work is penalized ←  →  Engagement work is recognized and rewarded CH. 3 - The road to actionable science is paved with good intentions Authors: Aparna Bamzai-Dodson1,2; abamzai@usgs.gov [corresponding author] Amanda E. Cravens3; aecravens@usgs.gov Renee A. McPherson2,4; renee@ou.edu 1U.S. Geological Survey North Central Climate Adaptation Science Center 2University of Oklahoma Department of Geography and Environmental Sustainability 3U.S. Geological Survey Fort Collins Science Center 4U.S. Geological Survey South Central Climate Adaptation Science Center Submitted to Annals of the American Association of Geographers 11 Feb 2022 Currently in review 3.0 Abstract To help stakeholders such as planners, resource managers, and policy and decision makers address environmental challenges in the Anthropocene, scientists are increasingly creating actionable science -- science that is useful, usable, and used. Engagement of stakeholders in the creation of scientific knowledge is one recommended approach to ground scientific objectives within the context of stakeholder needs and to produce outputs that are directly relevant to stakeholder plans, decisions, or actions. However, previous studies have demonstrated that there can be a gap between the goals and aspirations of projects intending to create actionable science and the resulting outputs. In this paper, we utilize the core principles for ethical research of respect for persons, beneficence, and justice from the Belmont Report (1979) to examine the perspectives of stakeholders engaged in climate adaptation science projects. We argue that poorly conceived stakeholder engagement can result in individual or community harm to stakeholders, and we challenge scientists to consider the broader ethical implications of engaging with these partners. 3.1 Introduction 3.1.1 Background As the Anthropocene unfolds, resource managers and scientists have identified a number of challenges where humans are the primary force behind environmental changes that are taking place in complex and interacting ways (Butler 2021; Crutzen 2006). Producing knowledge to address these challenges is common to geography as a discipline, with its focus on human-nature interactions, place-based solutions, and cross-scale connectedness (Larsen and Harrington 2021; Simm et al. 2021). However, such knowledge is also being produced in large quantities in disciplines outside of geography (e.g., wicked problems (Rittel and Webber 1973); action research (Brydon-Miller et al. 2003)), creating an opportunity to share ideas and lessons learned across disciplinary boundaries. In this paper, we intentionally choose to use some concepts and terminology that are found outside of geography to broaden the accessibility of our findings and to foster cross-disciplinary dialogue. To help decision makers identify possible solutions to these environmental challenges, including adapting to climate change, scientists[footnoteRef:1] have begun to focus on the production of actionable science –– science that is useful, usable, and used (Beier et al. 2017; Dilling and Lemos 2011; Winkler 2016). End users of science, such as these decision makers, are stakeholders[footnoteRef:2] in the research process, with their applications of scientific findings, data, and products ranging from being better informed to incorporating content into a plan, decision, or action (VanderMolen et al. 2020). Participation of stakeholders during the scientific process, also called stakeholder engagement, provides them with the opportunity to have input on decisions that may affect their communities or ecosystems of concern and can bring to light new perspectives on challenges and solutions that may not occur to scientists (Fiorino 1990). Additionally, stakeholder engagement can stimulate social learning by bringing individuals together around a shared problem and has the potential to improve the context-relevance of findings and outputs (Schmidt et al. 2020). However, there are many approaches to stakeholder engagement available to scientists (Bamzai-Dodson et al. 2021), raising tricky questions around who is allowed to participate, when and how that participation will occur, and how dissent will be managed (Sprain 2017). [1: We use scientists to refer broadly to those conducting research with a goal to generate new knowledge, recognizing that some readers may be uncomfortable with this term. We acknowledge that people from sectors and disciplines beyond science (e.g., practitioners) frequently act in this space. However, for reading clarity, we use the term "scientist" rather than a list throughout.] [2: We recognize that Tribal and Indigenous peoples and communities may be end users of science who hold unique rights to sovereignty and self-governance. These end users are more appropriately designated as rights-holders, not stakeholders, in the research process (Sarkki et al. 2021). In this paper, we nevertheless use stakeholder as a concise, overarching term for both rights and non-rights holders who have an interest in or concern about the topic of study.] Co-production is one approach to stakeholder engagement that emphasizes an iterative, two-way knowledge exchange between scientists and stakeholders and foregrounds stakeholder challenges to motivate the project design (Beier et al. 2017; Meadow et al. 2015). In these applied contexts, co-production is sometimes termed joint knowledge production (Hegger et al. 2012) to distinguish from its use in Science and Technology Studies to describe the mutual shaping of science and society (Jasanoff 2004). Co-production can occur at multiple stages throughout a single project (e.g., development of the research questions, selection of methods and techniques, collection of data, tailoring of outputs to support end-use) and also can accomplish long-term goals (e.g., co-evolution of institutional cultures, improved community wellbeing) across multiple project engagements (Wyborn et al. 2019). Co-production has been highlighted as a way to more equitably incorporate the diverse knowledge forms and professional networks of local experts into research and dissemination processes (Laursen et al. 2018). For many, these benefits have resulted in co-production being elevated as the “gold standard” strategy for stakeholder engagement. Although many scientists intend to use stakeholder engagement to create actionable science and systemic change, there can be a gap between the intended project goals and the achievement (and documentation) of outcomes. Previous studies have shown that scientists struggle to move from producing incremental improvements in data quality to creating transformational products that aid better decision making (Findlater et al. 2021) and that projects that engage stakeholders seldom result in significant changes in cultural norms or power dynamics (Jagannathan et al. 2020a). A broad systematic review of 478 studies at the science-policy interface found that while most aimed for ambitious knowledge exchange goals, only some (250 studies) conducted a data-driven evaluation of usability and even fewer (93 studies) conducted a data-driven evaluation of use (Karcher et al. 2021). Similarly, Pearman & Cravens (2022) found that creators of drought decision support tools emphasized the importance of stakeholder engagement principles to the tool creation process and to claimed successful adoption of those tools in decision making. Nevertheless, those same tool creators primarily conducted engagement in an informal manner and rarely conducted a formal evaluation of the usability or use of a tool (ibid). Despite existing evidence-based recommendations on the project roles that stakeholders can fill (Carney et al. 2009), strategies for including them (Daniels et al. 2020; Steger et al. 2021), and approaches for evaluating the resulting process and products (Owen 2021; Wall et al. 2017), many scientists still struggle to fulfill their claims of actionable science. At the same time, building off Arnstein’s (1969, p. 216) ethical caution against the “empty ritual of participation”, scientists are recognizing the potential harms to not carefully considering why stakeholder engagement is the right fit for a project, how information will be collected from and returned to stakeholders, and how stakeholder input will be used to inform project design and outcomes (Dilling et al. 2021; Lemos et al. 2018). When scientists do not reflect on these questions, the unintended consequences can include mismanaged expectations, often leading to exploitation of resources (e.g., time) and fatigue (Clark 2008) and sometimes even to outright hostility (Brittain et al. 2020). Poor engagement can be particularly harmful when working with under-resourced or under-served communities. An analysis of 125 climate studies in Indigenous communities found that 87 percent of them were conducted in an extractive way, utilizing Indigenous knowledge without adequate inclusion or recognition of Indigenous knowledge holders (David-Chavez and Gavin 2018). As Lemos et al. (2018, p. 723) contend, the gap between the goals and outcomes of stakeholder engagement must be closed to prevent it from "becoming an end in itself, glossing over the very values and goals that often inspire scientists and stakeholders to engage with one another." In other words, more engagement is not always the answer to improving outcomes, especially without consideration of both positive and negative consequences to stakeholders. In this paper, we argue that scientists must proactively consider the ethics of engaging with stakeholder partners, including when such engagement may not be considered human subjects research. We draw on literature from geography to summarize existing lessons that can be applied to actionable science. We also conduct novel analysis on the perspectives of stakeholders engaged in co-produced climate adaptation research projects to better understand how they perceive the impact of such engagement and the benefits and harms that they experience. 3.1.2 Dialogues from geography that inform stakeholder engagement Here, we highlight ideas from geography that underscore the importance and relevance of such literature to the production of actionable science and to the engagement of stakeholders in the research process. Underpinning this literature are the individual perspectives that scientists and stakeholders bring to a project, which contribute to what someone chooses to study, how they form research questions, what methods and data they consider relevant and appropriate, and the context(s) in which they elect to interpret their findings (Khagram et al. 2010). A key area of dialogue across human and physical geography surrounds reflexivity, definitions of which range from direct self-reflection by scientists about how their individual perspective influences their decisions to critical thinking by a research team about power and partnership dynamics (Boström et al. 2017; Montana et al. 2020). Reflexivity is a process by which one can examine their role as a scientist, their positionality, and how their choices influence their science (Hakkarainen et al. 2021; Holmes 2020). One goal of reflexivity can be to examine ethical considerations across multiple dimensions, including conflicts between scientist and stakeholder value systems and the power dynamics among competing interests (Brittain et al. 2020). This deliberative process can be especially important for establishing reciprocity with traditionally under-resourced or under-served stakeholder communities and ensuring equitable distribution of research benefits. For example, reflexivity might help scientists identify strategies to de-emphasize their role as an authority figure in conversations, setting an inclusive atmosphere and leading to more equitable consideration of ideas (Diver and Higgins 2014). Reflexivity can also help scientists select culturally appropriate language and mediums for communicating findings, ensuring that information is shared in the form that best fits community needs (ibid). An example of reflexivity in geography is embodied in critical physical geography (CPG), a call to integrate the insights and strengths from critical human geography and physical geography to better understand the impact and consequences of research on society (Lave 2014; Lave et al. 2018). CPG recognizes that choices made by scientists during knowledge production influence what is studied and how data and results are interpreted, which can have a resultant impact on decisions and policies informed by that evidence (Lane et al. 2018). As a result, CPG encourages scientists to question established norms in methods and analysis and devise new approaches to evaluate and make such choices (Tadaki 2017). CPG can teach scientists producing actionable science how reflexive consideration of their own invisible agenda is an opportunity to take responsibility for the legacy of the knowledge they produce (Tadaki et al. 2015). A CPG approach to actionable science for climate adaptation would interrogate whether such knowledge production is being led by the people and conducted in the places where climate impacts will be most keenly experienced (Colven and Thomson 2019). However, most scientists rarely discuss reflexivity and ethics in their work and when they do, they often struggle to move that knowledge from theory to practice (Ferraro et al. 2021). To help overcome barriers to reflexivity, Beck et al. (2021) provide practical prompts over four distinct dimensions: understand your purpose, establish real connections, inform and transform, and learn from the past. While it is not a realistic expectation that all scientists and stakeholders gain a deep understanding of the above knowledge before they work together, there are some principles gained from human subjects research that could be brought to bear. 3.1.3 Research ethics related to studies of human subjects Stakeholder engagement differs from human subjects research in that research is conducted in collaboration with other human partners as opposed to research data being collected from human subjects. However, human subjects research holds important historical lessons that can be applied to stakeholder engagement. Prompted by ethically abusive biomedical experiments such as those revealed during the Nuremberg War Crime Trials (International Military Tribunal 1949; Katz 1996) and the Tuskegee Syphilis Study (Brandt 1978), the Belmont Report outlines unifying ethical principles to which scientists working within the U.S. must adhere to protect the interests and rights of human subjects (United States 1978). The report describes three foundational principles and their application in a research setting: respect for persons (participants are able to provide fully informed and freely given consent), beneficence (benefits of participating are likely to outweigh the harms), and justice (vulnerable populations are not exploited when distributing research burdens and benefits) (ibid). Even today, the report remains the foundation for protective regulation by the U.S. Department of Health and Human Services, and the principles are codified in the widely used Institutional Review Board (IRB) process. In the decades since, continued examination has found that the Belmont Report principles are not perfect nor are they all-encompassing. Examples of identified gaps include overall conceptual vagueness, especially in how the three ethical principles relate to each other (Miller 2003); minimal consideration for the ethics of experimental design (Vollmer and Howard 2010); and an inability to cope with new concerns around big data and participant confidentiality (Friesen et al. 2017). Community-based participatory researchers struggle with the rigidity of the IRB approval process and implementation of these ethical principles, as their research approach requires shared project decision making and flexible protocols (Shore 2006). In addition, the Belmont Report focuses only on consideration of individual harms and ignores the potential for community harms (Tsosie et al. 2019), which is particularly important in respect to Tribal sovereignty, solidarity, and self-determination (Friesen et al. 2021; Saunkeah et al. 2021). Perhaps most importantly, researchers are building on these basic principles by examining the ethical decision space that scientists face outside of formal institutionalized processes for human subjects research (e.g., IRB approval). One case study described using an informal yet nuanced consent process that was continually renegotiated from the pre-proposal stage until well after the official end of the project (Cockburn and Cundill 2018), and an analysis of five case studies documented the importance to stakeholders of early conceptual benefits (e.g., better understanding of a system) in building a strong trust-based relationship that eventually yielded instrumental benefits (e.g., recommended actions to implement) (Ferguson et al. 2022). Other scholars have proposed supplementing the Belmont Report principles with appropriate representation (how human and non-human elements are represented in a project), self-determination (respect for power and sovereignty), reciprocity (ensuring direct and tangible benefits for partners), and deference (respect for all knowledge systems) (Wilmer et al. 2021). Even with these limitations, the Belmont Report principles can be useful for examining how scientists’ ethical responsibilities extend from human subjects to stakeholder engagement. The core principles of respect for persons, beneficence, and justice are intended to be foundational to research involving humans, so they arguably should also emerge in any kind of evaluation of research conducted in collaboration with human partners. Previous research in this area has primarily examined the perspective of the scientist on why they engage with partners (e.g., Clark, 2010; Pearman & Cravens, 2022; Steger et al., 2021), yet the voices that can most strongly speak to usability and reciprocity are those of the stakeholders. Here we focus on stakeholder perspectives to better understand the benefits and harms that they experience while participating in knowledge production. 3.2 Methods The data and results presented here are part of a broader initiative to describe and understand the impact of scientific projects funded by the U.S. Geological Survey North Central Climate Adaptation Science Center (NC CASC). This initiative included a document review of project proposals and reports, a survey of stakeholder participants in these projects, and interviews with a subset of surveyed stakeholder participants. This paper is an analysis of the interview data, for which IRB approval was obtained via Colorado State University. The full interview protocol and a description of the methods are available within Middleton (2020). Projects funded by the NC CASC are intended to produce regional science to aid natural and cultural resource managers in responding and adapting to the impacts of climate change, and investigators were encouraged from the proposal development stage onwards to utilize co-production as their stakeholder engagement approach. We use the NC CASC project portfolio as an exemplar case study that can provide generalizable results to other cases (Flyvbjerg 2006) because of each projects’ intent and claim to engage stakeholders in the co-production of actionable science. The primary investigators for sixteen distinct NC CASC projects were contacted to identify the stakeholders with whom they attempted to engage in co-production. Investigators returned a total of 188 unique contacts (all invited to complete a survey for the broader initiative). A subsample was generated using two main criteria: 1) identifying the one or two most highly engaged individuals for each of the projects, and 2) obtaining a diverse representation of agencies or institutions. Fifteen individuals from this subsample consented to an interview in which they were asked semi-structured questions about their roles, expectations, and outcomes while engaged in the project. Interviewees were federal, state, non-governmental (NGO), and community planners and decision makers. We use the term “Tribal” to describe the two interviewees who were enrolled members of federally recognized Tribes and “Indigenous” to describe the sole interviewee who was a non-federally recognized Native Hawaiʻian. Some interviewees disclosed that they held scientific position titles, such as ecologist or conservation specialist, but their primary duty was to serve as a technical expert and not to conduct novel research. Interviews were conducted primarily by video chat (or phone when not possible) individually, except for one case when two participants were interviewed together. Recordings (obtained with consent) were transcribed by an external service and checked for accuracy. For consistency, we continue to use the previously published interviewee labels (not numbered one through fifteen), although the category labels have no bearing on the results presented here. Interviews were coded in NVivo (QSR International Pty Ltd. 2020) by a single researcher using a modified grounded theory approach (Corbin and Strauss 2008). Table 3.1 is a list of all themes that emerged during initial coding. We then synthesized the wider literature surrounding stakeholder engagement to generate ideal definitions for what the three Belmont Report principles look like when operationalized in the context of co-production. First, we align respect for persons with the engagement best practice of including stakeholders in defining and consenting to project roles, goals, and expectations (Carney et al. 2009). Second, we align beneficence with the engagement best practice of prioritizing the development of research findings and products to benefit stakeholders (McNie 2007). Finally, we align justice with the engagement best practice of avoiding exploitation and extraction by proactively considering historic and present-day inequities and transparently building trust in under-resourced or under-served communities (Chief et al. 2015). Table 3.2 maps key emergent themes from the initial coding to the ideal definitions. In the Results and Discussion section, we examine the key emergent themes from our interview data and explore how our interviewees perceived adherence to these principles for each of the projects in which they were involved. Because interviewees represent neither a random nor representative sample of stakeholders, we do not provide quantitative results of our coding. Instead, our results describe the perspectives of highly engaged stakeholders and qualitatively identify narratives and concepts that warrant further exploration (Rust et al. 2017). We follow published standards for the best available social science (Charnley et al. 2017) to ensure the accuracy, reliability, and relevance of our findings. 3.3 Results and discussion This section comprises three subsections, each which summarizes stakeholder perspectives aligned with one of the Belmont Principles and its associated key emergent codes from Table 3.2. Discussion and interpretation of the data are presented alongside the results. Quoted material is lightly edited for brevity, clarity, and to ensure privacy (e.g., removal of filler words, false starts, and identifying information). Any changes do not alter intent or meaning. 3.3.1 Belmont principle #1: respect for persons Existing recommendations for stakeholder engagement in a research project include identifying the purpose for engagement, the role that the stakeholder will serve within the project, and the mechanism and process for engagement, all prior to the start of the project (Carney et al. 2009). These actions ensure that the stakeholder can clearly understand their role and freely consent to participating in the project. However, interviewees in our sample perceived a wide spectrum of understanding and definition around the roles, expectations, and engagement processes across the projects in which they participated. Stakeholders held positive perceptions about projects that established roles and processes early and repeated that information often to ensure that there was well-established mutual understanding and agreement. A federal resource manager shared “I can't stress enough, early on in the process, identify the right people to bring in” (RD2). One board member of a water conservancy district highlighted the importance of repeatedly convening the group: “At the beginning of every season, [project investigator] would hold a workshop with [project co-investigator] to get everybody back on board, remind them what we're doing, what the overall goals were, then, hit the field” (N1). The director of an Indigenous-led conservation NGO highlighted the cultural and relational importance that such processes can hold: “Indigenous attention to protocol and traditional ceremonial opening is everything because this is where we constantly recheck our relationship” (N4). When roles and processes were not well-established, some stakeholders held negative perceptions around the resulting outcomes. A conservation partnerships coordinator with the federal government commented on the ramifications of not defining roles early in the process: “We put stuff in the proposal[, but] none of that stuff was ever well thought out and that was a horrible, horrible problem for us in terms of communications” (RD5). However, a federal forest resource planner recognized and expressed frustration around the trade-offs that must be balanced when allocating project time between conducting engagement and research: “So much of the time that we spent was designing the process by which we were going to do the project. If we all were at a point where we knew how it was going to be designed, then you could just leap into the [science] part of it.” (D2). Stakeholders identified that social exchange in both informal and formal settings was key to establishing mutual respect and understanding and demonstrating genuineness during the research project. A faculty member at a Tribal college stated “the most important thing we can do is just get together and talk” (N3). Multiple stakeholders mentioned the importance of informal interactions, such as “occasionally sitting down and drinking a beer” (N1), to developing and strengthening relationships. This board member of a water conservancy district continued “To actually be out there in the field and working with these guys [...], I really value the people we have here on our public lands right now” (N1). The Tribal college faculty member also shared the importance of interactions outside of institutional structures in creating a sense of trust and genuineness: “Being able to visit with [the investigators] not necessarily in a formal structure, after hours, over dinner, over a beer, that kind of interaction is really, really critical... I always felt very comfortable, and I never had the sense that this was in any way an exploitative relationship” (N3). Stakeholders also expressed appreciation when scientists were willing to engage in practitioner-led spaces, as one extension specialist recounted: “A lot of these scientists that were involved do not attend [land manager organization] meetings[, but the investigator team’s] willingness to engage beyond just handing off a bunch of information [...] was really good” (R1). Stakeholders perceived that a weak relationship between themselves and the scientist(s) could lead them to be protective of their time and resources. Stakeholders also hinted that the stigma of being a bad partner can stay with a scientist beyond the span of a single project and influence future engagement. One federal park ecologist alluded to stakeholder wariness based on prior negative experience with the project investigator: “The [manager group] went into this [project] very cautiously. Part of it was probably who was leading the research” (RD3). A federal conservation specialist described how previous negative experiences with some investigators caused them to be protective of their group’s data with all outsiders: “People will tell you that we're a difficult group to work with because we won't share data unless we see a benefit to it or something... It takes time and effort to organize the data, compile it, give it to somebody, and then explain it, and then sometimes it gets used incorrectly” (R2). This interviewee continued on to explain how their protectiveness was motivated by a perceived waste of resources when data and information were used incorrectly: “It slays me to see some of this money squandered the way it is, so that's where I'm coming from” (R2). Stakeholders mentioned three distinct roles that they could play in the research process: a “cheerleader” (N1) or “promoter” (N3) who helps obtain funding or other support prior to or during the project; a provider of contextual information to the research team during the selection and design of research methods and outputs; and a verifier or disseminator of findings at the end of a project. The first role was described by a federal park manager as being “on the ground level of getting grant funding for it and writing supportive letters” (D3), while Indigenous and Tribal stakeholders perceived themselves as cheerleaders within the projects for their communities and as cheerleaders for the projects within their communities. One faculty member at a Tribal College described offering insights to the project team about “what we might try to do differently or what we might explore from a unique Indigenous lens” (N3), and a Tribal water manager expressed “the big hurdle was to get [the Tribal business council] to agree to the study, and I did that” (RD6). The second role was described by a state natural resource manager as “primarily providing technical assistance in the context of the needs of [our state] and our agency” (D1), while a federal forest ecologist characterized themselves as “the point person from the science perspective” (RD1). Finally, while stakeholders recognized the importance of their third role in bringing research to applications, they also perceived it as being too late to make adjustments to enhance usability if it was the only point when they were engaged. An extension specialist described how investigators “were “window shopping” information” (R1) when they brought together a large group of managers at the end of a project to see if anyone could use the final products. Importantly, stakeholders engaged poorly in a project often did not perceive themselves as being fully integrated team members; instead, they described being brought in as needed based on the priorities and interests of the scientist(s). A federal forest ecologist summarized this lack of integration into the project: “There was a sense that this is […] the scientist's project that they want to make useful to you, as opposed to going into it with ‘we are co-owners’” (RD1). One federal park ecologist shared their perception that investigators came in with preconceived notions about what was needed (“This is what we're going to do to help you out”) instead of truly listening to managers (“Hey, here's this opportunity. What do you guys think or need?”) (RD3). And a Tribal water manager expressed their feeling of false or token inclusion in a project by stating “we were at the table, but not on the menu” (RD6). Multiple stakeholders expressed a desire to see what the research team was doing “behind the curtain” (RD3) during the “radio silence” (RD5) between meetings and to play a more active role in the research process. An extension specialist shared that being engaged solely at the investigator’s behest could cause them to be cynical about intentions: “They call you three years in and say, ‘Hey, we need a fact sheet and a workshop. Can you put this together?’ And so you get kind of cynical about that approach, obviously” (R1). The first Belmont Principle, respect for persons, specifies that study participants must be able to provide fully informed and freely given consent. In the context of stakeholder engagement, we align this principle with open communication and agreement on the role of stakeholders in the project and how that role was established. Stakeholders mentioned that following best practices for meetings, such as mutually defining goals and objectives early, establishing rules of engagement together, and feeling that inclusivity was being prioritized (Golden et al. 2021), were essential to understanding their role and expected responsibilities. When stakeholders did not have a clear understanding of their roles and responsibilities, they mentioned frustration over wasted resources and time. In turn, these feelings could eventually impact the reputation of the scientists, leading stakeholders to engage with them cautiously or not at all in the future. Ideally, scientists should ensure that stakeholders have a clear understanding of and consent to their roles and responsibilities within a project, with continued check-ins throughout the lifetime of a project, (Carney et al. 2009) to ensure effective and respectful use of resources and time (Buxton et al. 2021; Clark 2008). 3.3.2 Belmont principle #2: beneficence Stakeholder engagement in the research process is predicated on the assumption that an upfront understanding of needs is directly linked to greater benefits at the project end (McNie 2007). However, some interviewees perceived that consideration of research benefits to them by scientists was a “check that box” (RD1) exercise in response to requests for proposals from funding agencies that require or encourage consideration of end-user needs. An extension specialist described the tension between stakeholder-driven objectives and those that are motivated by federal funding priorities: “The problem is if it's not an issue or a topic that has relevance among your constituent basis, then this is a fairly meaningless project” (R1). A federal conservation specialist also highlighted the pressure on investigators to conduct cutting edge science: “Researchers will take the money to address the question, but they will change it to fit some ecological theory that's hot at the moment or something that they think is more publishable” (R2), going on to say: “My experience with a lot of researchers [...] is that our role is to help them get the money and then the product just sorta goes its own path” (R2). This conservation specialist continued on to recognize that not all of their previous experiences had been negative and that there were research partners who were genuinely committed to creating actionable science: “Some of the folks that we work with are very good about making sure that our questions get answered[, …] so a lot just depends upon people” (R2). Tribal and Indigenous rights-holders expressed that scientists needed to consider the benefits and harms of the research project to the community as a whole in addition to the benefits and harms to an individual person or agency. The director of an Indigenous-led conservation NGO highlighted the importance of considering relationships and community from the start of a project: “The biggest component is the relational component -- how do we do this with each other in the community?” (N4). These rights-holders also asserted that scientists working within their communities need to improve their understanding of Tribal sovereignty and how it impacts the ways in which Tribal policies and actions are decided. A Tribal water manager shared an example of how cooperative decision making occurs within their Tribe: “We're a sovereign nation, and that sovereignty is such a strong statement that people don't really realize it… We meet collectively at a town hall meeting, every so often, and there's an issue that needs to be addressed and discussed, maybe a resolution will come out of that discussion, establishing Tribal law” (RD6). As Whyte (2013) argues, Traditional Knowledge is communally held and shaped, so in many communities, all members must be provided a transparent opportunity to engage in conversations about the risks and benefits of participating in a project. Rights-holders expressed that genuine commitment to community benefits was demonstrated by scientists who took the time to listen to their needs, transparently prioritized those needs, and developed themselves into trusted sources of information for the community. A faculty member at a Tribal college stated that scientists should be prepared to demonstrate their genuine commitment when engaging with their community: “We've had people for a century come and tell us they're going to save the Indian, so there's suspicion often and wariness, [... and] you need to be prepared to deal with it” (N3). And a Tribal water manager expanded on the process of building trust early by saying: “[The co-investigator] developed the trust with Tribal members and Tribal committees, and once that trust is established, then things happen a lot more smoothly” (RD6). This water manager continued on to describe the importance of identifying the key individuals with whom to build trust: “The very first thing that [the co-investigator] did [in a previous project] was address the elders and that's what she did here... Most people, if not all people, don't understand that” (RD6). Stakeholders perceived that the benefits of the research project could be more tangible when the team made a commitment to equitable inclusion and integration of non-research partners. A federal hydrologist expressed a positive attitude towards a research team that strove to be as inclusive as possible: “They listened to everything – they were very methodical in writing ideas down and capturing everything – so I knew that they were going to put on a good organized workshop, keep things moving, and come out with a great product” (N2). An extension specialist shared that their experience co-producing project outputs as a group was “an ideal situation” because “the modeling was characterized properly but the message was also delivered in a way that was palatable” (R1). However, even when the research team was genuinely committed to delivery of benefits to stakeholder partners and followed through on implementation of a robust engagement plan, stakeholders sometimes still saw project benefits as “[falling] short in providing actions” (D3). A federal forest resource planner stated that trying to identify a common set of needs across multiple stakeholder groups that could be met with a single product “was biting off way too much than could be done in that workshop” (D2). And a federal hydrologist confessed that, even after creation of what they considered to be a successful product, there was still a gap between information that was usable and information that would be used by managers: “As far as implementing these things, it still seems kind of abstract… Take these tools and actually put them on the ground? I just don't know how that's going to happen. If that's even going to happen” (N2). The second Belmont Principle, beneficence, emphasizes that the likely benefits of participating in a study should outweigh the potential harms that a participant might experience. When engaging stakeholders in a project, this principle aligns with scientists foregrounding benefits to the stakeholder and intentionally designing and producing outputs for use in plans, decisions, or actions. However, multiple interviewees perceived a lack of follow-through from their research partners towards addressing their needs and ensuring that they benefited from the process. Tribal and Indigenous partners mentioned the importance of considering community benefits and understanding the role of community-based decision making in the research process. Including and listening to diverse viewpoints, recognizing and prioritizing stakeholder needs, and demonstrating genuine commitment to stakeholder interests were all project characteristics mentioned as likely to increase the benefits to partners. Scientists need to ensure that (individual, organization, or community) benefits to the collaborating stakeholders are highlighted throughout the project and that outputs are developed primarily for stakeholder use and secondarily for other venues. 3.3.3 Belmont principle #3: justice To avoid exploitation of vulnerable populations, the burdens and benefits of engaging in the scientific process need to be divided fairly and equally amongst participants. Scientists who conduct engagement in under-resourced or under-served stakeholder communities need specialized skills and knowledge to prevent exploitative and extractive relationships, training for which is demonstrably lacking (Kirby et al. 2019). In our data, themes around exploitation, equity, and justice appeared primarily in our interviews with Tribal and Indigenous stakeholders, although we believe that these results are relevant to other communities that have experienced inequity and trauma. Tribal and Indigenous interviewees perceived that issues around environmental justice and social inequity needed to be addressed directly even when they were not a main project objective or when doing so was new and uncomfortable for most scientists. These rights-holders felt directly acknowledging these issues was critical to demonstrating genuine interest in their communities and building trust. The director of an Indigenous-led conservation NGO expressed that, while the trauma stemmed from historical events, the repercussions of those events were still present in their community even now: “Add to [this project], the wrinkle of an Indigenous culture and a local base community that sees itself in opposition to the large framework… Being able to step in there and then look at these historic traumas, how they're shaping the conversation even 200 years later. How do you right those unrightable wrongs? Nobody was present at the initial infraction. Everybody's living with a reverberation of […] a bomb that has gone off in slow motion” (N4). A faculty member at a Tribal college shared the importance of consideration of issues centered on justice and equity and that ignoring them could continue a lack of trust: “A lot of scientists can sometimes get uncomfortable, [but] you've got to make a linkage to issues of justice [and] of extreme income and wealth inequality... Trying to tell someone on a reservation that they should be super concerned about what's going to happen in 10 years, 25 years, 50 years, they're saying no, what I'm concerned about is we don't have running water... Being aware that there might be other more immediate problems that people are facing, you've got to be sensitive to do that.”; continuing on to say “Really addressing climate change will help them address some of those issues of social justice, of environmental justice, of education inequalities... You've got to make that connection because if you don't, [...] then they just tune out. ‘Oh god, here's another great scientist who has come to tell us all the things that we should be doing’” (N3). Stakeholders expressed that boundary actors (themselves or others) could pull project contributors together, help accelerate the relationship- and trust-building process by leveraging their prior relationships and knowledge, and produce more just and equitable outcomes. These boundary actors are people who span across and can navigate among multiple disciplinary, institutional, or cultural spaces (Goodrich et al. 2020). A Tribal water manager shared how their ability to act as a trusted representative on behalf of the project was integral to quickly generating feedback from the community and interpreting that feedback for the project team: “I was the key person or the catalyst that pulled [these meetings] together because there was a certain amount of trust that I had with the [...] individuals that we needed…. There has to be a trust developed in a reservation community environment before there are people willing to expound their thoughts and their concerns”; continuing on to say “You need a local person to articulate your thoughts and to take the local people and have their thoughts articulated back to you and just have a cordial meeting in the beginning about what can happen and what expectations your client might be thinking about” (RD6). The director of an Indigenous-led conservation NGO described their ability to cut through boundaries between partner groups on a project: “It's like everybody's in their own stratified slice, and they argue about where they are in the lasagna, but it's all cheese to me... I don't see the stratifications.” (N4). However, an extension specialist expressed the potential relational harm that boundary actors may experience when representing project scientists to individuals and groups with whom they’ve built trust: “You can think of trust with [on-the-ground] groups as a kind of capital [...] that you've earned over the years. I may burn a lot of my trust capital delivering this message, and I may get very little back” (R1). The third Belmont Principle, justice, reminds scientists that vulnerable populations should not be exploited when considering who bears the benefits and burdens of a study. We align this principle with a need for scientists to think about environmental justice and equity before engaging with under-resourced or under-served stakeholder communities. Tribal and Indigenous interviewees expressed that projects should not be conducted with their communities or on their lands without consideration of historical and present-day inequity and trauma. If scientists lack or are unable to obtain the necessary skills to address these issues, they can use boundary actors as trusted intermediaries instead. However, these boundary actors put their own partnerships with a community at risk on behalf of the scientist and can experience relational harm if engagement is poorly executed. Scientists must carefully contemplate and thoughtfully execute their intended engagement strategy to prevent these communities, or any boundary actors who act on behalf of the project team, from experiencing additional harms. The justice principle suggests a high standard that must be met for ethical reasons. Conversely, it suggests that project teams that do not have adequate time, resources, or expertise to address these issues would be better off not engaging with stakeholders within their project or perhaps even leaving a particular research question to be addressed by a team better situated to undertake it in a just manner. 3.4 Conclusions Our study investigated the perceptions of stakeholders about their involvement in co-produced climate adaptation research projects. Our qualitative analysis of the interview data was informed by the three Belmont principles: respect for persons, beneficence, and justice. Although stakeholder engagement does not usually fall under human subjects oversight, we argue that the Belmont principles can help guide ethical stakeholder engagement. Intentionally or not, scientists whose practices aligned with our operationalized versions of the Belmont principles were perceived to be transparent, committed to stakeholder interests, and trustworthy. In contrast, stakeholder engagement that was inconsistent with the Belmont principles caused individual or community harm, especially when inadequate consideration was given to why stakeholders needed to be engaged and how their contributions would be incorporated into a project. Although proactive practices by scientists may occur by coincidence or simple kindness, intentional focus on the Belmont principles may lead to more consistently ethical outcomes in climate adaptation. Importantly, our results demonstrate that even with the best of intentions, scientists can cause harm to the stakeholders they engage (in addition to failing to do good science) if they don’t pay attention to the ethics of engagement. Scientists who do not inform and ask stakeholders to consent to clearly defined roles and expectations can cause broken trust and create feelings of disrespect and wasted time and resources. Projects that prioritize using a cutting-edge scientific technique over producing decision-ready results are disingenuous and can result in extraction of knowledge from the community without returning value to them. Selection of an under-served community as an “interesting” study site without consideration of historical inequities can contribute to the continued exploitation of that community. Optimally, scientists should provide a high standard of care to the people they engage in knowledge production and strive for their actions to be beyond performative. If scientists are unable to commit to reflecting on the ethics of engagement, then they should consider not engaging with stakeholders. In particular, interviewees highlighted the importance of addressing issues of environmental justice and equity even when doing so would be uncomfortable for many scientists and could be perceived as crossing over into advocacy. Scientists often see themselves as honest brokers in policy and decision making settings (Pielke Jr 2007), individuals who provide factual information on possible outcomes without promoting any single choice. This desire to remain an honest broker can be reconciled with the request to consider justice and equity; scientists can focus on procedural justice –– advocacy for consistent, fair, and transparent processes that consider power imbalances (Ruano-Chamorro et al. 2022). Such an approach can result in more ethical and equitable engagement between project contributors, while allowing scientists to remain outcome-agnostic (ibid). While prioritizing procedural justice, scientists can use reflexivity to consider the authority that they hold within their role (Durose et al. 2021) and their ability to disrupt the balance of power within a community (Reyna et al. 2021). Ultimately, scientists must recognize and reciprocate the value of contributions they obtain from a community in the way that is seen as beneficial and requested by that community (ibid). We note that there are limitations to our study due to the scope of our data. Projects included in this analysis were carried out only in the U.S. Northern Great Plains, and interviewees were selected for high engagement and to represent a diversity of institutions. As a result, we acknowledge that our sample is geographically limited, small in size, and may miss key stakeholder populations. Importantly, our interviewees only included three Tribal and Indigenous individuals, although portions of their perspectives should still be generalizable Tribes and Indigenous communities as a whole and other under-served or under-resourced populations. Additionally, we did not explicitly include questions about previous scientist or stakeholder experience and training regarding relevant topics, such as facilitation or research ethics. Despite these limitations, our results help us understand the experiences of stakeholders engaged in these projects, examine how ethical those interactions were, and identify directions that warrant further research. Research and resource management challenges around climate adaptation can be enriched by our understanding of social phenomena (e.g., governance, norms), social processes (e.g., decision making, communications), and individual attributes (e.g., values, behaviors) (Bennett et al. 2017). However, 51 percent of forty nine cross-disciplinary ecologists surveyed in the United Kingdom ranked direct engagement with stakeholders as their first choice for improving the social and human dimensions of their work, while only 27 percent ranked working closely with social scientists on a research project first (Lowe et al. 2009). Many scientists working in this space are trained exclusively in biophysical disciplinary traditions, and when scientists who lack proper training and expertise choose to conduct empirical social science on their own, it can result in inappropriate application of methods and inadequate reporting of results (Martin 2020; Moon et al. 2016). Even when a social scientist with relevant expertise is included as part of a project team, it can be seen as a mechanism for facilitating the stakeholder engagement process and not as supporting a line of inquiry in its own right (Robinson et al. 2019). While we don’t expect that all scientists and stakeholders be trained in social science traditions, we strongly suggest that projects consider including at least one person who brings relevant skills to the table to reduce the risk of unintended harm. In this paper, we describe how poor engagement of stakeholders in co-produced climate adaptation projects can lead to individual or community harms and recommend that projects proactively consider ethical engagement. While we use climate change as the lens for this paper, we believe these ideas are applicable to stakeholder engagement to address other environmental problems. For example, the U.S. faces pressing questions around the quantity and quality of available future water resources under both a changing climate and a changing set of human activities, with increased use of citizen science as one highly recommended strategy for response by the scientific community (National Academies of Sciences Engineering Medicine 2018). Our findings could be applied to such a project by encouraging scientists to consider which citizens are engaged in data collection and how their roles are defined, how those data will be used and how benefits to the citizen are ensured, whether there are any community sensitivities around sharing such data openly, and how to limit data voids in vulnerable communities where citizens may not have time to volunteer. As a discipline, geography has a history of blending ideas from different traditions to think more broadly about the world, and geographers are uniquely poised to contribute across disciplinary boundaries to knowledge production to address environmental problems of the Anthropocene, including adapting to climate change. 3.5 Acknowledgments This work was funded by the U.S. Geological Survey South Central and North Central Climate Adaptation Science Centers. Jill Lackett and Lindsey Middleton conducted the data collection. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. We thank the members of ABD's dissertation committee, the Climate Adaptation Science Center Evaluation Working Group, and the Science of Actionable Knowledge Working Group for insightful discussions around what constitutes successful actionable science for climate adaptation. We thank Katherine R. Clifford for her early peer review and feedback, which honed and improved the concepts in this paper. Table 3.1 - All emergent codes All emergent codes from modified grounded theory approach. Emergent codes Agency / institution need for science Agency / institution planning cycles Agency / institution sources for scientific information Barriers to co-production Barriers to creating plans Barriers to adaptation action Barriers to obtaining scientific information Boundary spanning & trust Environmental justice & social inequity Establishment of roles and relationships Overall project goals / objectives Perception of co-production overall Perception of project benefits / beneficiaries Perception of project ownership Process for identifying needs / benefits Specific role of stakeholder Strengths of co-production Table 3.2 - Summary definitions and codes used for analysis Belmont principles, definitions, ideals, and key emergent codes used for analysis. Principle Belmont Report definition Operationalized co-production ideal Key emergent codes Respect for Persons Participants are able to provide fully informed and freely given consent * Stakeholders participate in defining and consent to their project roles and responsibilities * Mutually shape and understand project goals and expectations * Process for establishing roles and relationships * Specific project role of stakeholder * Perception of project ownership Beneficence Benefits of participating are likely to outweigh the harms * Hold project benefits to stakeholder (individual, organization, or community) in the foreground * Develop outputs primarily for stakeholder use and secondarily for other venues * Perception of project benefits / beneficiaries * Process for identifying needs / benefits Justice Burdens and benefits are distributed fairly and equally * Understand and consider historical and present-day inequity and trauma before engaging with under-resourced or under-served stakeholders * Build relationships of trust and transparency * Environmental justice & social inequity * Boundary spanning & trust CH. 4 - When does science achieve societal impact? Authors: Aparna Bamzai-Dodson1,2; abamzai@usgs.gov [corresponding author] Renee A. McPherson2,3; renee@ou.edu 1U.S. Geological Survey North Central Climate Adaptation Science Center 2University of Oklahoma Department of Geography and Environmental Sustainability 3U.S. Geological Survey South Central Climate Adaptation Science Center Intended for submission to The Professional Geographer 4.0 Abstract To cope with the complex environmental impacts and interactions of the Anthropocene, the research community is increasingly being asked to produce science that can directly support policy and decision making. To achieve such societal impact, scientists are engaging “out in the world” with stakeholders to better understand their needs and inform knowledge production. However, there are a wide variety of stakeholder needs, ranging from learning how to establish collegial relationships with scientists to obtaining specific information to ingest into a pre-existing decision process, many of which do not directly connect to traditional methods of measuring scientific impact (e.g., publication citations, journal impact factor). In this paper, we describe how concepts from the discipline of evaluation can be used to evaluate the societal impacts of evaluation and present a case study application. We developed and deployed a survey instrument that asked stakeholders engaged in climate adaptation research projects about their perceptions of the engagement process, their use of outputs and outcomes, impacts to relationships, and demographic information. We argue that applicants should consider evaluation from the beginning of project development to ensure that they have sufficient resources and expertise available. 4.1 Introduction 4.1.1 Background The defining characteristic of the Anthropocene is the impact of human activities on environmental systems (Crutzen 2006), with the discipline of geography contributing substantially to our understanding of the resulting complex socio-bio-physical effects and interactions (Annals of the AAG Special Issue: The Anthropocene; Butler 2021; Winkler 2016). To ensure that science is provided in an actionable form to support policy and decision making (useful, usable, and used; Dilling and Lemos 2011) when it is critically needed, geographers are interacting “out in the world” with information end-users (The Professional Geographer Focus Section: Out in the World: Geography's Complex Relationship with Civic Engagement; Barcus and Trudeau 2018), known more broadly as stakeholder engagement (Reed et al. 2018). Engagement of stakeholders in research projects has a demonstrated positive impact on subsequent information use for decision making (Nguyen et al. 2019). While geographers have a wide range of approaches available to them when engaging with stakeholders (Bamzai-Dodson et al. 2021), traditional definitions of research success most often focus on agency or academic, and not societal, impact (Meadow and Owen 2021; Robinson and Hawthorne 2018). Societal impacts are not well captured by traditional definitions of success for research impact, such as number of publications and citation metrics (Cozzens 1997; National Research Council 2012). Defining success for societal impact can be challenging because the needs of stakeholders can vary from learning how to work collaboratively with researchers (collegial engagement) to being generally better informed (conceptual information use) to taking specific on-the-ground action (instrumental information use) (Meadow et al. 2015; VanderMolen et al. 2020). To accommodate this diverse range of needs, evaluation approaches need to be specifically tailored to examine the impact and actionability of such information to a community (Block et al. 2018; Ford et al. 2013). In particular, it’s important to identify empirical methods for measuring and monitoring trust between the producers and users of scientific information, as trust plays a key role in the uptake of information for policy and decision making (Boschetti et al. 2016; Lacey et al. 2017). In this paper, we introduce concepts from the field of evaluation and provide an overview of how they may be integrated into the production of actionable science to help define indicators for and evaluate societal impacts. We focus on climate impacts and adaptation research, as it is one area where the production of actionable science is growing at a rapid pace. We present results from a case study application of these concepts to understand the societal impacts of research funded by the U.S. Geological Survey Climate Adaptation Science Center network and discuss how these findings can be further developed. We argue that deliberate consideration of success and explicit attention to evaluation can improve the actionability of science. 4.1.2 Evaluation theory and practice Evaluation helps individuals and organizations learn and improve program operations by testing the effectiveness of or changes in activities; it differs from assessment, which is intended to grade or score performance (Baylor et al. 2019; Patton 2018). The field of evaluation utilizes a variety of different theoretical approaches (e.g., summative, formative) and methodological techniques for operationalization, with a rich set of literature differentiating between the advantages and limitations of these approaches and techniques and identifying the appropriate contexts for their use (Hansen 2005; Preskill and Russ-Eft 2004; Weiss 1998). Therefore, no matter the context, it is incumbent upon the evaluator to initially determine the kind of evaluation required and ensure that they draw upon the appropriate best practices when designing the evaluation process. Here, we summarize a few key approaches to designing and conducting evaluations and provide examples of how they might be applied to climate impacts and adaptation research. Every evaluation must begin with an appraisal of what it is that specifically needs to be evaluated (the evaluand) and what aspects of the evaluand (for example, process or outcomes) are most appropriate for evaluation (Patton 2011, 2012). As part of this appraisal, the evaluator identifies the purpose of the evaluation and other contextual factors, such as the level of analysis or precision (ibid). Once the approach and method are identified, the evaluator selects suitable variables for measurement and analysis, which cover necessary aspects of the evaluand that are to be evaluated, while also being scientifically sound (measured reliably, scaled appropriately, etc.) (ibid). If the approach is quantitative or mixed-methods, then the evaluator also must ensure that statistical assumptions and analyses are logically sound and allow sufficient statistical power. Summative evaluation synthesizes the evidence of success at a given (most often retrospective, end of project) point in time and provides a single estimation of performance, while formative evaluation is an ongoing process of estimating performance that guides self-corrective actions (Patton 2012). However, the line between formative and summative evaluation is a fine one, as an initial summative evaluation may provide the baseline for or first step in a formative evaluation process (Taras 2008). The distinguishing feature between formative and summative approaches is whether or not actions can be modified as the process is carried out, and not the techniques or methods (Wiliam 2006). In fact, the techniques and methods used for conducting either kind of evaluation can be the same (Torrance 1993). Program-level evaluation examines whether the collective impact of a set of activities or policies is aligned with the overall goals and whether there are any unintended consequences (Donaldson 2007). In contrast to program-level evaluation, project-level evaluation allows for a better understanding of the perceived merits of investing resources in one initiative or activity versus another (Patton 2012). Evaluation of specific projects can provide crucial learning and development opportunities to improve organizational practices and contribute to overall program evolution (Donaldson 2007; Patton 2011). Process evaluation determines whether an activity has been implemented as intended by examining internal characteristics that influence the success or failure of the activity (Craig et al. 2008). Outcome evaluation determines whether an activity has achieved its intended goals (typically, some kind of system change) (ibid). In other words, process evaluation focuses on the “how?”, and outcome evaluation focuses on “to what ends?”. The challenge with outcome evaluation is that it requires targets that are clear, specific, and measurable. Since outcomes often do not manifest until substantial time has passed, it is also possible to identify intermediary outcomes that serve as crucial steps towards achieving the overall goal (Ardoin et al. 2015). All of these approaches intersect in layered ways when operationalized, and an evaluator should make intentional selections between them to meet the goals of the evaluation. For example, to enhance the investment of public funding for actionable science, an evaluator may elect to conduct a formative program-level process evaluation. This approach would help the funding program develop funding opportunities, proposal reviews, and project management that prioritize the use of information for policy and decision making. In contrast, to improve their understanding of the best practices necessary for successful creation of products by stakeholders and researchers and pitfalls to avoid, an evaluator may instead elect to conduct a summative project-level outcome evaluation. Although guidance for investigators is emerging on proactive consideration of societal impact during project development (Meadow and Owen 2021), evaluation of climate impacts and adaptation research has traditionally occurred in an ad hoc summative manner and has not been robustly informed by evaluation theory and practice. 4.1.3 Success and evaluation for actionable science Here, we present some applications of evaluation theory and practice to better understand the societal impact of producing actionable science from climate impacts and adaptation research. There are several traditional models and mechanisms for gathering quantitative measures of scientific impact, including research inputs such as the amount of funding obtained and research outputs such as the number of publications, their associated journal impact factor, and number of citations (Coryn et al. 2007) or the number of downloads of products from websites (Doemeland and Trevino 2014). However, none of these measures identifies whether or how the stakeholder used the information to make a decision because knowledge delivery does not equal knowledge use (National Research Council 2012). To evaluate actionability of science, an evaluator can focus on stakeholder perception of the process, such as workshop evaluations, or, more important to actionability, how well stakeholder input increases the usability, or even better, the actual use, of the research outputs in their job. In practice, evaluating information usability and use by policy and decision makers is notoriously difficult. Wall et al. (2017) provide an initial direction for evaluating the true impacts of actionable climate science, such as that agencies and managers find the science credible and the findings are explicitly applied in agency planning, resource allocation, or a policy decision. McNie (2013) suggests other options, such as evaluating whether “all relevant information was considered” or “whether the science was understood and interpreted correctly”. Quantifying these impact metrics is difficult, but evaluators can conduct evaluation through follow-up surveys to decision makers (e.g., Guido et al. 2013) or through quantitative or qualitative studies of the language in plans and decisions (e.g., VanLandingham and Silloway 2016). More tractable is the evaluation of stakeholder inputs that inform specific numerical model parameters or outputs. For example, models can be assessed to determine whether output quality increased as a result of stakeholder input, such as examining whether simulations of biomass availability improved after incorporating local knowledge of wildlife interactions and forage behavior (Beeton et al. 2019). More nuanced approaches to incorporating perspectives from stakeholders require deeper engagement and focus primarily on understanding how the stakeholder experienced or perceived the engagement. These approaches can include examining factors such as the time required to build the relationship, an understanding of how the project might influence the person or their community, and the nature of the interactions between scientists and users, including building trust (Boschetti et al. 2016; Lacey et al. 2017). Data collection may include surveys (particularly those using open-ended questions that allow people to describe what they experienced or why they hold a certain view) or semi-structured interviews. The iterative nature of some stakeholder engagement in actionable science means formative evaluation is possible through the use of longitudinal evaluation designs, such as the same survey administered multiple times during the development of a decision support tool to ensure that updates to the tool enhance usability (Klink et al. 2017). In situations where stakeholder engagement yields neither scientific nor societal impact, success may be defined in more intangible ways, including integration of stakeholders into the investigator team and satisfaction with the process (Wall et al. 2017). Here, methods for evaluation can focus on identifying and monitoring measurable outcomes on intermediary time scales. For example, a project team can design a conceptual logic model that captures stakeholder impact as a long-term outcome and identifies how to measure change at interim checkpoints (Colavito et al. 2019) or a theory of change-based framework where establishing and maintaining relationships are key social learning outcomes for an entire community of practice (Owen et al. 2019). Regardless of the approach selected, thinking strategically about evaluation from the front-end of a project ensures that appropriate information is collected throughout to monitor whether goals are being achieved and take corrective actions as needed. 4.2 Case study The case study data and results described in this paper are part of a broader evaluation of climate impacts and adaptation research projects funded by the U.S. Geological Survey (USGS) South Central and North Central Climate Adaptation Science Centers (CASC), two regional centers within a nationwide network. This network was established by the U.S. Department of the Interior to “provide climate change impact data and analysis geared to the needs of fish and wildlife managers as they develop adaptation strategies in response to climate change” (Salazar 2009). To achieve this mission, projects funded by the CASCs create products and tools that directly support resource managers in their development and implementation of climate adaptation plans and actions. Funders of actionable science, such as the CASCs, are in a position to influence the form and goals of research across many stages of the process, from setting the priorities that appear in a solicitation to identifying appropriate proposal review criteria to selecting which projects receive funding. Evaluation of and by funders of actionable science is critical to understanding whether actions taken across each of these stages and by individual projects support the overall goal of societal impact (Arnott 2021; Arnott et al. 2020a). From 2013 to 2016, the network was guided by the Federal Advisory Committee on Climate Change and Natural Resource Science, which produced a report recognizing the network’s achievements and providing recommendations on how to improve operations (ACCCNRS 2015). A key recommendation in this report was for USGS to develop an evaluation process to ensure that programmatic activities and funded projects align with the mission (ibid). Suggested evaluation categories include “relevance, quality, processes, accessibility, and impact of science products and services,” although no framework or method for carrying out this evaluation process is provided (ibid). USGS headquarters conducts annual internal and five-year external program-level reviews of the regional centers to examine overall operations and impact (USGS 2021) but does not pursue project-level evaluation. As a result, regional CASCs are developing and piloting their own supplemental project evaluation processes. The broader evaluation of South Central and North Central CASC projects included an analysis of project documentation, a survey of stakeholders engaged in the projects, and a focused set of interviews with highly engaged stakeholders. This paper focuses on the survey, which was intended to provide a summative project-level evaluation of process, outputs and outcomes, and impacts based on the perspectives of stakeholders. This approach was chosen because formative evaluation was not a consideration in the development of the funding program and enough time had elapsed that multiple years of projects had reached completion. Our hope was that evaluation of the entire suite of projects by the program office would provide us with sufficient data to allow us to compare characteristics between dissimilar types of projects (e.g., projects carried out at local scales in comparison to projects to create data at broad regional scales). We utilized an electronic survey of project stakeholders because it was a no-cost option and no resources other than limited staff capacity were dedicated to this evaluation effort. These limitations are commonplace in funding programs, making this case a suitable proxy for conditions faced by other actionable science funders. 4.3 Methods We contacted the primary investigators for twenty eight South Central CASC projects and sixteen North Central CASC projects to identify the stakeholders whom they engaged during the project, resulting in a total of 186 unique contacts for the South Central CASC and 188 unique contacts for the North Central CASC. All contacts were invited to complete the survey, the protocol for which is publicly available on USGS ScienceBase (Bamzai-Dodson et al. 2022) and the design for which is based on published indicators of usable science (Wall et al. 2017). Institutional Review Board (IRB) approval was obtained via The University of Oklahoma (IRB number 7457), and Paperwork Reduction Act approval was obtained from the U.S. Office of Management and Budget (control number 1090-0011). The survey was divided into four sections: process, outputs and outcomes, impacts, and demographics. The survey protocol was pre-tested on staff from across the nationwide CASC network, and their feedback was incorporated into the final version. Six questions asked respondents about the process of creating new knowledge together among investigators, resource managers, and decision makers, focusing on the nature and timing of interactions. Nine questions asked respondents about perceptions of the products developed through the project, including factors that promoted or limited their use by the individual or their agency. Six questions asked respondents about their partnership with the investigators, including what made it likely or unlikely for them to work together again. Four questions asked respondents for demographic information, such as the geography, sector, and professional role that they worked in. Questions were a mix of multiple choice, Likert scale, open-ended, and matrix table, based on best practices for effective survey design (Couper et al. 2001; Fowler 1995). Survey dissemination and collection of responses was carried out electronically using Qualtrics (Qualtrics 2005), with a release date of 7 December 2018 and a ninety-day dissemination window. Data collection was hampered due to the unprecedented U.S. federal government shutdown from 22 December 2018 to 25 January 2019. Federal contacts were re-invited on 1 July 2019 to take the survey during a second ninety-day dissemination window, but response rates remained low. Table 4.1 provides the response rate information per region, and Table 4.2 summarizes the demographics of respondents. All survey questions were optional to complete, so the total responses per question does not always equal the total number of complete responses (49). 4.4 Results 4.4.1 Process: engagement in the process of knowledge production Questions in this section of the survey were designed to examine the nature and focus of interactions between stakeholders and investigators during the process of knowledge production. More than half of the respondents (57.1 percent) indicated their engagement began prior to proposal development, with an additional 12.2 percent engaged during proposal development. Engagement during a project ranged from never (zero times per year) to at least every week (fifty two or more times per year), although most respondents (67.4 percent) were engaged between one to eight times per year. No respondents said that the level of interaction was too much; however, 16 percent said that there was too little interaction. Together, results across these three questions seem to indicate that most projects engaged in early, ongoing, and consistent interactions with stakeholders and that there is no such thing as too much interaction. One respondent described their experience being engaged in a project late and expressed appreciation for the investigators’ responsiveness to their input: “The investigative team was slow to involve those of us who were able to provide more local expertise into the design process, however they did exhibit remarkable flexibility in inviting/allowing that input and then adapting their process to better include such material/knowledge.” The phases of a project during which the most stakeholders reported interaction were definition of the problem (87.5 percent), selection of products (85.7 percent), and dissemination of findings (87 percent). The most stakeholders reported no interaction during the design of research methods (27.1 percent), the collection of project data (27.1 percent), and the analysis of project data (32.6 percent). Only one respondent (3.85 percent) indicated that a formal needs assessment was done as part of the project, and ten respondents (38.5 percent) indicated that needs were determined through informal conversation. Eleven respondents (23.4 percent) indicated that a formal risk or vulnerability assessment was conducted, and eighteen respondents (38.3 percent) indicated that risk or vulnerability were assessed through informal conversation. These findings suggest that stakeholders are being primarily engaged by investigators at key decision points related to the context, scoping, and products of a project and not when decisions are made about research design such as method selection, data collection, and data analysis. Investigators are also preferentially choosing to engage stakeholders in informal ways when determining the management context of a research project instead of following established formal strategies for assessing needs, risk, or vulnerability (e.g., scenario planning, structured decision making, systems engineering). 4.4.2 Outputs and outcomes: production and use of outputs Questions in this section of the survey were designed to determine the types of outputs and knowledge produced by projects and understand how they were used by stakeholders. The most common project output reported by respondents was data provision, ranging from disseminating observations (e.g., place-based phenological data) to projections (e.g., climate model data) (Figure 4.1). Respondents also reported receiving summarized information from investigators, such as two-page overviews of new findings and quarterly newsletters. Notably, some respondents remarked on more subtle relational outcomes such as “many relationships” and “a new world view.” One respondent provided the following feedback on the networking opportunities that their project provided: “The most fruitful and beneficial outcomes from this project will be the connections established between collaborators. It is difficult to quantify [the potential outcomes of new relationships] but I think bringing people to the table is, nonetheless, extremely valuable and worth supporting.” All respondents indicated that projects helped them both be better informed broadly about an issue and be better informed specifically about a particular problem. However, stakeholders indicated that projects were not useful to gain a new technical skill (25.8 percent), formulate policy (23.8 percent), or implement adaptation plans (13 percent). While respondents indicated that projects helped them to understand changes in weather and climate observations and model projections and to link those changes to impacts to resources or places that they manage, no respondents indicated that projects helped them identify, evaluate, or select potential adaptation strategies to cope with such impacts. These results indicate that although knowledge and outputs produced by these projects helped stakeholders be better informed on how and why climate change is important to the resources that they manage, they were not used to make specific adaptation decisions. Twenty-four respondents indicated that there were specific factors that they felt contributed to their use of project outputs and provided descriptions of these factors in open-ended replies. The most common factor was a strong partnership between the investigator and stakeholder, illustrated as “trust, relationships, open-mindedness on all sides” and “an attention to the relationship, protocol, transparency, and communication.” A number of respondents described contexts with a clear management challenge linked to a demonstrated information need, such as a “well defined management need to be explored” and “Federal mandated water settlement legislation.” Respondents also mentioned a number of different ways in which investigators were able to make broad results relevant to their specific management challenge, such as the creation of “fine spatial resolution climate products” and the provision of “alternatives to traditional drought indices.” Thirteen respondents indicated that there were specific factors that limited their use of project results. The most common barriers were a need for additional time to use results (19.2 percent) and resource constraints (15.4 percent). One respondent described how late engagement in a project could act as a barrier to information use: “The one area that could have been improved would have been upfront discussion of delivery mechanisms to achieve broader impacts. The proposal included a component of incorporating results into specific agency products, without talking to the agency manager for all of those products before the proposal was submitted.” Respondents also described a need for “continued data collection and processing,” especially in places where extreme weather events disrupted data continuity. No respondents indicated an issue with the quality of the science provided by investigators. All respondents left projects feeling better informed (conceptual use of outputs, VanderMolen et al. 2020), and no respondents used project information to make specific adaptation decisions (instrumental use of outputs, ibid), suggesting that projects fell short of the funding program’s intended goals. Stakeholders had confidence in the quality and integrity of scientific outputs and understood their broad relationship to management contexts but lacked time and resources to apply such information to specific decisions, plans, or actions. Some respondents shared that they lacked time and resources to use results, though it has been noted that moving from conceptual to instrumental use of information can partly be a factor of project maturity (Ferguson et al. 2022), and thus it is possible that revisiting respondents after additional time has passed may reveal stronger instrumental use of information. 4.4.3 Impacts: building of relationships and trust Questions in this section of the survey were designed to examine the impacts of participating in a project to the building of relationships and trust between stakeholders and investigators. Respondents reported positive feelings overall about their engagement in South Central and North Central CASC projects. 93.6 percent of respondents felt satisfied with their experiences with the investigator team, while 87.2 percent of respondents felt satisfied with their experiences with the project. All respondents agreed that investigators were honest, sincere, and trustworthy. 91.5 percent of respondents agreed that investigators appreciated and respected what they brought to the project. The same number of respondents agreed that investigators were committed to the engagement process. 89.4 percent of respondents agreed that the investigators took their opinion seriously in the course of discussions. Furthermore, all respondents said it was likely that they would use additional results generated by this investigator team. These results indicate that stakeholders still felt goodwill towards people as individuals, even though no stakeholders used project outputs to make adaptation decisions, falling short of the funding program’s project objectives. Respondents provided a range of reasons that would make it likely for them to work with the investigators again in the future (Figure 4.2). Many respondents mentioned the nature of their relationship as a team, citing a desire to work with “good people” where the “collaborative spirit and tone of mutual respect is great.” In addition to a positive team atmosphere, respondents mentioned the level of expertise of investigators, with one respondent pithily capturing this as “they know their shit.” One project investigator was identified as an “outstanding scientist and human being,” with the respondent adding that “[their] humility despite [their] great knowledge and intellect is inspiring.” Finally, respondents mentioned the importance of the relevance of findings, such as the “ability to provide useful products” and “good, practical, implementable results that were directly applicable to my agency's goals and strategies.” When provided the opportunity to give any other feedback on their experience, several respondents noted their appreciation for the integration into the project of informal knowledge or results. One respondent highlighted the investigators’ “willingness to more readily recognize and respond to non-peer reviewed (nascent) local research,” and another acknowledged that investigators were willing to implement “a demonstration project” for local stakeholders. A third respondent stated that they valued support for a project “that was not firmly deliverables based” because one of the main outputs was the creation of a collaborative network of individuals. 4.5 Discussion In this paper, we summarized a variety of approaches from the discipline of evaluation and described their relevance to defining success and evaluating the societal impacts of actionable science within the context of climate adaptation research. We presented a case study to demonstrate how to operationalize selected approaches from this literature and to evaluate the societal impact of the science produced by projects funded by the South Central and North Central CASCs. This case study was carried out via a survey of stakeholders engaged in those projects. Since virtually all respondents indicated satisfaction with projects and investigators, our ability to contrast projects and interpret differences between them was limited. Additionally, our case study was limited by the low survey response rate and relatively small sample size. As a result, although we weren’t able to use the collected data the way in which we originally intended, the sample size was still robust for examining characteristics of investigators and projects that stakeholders found satisfactory (Couper et al. 2001; Taherdoost 2017). Describing these characteristics allowed us to meet our intended objectives and provided lessons learned from early projects that can be applied to subsequent similar projects. Our results corroborate previous studies that have demonstrated that stakeholders prefer being engaged in projects early, often, and consistently (Ferguson et al. 2022; Steger et al. 2021). Previous research has shown that stakeholders may become fatigued or stressed with interactions that do not result in perceptible changes to the research agenda to prioritize stakeholder benefits (Clark 2008; Young et al. 2020). Our findings show that even interacting with investigators more than once a week was perceived as satisfactory and not as too much interaction, reinforcing the idea that stakeholder fatigue is not an issue within a project when there is an obvious connection between the reason for the interaction and a benefit to the stakeholder. While stakeholders left these interactions feeling better informed by the knowledge and outputs produced by projects, they could not connect this information for use in making specific adaptation decisions. Even so, stakeholders placed value on participation in these projects due to the relational benefits that they gained such as growing their professional network and conversing with scientific experts in informal settings. Importantly, in carrying out this case study, we learned that although surveys such as ours are an increasingly common way for programs to evaluate the societal impact of their activities, our results did not actually get us to that goal. Even though we took care to design a single evaluation process that built on appropriate theory, methods, and survey design, we discovered that each project came with its own unique objective regarding societal impact, which ideally needed an individually tailored evaluand and measures. Attempting to generate a summative “one size fits all” survey for such a broad set of objectives was inadequate and resulted in an ineffective evaluation, even if it helped identify characteristics of satisfactory projects. We recommend that future initiatives to examine societal impact for the CASCs, and other similar funding programs, consider that evaluation for each project be integrated up front into proposal development, such as asking investigators to create a logic model with measurable attributes. Such an approach would ensure that subsequent project evaluations would then be designed with specific purpose in mind and could ameliorate the issue of a low response rate. However, this additional request for inclusion of evaluation design and implementation can only be met with a matching provision of additional resources from funders. One unintended benefit of this study was that it fed into the broader conversation across the regional CASCs about whether it was possible to quantitatively measure the societal impacts of research projects that they funded. Since development and dissemination of this survey protocol, the Southeast CASC has carried out additional quantitative and qualitative research from which findings are still emerging. To date, their evaluation initiative has described the differing ways in which individuals and organizations use climate adaptation science (Courtney et al. in review) and the distinct pathways which projects that aim for societal impact must follow in comparison to projects that aim for high scientific impact (Hyman et al. in review). These network-wide conversations are a continued effort to apply concepts from evaluation theory and practice to the challenge of funding and conducting research that results in real societal impact. 4.6 Conclusion Evaluation is a critical component of understanding the societal impact of climate adaptation science that is intended to be actionable, yet many existing approaches fall short of achieving this goal. We set out to do program-wide evaluation at the project-level by creating a single survey instrument. While analyzing our data, we noted that the diverse array of project objectives meant that the single overarching survey did not contain enough nuance to evaluate individual projects. Instead, each project needed a tailored measurement tool that was developed with its unique objectives in mind. For example, we found that our survey could not capture the differing definitions of success between place-based projects for targeted stakeholders and projects producing large, regional-scale products for many stakeholders. Nor could our survey capture the differences between projects designed to build relationships and trust between people and those designed to provide context for making a specific decision. This study demonstrates the limitations of a program summatively evaluating projects and that evaluation needs to be embedded in each project from the start. In particular, funders of science can encourage applicants to proactively consider evaluation during proposal development and provide the resources to bring in relevant and necessary evaluation expertise to an investigator team. Geography, specifically, can highlight the nuanced role of place, space, and scale in defining success for actionable science and add to the broader understanding of how to better do actionable science in support of climate adaptation work. 4.7 Acknowledgments This work was funded by the U.S. Geological Survey South Central and North Central Climate Adaptation Science Centers. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. We thank the members of ABD's dissertation committee, the Climate Adaptation Science Center Evaluation Working Group, and the Science of Actionable Knowledge Working Group for insightful discussions around what constitutes successful actionable science for climate adaptation. Table 4.1 - Response rate information for each region South Central CASC North Central CASC # solicited 186 188 # total complete responses 24 (12.9 percent) 25 (13.3 percent) Table 4.2 - Demographics of respondents “Other” self-identified as part of a “federally supported partnership.” Local State Fed Tribal Agency University or college NGO or private Other Resource manager/ decision maker/ planner 12 0 8 1 Scientist/ technician/ researcher 2 12 4 0 Equally both 5 0 2 0 Figure 4.1 - Word cloud of data provided by projects Word cloud generated from forty open-ended responses to the question “What kinds of information, data, tools, or other products did this project provide you?” Figure 4.2 - Word cloud of reasons why respondents would collaborate again Word cloud generated from thirty seven open-ended responses to the question “From your perspective, what reason(s) would make it likely for you to work with this investigator team or the CSC again in the future?” CH. 5 - Conclusions 5.1 Summary This dissertation improves practices around the inclusion of stakeholders in the production of actionable climate adaptation science by examining their experiences and perspectives. Previous research has shown that research projects that include a component of stakeholder engagement can lead directly to societal impact (Ferguson et al. 2022). However, if engagement is done poorly and fails to accurately elevate the decision context, then, at best, the information produced is useless for decision making and, at worst, it can lead to counterproductive decision making and maladaptation (Barnett and O’Neill 2010). As a result, some of the key motivating questions of this dissertation included: Why and how are researchers engaging with stakeholders? What is the impact of these interactions and who benefits from them? How will researchers know when they’ve achieved their goal of societal impact? Examination of each motivating question emphasized the need to intentionally match approaches for engagement and evaluation with the specific project context at hand. In Chapter 2, I synthesized literature across a range of approaches to stakeholder engagement, and I presented this information as a framework for the production of actionable science. I described each approach in terms of both a project’s scientific objectives as well as its desired societal impact. I included barriers to implementation that might inform whether an approach would be feasible to carry out given a project’s specific resource or contextual constraints. Importantly, I also provided guiding thought questions to help researchers use the framework to support their selection of an approach and creation of a stakeholder engagement plan. These questions were designed so that working through them does not result in one “right” answer; rather, they encourage researchers to consider tradeoffs across approaches and determine what is “right” for their project. Additionally, this framework provides researchers, funders, and practitioners with a common language for demonstrating the importance of both carrying out and allocating funding towards stakeholder engagement activities. In Chapter 3, I explored the perspectives of fifteen highly engaged stakeholders on the benefits and harms they experienced while engaging in actionable science projects. Most stakeholder engagement (research with people) does not fall under human subjects research oversight (research on people) and does not obtain ethical review from entities such as an Institutional Review Board. I organized the results using operationalized definitions of the three Belmont Principles (respect for persons, beneficence, and justice) and found that stakeholder engagement that did not align with these principles resulted in harms to those engaged. For example, some stakeholders mentioned that they felt that they were included by researchers as proposal partners during the process of applying for funding but that their needs were deprioritized after funding was obtained. These results demonstrated that researchers must reflect on the ethics of stakeholder engagement and proactively ensure that stakeholders receive benefits and do not experience harms when being engaged in a project. In Chapter 4, I used theory and practice from the discipline of evaluation to develop a survey tool to examine the societal impact of actionable climate adaptation projects. I deployed the tool in a case study of climate adaptation projects, and I used the data to summarize the characteristics of projects considered successful by stakeholders engaged in the research. While these findings can help broadly inform how future projects might be carried out, I found that using a single survey instrument did not capture the detailed differences between types of projects, limiting my ability to conduct intercomparison and more nuanced interpretation of the data. Together, these three chapters contribute to our existing geographical knowledge of how researchers and decision makers are navigating complex environmental challenges in the Anthropocene. Such knowledge is also being produced and shared in other disciplinary and transdisciplinary venues (e.g., climate services, World Meteorological Organization 2014; action research, Brydon-Miller et al. 2003) with limited exchange in and out of geography. The work presented in this dissertation connects geographic knowledge with this broader community, amplifying scholarly contributions from geography and enriching the overall dialogue. Moreover, this work adds to critical physical geography scholarship by applying a critical lens to publicly funded climate adaptation research and encouraging reflexive thinking in biological and physical scientists conducting this research. 5.2 Recommendations Based on these findings, I recommend that researchers doing actionable science must carefully consider the goals for societal impact and the design of their stakeholder engagement as early in the project development process as possible, selecting approaches for fit instead of convenience. While not all researchers receive training in human subjects research or engagement practices, every individual can pause and reflect on questions such as: Why am I asking this person for their time? What will they get in return? How will I change my research plan based on their input? These reflexive practices can help ensure that researchers give as much thought to designing their engagement plan as they do to the rest of their project design, elevating consideration of engagement to a core scientific decision. Over time, as such practices become habits, new generations of researchers will begin to pause and reflect on more nuanced questions such as: Which voices are being left out of the dialogue? How does my role as an expert alter the power dynamics amongst individuals in this conversation? In addition to practicing reflexive thinking, researchers can ensure that project teams include the relevant and necessary expertise to rigorously conduct engagement and evaluate societal impact. As funders of science increasingly include requests that proposed projects address real world challenges, I recommend that they also prioritize making resources available to support additional expertise on a project, such as salary for personnel with outreach and evaluation skills. Without these additional resources, investigators must either choose to reduce the proposed scientific impact of their project or conduct engagement and evaluation work without spending much thought or effort on this work. Unfortunately, many researchers face promotion processes that place heavy weight on academic output and peer-reviewed publications, resulting in more proposals choosing to under-resource engagement and evaluation instead of reducing scientific impact. Additional resources are necessary to ensure that stakeholder engagement is conducted in an ethical manner that prevents participants from experiencing harm. 5.3 Next steps The findings in this dissertation demonstrate that intentional design of stakeholder engagement and project evaluation can facilitate better actionable science. In the course of conducting this research, I had to set aside many additional unresolved questions, leaving open the possibility for next steps and future research directions. First, from the interviews with highly engaged stakeholders, it was clear that each individual had a different definition for actionable science and what it looks like in successful practice. There is still substantial room to go beyond the work in Chapters 2 and 3 to generate even more evidence-based guidance for what successful societal impact means and how researchers should navigate engagement to reach such a goal. Second, funders of science are starting to engage with stakeholders to produce the research agendas that direct what priorities appear in solicitations, how proposals are reviewed and evaluated, and which projects are selected for funding. However, there is a paucity of research that examines the influence of such engagement on the societal impact of resulting projects, especially regarding inclusion and whether resources and results are equitably distributed. Third, there is still a significant need to describe how to operationalize definitions of success and evaluation for the actionable science research community. The work comprising Chapter 4 is a starting point that demonstrates that there needs to be additional guidance on the contexts in which different evaluation methods should optimally be deployed. Finally, while the work in this dissertation was centered on a case study of climate adaptation, future research could examine its potential application to other complex environmental challenges that need actionable science (e.g., water rights and water management, managing ecosystems for biodiversity and to meet human needs). References ACCCNRS, 2015: Report to the Secretary of the Interior. Advisory Committee on Climate Change and Natural Resource Science, https://www.sciencebase.gov/catalog/item/5c1d05d3e4b0708288c9bc2a. Archie, K. M., 2014: Mountain communities and climate change adaptation: barriers to planning and hurdles to implementation in the Southern Rocky Mountain Region of North America. Mitig Adapt Strateg Glob Change, 19, 569–587, https://doi.org/10.1007/s11027-013-9449-z. Archie, K. M., L. Dilling, J. B. Milford, and F. C. Pampel, 2014: Unpacking the ‘information barrier’: Comparing perspectives on information as a barrier to climate change adaptation in the interior mountain West. Journal of Environmental Management, 133, 397–410, https://doi.org/10.1016/j.jenvman.2013.12.015. Ardener, S., 2005: Ardener’s “Muted Groups”: The genesis of an idea and its praxis. Women and Language: WL, 28, 50-54,72. Ardoin, N. M., K. Biedenweg, and K. O’Connor, 2015: Evaluation in Residential Environmental Education: An Applied Literature Review of Intermediary Outcomes. Applied Environmental Education & Communication, 14, 43–56, https://doi.org/10.1080/1533015x.2015.1013225. Arguez, A., I. Durre, S. Applequist, R. S. Vose, M. F. Squires, X. Yin, R. R. Heim, and T. W. Owen, 2012: NOAA’s 1981–2010 U.S. Climate Normals: An Overview. Bulletin of the American Meteorological Society, 93, 1687–1697, https://doi.org/10.1175/BAMS-D-11-00197.1. Arnott, J. C., 2021: Pens and purse strings: Exploring the opportunities and limits to funding actionable sustainability science. Research Policy, 50, 104362, https://doi.org/10.1016/j.respol.2021.104362. Arnott, J. C., C. J. Kirchhoff, R. M. Meyer, A. M. Meadow, and A. T. Bednarek, 2020a: Sponsoring actionable science: what public science funders can do to advance sustainability and the social contract for science. Current Opinion in Environmental Sustainability, 42, 38–44, https://doi.org/10.1016/j.cosust.2020.01.006. ——, K. J. Mach, and G. Wong-Parodi, 2020b: Editorial overview: The science of actionable knowledge. Current Opinion in Environmental Sustainability, 42, A1–A5, https://doi.org/10.1016/j.cosust.2020.03.007. Arnstein, S. R., 1969: A Ladder Of Citizen Participation. Journal of the American Institute of Planners, 35, 216–224, https://doi.org/10.1080/01944366908977225. Averyt, K., and Coauthors, 2017: Regional Climate Response Collaboratives: Multi-institutional Support for Climate Resilience. Bulletin of the American Meteorological Society, 99, 891–898, https://doi.org/10.1175/BAMS-D-17-0183.1. Bamzai, A., 2007: Global Changes in Precipitation Extremes. https://dukespace.lib.duke.edu/dspace/handle/10161/366 (Accessed April 1, 2022). Bamzai, P. N. K., 1994: Culture and Political History of Kashmir. Volume 3: Modern Kashmir. M.D. Publications,. Bamzai-Dodson, A., A. E. Cravens, A. Wade, and R. A. McPherson, 2021: Engaging with stakeholders to produce actionable science: a framework and guidance. Weather, Climate, and Society, 13, 1027–1041, https://doi.org/10.1175/WCAS-D-21-0046.1. ——, J. Lackett, and R. A. McPherson, 2022: CASC Project Evaluation Survey Template. Barcus, H. R., and D. Trudeau, 2018: Introduction to Focus Section: Out in the World: Geography’s Complex Relationship with Civic Engagement. The Professional Geographer, 70, 270–276, https://doi.org/10.1080/00330124.2017.1365307. Barnett, J., and S. O’Neill, 2010: Maladaptation. Global Environmental Change, 20, 211–213, https://doi.org/10.1016/j.gloenvcha.2009.11.004. Barry, J., and M. Thompson-Fawcett, 2020: Decolonizing the Boundaries between the ‘Planner’ and the ‘Planned’: Implications of Indigenous Property Development. Planning Theory & Practice,. Bartlett, C., M. Marshall, and A. Marshall, 2012: Two-Eyed Seeing and other lessons learned within a co-learning journey of bringing together indigenous and mainstream knowledges and ways of knowing. J Environ Stud Sci, 2, 331–340, https://doi.org/10.1007/s13412-012-0086-8. Bathke, D., T. Haigh, T. Bernadt, and N. Wall, 2019: Drought Scenario-Based Exercises. National Drought Mitigation Center, https://digitalcommons.unl.edu/ndmcpub/19/. Baylor, R., and Coauthors, 2019: Implementing Developmental Evaluation: A Practical Guide for Evaluators and Administrators. U.S. Agency for International Development, https://www.usaid.gov/GlobalDevLab/MERLIN/DEPA-MERL/uptake-developmental-evaluation-de/practical-guide-evaluators-administrators (Accessed December 23, 2020). Beaury, E. M., E. J. Fusco, M. R. Jackson, B. B. Laginhas, T. L. Morelli, J. M. Allen, V. J. Pasquarella, and B. A. Bradley, 2020: Incorporating climate change into invasive species management: insights from managers. Biol Invasions, 22, 233–252, https://doi.org/10.1007/s10530-019-02087-6. Beck, J. M., K. C. Elliott, C. R. Booher, K. A. Renn, and R. A. Montgomery, 2021: The application of reflexivity for conservation science. Biological Conservation, 262, 109322, https://doi.org/10.1016/j.biocon.2021.109322. Bednarek, A. T., and Coauthors, 2018: Boundary spanning at the science–policy interface: the practitioners’ perspectives. Sustain Sci, 13, 1175–1183, https://doi.org/10.1007/s11625-018-0550-9. Beeton, T. A., S. M. McNeeley, B. W. Miller, and D. S. Ojima, 2019: Grounding simulation models with qualitative case studies: Toward a holistic framework to make climate science usable for US public land management. Climate Risk Management, 23, 50–66, https://doi.org/10.1016/j.crm.2018.09.002. Beier, P., L. Hansen, L. Helbrecht, and D. Behar, 2017: A How-to Guide for Coproduction of Actionable Science. Conservation Letters, 10, 288–296, https://doi.org/10.1111/conl.12300. Beierle, T., 2002: The Quality of Stakeholder‐Based Decisions. Risk Analysis, 22, 739–749, https://doi.org/doi:10.1111/0272-4332.00065. Bennett, N. J., and Coauthors, 2017: Conservation social science: Understanding and integrating human dimensions to improve conservation. Biological Conservation, 205, 93–108, https://doi.org/10.1016/j.biocon.2016.10.006. Bierbaum, R., and Coauthors, 2013: A comprehensive review of climate adaptation in the United States: more than before, but less than needed. Mitig Adapt Strat Gl, 18, 361–406, https://doi.org/10.1007/s11027-012-9423-1. Bisbal, G. A., 2019: Practical tips to establish an actionable science portfolio for climate adaptation. Sci Public Policy, 46, 148–153, https://doi.org/10.1093/scipol/scy070. Block, D. R., E. Hague, W. Curran, and H. Rosing, 2018: Measuring Community and University Impacts of Critical Civic Geography: Insights from Chicago. The Professional Geographer, 70, 284–290, https://doi.org/10.1080/00330124.2017.1366777. Bojovic, D., A. L. St. Clair, I. Christel, M. Terrado, P. Stanzel, P. Gonzalez, and E. J. Palin, 2021: Engagement, involvement and empowerment: Three realms of a coproduction framework for climate services. Global Environmental Change, 68, 102271, https://doi.org/10.1016/j.gloenvcha.2021.102271. Boschetti, F., C. Cvitanovic, A. Fleming, and E. Fulton, 2016: A call for empirically based guidelines for building trust among stakeholders in environmental sustainability projects. Sustain Sci, 11, 855–859, https://doi.org/10.1007/s11625-016-0382-4. Boström, M., R. Lidskog, and Y. Uggla, 2017: A reflexive look at reflexivity in environmental sociology. Environmental Sociology, 3, 6–16, https://doi.org/10.1080/23251042.2016.1237336. Boy, G. A., 2017: The Handbook of Human-Machine Interaction: A Human-Centered Design Approach. CRC Press,. Bracken, L. J., H. A. Bulkeley, and G. Whitman, 2015: Transdisciplinary research: understanding the stakeholder perspective. Journal of Environmental Planning and Management, 58, 1291–1308, https://doi.org/10.1080/09640568.2014.921596. Bradbury, H., S. Waddell, K. O’ Brien, M. Apgar, B. Teehankee, and I. Fazey, 2019: A call to Action Research for Transformations: The times demand it. Action Research, 17, 3–10, https://doi.org/10.1177/1476750319829633. Brandt, A. M., 1978: Racism and Research: The Case of the Tuskegee Syphilis Study. The Hastings Center Report, 8, 21–29, https://doi.org/10.2307/3561468. Bremer, S., and S. Meisch, 2017: Co-production in climate change research: reviewing different perspectives. Wiley Interdisciplinary Reviews: Climate Change, 8, e482-n/a, https://doi.org/10.1002/wcc.482. Briley, L., D. Brown, and S. E. Kalafatis, 2015: Overcoming barriers during the co-production of climate information for decision-making. Climate Risk Management, 9, 41–49, http://dx.doi.org/10.1016/j.crm.2015.04.004. Brittain, S., and Coauthors, 2020: Ethical considerations when conservation research involves people. Conservation Biology, 34, 925–933, https://doi.org/10.1111/cobi.13464. Brydon-Miller, M., D. Greenwood, and P. Maguire, 2003: Why Action Research? Action Research, 1, 9–28, https://doi.org/10.1177/14767503030011002. Bush, V., 1945: Science, the endless frontier: a report to the President. U.S. Government Printing Office, https://www.nsf.gov/about/history/vbush1945.htm. Butler, D. R., 2021: The Anthropocene: A Special Issue. Annals of the American Association of Geographers, 111, 633–637, https://doi.org/10.1080/24694452.2020.1859312. Buxton, R. T., and Coauthors, 2021: Avoiding wasted research resources in conservation science. Conservation Science and Practice, 3, e329, https://doi.org/10.1111/csp2.329. Carney, S., L. Whitmarsh, S. A. Nicholson-Cole, and S. Shackley, 2009: A dynamic typology of stakeholder engagement within climate change research. Tyndall Center for Climate Change Research, Working Paper, 128. Carter, S. K., and Coauthors, 2020: Bridging the research-management gap: landscape science in practice on public lands in the western United States. Landscape Ecology, 35, 545–560, https://doi.org/10.1007/s10980-020-00970-5. Cash, D., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, and J. Jäger, 2002: Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making. KSG Working Papers Series RWP02-046,. Cash, D. W., J. C. Borck, and A. G. Patt, 2006: Countering the Loading-Dock Approach to Linking Science and Decision Making: Comparative Analysis of El Niño/Southern Oscillation (ENSO) Forecasting Systems. Science, Technology, & Human Values, 31, 465–494, https://doi.org/10.1177/0162243906287547. Charnley, S., and Coauthors, 2017: Evaluating the best available social science for natural resource management decision-making. Environmental Science & Policy, 73, 80–88, https://doi.org/10.1016/j.envsci.2017.04.002. Chief, K., and Coauthors, 2015: Guidelines for Considering Traditional Knowledges in Climate Change Initiatives. Social Science Research Network,. Christel, I., D. Hemment, D. Bojovic, F. Cucchietti, L. Calvo, M. Stefaner, and C. Buontempo, 2018: Introducing design in the development of effective climate services. Climate Services, 9, 111–121, https://doi.org/10.1016/j.cliser.2017.06.002. Clark, T., 2008: “We’re Over-Researched Here!”: Exploring Accounts of Research Fatigue within Qualitative Research Engagements. Sociology, 42, 953–970, https://doi.org/10.1177/0038038508094573. ——, 2010: On ‘being researched’: why do people engage with qualitative research? Qualitative Research, 10, 399–419, https://doi.org/10.1177/1468794110366796. Clifford, K. R., W. R. Travis, and L. T. Nordgren, 2020: A climate knowledges approach to climate services. Climate Services, 100155, https://doi.org/10.1016/j.cliser.2020.100155. Cloyd, E., S. C. Moser, E. Maibach, J. Maldonado, and T. Chen, 2016: Engagement in the Third U.S. National Climate Assessment: commitment, capacity, and communication for impact. Climatic Change, 135, 39–54, https://doi.org/10.1007/s10584-015-1568-y. Cockburn, J., and G. Cundill, 2018: Ethics in Transdisciplinary Research: Reflections on the Implications of ‘Science with Society.’ The Palgrave Handbook of Ethics in Critical Research, C.I. Macleod, J. Marx, P. Mnyaka, and G.J. Treharne, Eds., Springer International Publishing, 81–97. Colavito, M. M., S. F. Trainor, N. P. Kettle, and A. York, 2019: Making the Transition from Science Delivery to Knowledge Coproduction in Boundary Spanning: A Case Study of the Alaska Fire Science Consortium. Wea. Climate Soc., 11, 917–934, https://doi.org/10.1175/WCAS-D-19-0009.1. Collins, C. S., and C. M. Stockton, 2018: The Central Role of Theory in Qualitative Research. International Journal of Qualitative Methods, 17, 1609406918797475, https://doi.org/10.1177/1609406918797475. Colven, E., and M. J. Thomson, 2019: Bridging the divide between human and physical geography: Potential avenues for collaborative research on climate modeling. Geography Compass, 13, e12418, https://doi.org/10.1111/gec3.12418. Corbin, J., and A. Strauss, 2008: Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory. 3rd edition. SAGE Publications,. Coryn, C. L. S., J. A. Hattie, M. Scriven, and D. J. Hartmann, 2007: Models and Mechanisms for Evaluating Government-Funded Research. American Journal of Evaluation, 28, 437–457, https://doi.org/10.1177/1098214007308290. Couper, M. P., M. W. Traugott, and M. J. Lamias, 2001: Web Survey Design and Administration. Public Opinion Quarterly, 65, 230–253, https://doi.org/10.1086/322199. Courtney, S., A. Hyman, K. S. McNeal, L. C. Maudlin, and P. Armsworth, in review: Development of a survey instrument to assess individual and organizational use of climate adaptation science. Environmental Science & Policy,. Cozzens, S. E., 1997: The knowledge pool: Measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning, 20, 77–89, https://doi.org/10.1016/S0149-7189(96)00038-9. Craig, P., P. Dieppe, S. Macintyre, S. Michie, I. Nazareth, and M. Petticrew, 2008: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ, 337, a1655, https://doi.org/10.1136/bmj.a1655. Crausbay, S. D., and Coauthors, 2020: Unfamiliar Territory: Emerging Themes for Ecological Drought Research and Management. One Earth, 3, 337–353, https://doi.org/10.1016/j.oneear.2020.08.019. Cravens, A., and N. Ardoin, 2016: Negotiating credibility and legitimacy in the shadow of an authoritative data source. Ecology and Society, 21, https://doi.org/10.5751/ES-08849-210430. Crutzen, P. J., 2006: The “Anthropocene.” Earth System Science in the Anthropocene, E. Ehlers and T. Krafft, Eds., Springer, 13–18. Daniels, E., S. Bharwani, Å. Gerger Swartling, G. Vulturius, and K. Brandon, 2020: Refocusing the climate services lens: Introducing a framework for co-designing “transdisciplinary knowledge integration processes” to build climate resilience. Climate Services, 19, 100181, https://doi.org/10.1016/j.cliser.2020.100181. David-Chavez, D. M., and M. C. Gavin, 2018: A global assessment of Indigenous community engagement in climate research. Environ. Res. Lett., 13, 123005, https://doi.org/10.1088/1748-9326/aaf300. Davidson, S., 1998: Spinning the wheel of empowerment. Planning, 1262, 14–15. DeCrappeo, N. M., G. A. Bisbal, and A. M. Meadow, 2017: A Path to Actionable Climate Science: Perspectives from the Field. Environmental Management, 61, 181–187, https://doi.org/10.1007/s00267-017-0960-y. Dilling, L., and M. C. Lemos, 2011: Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environmental Change, 21, 680–689, http://dx.doi.org/10.1016/j.gloenvcha.2010.11.006. ——, K. Lackstrom, B. Haywood, K. Dow, M. C. Lemos, J. Berggren, and S. Kalafatis, 2015: What Stakeholder Needs Tell Us about Enabling Adaptive Capacity: The Intersection of Context and Information Provision across Regions in the United States. Weather Clim Soc, 7, 5–17, https://doi.org/10.1175/Wcas-D-14-00001.1. ——, M. C. Lemos, and N. Singh, 2021: Commentary: First, do no harm: Scaling usable knowledge for just and equitable outcomes. Global Environmental Change, 71, 102404, https://doi.org/10.1016/j.gloenvcha.2021.102404. Diver, S., 2017: Negotiating Indigenous knowledge at the science-policy interface: Insights from the Xáxli’p Community Forest. Environmental Science & Policy, 73, 1–11, https://doi.org/10.1016/j.envsci.2017.03.001. ——, and M. Higgins, 2014: View of Giving Back Through Collaborative Research: Towards a Practice of Dynamic Reciprocity. Journal of Research Practice, 10. Doemeland, D., and J. Trevino, 2014: Which World Bank reports are widely read ? The World Bank, http://documents.worldbank.org/curated/en/387501468322733597/Which-World-Bank-reports-are-widely-read (Accessed May 22, 2020). Donaldson, S. I., 2007: Program theory-driven evaluation science: strategies and applications. Lawrence Erlbaum Associates,. Doswell, C. A., A. R. Moller, and H. E. Brooks, 1999: Storm Spotting and Public Awareness since the First Tornado Forecasts of 1948. Wea. Forecasting, 14, 544–557, https://doi.org/10.1175/1520-0434(1999)014<0544:SSAPAS>2.0.CO;2. Durose, C., B. Perry, L. Richardson, and R. Dean, 2021: Leadership and the hidden politics of co-produced research: a Q-methodology study. International Journal of Social Research Methodology, 0, 1–21, https://doi.org/10.1080/13645579.2021.1960738. Enquist, C., and Coauthors, 2017: Foundations of translational ecology. Frontiers in Ecology and the Environment, 15, 541–550, https://doi.org/10.1002/fee.1733. Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor, 2016: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geoscientific Model Development, 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016. Ferguson, D. B., A. M. Meadow, and H. P. Huntington, 2022: Making a Difference: Planning for Engaged Participation in Environmental Research. Environmental Management, https://doi.org/10.1007/s00267-021-01585-5. Ferraro, K. M., A. L. Ferraro, and N. R. Sommer, 2021: Challenges facing cross-disciplinary collaboration in conservation ethics. Conservation Science and Practice, n/a, e523, https://doi.org/10.1111/csp2.523. Few, R., K. Brown, and E. Tompkins, 2007: Public participation and climate change adaptation: avoiding the illusion of inclusion. Climate Policy, 7, 46–59, https://doi.org/10.1080/14693062.2007.9685637. Filho, W. L., ed., 2015: Handbook of Climate Change Adaptation. 1st ed. Springer-Verlag Berlin Heidelberg,. Findlater, K., S. Webber, M. Kandlikar, and S. Donner, 2021: Climate services promise better decisions but mainly focus on better data. Nat. Clim. Chang., 11, 731–737, https://doi.org/10.1038/s41558-021-01125-3. Fiorino, D. J., 1990: Citizen Participation and Environmental Risk: A Survey of Institutional Mechanisms. Science, Technology, & Human Values, 15, 226–243, https://doi.org/10.1177/016224399001500204. Fisher, J. R. B., S. A. Wood, M. A. Bradford, and T. R. Kelsey, 2020: Improving scientific impact: How to practice science that influences environmental policy and management. Conservation Science and Practice, 2, e210, https://doi.org/10.1111/csp2.210. Flyvbjerg, B., 2006: Five Misunderstandings About Case-Study Research. Qualitative Inquiry, 12, 219–245, https://doi.org/10.1177/1077800405284363. Ford, J. D., M. Knight, and T. Pearce, 2013: Assessing the ‘usability’ of climate change research for decision-making: A case study of the Canadian International Polar Year. Global Environmental Change, 23, 1317–1326, https://doi.org/10.1016/j.gloenvcha.2013.06.001. Fowler, F. J., 1995: Improving Survey Questions: Design and Evaluation. SAGE Publications,. Friesen, P., L. Kearns, B. Redman, and A. L. Caplan, 2017: Rethinking the Belmont Report? The American Journal of Bioethics, 17, 15–21, https://doi.org/10.1080/15265161.2017.1329482. ——, E. Doerksen, and A. Gunay, 2021: Solidarity without Sovereignty: Extending the Belmont Principles Further? The American Journal of Bioethics, 21, 25–27, https://doi.org/10.1080/15265161.2021.1965250. Fyfe, A., K. Coate, S. Curry, S. Lawson, N. Moxham, and C. M. Røstvik, 2017: Untangling Academic Publishing: A history of the relationship between commercial interests, academic prestige and the circulation of research. Zenodo,. Gasson, S., 2003: Human-Centered vs. User-Centered Approaches to Information System Design. Journal of Information Technology Theory and Application (JITTA), 5. Gerlak, A. K., T. Heikkila, and J. Newig, 2020: Learning in environmental governance: opportunities for translating theory to practice. Journal of Environmental Policy & Planning, 22, 653–666, https://doi.org/10.1080/1523908X.2020.1776100. Gluckman, P., and J. Wilsdon, 2016: From paradox to principles: where next for scientific advice to governments? Palgrave Communications, 2, 1–4, https://doi.org/10.1057/palcomms.2016.77. Golden, N., K. Devarajan, C. Balantic, J. Drake, M. T. Hallworth, and T. L. Morelli, 2021: Ten simple rules for productive lab meetings. PLOS Computational Biology, 17, e1008953, https://doi.org/10.1371/journal.pcbi.1008953. Goodrich, K. A., K. D. Sjostrom, C. Vaughan, L. Nichols, A. Bednarek, and M. C. Lemos, 2020: Who are boundary spanners and how can we support them in making knowledge more actionable in sustainability fields? Current Opinion in Environmental Sustainability, 42, 45–51, https://doi.org/10.1016/j.cosust.2020.01.001. Guido, Z., D. Hill, M. Crimmins, and D. Ferguson, 2013: Informing Decisions with a Climate Synthesis Product: Implications for Regional Climate Services. Weather Clim Soc, 5, 83–92, https://doi.org/10.1175/wcas-d-12-00012.1. Hakkarainen, V., K. Mäkinen-Rostedt, A. Horcea-Milcu, D. D’Amato, J. Jämsä, and K. Soini, 2021: Transdisciplinary research in natural resources management: Towards an integrative and transformative use of co-concepts. Sustainable Development, 1–17, https://doi.org/10.1002/sd.2276. Hampton, C., D. Reeping, and D. S. Ozkan, 2021: Positionality Statements in Engineering Education Research: A Look at the Hand that Guides the Methodological Tools. Studies in Engineering Education, 1, 126–141, https://doi.org/10.21061/see.13. Hansen, H. F., 2005: Choosing Evaluation Models. Evaluation, 11, 447–462, https://doi.org/doi:10.1177/1356389005060265. Hassol, S. J., 2008: Improving How Scientists Communicate About Climate Change. Eos, 89, 106–107, https://doi.org/10.1029/2008EO110002. Hegger, D., M. Lamers, A. V. Zeijl-Rozema, and C. Dieperink, 2012: Conceptualising joint knowledge production in regional climate change adaptation projects: success conditions and levers for action. Environmental Science & Policy, 18, 52–65, https://doi.org/10.1016/j.envsci.2012.01.002. ——, A. V. Zeijl-Rozema, and C. Dieperink, 2014: Toward design principles for joint knowledge production projects: lessons from the deepest polder of The Netherlands. Reg Environ Change, 14, 1049–1062, https://doi.org/10.1007/s10113-012-0382-6. Holmes, A. G. D., 2020: Researcher Positionality - A Consideration of Its Influence and Place in Qualitative Research - A New Researcher Guide. Shanlax International Journal of Education, 8, 1–10, https://doi.org/10.34293/education.v8i4.3232. Human, B. A., and A. Davies, 2010: Stakeholder consultation during the planning phase of scientific programs. Marine Policy, 34, 645–654, https://doi.org/10.1016/j.marpol.2009.12.003. Hyman, A., S. Courtney, K. S. McNeal, L. Bialic-Murphy, C. Furiness, and P. Armsworth, in review: Distinct pathways to stakeholder use versus scientific impact in climate adaptation research. Conservation Letters,. IDEO.org, 2015: The Field Guide to Human-Centered Design. 1st edition. IDEO.org / Design Kit, 192 pp. Imenda, S., 2014: Is There a Conceptual Difference between Theoretical and Conceptual Frameworks? Journal of Social Sciences, 38, 185–195, https://doi.org/10.1080/09718923.2014.11893249. International Military Tribunal, 1949: The Nuremberg Code. U.S. Government Printing Office,. IPCC, 2018: Summary for Policymakers. Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, sustainable development, and efforts to eradicate poverty, V. Masson-Delmotte et al., Eds. ——, 2022: Climate Change 2022: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. H.O. Pörtner et al., Eds. Cambridge University Press,. Jagannathan, K., J. C. Arnott, C. Wyborn, N. Klenk, K. J. Mach, R. H. Moss, and K. D. Sjostrom, 2020a: Great expectations? Reconciling the aspiration, outcome, and possibility of co-production. Current Opinion in Environmental Sustainability, 42, 22–29, https://doi.org/10.1016/j.cosust.2019.11.010. ——, A. D. Jones, and I. Ray, 2020b: The making of a metric: Co-producing decision-relevant climate science. Bull. Amer. Meteor. Soc., https://doi.org/10.1175/BAMS-D-19-0296.1. Jansen, T., L. Claassen, I. van Kamp, and D. R. M. Timmermans, 2019: Understanding of the concept of ‘uncertain risk’. A qualitative study among different societal groups. Journal of Risk Research, 22, 658–672, https://doi.org/10.1080/13669877.2018.1503614. Jantarasami, L., J. Lawler, and C. Thomas, 2010: Institutional Barriers to Climate Change Adaptation in U.S. National Parks and Forests. Ecology and Society, 15, https://doi.org/10.5751/ES-03715-150433. Jarvis, R. M., and Coauthors, 2020: Navigating spaces between conservation research and practice: Are we making progress? Ecological Solutions and Evidence, 1, e12028, https://doi.org/10.1002/2688-8319.12028. Jasanoff, S., ed., 2004: States of Knowledge: The Co-production of Science and the Social Order. Routledge,. Johnson, C. J., and M. P. Gillingham, 2004: Mapping uncertainty: sensitivity of wildlife habitat ratings to expert opinion. Journal of Applied Ecology, 41, 1032–1041, https://doi.org/10.1111/j.0021-8901.2004.00975.x. Kalafatis, S. E., M. C. Lemos, Y.-J. Lo, and K. A. Frank, 2015: Increasing information usability for climate adaptation: The role of knowledge networks and communities of practice. Global Environmental Change, 32, 30–39, https://doi.org/10.1016/j.gloenvcha.2015.02.007. Karcher, D. B., C. Cvitanovic, R. M. Colvin, I. E. van Putten, and M. S. Reed, 2021: Is this what success looks like? Mismatches between the aims, claims, and evidence used to demonstrate impact from knowledge exchange processes at the interface of environmental science and policy. Environmental Science & Policy, 125, 202–218, https://doi.org/10.1016/j.envsci.2021.08.012. Katz, J., 1996: The Nuremberg Code and the Nuremberg Trial: A Reappraisal. JAMA, 276, 1662–1666, https://doi.org/10.1001/jama.1996.03540200048030. Kemp, K. B., J. J. Blades, P. Z. Klos, T. E. Hall, J. E. Force, P. Morgan, and W. T. Tinkham, 2015: Managing for climate change on federal lands of the western United States: perceived usefulness of climate science, effectiveness of adaptation strategies, and barriers to implementation. Ecology and Society, 20, 17, https://doi.org/Artn 1710.5751/Es-07522-200217. Khagram, S., and Coauthors, 2010: Thinking about knowing: conceptual foundations for interdisciplinary environmental research. Environmental Conservation, 37, 388–397, https://doi.org/10.1017/S0376892910000809. Kimmerer, R. W., 2014: Returning the Gift. Minding Nature. Kirby, C. K., C. Haruo, K. P. Whyte, J. C. Libarkin, C. Caldwell, and R. Edler, 2019: Ethical collaboration and the need for training: Partnerships between Native American Tribes and climate science organisations. Gateways: International Journal of Community Research and Engagement, 12, ID 5894-ID 5894, https://doi.org/10.5130/ijcre.v12i1.5894. Kirchhoff, C. J., M. C. Lemos, and S. Dessai, 2013: Actionable Knowledge for Environmental Decision Making: Broadening the Usability of Climate Science. Annu Rev Env Resour, 38, 393–414, https://doi.org/10.1146/annurev-environ-022112-112828. ——, R. Esselman, and D. Brown, 2015: Boundary organizations to boundary chains: Prospects for advancing climate science application. Climate Risk Management, 9, 20–29, http://dx.doi.org/10.1016/j.crm.2015.04.001. Klemm, T., and R. A. McPherson, 2018: Assessing Decision Timing and Seasonal Climate Forecast Needs of Winter Wheat Producers in the South-Central United States. Journal of Applied Meteorology and Climatology, 57, 2129–2140, https://doi.org/10.1175/JAMC-D-17-0246.1. Klenk, N., A. Fiume, K. Meehan, and C. Gibbes, 2017: Local knowledge in climate adaptation research: moving knowledge frameworks from extraction to co-production. Wiley Interdisciplinary Reviews: Climate Change, 8, e475-n/a, https://doi.org/10.1002/wcc.475. Klink, J., V. Koundinya, K. Kies, C. Robinson, A. Rao, C. Berezowitz, M. Widhalm, and L. Prokopy, 2017: Enhancing interdisciplinary climate change work through comprehensive evaluation. Climate Risk Management, 15, 109–125, https://doi.org/10.1016/j.crm.2016.11.003. Kruk, M. C., R. Vose, R. Heim, A. Arguez, J. Enloe, X. Yin, and T. Wallis, 2018: Drought Amelioration: An Engagement-to-Implementation Success Story. Bull. Amer. Meteor. Soc., 99, 2457–2462, https://doi.org/10.1175/BAMS-D-17-0177.1. Lacey, J., M. Howden, C. Cvitanovic, and R. M. Colvin, 2017: Understanding and managing trust at the climate science–policy interface. Nat Clim Change, 8, 22–28, https://doi.org/10.1038/s41558-017-0010-z. LaChapelle, P. R., S. F. McCool, and M. E. Patterson, 2003: Barriers to Effective Natural Resource Planning in a “Messy” World. Society & Natural Resources, 16, 473–490, https://doi.org/10.1080/08941920309151. Land, M., B. Macura, C. Bernes, and S. Johansson, 2017: A five-step approach for stakeholder engagement in prioritisation and planning of environmental evidence syntheses. Environmental Evidence, 6, 25, https://doi.org/10.1186/s13750-017-0104-0. Lane, S. N., C. Biermann, and R. Lave, 2018: Towards a Genealogy of Critical Physical Geography. The Palgrave Handbook of Critical Physical Geography, R. Lave, C. Biermann, and S.N. Lane, Eds., Springer International Publishing, 23–47. Larsen, T. B., and J. Harrington, 2021: Geographic Thought and the Anthropocene: What Geographers Have Said and Have to Say. Annals of the American Association of Geographers, 111, 729–741, https://doi.org/10.1080/24694452.2020.1796575. Latulippe, N., and N. Klenk, 2020: Making room and moving over: knowledge co-production, Indigenous knowledge sovereignty and the politics of global environmental change decision-making. Current Opinion in Environmental Sustainability, 42, 7–14, https://doi.org/10.1016/j.cosust.2019.10.010. Laursen, S., N. Puniwai, A. S. Genz, S. A. B. Nash, L. K. Canale, and S. Ziegler-Chong, 2018: Collaboration Across Worldviews: Managers and Scientists on Hawaiʻi Island Utilize Knowledge Coproduction to Facilitate Climate Change Adaptation. Environmental Management, 62, 619–630, https://doi.org/10.1007/s00267-018-1069-7. Lave, R., 2014: Engaging within the Academy: A Call for Critical Physical Geography. ACME: An International Journal for Critical Geographies, 13, 508–515. ——, C. Biermann, and S. N. Lane, 2018: Introducing Critical Physical Geography. The Palgrave Handbook of Critical Physical Geography, R. Lave, C. Biermann, and S.N. Lane, Eds., Springer International Publishing, 3–21. Ledingham, J. A., 2003: Explicating Relationship Management as a General Theory of Public Relations. Journal of Public Relations Research, 15, 181–198, https://doi.org/10.1207/S1532754XJPRR1502_4. ——, and S. D. Bruning, 1998: Relationship management in public relations: dimensions of an organization-public relationship. Public Relations Review, 24, 55–65, https://doi.org/10.1016/S0363-8111(98)80020-9. Lee, K. A., J. R. Lee, and P. Bell, 2020: A review of Citizen Science within the Earth Sciences: potential benefits and obstacles. Proceedings of the Geologists’ Association, https://doi.org/10.1016/j.pgeola.2020.07.010. Leitch, A. M., J. P. Palutikof, D. Rissik, S. L. Boulter, F. N. Tonmoy, S. Webb, A. C. P. Vidaurre, and M. C. Campbell, 2019: Co-development of a climate change decision support framework through engagement with stakeholders. Climatic Change, 153, 587–605, https://doi.org/10.1007/s10584-019-02401-0. Lemos, M. C., C. J. Kirchhoff, and V. Ramprasad, 2012: Narrowing the climate information usability gap. Nature Climate Change, 2, 789–794, https://doi.org/10.1038/Nclimate1614. ——, ——, S. E. Kalafatis, D. Scavia, and R. B. Rood, 2014: Moving Climate Information off the Shelf: Boundary Chains and the Role of RISAs as Adaptive Organizations. Weather Clim Soc, 6, 273–285, https://doi.org/10.1175/WCAS-D-13-00044.1. ——, and Coauthors, 2018: To co-produce or not to co-produce. Nat Sustain, 1, 722–724, https://doi.org/10.1038/s41893-018-0191-0. ——, K. S. Wolske, L. V. Rasmussen, J. C. Arnott, M. Kalcic, and C. J. Kirchhoff, 2019: The Closer, the Better? Untangling Scientist–Practitioner Engagement, Interaction, and Knowledge Use. Wea. Climate Soc., 11, 535–548, https://doi.org/10.1175/WCAS-D-18-0075.1. Lowe, P., G. Whitman, and J. Phillipson, 2009: Ecology and the social sciences. Journal of Applied Ecology, 46, 297–305, https://doi.org/10.1111/j.1365-2664.2009.01621.x. Lubchenco, J., 1998: Entering the Century of the Environment: A New Social Contract for Science. Science, 279, 491–497, https://doi.org/10.1126/science.279.5350.491. ——, 2017: Delivering On Science’s Social Contract. Michigan Journal of Sustainability, 5, http://dx.doi.org/10.3998/mjs.12333712.0005.106. Lynam, T., W. de Jong, D. Sheil, T. Kusumanto, and K. Evans, 2007: A Review of Tools for Incorporating Community Knowledge, Preferences, and Values into Decision Making in Natural Resources Management. Ecology and Society, 12, https://doi.org/10.5751/ES-01987-120105. Mach, K. J., and Coauthors, 2020: Actionable knowledge and the art of engagement. Current Opinion in Environmental Sustainability, 42, 30–37, https://doi.org/10.1016/j.cosust.2020.01.002. Magliocca, N. R., and Coauthors, 2018: Closing global knowledge gaps: Producing generalized knowledge from case studies of social-ecological systems. Global Environmental Change, 50, 1–14, https://doi.org/10.1016/j.gloenvcha.2018.03.003. Martin, V. Y., 2020: Four Common Problems In Environmental Social Research Undertaken by Natural Scientists. BioScience, 70, 13–16, https://doi.org/10.1093/biosci/biz128. Matsui, K., 2015: Problems of Defining and Validating Traditional Knowledge: A Historical Approach. International Indigenous Policy Journal, 6, https://doi.org/10.18584/iipj.2015.6.2.2. Maudlin, L. C., K. S. McNeal, H. Dinon-Aldridge, C. Davis, R. Boyles, and R. M. Atkins, 2020: Website Usability Differences between Males and Females: An Eye-Tracking Evaluation of a Climate Decision Support System. Wea. Climate Soc., 12, 183–192, https://doi.org/10.1175/WCAS-D-18-0127.1. McCarthy, I. P., D. Hannah, L. F. Pitt, and J. M. McCarthy, 2020: Confronting indifference toward truth: Dealing with workplace bullshit. Business Horizons, 63, 253–263, https://doi.org/10.1016/j.bushor.2020.01.001. McKinley, D. C., and Coauthors, 2017: Citizen science can improve conservation science, natural resource management, and environmental protection. Biological Conservation, 208, 15–28, https://doi.org/10.1016/j.biocon.2016.05.015. McNie, E. C., 2007: Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature. Environmental Science & Policy, 10, 17–38, https://doi.org/10.1016/j.envsci.2006.10.004. ——, 2011: Western Water Assessment Program Evaluation - Summative and Formative Evaluations: 2011. Sedna Organization Consulting, LLC,. ——, 2013: Delivering Climate Services: Organizational Strategies and Approaches for Producing Useful Climate-Science Information. Weather Clim Soc, 5, 14–26, https://doi.org/10.1175/WCAS-D-11-00034.1. Meadow, A. M., and G. Owen, 2021: Planning and Evaluating the Societal Impacts of Climate Change Research Project: A guidebook for natural and physical scientists looking to make a difference. https://www.rie.arizona.edu/sites/default/files/Meadow-Owen_Societal-Impacts_Guidebook.pdf (Accessed March 7, 2022). ——, D. B. Ferguson, Z. Guido, A. Horangic, G. Owen, and T. Wall, 2015: Moving toward the Deliberate Coproduction of Climate Science Knowledge. Weather Clim Soc, 7, 179–191, https://doi.org/10.1175/Wcas-D-14-00050.1. Middleton, L., 2020: Science communication and coproduction: applying the theory of motivated information management to the science-policy interface. Colorado State University, https://hdl.handle.net/10217/212047. Miller, R. B., 2003: How the Belmont Report Fails. Essays in Philosophy, 4, 119–134, https://doi.org/10.5840/eip2003423. Molinsky, A., 2007: Cross-Cultural Code-Switching: The Psychological Challenges of Adapting Behavior in Foreign Cultural Interactions. The Academy of Management Review, 32, 622–640, https://doi.org/0.5465/amr.2007.24351878. Montana, J., L. Elliott, M. Ryan, and C. Wyborn, 2020: The need for improved reflexivity in conservation science. Environmental Conservation, 47, 217–219, https://doi.org/10.1017/S0376892920000326. Moon, K., T. Brewer, S. Januchowski-Hartley, V. Adams, and D. Blackman, 2016: A guideline to improve qualitative social science publishing in ecology and conservation journals. Ecology and Society, 21, https://doi.org/10.5751/ES-08663-210317. Morisette, J. T., and Coauthors, 2017: Crossing Boundaries in a Collaborative Modeling Workspace. Society & Natural Resources, 30, 1–10, https://doi.org/10.1080/08941920.2017.1290178. Moser, S. C., and J. A. Ekstrom, 2010: A framework to diagnose barriers to climate change adaptation. Proceedings of the National Academy of Sciences, 107, 22026–22031, https://doi.org/10.1073/pnas.1007887107. Naskar, M., K. Roy, G. Karnatak, S. K. Nandy, and A. Roy, 2018: Quantifying climate change induced threats to wetland fisheries: a stakeholder-driven approach. Environ Dev Sustain, 20, 2811–2830, https://doi.org/10.1007/s10668-017-0018-6. National Academies of Sciences Engineering Medicine, 2018: Future Water Priorities for the Nation: Directions for the U.S. Geological Survey Water Mission Area. The National Academies Press, 110 pp. National Research Council, 1978: Knowledge and Policy: The Uncertain Connection. The National Academies Press,. ——, 2009: Informing Decisions in a Changing Climate. The National Academies Press,. ——, 2012: Using Science as Evidence in Public Policy. The National Academies Press, https://doi.org/10.17226/13460. Naugle, D. E., B. W. Allred, M. O. Jones, D. Twidwell, and J. D. Maestas, 2020: Coproducing Science to Inform Working Lands: The Next Frontier in Nature Conservation. BioScience, 70, 90–96, https://doi.org/10.1093/biosci/biz144. Nguyen, V. M., N. Young, J. W. Brownscombe, and S. J. Cooke, 2019: Collaboration and engagement produce more actionable science: quantitatively analyzing uptake of fish tracking studies. Ecological Applications, 29, e01943, https://doi.org/10.1002/eap.1943. NIDIS, 2016: The National Integrated Drought Information System Implementation Plan - 2016 Update. National Oceanic and Atmospheric Administration, https://www.drought.gov/sites/default/files/2020-06/Implementation-Plan-December-2016-Update.pdf (Accessed February 26, 2021). Norström, A. V., and Coauthors, 2020: Principles for knowledge co-production in sustainability research. Nat Sustain, 1–9, https://doi.org/10.1038/s41893-019-0448-2. Oakley, N. S., and B. Daudert, 2016: Establishing Best Practices to Improve Usefulness and Usability of Web Interfaces Providing Atmospheric Data. Bulletin of the American Meteorological Society, 97, 263–274, https://doi.org/10.1175/BAMS-D-14-00121.1. O’Haire, C., and Coauthors, 2011: Appendix G, Strengths and Limitations of Stakeholder Engagement Methods. Engaging Stakeholders To Identify and Prioritize Future Research Needs, Methods Future Research Needs Reports, Agency for Healthcare Research and Quality (US). Oliver, K., A. Kothari, and N. Mays, 2019: The dark side of coproduction: do the costs outweigh the benefits for health research? Health Research Policy and Systems, 17, 33, https://doi.org/10.1186/s12961-019-0432-3. Orbe, M. P., 1998: From The Standpoint(s) of Traditionally Muted Groups: Explicating A Co-cultural Communication Theoretical Model. Communication Theory, 8, 1–26, https://doi.org/10.1111/j.1468-2885.1998.tb00209.x. Owen, G., 2021: Evaluating socially engaged climate research: Scientists’ visions of a climate resilient U.S. Southwest. Research Evaluation, 30, 26–38, https://doi.org/10.1093/reseval/rvaa028. ——, D. B. Ferguson, and B. McMahan, 2019: Contextualizing climate science: applying social learning systems theory to knowledge production, climate services, and use-inspired research. Climatic Change, 157, 151–170, https://doi.org/10.1007/s10584-019-02466-x. Patton, M. Q., 2011: Developmental evaluation: applying complexity concepts to enhance innovation and use. New York : Guilford Press,. ——, 2012: Essentials of Utilization-Focused Evaluation. SAGE,. ——, 2018: Evaluation Science. American Journal of Evaluation, 39, 183–200, https://doi.org/10.1177/1098214018763121. Pearman, O., and A. E. Cravens, 2022: Institutional barriers to actionable science: Perspectives from decision support tool creators. Environmental Science & Policy, 128, 317–325, https://doi.org/10.1016/j.envsci.2021.12.004. Phillipson, J., P. Lowe, A. Proctor, and E. Ruto, 2012: Stakeholder engagement and knowledge exchange in environmental research. Journal of Environmental Management, 95, 56–65, https://doi.org/10.1016/j.jenvman.2011.10.005. Pielke Jr, R. A., 2007: The Honest Broker: Making Sense of Science in Policy and Politics. Cambridge University Press, 198 pp. Plattner, H., C. Meinel, and L. Leifer, eds., 2011: Design Thinking: Understand – Improve – Apply. Springer-Verlag,. Porter, K., and Coauthors, 2011: Overview of the ARkStorm scenario. U.S. Geological Survey,. Preskill, H., and D. Russ-Eft, 2004: Building evaluation capacity. Sage Publications,. Prior, J., 2013: Community engagement to resolve climate adaptation conflicts. Climate Adaptation Futures, John Wiley & Sons, Ltd, 167–176. Pulver, S., N. Ulibarri, K. Sobocinski, S. Alexander, M. Johnson, P. McCord, and J. Dell’Angelo, 2018: Frontiers in socio-environmental research: components, connections, scale, and context. Ecology and Society, 23, https://doi.org/10.5751/ES-10280-230323. Purdy, A. J., and Coauthors, 2019: Designing Drought Indicators. Bull. Amer. Meteor. Soc., 100, 2327–2341, https://doi.org/10.1175/BAMS-D-18-0146.1. QSR International Pty Ltd., 2020: NVivo. Qualtrics, 2005: Qualtrics. Qualtrics,. Reed, M. S., and Coauthors, 2018: A theory of participation: what makes stakeholder and public engagement in environmental management work? Restoration Ecology, 26, S7–S17, https://doi.org/10.1111/rec.12541. Reyna, V., R. Villegas, M. Simrak, and M. Kwakwa, 2021: Ethical Complexities of Civically Engaged Research. PS: Political Science & Politics, 54, 734–737, https://doi.org/10.1017/S1049096521000846. Richards, R., M. Sanó, A. Roiko, R. W. Carter, M. Bussey, J. Matthews, and T. F. Smith, 2013: Bayesian belief modeling of climate change impacts for informing regional adaptation options. Environmental Modelling & Software, 44, 113–121, https://doi.org/10.1016/j.envsoft.2012.07.008. Rittel, H. W. J., and M. M. Webber, 1973: Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169, https://doi.org/10.1007/BF01405730. Robinson, J. A., and T. L. Hawthorne, 2018: Making Space for Community-Engaged Scholarship in Geography. The Professional Geographer, 70, 277–283, https://doi.org/10.1080/00330124.2017.1366775. Robinson, K. F., A. K. Fuller, R. C. Stedman, W. F. Siemer, and D. J. Decker, 2019: Integration of social and ecological sciences for natural resource decision making: challenges and opportunities. Environmental Management, 63, 565–573, https://doi.org/10.1007/s00267-019-01141-2. Rose, D. C., and Coauthors, 2018: The major barriers to evidence-informed conservation policy and possible solutions. Conserv Lett, 11, e12564, https://doi.org/10.1111/conl.12564. Rosendahl, D., R. McPherson, A. Wootten, E. Mullens, J. Blackband, and A. Bryan, 2019: Making Sense of Local Climate Projections. Eos, 100, https://doi.org/10.1029/2019EO136493. Ross, S. L., and Coauthors, 2013: SAFRR (Science Application for Risk Reduction) Tsunami Scenario--Executive Summary and Introduction: Chapter A in The SAFRR (Science Application for Risk Reduction) Tsunami Scenario. U.S. Geological Survey,. Rouse, W., 2007: People and Organizations: Explorations of Human-Centered Design. John Wiley & Sons, Ltd,. Rowe, G., and L. J. Frewer, 2005: A Typology of Public Engagement Mechanisms. Science, Technology, & Human Values, 30, 251–290, https://doi.org/10.1177/0162243904271724. Ruano-Chamorro, C., G. G. Gurney, and J. E. Cinner, 2022: Advancing procedural justice in conservation. Conservation Letters, e12861, https://doi.org/10.1111/conl.12861. Rumore, D., T. Schenk, and L. Susskind, 2016: Role-play simulations for climate change adaptation education and engagement. Nature Climate Change, 6, 745–750, https://doi.org/10.1038/nclimate3084. Rundstrom, R., 2009: Counter-Mapping. International Encyclopedia of Human Geography, R. Kitchin and N. Thrift, Eds., Elsevier, 314–318. Rust, N. A., and Coauthors, 2017: Quantity Does Not Always Mean Quality: The Importance of Qualitative Social Science in Conservation Research. Society & Natural Resources, 30, 1304–1310, https://doi.org/10.1080/08941920.2017.1333661. Saarman, E., and Coauthors, 2013: The role of science in supporting marine protected area network planning and design in California. Ocean & Coastal Management, 74, 45–56, https://doi.org/10.1016/j.ocecoaman.2012.08.021. Salazar, K., 2009: Department of the Interior Secretarial Order 3289: Addressing the Impacts of Climate Change on America’s Water, Land, and Other Natural and Cultural Resources. Samson, E., P. E. Hirsch, S. C. F. Palmer, J. W. Behrens, T. Brodin, and J. M. J. Travis, 2017: Early Engagement of Stakeholders with Individual-Based Modeling Can Inform Research for Improving Invasive Species Management: The Round Goby as a Case Study. Front. Ecol. Evol., 5, https://doi.org/10.3389/fevo.2017.00149. Sarewitz, D., and R. A. Pielke Jr, 2007: The neglected heart of science policy: reconciling supply of and demand for science. Environmental Science & Policy, 10, 5–16, http://dx.doi.org/10.1016/j.envsci.2006.10.001. Sarkki, S., H. I. Heikkinen, and A. Löf, 2021: Reindeer Herders as Stakeholders or Rights-Holders? Introducing a Social Equity-Based Conceptualization Relevant for Indigenous and Local Communities. Nordic Perspectives on the Responsible Development of the Arctic: Pathways to Action, D.C. Nord, Ed., Springer Polar Sciences, Springer International Publishing, 271–292. Saunkeah, B., J. A. Beans, M. T. Peercy, V. Y. Hiratsuka, and P. Spicer, 2021: Extending Research Protections to Tribal Communities. The American Journal of Bioethics, 21, 5–12, https://doi.org/10.1080/15265161.2020.1865477. Scheufele, D. A., and S. Iyengar, 2017: The State of Framing Research. The Oxford Handbook of Political Communication, Oxford University Press. Schlesinger, W. H., 2010: Translational Ecology. Science, 329, 609–609, https://doi.org/10.1126/science.1195624. Schmidt, L., T. Falk, M. Siegmund-Schultze, and J. H. Spangenberg, 2020: The Objectives of Stakeholder Involvement in Transdisciplinary Research. A Conceptual Framework for a Reflective and Reflexive Practise. Ecological Economics, 176, 106751, https://doi.org/10.1016/j.ecolecon.2020.106751. Sharfstein, J. M., 2016: Banishing “Stakeholders.” Milbank Q, 94, 476–479, https://doi.org/10.1111/1468-0009.12208. Shore, N., 2006: Re-Conceptualizing the Belmont Report. Journal of Community Practice, 14, 5–26, https://doi.org/10.1300/J125v14n04_02. Simm, D., A. Marvell, and A. Mellor, 2021: Teaching “wicked” problems in geography. Journal of Geography in Higher Education, 45, 479–490, https://doi.org/10.1080/03098265.2021.1956883. Simonsen, J., and T. Robertson, eds., 2013: Routledge International Handbook of Participatory Design. 1st Edition. Routledge, 320 pp. Singletary, L., W. Evans, L. Sicafuse, and L. Maletsky, 2015: Evaluation Resource Guide for Joint Fire Science Program Fire Exchanges. University of Nevada Cooperative Extension,. Sprain, L., 2017: Paradoxes of Public Participation in Climate Change Governance. The Good Society, 25, 62–80, https://doi.org/10.5325/goodsociety.25.1.0062. Stedman, R. C., N. A. Connelly, T. A. Heberlein, D. J. Decker, and S. B. Allred, 2019: The End of the (Research) World As We Know It? Understanding and Coping With Declining Response Rates to Mail Surveys. Society & Natural Resources, 32, 1139–1154, https://doi.org/10.1080/08941920.2019.1587127. Steger, C., and Coauthors, 2021: Science with society: Evidence-based guidance for best practices in environmental transdisciplinary work. Global Environmental Change, 68, 102240, https://doi.org/10.1016/j.gloenvcha.2021.102240. Symstad, A. J., N. A. Fisichelli, B. W. Miller, E. Rowland, and G. W. Schuurman, 2017: Multiple methods for multiple futures: Integrating qualitative scenario planning and quantitative simulation modeling for natural resource decision making. Climate Risk Management, 17, 78–91, https://doi.org/10.1016/j.crm.2017.07.002. Tadaki, M., 2017: Rethinking the role of critique in physical geography. The Canadian Geographer / Le Géographe canadien, 61, 73–83, https://doi.org/10.1111/cag.12299. ——, G. Brierley, M. Dickson, R. Le Heron, and J. Salmond, 2015: Cultivating critical practices in physical geography. The Geographical Journal, 181, 160–171, https://doi.org/10.1111/geoj.12082. Taherdoost, H., 2017: Determining Sample Size; How to Calculate Survey Sample Size. International Journal of Economics and Management Systems, 02. Taras, M., 2008: Summative and formative assessment: Perceptions and realities. Active Learning in Higher Education, 9, 172–192, https://doi.org/10.1177/1469787408091655. Taylor, K. E., R. J. Stouffer, and G. A. Meehl, 2012: An Overview of CMIP5 and the Experiment Design. Bull. Amer. Meteor. Soc., 93, 485–498, https://doi.org/10.1175/BAMS-D-11-00094.1. Torrance, H., 1993: Formative assessment: Some theoretical problems and empirical questions. Cambridge Journal of Education, 23, 333. Trisos, C. H., J. Auerbach, and M. Katti, 2021: Decoloniality and anti-oppressive practices for a more ethical ecology. Nat Ecol Evol, 1–8, https://doi.org/10.1038/s41559-021-01460-w. Tsosie, K. S., J. M. Yracheta, and D. Dickenson, 2019: Overvaluing individual consent ignores risks to tribal participants. Nat Rev Genet, 20, 497–498, https://doi.org/10.1038/s41576-019-0161-z. Turnhout, E., T. Metze, C. Wyborn, N. Klenk, and E. Louder, 2020: The politics of co-production: participation, power, and transformation. Current Opinion in Environmental Sustainability, 42, 15–21, https://doi.org/10.1016/j.cosust.2019.11.009. United States, 1971: ALASKA NATIVE CLAIMS SETTLEMENT. ——, 1978: The Belmont report: ethical principles and guidelines for the protection of human subjects of research. U.S. Department of Health and Human Services, and National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html. USGS, 2021: Program Evaluation | Climate Adaptation Science Centers. U.S. Geological Survey,. https://www.usgs.gov/programs/climate-adaptation-science-centers/program-evaluation (Accessed October 25, 2021). VanderMolen, K., A. M. Meadow, A. Horangic, and T. U. Wall, 2020: Typologizing Stakeholder Information Use to Better Understand the Impacts of Collaborative Climate Science. Environmental Management, 65, 178–189, https://doi.org/10.1007/s00267-019-01237-9. VanLandingham, G., and T. Silloway, 2016: Bridging the Gap between Evidence and Policy Makers: A Case Study of the Pew‐MacArthur Results First Initiative. Public Administration Review, 76, 542–546. Vollmer, S. H., and G. Howard, 2010: Statistical Power, the Belmont Report, and the Ethics of Clinical Trials. Sci Eng Ethics, 16, 675–691, https://doi.org/10.1007/s11948-010-9244-0. Wall, T., A. M. Meadow, and A. Horangic, 2017: Developing Evaluation Indicators to Improve the Process of Coproducing Usable Climate Science. Weather Clim Soc, 9, 95–107, https://doi.org/10.1175/WCAS-D-16-0008.1. Webb, C., 1992: The use of the first person in academic writing: objectivity, language and gatekeeping. Journal of Advanced Nursing, 17, 747–752, https://doi.org/10.1111/j.1365-2648.1992.tb01974.x. Weiss, C. H., 1998: Have We Learned Anything New About the Use of Evaluation? American Journal of Evaluation, 19, 21–33, https://doi.org/10.1177/109821409801900103. West, S., L. J. Haider, S. Stålhammar, and S. Woroniecki, 2020: A relational turn for sustainability science? Relational thinking, leverage points and transformations. Ecosystems and People, 16, 304–325, https://doi.org/10.1080/26395916.2020.1814417. WGA, 2004: Creating a Drought Early Warning System for the 21st Century. Western Governors’ Association, https://www.drought.gov/sites/default/files/2020-06/200406_WGA_NIDIS_Report.pdf (Accessed February 26, 2021). White, D. D., A. Wutich, K. L. Larson, P. Gober, T. Lant, and C. Senneville, 2010: Credibility, salience, and legitimacy of boundary objects: water managers’ assessment of a simulation model in an immersive decision theater. Science and Public Policy, 37, 219–232, https://doi.org/10.3152/030234210X497726. Whyte, K., 2020: Too late for indigenous climate justice: Ecological and relational tipping points. WIREs Climate Change, 11, e603, https://doi.org/10.1002/wcc.603. Whyte, K. P., 2013: On the role of traditional ecological knowledge as a collaborative concept: a philosophical study. Ecological Processes, 2, 7, https://doi.org/10.1186/2192-1709-2-7. ——, 2014: Justice forward: Tribes, climate adaptation and responsibility. Climate Change and Indigenous Peoples in the United States: Impacts, Experiences and Actions, J.K. Maldonado, B. Colombi, and R. Pandya, Eds., Springer International Publishing, 9–22. Wiliam, D., 2006: Formative Assessment: Getting the Focus Right. Educational Assessment, 11, 283–289. Wilkins, E. J., and Coauthors, 2019: Rural-urban differences in hunting and birdwatching attitudes and participation intent. Human Dimensions of Wildlife, 24, 530–547, https://doi.org/10.1080/10871209.2019.1661046. Wilmer, H., and Coauthors, 2021: Expanded Ethical Principles for Research Partnership and Transdisciplinary Natural Resource Management Science. Environmental Management, https://doi.org/10.1007/s00267-021-01508-4. Wilson, M. A., and R. B. Howarth, 2002: Discourse-based valuation of ecosystem services: establishing fair outcomes through group deliberation. Ecological Economics, 41, 431–443, https://doi.org/10.1016/S0921-8009(02)00092-7. Winkler, J. A., 2016: Embracing Complexity and Uncertainty. Annals of the American Association of Geographers, 106, 1418–1433, https://doi.org/10.1080/24694452.2016.1207973. Wong-Parodi, G., K. J. Mach, K. Jagannathan, and K. D. Sjostrom, 2020: Insights for developing effective decision support tools for environmental sustainability. Current Opinion in Environmental Sustainability, 42, 52–59, https://doi.org/10.1016/j.cosust.2020.01.005. World Meteorological Organization, 2014: Implementation Plan of the Global Framework for Climate Services. https://gfcs.wmo.int//implementation-plan (Accessed November 9, 2020). Wright, P., and J. McCarthy, 2010: Experience-Centered Design: Designers, Users, and Communities in Dialogue. Synthesis Lectures on Human-Centered Informatics, 3, 1–123, https://doi.org/10.2200/S00229ED1V01Y201003HCI009. Wyborn, C., A. Datta, J. Montana, M. Ryan, P. Leith, B. Chaffin, C. Miller, and L. van Kerkhoff, 2019: Co-Producing Sustainability: Reordering the Governance of Science, Policy, and Practice. Annual Review of Environment and Resources, 44, https://doi.org/10.1146/annurev-environ-101718-033103. Young, N., S. J. Cooke, S. G. Hinch, C. DiGiovanni, M. Corriveau, S. Fortin, V. M. Nguyen, and A.-M. Solås, 2020: “Consulted to death”: Personal stress as a major barrier to environmental co-management. Journal of Environmental Management, 254, 109820, https://doi.org/10.1016/j.jenvman.2019.109820. Appendix A - Survey data public summary All survey questions were optional to complete, so the total number of responses per question differs. Questions with no results or with identifying information in the results are not included. 2.2 - When did the investigator team first contact you regarding the project? (select one) # Answer % Count 1 I (or someone from my agency) contacted the investigator team first 24.49% 12 2 Prior to writing the proposal 32.65% 16 3 During the proposal writing process 12.24% 6 4 After the project was selected for funding 28.57% 14 5 After the project was completed 2.04% 1 Total 100% 49 2.3 - How frequently did you interact with the investigator team during the lifetime of the project? (select one) # Answer % Count 1 Never (0 times a year) 2.04% 1 2 Rarely (1-3 times a year) 28.57% 14 3 Roughly every other month (4-8 times a year) 38.78% 19 4 Roughly every month (8-14 times a year) 12.24% 6 5 Multiple times a month (15-51 times a year) 14.29% 7 6 At least every week (52 or more times a year) 4.08% 2 Total 100% 49 2.4 - What was your perception of the frequency of interaction? (select one) # Answer % Count 1 We interacted far too much 0.00% 0 2 We interacted slightly too much 0.00% 0 3 We interacted the right amount 84.00% 42 4 We interacted slightly too little 10.00% 5 5 We interacted far too little 6.00% 3 Total 100% 50 2.5 - How did the investigator team interact with you in each of the relevant phases of the project? (select all that apply) # Question Email correspondence Telephone or video calls Face-to-face interactions No interaction Total 1 Definition of the management or research problem(s) 35.71% 30 21.43% 18 35.71% 30 7.14% 6 84 2 Articulation of the research question(s) 35.06% 27 19.48% 15 35.06% 27 10.39% 8 77 3 Design of the research methods 37.68% 26 17.39% 12 26.09% 18 18.84% 13 69 4 Selection of project products 38.67% 29 24.00% 18 28.00% 21 9.33% 7 75 5 Collection of project data 35.29% 24 14.71% 10 30.88% 21 19.12% 13 68 6 Analysis of project data 31.25% 20 18.75% 12 26.56% 17 23.44% 15 64 7 Interpretation of results 35.14% 26 18.92% 14 33.78% 25 12.16% 9 74 8 Creation of tools or other project products 41.54% 27 16.92% 11 30.77% 20 10.77% 7 65 9 Dissemination of findings 38.16% 29 21.05% 16 32.89% 25 7.89% 6 76 Recoded: # Question Interaction No interaction Total 1 Definition of the management or research problem(s) 87.50% 42 12.50% 6 48 2 Articulation of the research question(s) 83.33% 40 16.67% 8 48 3 Design of the research methods 72.92% 35 27.08% 13 48 4 Selection of project products 85.71% 42 14.29% 7 49 5 Collection of project data 72.92% 35 27.08% 13 48 6 Analysis of project data 67.39% 31 32.61% 15 46 7 Interpretation of results 80.43% 37 19.57% 9 46 8 Creation of tools or other project products 84.78% 39 15.22% 7 46 9 Dissemination of findings 86.96% 40 13.04% 6 46 2.6 - How did the investigator team identify your agency's needs? (select one) # Answer % Count 1 A formal needs assessment was conducted by the investigator team. 3.85% 1 2 The investigator team determined your agency's needs through informal discussion(s). 38.46% 10 3 Agency needs were determined by my organization or another investigator (team). 11.54% 3 4 No assessment of agency needs was conducted. 26.92% 7 5 I don't know. 19.23% 5 Total 100% 26 2.7 - How did the investigator team assess the vulnerability of or risk to the resources and/or landscapes that your agency manages? (select one) # Answer % Count 1 A formal vulnerability or risk assessment was conducted by the investigator team. 23.40% 11 2 The investigator team assessed vulnerability or risk through informal discussion(s). 38.30% 18 3 Vulnerability or risk were determined by my organization or another investigator (team). 12.77% 6 4 No assessment of vulnerability or risk was conducted. 12.77% 6 5 I don't know. 12.77% 6 Total 100% 47 3.2 - What kinds of information, data, tools, or other products did this project provide you? 3.3 - This project provided me with information, data, tools, or other products that... (select one for each) # Question Agree Disagree Neither agree nor disagree Not applicable Total 1 Included user documents or guidance 78.72% 37 6.38% 3 4.26% 2 10.64% 5 47 2 Were easy to access 87.23% 41 0.00% 0 6.38% 3 6.38% 3 47 3 Were easy to use 76.09% 35 0.00% 0 17.39% 8 6.52% 3 46 4 Appeared rigorous and credible 85.11% 40 0.00% 0 8.51% 4 6.38% 3 47 3.4 - The investigator team and project helped me to: (select all that apply) # Answer % Count 1 Understand variability and changes in weather and climate in the places that I work 37.70% 23 2 Link variability and changes in weather and climate to impacts to resources or places that I manage 45.90% 28 3 Identify, evaluate, or select potential adaptation strategies to cope with these impacts 0.00% 0 4 other 16.39% 10 Total 100% 61 3.5 - Were the project results useful for you or your colleagues in the following ways? (select one for each) # Question Extremely useful Moderately useful Not useful Total 1 Being better informed broadly about an issue 71.43% 30 28.57% 12 0.00% 0 42 2 Being better informed specifically about a particular problem 70.45% 31 29.55% 13 0.00% 0 44 3 Gained a new technical skill 35.48% 11 38.71% 12 25.81% 8 31 4 Formulated policy 9.52% 2 66.67% 14 23.81% 5 21 5 Informed management plans 50.00% 18 41.67% 15 8.33% 3 36 6 Informed management actions 48.48% 16 45.45% 15 6.06% 2 33 7 Informed adaptation plans 48.48% 16 42.42% 14 9.09% 3 33 8 Implemented adaptation plans 52.17% 12 34.78% 8 13.04% 3 23 9 Provided training or education within your organization 64.29% 18 28.57% 8 7.14% 2 28 10 Provided training or education outside of your organization 71.43% 20 25.00% 7 3.57% 1 28 11 Other 0.00% 0 0.00% 0 0.00% 0 0 3.6 - Please briefly describe how your agency used project results to inform any agency plans or actions. 3.7 - Were there any specific factors that promoted the use of project results (select one) # Answer % Count 1 Yes 52.17% 24 2 No 47.83% 22 Total 100% 46 3.8 - Please briefly describe the specific factors that you feel promoted the use of project results. 3.9 - Did anything limit the use of project results? (select one) # Answer % Count 9 Yes 28.89% 13 10 No 71.11% 32 Total 100% 45 3.10 - From your perspective, what factors may have limited the use of the project results? (select all that apply) # Answer % Count 1 Science was not of sufficient quality 0.00% 0 2 Results did not match scale of decisions or plans 7.69% 2 3 Production of results did not match the timeline for decision making or planning 7.69% 2 4 Resource managers / decision makers were not engaged sufficiently by the investigators 3.85% 1 5 Results were not communicated in a way that is understandable by resource managers / decision makers 3.85% 1 6 Tools and other products were not designed in a way that is functional for resource managers / decision makers 3.85% 1 7 Resource managers / decision makers need additional skills or training to make use of the results 3.85% 1 8 Did not match existing internal priorities of your organization 7.69% 2 9 Internal priorities of your organization shifted 3.85% 1 10 Resource constraints within your organization 15.38% 4 11 Other 23.08% 6 12 Resource managers / decision makers need additional time to make use of the results 19.23% 5 Total 100% 26 4.2 - How do you feel about your experiences with this investigator team? (select one) # Answer % Count 1 Extremely satisfied 72.34% 34 2 Somewhat satisfied 21.28% 10 3 Neither satisfied nor dissatisfied 4.26% 2 4 Somewhat dissatisfied 2.13% 1 5 Extremely dissatisfied 0.00% 0 Total 100% 47 4.3 - How do you feel about your experiences with this project? (select one) # Answer % Count 1 Extremely satisfied 57.45% 27 2 Somewhat satisfied 29.79% 14 3 Neither satisfied nor dissatisfied 8.51% 4 4 Somewhat dissatisfied 4.26% 2 5 Extremely dissatisfied 0.00% 0 Total 100% 47 4.4 - Please rate how strongly you agree or disagree with the following statements. (select one for each) # Question Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Total 1 The investigator team appreciated and respected what I brought to the project. 72.34% 34 19.15% 9 6.38% 3 2.13% 1 0.00% 0 47 2 The investigator team took my opinion seriously in the course of discussions. 72.34% 34 17.02% 8 8.51% 4 2.13% 1 0.00% 0 47 3 The investigator team was committed to the process. 78.72% 37 12.77% 6 8.51% 4 0.00% 0 0.00% 0 47 4 The investigator team was honest and sincere. 87.23% 41 12.77% 6 0.00% 0 0.00% 0 0.00% 0 47 5 The investigator team was trustworthy. 87.23% 41 12.77% 6 0.00% 0 0.00% 0 0.00% 0 47 4.5 - Based on your experiences with this investigator team and project, in the future, would you: (select one for each) # Question Extremely likely Somewhat likely Neither likely nor unlikely Somewhat unlikely Extremely unlikely Total 1 Use additional results generated by this investigator team 76.60% 36 23.40% 11 0.00% 0 0.00% 0 0.00% 0 47 2 Work with this investigator team again on another project 84.44% 38 13.33% 6 0.00% 0 2.22% 1 0.00% 0 45 3 Work with another investigator team on a CSC-funded project 58.70% 27 36.96% 17 4.35% 2 0.00% 0 0.00% 0 46 4 Request climate-relevant adaptation support directly from the CSC 45.65% 21 34.78% 16 19.57% 9 0.00% 0 0.00% 0 46 4.7 - From your perspective, what reason(s) would make it likely for you to work with this investigator team or the CSC again in the future? 5.2 - What is the primary focus area of your work? (select all that apply) # Answer % Count 1 Wildlife 21.78% 22 2 Fish 6.93% 7 3 Water 19.80% 20 4 Habitat or vegetation 23.76% 24 5 Cultural resources 6.93% 7 6 Infrastructure 3.96% 4 7 Agriculture 10.89% 11 8 Other 5.94% 6 Total 100% 101 5.3 - Would you consider your role primarily to be? (select one) # Answer % Count 1 Resource manager/decision maker/planner 45.65% 21 2 Scientist/technician/researcher 39.13% 18 3 Equally both 15.22% 7 Total 100% 46 5.5 - Please provide any additional feedback on your experience with this investigator team and project that you feel was not covered in the survey questions: