A more nuanced analysis of evidence-based decision-making by Canada’s protected area managers: a comment on Lemieux et al. (2021)

In protected areas management, critical decisions demand appropriate use of evidence (Sutherland et al. 2004, 2019). Because the management of protected areas is conducted in a cultural, social, economic, and ecological interface (Robinson 2011), an integrated, multidisciplinary perspective should guide the selection, evaluation, and interpretation of relevant and sufficient bodies of evidence to inform conservation action. Erroneous conclusions based on flawed analyses and interpretations are problematic, and could result in misdirected time, energy, and resources, or worse (i.e., poor conservation outcomes with repercussions for people).

Given the importance of appropriate use of evidence, Lemieux et al. (2021) claimed in their recent article in this journal that evidence use by Canadian protected area managers is in dramatic decline since 2013 is, on the surface, highly disturbing. But closer examination of both their specific analyses and the broader context paints a more nuanced picture of the situation. While we strongly agree with Lemieux et al. (2021) that appropriate use of evidence is a key element of effective protected area management, we have concerns regarding their characterization of the types of evidence that are appropriate for various kinds of questions faced by protected area managers, and we challenge some of their survey approaches and interpretation. Moreover, we suggest important concepts are missing from their paper, so we expand on their recommendations in the context of additional barriers to the successful use of evidence to inform conservation in Canada’s protected areas. We also extend and, in some cases, amplify recommendations arising from the Lemieux et al. (2021) paper given our common interest in evidence and its application.

Evidence is diverse, its characterization nuanced. How much evidence is needed, and the detail and specificity required, should be related to the conservation action under consideration. The relative effect of the decision to act (or not), the applicability of the generic (nonlocal) evidence to the specific conservation question, and the burden and standard of proof required (Salafsky and Redford 2013) by the impact (locally to globally) of any management decision all help determine the extent and focus required of the evidence. The burden of proof can be defined by the reversibility of the decision or process, the certainty of the outcome, the relative consequences of acting in error, and the urgency of the conservation problem at hand (Salafsky and Redford 2013). The higher the burden of proof, the more thorough the collection, documentation, and evaluation of evidence would need to be. Assembling and evaluating the necessary evidence can be part of a process that brings relevant partners together for structured and adaptive discussions to define the conservation targets and the appropriate actions to achieve them, as per the Conservation Standards (Conservation Measures Partnership 2020).

Lemieux et al. (2021) argued that there is a rank order of evidence valuation with anecdotal local knowledge at the bottom of the pyramid and peer-reviewed papers at the top. Unfortunately, this hierarchy is faulty at both ends of its spectrum. On one end, the represented rank order of evidence fails to consider the situation-dependent context of many protected area management questions wherein Indigenous Knowledge or local knowledge may be most relevant and useful. On the other end, it ignores the immense power and sometimes increased management relevance of systematic reviews over other evidence synthesis methods (Thomas-Walters et al. 2021). With various types of evidence across the spectrum underrated or unconsidered (e.g., evidence syntheses) by their survey approach and interpretation, the claims of Lemieux et al. (2021) that evidence use is falling and failing protected areas and conservation are based on an incomplete view.

Perhaps what is needed is not more adherence to ideologies and ranking systems, but rather a systematic and documented process to thoughtfully determine what the best evidence may be for a given question of management concern. As such, we recommend a process where all types of evidence available (including knowledge held by rightsholders and stakeholders) are considered in the context of the import, time sensitivity, and uniqueness of the particular conservation question. This allows the value of each piece of evidence to be considered in relation to its reliability, directionality, strength, and relevance (Salafsky et al. 2019) and then subjected to critical review (Irvine et al. 2021).

This type of evidence-matching process is well established and tested in medical and judicial realms and continues to develop in the conservation domain. In conservation, peer-reviewed literature is most valuable when the generic evidence it contains has spatial or ecological relevance to the conservation action being contemplated and when the burden of proof is high, then more than local evidence is required to ensure the more overarching claims are supported by the evidence in the literature (Salafsky and Redford 2013). Often, peer-reviewed literature or a systematic review will point to what local evidence still needs to be generated through monitoring by pointing out that the weight of evidence for a conservation action to have a certain effect is relatively high, but that the particular outcome locally is unknown (Stewart et al. 2005). However, not only does the internal validity (reliability) need to be determined for each piece of evidence (Pullin and Stewart 2006), but the external reliability (relevance) also needs to be considered (Pullin et al. 2020). A locally sourced anecdote that speaks directly to the management question (e.g., observation of the presence of an imperilled species in a given area) could be far more useful than a systematic review that is only marginally relevant to the immediate issue. This need to combine both local and external sources of evidence is particularly true in situations in which we are not doing “black box” analyses of the effectiveness of a specific intervention (e.g., using insecticides to control pest species), but rather are using complex conservation strategies (e.g., using a combination of predator management, visitor management, and conservation breeding approaches to restore an endangered population of caribou).

Furthermore, evidence complacency, or the seeming reluctance of conservation practitioners to use peer-reviewed science (Sutherland and Wordley 2017), needs to be carefully differentiated from using appropriate local and Indigenous Knowledge that adequately answers the questions at hand. Consciously choosing the correct type and level of evidence is a judgement that considers all aspects of the problem rather than appealing by default to a perceived authority through the use of peer-reviewed literature that may or may not be relevant. Peer review is imperfect (Henderson 2010), and there may be so-called grey literature outputs (e.g., local monitoring reports) that may be highly relevant to a given topic. Moreover, generic effect sizes predicted across studies may not provide sufficient local relevance (Dicks et al. 2014). The complacency issue also applies to blinkered academics (O’Connell and White 2017) who may automatically assume that their work is going to be the most relevant to conservation practice when, unless the work is specifically aligned to support managers of protected areas, it may be very far from useful.

Although Lemieux et al. (2021) identified a very strong link between protected area management questions and evidence syntheses, they failed to ask their survey respondents about their use or valuation of this body of work. This is despite other recent research showing that if high-quality syntheses are readily available and relevant, they are likely to be used to support and enable evidence-based decision-making in conservation in Canada (Thomas-Walters et al. 2021).

When is evidence and knowledge held by protected areas staff most valuable? We would argue it is when the generic evidence available does not answer the question of management concern, when the system is particularly unique so that extrapolation is spurious, when language barriers don’t allow the use of other evidence sources, and when it is high-quality work using scientifically defensible and rigorous design and analysis. Staff-derived evidence is also most usable when the burden of proof is relatively low since the work can be evaluated, action taken, and adaptive management and monitoring initiated efficiently, freeing up time to allocate more effort where required. This partially addresses the capacity overload situation for protected area managers identified by Lemieux et al. (2018) where managers were less likely to use nonstaff-derived evidence the longer they were with an organization. Triage and appropriate calibration of the level of rigour, given the burden of proof required, is a key concept to manage the burnout that is all too common in conservation organizations.

In addition to their mischaracterization of evidence, we have identified both methodological and analytical issues that fail to support Lemieux et al.’s (2021) thesis that there has been a dramatic decline in the use of evidence by protected area managers in Canada over the past few years.

From a methodological perspective, it is arguable whether the sample of respondents in the surveys that Lemieux et al. (2021) conducted is representative of Canadian protected area managers as a whole. Of the respondents, 46% of them work in protected areas management in Alberta. And yet, Alberta supports only 12% of Canada’s population and only 15% of its protected areas (Canada and Environment and Climate Change Canada 2020). In addition, when compared to the representation by region from the 2013 baseline data, Alberta was still high, but only comprised 27% of the survey sample at that time. Comparing evidence use patterns across time using groups with substantively different regional composition is dubious. Drawing strong inferences from these data despite their relatively small sample size and variable purposive sampling between years without controlling for regional bias is presumptuous; the differences noted could be mainly driven by respondents from a single natural resource management agency in one province (i.e., Alberta Environment and Parks) that has been potentially influenced by institutional culture and geopolitical changes including an election in April 2019 that saw a change in governing political power. The fact that the survey spanned frontline staff and high-level managers may also introduce variance in responses. Presumably frontline staff are dealing with different decisions (in scale and consequence) than managers based at headquarters. From Lemieux et al. (2021) we are unable to assess the change in the community surveyed across years or the extent to which one’s position and role within an organization may also influence perspectives.

From an analytical perspective, Lemieux et al. (2021) stated that “Notably, both local and Indigenous forms of knowledge have increased in perceived value since the 2013 survey”—directly contradicting their own survey results reported in table 2. First, Indigenous Knowledge has increased in valuation very slightly from a mean of 2.9 to a mean of 3.1, but the change is not significant and therefore does not even allow the calculation of a Cohen’s d value (see Lemieux et al. 2021). Local knowledge has in fact decreased in its mean valuation from 3.2–2.9 (delta of −0.25), which is statistically significant and shows a moderate Cohen’s d rating. The relative valuation of staff assessments has also declined in a statistically significant way from 2013 to 2019 (mean of 3.8 dropping to 3.4), whereas valuation of peer review has increased with a moderate Cohen’s d value. So, some of their arguments in the paper appear not to be grounded in the actual survey data.

Furthermore, in 2019 the research team did not collect data on the organizational affiliation of the survey respondents. The rationale for not asking for such information was to “… elevate anonymity and encourage more fulsome responses regarding their agency’s knowledge mobilization capabilities and challenges.” This decision has rather significant implications for how the findings can be interpreted and operationalized. As one example, within Ontario, it is unknown how many of the respondents were from governments and how many were from regional conservation authorities. Consequently, we’re unable to look for any bright spots and dark spots at the organizational level where activities are implemented from which to learn or draw cautionary tales. For instance, a regional government responsible for a few small parks is less likely to have internal science support capacity, access to academic library subscriptions, or the logistical and staff support to engage with Indigenous communities or other relevant communities and knowledge holders or users in a meaningful and respectful way. Conversely, there’s an expectation that such capacity exists certainly at the federal level but also in some of the larger provincial agencies. The analysis assumes that there are no differences in institutional culture or capacity which is not constructive. In fact, we submit that there may be agencies and organizations that are leading the way not just nationally but internationally in this respect (e.g., canadiancebc.com/projects/), but they are not identified so it is impossible to learn from them and better address the identified barriers.

The provocative title of the article implies a much stronger effect than is substantiated in the paper itself and continues to perpetuate a maladaptive division between knowledge generators and protected area managers.

We also believe that Lemieux et al. (2021) based their survey on an incomplete list of potential barriers to evidence use. The top three barriers to evidence use that they identified were limited financial resources, lack of staff, and lack of time. While we agree that these are often important obstacles, we think that other vital barriers were omitted from their survey.

One of the greatest barriers to evidence use in our experience is a careful and detailed framing of the conservation problem. Without delineating clear claims, it is incredibly difficult to define the relevant evidence base and evaluate the collected weight of evidence supporting or refuting the claim (Irvine et al. 2021). Training in the evaluation and assessment of evidence is critical for protected area managers to allow improved formulation of the problems and thoughtful consideration of the evidence and the conservation questions at hand (Conservation Measures Partnership 2020; Downey et al. 2021). Although the availability of evidence to conservation practitioners outside of academia has never been greater, with an estimated 45% of available literature in open access journals in 2018 (Piwowar et al. 2018), accessing, filtering, and interpreting journal articles still takes time and resources, which remain a limiting factor for survey respondents in carrying out effective conservation in Canada (Lemieux et al. 2021) and globally (Christie et al. 2021). The clear framing of the conservation question and the explicit consideration of the burden of proof for the question at hand may help address this barrier to a degree since derivation of the claims supporting or refuting the proposed action can help narrow the field of consideration for the evidence base. The solution to the barrier of “information swamping” is confounded by Lemieux et al.’s (2021) failure to differentiate between peer-reviewed literature papers and evidence syntheses. They state “…while ‘more [evidence] is better’…there is a danger of swamping decision-makers with research and monitoring findings and overwhelming their ability to synthesize them into conclusions that are timely and meaningful (Items 5, 9, and 13; table 5).” And yet, evidence syntheses, focused on management questions, aggregate research findings and conclusions in meaningful ways that allow decision-makers to draw conclusions and inform decisions in timely ways based on the totality of the evidence while acknowledging bias and uncertainty (Cook et al. 2017). Furthermore, research demonstrates that decision-makers will preferentially use evidence synthesis of the highest level if they are available (Thomas-Walters et al. 2021).

A second overlooked barrier is the meaningful articulation and framing of the conservation question with Indigenous communities’ perspectives equally weighted throughout the entire process. In Lemieux et al.’s (2021) table 1, the context of protected areas decisions and how they select and use evidence from different knowledge systems is not highlighted and it is a significant barrier to ensure all types of evidence are used to inform conservation. The underrepresentation of Indigenous persons in the natural sciences as well as the ignorance of some researchers and protected area managers about Indigenous law and how to respectfully interact with those communities (Wong et al. 2020) need to be addressed by individuals as well as institutions (e.g., by Indigenizing conventional resource management models; Hessami et al. 2021) before trust can be built to blend the evidence bases (Kadykalo et al. 2021). This is combined with the colonial structuring of most national and provincial–territorial protected areas, including the intentional exclusion of First Nations from their use and management (Binnema and Niemi 2006), and it creates a significant barrier. However, there is increased understanding of the pivotal importance and wisdom of Indigenous Knowledge (as reflected empirically by the higher species richness of Indigenous protected areas; Schuster et al. 2019) and Knowledge Holders, academics, and land area managers are working to determine ways to equally weight Indigenous Knowledge (e.g., Schang et al. 2020; Reid et al. 2020), as noted by Lemieux et al. (2021). Uncertainty about how to do this blending well is a significant barrier in Canada (Cooke et al. 2016; Buxton et al. 2021), but it is also an area inviting great innovation being bridged through careful and kind use of principles such as Ethical Space and Two-Eyed Seeing (Reid et al. 2020; M’sɨt No’kmaq et al. 2021) and through well-established co-management principles in some protected areas (Lee et al. 2021).

A third barrier not explicitly discussed in Lemieux et al. (2021), and potentially very important in their data set, is the political landscape in which protected area managers work. Globally, we have seen the rise of populism that aligns evidence-based and scientific thinking with the political parties on the left of the spectrum despite the position of scientists that they are politically agnostic. While this may not influence individual protected area managers’ approaches, it assuredly affects the resourcing and mandate direction of agencies run by provincial–territorial, federal, and First Nations governments. Without carrying on into advanced scientific study, the levels of scientific literacy and understanding in the public are staggeringly low with 77% of US citizens unable to describe the idea of a scientific study and 36% incorrectly describing the concept of probability (Scheufele and Krause 2019); even those persons who understand scientific concepts correctly do not always carry that knowledge into their policy preferences (Scheufele and Krause 2019). Some causes elucidated for the perception of the diminishing role of evidence in decision-making in government agencies from other research were inferred by respondents to be: political and socio-economic interference (28%), decreased institutional resources and capacity (9%), and science integrity (9%) (Kadykalo et al. 2021). More work is needed to understand the socio-political context for decision-making and how that varies over time and jurisdiction.

A fourth barrier not examined clearly in Lemieux et al. (2021) is the lack of an accepted and universal method for documenting the evidence base and process for making conservation decisions. While judicial (e.g., reasons for judgements and cases in brief that are publicly available), and medical (e.g., patient records including clinical judgements at decision points that are open to professional review organizations) realms have prescriptive and accepted approaches for documenting technical or legal decisions, ecologists and protected area managers have no such standard required documentation. The lack of testing in the court system of decisions leading to conservation outcomes as well as other factors like the lack of a national governing organization for ecologists (only some provinces require Professional Biologist certifications requiring membership and Continuing Professional Development) have resulted in a documentation-poor profession. Lemieux et al. (2018) stated in their earlier paper to which this one is compared, “However, as Dicks et al. (2014) state, the personal experience of a decision-maker, often essential for effective action, can also lead ‘…to the propagation or entrenchment of poor or untested practice, a risk that is reduced if the sources of evidence and experience used to inform decisions are transparently recorded’ ”. The problem is therefore not in the type of evidence used if the local knowledge is the most relevant and helpful to the question at hand, but the documentation demonstrating that the local knowledge is credible and reliable. Simple tools such as summaries of evidence supporting each key decision claim and an expert evaluation (internal or external depending on the burden of proof) of a compiled evidence library can be streamlined and institutionalized without taking much additional time and providing a much more confident basis upon which to take conservation action. Over time, this could also result in a catalogue of informed decisions from which to learn and adapt monitoring and follow-up conservation actions.

Increased financial investment was recommended by Lemieux et al. (2018) and is amplified here, but simply more money spent without a strategic and analytic approach to prioritization and all the barriers to evidence use will not necessarily increase conservation benefit (Joseph et al. 2009) as demonstrated repeatedly in countries around the world that are struggling with similar conservation predicaments to Canada.

Lemieux et al. (2021) argued that “transformations are needed in four fundamental areas: (i) building a more relevant evidence base, (ii) (re)establishing collaborative forums for knowledge sharing, (iii) embedding researchers and Indigenous Knowledge Holders within protected areas organizations, and (iv) making a commitment to accountability, transparency, and reporting with respect to knowledge management.” While we concur with the need for transformation to occur in these areas, we have endeavoured through this response to identify additional barriers that were missed in the original paper that need to be addressed to remedy the issue of any diminishing evidence use. Unfortunately, these were not explored by Lemieux et al. (2021); therefore, the survey results reported in their paper do not make it clear exactly which barriers the protected areas community is experiencing and how they can best be overcome. The barriers we outline above are those we have experienced or noted through work in protected area management and conservation science, but likely there are other barriers which would emerge from a more comprehensive and representative survey of the Canadian protected area management community.

In conclusion, understanding evidence use in environmental decisions and institutionalizing robust and transparent processes at all levels of government is critical to ensure limited conservation resources are allocated wisely and that decisions benefit biodiversity and people. Here we highlight that there is a shift underway in how evidence is considered, documented, and blended among cultures within protected areas management in Canada. The decline in evidence use noted by Lemieux et al. (2021) may be occurring, but we believe the issue requires more nuanced examination and consideration of the appropriate and relevant evidence base rather than applying a simpler frame. Work such as that conducted by Lemieux et al. (2021) is important, but we need to adopt a more considered and considerate approach when evaluating the current state of environmental decisions in Canada and what we need to do to make them better. Arguably, these issues extend well beyond the protected areas of Canada and are germane to managed systems and natural resources around the globe.