Policy formulation, policy advice and policy appraisal: the distribution of analytical tools

At its heart, policy analysis is what Gill and Saunders (1992, pp. 6‒7) have characterized as ‘a method for structuring information and providing opportunities for the development of alternative choices for the policymaker’. An important part of the process of policy formulation, policy analysis involves policy appraisal: providing information or advice to policymakers concerning the relative advantages and disadvantages of alternative policy choices (Mushkin 1977; Wildavsky 1979; Sidney 2007; Howlett et al. 2009). Such advice comes from a variety of different actors operating in a wide range of venues both internal and external to government. And policy workers operating in these venues employ many different types of analytical techniques or ‘policy formulation tools’ in this effort (Mayer et al. 2004; Colebatch et al. 2011). These tools generally are designed to help evaluate current or past practices and aid decision making by clarifying or eliminating some of the many possible alternative courses of action mooted in the course of policy formulation. They play a significant role in structuring policymaking activity and in determining the content of policy outputs and thus policy outcomes (Sidney 2007) and are a worthy subject of investigation in their own right. Unfortunately, although many works have made recommendations and suggestions for how formulation should be conducted (Vining and Weimer 2010; Dunn 2004), very few works have studied how it is actually practiced, on the ground (Colebatch 2005 and 2006; Colebatch and Radin 2006; Noordegraaf 2011). This lack of knowledge is generally true of many of the tasks and activities involved in policy formulation (DeLeon 1992;

At its heart, policy analysis is what Gill and Saunders (1992, pp. 6-7) have characterized as 'a method for structuring information and providing opportunities for the development of alternative choices for the policymaker'. An important part of the process of policy formulation, policy analysis involves policy appraisal: providing information or advice to policymakers concerning the relative advantages and disadvantages of alternative policy choices (Mushkin 1977;Wildavsky 1979;Sidney 2007;Howlett et al. 2009).
Such advice comes from a variety of different actors operating in a wide range of venues both internal and external to government. And policy workers operating in these venues employ many different types of analytical techniques or 'policy formulation tools' in this effort (Mayer et al. 2004;Colebatch et al. 2011). These tools generally are designed to help evaluate current or past practices and aid decision making by clarifying or eliminating some of the many possible alternative courses of action mooted in the course of policy formulation. They play a significant role in structuring policy-making activity and in determining the content of policy outputs and thus policy outcomes (Sidney 2007) and are a worthy subject of investigation in their own right.
Unfortunately, although many works have made recommendations and suggestions for how formulation should be conducted (Vining and Weimer 2010;Dunn 2004), very few works have studied how it is actually practiced, on the ground (Colebatch 2005 andColebatch and Radin 2006;Noordegraaf 2011). This lack of knowledge is generally true of many of the tasks and activities involved in policy formulation (DeLeon 1992;Linder and Peters 1990), and data is limited on virtually every aspect of the policy appraisal activities in which governments engage and on the nature of the advice they receive in so doing (Page 2010;Page and Jenkins 2005).
Fortunately, however, some progress has been made on this front in recent years as evidence has begun slowly to be gathered on the nature of policy work and the different types of analytical tools practiced in different venues by different actors (Mayer et al. 2004;Boston et al. 1996;Tiernan 2011;Sullivan 2011). Several analysts, for example, have made considerable progress in mapping many of the activities involved in both ex post and ex ante policy evaluation Hertin et al. 2009;Turnpenny et al. 2008 and. And these efforts have been joined by other work done in Australia and elsewhere on regulatory impact assessments and the use of other similar tools and techniques in formulation activities (Carroll and Kellow 2011;Rissi and Sager 2013).
More recently the authors and their colleagues published a series of studies examining the activities of governmental and non-governmental policy actors in Canada which has helped push the frontiers of knowledge on these subjects forward. These studies have joined others in probing the backgrounds and activities of professional policy analysts in government (Bernier and Howlett 2011;Howlett and Newman 2010;Howlett and Wellstead 2011;Howlett and Joshi-Koop 2011); those working for NGOs (Evans and Wellstead 2013); ministerial staffers (Eichbaum and Shaw 2007;Shaw and Eichbaum 2012;Connaughton 2010;Fleischer 2009); policy consultants (Saint-Martin 1998a;1998b;Speers 2007;Perl and White 2002) and many other prominent members of national and sub-national policy advisory systems (Dobuzinskis et al. 2007;Halligan 1995;Craft and Howlett 2012a).
Consistent with the pattern found in the UK by Page and Jenkins (2005), Australia (Tiernan 2011), New Zealand (Eichbaum andShaw 2011), and Ireland (Connaughton 2010), these studies have found most policy workers in Canadian government to be engaged primarily in process-related tasks and activities. However, the work published to date has several limitations. First, although it has distinguished between regional and central level activities (Wellstead et al. 2009;Wellstead and Stedman 2010) and has found some significant variations in analytical tools practiced at these levels, it has generally not distinguished very carefully between different organizations and functions of government within departments and units (for an exception to this rule see Howlett and Joshi-Koop 2011).
Second, it has generally explored differences between government-based and non-governmental analysts and analysis, without taking into account the activities of the 'third set' of so-called 'invisible' analysts (Speers 2007); that is, the ever-growing legion of consultants who work for governments on policy matters, in some cases supplanting or replacing internal analysis and analysts (Howlett and Migone 2013;Momani 2013;Lindquist and Desveaux 2007). A more complete picture of policy formulation tools and the roles played by policy analysts in these venues is needed if the nature of contemporary policy work, analytical techniques and formulation activities is to be better understood.
This chapter addresses both these concerns. First, it briefly summarizes the results of published national and sub-national surveys conducted in 2006-2009 of internal Canadian policy analysts and sets out what is known about their formulation and appraisal activities, focusing on the techniques they employ in their work. Second, it re-examines the original datasets used in these studies to tease out their findings with respect to differences in the use of analytical tools across departments and functional units of government. Third, it draws on two new surveys of policy consultants and those who manage them completed in December 2012, and two surveys of NGO analysts conducted in 2010-2011, in order to compare and assess what kinds of tools are practiced by the private sector and nongovernmental counterparts of professional policy analysts in government.
Since the questionnaires used in the studies are almost identical, this data provides useful material that can start to fill out a comprehensive picture of similarities and differences across different venues for policy work. Combined, the data from these three studies provides more precise description of the frequency of use of specific kinds of policy formulation tools used in government and in other policy formulation venues outside government. As the chapter shows, the frequency of use of major types of analytical tool in policy formulation is not the same between the different sets of actors and also varies within venues of government by department and agency type. Nevertheless some general patterns in the use of policy appraisal tools in government can still be discerned, with all groups employing process-related tools more frequently than 'substantive' content-related technical ones, reinforcing the procedural orientation in policy work identified in earlier studies.

THE 'LUMPY' HYPOTHESIS: THE (UNEVEN) DISTRIBUTION OF POLICY ANALYSIS ACROSS GOVERNMENT
In his contribution to a 2007 book on the state of the art of policy analysis in Canada, the former head of the federal government Policy Research Initiative (Voyer 2007) suggested that the distribution of analytical capacities among government agencies was by nature 'lumpy' or uneven. That is, different units do not just have different supplies of analytical services -the usual subject of academic analyses -but also different demands. Therefore, in practice, not all units require the same capacity or capabilities in terms of policy analysis and aggregate measures of overall government capacity require nuanced application with respect to determining the needs and gaps encountered by specific agencies and activities.
It is also the case that the venues of policy research extend beyond the governmental confines which Voyer (2007) discussed. That is, policy analysis and advice is not the exclusive purview of professional analysts in government agencies but extends beyond them to the non-governmental sector in the form of analysis conducted by consultants and by a range of NGOs, including think tanks and research councils among others (Craft and Howlett 2012a). The distribution of capacities among nongovernmental policy workers is even less well understood than within governments -until very recently virtually the exclusive focus of research into policy work -and the relationships existing between the governmental and non-governmental components of policy advisory systems are also almost completely unknown.
A plausible hypothesis, however, is to suggest that Voyer's 'lumpiness thesis' within government can be extended to the external components of overall policy advisory systems. That is, given supply and demand conditions overall and within each organization, not only should the distribution of policy formulation tools, tasks and capacities be expected to vary across governments, but also across non-governmental analysts, and between governmental and non-governmental actors, as well. 1 In what follows empirical evidence from the above-mentioned three sets of surveys into the activities of professional analysts in government, policy consultants, and analysts working for NGOs in Canada undertaken by the authors over the period 2006-2013 is presented, along with data examining the distribution of capacities within government. This data allows us to examine for the first time the distribution of techniques across governmental and non-governmental venues in some detail.

DATA AND METHODS
The first set of surveys mentioned above focused on the activities of professional policy analysts employed by federal and provincial governments in 2006-2009. This set of 15 studies examined the behaviour and attitudes of core civil service policy actors in all senior Canadian 'policy bureaucracies' (Page and Jenkins 2005); that is, a 'typically' structured, Weberian, multi-level (federal) system of professional policy advice (Halligan 1995;Waller 1992). 2 Data on the federal government came from two surveys conducted in 2006-2007. The first was a census of 1937 people identified by members of the Regional Federal Council (an organization of senior federal civil servants located outside Ottawa) from all provinces and territories that undertook policy-related work. The second was a random sample of 725 National Capital Region-based (Ottawa-Hull) policy employees identified from the Government Electronic Directory of Services (GEDS) (Wellstead and Stedman 2010;Wellstead et al. 2009). The response rates were 56.8 per cent (n51125) and 56.4 per cent (n5395) respectively, giving a total sample of 1520 policy workers.
Provincial and territorial data were collected from each sub-national jurisdiction in 13 separate surveys conducted in late 2008 and early 2009. Respondents were identified from job titles listed in publically available sources such as online government telephone directories, organizational charts and manuals and members of Public Service Commissions (Howlett 2009;Howlett and Newman 2010). This yielded a population of 3856 policy-based actors and 1357 responses were received for a response rate of 35.2 per cent. The total population surveyed across the federal, provincial and territorial governments was thus 6518 with an overall combined national response rate of 2877 or 44.1 per cent.
While the survey questionnaires used in these studies were very similar, they were not identical and some questions relevant to this inquiry relating to tools of analysis were not included in the federal survey. Also the range of ministries and units varies by province and territory, meaning it is difficult to arrive at an aggregate depiction of intra-governmental structure required for the analysis. As a result, the largest single provincial case, Ontario, is used as a proxy for the provincial and territorial professional policy analyst community and occasionally for the federal or national levels as well. This is reasonable since (a) Ontario has by far the largest number of respondents in the survey so the results closely approximate the overall provincial and territorial findings and (b) separate analysis of the federal and provincial cases revealed a general pattern of close similarities between analysts working in the two levels of government (Howlett and Wellstead 2012). The second set of surveys was conducted in 2010-2011 to probe the situation with non-governmental analysts employed by think tanks and research institutes. Two surveys were conducted: (1) a government-based, 192 variable (45 questions) questionnaire, designed in part from previous capacity surveys by Howlett (2009) and Wellstead et al. (2009) and intended to capture the dynamics of NGO-government interactions; and (2) an NGO-based, 248 variable questionnaire (38 questions). Questions in both surveys addressed the nature and frequency of the tasks performed by analysts, the extent and frequency of their interactions with other policy actors, and their attitudes towards and views of various aspects of policy-making processes, as well as questions addressing their education, previous work, and on-the-job training experiences. Both also contained standard questions relating to age, gender, and socio-economic status. The survey was delivered to 2458 provincial policy analysts and 1995 analysts working in the NGO sector in the Canadian provinces of Ontario, Saskatchewan, and British Columbia. Four policy communities were selected for this survey: environment, health, immigration, and labour. The specific provinces and policy sectors dealt with in this study were chosen because they represent heterogeneous cases in terms of politics, history and economic and demographic scale.
Like the governmental studies, mailing lists for both surveys were compiled, wherever possible, from publicly available sources such as online telephone directories, using keyword searches for terms such as 'policy analyst' appearing in job titles or descriptions. In some cases, additional names were added to lists from hard-copy sources, including government organization manuals. Based on preliminary interviews with NGO representatives, it was clear that many respondents undertook a variety of non-policy related tasks in their work. As a result, the search was broadened to include those who included policy-related analysis in their work objectives. Due to the small size of both study populations, a census rather than sample was drawn from each. The unsolicited survey in January 2012 used Zoomerang ® , an online commercial software service. A total of 1510 returns were collected for a final response rate of 33.99 per cent. With the exception of the NGO respondents on labour, the percentage of respondents corresponded closely with population expectations developed by the authors.
The third set of surveys was conducted in 2012-2013 to assess the activities of external consultants hired by governments. Two surveys were conducted, one of government managers involved in contracting consultants and the other of consultants themselves. Both were surveyed in order to help understand how consultants' policy advice is solicited, developed, transferred, and used in the context of the Canadian policy advisory system. The consultants' survey was administered to representatives of companies that had performed policy work for various levels of government in Canada between 2004 and 2012. The consultants were identified through sampling of over 34,000 contracts from 10,000 companies contained in the federal government's Proactive Disclosure database of procurement contracts.
The consultants' survey contained 45 questions on similar subjects as the earlier federal, provincial and NGO surveys and was administered on line (SurveyMonkey®) in December 2012 to 3228 email addresses obtained for consulting firms involved in policy work. Three hundred and thirty-three complete responses and 87 partial ones were received for a total of 420 responses and a response rate of 13 per cent. Like the NGO study, the consultant survey questionnaire was designed to replicate as far as possible the exact questions asked of federal, provincial and territorial permanent policy analysts by the authors in 2009-2010 in order to allow meaningful comparisons between these actors and others in the Canadian federal policy advisory system.

FINDINGS
In what follows, some of the results of the three surveys are presented. The first set of findings is derived from the federal/provincial/territorial survey and deals with the original 'lumpiness' hypothesis concerning the expectation of analysis and analytical tools varying by venue or location within government. The second set of results addresses the situation of non-governmental policy workers.

The Distribution of Capacities within Government: Venues and Tools
In general, most studies of the use of sophisticated policy analytical tools and techniques in government have highlighted that such use requires several pre-conditions to be met. On the supply side, agencies undertaking such analyses require (a) access to high quality quantifiable data or information (Vining and Boardman 2007) and (b) the human resource and managerial capability to both demand and supply such analysis (Howlett 2009). But not all agencies meet these criteria or have not done so at all times and in all circumstances. Since existing studies have not examined each agency in detail, as pointed out above, exactly which kinds of agencies exhibit strength in which areas is uncertain and under-explored.
Furthermore, on the demand-side, not all departments have the need for the same kinds of data and information and therefore can also be expected to exhibit a different pattern of use of specific analytical tools. Thus for example, some agencies like Finance or Treasury Board typically deal with relatively easily quantifiable issues (budgets, revenues and expenditures respectively) usually with plentiful historical and contemporary data assumed to be very accurate and precise, and are well resourced and able to hire staff or consultants who are interested in and can utilize this kind of evidence. They have always employed highly technical forms of analysis and are likely to continue to do so into the future. Other agencies, however, such as those dealing with social or environmental policy deal with less quantifiable or contested data and may not be interested in or able to use the kinds of information that other agencies utilize. Still others fall in between -for example, many health or housing or transport agencies -who may have high quality data available but may only use it at some times but not others. And finally others may not have access to the data they need even if they are willing and are potentially or actually capable of using it (Howlett and Joshi-Koop 2011;Craft and Howlett 2012b).
The survey of provincial and territorial officials provides some insight into this question. The top ten policy-related analytical tools employed by policy analysts for five selected departments in the Ontario case are shown in Table 8.2. Brainstorming (91.2 per cent) is the most used and the analysts working on environmental issues tend to use this tool the most (94.8 per cent). Consultation exercises come a distant second at 76.3 per cent, with analysts working on education issues using this tool the most at 82.1 per cent. Risk analysis and checklists are ranked third and fourth respectively with health analysts (74.3 per cent) and environmental analysts (70.7 per cent) the most frequent users.
Cost-benefit analysis and scenario analysis, often thought to be fundamental tools employed in policy analysis, are in fact ranked fifth and sixth, although, not surprisingly, finance departments are the top users for both (74.3 and 63.5 per cent respectively). The next highest-ranked tool is expert judgement and elicitation, used the most by the environmental department (63.8 per cent). Finance departments also, not surprisingly, use financial impact analysis (73 per cent) and cost-effectiveness analysis (58.1 per cent) the most in their field of work. Focus groups are rarely used by such units (27 per cent) but are much more commonly employed by education analysts (46.3 per cent).
There are thus distinct differences across intra-governmental policy venues with respect to the kinds of analytical tools used. Finance is the dominant user of every 'technical' type of analysis except risk analysis and scores low on 'consultation' activity and other 'soft' tools, while transportation scores lowest on both measures. Environment scores lowest on most 'hard' tools and high on tools such as expert elicitation. Education is also low on most 'hard' tools although it is higher on financial impact analysis and health is low on most tools although high on the use of risk analysis.
This suggests, as Voyer (2007) intimated, that governmental units have their own particularities and needs. But some general conclusions can also be drawn from these figures about the nature of hard/soft tools used, based on the general nature of the tasks each unit is assigned. That is, this evidence suggests that differences in the distribution (supply and demand) for analysis can be traced back to the fundamental task or mission of each agency. This is very much along the lines Voyer (2007) initially suggested. The three tables above provide additional evidence of this supposition. Table 8.3 looks at the entire provincial and territorial dataset and finds differences in the use of tools of evidence-based or evidence-informed policy analysis among six major activity areas with more of this kind of activity found in health, the field where the idea of evidence-based policymaking originated. Table 8.4 looks at several aspects of the tasks faced by analysts in different units and finds significant variations across sectors. Finally, Table 8.5 provides a self-assessment made by the analysts

The Overall Distribution of Capacity between Governmental and Nongovernmental Actors
In this section we address the larger, extended, version of the Voyer thesis; that is, we extend the analysis of tools and venues for policy formulation beyond different units of government to address differences in capacity and techniques across different venues outside governments. Here the two key groups to be compared with professional analysts inside government are professional consultants who worked on a temporary contract basis for governments, and analysts located in the NGOs with whom government officials, and consultants, interact. This analysis begins by comparing the backgrounds and training of the two groups of internal and external advisors. Comparing the level of formal education between analysts and consultants and NGOs, about 75 per cent of the policy consultants have a graduate or professional degree, with 23 per cent having only a lower-level university degree. This is much higher than the internal part of the professional analytical community where about 56 per cent of the policy analysts have some graduate or professional education. For those working in NGOs, the level of formal education is evenly split relative to the analysts and consultants at 51 per cent with a senior degree and 44 per cent with a lower-level one (Evans and Wellstead 2013). This suggests that the range of qualifications found in the internal and external parts of the professional analytical community differ, with policy consultants tending to be more qualified (based on graduate and professional accreditations) than policy analysts in government or those working for NGOs. The level of formal education can influence the type of policy tools which are used in formulation. More important than general educational level, however, for our purposes, are differences in specialized training in specific subjects such as public policy and, especially, policy analysis and evaluation. Here the differences between internal and external analysts were less obvious as about 40 per cent of policy consultants and about the same number of policy analysts in government had taken three or more policy-related courses at the post-secondary level. However, only 20 per cent of the NGO policy workers surveyed had done similar courses. Almost 70 per cent of NGOs, compared with 47 per cent of policy consultants and 58 per cent of governmental policy analysts did not have any specific post-secondary courses on formal policy analysis or evaluation.
The areas of training also differ. Policy consultants tend to have a university degree in economics, business management, engineering, political science and public administration, with these five fields accounting for about 85 per cent of degrees (allowing for multiple degrees) conferred. In comparison, the five leading degree fields of internal policy analysts were political science, business management, economics, public administration and sociology, in that order. These five fields accounted for about 60 per cent of degrees (allowing for multiple degrees) conferred, while a wide range of other social science, law and humanities degrees accounted for another 40 per cent of credentials (Howlett and Newman 2010). The top five fields for NGOs, on the other hand, are general social sciences, business management, arts and humanities, political science and public administration (Evans and Wellstead 2013).
There are similarities in these fields of study, of course, as business management features highly in all three, but overall many analysts in government tend to be educated in political science and public administration, consultants in economics and analysts working for NGOs in sociology. This suggests a certain amount of self-selection by intellectual orientation among analysts employed in each area. However, it also highlights the lack of training in all venues encompassing areas such as the natural sciences, engineering and law, which are often thought to account for a sizable component of all three groups.
Further survey questions inquired into specific aspects of the organization of policy work in each area. Policy consultants (84 per cent) and NGOs (68 per cent) tend to work in groups of one to five, while only 10 per cent of consultants and 15 per cent of NGOs work in groups of six to ten (Evans and Wellstead 2013). This is in contrast to policy analysts in government where almost 65 per cent work in units of fewer than ten employees and 30 per cent in units of fewer than five full-time equivalent employees (Table 8.6) (Howlett and Newman 2010). This suggests that whatever skills consultants and NGO workers have individually represents the sum of the policy formulation tools which they can bring to bear on a subject, while policy analysts in government, not surprisingly, are much better resourced as a team. This variation in organizational capacities is reflected in the kinds of roles or tasks taken on by different group members. While this question was not asked of NGO members, policy consultants and analysts share similar types of roles but not with the same frequency. Policy consultants, for example, take on the roles of advisor (62 per cent), analyst (58 per cent), and researcher (50 per cent) in their respective consultancies, while for policy analysts the advisors make up 80 per cent, the analysts 74 per cent and the researchers only 41 per cent. The top three policy-related tasks which policy consultants undertake include research and analysis (83 per cent), providing advice (77 per cent), and providing options on issues (61 per cent). Besides policy development, however, policy consultants have to fulfill functions of project management (48 per cent), communications (41 per cent), and programme delivery (36 per cent). Policy analysts in government are more focused and very high percentages of analysts undertake research and analysis (93 per cent), provide advice (92 per cent), and prepare briefing notes or position papers (91 per cent). In comparison, NGO-based analysts most commonly consult with stakeholders (96 per cent), identify policy issues (94 per cent), and consult with decision makers (91 per cent) (Evans and Wellstead 2013) (see Table 8.7).
When it comes to their preferred analytical tools, this question was only asked of consultants and analysts in government and not of NGO respondents. From a list of 20 policy-related analytical tools, the top two employed by policy consultants are brainstorming (70 per cent) and consultation exercises (67 per cent), much the same as policy analysts. However the third choice is quite different and revealing, with focus groups (57 per cent) being the third most used tool among consultants rather than risk analysis (68 per cent) as it is for analysts (Howlett and Newman 2010) (see Table 8.8).
A fuller description of the tools used by each group of analysts and a comparison of similarities and differences is set out in Tables 8.9 and 8.10.

CONCLUSION
Until recently, only very weak, partial, dated, and usually anecdotal information existed on the situations found in different government and nongovernmental venues with respect to the activities of the policy analysts found in these locations.
In the case of the US, Arnold Meltsner (1976) long ago observed that analysts undertook a number of roles in the policy-making process but emphasized their specialist training and expertise in sophisticated methods     (2000), Nancy Shulock (1999) and Sean Gailmard and John Patty (2007), however, argued that the use of such techniques was exaggerated and that many analysts engaged more often in more process-related activities.
In the United Kingdom and Germany as well, contrary to the early picture of carefully recruited analysts trained in policy schools to undertake specific types of microeconomic-inspired policy analysis presented by Meltsner (Weimer and Vining 1999), investigators such as Edward Page and Bill Jenkins (2005) and Julia Fleischer (2009) found that British and German policymaking typically featured a group of 'policy process generalists' who rarely, if ever, dealt with policy matters in the substantive areas in which they were trained and had very little training in formal policy analysis. The extent to which this average picture accurately described the situation in all venues within a country and within governments, however, has remained an open question until now.
Overall the data presented in this chapter display a picture of government, as a whole, exhibiting an uneven distribution of capacities and technical capabilities and utilization practices across different organizational and thematic venues. The data show that some departments and agencies -such as Finance -enjoy favourable circumstances which allow them to practice sophisticated analytical techniques while others may only meet these criteria from time to time depending on various factors or their task environments. Important here, for example, is the nature of the internal and external training analysts receive, their job expectations and work descriptions, the nature of the issues and tasks they commonly face in their work, and managerial demands and leadership.
Some of this unevenness within government can be offset through the use of external consultants or reliance on NGOs to provide analysis, and new data presented in this chapter suggest that the capacities and techniques of analysis practiced by analysts in government consulting and in non-government venues are indeed different from those found internally. Formal education levels, disciplinary background and policyrelated training are not the same in venues outside of government as they are internally. There are some signs of a complementary relationship between internal analysts and consultants, as in general the consultants are better educated and trained relative to analysts and are able to bring a different skill set to formulation processes (Lindquist and Desveaux 2007;Lindquist 2009). The NGO sector, on the other hand, is very underdeveloped by comparison with either group and is unlikely to replace or supplement either.
The existence of such internal and external distributions of capacities and analytical practices is a situation which has significant implications for policy formulation in government and for the role played in it by advice stemming from the NGO and private sector. Although the full implications of these differences in tool use and policy work across venues remain to be spelled out, they suggest a pattern, in Canada at least: of increasing sophistication in analysis and policy work as one moves from the non-governmental sector to the governmental one, and within government from more socially involved agencies to more economically oriented ones, with policy consultants able to augment internal activities. While additional cross-national studies are needed to determine how common this pattern is, it is compatible with most of the limited work done to date examining the situation with respect to policy advice, policy formulation and the utilization of analytical techniques in countries such as the Netherlands, Australia and New Zealand, the UK and the US. 3 NOTES 1. A subordinate hypothesis would be to expect that some aspects of non-governmental capacities could be used to bolster gaps in the governmental level, and possibly viceversa, so that the relationship between the two components of the policy advisory system would be a complementary, synergistic one, rather than a purely duplicative or redundant one. Thus as John Halligan suggested: The conventional wisdom appears to be that a good advice system should consist of at least three basic elements within government: a stable and reliable in-house advisory service provided by professional public servants; political advice for the minister from a specialized political unit (generally the minister's office); and the availability of at least one third-opinion option from a specialized or central policy unit, which might be one of the main central agencies. (Halligan 1995, p. 162) This is a subject of another research project currently underway among the authors. 2. A Westminster-style parliamentary democracy, Canada features a very decentralized form of federalism in which ten provincial (and to a lesser extent, three territorial) governments exercise exclusive control over significant areas of governmental activity including education, urban affairs, healthcare, natural resources and many important social welfare programmes (Howlett 1999). Other important areas such as immigration, agriculture, criminal law and environmental policy are shared with the federal government. While the territorial governments and some provincial ones -such as Prince Edward Island with a population of only 140 000 -are quite small, others such as the Province of Ontario (population 13 000 000) are as large, or larger, than many national governments. Given this circumstance, data were collected from two online sets of surveys: one covering federal employees and the other covering the provincial and territorial governments. 3. See above on the US and the UK. Similar findings have been made in the cases of the Netherlands, Australia and New Zealand, by Robert Hoppe and Margarita Jeliazkova (2006), Patrick Weller and Bronwyn Stevens (1998) and Jonathan Boston and his colleagues (1996), respectively.