Evaluation Report - Participant Funding Program

Evaluation Report - Participant Funding Program [PDF 882 KB]

Evaluation Report
Participant Funding Program
National Energy Board
Approved by the Chair and CEO | January 4, 2016

Copyright/Permission to Reproduce

Table of Contents

• List of Acronyms
• Executive Summary
• Part 1. Background
  • 1.1 Evaluation Rationale
  • 1.2 Program Background
• Part 2. Evaluation Approach and Methodology
  • 2.1 Evaluation Objectives
  • 2.2 Methodology
  • 2.3 Limitations
• Part 3. Findings
  • 3.1 Relevance
  • 3.2 Effectiveness
  • 3.3 Efficiency and Economy
• Part 4. Conclusions and Recommendations
  • 4.1 Demonstration of Relevance
  • 4.2 Demonstration of Effectiveness: Achievement of Outcomes
  • 4.3 Demonstration of Efficiency and Economy
• Annex 1: PFP Logic Model
• Annex 2: History of the Participant Funding Program
• Annex 3: Evaluation Matrix
• Annex 4: Management of the Evaluation
• Annex 5: References
• Annex 6: List of Hearings Eligible for Participant Funding
• Annex 7: Management Response

List of Acronyms

Executive Summary

Background

This evaluation report presents the findings, conclusions and recommendations of the National Energy Board’s (NEB, Board) Participant Funding Program (PFP or “the program”) evaluation. The evaluation examines the program’s relevance, effectiveness (performance), efficiency and economy for the period of June 2010 to June 2015 during which the initial Terms and Conditions (2010) of the PFP were applicable. The evaluation is mandatory and meets the requirements of the Financial Administration Act^{Footnote 1}, the 2008 Treasury Board Secretariat (TBS) Policy on Transfer Payments^{Footnote 2} and the 2009 TBS Policy on Evaluation^{Footnote 3}; all of which require an evaluation of grants and contributions every five years. The evaluation was undertaken between November 2014 and September 2015 and focused on the achievement of immediate and intermediate outcomes as well as program design and implementation.

This evaluation analyzes data from the past five years of the PFP, obtains feedback from internal and external stakeholders and examines relevant documentation in order to assess outcomes achieved as well as challenges and opportunities in program implementation.

Key Findings

Relevance

The Participant Funding Program is aligned with the roles, responsibilities and priorities of the federal government as well as the mandate and priorities of the Board. There is a continuing need for PFP as a result of the National Energy Board Act (NEB Act) and Canadian Environmental Assessment Act, 2012 (CEAA 2012). Stakeholders also agree that the funding program is needed though opinions were divided on whether the program should maintain its current budget for contributions given its purpose of being a supportive yet modest funding program or whether it should establish an increased level of participant funding overall for the program, therefore changing its purpose.

Achievement of Outcomes

The Participant Funding Program demonstrates that it takes steps to make the program accessible to potential applicants and it promotes fairness and transparency through a structured application process and publically reporting on the results of funding opportunities.

For example, the PFP mainly uses a section of the NEB website to communicate information about the program such as current funding opportunities and funding reports. The NEB also creates and disseminates other public information materials to target applicants, however some updates are required. Engagement sessions conducted by other staff with Aboriginal groups or open houses are also used to promote the program as well as a newspaper campaign to launch each funding opportunity.

Internal stakeholders generally support the objectives of the PFP and have the information they need in their work. External stakeholders agree that contribution awards help to off-set the cost of participating in hearing. Stakeholders generally agree that the program is accessible and fair, but some have found that the funding process could be adjusted to allow more time to prepare for a hearing. As well applicants that do not receive a funding award suggest that a more comprehensive reason should be provided.

The PFP determines funding for contributions on a project-by-project basis based on information such as the size and location of the project, the number of Aboriginal groups in the area, available resources and the funding level set for similar projects in the past. Based on data from the program and finance, the budget for contributions each fiscal year has been adequate to make payments to recipients and the PFP re-profiles the balance of unused contributions to the next fiscal year when the demand for payments is anticipated. The program demonstrates that it takes action to substantially adjust the hearing envelope for hearings that involve more affected people, greater complexity and a larger project scope than past hearings. This requires a formal request for additional funds through the Treasury Board Secretariat.

Most recipients of participant funding awards participate in the hearing and access their funding. The PFP has tried different methods to determine whether the program supports participation in a hearing such as feedback surveys within applications and invoices as well as having a socio-economic specialist from the Applications Business Unit summarize the nature and level of participation of funded applicants. These methods, though, have not been consistent and stakeholders suggest that determining adequate levels of participation is challenging because it is subjective. There is an opportunity to consistently compile and analyze data to determine trends in PFP related hearings and to support assessment of PFP outcomes.

Design and Implementation

The program has developed tools as well as a process map to guide program processes as well as templates and informational material for the public. Considerable efforts have been made at the five year mark of the program by current staff to gradually document and update key program material and in light of the new PFP Terms and Conditions (2015). Additional work is needed to create a new logic model and corresponding performance framework with outcomes that are measurable. This will help the program to position itself for consistent data collection and allow it to analyze and update the program as required.

Over the past five years the program’s approach has been to set a funding envelope and then either approve awards following the application deadline or review applications individually or in batches as they come in. Firm deadlines have established greater certainty in the process for applicants and program staff whereas processing applications as they are submitted may have helped to spread out the workload. More recently the PFP has introduced funding caps for individuals or groups in order to set modest expectations for applicants in their funding request. While this approach may help to manage the workload of the program, comments from internal stakeholders, including members of the funding review committee, state that over the past five years, when reviewing applications on a rolling basis without knowing the full demand for the particular funding opportunity makes it more challenging to allocate limited dollars.

There is variation in the timing of PFP in relation to other key hearing activities and stakeholders suggest that timelines and the approach to the Application to Participate (ATP) in relation to the Application for Funding should be examined in order to make improvements to the timeliness and accessibility of the PFP.

Finally, recipients of awards must be granted intervenor status through the Application to Participate process, which sometimes occurs before, during or after the PFP process. While the newer application form requests the applicant to state whether they already know they are an intervenor, the PFP’s process map does not include the steps to be taken by staff in confirming the participation status of a given applicant. This is important given that funding is contingent on the participation level granted by the Board.

Efficiency and Economy

The Participant Funding Program appears to be improving the efficiency of program delivery given the working context and works towards solutions to continuously improve. For example the program updated its delegated financial signing authority matrix to improve the timing of some of its activities.

Hearings overlap fiscal years and personnel and O&M costs may be incurred in a different year than when the awards are offered, signed or paid. Thus assessing whether the program used its resources economically for each funding process would be limited in its usefulness when analyzing whether the program operates in an economical manner. From the data though, the cost of the Funding Review Committee (FRC) makes up most of the operating and maintenance (O&M) expenditures and individual hours and billing are variable but overall the cost of a FRC is minimal (between 0.1% and 5%) when compared to the total funding requested or awarded per funding opportunity.

The PFP is a small program within the Applications Business Unit and operates on its own budget (fiscal year) set in 2010 for direct costs (contributions) and indirect costs (personnel and operating and maintenance). It is supported by two full time staff - one funded by the PFP personnel budget - the other through the Applications Business Unit. The need for the additional full-time equivalent (FTE) resource was required to allow the program to manage the variable workload. Thus the original budget has not been adequate to support the required staff to deliver the program. As well, although the program has budgeted its indirect costs to total program costs within its target of 13%, fewer contribution payments made then budgeted per fiscal year together with an extra FTE makes it appear that the actual program administration costs are a large share of the program’s overall budget.

Due to the nature of the program, the program manager has had to manage the budget and cash flow by both fiscal year and project. Recent changes to the Terms and Conditions (June 2015) of the program expand its scope and flexibility. For example the Funding Review Committee is no longer mandatory and other types of hearings will also be required to offer funding. Work, though, remains to clarify and implement these particular changes introduced by the Terms and Conditions.

Data and analysis related to advertising PFP in newspapers (which is not paid for by the PFP budget) was also found to be missing for the past five years. The evaluation also noted that assessing the level of effort for the program by quantifying the hours of work by staff in relation to the program is at best an estimate, since staff do not accurately record their hours of work. Recording work related to the PFP as accurately as possible is important so that the program can assess the level of effort involved on operating the program and its related activities.

The program has not had a useful electronic tool to manage information on each funding opportunity, for each applicant and overall data thus creating an unnecessary administrative burden. The records management system (RDIMS) is not the right fit for a contributions program. As a result, the program has some incomplete and inaccurate records and data. This has an effect on the ability to analyze information and to contribute to performance management. The program has already identified this risk and has started to work on a new IM/IT solution but there is still further work and time required for it to be implemented.

Recommendations

1 It is recommended that in response to the new Terms and Conditions (June 2015), a logic model should be developed and corresponding performance measures that are SMART (specific, measurable, attainable, relevant and timely). Performance measures should be consistently tracked and reported. The program should also develop an approach to monitoring contribution agreements for compliance with terms and conditions in the agreement (e.g. the recipient’s deliverables).

2 It is recommended that the staff involved in the PFP and Hearings work together to assess the hearing process timelines and key deliverables in order to realize efficiencies in the process and to allow for better planning of PFP and ATP. Integration of timelines for the PFP and ATP within the hearing process to the extent possible as well as consistency in the timing of key activities would help to ensure a fair, timely, accessible and transparent funding program.

3 It is recommended that the PFP work with other business areas (e.g. Applications, Communications and Finance) on a continuous basis to articulate their shared processes and to update and maintain internal and external documentation related to PFP in order to ensure that the program is accessible and communicated consistently both internally to staff and to the public. Good communication and coordination of information is important since the program requires assistance and advice from other internal services at the NEB. As well, frequent changes to the program design demands clear and consistent information to staff and the public.

4 It is recommended that the PFP assess its responsibility for budgeting and managing its communications costs and examine its approach for effectiveness in reaching eligible participant groups. Communications (including advertising) are an indirect cost to operating the program and it is important to assess various approaches to ensure that the resources are adequate and are used in a way that adds value and contributes to achieving program objectives.

5 It is recommended that the PFP document its approach for determining its use of an external FRC. This guideline should have clear rules to ensure the predictability, transparency and fairness of the program and its process for reviewing applications and recommending awards. In developing an approach, the program could assess the level of effort and cost to coordinate a Funding Review Committee and determine the situations where it might be economical to use an FRC.

6 It is recommended that the program make further improvements to efficiencies in its process overall by implementing a unique and centralized system to collect, analyze and report information related to specific projects and the program overall. Such a system could also improve workflow and control of the process and program documentation.

7 It is recommended that the Participant Funding Program use available data to track and monitor the level of contributions required as well as indirect costs (including all roles that are critical to operating the program). Tracking and monitoring all direct and indirect costs will allow the program to better assess its budgetary needs over time.

1 BACKGROUND

1.1 Evaluation Rationale

This is the first evaluation of the Participant Funding Program at the Board and presents several findings, conclusions and recommendations. The evaluation examined the program’s relevance, effectiveness (performance), efficiency and economy during the period June 2010 to June 2015. Given that the program is still in its early years, the focus of the evaluation was on the achievement of immediate and intermediate outcomes. As well, the evaluation responds to the request from PFP management to assess whether internal stakeholders are aligned with PFP objectives and have the information they need. The evaluation was substantially carried out between November 2014 and September 2015.

The evaluation report is organized as follows:

• Section 1: Background
• Section 2: Evaluation Approach and Methodology
• Section 3: Findings
• Section 4: Conclusions and Recommendations

Annexes 1 to 7 contain additional information such as a logic model (that illustrates inputs, outputs and outcomes of the PFP), history of PFP, evaluation matrix, references, management of the evaluation, list of projects and hearings eligible for participant funding and the management response.

1.2 Program Background

The National Energy Board is an independent federal, quasi-judicial regulator established in 1959 to promote safety and security, environmental protection and economic efficiency in the Canadian public interest within the mandate set by Parliament for the regulation of pipelines, energy development and trade. Its main responsibilities are established in the National Energy Board Act. Certain pipeline project applications to the Board trigger a public hearing. The NEB issues a Hearing Order providing details about the hearing process and how persons who are directly affected or who have relevant information or expertise may apply to participate. Participants are given the opportunity to make their case by submitting evidence on the hearing record and to appear at the oral portion of the hearing in order to provide their views and evidence. This informs the decision-making about the proposed project under review. The final decision whether or not to approve the application for a major pipeline project is made by the federal government (Governor in Council) based on the recommendation from the Board.^{Footnote 4}

The Board established the PFP in 2010 to facilitate the participation of the public in hearings with an oral component held under the National Energy Board Act^{Footnote 5}. The initial Terms and Conditions were approved by the Treasury Board Secretariat (TBS) on 24 June 2010 and the program was launched in September 2010. The Terms and Conditions of the program were recently modified on 2 June 2015.

The purpose of the PFP at the National Energy Board is^{Footnote 6}:

• To ensure more timely processes and meaningful public engagement in project reviews
• To enhance the quality, thoroughness and credibility of the reviews, and reduce the risk of time-consuming and costly delays because of challenges due to adequacy of process
• To enable the exercise of the Canadian Environmental Assessment Act, 2012 (CEAA 2012) on the part of the NEB

The objective of the PFP at the National Energy Board is^{Footnote 7}:

• To facilitate effective public participation ensuring that public concerns and values are taken into consideration during the NEB’s hearing process.

The following groups or individuals are eligible to apply for participant funding:

• landowners or individuals living on or near the proposed project area
• unincorporated non-industry not-for-profit group or association
• incorporated non-industry not-for-profit organization;
• Aboriginal groups (e.g., First Nations, Métis, or Inuit)
• other affected groups or individuals (e.g., community organizations that would be affected by the project)

Eligible applicants must also demonstrate in their application that they meet one of the following:

• a direct, local interest in the project, such as living or owning property near the project area;
• local community insights or Aboriginal traditional knowledge about the proposed project;
• an interest in potential project impacts on treaty lands, settlement lands, traditional territories or
• related claims and rights; or
• plans to provide expert information relevant to the proposed project.

Another mandatory requirement for those with a funding award is that they must also apply for and obtain status as an intervenor in the hearing. This is a separate and independent decision-making process carried out at the NEB to determine who may participate in a hearing and how. Intervenors may file written evidence, ask questions and provide final arguments.^{Footnote 8}

For further illustration of the program outcomes, outputs and inputs, refer to the PFP Logic Model in Annex 1.

1.2.1 History and context

The National Energy Board officially launched its Participant Funding Program in late 2010; however the funding program was contemplated much earlier. A request from the Minister of Natural Resources Canada in late 1995 to review the best method to introduce a funding program resulted in a report a few months later and presented some options to consider under the legislative framework at that time.^{Footnote 9}

The Participant Funding Program, modeled on CEEA’s program, had the goal of providing a modest level of funding to facilitate participation of intervenors in an oral hearing. It is not intended to cover all the costs of participation and the NEB encourages eligible recipients to collaborate with others and seek other additional sources of funding.

The new Terms and Conditions approved in June 2, 2015 introduce some key changes to the scope of the program including:

• expanding the eligibility of hearings to include environmental assessments of designated projects (as defined by CEAA 2012) for which the Board is the responsible authority as well as written facilities hearings;
• the Funding Review Committee will be optional; and
• applicants have to demonstrate that they plan to provide value-added information that contributes to a better understanding of the potential impacts of a project.

The PFP has also introduced recent changes in the design and delivery of the program such as:

• the program no longer requires recipient costs to be classified by a priority level
• eligible costs will be reimbursed from the date the company files its project application with the NEB rather than the effective date of the contribution agreement.
• expected deliverables from the recipient are listed in the contribution agreement

Annex 2 also summarizes the history of PFP and some key actions and changes related to the program.

1.2.2 Program delivery

When the program received approval in 2010, it first operated out of the Office of the Secretary of the Board in order to operate arms-length from regulatory processes. There were a few projects in the first years of the program with few applicants for funding; however this changed in 2012 with the Enbridge Line 9B project application. Around the same time, due to the increasing complexity of hearings and the introduction of time limits, the NEB decided to situate the PFP and its staff within the Applications Business Unit. This was to allow the program staff and those responsible for managing the hearing process to work together to manage their respective process timelines. The program maintains its independence from the project application review process, by working exclusively with Hearing Management on process and with engagement staff to provide information about PFP that can be shared with the public. The PFP is the only transfer payment program administered by the Board.

A full-time manager and coordinator manage the PFP. A small Steering Committee of four staff, comprised of senior NEB staff and the PFP Manager, provides strategic and technical advice to the program and its delivery. Other staff (e.g. legal counsel) is invited as required to join the meetings and certain discussions.

The Steering Committee, recommends a funding envelope for each project based on certain criteria (Table 1). In the past, the Chief Operating Officer (COO) would approve the funding envelope; however with updates to delegated signing authority in 2012 and 2015, the leader of the Applications Business Unit is now responsible for this role. Once the funding envelope is approved, it is announced publically.

Table 1: Funding Level Criteria

The PFP Manager establishes a Funding Review Committee (FRC) by drawing from the pool of external members approved by the Executive Vice-President, Regulatory. The Committee usually consists of at least three people, including the PFP Manager and at least one person not connected to government. Committee members must be independent, which means they must have no interest or financial stake in the proposed project. The FRC reviews and evaluates the merit of each application and prepares a report recommending how funding should be allocated. They invoice their time to the program and are all paid the same per diem (pro-rated per hour).

While the authority for PFP awards resides with the Executive Vice-President, Regulatory, most funding recommendation memos were addressed to the Chief Operating Officer when approving awards. PFP processes and templates have since been updated to align with the revised Terms and Conditions and Financial Delegation Matrix approved in June 2015.

Originally, the PFP required applicants to prioritize costs by high, medium and low in accordance with a chart provided in the PFP Funding Guide^{Footnote 11}. For example legal, expert and travel costs were given high priority whereas translating information was considered medium priority. High priority costs were to be paid first followed by medium and low if there were funds remaining from the applicant’s award. In December 2014 the program updated its Funding Guide1^{Footnote 12} and removed the distinction between priority costs. Instead it introduced funding maximums for each type of eligible recipient.

Contribution Agreements are sent to applicants that receive an award. For the last five years, the Contribution Agreement only allowed reimbursement of expenses incurred after the agreement was signed by both the recipient and the NEB’s Vice President, Applications (now changed to the Director accountable for the PFP). In 2015, the Contribution Agreement template was revised to allow reimbursement of eligible costs incurred from the date the company files the project application with the Board to the date the hearing record closes. This approach ensures that the eligible costs are based on the same date for all applicants versus when their agreements are signed. As mentioned before, the participant must also obtain intervenor status if they haven’t already for the contribution agreement to remain valid.

To receive payment for eligible expenditures, recipients are required to use the claim template provided by the program and submit their request, typically within 60 days from the date the hearing closes. The program assesses the claims and supporting documentation for eligibility and then makes a payment request through the Finance group at the NEB. The original Terms and Conditions of the PFP required the Vice President, Applications to certify payments according to the Financial Administration Act (FAA). The new Terms and Conditions (June 2015) for the program now place this responsibility with the PFP Manager. The program indicates that this reduces the time to approve payments.

1.2.3 Resources

The PFP currently has an annual budget of just over $1.5 million, which includes $1.3 million of funding set aside for contribution payments and the rest for one full-time staff member and operating and maintenance costs (O&M). See Table 2 for the breakdown. Operating and maintenance funding is used for the following: Funding Review Committee fees, consulting services, temporary help and training and travel for PFP staff.

Table 2: Budget set in 2010^{Footnote 13}

The budget for contribution payments in FY 2012-13 was planned in 2010 at a significantly higher level due to an anticipated project and higher level of public participation expected for that time; however this particular project application was not submitted, thus the program requested and received approval from Treasury Board to re-profile the budget for contributions to future years to meet the demand for other eligible hearings.

Table 3 below reflects the updated budget for contributions.

Table 3: Modified Budget for Contributions^{Footnote 14}

Table 4: Actual Expenditures^{Footnote 19}

The expenditures on personnel, as shown in Table 4 above, are higher than budgeted since 2012-13 because it includes both the manager and coordinator position. Although the PFP coordinator position is funded by the Applications Business Unit, it is considered a necessary role to operate the program.

The budget for indirect costs (Table 2) compared to the overall budget for the program (using the modified budget for contributions from Table 3) shows that the program was in line with its target of budgeting 13% of indirect costs to overall budget. However, as shown below in Table 5, the actual indirect cost versus total actual spending has exceeded the 13% target. This is partially due to including the second FTE position in the actual expenditures and there were fewer contribution payments than what was budgeted for.

Table 5: Comparison of indirect costs to total program cost (budgeted and actual)

In FY 2012-13 the program spent a greater level of its O&M budget than previous years in order to advertise for new members to its Funding Review Committee. The PFP advertising costs for each project, however, are paid by the overall hearing budget and not by the program’s budget. Table 6 below provides another way to examine contributions, and shows the amounts awarded and signed versus paid each fiscal year. This table demonstrates that contributions are not always paid in the same year they are awarded, thus necessitating a close watch over cash flow by the program.

The number of hearings is variable and sometimes take place over more than one fiscal year. Overall though, contributions and expenditures increased each year since the PFP was introduced.

Table 6: Contribution Awards and Payments

2 EVALUATION APPROACH AND METHODOLOGY

This evaluation was conducted in accordance with the TBS Policy on Evaluation (2009) and addresses the core evaluation questions related to relevance and performance, efficiency and economy, namely:

1. Alignment with federal roles and responsibilities;
2. Alignment with government priorities;
3. The continued need for the program;
4. Achievement of expected outcomes; and
5. Demonstration of efficiency and economy.

The results of the evaluation will be used by the NEB to inform future decisions about program design and to further develop direction for performance management.

2.1 Evaluation Objectives

During the planning phase for the evaluation and in consultation with the PFP staff, a logic model, evaluation questions and outcome measures were defined.

A PFP logic model (Annex 1) was developed specifically for the PFP Evaluation in consultation with PFP staff and the following documents:

• The Treasury Board Submission approving the NEB PFP Terms and Conditions
  (approved June 24, 2010);
• NEB PFP Funding Guide (2010 and 2014);
• NEB PFP informational material, website, internal process and procedures; and
• Draft PFP Performance Measurement Framework (2013).

The logic model addresses the original performance measures from the Terms and Conditions of the program and the evaluation matrix as shown in Annex 3 incorporates these as well as relevant indicators. Given that this is the first evaluation for a program that has existed for only five years, the evaluation focuses on immediate and intermediate outcomes from the logic model as well as on the program design and implementation.

In early July 2015, the NEB Evaluation Committee and the NEB’s Chair and CEO endorsed the Terms of Reference and associated evaluation questions for the Evaluation of the PFP. These are shown in Table 7 below along with a conclusion:

Table 7: PFP Evaluation Questions

2.2 Methodology

This evaluation assessed and analyzed program documentation, conducted interviews and obtained feedback from internal and external stakeholders using systematic questions. Using the program and financial data that was available from the NEB, qualitative and quantitative analysis was also done.

The program manages funding opportunities for hearings that take place over one or more years as demonstrated by Table 6 above, such that contributions are usually paid in another year than when they are awarded. As a result, the program manages individual funding opportunities within the overall budget set for the program each fiscal year. Program staff implemented a process to track and update cash flow by project and fiscal year which involves forecasting when contribution funds will be paid to ensure that funds will be available when required. This evaluation generally takes a project-by-project approach to analyze the data for each hearing’s funding opportunity to answer several evaluation questions but the personnel and O&M costs are analyzed by fiscal year as they are budgeted in this way and not by project.

Interviews [27] were conducted either in-person, by telephone or via email with the help of an interview guide. Comments from interviewees are reported in an aggregate manner with no references to specific individuals. Interview notes were analyzed by indicator to determine relevant information and key themes. See Table 8 below.

Table 8: Summary of Interviews

External Feedback

Those that had applied for funding under the PFP at least once were invited to respond to certain questions and provide their thoughts. This survey was contracted out and took place between February and March 2015. The response rate was 18% (from a total of 44 groups/individuals).

Internal Feedback

Those at the NEB who might have relevant knowledge or experience with the program were invited to respond to several evaluation questions in July 2015 (189 staff). The response rate was nearly 50%.

2.3 Limitations

The evaluation used multiple lines of evidence to ensure that observations were consistent with the information made available through interviews, document review and data analysis. The limitations to the evaluation methodology are:

• Low response rate/feedback from external stakeholders
• Accessibility of financial or program data to assess the indicators and conclude on the measures for the evaluation.

3 FINDINGS

3.1 Relevance

Evaluation questions in this section include:

1. Is the PFP aligned with the roles, responsibilities and priorities of the federal government?
2. Is the PFP aligned with the NEB mandate, strategic outcome and priorities?
3. Does the PFP continue to address a demonstrated need?

3.1.1 Federal Roles and Responsibilities

Findings and Supporting Evidence
The Participant Funding Program is aligned with federal roles and responsibilities.

In 2004, the Smart Regulation Strategy^{Footnote 20}, a report to the Government of Canada, recommended that “participant funding must be recognized as an essential element of environmental assessment to enable citizens to participate in the assessment process”.

In Budget 2010^{Footnote 21} the Government stated that participant funding programs will be established by both the Canadian Nuclear Safety Commission (CNSC) and the Board “to ensure the timely and meaningful engagement of the public, stakeholders and Aboriginal peoples in the review of major energy projects”. At the same time responsibility for conducting environmental assessments for energy projects was delegated from the Canadian Environmental Assessment Agency to the Board for projects under its area of expertise.

The CEAA 2012 in Section 58(1) requires a responsible authority to establish a participant funding program to facilitate the participation of the public in the environmental assessment of any designated project under its responsibility.

The Jobs and Economic Growth Act 2010^{Footnote 22} amended the NEB Act by adding section 16.3: “For the purposes of this Act, the Board may establish a participant funding program to facilitate the participation of the public in hearings that are held under section l24”.

3.1.2 Alignment with Government Priorities and NEB Strategic Outcomes

Findings and Supporting Evidence

The Participant Funding Program is aligned with government priorities and the NEB Strategic Outcomes.

In the NEB’s Program Alignment Architecture (PAA), the PFP program falls under the program Energy Regulation and sub-program: Energy Regulation Implementation, Compliance Monitoring and Enforcement. Since 2013, it includes the statement: “...the NEB actively involves those affected by its regulatory decisions through public hearings and other engagement activities”.

A review of the Reports on Plans and Priorities (RPP) shows that:

• In FY 2006-07, the NEB committed to introducing participant funding, acknowledging that hearing participants and landowners argued that a lack of support puts them at a disadvantage during specific hearings due to the cost of technical experts or lawyers to represent them or the cost of taking time to participate. The NEB sought to make hearing processes more accessible to a broad range of stakeholders.
• Over the next few years, the NEB reported its progress in its RPPs which included work to obtain the authority to directly provide funding to qualified public participants. It also stated that a key motivator in developing the PFP was to achieve regulatory efficiency and effectiveness.^{Footnote 23}

A review of the Departmental Performance Reports (DPR) shows that:

• In FY 2009-10, the NEB states that implementing a participant funding program will support eligible stakeholders in effectively engaging in NEB regulatory processes.
• In FY 2010-11, the NEB states that it implemented the PFP to enhance the timely, meaningful engagement of the public in oral facility hearings.

A performance measurement policy from when PFP was first launched in 2010, states that the planned result for PFP is the meaningful participation of the public in oral hearings to review the impacts of proposed projects.

3.1.3 On-going Need

Findings and Supporting Evidence

There is demand for funding each time it is announced for a proposed project hearing and this demand is met by providing a contribution towards eligible costs related to the hearing.

Feedback from external stakeholders indicates that their participation was made possible through the funding. Internal stakeholders note that demand for PFP has increased in recent years, which suggests there is a continued need for the program.

The Participant Funding Program is made available to eligible applicants to prepare for and participate in public hearings related to the following applications under the NEB Act as found in Table 9 below:

Table 9: Hearings with PFP

Between 2011 and June 2015, the NEB administered participant funding for 10 hearings, nine of which have concluded. At the time of the evaluation, there were seven other hearings in various stages of the PFP process. Annex 6 contains a detailed list of the project names and companies that applied. For the purposes of this evaluation, short-form names for the projects are used consistently throughout to illustrate the data and analysis. Table 10 below shows that each funding envelope has varied over the years, and is generally around $200,000 on average (not including Trans Mountain Expansion or Energy East).

The Program usually awards the full funding envelope for each hearing. Early in the program, though, the total request from applicants was sometimes below the amount of funding available as shown in Figure 1 below, and after assessing the applications the total amount awarded was even lower.

When compared to the overall PFP budget of $8,413,619 (Table 2) that was available for contributions between FY 2010-11 and FY 2013-14^{Footnote 24}, less than half (approximately 33%) of this has been made available in funding envelopes and even less has been awarded. Factors affecting the funding envelope include the type and scope of the project and the level of interest from the public. Figure 2 illustrates this further for this time period.

Table 10: Funding envelopes announced by project and fiscal year

Figure 1: Funding offered, requested and awarded by completed hearing

Figure 2: Snapshot of the PFP from FY 2010-11 to FY 2013-14

Unused contribution funds lapse unless they are re-profiled each year for another hearing, which is usually the case for the PFP since it is a demand-driven program that manages a budget based on information that is not necessarily available at the beginning of each fiscal year. Two of the indicators identified in the evaluation matrix regarding the need for the Program are:

• Demand for participant funding by the number of applications; and
• Percentage of funds available for a given project that were awarded and used by participants in the hearing.

As shown in Table 11 below, the number of applicants and awards has varied by project. For all 10 hearings listed in Table 9, 76% of applicants have been awarded funding, and of those, 86% signed their contribution agreement. Not including Trans Mountain Expansion Project, which is still in the process of making contribution payments, of the 38 contribution agreements signed, 87% received payment. The majority of applications for funding have come from Aboriginal groups; however Line 9B and Trans Mountain Expansion Projects saw an increase in applications from non-governmental organizations/not-for-profits and individuals.

Table 11: Number of Applications and Contributions Awarded, Signed and Paid

There are a several reasons why the number of awards is not the same as the number of contribution agreements or the number of awards paid. Recipients may have declined their agreement by not signing, later withdrew from the hearing or failed to send in their invoices to the NEB for reimbursement.

In the first few hearings that offered participant funding, those applying for funding were likely to receive it. As well these applicants tended to be Aboriginal groups. The other types of groups attending and participating in the hearings were commonly groups that are ineligible for participant funding such as energy industry organizations, government groups or for-profit organizations. Internal feedback also points to the Aboriginal engagement activities where staff directly provided attendees with information about the hearing process and availability of participant funding as being a potential contributor to the higher number of Aboriginal groups making applications for funding.

On average, the awards have ranged from $1,250 for an individual to $550,000 for a group. Usually the awarded funds were accepted by recipients though there were 17 cases across several hearings amounting to $126,450 where the contribution agreement was declined or not signed by the recipient, thus voiding the award. This results in a lower total dollar value of awards paid out compared to that awarded. Table 12 shows the awards and amounts paid up to FY 2014-15 for the projects

Table 12: Funding available versus total funds paid to participants

There is no obvious trend in the number of applications received during the period reviewed by the evaluation or in the amounts requested or awarded. Recently though, there have been more applications for funding; and the funding envelopes have also increased to meet anticipated demand. The number of applications for funding or interest in participating in the hearing is likely influenced by a variety of factors including the nature of the project and its location, the information in the project description or application and awareness of PFP.

In response to the question “Would you still have been able to participate in the NEB hearing without funding from PFP”, most respondents stated that would have not been able to participate without funding [5/6]. As stated before, caution should be taken when interpreting results from external feedback given the low response rate to the survey.

Responses from internal stakeholders were mixed, with some pointing out that funding levels could better match demand, or that additional funding is required.

3.2 Effectiveness

Evaluation questions in this section include:

Immediate Outcomes

Intermediate Outcomes

• 4f: Does the PFP support adequate levels of public participation in oral hearings?
• 4g: Does the PFP support adequate levels of Aboriginal participation in oral hearings?
• 4h: Do applicants with participant funding participate in the project review process?

3.2.1 Immediate Outcomes

An immediate outcome is “an outcome that is directly attributable to a policy, program or initiative’s outputs”.^{Footnote 27} The outcomes and the outputs of the PFP are listed in the Logic Model (Annex 1).

3.2.1.1 Alignment with program objectives

Findings and Supporting Evidence

Internal stakeholders have the information they need to perform their work.

Internal stakeholders indicate that they understand and support the objectives of the PFP and believe that participant funding helps to remove the financial barrier to participation. They also raise the point that it is important for the program to continually manage expectations since it has limited funding.

The majority of internal stakeholders [62/91] providing feedback have the information about the program that they need to perform their work. They notice that the program has been proactive in making recent improvements to internal communication, there are updated process documents and information about the program is shared. Stakeholders encourage continuous work to formalize internal processes for interacting with other areas of the organization especially to consistently implement each funding opportunity within the hearing process and communicate information to potential applicants and participants.

3.2.1.2 Public awareness

Findings and Supporting Evidence

The PFP process and funding opportunities are, in general, communicated to targeted groups; however there are ways to better align internal efforts on communicating accurate, timely and consistent information to the public.

The Participant Funding Program uses several methods to communicate details about the program and funding opportunities to the public and Aboriginal groups. Newspaper Ads and the NEB’s website are used to launch a funding opportunity. The website also contains further information about current projects and their status, application deadlines, funding reports, templates and further description about the program. There are also quick links to the funding program through major project webpages.

Staff that conducts Enhanced Aboriginal Engagement (EAE) through the Applications Business Unit, coordinates with the Major Projects Management Office (MPMO) at Natural Resources Canada (NRCan) to mail letters to specific groups (determined to be potentially affected by the proposed Project) about a particular project and funding opportunity. These letters are usually sent around the same time as the PFP news release or shortly after. These engagement activities with the Aboriginal groups or Open Houses with the public are also ways in which the NEB provides further information about the hearing process and participant funding.

It is important that the program maintain relevant PFP information in external publications. For example, in September 2010, the NEB published a Guide for Landowners and the Public which summarizes the hearing process and provides information about the regulatory life cycle, land agreements, right of entry, and the complaints process. This guide is available in print or electronically on the NEB website. It does not have any information about the participant funding program though it is addressed to one of the target audiences of the funding program. This could be because the guide was introduced around the time that the PFP began and has not been updated since.

When the NEB issues a Hearing Order, it requires the company to serve a copy on certain persons, agencies, Aboriginal groups, individuals and landowners. The NEB also requests the company to publish the Notice of Hearing in certain publications by a specific date. The Notice may also mention the PFP. In the first several hearings the application deadline occurred prior to the Hearing Order and thus the Hearing Order does not mention the funding program. Other more recent Hearings have briefly mentioned the availability of participant funding or promise further information in subsequent updates.

Although the PFP is advertised by newspaper, the program has never paid for these Ads through its O&M budget, even though ‘communication’ is considered an indirect administrative cost for the program. The only advertising paid for through the PFP budget was to recruit Funding Review Committee members a few years ago.

Feedback from external stakeholders suggest that they hear about funding through the NEB website or information session [2/8], through public notice [1/8], local media (newspaper, radio, TV) [1/8] or a colleague [1/8]. Several internal stakeholders were of the opinion that it could be confusing for the public to differentiate between the Application to Participate process and the Application for Funding, since they are managed by different areas of the Board and have separate templates and deadlines but similar requirements to justify participation or funding. Internal stakeholders that have a role in interacting with Aboriginal groups and the public indicate that they have become aware of funding later in the process. Staff did not provide any reasons why this may be the case.

Other internal stakeholders advised that the program should work closely with communications and that the various business units and teams should integrate their messages along with the PFP to ensure consistency. Other comments were that if engagement staff received more training from the program they could better support it by responding to these external questions from the public.

3.2.1 Accessibility

Findings and Supporting Evidence

The Program could improve the templates and performance measurement and define a systematic approach to collecting feedback and data to determine whether it is meeting its various objectives.

Templates and Tools

The Treasury Board Policy on Transfer Payments (2008) expects that “transfer payment programs are accessible, understandable and useable by applicants and recipients.”^{Footnote 28}

Half of the participants surveyed [4/8] suggest that they experienced challenges in completing their application for funding related to the process and form. A few suggested that it was easy to complete [2/8]. Participants state that they found the application form repetitive in some areas and they were uncertain how to fit their expenses into the funding categories. The majority of respondents indicate that they used additional help such as a consultant to complete the application [6/8]. Participants also mentioned that they are uncertain what criteria are used to judge their application.

Internal stakeholders also provided feedback and suggest that the application process needs consistency and streamlining as well as clearer guidelines. Others point out that clear operating rules were initially lacking for the program, but this has been improving. They also note that the program demonstrates that it is flexible and also makes adjustments within the constraints of the PFP Terms and Conditions and in particular, they note that program documentation has improved. Internal stakeholders also thought that while the changes the program introduces may be positive, it can be confusing for staff and applicants when changes are made to the program while carrying out the funding process for a particular hearing and when these changes are not readily communicated in advance.

The PFP has recently updated or created new process documents, prepared a detailed process map, application templates, participant funding guide and news release template. It also can demonstrate that it has developed information that engagement staff can use to communicate the program, it has updated the program’s public website and it is also working to update and further improve other forms such as invoice claims.

Some external and internal stakeholders have suggested that ideally the Application to Participate (online-based form process) and the Application for Funding (paper-based form), could use the same online platform to accept applications. This could reduce some of the administrative work of duplicating similar information between these two applications. It could also be more efficient for applicants and staff to have information accessible in one spot.

Performance Measurement

The TBS Policy on Transfer Payments (2008)^{Footnote 29} has certain expectations for the design and delivery of transfer payment programs. The Policy requires a performance measurement strategy to be established at the time of program design and that it is maintained and updated throughout its life cycle to effectively support an evaluation or review of relevance and effectiveness of each transfer payment program.”

The TBS Directive on Transfer Payments (2008)^{Footnote 30} requires a documented assessment of a performance measurement strategy including performance measures, indicators and supporting data requirements and data collection strategy.

The PFP has a Performance Measurement Policy (2010), a Performance Measurement Strategy (2010) and a draft Performance Measurement Framework (2013). These documents state that the planned result for the program is “meaningful participation of the public in oral hearings to review the impacts of proposed projects”. Performance measures and indicators were also identified for measuring program outcomes and efficiency in delivery:

• Number of applicants for participant funding per hearing
• % of applications approved for funding.
• % of recipients who agree that the funding provided assisted their ability to participate.
• Average # of days from signing of contribution agreements to first payment of funds.
• Average # of days from receipt of recipient of progress reports to disbursement of next funding tranche.

There is evidence that the program has made several attempts to compile data in response to these performance measures for the first five hearings; however analysis is incomplete, some of the calculations are inaccurate or have not been updated. The evaluation found that useful data exists in multiple systems but it is time consuming to review and analyze. As a result, additional work was required to compile information from original sources in order to confirm the data and analyze it for this evaluation.

The PFP developed a template that requested the socio-economic specialist, who was involved in a particular application review, to complete following the corresponding hearing. It states that the purpose of the template is for the PFP and steering committee to understand the amount of involvement by participants so appropriate funding allocations can be made.

It was created by the program in 2011 and was used for 7 of the 9 completed hearings reviewed in the evaluation. The template documents the name of the group or individual, how they participated, and whether they applied for and received funding. Several staff mentioned that they were unaware of the origins of this template, the intent or whether it was a useful exercise. It is also unclear whether this template was used to monitor the obligations of the recipient against their contribution agreement.

Monitoring Payments

The TBS Directive on Transfer Payments (2008)^{Footnote 31} also requires an assessment of the risks associated with applicants and recipients and measures to manage the risks.

The TBS Guideline on Transfer Payments^{Footnote 32} specifies that if recipient audits are selected as a way to monitor transfer payments, then a risk-based plan should be developed for such audits. Program staff indicates that a recipient audit has never occurred and that recipient audits are not used as a monitoring tool because there is no process in place; however the standard contribution agreement template signed with each recipient states that it is a possibility. There is also a provision in the contribution agreement requiring the recipient to maintain relevant documentation for a certain period of time (varies from 24 or 36 months). Internal stakeholders stated in interviews that they are interested in implementing a recipient audit process, which would require a risk profile for each recipient as well as criteria for assessing risk.

The program has changed the contribution agreement template several times. A review of the contribution agreements from each funding opportunity demonstrates that at the beginning of the program, the obligations of recipients was limited to the participant agreeing to participate in the oral hearing process, using their funding on eligible costs identified in the Application approved by the NEB and filing their information on the public record. Usually the agreement was effective once signed by both the awardee and the NEB. Final invoices were due within 60 days following the end of the hearing. The program used to offer an advance of up to 75%. This changed to 50% for a few hearings but is more recently back to 75%. The current contribution agreement template articulates the obligations of the recipient in more detail, has made the effective date of the agreement from the date the project application is filed versus when it is signed. Record keeping requirements from the recipient have also changed from “up to and including 24 months after the final contribution is made under the agreement” to “36 months after final payment”.

PFP staff indicate that a performance monitoring framework for the new Terms and Conditions (June 2015) has not been put in place yet as measures have not yet been identified. Staff also indicates that the hearing process lacks measures that can generate information that would be useful for performance management of the program. Overall, feedback from staff suggests that the program could improve performance measurement and a systematic approach to collecting feedback and data to determine whether it is meeting its objectives.

Originally the PFP would ask one question in the application form to know how applicants learned about the program. As well, some of the claim form templates also had one yes/no question to obtain feedback on whether the funding award made it easier for the applicant to participate in the hearing. PFP staff note in their review of previous applications and claim forms that these questions were not asked each time and the program was not clear enough in its documentation or analysis on how this feedback was obtained or what the results mean. As well, there are times the recipients did not answer the brief survey questions.

The program changed the application template in December 2014 and incorporated a longer survey in order to have immediate feedback from applicants. The PFP Guide also reiterates the request for applicants to complete the survey in order to help the NEB to evaluate the effectiveness of the program in meeting program objectives and to help identify potential improvements in program administration. The survey asks, for example, how applicants learned about the funding opportunity, if they find the form understandable and easy to fill out and provide suggestions for how to improve it. Since introducing this update, the program is tracking responses and comments. This evaluation did not consider this evidence because it solicits feedback for hearings that are outside the scope of the evaluation.

3.2.1.4 Timeliness

Findings and Supporting Evidence

The program could better define timelines in the process to allow award recipients adequate time for preparing to participate in the hearing process.

Preparing for Participation

The TBS Policy on Transfer Payments (2008) requires “reasonable and practical departmental service standards for transfer payment programs”.

Participant feedback suggests that for some the public notice about the availability of funding was given too late [4/8]. A few said it was just right [2/8] and the rest had no response. One participant found that funding was too late for them to consider an expert witness and another similarly stated that the approval process should be quicker so there is adequate time to prepare for a hearing. Participants indicate that they could have used more time to prepare effectively for the hearing they were involved in [3/8], some said they didn’t have enough time [2/8] and a few were uncertain [3/8]. Complaints tracked by the program also indicate that applicants and participants are concerned about the timing of the process and thus the impact on their ability to participate.

There were also a number of comments from internal stakeholders related to the timing of funding. They note that the PFP was introduced in 2010 whereas the Application to Participate was formalized later in 2012 following changes to the NEB Act. This change has introduced some challenges and confusion in explaining the two processes which are kept separate but are also related with the funding award dependent on a successful application to participate as an intervenor. The funding application requests applicants to provide a response to their intervenor status however the internal process map for the program has not documented for staff how and when to confirm the intervenor status of funding applicants.

A staff member pointed out that the PFP could be more in sync with the hearing process timelines (or vice versa) such that once funding is approved it is readily available. Others suggested reviewing the timing of the PFP process, ensure timely notification and make funding available sooner in order to allow recipients the time to prepare for the hearing. Having a deadline of 30 or 60 days before the first date of the hearing makes it challenging for the program to carry out its processes in a timely fashion especially if applications are received closer to this deadline and applications are reviewed on a rolling basis. According to the June 2010 Terms and Conditions of the PFP, funding was eligible after the contribution agreement is signed, thus a funding deadline close to the hearing leaves very little time for an applicant to use the funding on the costs they incur to prepare for the hearing. Others suggested that the Application to Participate (ATP) process could occur first so that the PFP could dedicate its resources to assessing applications that have already confirmed that the applicant has intervenor status.

The first six PFP news releases included the deadline to submit an application. This changed for subsequent news releases to applying ‘as soon as possible’. Variation was observed in the deadlines, and the program has continued to use a “60 days before the first day of the oral portion of a hearing” deadline, later updating this on the NEB website to the actual date once known. In the past, deadlines likely provided more certainty for potential applicants in organizing their time. Recommending potential applicants to “apply as soon as possible” may also cause undue confusion.

The following calculations were made using program data:

• Nine out of ten hearings examined in this evaluation had a Project Description^{Footnote 33}. It took an average of 63 days (or range of 28-113 days) to announce the availability of funding for a given project following receipt of the Project Description. This time period is used to prepare the PFP process and includes determining the funding envelope.
• Letters rom the NEB and MPMO to potentially impacted Aboriginal groups were usually sent out between 0-21 days following the PFP notice of funding.
• For the ten hearings reviewed, applicants had between 29 and 494 days to apply for funding. The first six hearings eligible for participant funding had a similar time frame to apply (29-63 days) whereas the following four hearings ranged from 219-494 days.
• For the five projects that had an application deadline that preceded the Hearing Order, the List of Parties (lists the approved participants and level of participation), was issued 106 to 270 days after the PFP deadline. This means that those awarded participant funding would have waited several months for confirmation of intervenor status and thus confirmation that their expenses could be reimbursed in accordance with the conditions of their Contribution Agreement. In these cases, there was an average of five months between the date the List of Parties was issued and the first date of the hearing, which is time that recipients could use to prepare to participate. The PFP Guide (2010) mentions that a successful application for participant funding does not guarantee that an applicant will be successful at obtaining intervenor status and cautions applicants about this possibility prior to incurring costs.
• One project did not file a Project Description, and the hearing followed a different timeline with the PFP process commencing after the Hearing Order was issued (Figure 4). While this resulted in a smaller gap in time (12 days) between the PFP deadline and the date the List of Parties was confirmed, there was less time available to prepare (approximately 3 months) compared to the other hearings.
• The other four more recent hearings had the deadline to apply for PFP occur after the ATP deadline and issuance of the List of Parties. As a result, participants also had around 5 months to prepare for the hearing.
• In general, the List of Parties was issued before the contribution agreements were signed by the NEB or close to the date when some of the first agreements for a given hearing were signed. When comparing the date the List of Parties was issued to the date the applicant submitted their signed application, there were 69 applications in which the intervenor status of the applicant applying for funding would have been known. The other 88 applications for funding were received between 6-231 days before the intervenor status of the applicant was confirmed. Either situation could be applicable within a particular hearing and its funding opportunity especially when applications were reviewed and awarded as they were submitted.
• Funding is meant to help participants prepare for and participate in public hearings. On average, each funding opportunity has allowed at least 30-60 days to apply. More recent hearings allow far more days (225-494 days). Applicants sometimes sign their contribution agreement on the day they receive it and sometimes send it back to the NEB for its signature more than 2 months later. Once the program receives the signed contribution agreement from the applicant, it proceeds to sign it usually the day received or within the next 3 weeks. Table 13 below indicates how long applicants can expect, on average, between applying for funding and receiving their signed contribution agreement from the NEB, while Figures 3-7 illustrate the hearing process for a few projects and a calculation of timelines between key milestones. These figures demonstrate the variation in approach to the process, timelines and the order in which key activities are carried out.

Table 13: Average number of days between applying for funding and the effective date of the contribution agreement

• The program has not published service standards for its program delivery in the past, but more recently, it has committed publically for certain funding opportunities to issue a funding decision to an applicant within 60 days after the announced deadline.
• Six participants indicated that they contacted the NEB to raise concerns or questions about the application process and found that the response from the Board was either timely, very timely or neither [6/8]. Half were satisfied with how their concern/ question was addressed while two were very unsatisfied. Some thought that the NEB representative understood their concern; one applicant thought they were given misleading information on the likelihood of funding; and another commented that they found the representative helpful but was unsure what questions to ask due to unfamiliarity with the process.

Figure 3: Vantage Hearing Timelines

Figure 4: Line 9 Reversal Hearing Timelines

Figure 5: Line 9B Hearing Timelines

Figure 6: North Montney Hearing Timelines

Figure 7: Trans Mountain Expansion Hearing Timelines

Note:
The NEB holds hearings for Oral Traditional Evidence (OTE) to acknowledge that Aboriginal peoples have an oral tradition for sharing certain information and writing may not be an adequate way to share it. OTE forms part of the evidence that a Hearing Panel will consider when assessing the Project.

Reimbursing Claims

Participants indicate in their feedback that they either estimated their costs accurately or underestimated them.

In the past, the PFP used to send the expense claims and supporting documentation from applicants to finance staff for processing. Later, the process was changed to the program assessing the claims for completeness and accuracy and then forwarding a payment request to finance staff. The program has also recently updated delegated signing authority to make the process more efficient. The program usually has a few interim payment requests and rarely has a request for an advance of funds. All recipients must submit a final claim within 60 days of the close of the hearing. The PFP Coordinator is responsible for reviewing the claims when they are first submitted, verifying the expenses and retaining the documentation. The finance team reviews the payment request and follows-up with the program if there are any concerns or questions. The finance team also processes payment requests for the funding review committee following receipt of their invoice. In order to clarify how it will work with the PFP overall, finance staff documented the roles and responsibilities.

During the evaluation, it was understood that the approved payment requests are forwarded by finance staff to another government processing center for actual payment to the recipient. Originally the evaluation intended to calculate the time to reimburse a complete claim, however this information about actual payment dates would reside in another government department and if there were a service standard, if would be outside the program’s control.

3.2.1.5 Transparency

An expected result of the TBS Policy on Transfer Payments (2008) is “applicants and recipients are engaged in support of innovation, continuous improvement and the establishment of fair, transparent and positive relations with them.”^{Footnote 34}

Findings and Supporting Evidence

The Program is fair and transparent, providing information and instructions on each funding opportunity as well as publically sharing the names of recipients and their awards. The program demonstrated that it is working to have greater transparency in its assessment approach and feedback to applicants.

Transparency and fairness of the process

Five out of eight participants responded that the funding application process is clear and fair. They also thought it had improved in the last few years. Half of the participants either agreed or strongly agreed that the PFP documentation was clear about the information required to apply. One respondent found that the instructions were clear, the examples helped to complete the form, and through its website, the NEB clearly explained which costs were eligible. On the question of whether the eligibility requirements were clear, participants overall either agreed or were neutral.

Staff and the FRC members suggested that receiving applications on a rolling basis and then assessing them as they come makes it challenging to allocate the limited funding envelope in a fair manner. Fairness is best achieved by consistency in the process. Staff also acknowledges that the program has to be flexible in order to ensure fairness, such that when the company proposes a new route during the hearing process that may affect new people, the PFP has responded by opening up the process again if it has already closed.

When the Application to Participate (ATP) was introduced in 2012, the process and decisions makers were kept separate from the funding program in order to keep participation and funding decisions fair and independent.

Funding recommendations

Participant feedback indicates that for those that who didn’t find the process clear and fair, they were mainly concerned with the lack of clarity in the application or feel they were not provided clear reasons why their application for funding was denied even after requesting an explanation. For those participants who had their application for funding denied [2/8], they report that they did not receive an explanation. One participant thought there is a lack of transparency from the funding review committee and the process to determine the allocation or amounts for funding. Another perceived that the program favours some groups over others. Participants suggest that when they receive a decision on funding it should include the assessment criteria and its rating. This type of information and feedback could help an applicant on their future applications for funding. A review of funding notices to those denied confirms this opinion - there was no specific explanation tailored to the applicant that received a notice. The program states that it has since addressed this starting in 2014.

Internal stakeholders also provided feedback, and suggested that the program could enhance transparency in its reasons for decisions on funding. They indicate that they are aware of the opinions of applicants that the funding decisions could be clearer and that there could be an improved response to complaints. Others commented that they already see evidence of this challenge being resolved. Interviews with staff confirm this - the program is looking to give more comprehensive and specific reasons why applicants do not receive funding or indicate for those that do, why the amounts are less then requested. This will improve consistency and quality of review. As well, the funding review committee is formalizing their analysis and recommendations.

In its 2014 Annual Report^{Footnote 35}, the NEB indicates that it has responded to this type of feedback to make decisions more transparent by including reasons.

Disclosing Awards

The NEB is required to demonstrate transparency by reporting all decisions by the Funding Review Committee. Thus it publishes the list of awards per funding opportunity to its website once all the awards have been made for a particular project. Currently there are 4 projects with a funding report available on the website and staff indicated that they plan to publish all past hearings as well. Program staff is also evaluating its approach to assessing the information shared through the website to ensure consistency across documentation, and to prepare for information to be re-launched through the canada.ca web renewal project^{Footnote 36}.

The NEB also discloses contribution awards over $25,000 on a quarterly basis to meet the requirements of proactive disclosure in the federal government.

Legal Challenges

In its 2014 Annual Report, the NEB reports that an application for judicial review was filed June 10, 2014 with respect to a decision for the participant funding program which denied the applicants’ request for funding for the Trans Mountain Expansion project application. A hearing took place on March 23, 2015 and the courts recently dismissed this application on September 22, 2015^{Footnote 37}.

3.2.2 Intermediate Outcomes

An intermediate outcome is “an outcome that is expected to logically occur once one or more immediate outcomes have been achieved”.^{Footnote 38}

3.2.2.1 Participation of the public and Aboriginal groups in oral hearings

Findings and Supporting Evidence

Most of the eligible public and Aboriginal groups that applied for funding were awarded.

Internal stakeholders were asked whether funding enabled adequate levels of public and Aboriginal participation. Half of those that responded to this question indicate that they did not know and suggested that the program should focus on meaningful participation as opposed to assessing ‘adequate participation’. In response to whether funding led to more meaningful public participation, again half the respondents indicated that they did not know while a third of respondents thought it had. Many, however, suggested that overall it is challenging to define or assess whether PFP facilitates meaningful participation. Many commented that it could be misleading to attribute meaningful participation solely to the funding program. Anecdotally, however, some staff heard from participants that participant funding helped them to participate more fully in the hearing process.

Participation levels may have gone up over time; however staff point out that assessing the adequacy of numbers participating in a hearing is difficult. Internal stakeholders felt they do not have the information to determine this outcome. They recommended that the hearing process could incorporate measures and help to assess this outcome.

Internal stakeholders were also asked whether participant funding led to more evidence (written or oral) for oral hearings. The majority of respondents (66%) indicated that they did not know while some said yes (25%) and the rest answered no or did not answer. Respondents also provided mixed comments with some again suggesting that it is hard to assess this and report on it. A few suggested that they noticed the evidence becoming more sophisticated but where unsure whether this could be attributed to PFP or if it reflects that participants are more aware about hearings. Others have found that the quality of evidence may have increased but not necessarily the quantity.

The PFP has done preliminary work to consolidate data and document the nature of participation by those awarded funding by counting the number of items on the record and noting the information that was filed for the hearing. It was not as clear how this information is being used by the program for analysis and to assess its objectives or measures of success.

The purpose of the PFP is to also encourage effective public participation that ensures public concerns and values are taken into consideration. The funding guide for the program specifies that to apply for funding, applicants have to demonstrate the following:

• Proposed activities must relate to the “list of issues”
• Proposed activities must be unique (not duplicating the information available from other sources)
• An applicant must demonstrate an interest in the proposed project that relates to one the following:
  • direct, local interest;
  • local community insights and/or Aboriginal traditional knowledge about the proposed project;
  • an interest in how the potential project may affect Aboriginal lands or rights, or
  • expert information relevant to the proposed project.

In July 2012, changes made to the National Energy Board Act include the addition of section 55.2, which sets out when the Board will allow a person to participate in a hearing. Applicants must demonstrate to the Board’s satisfaction that they stand to be directly affected by a project or have relevant information and expertise that could help the Board gain a greater understanding of the project under consideration.

Internal stakeholders responded to the question whether they think funded intervenors provided oral hearings with information that contributes to a better understanding of the impacts of a project and informs the Panel’s recommendation/decision - and a large majority of respondents (65%) said they did not know. Around one-third said yes. Comments in relation to this question indicate that again, internal stakeholders are uncertain whether participant funding plays a role in whole or part or they suggest that it would depend on the project and the Hearing Panel assessing the usefulness of information. For those that answered yes to the question, they felt that funding has allowed people to raise local issues and suggest that evidence from studies/review is informative. One staff member noted that since the program began, the evidence has improved in quality. Internal stakeholders suggest that the program staff should examine how to assess ‘value-added information’ for each hearing and overall if it is an objective and ultimate outcome for the PFP.

Data from the program indicates that the PFP has not had many applications from non-Aboriginal groups in the first five years it has existed. Figure 8 shows that the number of applications from Aboriginal groups is almost double that of non-Aboriginal individuals or groups for ten hearings examined in this evaluation. Internal stakeholders suggest in their feedback that they have observed that participation by Aboriginal groups has been increasing with funding and that the information from these groups about the potential impacts of proposed projects is useful.

Figure 8: Number of Applications from Aboriginal and non-Aboriginal applicants

A comparison of the different types of intervenors for the last nine hearings that have concluded shows that many of those who participated would have been ineligible for funding (e.g. companies, government). Table 14 illustrates this below. There was very little participation by non-Aboriginal groups or individuals or associations when the PFP was introduced in 2010, but this changed starting in 2012 with the Line 9B hearing.

Table 14: Type and number of approved intervenors for nine completed hearings

Table 15 shows, for the nine completed hearings, the number of funding awards and applications compared to the number of intervenors that participated in the hearing that were eligible to apply for funding. The dates indicate the period from announcing participant funding to the decision or report by the Board on the proposed project. Overall of the intervenors that were eligible to apply for funding, 45% had an award.

Table 15: Number of intervenors with funding

There is no distinct trend though in eligible intervenors with funding awards over the years as illustrated in Figure 9.

Figure 9: Percent of eligible intervenors for funding with an award

As already indicated, the majority of funding applications has come from Aboriginal groups and Table 16 below illustrates this further. The number of Aboriginal groups that actually applied for funding tends to be lower than the number of groups determined as potentially impacted by the MPMO and NEB. Overall, though, Aboriginal groups usually represent 100% of applications in the early years of the PFP.

Table 16: Funding for Aboriginal groups and their participation in the past nine completed hearings

3.2.2.2 Participation of Funded Intervenors

Findings and Supporting Evidence

Most applicants that were awarded accessed their funding.

Participants indicate that they would have not been able to participate without funding from the program [5/6]. These five participants that received funding state that they submitted oral or written evidence during the hearing; however they were split in their opinions on how effectively they were able to present their views and raise their concerns. While three believed they were effective or somewhat effective, two said they didn’t feel effective and the other participant said they didn’t know. Participants indicated that with regards to the benefit of eligible expense categories, it was either positive or had no impact. For those that indicated it was positive, they suggested it was legal representation, expert information and the travel to the hearing that helped as well as community consultation, collecting information and providing honoraria.

Most applicants that were awarded usually accessed their funding as shown in Table 17 below for the nine completed hearings. There were several instances where applicants withdrew from the hearing after their award was approved and signed or their funding was terminated because they did not participate in the hearing. Sometimes awardees did not submit final invoices. In summary:

• 75% of the number of applications were awarded
• 87% of the numbers of signed agreements were paid.
• 30% of the amount requested was awarded
• 95% of awards were signed
• 86% of signed awards were paid.

Table 17: Overview of nine completed hearings

3.3 Efficiency and Economy

The TBS Evaluation Policy (2009) defines both efficiency and economy. Efficiency is the extent to which resources are used such that outputs are produced at a minimum cost. Economy is minimizing the use of resources (cost) to achieve expected outcomes.

The NEB is required to have a participant funding program and there is no alternative at this time. Thus the evaluation of efficiency and economy focussed on obtaining and analyzing performance measurement data to examine optimal utilization of resources and possible areas for improvement.

Evaluation questions in this section include:

1. 5a: Is the PFP delivered efficiently in comparison with similar programs? Are there ways to improve program delivery?
2. 5b: Are the PFP resources (dedicated staff and the Funding Review Committee) appropriate/adequate to deliver the program?

3.3.1 Efficiency of Program Delivery

Findings and Supporting Evidence

Program efficiency has improved over time; however program administration can be streamlined.

The TBS Directive on Transfer Payments (2008) defines efficiency of payment as those payments that can be processed quickly and that documentation sought in support of payments is only that which is necessary and sufficient.

External stakeholders indicate in their feedback that they experience an average to high level of administrative effort related to the PFP process in general. In particular, the application and claims process takes extra time and payments can be slower than expected. External stakeholders suggest that the claim forms could be clearer and more examples of eligible costs and level of costs would also be helpful.

Internal stakeholders also readily provided feedback about PFP related to program delivery and process. Almost half of respondents believe that the administration of PFP has improved over the last few years and the other half did not know for sure. One respondent commented that it is efficient for PFP to be implemented by the NEB. A few suggested that so far, having PFP restricted to oral hearings has hindered the Panel from choosing the optimal format for the hearing - which in some cases is a written process. By choosing an oral hearing, the Panel has been able to ensure that participant funding would be made available. With the recent changes to the Terms and Conditions to include other hearings other than oral this issue will likely be resolved.

Internal stakeholders also suggested that funding could be made available earlier in the process to ensure that participants will be assured that they have funding in place to support their preparation activities for the hearing. With the new Terms and Conditions, however, eligible activities will now be funded from when the date the company files their application instead of when the contribution agreement is signed. This may make a difference, but the applicant is still required to obtain intervenor status, for which the timing is not necessarily before the funding award and contribution agreement.

A few stakeholders also suggested that applicants could be further encouraged to collaborate and cooperate to minimize duplication of information, though others mentioned that it could be challenging or unrealistic to achieve this. Other general comments relate to performance management, with the suggestion that there could be better tracking and measurement of the expected outcomes of the Program and that the hearing process should take part in helping to assess this. This evaluation notes that the newer Application Form for funding requests the applicant to provide suggestions to improve the application form. The Application Guide encourages applicants to respond to this request in order to help the NEB evaluate the Program’s effectiveness in meeting program objectives and to identify potential improvements in administration.

Information Management

The PFP has identified that the current content management software (known as the Records Documents and Information Management System, or RDIMS) generally used at the NEB is not optimal for the nature of work involved in administering a funding program. It is time consuming to manage information on this system and it lacks the tools necessary to easily organize information by project and applicant or to compile and analyze qualitative or quantitative data. A review of documentation on the current system demonstrates that there have been multiple and incomplete attempts to organize and track information. The program should have a better tool for program management especially since it expects that it will experience growing demand and thus administrative work.

The PFP has already recognized this as an opportunity and is working to review the issue and establish a new system for program management. This new system will address case management and improve workflow, thus allowing program staff to better manage the workload and track data in a practical way.

The system will be accessible by PFP staff only. During the evaluation, the program indicated it had already piloted a solution and was working with other interested departments about the options to permanently implement this solution.

Benchmarking

There are some similarities in terms of goals and in the design of the participant funding programs at the NEB, the CNSC and the Canadian Environmental Assessment Agency (CEAA). Those involved with the NEB participant funding program are not necessarily familiar with the details or outcomes of these other programs to draw conclusions about the efficiency of the PFP in relation to funding programs at CNSC and CEAA.

CEAA has had a participant funding program since 1990 and first evaluated it in March 2009 and is currently conducting its second evaluation in 2015. The CNSC was approved to introduce a participant funding program at the same time as the NEB and has evaluated its program in 2015.

CNSC performed a comparative analysis for its recent evaluation covering FY 2011-12 to FY 2013-14. CNSC calculates, for all three organizations, the funds awarded (output) per full-time equivalent (FTE) as a measure of efficiency such that a higher number of recipients per program FTE indicates greater efficiency. It found comparable levels of efficiency for all three organizations. Calculations by CNSC were based on 2 FTE at the NEB.

The evaluation of the NEB’s program though takes into account that it has one FTE funded by the PFP and the second FTE funded by the Application Business Unit and thus efficiency calculations can also be done using one FTE. As well it is important to note that other staff are involved in processing and awarding applications but aren’t directly supported by the PFP budget (e.g. records, mail room and finance). Another consideration is that time and effort is spent reviewing applications (including review by the FRC) even if they do not result in a contribution agreement or the project is cancelled by the company prior to the hearing.

In order to compare to CNSC, which examined a shorter time frame between FY 2011-12 and FY 2013-14, the numbers have been extracted for the NEB for this time period only in order to make a comparison as shown in Table 18 below.

Table 18: Comparison between CNSC and NEB^{Footnote 39}

Due to the timing of PFP and its process which usually overlaps fiscal years, applications for funding may occur in a different fiscal year than when the contribution agreement is awarded and signed. This explains the large difference between the value of funding requested, awarded and paid.

Data from the Treasury Board Secretariat^{Footnote 40} shown in Table 19 below, demonstrates that CEAA has the largest budget for contribution payments and also makes the largest share of payments in relation to its budget compared to the NEB or CNSC for each year from 2011-12 to 2013-14 and overall. The CNSC has increased the amount of contribution payments each year while the National Energy Board has been variable, due to an overestimation of the budget for contributions FY 2012-13.

Table 19: Comparison of Contribution Payments against Budget for CEAA, CNSC and NEB from 2011-12 to 2013-14

3.3.2 Adequacy of resources for program delivery

Findings and Supporting Evidence

The workload can be variable depending on the number of funding opportunities underway and the number of applications to process.

The Participant Funding Program has a modest budget for staffing and supports a full-time Manager while the Applications Business Unit funds the full-time co-ordinator position to ensure there is adequate staffing to administer the program. Over the last five years, the co-ordinator position has had some turnover - which has caused, according to internal stakeholders, a risk of continuity in knowledge for the program. Stakeholders have noted, though, a recent focussed effort on improving process documentation which should help to mitigate the risk of staff turnover.

The program is also supported by the Financial Services staff at the NEB for the processing of payments. Finance staff estimate that they spend 5 to 10% of their time supporting PFP which they prioritize over their other work. They are able to process payment requests within 2-3 days before sending these on to another government department for direct payment to recipients. Finance is also impacted by the program’s staff turnover and has stated that it can get time consuming to educate PFP staff in financial management.

During interviews with internal stakeholders, no major concerns were raised about the adequacy or appropriateness of PFP resources except to acknowledge that the workload is variable depending on where in the process the program is on a given day for a particular hearing. It also depends on the number of individual PFP opportunities in progress and the demand in terms of applications. Workload can be unpredictable since it is a demand driven program. Internal stakeholders indicated that it is important though for those working for or in relation to the PFP, that they have adequate training and experience in transfer payment programs and financial management in order to deliver the program effectively and efficiently.

Program Delivery Costs

The program has a specific time code that staff can use to record their hours of work if related to the program. This is a way of tracking the level of involvement. Data from the TIME system is useful for analysis if the system is used properly. A review of the hours coded to the program though shows inconsistent use of the time code which makes the data an approximation at best. Table 20 below illustrates the data that is available by fiscal year^{Footnote 41}.

Table 20: Hours spent on the PFP

Over the last five years, 60 people have coded time to the PFP for a total of 15,759 hours. The bulk of time spent on PFP is from directly involved staff (PFP Coordinator, Director, Manager and Steering Committee). This table does not include the hours billed by the Funding Review Committee for this time period (530 hours).

Originally, the participant funding program was expected to use a maximum of 13% of its total funding per year on administration which is made up of salary, operating & maintenance costs, travel, communications, translation and professional fees for the Funding Review Committee. As well, it was expected that the administration costs would vary by year and could potentially decrease after the program had finalized its design and start-up and gained more experience.

Since contribution funds for a given hearing may be paid out over multiple years and administration costs may be incurred in a different year than when the actual contribution payment is made, it is difficult to calculate whether the program has achieved this administrative overhead target of 13%. According to the original budget for the program, an average of 9% was budgeted towards personnel and operation & maintenance costs and 91% on average towards contribution payments.

Data for the last five years of the PFP was obtained from program and finance staff. Not including FY 2010-11 (because there were no contribution payments made), personnel and O&M costs represented 24-61% of total spending (40% on average) and contribution payments represented the remaining 39-76% (60% on average) of total spending as shown in Table 21 below.^{Footnote 42} Thus the data indicate that the program has spent more per year on administering the program than it planned for when compared to the level of contribution payments. This is due in part to including the coordinator position as part of actual spending and, as staff note, the contribution payments follow the hearing schedule, so when a hearing concludes, most of the payments are made. Payments may occur in another fiscal year from when the contribution agreement was signed which results in variable level of contribution payments each fiscal year.

Table 21: Ratio of Costs to Spending

Using the data provided by the program and finance staff, a few other calculations were made to understand the average costs of PFP over the past five years:

• The ratio of budgeted program costs to overall budget ranged from 7-12%
• The ratio of program costs to funding envelopes was 19%.
• The ratio of program costs to value of awards offered (before agreements signed) was 19%. This is 20% when compared to signed contribution agreements.
• The ratio of program costs to contribution payments was 53% (note that costs have been incurred to administer the program for Trans Mountain Expansion, however payments are still ongoing in another fiscal year from when they were awarded).
• Overall, 100% of the personnel and O&M budget has been used to operate the program, with some variation from year to year. The O&M cost has stayed within budget for the first three years of operation. The other more recent years have spent more than budgeted as shown in Table 22 below. The largest share of O&M spending each year is usually due to fees for Funding Review Committee members.

Table 22: O&M budget used to administer the program each fiscal year

Table 23: # of applications, awards and signed contribution agreements

• Over the five year period examined by this evaluation, there have been a total of 157 funding applications for 10 hearings. The costs for salary (2 positions) as well as O&M totals $797,869.42. Thus an average of $5,082 is spent to process each application.
• Alternatively, over the last five years, as illustrated in Table 23 above, there have been a total of 120 awards for 10 hearings. This represents a cost of $6,649 per award. If calculating the cost per signed contribution agreement for 10 hearings, this represents a cost of $7,746 per funded participant to administer the program. These calculations are used to demonstrate that there are costs incurred to process applications, even if they don’t result in awards or signed agreements.

Advertising

Communications staff at the NEB create the advertising plan overall for a hearing which includes participant funding. The Secretary’s Office approves the advertising budget. Placing individual ads require approval from Management Services to comply with the Financial Administration Act (section 52). The NEB is responsible for tracking advertising costs and reporting through the government-wide Advertising Information System (AdMIS). The evaluation noted from staff that tracking actual advertising costs for PFP has begun in the past fiscal year. They indicate that to review past data on the allocation of advertising costs to the PFP versus other hearing related advertising would be a cumbersome process since this information is not readily available. Staff indicate that going forward they will be tracking advertising in more detail to support analysis as required.

This evaluation examined the currently available data that identifies PFP advertising costs and found that in FY 2014-15 the actual cost for advertising the PFP would have been over 200% of the program’s O&M budget. This supports the observation made by staff that the PFP advertising costs are kept as part of the overall hearing costs because the program’s O&M budget is limited.

Nevertheless, staff suggest that the PFP advertising should be both tracked and accounted for through the program’s budget rather than the hearing budget overall. This would allow PFP staff direct access to their data and initiate their PFP news release sooner rather than waiting for sign-off on the overall advertising plan for a hearing and also allow them to track and analyze the effectiveness of the advertising process.

Funding Review Committee

The Funding Review Committee (FRC) is also funded through the PFP budget and started off with compensation of $500 per diem for the first 6 eligible hearings. In 2013, this was raised to the current amount of $1000 per diem (pro-rated per hour). The program could not produce documentation with rationale for this change, however several staff mentioned that it could have been based on some comparison with other government consultant rates or based on feedback from committee members.

In 2012-13, the PFP spent a part of its budget to recruit members for its external funding review committee. Total costs incurred for newspaper advertising was around $30,000. Interviews with the Funding Review Committee members indicate that only a few found out about the opportunity through advertising, and rather they were informed by their network.

For the last five years, a Funding Review Committee was required to review applications, in order to mitigate the risk of bias. The program’s newer Terms and Conditions (June 2015) now state that it they may use a funding review committee to assess applications. Staff support using an independent committee to make recommendations for awards but indicate that it can be time consuming to coordinate an external committee especially with an increasing number of applications to review against a limited funding envelope. Members of the Funding Review Committee also indicate that it can be challenging to allocate funding on a rolling basis rather than waiting for all the applications to arrive and thus making an informed assessment and allocation. Staff suggests that they have already streamlined the work required by a FRC in allocating amounts by recently introducing funding caps per applicant or group.

The invoices and payments to FRC members were also assessed. Members receive an appointment letter stating which hearing they are assigned to assess funding applications for. It states the per diem and the other costs (such as travel) that will be reimbursed should they arise. Very recently, the program has added the estimated value of the agreement with the FRC member. Members are required to submit invoices detailing the nature of their work and time spent on it in order to claim their professional fees. As noted before, the per diem rate doubled from $500 to $1000 in FY 2013-14. For the ten hearings examined in this evaluation, the average FRC hours has been around 3.4 hours per application but has ranged on average from 0.6 to 9.1 hours. Figure 10 below illustrates the variation and range of FRC hours by project

The cost of a committee member to review each application ranges on average from $40 to $1,207 which is an average of $418 per application. This range in cost by member may be explained by the variation in hours. Most funding opportunities had at least two external members and sometimes three. Cases were observed where a FRC member would submit an invoice with more than three times as many hours as another member for the same funding opportunity review. No further work was done though in this evaluation to determine the possible explanations for this as the PFP staff have since changed and there was no evidence of the program analyzing the FRC and trends in cost or effort.

Approximately $95,822 has been spent on the FRC ($30,212 on advertising for FRC members in FY 2012-13 and $65,610 in professional fees for the ten hearings examined in the evaluation). This represents 2.3% of total funding awarded and around 2.4% of the value of contribution agreements signed.

Figure 10: Hours spent by the FRC to review applications

There has been administrative work related to hearings that were ultimately changed to written hearings or cancelled by the company, as well there would have been administrative costs in FY 2014-15 that relate to hearings which are still in progress or have not been awarded or paid out within the time period under review by this evaluation.

The program has identified the possibility of re-evaluating how it uses the external funding review committee especially in light of the more recent changes to the design of the program.

Table 24 and Table 25 below summarize the amounts requested by application and amount awarded respectively including the % each range represents in relation to the total number of applications or awards. The Tables account for ten hearings (Vantage to TMX)

Table 24: Breakdown of 157 Applications from for ten hearings

Table 25: Breakdown of 120 Awards for ten hearings

This data shows that 50% of applications requested more than $75,000, however only 9% were awarded this amount or more. Applicants were usually awarded less than they asked for in their application. This could be due to the funding application request including ineligible activities, or the higher level of demand for funding against a limited funding envelope which results in a smaller amount available for each award. The FRC reviews each application and as outlined in the Program Funding Guide, assesses the following^{Footnote 43}:

• eligibility of the applicant
• the applicant’s interest in the proposed project
• the potential for the proposed project to impact the applicant
• how important the participation of the applicant will be to the hearing
• whether anyone else has proposed to do the same work
• reasonableness of requested costs
• whether the applicant has other sources of funding
• whether expense proposed are high, medium or low priority

Following its review, the FRC provides a brief summary of reasons for its recommendation of awards in its report to the NEB. The data demonstrate that the FRC usually awards the full funding envelope and recommends the amount for each individual award, which is always less than what the applicants ask for. Over the last five years of the PFP, funding requests have almost always been higher than each funding envelope. Around $3.45 million has been requested over the last nine completed hearings, however only $1.03 million has been awarded from funding envelopes totalling $1.24 million. The difference between the amount requested and awarded is $2.42 million.

As stated previously, there are likely several factors contributing to this large difference between amount requested and awarded. For example, demand for funding might be less than the funding envelope for a given hearing. As shown in Table 26 below, there were two hearings where the total amount requested was less than the funding envelope. After the FRC reviewed the applications, the amount awarded was even less. Another factor could be the level of experience applicants have in filling in their applications and uncertainty how to calculate anticipated costs, or activities proposed could be ineligible as determined by the program or FRC. Another factor (which has since changed), was the lack of individual or group funding maximums to guide the applicant’s assessment of proposed expenses and total amount requested.

Table 26: Comparison of funding request to funding envelope

Demand for funding continues to be much higher than the level of funding available; however to conclude that there is a cost savings due to the assessment by the FRC should also take into account that the terms of the program require a funding review committee to assess applications against certain criteria and eligible costs before recommending awards. There is also difficulty in concluding whether the number of awards or the level of award has an effect on the program such that higher numbers and dollars would entail more time and effort by the program to manage and oversee. It is more likely that there is a direct link between the number of applications or number of funding opportunities occurring at the same time and an increase in program effort since each application is processed by the program and assessed by the FRC.

Implementing new Terms & Conditions

The new Terms and Conditions (2015) also introduce changes that may alter the demand to programs resources. Participant funding is no longer restricted to oral hearings but is extended to written public facilities hearings as well as the environmental assessment of designated projects under CEAA 2012 regulated by the Canada Oil and Gas Operations Act and for projects not subject to a public hearing under section 24 of the NEB Act but require participation by the public in an environmental assessment under CEAA 2012.

Meetings with internal stakeholders indicate that work remains to be done to implement the changes proposed by the new Terms and Conditions. The program is also still assessing its timelines for announcing funding, setting deadlines and determining how a funding review committee will operate under these changes.

4 CONCLUSIONS AND RECOMMENDATIONS

Overall, it is evident that the program has undergone a few phases of growth and development as it gains experience and knowledge. This section presents the evaluation’s conclusions based on the findings presented in Section 3, and makes recommendations based on that analysis. Some of the recommendations address more than one core evaluation question.

4.1 Demonstration of Relevance

Based on the evidence presented in Section 3.1, the Participant Funding Program is aligned with the roles, responsibilities and priorities of the federal government as well as the government priorities and the mandate, strategic outcome and priorities of the National Energy Board. There is a continuing need for PFP, primarily as a result of the National Energy Board Act and the Canadian Environmental Assessment Act (2012).

4.2 Demonstration of Effectiveness: Achievement of Outcomes

The logic model created for this evaluation was based on the program description from the 2010 PFP submission. A review of the original performance management framework reveals gaps in data collection and analysis.

This evaluation addresses the immediate and intermediate outcomes set for the program when it was initially introduced; however feedback from stakeholders as well as limited processes to collect and analyze information makes it difficult to fully conclude whether funding from the NEB’s PFP supports adequate levels of participation by the public or Aboriginal groups in oral hearings. As a result, this will make it more difficult to assess in a future evaluation whether the ultimate outcome of the PFP is achieved. This finding is important given the changes in June 2015 to the Terms and Conditions of the program and when determining the new program logic model and a performance management framework.

The program has made recent improvements and updates to its process-related documentation. A review of how the PFP fits into the overall hearing process, though, shows variation over the past five years regarding the timelines and key milestones. Feedback from stakeholders suggests that having a separate process to apply to participate (ATP) versus applying for funding is confusing and could be more efficient if combined or if their timelines are adjusted with respect to other administrative processes for the hearing. There was also concern about confusion caused by changes to the funding process that are not communicated right away or in advance of the change.

The PFP works with other business areas at the NEB to communicate information about the program. The website is used to provide information about current funding opportunities, engagement staff present information to the public and Aboriginal groups and newspaper advertising is used as one way to launch a funding opportunity. Staff from the finance team and applications and communications business units is essential to delivering the PFP and as such a common understanding of shared processes with the program is essential for its delivery.

Stakeholders support the objectives of the PFP and their feedback indicates that funding can off-set the cost of participation in the NEB hearings. Over the last five years, the number of applications has been variable, however most applicants with an award access their funding.

The PFP advertises each funding opportunity in newspapers. As advertising is funded outside of the PFP budget and managed by communications staff, there is a lack of analysis about the effectiveness of advertising and the benefits of this approach.

4.3 Demonstration of Efficiency and Economy

The PFP has used an external Funding Review Committee for the past five years to review applications and make funding recommendations. The program recently revised its Terms and Conditions (June 2015), which makes a FRC no longer mandatory.

The program has not been operating with a useful electronic platform for program management, creating an internal administrative burden and a risk of incomplete and inaccurate records and data.

• Recommendation 6: It is recommended that the program make further improvements to efficiencies in its process overall by implementing a unique and centralized system to collect, analyze and report information related to specific projects and the program overall. Such a system could also improve workflow and control of the process and program documentation.

When the PFP was first developed in 2010, the budget was set for FY 2010-11 to FY 2014-15 and ongoing. It was based on information and NEB hearing processes available at the time. Changes to legislation as well as the expanded scope of the PFP following the new Terms and Conditions introduced in June 2015 are not accounted for or anticipated by the budget set for the program in 2010. As well, inconsistent use of recording timesheets for work related to PFP hinders the ability to analyze and account for all staff time that contributes to the program.

• Recommendation 7: It is recommended that the Participant Funding Program use available data to track and monitor the level of contributions required as well as indirect costs (including all roles that are critical to operating the program). Tracking and monitoring all direct and indirect costs will allow the program to better assess its budgetary needs over time.

Annex 1: Participant Funding Program Logic Model

Annex 2: History of the Participant Funding Program

Annex 3: Participant Funding Program Evaluation Matrix

RELEVANCE

Core Issue #1 - Alignment with federal roles and responsibilities
Assessment of the role and responsibilities of the federal government in delivering the program

Core Issue #2 - Alignment with government priorities
Assessment of the linkages between program objectives, federal government priorities and departmental strategic outcomes

Core Issue #3 - Continued need for the program
Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.

PERFORMANCE

Core Issue #4 - Achievement of expected outcomes
Assessment of progress toward expected outcomes with reference to performance targets, program reach and program design, including the linkage and contribution of outputs to outcomes.

PERFORMANCE

Core Issue #4 - Achievement of expected outcomes
Assessment of progress toward expected outcomes with reference to performance targets, program reach and program design, including the linkage and contribution of outputs to outcomes.

EFFICIENCY AND ECONOMY

Core Issue #5 - Demonstration of efficiency and economy
Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.

Annex 4: Management of the Evaluation

Roles and Responsibilities

The evaluation was carried out by an evaluation specialist at the NEB responsible for all phases of the evaluation (planning, conducting and reporting). The Director of Corporate Performance reviewed and provided feedback on the terms of reference, evaluation plan, logic model, findings, draft and final reports as well as recommendations and participated in key meetings.

Program and senior staff, as well as senior management reviewed the draft report and provided feedback and comments. The Evaluation Committee at the NEB, chaired by the Chief Operating Officer, is responsible for reviewing evaluation plans, reports, management responses and recommending approval to the Chair/CEO. The Committee met in November and provided feedback. The Evaluation Report was updated and presented to the Chair/CEO for approval.

Contracts

The external online survey was contracted to Prairie Research Associates Inc. (PRA) and took place in between February 27 and March 30, 2015. The survey was designed in consultation with the NEB and sent to 44 past applicants to the program. A report was prepared and submitted by the contractor in April 2015 which summarizes the results and comments from the respondents. The questions focused on participant awareness of NEB hearings and funding, communication with the NEB, the processes for applying and receiving notification of a decision on funding, participation in the hearing and the payments process.

The report provides a caution about interpreting the results, though; because the response rate was only 18% (8 participants completed the survey). Nevertheless, the results are mentioned throughout this evaluation in addition to other lines of evidence such as complaints by applicants or applicant responses to surveys built into the application form.

Challenges

With the approval of the Terms of Reference for the Evaluation in early July 2015, this left a very tight timeline to collect and analyze data, conduct interviews and writing to achieve a final published report by the original deadline of September 30, 2015. The timelines were revised and resulted in a draft report in October 2015 and subsequent review of the evaluation report from the program and management took place in November/December 2015.

Timelines

Table 27 illustrates the planned tasks (highlighted in grey) versus the actual tasks if they differed (highlighted in blue). Work on planning the evaluation started in late FY 2014-15, however due to a change in staff the evaluation core work occurred between July and October 2015.

Table 27: PFP Evaluation Timelines

Annex 5: References

Reports & Guidance

• Canadian Nuclear Safety Commission. Participant Funding Program Evaluation (2015)
• External Advisory Committee on Smart Regulation: Smart Regulation, a Regulatory Strategy for Canada (2004)
• Federal Court Decision: 2015 FC 1005 (2015)
• Public Works and Government Services Canada: Budget 2010
• National Energy Board: Annual Report (2008)
• National Energy Board: Departmental Performance Reports 2006-07 to 2014-15
• National Energy Board: Reports on Plans and Priorities 2006-07 to 2014-15
• National Energy Board: Guide for Landowners and the Public (2010)
• National Energy Board: Filing Manual
• National Energy Board. Hearing Process Handbook: A Guide to NEB Hearings (2013)
• National Energy Board Report: Intervenor Funding Options (March 1996)
• National Energy Board: Participant Funding Guide (2010)
• National Energy Board: Participant Funding Guide (2014)
• Treasury Board Secretariat: Assessing Program Resource Utilization When Evaluating Federal Programs (2013)
• Treasury Board Secretariat Guideline on Transfer Payments
• Treasury Board Secretariat: InfoBase
• Treasury Board Secretariat, Centre of Excellence for Evaluation: Results-based Management Lexicon.

Legislation

• Canadian Environmental Assessment Act (2012)
• Financial Administration Act
• Jobs and Economic Growth Act
• National Energy Board Act

Policies & Directives

• Treasury Board Secretariat Directive on Evaluation (2009)
• Treasury Board Secretariat Policy on Evaluation (2009)
• Treasury Board Secretariat Policy on Transfer Payments (2008)

Annex 6: List of Hearings Eligible for Participant Funding

The following table shows the hearings that are or have been eligible for participant funding. Company names and Project Names are from the Project Applications. The first date is the date of the PFP news release and the second date is the date of the Decision/Report issued by the NEB. If not available yet, this is indicated by “TBD” (to be determined).

Annex 7: Management Response

Date modified:

2023-06-09