Language selection

Search

Summative Evaluation of the JURIST and IMPACT Justice Projects - Executive summary

The findings, conclusions, lessons learned and recommendations in this document are those of a third-party contractor. Their findings do not necessarily reflect the views of the Government of Canada. We do not guarantee the accuracy of the information provided in this report.

On this page

Purpose and object of the evaluation

Canada’s Department of Foreign Affairs, Trade and Development (DFATD) (now known as 抖淫视频 (GAC) commissioned this combined evaluation of two projects implemented in the Caribbean: the Judicial Reform and Institutional Strengthening (JURIST) project and the Improved Access to Justice in the Caribbean (IMPACT Justice) project. The rationale for an evaluation that combines both projects is that they were designed by GAC and its local implementing partners (IPs) to be complementary.  

The purpose of the evaluation was to assess the achievement of results and inform decision-making regarding the Government of Canada’s (GoC) work in the justice reform sector within the Caribbean. The object of the evaluation was the two completed projects, JURIST and IMPACT Justice. The scope of the evaluation included both of these projects in their entirety, from inception to their closure, a period of nine fiscal years.

Key Stakeholders of the evaluation

This evaluation was undertaken for the three key stakeholders: the project IPs—the Caribbean Court of Justice (CCJ) for JURIST and the University of the West Indies (UWI) for IMPACT Justice; and Government of Canada (GoC), the donor; and the evaluation’s commissioning agent. This external, independent, summative evaluation took place more than a year after the projects ended in March 2023.

Objectives of the evaluation

The specific objectives of the evaluation were to a) assess the effectiveness of the projects and the integration of crosscutting themes, b) identify the factors that facilitated or constrained results, c) assess the relevance and sustainability of results and d) provide findings, lessons learned, conclusions and recommendations on Canada’s programming in the justice sector in the Caribbean.

Summary description of the projects evaluated

JURIST was contracted through a five-year $19.4 million Contribution Arrangement signed by the Government of Canada (GoC) in 2014. In 2019 the project was granted a four-year no-cost extension to March 2023. Caribbean governments and institutions provided $4.0 million of additional in-kind contributions. The CCJ implemented JURIST on behalf of the Conference Heads of Judiciary (CHJ) of CARICOM. The project was designed to support judiciaries across English-speaking Caribbean countries in their own reform efforts, focusing on a) gender-responsive delivery of court services, b) public education and knowledge management, and c) court capacity.

IMPACT Justice was contracted through a five-year $19.2 million Contribution Agreement, also signed by the GoC in 2014. In 2019, IMPACT Justice was granted a three-year, no-cost extension to March 2022. In 2022, the project was granted an additional one-year no-cost extension, and thus, the project officially ended in March 2023. Caribbean governments and institutions provided an additional in-kind contribution of $3.0 million. The project was implemented by UWI and focused on 14 CARICOM member states. Its primary partners were governments, regional institutions, and civil society institutions. The project had four main components: a) model laws and treaties, b) continuing legal professional development, c) legal information and education, and d) alternative dispute resolution (ADR).

The ultimate outcome that JURIST was expected to contribute to was: A judicial system that is more responsive to the needs of women, men, youth, business, and the poor. The ultimate outcome that IMPACT Justice was expected to contribute to was: Enhanced access to justice benefitting men, women, youth, and businesses in the CARICOM Regions. Stakeholders of the two projects included the donor (GoC), implementing partners (UWI and CCJ), intermediaries, others (such as individuals, groups, or government representatives that had an interest in a particular project activity or outcome), and beneficiaries.

Approach and methodology of the evaluation

Using GAC’s Evaluation Design Matrix, the evaluation was structured around eight specific questions related to relevance, effectiveness, efficiency, and sustainability. It applied a non-experimental approach which relied on both quantitative and qualitative data. The approach supported a triangulated evidence-based assessment of what was implemented and what could be attributed to the project. The evaluation considered multiple variables simultaneously. The analysis did not attempt to statistically measure the significance of the relationships between these variables. Instead, inferences were expressed as evidence-based findings. The main methods used for primary and secondary data collection included a) document review, b) key informant interviews, and c) expert panels. Post-data collection debriefing sessions were used to further validate findings.

The Evaluation Team (ET) started with a set of documents made available from within GAC’s electronic project database. This e-library expanded during the data collection and analysis phase as the evaluators added relevant documents and information from internet searches. The evaluation focused on a sample of eight countries (Antigua and Barbuda, , , , , , Trinidad and Tobago) to allow a deeper level of analysis. In total, 80 KIs were interviewed, of which 49 were women.

Limitations

The evaluation took place more than a year after these projects closed. There had been significant staff turnover, and there were no reported plans for another phase of these projects, which may have motivated local interest. Given this reality, the ET found significant attrition in stakeholder interest. Finding the right people with knowledge of how the projects were planned and implemented was sometimes challenging. To mitigate this, the ET worked closely with GoC staff based in Barbados and relied on their established connections and relationships with CCJ, UWI, and other intermediary stakeholders in the region. Data collection from document review was core to the evaluation’s methodology, with KIIs playing a complementary role.

Evaluation Findings

  1. Both projects' designs were participatory, directly involved essential stakeholders, and responsively adjusted throughout their entire project cycles.
  2. GoC’s decision during the design process to contract established local institutions, CCJ and UWI, as executing agencies helped assure the ongoing relevance of both projects to regional needs.
  3. When assessing results relative to targeted project-specific key performance indicators, most (87 percent) of expected immediate and intermediate outcomes were partially to fully achieved.
  4. Evidence of outcome achievement for JURIST was strongest for a) accelerated judicial reform consistent with best practice standards, b) improved gender-sensitive court and judicial service delivery, and c) reduced court case backlog.
  5. Evidence of outcome achievement for IMPACT Justice was strongest for a) new model legislation, b) more integrated GE considerations within model legislation and legal education materials and courses, and c) improved knowledge management.
  6. For JURIST, outcomes where progress was the most disappointing included a) development of CCJ’s envisaged knowledge management system (CJIS), b) integration of effective ADR into the court system, and c) improved court capacity for ongoing public education and engagement.
  7. For IMPACT, the outcome for which progress was the most challenging was the establishment of community-based ADR networks and service points.
  8. The unexpected positive outcomes most frequently cited were a) project contribution to disaster preparedness and response efforts and b) the high-profile nature of the projects, which provided GoC with additional diplomatic standing and space to support a range of value-based development initiatives in the region.
  9. The most frequently cited negative outcome of these two large projects—implemented in the same sector and with similar timelines and budget sizes—was the missed opportunity to collaborate effectively in the overlapping areas of knowledge management, public education, and ADR.
  10. Project-developed and promoted capacity development strategies were mostly limited in scope, and instead of taking a holistic systems-approach, tended to focus on easily measured activities and outputs.
  11. JURIST's consistent close partnership with the Caribbean Association of Judicial Officers (CAJO) and the Caribbean Academy for Law (CAL) helped build the relevance and capacity of these important regional organizations.
  12. GoC’s localization strategy—using local rather than Canadian or UN executing agencies as project implementors—supported ownership, contributed to local institutional development, and supported the achievement of results.
  13. These large projects each had a wide range of initiatives, and both tended to be responsive and had difficulty focusing on annual work plans and budgeting strategically.
  14. The projects had two separate disconnected logic models and were implemented as such rather than the PMUs actively seeking opportunities to create synergies across related expected outcomes.
  15. It was often difficult for GoC to know what kind of organizational support (mentoring, coaching, facilitating, instructing, etc.) was sufficient and appropriate when interacting with IPs on design and performance concerns, and IPs were not always clear on what they needed when asking for help.
  16. While the application of a gender lens was not fully integrated during the design phase as a cross-cutting theme for either project, gender equality (GE) became increasingly central to the projects after the introduction of the Feminist International Assistance Policy (FIAP).
  17. There is virtually unanimous consensus among key project stakeholders that the integration of GE was important and successful with the following three results most frequently cited: a) development of a range of gender-sensitive products (legislation, legal education materials, and client service standards), b) creation of a model court and operational guidelines for adjudication of sexual offenses, c) new awareness within the legal system of GE as an issue.
  18. Good governance was an inseparable underlying theme of both projects, given the foundational role the rule of law plays in the democratic Caribbean countries targeted for justice-sector reform.
  19. Advocating respect for human rights was integrated into the design of both projects and integrated as cross-cutting activities within annual approved workplans, e.g., targeting people with disabilities and Indigenous populations that have difficulty accessing the justice system.
  20. Environmental sustainability had limited relevance as a cross-cutting issue and, at times, distracted project managers from the central theme of justice reform.
  21. JURIST’s design was informed by and aligned with a globally recognized international framework for court excellence.
  22. The key factors that support the continued sustainability of project results include a) a committed regional agenda for change, b) well-positioned change leaders, and c) the momentum of outcomes achieved thus far.
  23. Key factors that could constrain the achievement of sustainability include a) regional diversity and shifting priorities, b) lack of financial and human resources within small countries), c) tendency for change agents to work in silos rather than actively collaborate, and d) complexity of the remaining issues.

Conclusions

Relevance

Input and direction from a wide range of appropriate local stakeholder representatives assured ongoing relevance. Canada’s commitment responded to the region’s own justice reform efforts and priorities. Strategically, the projects were designed to align with the areas of judicial reform that CARICOM states themselves had formally selected for priority action. Using local IPs assured ownership and sustained local involvement, and ongoing relevance of the projects to both national and regional needs. The established relationships these two organizations have in the region, and their understanding of the local context helped them navigate the complex political nature of regional justice reform.

Effectiveness (Outcomes)

JURIST accelerated judicial reform at regional and national levels. The Project made a meaningful contribution and positively affected the lives of the regional stakeholders it touched. The capacity of judicial officers and courts in the region improved, influenced by a wide range of reform initiatives. Court case backlogs were reduced in at least 4 countries. Achievement was more limited for the development of CCJ’s knowledge management system, integration of effective ADR into the court system, and public education and engagement.

Outcome achievement for IMPACT Justice was strongest for model legislation, integration of GE considerations within model legislation, production of legal education materials and courses, and improved knowledge management. Positive outcome from the project’s legislative work continues to ripple out well after the project ended. The latest iteration (vLex) connects UWI’s database to a global platform that supports technical innovation, security, and maintenance. 

For IMPACT Justice, the establishment of community-based ADR networks and service points was disappointing. While the number of individuals trained at both introductory and advanced levels was impressive, this did not lead to established networks, nor a region-wide training curriculum or accreditation system.

Because these projects were large and broad, they attracted regular media attention and often intersected with other GoC interests within the region. This helped place Canada in a leadership position within justice reform across the region which in turn gave it diplomatic recognition and standing. The projects reinforced the perspective in the region of Canada as a reliable and values-driven development partner.

Effectiveness (Capacity Building)

Project-developed and promoted capacity-building strategies tended to focus on easily measured activities and outputs instead of taking a holistic systems approach. To improve CLPD, IMPACT Justice focused on developing a Model Legal Profession Bill, offering a range of continuing education events. These initiatives did not come together as a comprehensive strategy. Instead, the measures of success were counts of people (for example, attorneys engaged in one-off events).

The case of IMPACT Justice’s strategy for improving ADR was similar. The expected result was a region-wide ADR training policy and accreditation system, and accessible ADR networks and service points. However, there was no explicit strategy to achieve this. Instead, the project focused on a narrower training-of-trainers strategy and training people. A positive exception was the rollout of Restorative Justice practice in schools. Both IMPACT Justice and JURIST worked on ADR support in their own silos.

To improve the capacity of courts to deliver responsive services, the strategy adopted by JURIST was to make performance standards more transparent, sensitize and train judicial officers and court staff, open specialized courts, create operational manuals and bench books, further develop the CJIS, and improve case management using new technology. However, these were mostly managed as separate initiatives rather than guided by an overarching strategy with ongoing monitoring to determine what worked best and what needed to be adjusted. A performance highlight was the close partnership by JURIST with the Caribbean Association of Judicial Officers (CAJO) which helped build the relevance and capacity of this important regional organization.

Integration of Gender Equality

The integration of GE contributed to the development of a range of gender-sensitive products and new awareness within the legal system of GE as an issue. A gender lens was apparent during the design and early implementation process as a cross-cutting theme and was applied even more consistently once the FIAP was announced. Aligned with the FIAP, both projects also include initiatives to engage men to address GE more holistically.

Influenced by FIAP, the establishment of a Sexual Offense Model Court (SOMC) was a key additional initiative for JURIST. The project achieved success in developing comprehensive court procedures and guidelines aimed at addressing sexual offense cases with both efficiency and sensitivity to the experiences of victims. These procedures and guidelines streamlined the judicial process and will likely help reduce delays and minimize the distress that victims of sexual abuse often face.

Neither project had a comprehensive or dedicated strategy outlining how GE would be addressed throughout the entire project lifecycle. Instead, the approach was to incorporate aspects of GE into specific initiatives and activities of the projects as these were designed and implemented. Late inclusion of GE specialists within PMU teams meant that their roles were more reactive than proactive, focusing on specific gender-related tasks rather than influencing the overall direction or management of the projects.

The projects struggled to effectively engage the public on issues related to GE. There were missed opportunities to fully integrate GE into public awareness and education campaigns.

Efficiency (Management)

There were many initiatives in which the two projects could have collaborated more directly. The projects found it difficult to share strategies, integrate planning and successfully find synergies. Greater advocacy of systems thinking – seeing the whole system rather than being overly focused by GoC on quantitative KPIs – may have helped PMU staff connect their areas of work across the projects and gain a deeper understanding of key interrelationships.

The decision to contract UWI and CCJ was an innovative and values-driven move by GoC that supported ownership, added to local institutional development, and helped to achieve results. Localization did not automatically lead to smoother programming, and it required additional skills and resources from GoC. Tensions around issues of legitimacy and representation had to be carefully navigated and managed, and competition between the IPs was an unexpected challenge. Additional resources and mentoring were required to help local entities comply with GoC’s planning, budgeting, monitoring, and reporting requirements.

Professionally facilitated participatory reflection events during which performance was openly discussed and reviewed, and priorities updates were too infrequent. That said, when these events did occur, they were well received and made contributions to relevance, effectiveness, and efficiency.

Crosscutting Themes

Good governance and advocating respect for human rights were inseparable underlying themes for both projects. Access to justice and fair adjudication pointed reform-minded change leaders to poverty and the most vulnerable, marginalized poor. The Disability and Inclusion Awareness Guidelines for Judiciaries and Judicial Officers, released by CAJO in 2023, suggest that a focus on human rights and accessibility will continue. Both projects made formal efforts to integrate environmental sustainability as a cross-cutting theme. However, relevance and results related to this theme were mostly pro forma and tenuous.

Sustainability

Evidence of sustainability is positive. The legislation developed by IMPACT Justice will continue to serve as models and influence adjudication with greater sensitivity to GE and human rights. Trained legislative drafters are available in the region to engage in the ongoing process of updating legislation. While progress on legal education was limited, attorneys and their bar associations were further sensitized, and calls for CLPD with accountability frameworks continued. UWI’s agreement with vLex promises a more sustainable and sophisticated legal information database.

For JURIST, customer-focused and gender-responsive court and judicial service delivery improvements in the region were strengthened without obvious backsliding or loss of commitment to the positive changes made. There are new ways of thinking that can be attributed to the project. Change leaders continue to draw on the national, regional, and global contacts and networks made through JURIST. The CCJ is an active member of the International Consortium for Court Excellence and is committed to the reform framework provided by this association.

Looking across CARICOM, the momentum of measurable outcomes already achieved is not likely to be reversed, given the region’s long democratic traditions and formal commitments in this direction. Key factors that will likely continue to challenge the ongoing sustainability of justice reform efforts include regional diversity and the lack of financial and human resources within small countries.

Lessons Learned

Recommendations

  1. To better conceptualize, coordinate and manage GoC contributions in the Caribbean, GoC should clarify its overall programming intentions for the region.
  2. To add to learning for how to move a localization agenda forward, GoC should use the practical experience of locally led development which these projects offer.
  3. As a final capstone for these two significant projects, GoC and the two IPs should use the final report of this evaluation to organize a celebratory event in the region.
  4. When supporting efforts that aim for systems-level change, GoC and its IPs should build in regular (at least annual) professionally facilitated reflection, performance review, and priority-setting events.
  5. For complex programming such as justice-system reform, GoC should fully resource a central role for a capable, on-the-ground, external MERL expert.
  6. When programming for capacity development, UWI and CCJ should move beyond a quantitative focus on numbers trained and look to fund expertise in behavioural change education.
  7. To encourage and support ongoing judicial reform across CARICOM member states, CCJ should continue to actively engage with the International Framework for Court Excellence (IFCE) provided through its membership in the International Consortium for Court Excellence (CCI).
Date modified: