Undertaking effective M&E during COVID-19 – reflecting on lessons learned from the Agulhas independent evaluation of the Initiative for Climate Action Transparency, supported by CIFF
by Mark Kelleher, Catherine Cameron and Katerina Cerna In the first few months of 2020 one issue has dominated worldwide: the COVID-19 pandemic, and the way that this emergency has changed patterns of living and working. Commentators have pointed to a total re-evaluation of work as we know it and business leaders have heralded the […]
04.05.20
[addtoany]
by Mark Kelleher, Catherine
Cameron and Katerina Cerna
In the first few months of 2020 one issue has dominated worldwide: the COVID-19 pandemic, and the way that this emergency has changed patterns of living and working. Commentators have pointed to a total re-evaluation of work as we know it and business leaders have heralded the emergence of a “new normal” of remote working[1].
However, one
wonders how many of these changes are ‘revolutionary’, and how many are changes
that many have wanted to make for a while. Particularly relevant for our work
on climate change – the working methods we see today brought on by COVID-19 are
synonymous with the sorts of changes that are routinely recommended as ways to reduce
our carbon footprint.
To illustrate how organisations can undertake effective Monitoring and Evaluation work during COVID-19 (and in future) we reflect on the approach and methodologies Agulhas used for a recent evaluation of the Initiative for Climate Action Transparency (ICAT) supported by The Children’s Investment Fund Foundation (CIFF) and the practical lessons on remote working that we learned along the way.
Although
COVID-19 was not a pertinent issue for this particular evaluation, at the design
phase we worked with the CIFF and ICAT to make our evaluation as low-impact and
collaborative as possible. This meant that the way our international team of
researchers – split across both hemispheres – conducted this evaluation took
advantage of virtual working, involved a minimum of travel, and utilised the
knowledge and cooperation of existing stakeholder partnerships. The methodology
for this evaluation was designed to provide both robustness and flexibility in the
cross-triangulation of data from document reviews, interviews, a workshop and two
on-line country stakeholder surveys.
Let’s have a look at the five steps we
followed:
Step One: Starting from the source material. Desk
reviews are an important part of any evaluation so make sure that you have all
the material you need to begin.
✪ Ensuring early, open and transparent
access to project documents is important.
Step Two: Phone
calls and videoconferencing are important tools to use when conducting remote
evaluations. We held monthly check-ins with the ICAT Secretariat and CIFF; and
conducted remote semi-structured interviews.
✪ Where possible use an online meeting tool that the key stakeholders and their network are already familiar with.
Step Three: Other
methodologies, for example using online surveys, support in-depth data collection
from larger sample sizes. Given the high number of countries that ICAT is supporting,
(over 30) it was clear that we could not seek to interview all national
stakeholders in person. In order to include voices from all participating
countries in the evaluation equally, we developed an online survey for key
national stakeholders to complete. It is extremely important to share your
survey design with key stakeholders, in order to refine it thoroughly before it
is sent out.
✪ Using the communication networks of the
organisation you are evaluating greatly improves survey response rates.
Step Four: Findings
need to be triangulated and sense-checked. A survey generates a lot of data, sense-checking
was done through an interactive workshop with the ICAT network. The workshop
was held (pre-C19) in London although some stakeholders joined remotely from
around the world. Since the lockdown we have held similar sense checking workshops
for other assignments using only online participation.
✪ Running
group sessions in an online environment allows participants to join that would not
otherwise be able to make it in person, and places all participants on a level
playing field.
Step Five: Case studies help to bring out in-depth
examples of a programme’s impact. For
the case studies we conducted of six ICAT partner countries, we were able to
complete these remotely through a combination of online interviews, desk
reviews and the fact checking of ICAT’s implementing partners.
✪ Carrying
out these case studies in-country would have been highly time and carbon
consuming, which we were able to avoid.
The carbon emissions saved
By doing the country case studies remotely we saved almost 22 tonnes of carbon dioxide by not taking the flights shown on the map.[2]
Some fieldwork and data collection
methods are quite difficult to adapt to remote working.
In environments
where IT access is a challenge, video conferencing and skype calls may not be
an option. For ICAT, we were conducting interviews with professional staff, largely
capital city based, working in an office environment. However, in cases where one
needs to gather broader public input this can be challenging. Particularly in
cases where partners are located in developing countries with already low
capacities, an in-country visit may be required to advance the process for the ICAT
leadership team. However, adaptability and flexibility is key, and it is always
useful to consider alternatives. In the case of the evaluation team there are a
number of adaptive data collection methods that can be used for remote
fieldwork.
✪ Our colleagues at the Dala Institute in Indonesia have helpfully produced a blog on some of the challenges and solutions when carrying out fieldwork on the ground involving community members.[3]
Holding seminars and workshops online loses
some human contact and discussion ‘in-the margins’.
Although we are
all rapidly adapting to new methods of working, anyone that has participated in
an online meeting or workshop over the past month will already know that these
can be a little slower, that the conversation doesn’t always flow quite as well,
and there can be glitches in even the best wifi.
✪ Allow the space for everyone to log-on, to respond during the meeting and build regular breaks into the agenda in order to keep the conversation fresh and engaging. Divide the meeting into key segments so that busy stakeholders can join for key sessions.
We may need to change what is being
measured.
In addition to
adapting in terms of HOW we are conducting our evaluations by finding other
ways to collect data (e.g. phone-based, online surveys, expanding desk
reviews), we may also need to change WHAT we are measuring in cases where
disruptions to programme implementation or delays in measurement are a threat
to validity (for example, doing an endline just after/later than planned in an evaluation
could lead to lower statistical power and a null result, which may be due to
COVID19 and not to programme failure).
✪
This is more pertinent to health evaluations rather
than the case study here.
There may be ethical issues to moving to online
data collection.
For example
bias from excluding those without phones or access to power to charge phones, or
the ethics of calling those who consented to healthcare follow-up, but not to research
where databases do exist, shared phones, risks of data breaches.
✪ Colleagues at 3iE have produced a useful blog that reviews some of these issues.[4]
To conclude:
It is possible
to conduct efficient, robust, productive evaluations remotely. This requires some
planning ahead, and access to good communications technology, but can save on
costs and carbon. And in fact, during lockdown conditions, there is potentially
an opportunity for teams working on climate transparency to intensify some of
their data gathering and analysis work, while other activities are being placed
on hold.
Finally, the first quarter of 2020 has seen an unprecedented fall in CO2 emissions and other pollutants worldwide, with analysis showing that the CO2 reduction this year will be greater than any year previously[5].
Post 2020, it will be difficult – if not impossible – to try to maintain that so many meetings need to be conducted face-to-face, that each interaction needs to be finalised with a handshake: after all, virtual elbow bumps are available.
