Project EVALUATION
SEPTEMBER – OCTOBER 2019
Background and Context
Lutheran World Relief is implementing two projects in Indonesia: Improved Coffee and Cocoa Producer Organizations in Bali and NTT, and Youth in Sustainable Agriculture and Entrepreneurship in Flores.
Improved Coffee and Cocoa Producer Organization
Improving Coffee and Cocoa Producer Organizations in Bali and NTT is a three-year project designed to improve the capacity of local farmer organizations. Coffee and cocoa producers face a wide range of challenges attempting to produce good quality cocoa and coffee at the farm level. These include but are not limited to: a lack of agricultural and business knowledge and growing and primary process skills, a lack of money to invest in their farms, and a lack of access to markets and financial institutions. Some farmers understand they will be in a better bargaining position when they are members of a registered and recognized organization. However, the existing farmer organizations need to be improved to meet market requirements and to respond to farmer expectations. The project is focusing on building the capacity of farmers’ organizations so that they can better serve their members in these three functional areas: 1) better product quality, 2) better service in obtaining finance, and 3) better access to markets.
GOAL: Increase cooperatives' capacity to deliver better services to members
OUTCOME 1: Cooperative in Bali and Flores have increased their access to financing to buy more f their members' produce
OUTCOME 2: Cooperative in Bali and Flores has increase their access to various market to have better price for farmers' product
Youth in Sustainable Agriculture and Entrepreneurship
The participation of youth in agriculture is decreasing national wide due to irregular and unstable income. Likewise, food crop, in coffee commodity, there is limited number of youth who actively engaged. Indonesia Statistic Board noted that number of farmer has decreased and 5 million households are no longer farmer. The age of farmers is getting older. 60.8% of farmers are at above 45 year old with limited education background (basic school only) and lack of new technology adoption. In contrast, regeneration process particularly on food related farming is very slow and youth prefer to work in. Youth viewed agriculture did not provide regular income since the profit will be received only during harvest session instead of monthly basis. Moreover, agriculture did not also give stable income during its harvest due to many factors, such as production failure and market price fluctuation. Consequently, youth preferred to work in the city for different types of job, including daily labor. Combined with selling farming land and convert it to the infrastructure, such as hotel, shops, and tourist facility, this, furthermore, has caused decreased agriculture productivity industry sector instead of farming
There are many variables influence youth’s (within farmer families) passion to be farmers, including sex and main activity/occupation, interest, ambition, and knowledge on farmer regeneration This project is aimed to increase youth participation in agriculture by proving that agriculture is promising income source through creating entrepreneurship opportunities and improve on-farm practice. Youth will be engaged not only in farming and cultivating coffee or other crops, but also in building entrepreneurship utilizing value added from their farming product.
GOAL: Increased the number of youth (men and women) who participated in agriculture due to promising income
OUTCOME 1: Increased off-farm entrepreneurship skills and opportunity
OUTCOME 2: Increased knowledge and skills of youth (men and women) on on-farm management
Evaluation Purpose And Use
These two projects will be finished in September/October. The end term evaluation has been planned by engaging an external consultant to assess the progress made by the project in achieving the overall goal and objectives of the project, as well as the overall effectiveness of the interventions, document major learnings on what worked well and what didn’t which would help LWR design better programs in the future. This is especially important as LWR implements its new Rural Economy and Livelihood (REAL) strategy.
Evaluation Scope
The objectives of the evaluation are to assess:
· The achievement of projects’ outcomes based on their respective set of indicators/ targets to date as well as the projects key evaluation questions.
· The project performance based on OECD-DAC evaluation criteria.
· Analyze the project approach, strengths and weaknesses; issue and challenges of major components of this project and provide recommendation for future
The assessment will be based on OECD-DAC criteria given below.
· Relevance or appropriateness of the project: To assess whether the project interventions have connected properly to the need of the target community.
· Efficiency: To measure the extent to which the results both quantitative and qualitative are achieved as a result of the interventions or project activities or were there any major deviations made for the project success in response to external context, if any.
· Effectiveness: To measure the extent to which the project achieved the outcomes and indicators outlines in the project; allowed for participants participation in the implementation of the project interventions and developing their ownership and accountability and the extent to which LWR’s accompaniment contributed in meeting the project objectives and outcomes?
· Impact: What were the changes or effects both positive and negative or intended and unintended on the community, beneficiaries ,gender and village groups/SHGs ;What are the community/beneficiaries perspective on the immediate and intermediate effects (quotes can be collected to provide evidence on participant’s perspective)and what were the partner staff perspectives on the effects
· Sustainability: Measure the extent to which the project results and impacts or positive outcome at the community level are sustainable in the longer terms (quotes can be collected that can provide evidence on participant’s perspective).
Evaluation Methodology
The Consultant should work in close coordination with LWR team to carry out the study. The Consultant is expected to conduct:
Desk Review
The Consultant will complete a desk review of key program documentation including project design documents, monitoring and progress reports, and other relevant publications. The purpose of the desk review is twofold:
i. to better understand program operations on the ground;
ii. to become familiar with existing analytical frameworks used to evaluate such types of programs; and
The evaluation will also use quantitative and qualitative method in this study.
Quantitative Study
Administer a rapid beneficiary survey to an estimated 450 households. The table below gives the distribution of respondents per project and location that should be included in the study.
Project
Improved Coffee and Cocoa Producer Organization
Sample Size
90% CL, 5% CI
222
Youth in Sustainable Agriculture and Entrepreneurship
Sample Size
90% CL, 5% CI
207
The draft questionnaire in English will be provided to the Consultant after Contract signing. The Consultant is expected to translate, pre-test, and finalize the questionnaire. Enumeration will be done using Android devices. LWR will provide the Consultant with access to Commcare for enumeration purposes.
Qualitative Study
The Consultant is expected to conduct qualitative study as well. The qualitative study is intended to provide better insights into the changes that happened in the community as well as inform policymakers and implementers about how the project were implemented and maintained. As such, the qualitative study is envisaged to go deep into the discussions rather than go wide and cover more villages. The Consultant is expected to implement two approaches: FGD and KII. The proposed number for each method per project is given below:
Project
Improved Coffee and Cocoa Producer Organization
FGD 9
KII 3
Youth in Sustainable Agriculture and Entrepreneurship
FGD 8
KII 4
Qualitative data shall be captured using key notes using the template that will be provided by LWR.
Description of Tasks
Task 1 – Write Inception Report and Work Plan
The Consultant shall submit an inception report and work plan that includes the following:
Understanding of the engagement and tasks required
Substantive input into design of draft data collection instruments, methodology and overall study approach
Team composition and recruitment plan
Numbers, dates, duration, and location of training sessions; including composition of training teams, agenda, preparation of logistics, and reporting arrangement to the LWR Evaluation Coordinator
A list of permits and letters required to conduct fieldwork and when those will be obtained
Expected duration of survey implementation and work plan for Field Teams including supervision plan
Expected date and duration of the Data Cleaning activities, including delivery dates of the different datasets
Task 2 – Prepare for field work
Preparation for field work entails a number of subtasks that must be completed in order to ensure the quality of data that will be collected. It includes:
· Recruitment and selection of field teams and data processing teams.
Interviewers and facilitators will be responsible for collecting the data in the field.. Interviewers that will be selected must have demonstrated experiences in administering surveys or facilitating discussions. Each field team must be headed by a field supervisor which will oversee the over-all management of field work of the team including backchecking and other data quality processes on field. Data processin team are responsible for ensuring that data collected in the field are free from errors prior to analysis. They conduct the data transformations as required by the Consultant.
· Prepare research permits
In the event that the community or municipality will require permits for field work, it is the responsibility of the Consultants to obtain these permits.
· Finalization of data collection instruments
All data collection instruments must be tested before being used. Testing procedures could be pilot testing or cognitive pretesting. The objectives of the testing are:
o allow the Consultant to familiarize itself with the instruments,
o test the logistics (such as field mobilization and demobilization, potential locations of basecamps) and time necessary for the surveys
o test the draft questionnaires. Testing the draft questionnaires will include identifying any exercises/questions that are not working as currently framed.
· Familiarize with the use of COMMCARE
LWR prefers data collection using COMMCARE. Consultant must familiarize themselves with the use of the software before field work.
· Training of Interviewers and Facilitators
LWR strongly prefers that all field team members be trained in one central location. If Consultant proposes training in multiple locations, the proposal should detail how the service provider will ensure consistency across training locations and common understanding of questionnaires and data collection protocols.
Note that the Consultant is required to begin fieldwork within one week of the end of training. If there is a delay of more than one week, the consultant will be required to do a one-day refresher training to all interviewers at its own expense.
Task 3 – Collect Data
· Data Collection (Quantitative)
LWR will provide the consultant with a list of person to interview. Field interviewers must follow the list and if for some reasons, the list was not followed, the Consultant is required to reinterview at its own expense.
· Data Collection (Qualitative)
Appropriate venues for the conduct of FGDs must be identified and booked before the activity. Invitations for participants to participate must be sent out a couple of days before the discussion. In the event that, less than 4 participants showed up for the FGD, the activity will still continue, with the facilitator adopting KII approaches. For KIIs, appointment must be booked in advance and reconfirmed a couple of days before the interview. Data recording, if allowed, and extensive note-taking should be done for each activity.
· Data Quality
The Consultant is responsible for collecting the highest quality data possible and subjecting interviewers, facilitators, and data to great scrutiny to ensure quality. It is recommended Consultant implement three kinds of data quality checks.
o Observation: Supervisors accompany the interviewers while they are doing the interview to observe and support interviewers. With this type of checking, supervisors will find out if the interviewer understands interview questions, effectively conveys them, and can coach the interviewer on how to improve if problems arise. Supervisors must accompany interviewers for at least 10% of interviews.
o Supervisors are required to revisit households and ask some of the (sample) questions in the questionnaires to the same respondents.. The purpose of this procedure is to verify that interviewer actually met the respondents and conducted the interview. If there are significantly different answers between the two interviews, supervisors should clarify the difference. If the difference is because the interviewer wrongly posed the question or misunderstood the question, supervisors should brief the team to clarify the misunderstanding. If any interviewer is found to be falsifying or manipulating data, he/she must be fired and replaced. Replacement interviewers must re-interview all respondents interviewed by the interviewers who falsified data. The Service Provider must ensure that at least 5% of interviews are audited (ie, households revisited).
o Checking the completed questionnaires: supervisors will check 20% of completed questionnaires to identify errors made by the interviewers.
· Methodological Report on field work implementation
The Consultant shall write a methodological report of no more than 5 pages (excluding annexes) detailing implementation of the fieldwork, any issues and problems that arose and how they were managed. The report should include:
Task 4 – Data cleaning, analysis, and draft report
Data processing team is responsible for starting the data cleaning process as soon as they receive data and other supporting documents from the field. This process includes checking the accuracy of data received, such as respondents’ identification and linkages among questions (meaning the skip patterns and relationships across questions). If data processing staff requires clarification from the field and the team is still in the field, they should clarify discrepancies with the field team.
Data processing team should also quality check the data captured qualitatively. The notes will be compared to the recording done during the activity to ensure that all important points are captured in the notes.
Data tables will be generated for quantitative data. Latent level of analysis will be done for qualitative data where relevant notes are tagged and categorized.
All answers or notes that are in the local language must be translated in English.
The Consultant is expected to first submit a draft report for comments and review of LWR. Clean raw data sets for quantitative data including frequencies and cross-tabulations must be submitted along with the draft in CSV format, and when available, Stata 11 format. FGD and KII notes along with a summary of key qualitative findings must be submitted in English and in Word format..
Task 5 – Final Report submission
The Consultant must submit the final report addressing all the comments made by LWR. A tracking table of comments made and responses must be included in the Annex..
Evaluation Deliverables
1) Evaluation plan (inception plan, inception report): The plan should be prepared by the evaluators before going into the full-fledged data collection exercise. It should detail the evaluators’ understanding of what is being evaluated and why, showing how each evaluation question will be answered by way of: proposed methods, proposed sources of data, sampling methodology, and data collection procedures.
2) Data collection preparation: The evaluator will recruit and train the enumerator. The evaluator also responsible in managing the permit for data collection
3) Draft evaluation report: The evaluator will submit a draft report by October 15. The LWR country office will review the draft evaluation report to ensure that it follows the terms of reference, addresses all of the evaluation questions, and follows any agreed upon format. The report should be submitted along with raw data collected during the evaluation process for triangulation if required.
4) Final report: within 07 days of receiving comments from LWR.
Evaluation Timeframe
The evaluation will be started on September 5 for 22 days work at the maximum, excluding travel time to project location.
Evaluator Qualifications And Application Procedures
The evaluation seeks for a person meeting the following requirements:
Qualification:
· Academic qualification in Social sciences
· Minimum 10 years of working experience in evaluating project and in the field of agriculture value chain;
· Experience in using CommCare as evaluation data collecting tools is preferred
· The evaluators’ independence from any organizations that have been involved in designing, executing or advising any aspect of the intervention that is the subject of the evaluation.
Skills:
· Writing and communication will be in English, and he/she must have excellent communication skills in English. He/she should be also able to communicate in Hindi while data collection in the project area;
Competencies:
· Demonstrates commitment to LWR’s mission, vision and values;
· Highly knowledgeable of participatory monitoring and evaluation processes and experienced in evaluation of projects on agriculture value chain (on coffee and cocoa would be an added advantage);
· Ability to deliver quality report within given time frame;
Application Instruction:
All interested candidates should submit an updated CV and Evaluation Proposal along with evaluation cost and terms of payment and three (3) professional references (not exceed 350kb) to Jobsindo@lwr.org. Please put “Evaluation”_<your name>” in the subject line and put your name in the file name on CV and Proposal (sample: CV_<your name>, Proposal_<your name>
Application deadline: 2 September 2019. Only shortlisted applicants will be invited to interview. No telephone inquiries please.
