Adobe Systems Policy evaluations – funds and programs Filip Hrůza Adobe Systems 2 Aim This lecture aims to provide an overview and introduction of both general type of evaluation methods and tools with focus on their use in pracice: micro- and macro-evaluations. Meanwhile the qualitative microevaluations can provide decision-makers with very deep insight in functioning of evaluated policy/program/intervention, quantitative macroevaluations can contribute to policy evaluation with top-down analytical view of how chosen policy setting influences greater area such as region, country or etc. Adobe Systems 3 Source: http://www.equitesante.org/ Adobe Systems 4 Introduction •every public policy => political assignment and ideas (political parties) => goals, tools and funds • •how to find out whether PP goals are fullfiled by chosen tools? => needs to have a set of relevant tools to evalute it all properly • •major development in 20th century • •generally there can be identified two general approaches - microeconomic (bottom-up approach) and macroeconomic (top-down approach) • •recent development is significantly powered by growing role and use of ICT and internet, open and big data and participation tools • •according to growing complexity it is becoming highly desirable to use different tools to evaluate policies because of its complementarity (different views => BIG PICTURE) • Adobe Systems 5 Source: https://www.cartoonstock.com Adobe Systems 6 Source: Government of Canada: Evaluation of the 2009 Policy on Evaluation Adobe Systems 7 Introduction Hallet and Untiedt (2001) present three main reasons why public spending should be monitored and evaluated: 1)To evaluate the effectiveness or efficiency of expenditures associated with certain policy in order to achieve certain objectives and the possibility of comparison with alternative policies; 2)To provide a guide to decision-makers on how to achieve the intended goals by using the most efficient way of spending public funds; 3)3) To ensure liability to taxpayers through a view to the most efficient use of public funds. Adobe Systems 8 Source: https://texaspolitics.utexas.edu/ Adobe Systems 9 Introduction Lopez-Rodriguez and Faina (2014) and the European Commission outline two main approaches of impacts of EU cohesion policy assesments: 1) 1)Theory-based evaluations (literature reviews, text analysis, interviews and questionnaires or case stuies etc.); 2) 2)Counterfactual analysis (econometrics, macroeconomic models or input-output models etc.). The complementarity of both (qualitative and quantitave) approaches lies in the interconnection of the knowledge acquired by the individual methods. Adobe Systems 10 Introduction Other policy evaluation typology: 1) Ex-ante evaluations (planning and forecasting) 1) 2) Process evaluations (implementation) 1) 3) Ex-post evaluations (assessment and accountability) Source: Venable et al. (2012) Adobe Systems 11 Introduction Causal versus non-causal models 1) Causal models •Measuring the impacts of various factors (independent variables) on results of certain programs or projects (dependent variables) •E.g. Simulation models, input-output models, microeconomic models or statistic models •Used to assess of impacts of different projects/programs/politics, when you are able to prove the relationship between dependent and independent variables •Presence of risks => potential risk of misleading results •Example: Development of infrastructure (e.g. Highway network) around cities and its impact on influenced cities => economic development of influenced cities (in comparison to others). Is the economic growth the consequence of development of such infrastructure or the growth would be reached even without this infrustructure because these cities have bigger potential for economic growth than other cities. The solution could be reached by using further suiable detailed method (case study, interviews, document analysis etc.) 2) Non-causal models •E.g. Observing of participants, interviews, case study, etnografic methods, preference revealing etc. •Political dimension problem within taking the political decisions when deciding about the projects/programs/politics in public sector •Primary vs secondary data 1) Adobe Systems 12 Policy evaluation: How it should be? Adobe Systems 13 Policy evaluation: How it usually is? ? Adobe Systems 14 Best practice: Denmark Adobe Systems Micro-evaluations •Case study • •Cost-benefit analysis • •Expert panels • •Focus groups • •Interviews • •Multi-criteria analysis • •SWOT analysis • Adobe Systems Micro-evaluations SWOT analysis •Analysis of strong (S) and weak (W) sides, opportunities (O) and threats (T) •Aim: get the subjet/service/product from A => B (positioning) •Originally from private sector within the strategic planning Framework and then followed by use in public sector for evaluation of public projects or development… •Paradigms: general vs concrete, shallow vs deep (detailed), subjective vs objective, pragmatic empiricism • •Disadvantages (limits): •poor performance / method usage (speed of execution, depth of execution, poor interpretation (see causal analysis), •replacing a location in a matrix), •creating an analysis for already decided goals (bias), •brainstorming limits, •organization goals vs. community goals, •the width of the working team Adobe Systems Micro-evaluations Focus groups •length: app. 60-90 min. •a narrow group of interested people (about 7 people) •together talks about a pre-selected theme •the advantage it is not time-consuming •the positive of method is the development of the debate and the raise of new opinions / views Round table •length: 90 minutes + •a large to potentially representative group •more issues and themes •advantage is the interconnection and confrontation of the city / city management with the citizens •higher objectivity => objectification of the findings Adobe Systems Micro-evaluations Case study •for a more detailed description and analysis of the project •data collection both qualitative and quantitative •critical analysis, implementation analysis, or project impact analysis •structure and form Interview •we need to find out a specific type of information or get a more comprehensive view •types of interviews: free, semi-structured and structured •appropriate sample selection + interviewer experience Survey •generalize to the segment of the population they represent (a representative sample of respondents). •rules for creating a questionnaire •closed questions vs. open vs. semi-open questions (other answer) •test the questionnaire •encoding data for further processing Adobe Systems 19 Source: European Commission (2016) Source: Office of the Government of the Czech Republic (2018) QUEST III R&D Adobe Systems 20 Source: European Commission (2016) RHOMOLO Adobe Systems 21 Examples and case studies Government of Canada: Evaluation of the 2009 Policy on Evaluation https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/centre-excellence-eva luation/evaluation-2009-policy-evaluation.html EVALSED - The resource for the evaluation of Socio-Economic Development : Sourcebook - Method and techniques http://ec.europa.eu/regional_policy/en/information/publications/evaluations-guidance-documents/2013 /evalsed-the-resource-for-the-evaluation-of-socio-economic-development-sourcebook-method-and-techni ques Adobe Systems 22 References Hallet, M. a Untiedt, G. (2001) The potential and limitations of macroeconomic modelling for the evaluation of EU Structural Funds illustrated by the HERMIN model for East Germany. Informationen zur Raumentwicklung. Heft 6/7.2001, s. 451-463. López-Rodríguez, J. a Faína, A. (2014) Rhomolo and other methodologies to assess The European Cohesion Policy. Investigaciones Regionales, 29, s. 5-13. Europena Commission (2014) Sixth report on economic, social and territorial cohesion. [online]. Available at: Venable, J.R., Pries-Heje, J. and Baskerville, R. (2012) A Comprehensive Framework for Evaluation in Design Science Research. Proceedings of the 7th international conference on Design Science Research in Information Systems: advances in theory and practice Office of the Government of the Czech Republic (2018) Dopad ESI fondů na HDP ČR: simulace modelů QUEST III a RHOMOLO. Available at: