Vol.4 No.1 2011

36/78

Research paper : Formation of research strategy and synthetic research evaluation based on the strategy (N. Kobayashi et al.)−33−Synthesiology - English edition Vol.4 No.1 (2011) Answer (Naoto Kobayashi)Basically, since the strategy formation and research evaluation are inextricably associated with each other, the evaluator should ideally be involved from the time of research strategy formation. However, the viewpoints of research strategy formation and evaluation are slightly different, and it is not desirable for the same entity to execute the P and C of the PDCA cycle. Therefore, I think only some members should overlap in the committees for research strategy formation and research evaluation.4 Feedback in evaluationComment (Motoyuki Akamatsu)In subchapter 4.2, you discuss the importance of feedback, and I think everyone will agree. However, if the feedback is provided offline in writing, I think you loose the opportunity for abduction based on the discussion between the research executant and the evaluator. If possible, please include a discussion on how this feedback should be done.Also, a program is expected to run for five to seven years, and I imagine that the feedback loop is in the order of five years. In general, I don’t think it is easy to reflect the result of a review from five years ago in the current program. That is because in a program of five to seven year scale, it is difficult to determine whether the objective was finally achieved immediately after the completion of the program, and the immediate feedback to the next step is difficult.Answer (Naoto Kobayashi)As you indicate, the evaluator and the research promoting group should not be antagonistic, but they should “walk side by side”. Since the act of research is a repetition of abduction and validation, a feedback without the discussion on abduction loses its meaning.For the feedback cycle, as you indicate, it is difficult to evaluate a program unless about three years have passed after completion. However, when the R&D, technological development, and system development are done in society, they inevitably become contiguous due to the social demand. Specifically, the FP7 framework in Europe is a seven-year (2007~2013) program. An intermediate evaluation was conducted in autumn 2010, and the appraisal of the FP8 (2013~2019), the next framework, will be started in 2011 based on this evaluation. In practice, the programs are sequential and the feedbacks are done in extremely short span of time.5 Evaluation of the research program and evaluation of the research projectComment (Kazunori Nakamura)In order to apply this research evaluation method, it is necessary to apply synthetic research evaluation in the process of strategy formation of the research program. Therefore, I don’t think it is readily applicable to a general research project that has not passed through this process.Answer (Naoto Kobayashi)A research project has, in a sense, a simple structure compared to a research program. There, the research objective, method, results, and expected outcome are contained in a small area, but has a fractal structure to the research program. For example, it is possible to apply the evaluation using the aforementioned three aspects (1) progress, (2) depth, and (3) phase, and it is also possible to apply the evaluation method that synthesizes the three. However, the execution of a research project is positioned as one element of the research program within the research strategy, and the appraisal of the research strategy can be simplified.6 Verification of the case of research evaluationComment (Motoyuki Akamatsu)Since this is a paper on research evaluation, it is desirable if you present an evaluation of an actual research. If there are reviews and discussions from the members of the evaluating committee, Evaluation Department, and the evaluated entity, the paper itself will become abductive and interesting.Answer (Kenta Ooi)In the Second Medium Term Target Period Research Unit Evaluation Report published in May 2010, the comments by the external evaluation committee members, the research unit heads, and the coordinators were analyzed, the characteristics and issues of the evaluation system in the Second Term were summarized, and the improvement points were organized. For the evaluation system, there were many who highly evaluated the current system, such as the “evaluation from the external evaluation committee members” and the “introduction of the viewpoint of outcome taking into account the exit to industry and society”. On the other hand, many issues that must be improved were pointed out, for example, the “need for flexibility in handling of various types of R&D such as the bottom-up research or the long-term research”, the “reduction of evaluation load”, and “further utilization of the evaluation results”. Being aware of these improvement issues, the evaluation from the viewpoint of outcome will be continued for the Third Term evaluation system, and efforts will be made to raise the immediacy of effect. It is not even 10 years since the research evaluation started at AIST. As you indicated, it is important to make improvements based on the verification of hypotheses, toward a better evaluation system.For specific examples at AIST in this paper, we discuss the issues of the evaluation system currently used at AIST from the perspective of an ideal synthetic evaluation based on the research strategy. To actually apply the synthetic evaluation system, I think it is necessary to design the system as a whole including the strategic research advancement, rather than cutting out the evaluation system only. I think it is necessary to conduct the modeling and the hypothesis verification for the AIST system in a larger framework.7 Logic model and synthetic research evaluationComment (Kazunori Nakamura)The example of Nagasaki prefecture is described as a case where the logic model was applied to advance the strategic R&D, and the Research Evaluation Subcommittee and the Research Enterprise Evaluation Committee were held utilizing the logic model. I think you should clarify the basic differences between the evaluation based on the logic model in this case and the synthetic research evaluation described in this paper.Answer (Osamu Nakamura)It was presented as one of the examples of “R&D evaluation that seemed to have incorporated the thinking of synthetic research evaluation by trial-and-error”, and so there is no basic difference.As mentioned in this paper, the mission of the Science and Technology Promotion Bureau of Nagasaki Prefectural Government is to contribute to create energetic Nagasaki prefecture where people can live with dreams for the future by utilizing science and technology. To evaluate whether the research institutes quickly recognize the needs of the local companies and product districts, and set the research topics that can achieve the demanded results, we asked to review all the running projects and summarize them into the logic model at first. That is because the application of the logic model is effective to clarify the strategic logic of the scenario.Each evaluation committee checked the positioning of each research topic based on the logic model, evaluated whether the

※このページを正しく表示するにはFlashPlayer9以上が必要です