Assessment

NOTES TO THE INSTRUCTOR  

• This section is designed to be brief, but it is nonetheless important. The key point to be made is that workshop instructors, as well as faculty participants, should recognize the importance of assessing the impact of their efforts.
• The criteria for choosing assessment goals were developed by the participants in the original consensus conference that ultimately culminated in the creation of this workshop curriculum.
• To help support those interested in assessing impact, examples are provided of approach and content for assessment tools used in the creation of the curriculum.

---

Mentoring, as with other forms of teaching, is intended to produce a positive impact. However, that impact is not guaranteed. For this reason, effective teaching is defined in part by assessing whether goals have been met.

Criteria for choosing assessment goals

Choosing among the many possible outcomes and measures should begin with whether a particular outcome meets appropriate criteria, such as:

1. Important: The goal should address something that is particularly relevant (important) to the ethical or responsible conduct of science.
2. Deficient: Some things that are important may not in fact be lacking. The goal should address something that needs improvement or correction because it is deficient.
3. Independent: Even if something is important and deficient, it could be secondary to some other goal. Meeting the goal should be independent of first needing to meet other goals.
4. Amenable to Intervention: Even if something is important and deficient, we may have no realistic way to repair that deficit. The goal should be something for which we have, or we could reasonably produce or acquire, an intervention that would enable us to make a change.
5. Measurable: It is possible that there is something that we can change by intervention that is both important and deficient, but we have no means to assess our impact. The goal should be something for which we have the tools for defining measurable outcomes. [NOTE: Measurable outcomes can also include qualitative findings. The key is to have something credible to convince ourselves and others that there is some value added because of our efforts.]
6. Magnitude: It is possible that there is something that we can change by intervention that is important, deficient, and measurable, but the magnitude of our impact might be too small to be considered cost effective. The goal should be something for which we can produce a change of sufficiently large magnitude.
7. Feasible: Even if something reasonably meets all of the above criteria, it may not in fact be practical or feasible in the research environment because of the amount, type and availability of resources required or because of the characteristics of the research environment. The goal should be something that is feasible.

Assessment Plan for this Curriculum

One example of an assessment strategy is what was done for this workshop during its development. The items below could readily be adopted or modified for assessing future iterations of this workshop curriculum and/or the impact of faculty adoption of one or more of the approaches proposed in the workshop. If workshop instructors or faculty participants are interested in using either approach, contact Michael Kalichman or Dena Plemmons for access to the surveys used on SurveyMonkey.

Faculty Feedback

Prior to the workshop and six months after the workshop, faculty could be asked to complete a brief (2-3 minutes) online survey. Although names and e-mail addresses would be used to invite their participation in the survey, identifying information can be de-coupled from the data and not be part of any analysis, summary, or publication.

In addition to feedback on which of the proposed approaches were attempted, two primary questions to be answered are:

1. Do you perceive that the proposed approaches are feasible, relevant, and effective?
2. Do you have observations or experiences consistent with the presumption of a positive impact?

Student Feedback

Prior to the workshop and six months after the workshop trainees could be asked to complete a brief (2-3 minutes) online survey. Although trainee names and e-mail addresses would be used to invite participation in the survey, their identifying information can be de-coupled from the data and not be part of any analysis, summary, or publication.

In addition to feedback on which of the proposed approaches were attempted, the two primary questions to be answered are:

1. Do the students perceive that the proposed approaches are relevant and effective?
2. Do the students report outcomes consistent with the presumption of a positive impact?

The content of the surveys used is summarized on the following two pages.

---

Faculty Feedback Questions

1. During the most recent academic term, which of the following strategies did you use as a basis for discussion with one or more of your trainees (graduate students and/or post-docs)?

Strategy	Yes/No
Code of ethics or conduct for your research profession
Items on a checklist of research ethics topics
A real or fictional case to demonstrate research ethics issues
An Individual Development Plan establishing responsibilities for you and your students
A group policy addressing research ethics issues

2. For each of the above strategies that you used:

a. Did you use this strategy in the context of a group meeting (e.g., journal club, discussions of data or research strategies) and/or one-on-one?

 Using a scale of agree/neutral/disagree, please rate the following statements:
 In my particular research group, this strategy for teaching research ethics is

b. Feasible (it can be done)
c. Relevant (it is meaningful to our practice of research)
d. Effective (it helps to promote research integrity)

A. How many trainees are part of your research group?

               Graduate students _____   Post-docs ________

B. Over the most recent academic term, how many hours did you discuss research ethics issues with one or more of your trainees (graduate students and/or post-docs)?

In the context of:	Hours
One or more of the proposed strategies?
Other conversations?

 

3. Please note any observations you’ve had that speak for or against the effectiveness for your research group of any of the above strategies you have used.

4. Please share with us any other strategies, whether purposeful or ad hoc, you have successfully used to generate discussions about research ethics in your research group.

5. Please provide any other comments you may have.

---

Student Feedback Questions

1. During the most recent academic term, which of the following strategies did your research mentor use as a basis for discussion with you?

Strategy	Yes/No
Code of ethics or conduct for your research profession
Items on a checklist of research ethics topics
A real or fictional case to demonstrate research ethics issues
An Individual Development Plan establishing responsibilities for your research mentor and you
A group policy addressing research ethics issues

2. For each of the above strategies that your research mentor used:

A. Did your mentor use this strategy in the context of a group meeting (e.g., journal club, discussions of data or research strategies) and/or one-on-one?

Using a scale of agree/neutral/disagree, please rate the following statements:

In my particular research group, this strategy for teaching research ethics is

B. Relevant (it is meaningful to our practice of research)
C. Effective (it helps to promote research integrity)

3. Over the most recent academic term, how many hours did you discuss research ethics issues:

With:	Hours
Your research mentor?
Others?

4. If the number of hours in question 3 was >0, then what impact, if any, did those conversations have on you?

5. Could you briefly describe any other approaches your mentor has used to generate discussions about research ethics in your research group?

6. Please provide any additional comments you may have.