upThe first of Carnegie Mellon University's annual NSF Engineering Education Scholars Workshops was held July 21-26, 1996. The purpose of the week-long workshop was to provide new engineering faculty with information and skills that would enable them to more effectively assume the responsibilities of an academic career. The workshop included a variety of lectures, panel discussions and activity sessions, each addressing topics such as designing and teaching effective classes, advising graduate students, writing research proposals, and dealing with diversity issues. The first-year, short-term evaluation of this workshop included field observations, evaluation surveys, and a pretest/post-test evaluation of learning on intended educational objectives. The preliminary long-term evaluation involved tracking the on-line discussions of the network of new engineering educators, established during the workshop. The workshop was highly successful in meeting general goals, such as: providing participants with an introduction to basic pedagogy and research fund-raising skills; providing them with an exposure to current trends in and future directions for engineering education, and; fostering the development of a professional network among these new engineering faculty. The workshop was somewhat less successful, however, in achieving its specific educational objectives. Learning on these objectives might be improved by reducing the somewhat overwhelming pace of the workshop agenda, and by more consistently modeling the characteristics of good pedagogy espoused by the workshop presenters. Two significant achievements of the workshop were the extraordinarily high level of social bonding fostered by the experience, and an imparted feNor among participants for becoming good teachers, with a sense of empowerment for achieving that ambition.
Carnegie Mellon University (CMU)'s Engineering Education Scholars Workshop (EESW) is a National Science Foundation (NSF) funded program designed to teach basic pedagogy, mentoring skills, and research fund-raising skills, and to foster the development of a support network among a selected group of new and future engineering faculty. Four general goals were identified for the workshop [ I, p. I ]. These included:
In addition to these general goals, ten specific educational objectives were identified [1, p.2]. Participants were expected to be able to do the following by the end of the workshop:
One additional educational objective was added to this list by the program co-chairs, while they were preparing for the 1996 EESW. This was:
To address these goals, CMU's EESW program was designed as a one-week long, intensive program of 25 lectures, panel discussions, working sessions and social events. The agenda was quite full, and included sessions on each of the following topics, each directly related to one of the educational goals described above:
In addition to these, there was a keynote speech on, "Exercising Leadership in a Balanced Academic Career," and sessions on the use of case studies in teaching engineering, on balancing the professional responsibilities of faculty life, and on diversity issues. The EESW scholars also participated in a videotape review session, in which a ten-minute taped sample of their teaching was critiqued. In addition, they completed an engineering course design project, to practice applying information learned in several of the workshop sessions. Finally, there were a number of social activities, including a dinner reception, a trolley tour of Pittsburgh, a river-boat dinner cruise, and a formal breakfast on the last day of the workshop.
The workshop sessions took place primarily in three academic spaces on the Carnegie Mellon campus: a spacious, semi-circular, tiered lecture hall; a regular classroom furnished with movable chairs with desk-arms, and; a civil engineering laboratory with black lab-benches encircling columns, and an open area in one comer with long conference tables, folding chairs, and a small blackboard. Participants were housed in one of the university's more modem dormitories, a five-minute walk from the academic spaces where the sessions took place. The campus catering service provided meals, and morning and afternoon snacks. Participants were also given access to a semi-private recreation room in the dormitory.
Participants were provided with a number of resource materials during the workshop. These included a copy of, "The New Professor's Handbook" [2], a workshop booklet entitled, "Preparing Engineering Faculty of the Future," and a series of supplemental handouts which were passed out during the workshop sessions. They were also given a laptop computer, on loan from CMU for use during the week.
The workshop was attended by a group of 30 participants from across the United States, who were either intending to pursue, or were currently in their first year of an academic career in an engineering discipline. These participants were selected by a committee review of complete application packages which included a curriculum vita, a personal statement of interest, a letter of recommendation, and some demographic background information. The resulting participant group was quite diverse, including a variety of racial/ethnic backgrounds. The group included: 14 white women, 9 women of diverse color, 5 white men, and 2 men of diverse color. In addition to this, the group included a range of professional backgrounds. Just over half (19) of the participants were graduate students or post-doctoral fellows, and the rest ( 11) were first-year faculty (visiting faculty or tenure-track, assistant professors). Several participants had worked in industry before returning to academia to complete their Ph.D.s or to become faculty members. The one dimension along which the diversity of the group was biased was gender.
The evaluation design for CMU's EESW program involves an annual short-term, formative evaluation, and a long-term evaluation of the overall impact the workshop has on the academic career paths of the participants. The short-term evaluations include three major data sources, used to triangulate, and thereby clearly identify areas of success and areas of concern in the workshop experience. These data sources include field observations and informal interviews conducted by the evaluator during the workshop, a pretest and post-test of participants' learning, and participant and presenter evaluation surveys.
Field Observations & Informal Interviews
Throughout the workshop, I acted as a non-participant observer at all academic workshop sessions, all formally scheduled social activities, and at several informal gatherings. During this time, I followed a goal-free observation strategy, using the "full field note" method [3], which involves a process of taking extensive, handwritten field notes during the workshop events. To maintain neutrality, the field notes included not only my perceptions, but also records of the physical manifestations which lead to those perceptions. For example, rather than merely recording that participants seemed interested in a presentation, I would record that I noticed them sitting forward in their seats, taking lots of notes, or urgently waving their hands in the air to ask questions. In order to minimize any disturbance which might be caused by my role as observer, participants and presenters were informed that, rather than to evaluate them, my role was to gather information about the workshop experience and how, if at all, it changed the way in which participants thought about certain issues. They were encouraged to complete all evaluation forms, and to speak with me regarding any problems they had or any suggestions they would like to offer. Any such discussions were recorded in the field notes as "informal interviews." The field notes and informal interviews were transcribed and coded for content, according to established qualitative methods [4,5]. The patterns which emerged from this analysis are described anecdotally, with supporting excerpts from field notes, and from participant and presenter comments.
Pretest and Post-test Evaluations of Learning
A pretest for prior knowledge and post-test of learning were designed to determine the short-term educational impact of the workshop experience. At registration, pretests were handed to each participant, who was instructed to complete the pretest and hand it in the next morning. Post-tests were administered during the last session of the last day of the workshop. All thirty participants submitted both a completed pretest and a completed post-test. Each question designed to test a specific educational objective was also given to the relevant workshop session presenters, who were asked to provide a "key" solution, representing what they thought to be the most important ideas for the participants to glean from their session. Their responses were combined to produce a single "key" for the tests. Then, for each participant, each pretest/post-test question pair was assigned one of the following four codes:
+ if his/her thoughts on the question changed during the workshop, to MORE closely align with the main ideas the presenters listed on their "key".
O+ if there was no significant change in his/her thoughts on the question over the course of the workshop, and s/he had accurate prior knowledge on the topic.
O- if there was no significant change in his/her thoughts on the question over the course of the workshop, and his/her prior knowledge had little or no relationship to the material which was presented.
-- if his/her thoughts on the question changed during the workshop, to LESS closely align with the main ideas the presenters listed on their "key".
To provide a mechanism for comparing the relative effectiveness of the workshop in meeting the different educational objectives, an "index of learning" was then calculated for each question. This was done by assigning a value of +2 to each "+" code, +1 to each "O+" code, -1 to each "O-" code, and -2 to each "-" code, and then summing across participants.
Evaluation Surveys
Anonymous evaluation surveys were administered to presenters and participants, to assess their perceptions of the quality and utility of the sessions and of the overall workshop experience. These surveys included Likert-scales regarding the overall quality, and amount of time spent on each session, and open-ended questions regarding things like workshop administration and logistics, the utility of information presented, and general comments. The surveys were designed with careful attention to neutrality and clarity in wording, according to established guidelines [6]. Surveys were distributed to the presenters at the end of each session. They were asked to fill out the evaluation form before leaving, or to return it by mail before the end of the week. Approximately 60% of the presenters returned completed forms. Surveys were distributed to participants each evening, just after the last working session of the day. The forms were completed at that time, and returned before they went to dinner. Between 90% and 100% of the participants filled out their forms on any given day.
The remainder were returned blank. The quantitative data from these survey forms were entered into a spread-sheet, from which response frequency distributions were plotted. Answers to the open-ended questions and comments were analyzed for content in the same manner as the field notes.
Long Term Evaluation
The long term evaluation involves tracking the academic career paths of workshop participants, to assess the extent to which the workshop affects the ways in which, or the comfort with which, they approach the responsibilities of their new academic posts. Qualitative data for this is being gathered through annual surveys and interviews with workshop alumni. These data will be analyzed to identify patterns across participants in each cohort as well as over time for individuals. This paper, however, is limited in scope to a review of the highlights of the short-term evaluation of CMU's 1996 EESW program, and to a few, preliminary, long-term evaluation findings.
CMU's 1996 EESW program was highly successful in meeting the four general workshop goals, described in the introduction of this paper. The workshop was less successful in achieving the eleven specific educational objectives set for the workshop program. However, significant strides were made towards meeting those objectives. In addition to these intended outcomes, two significant achievements of this workshop were the high level of social bonding fostered by the experience, and the imparted fervor for becoming good teachers, combined with a sense of empowerment on the part of participants for achieving that ambition. Two general areas of concern also emerged from the evaluation data. The first of these involved the overwhelming pace of the workshop and a resulting information overload, which caused some amount of complaining about fatigue, and an observable decrease in attentiveness to learning activities over the course of the week. The second area of concern involved the lack of consistency with which workshop presenters modeled the precepts of good pedagogy, which they espoused and discussed throughout the week. Each of these general areas of success and concern if discussed briefly in the following sections.
Effectiveness in Meeting General Workshop Goals
The first of the four general workshop goals set forth in the proposal was to provide "professional development not traditionally offered in graduate programs, such as explicit training in understanding how students learn, planning courses, and using a variety of techniques and strategies to facilitate learning (including interactive lectures, problem-based learning, writing, and technology)." Considering that, traditionally, no formal training on pedagogy is offered to engineering graduate students, this workshop goal was very well met. Over the course of the week, ten full sessions were dedicated to introducing participants to fundamental concepts from general education theory and to specific teaching methods which could be used in engineering courses. In addition, participants were provided opportunities to discuss that information both in formal sessions and during informal interactions and, in a limited fashion, to practice applying that information to teaching-related tasks, assigned in sessions and as "homework." The actual amount of learning which took place as a result of this training is discussed further, below, in the section on the Effectiveness in Meeting Specified Educational Objectives.
The second general workshop goal was to present "information on the role of engineering faculty members in setting up and conducting research, focusing on issues that are not usually encountered as a graduate student such as securing funding, supervising graduate research, and balancing the various responsibilities of a faculty member." Five of the workshop sessions were dedicated to providing participants with an exposure to information on obtaining funding for research, on mentoring and supervising graduate students, and on balancing faculty responsibilities. No opportunity was provided for guided practice in any of these areas, and learning on these topics (discussed below) was not extensive. However, mastery of the skills associated with these topics was not a general goal of the workshop, and therefore, as far as providing exposure to these topics is concerned, the workshop did achieve its goal.
The third of the general goals for the EESW program was to provide "an opportunity to explore how engineering education in the 21st century can integrate new populations and technologies, sustain high ethical standards, and value teaching as a scholarly activity equal to discovery." Eight sessions addressed these issues in some capacity. Participants were not provided with opportunities to actively explore these issues, however, they were exposed to information on current trends in and future directions for engineering education, on diversity issues, on ethics in teaching engineering, and on the use of technology in the classroom.
The fourth general workshop goal was to provide "intellectual and social support with colleagues at a critical juncture in their careers with an eye towards future interaction as their careers develop." This particular goal was met to an extent beyond reasonable expectation. In addition to the intellectual and social support provided by the presentations and by the networking opportunities inherent in spending a week among like-minded professionals, this workshop fostered a level of social bonding, enthusiasm and empowerment among participants and between participants and workshop staff, which is out of the ordinary for adults at a professional conference.
On The Quality of Social Interactions
The social atmosphere of the workshop was friendly and energetic, and the attitudes of participants, coordinators and presenters, alike, seemed extremely positive, throughout the entire week. Evidence of this is found in survey responses, and in the field notes, which show many instances of animated conversation and social banter. This was true of the interactions among participants as well as between participants and workshop staff, and between participants and presenters. Even as fatigue began to take its toll on participants' focus on workshop learning activities, their enthusiasm for the workshop experience remained high. As one participant noted on her final evaluation survey form:
Conversations during breaks, meals, and social activities were animated, and ranged in content from discussions of theory presented in workshop sessions, to the sharing of personal experiences in teaching engineering (e.g. classroom management problems and tips for solving them), to arguments about social issues faced by new women engineering faculty (e.g. the liability of holding a technical Ph.D. when trying to get a date), to exhibitions of playfulness (e.g. mimicking and making fun of the Pittsburgh trolley tour-guide). Many participants mentioned that they found these informal conversations with each other to be very useful, both for information exchange and for developing personal and professional connections with like-minded people. In the words of two workshop participants:
The positive social atmosphere may have been due, in part, to having an unusually good group chemistry among the participants. However, it may also be attributed, in part, to specific efforts on the part of the workshop staff. For example, the co-chairs encouraged participants to dress comfortably and casually, and did so themselves. In addition, the workshop staff made themselves approachable and accessible to the participants (one of them stayed in the dormitory with the participants), joking and casually chatting with them between sessions. In doing this, the co-chairs came across as peer mentors and friends, rather than as aloof experts and administrators. The workshop co-chairs also acted as "master teachers" by modeling the behaviors presented in the workshop as being critical to good teaching and good mentoring. They exuded enthusiasm about the material being presented, and concern that both the logistical and educational needs of the scholars be met. They set a comfortable tone for the learning experience, laid out clear learning objectives, and made an effort to ensure that the workshop taught to those objectives. In doing this, they spread excitement among the participants about providing good educational experiences for engineering students. Evidence for this is found in the enthusiasm with which participants shared and requested additional "tips for good teaching," which they could immediately put to work in their classrooms. Participants also indicated this in their responses to the survey question, "What else did you learn during this workshop that will be useful to you in your career as an engineering educator?" For example, several participants noted that teaching can be fun and rewarding:
Nineteen participants said that they had gained a sense of empowerment to continuously improve their pedagogical skills. They learned, for example:
Eleven participants indicated a new or reinforced motivation to provide engineering students with the best possible educational experiences. They learned, for example:
In addition to this, twenty seven of the thirty participants noted that one of the most important things they gained from the workshop experience was their connection to a professional network of new engineering faculty. When asked, "What else of value can you take away from this experience?" they responded, for example:
The social atmosphere of the workshop also fostered an exceptional amount of interpersonal bonding among the participants. This is supported by the group's persistent chanting, in sing-song voices, of mantras they had formulated to capture the essence of their learning and of their shared experience. For example, "Coverage is the enemy!!" and, "This is going on the evaluation.!!" It is also supported by the impromptu performance of a song, "The NSF Engineering Education Scholars Workshop Blues," which was written and sung by one participant, accompanied by another participant (who was also a gifted pianist), for the group at the formal breakfast on the last day. This impromptu performance earned the performers peals of laughter and grins of enjoyment, followed by a standing ovation. Finally, departures were accompanied by lots of hugging, the flashing of cameras, ride-sharing to the airport, and hurried exchanges of telephone numbers and e-mail addresses. Clearly, the social atmosphere of the workshop was highly inspirational for the participants. As one individual put it,
Effectiveness in Meeting Specified Educational Objectives
In addition to meeting the four general workshop goals, the EESW was expected to help participants to achieve the eleven specific educational objectives described in the introduction of this paper. Learning on each of these objectives was assessed using a written pretest and post-test of participants' knowledge, and analyzed according to methods discussed in the evaluation methods section, above. The results of this analysis is summarized in Table 1, below, where the educational objectives are listed in rank-order (from most learning to least learning), according to the relative educational impact of the workshop experience in producing learning on each objective. In cases where indices of learning were equal, the educational outcome which earned the most "+" codes for learning was assigned the higher rank. These indices are on a scale of-60 to +60, in which -60 represents the maximum possible negative educational impact, 0 represents no educational impact, and +60 represents the maximum possible positive educational impact. It is important to note that these indices of learning do not provide a real "measure" of learning as much as they provide a mechanism for comparing the overall learning achieved on each objective. A theme emerged from an analysis of the educational processes used in the workshop sessions related to the least successfully met educational objectives, as compared to the educational processes used in the more successful sessions. This involved the modeling (or lack thereof) of"good pedagogy."
On Modeling the Characteristics of "Good Pedagogy"
The basic model of "good pedagogy" which was presented in several of the workshop sessions, and which is outlined in many introductory education textbooks [2,7], states that well-planned lessons should include certain key characteristics which enhance learning. These include: providing a clear statement of objectives for the lesson at the outset, and spending 10-15 minutes at the end of a lesson to recap main ideas and provide closure; presenting new information in a manner which explicitly roots out and addresses prior knowledge; providing an opportunity for guided practice in applying that new knowledge with immediate feedback from the teacher and; providing an opportunity for independent practice, to reinforce the new knowledge prior to lesson closure and assessment.
There is a fairly strong correlation between the ranks of the educational objectives in terms of educational impact, as seen in Table 1, and the extent to which the above-described characteristics of good pedagogy were included in the related workshop sessions. For example, the only session which incorporated all of these characteristics, in the proper order, was the only session relevant to the educational objective on cognitive processes in learning, which earned the highest scores on the pretest/post-test evaluation of learning. Excerpts from the field notes, illustrate this modeling of good pedagogy:
In contrast, the educational objectives which earned the lower scores for learning were related to workshop sessions which incorporated few, if any of those characteristics of good pedagogy to any great extent. An excerpt from the field notes for one of those sessions illustrates this point:
upOn Content Coverage and Pacing
Another precept of good teaching which was mentioned in three of the sessions, but not consistently followed over the course of the workshop, is the idea of teaching for mastery of basic concepts rather than teaching for coverage. This idea became a mantra of the workshop as participants would repeatedly quote, "Coverage is the enemy!" What this means in practice is that, if mastery and retention of new material is the desired outcome, sufficient time needs to be allowed during the learning process for students to process and "digest" new ideas, making appropriate links to existing schemas, so that they can later retrieve and use the new knowledge. Students also need to be alert and attentive in order for this to occur. This learning process requires more time than was available during the 1996 workshop.
The content presented in the workshop covered a broad range of important topics, and was accurate to the most current literature on teaching and learning, and on the key issues currently faced by engineering education. Sixteen of the thirty participants commented on their surveys that all of the presented information was interesting and potentially useful. They also thought that each of the topics could easily consume an entire week by themselves, if thoroughly discussed. For example:
The pace at which this information was presented, however, tended to be overwhelming for the participants. Every one of the thirty workshop participant mentioned this issue, at least once, in the comments section of their survey forms. For example:
The comment of this last participant hit the proverbial nail on the head. Too much information was presented in too short a time during many of the sessions, with too little time set aside for participants to process it, practice using it, and obtain feedback on their learning. This resulted in decreased short-term learning, as evidenced in the results of the pretest/post-test evaluation of learning. It may also have a negative impact on participants' long-term retention and use of the pedagogical concepts they did learn.
Although the long-term evaluation of the CMU EESW program is only beginning, there is some evidence, already, that at least some of the participants' learning on pedagogy has been retained, and is being implemented as they begin their faculty careers in engineering. There is also evidence that the professional support network, which the co-chairs had hoped to establish through the workshop, is alive and active. In part, these results are self-reported, as in a paper presented by Keri Hombuckle at the 1997 ICEE conference [8]. In addition, during the seven months immediately following the workshop, a total of thirty-three lengthy (on the order of one type-written page) email messages were posted, by fourteen different 1996 EESW participants and staff members, to a distribution list set up by the workshop administrators to facilitate continued interactions among participants. The topics of these posts included academic job-opening announcements, personal and professional news updates (relocations, babies born, grants won, etc.), humorous anecdotes, requests for advice on pedagogy, responses to those requests for advice, and information regarding resource materials useful to engineering faculty.
There are a number of ways in which the Carnegie Mellon University EESW program might be improved for future years. The most important two of these would be: ( 1 ) to reduce the amount of material covered, and the workshop pacing, to encourage better mastery and retention of presented information, and; (2) to more consistently model the precepts of good pedagogy throughout the workshop. However, the workshop program can be considered highly successful, on most counts, even in its first year. Evidence to support this was present throughout the field notes, which showed the program to be successful in meeting all four of the general workshop goals. Success was also evident in the pretest/post-test results, which indicated that at least some learning had been achieved on every one of the educational objectives of the workshop. Finally, and perhaps most telling, success was evident in the overwhelmingly positive response of the participants.
The 1996 EESW participants left Carnegie Mellon University with more knowledge and enthusiasm than they had arrived with, regarding their future careers in academe:
They gained a broader perspective on what it means to be an engineering faculty member:
They gained a sense of empowerment and community:
They gained an appreciation of professional development for engineering faculty, which goes beyond keeping up on the research in their respective fields:
And they left with gratitude for the workshop experience:
1. Ambrose, S.A. and Davidson, C.I. (1997). Preparing Engineering Faculty of the Future: The NSF Engineering Education Scholars Workshop. To be published in Proceedings, Frontiers in Education Conference, Pittsburgh, PA, November 5-8, 1997.
2. Davidson, C.I. and Ambrose, S.A. ( 1994). The New Professor' s Handbook: A Guide to Teaching and Research in Engineering and Science. Bolton, MA: Anker Publishing Co.
3. Olson,S. (1976). Ideas and Data: Process and Practice of Social Research. Homewood, IL: The Dorsey Press.
4. Schatzman, L. & Strauss, A. (1973). Field Research: Strategies for a Natural Sociology. Englewood Cliffs, NJ: Prentice-Hall, Inc.
5. Patton, M.Q. (1990). Qualitative Evaluation & Research Methods (2nd Ed.). Newbury Park, CA: Sage Publications.
6. Sudman, S. & Bradburn, N. (1982). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco, CA: Jossey-Bass, Inc.
7. Burden, P. and Byrd, D. (1994). Methods for Effective Teaching. MA: Allyn & Bacon Division of Simon & Schuster, Inc.
8. Hornbuckle, K.C.; van der Meulen, M.C.H.; Ambrose, S.A.; and Davidson, C.I. (1997). The National Science Foundation's Engineering Education Scholars Workshop: Strategies for the New Professor. In Proceedings, International Conference on Engineering Education: Progress through Partnerships, Chicago, IL. August 14-15, 1997.
Back to Table of
Contents