LOHMANN, Jack R.
Associate Dean for Academic Affairs, College of Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA 30332-0360, jack.lohmann@coe.gatech.edu
Abstract: Throughout the United States, state legislative bodies, university governing boards, and regional and professional accreditation organizations are increasingly requiring universities and colleges to demonstrate that they have appropriate self-regulating processes in place to achieve their stated missions and goals. Central to these processes is that they be based on outcomes assessment and be focused on continuous improvement. This paper presents an overview of Georgia Tech's experiences in designing and developing outcomes-based assessment programs for all its undergraduate and graduate programs with an emphasis on those experiences within the College of Engineering. It also presents the current efforts to implement the programs within a continuous improvement process that is intended to be sustainable and broadly responsive to the needs of multiple external constituents. It concludes with seven suggestions for those embarking on developing outcomes-based assessment programs.
Keywords: ABET, accreditation, assessment, EC 2000, outcomes
Universities and colleges throughout the United States are increasingly being required by external governing bodies and accreditation organizations to demonstrate that they have appropriate self-regulating processes in place to assure that they are achieving their stated missions and goals [6][7][8]. A central element required of these self-regulating processes is that they be based on outcomes assessment and be focused on continuous improvement [2][9]. The new Engineering Criteria 2000 (EC 2000) issued by the Accreditation Board for Engineering and Technology (ABET), the accrediting organization for U.S. engineering education and engineering technology programs, is an example. These criteria are dramatically altering the manner in which engineering colleges evaluate their curricula, interact with their major stakeholders, and seek accreditation [2][3][4]. These criteria, and similar expectations by other external constituents, are also placing additional demands on limited university resources and faculty time to respond to these new requirements [5]. At the Georgia Institute of Technology, all degree programs, both undergraduate and graduate degrees, now have outcomes-based assessment programs. They have been developed over the past several years either as the result of a mandate by the Board of Regents, the governing board of the University System of Georgia, or to meet the new professional accreditation standards, such as those by ABET, the Computing Sciences Accreditation Board (CSAB), the National Architectural Accrediting Board (NAAB), and the American Assembly of Collegiate Schools of Business (AACSB), or to fulfill the requirements for accreditation by the Southern Association for Colleges and Schools (SACS), the accrediting body for secondary and post-secondary institutions located in the southeastern region of the United States. Maintaining these potentially resource-intensive programs has become a considerable challenge. The challenge is to implement sustainable outcomes assessment programs that facilitate the Institute's efforts to improve its educational programs while also demonstrating to multiple external constituents that these programs are effective means of self-regulation.
This paper describes some of Georgia Tech's experiences in designing and developing outcomes-based assessment programs and how it is now implementing them in a process intended to be both sustainable and broadly responsive to the needs of multiple external constituents. An emphasis of the paper is on the experiences and activities as they relate to the College of Engineering.
The paper first presents a brief history of Georgia Tech's activities in the area of outcomes assessment. Because outcomes assessment has been driven by both internal initiatives and external requirements, the Institute now finds itself, somewhat predictably, at the juncture of consolidating and refining the results of these activities into an overall effort that is affordable and effective. Thus, the subsequent two sections summarize the current state of affairs. The first of these two sections presents what the College has concluded are the basic elements needed for an effective outcomes assessment plan. While there clearly are many elements which may be included in an outcomes assessment plan, the College has identified four elements which have been found to be most helpful for its internal needs and that are common to the needs of its principal external constituents. The process to implement these plans is then presented. The process is designed to provide adequate documentation to satisfy both internal needs for self-evaluation and external requirements for self-regulation without requiring a significant administrative effort. Finally, the paper concludes with seven suggestions for those embarking on developing outcomes-based assessment programs.
The creation of outcomes assessment programs at Georgia Tech evolved during most of the 1990s. The foundations for these programs were laid largely during the first half of the 1990s through a number of internally-initiated events focused on continuous quality improvement. The translation of some of these activities into formal outcomes-based assessment programs, however, occurred largely during the later half of the 1990s as a result of the need to meet the requirements of two re-accreditation reviews.
In 1992, three events laid the foundations for the first College of Engineering and Institute efforts in outcomes assessment. The first event was the development of a College of Engineering Strategic Plan in conjunction with the development of strategic plans for each School within the College. An important part of the strategic planning process was the identification of sets of programs at peer institutions from which the College and Schools would benchmark. The College identified initially twelve peer colleges, and each School identified its own set of peer schools or departments. These sets have been refined over time as a result of surveys of Georgia Tech undergraduate and graduate students, alumni, and employers of Georgia Tech graduates. Considerable information has been gathered regularly from these programs. The second event was an award from IBM whose purpose was to facilitate the implementation of Total Quality practices across Georgia Tech. While the grant affected the administrative processes of Georgia Tech more than it did the academic processes, it nonetheless helped move the Institute further down the path toward better continuous improvement processes. Finally, the third event was a mandate from the Board of Regents of the University System of Georgia that all undergraduate programs within the university system would develop and implement outcomes-based assessment programs over a multi-year period [8]. This mandate initiated Georgia Tech's first Institute-wide effort in outcomes assessment.
By the mid-1990s, several other activities continued to build off these early efforts and lend more focus to outcomes assessment. The College of Engineering formed an Ad Hoc Engineering Curriculum Assessment Committee comprised of representatives from several companies that were known for their excellence in continuous quality improvement. Also included on the committee were faculty, staff, and students from the Institute. The purpose of the committee was to learn and adapt from industry those continuous improvement practices that might be applicable to academe. Among the principal results from this activity was the identification of the College of Engineering's key stakeholders from which the College now seeks feedback on a more regular and systematic basis. A major update of the Institute Strategic Plan was performed in 1995, and in 1996, the Board of Regents mandated the conversion of the University System from the quarter system to a semester system by the fall of 1998 (although Georgia Tech was given an extension until the fall of 1999 because of the disruptions associated with its serving as the host of the 1996 Olympic Village). Needless-to-say, much of the previous strategic planning, benchmarking, and efforts in continuous improvement and outcomes assessment were helpful in planning and designing the new curricula.
During the late 1990s, several events coalesced these activities and resulted in formal, published outcomes assessment programs. Principal among them for the College of Engineering were two re-accreditation reviews. One review occurred in 1997 when Georgia Tech served as one of the first five institutions to pilot ABET's new accreditation criteria, EC 2000. Ten engineering baccalaureate programs and one masters program were reviewed [2][7]. The other review occurred in 1998 and involved eighty graduate programs across the Institute as part of a special re-accreditation visit by SACS [6]. Because outcomes assessment is now a part of all the Institute's undergraduate and graduate programs, the Institute recently hired a full-time Director of Assessment to provide guidance and counsel on improving the individual unit plans and to facilitate more cohesion among them across the Institute. Within the College of Engineering, this is aided by support from SUCCEED, an engineering education coalition sponsored by the National Science Foundation, which targets outcomes assessment as one of its four major areas of activity [10].
Georgia Tech's experience in designing and developing outcomes-based assessment programs was anything but straightforward. It required the involvement of a large number of faculty, and many, many meetings. However, a guiding philosophy throughout their development was to create programs that would work within the existing resource allocations, utilize well assessment activities that were already in place, provide a good baseline of initial results, and, of course, meet the external needs required, e.g., EC 2000, SACS. Although each effort attempted to build on the preceding activities and to capitalize on the growing list of "lessons learned", a considerable amount of time and resources were required for each effort because of the expanding list of internal and external constituents involved and an expanding list of needs and requirements [2][3].
The principal challenge now is to implement the programs so that they are both broadly responsive and sustainable; specifically, they need to provide useful information for multiple constituents and within a framework of limited resources and faculty time. Within the College of Engineering, the approach to meeting that challenge has been twofold. First, the College is refining its current assessment programs to focus on those elements which it has found to be most useful for its own purposes and also most common among its principal external requirements. Second, the College has begun to implement the programs in a process designed to emphasize the systematic documentation of important assessment results while also minimizing the effort needed to administer them. Sharpening the focus of the current assessment programs is described next, followed by description of the process in which the assessment programs are being administered.
While the outcomes assessment plans developed to date reflect Georgia Tech's evolving philosophy of the self-evaluation of its educational programs, their development was also guided considerably by the expectations of external constituents, especially re-accreditation guidelines. For example, the undergraduate assessment plans in architecture, computing, and engineering reflect the guidelines of their respective accreditation boards, whereas the undergraduate assessment plans in the sciences and liberal arts reflect more the guidelines from the Board of Regents. At the graduate level, all the Institute's plans mirror a response to the requirements set forth by SACS, but they were also influenced by their respective undergraduate plans that were developed earlier. As such, the plans presently cover a multitude of elements related to assessment. For example, among the elements to be found in many plans are: summaries of unit strategic plans with their associated mission and vision statements; overviews of the unit administrative structures, and assessment responsibilities and procedures; descriptions of the unit strengths, challenges, faculty and facilities; tables of data presenting institutional support and resources; discussions of the program purpose and educational objectives; lists of expected student outcomes, program outcomes, and assessment methods; and examples of use of results. While these elements satisfied the particular reporting requirements for re-accreditation, relatively few of these elements are essential to the on-going assessment of a program's performance.
In general, the College of Engineering has found that four elements seem to be the most useful for its internal needs and also common to the requirements of its principal external constituents (Board of Regents, ABET, and SACS). The four elements are: 1) a concise statement of the purpose of a degree program and its general educational objectives; 2) a list of the principal expected outcomes to be achieved by its graduates; 3) a list of the methods used to assess student achievement of the expected outcomes; and 4) a description of the process used to systematically document the use of assessment results.
In large measure, these elements address the four basic questions that most accreditation visitors have in mind when reviewing a degree program, and certainly what faculty should have in mind when evaluating their curricula, namely: 1) What kind of career/lifetime preparation does the degree program seek to provide?; 2) What kinds of abilities are students expected to exhibit after graduating from the program?; 3) How does the program assess achievement of these skills and abilities by its graduates?; and 4) How does the program systematically evaluate and act upon the assessment results it collects?
The College of Engineering has spent a considerable amount of effort developing and refining the first three elements [2]. The last element, however, is one that is receiving increased attention. While the college can readily provide examples of the use of assessment results, it has learned as a consequence of two re-accreditation reviews that demonstrating there is a systematic process in place to document and act upon assessment results is quite a different matter. Some observations about Georgia Tech's experiences in addressing each of these four elements are the following.
Developing a degree program's purpose and general educational objectives was relatively straightforward. Because the Institute, College, and Schools had already developed strategic plans, mission and vision statements existed at each level. Although the strategic plans were developed with a much broader perspective in mind than specific degree programs, they often contained either explicit or implicit statements relevant to the degree programs. Thus, in many cases, units were able to readily develop brief statements describing the purpose of their degree programs and their general educational objectives by identifying or enhancing those statements within the strategic plans that related to the degree programs. Ideally, of course, assessment planning and strategic planning should be mutually supportive, that is, assessment results should inform strategic planning, and strategic planning should inform what should be assessed. In the first pass at developing assessment plans, strategic planning informed assessment planning, but not vice versa. However, the Institute is now updating its strategic plan from 1995, including those at the college and school level, and it is explicitly incorporating into the process mechanisms to link strategic planning with outcomes assessment. This will be an important step toward assimilating outcomes assessment into on-going Institute planning processes.
The principal expected outcomes for graduates for each degree program were created pretty much from scratch. This element, unlike the other elements, was one in which the College (and Institute) had not addressed significantly in other related efforts, such as strategic planning, and it required considerable time and effort. To a large degree, developing written expected outcomes required the faculty to state, explicitly, what had been previously mostly their unstated expectations of their graduates. Faculty and students were involved in the process, as well as employers, others in industry, and alumni. The expected outcomes were, in large part, a natural extension of the program purpose and educational objectives. However, in some cases, during the process of developing the expected outcomes, it became clear that the program purpose and educational objectives needed some rethinking. As to be expected in attempting to do something this new, at all levels, and across the Institute, some expected outcomes were stated well and were quite specific, while others could still benefit from further refinement.
Developing a list of the methods used to assess student achievement of the expected outcomes depended mostly on identifying the feedback mechanisms that were currently in use. The Institute, College, and many Schools had long been using surveys and interviews of alumni, students, and employers, monitoring the results from national examinations and competitions, and seeking feedback from various faculty, student, and industry groups. Thus, developing a list was mostly a matter of recognizing that many of the feedback mechanisms already in use were, in fact, methods of assessment, even if they had not otherwise been thought of or described in those terms before. A major challenge, however, was attempting to map meaningfully how the methods employed were believed to assess the outcomes. More often than not, the methods employed were indirect measures at best, and they often were measures of multiple outcomes. As such, the College approach has been to view the results of these methods more collectively than individually. That is, while an individual method may have its limitations, collectively, all the methods provide a reasonably informative picture.
Finally, presenting documentation that assessment results are collected was fairly easy, e.g., benchmarking data, survey analyses, minutes of advisory board meetings; however, presenting documented evidence of the systematic use of assessment results was not an easy task. Rarely are important curriculum changes the result of a single survey, or one meeting of an advisory board, etc. They are typically the result of input from many sources, often over a period of time, and frequently involve many faculty meetings and committees. As such, assembling materials in a manner so as to document the connection between a given result and a specific curriculum change is very difficult. Historically, academia has not thought about documenting and archiving such materials in a manner suitable for accreditation or other external reporting requirements. Demonstrating that one has "closed the loop" between assessment results and curricular change is a worthy goal, but it certainly is not a straightforward process. As such, one area in which the College is devoting additional attention in its assessment plans is to include a section on the system of record keeping used to document the use of results. The intention is to bring more clarity as to the system of record keeping the units will employ to document regularly the use of assessment results.
Now that the Institute has met the challenge of creating outcomes assessment programs for all its degrees, it now must meet the challenge of sustaining them as part its on-going operations. Initially, this was felt not to be a particularly significant challenge because most of the assessment methods used already had on-going data collection cycles, i.e., some data was collected quarterly, some annually, some bi- or tri-annually. It was believed that evaluation of the data collected would naturally "drive" the process of assessment. For purposes of the College's self-evaluation, this was a reasonable belief and a reasonably effective process for quite some time. The College has long been collecting data, seeking feedback, and acting on the results to improve its curriculum. However, what the College has not been doing is documenting systematically how the data collected connects to program improvements. In essence, for purposes of demonstrating adequate self-regulation, external constituents required more archival evidence than the College had been accustomed to collecting. The challenge, then, has been to create a process that meets these external expectations, provides useful information for self-evaluation, and is not a significant time consuming task.
To this end, the Institute is now implementing a process that is comprised of two elements. The process requires that each degree program: 1) prepare a brief annual report summarizing its major assessment activities each year; and 2) conduct a review, and prepare a brief report, of its overall performance every five years.
The first round of annual reports is due in the fall of 1999. The purpose of the reports is to provide a mechanism to document, and archive, on a regular basis the use of assessment results. Each academic unit will prepare a summary report highlighting the assessment efforts of its degree programs for the past year. By keeping the reports brief, it is intended to keep them: focused on the major areas for improvement; suitably informative for dissemination across the Institute; and more likely to be acted upon. Over time these reports will form the basis of a system of record keeping that is intended to document the systematic use of assessment results.
The annual reports will highlight three areas: significant modifications to the program assessment plans, if any; noteworthy results of assessment activities and their dissemination (within and outside the degree program); and major actions proposed or taken as a consequence of the evaluation of the assessment results. An annual reporting cycle was selected primarily because much of what occurs in academia occurs around the academic year. Additionally, more frequent reporting was viewed as too burdensome, and less frequent reporting would not likely be viewed by external constituents as a sufficiently "on-going" and "continuous" process.
The first round of five-year program reviews for the Institute is scheduled for AY 1999-2000 (although the College of Engineering actually began its cycle in the spring of 1999). The purpose of the reviews is very similar to the annual reports except that they are intended to provide an opportunity for longitudinal evaluation of assessment results and to conduct a periodic evaluation of the effectiveness of the assessment process itself. Within the College of Engineering, these program reviews will become part of the School-wide reviews that are now conducted on a similar cycle. The School-wide reviews are conducted with the assistance of an external committee of visitors that interacts with faculty, students, staff, and administrators, mostly within the School, and provides the School, College, and Institute with their thoughts and comments on the School's overall directions, operations, and performance in all areas. Although evaluation of the School's degree programs has always been a part of the review, it has not previously been conducted from the perspective of outcomes assessment. Future School reviews will include within them a review of the performance of the School's degree programs, as has been done before, but it will also include an evaluation of the School's assessment plans and its implementation of them.
The program reviews will be phased in so that ultimately approximately one-fifth of the Institute degree programs are reviewed each year. All degree programs will have completed their first five-year review by 2003, in time for the next general ABET review for engineering that occurs that year as well as for the next general SACS review for all programs scheduled for 2004.
As mentioned before, as a result of either the policies of the Board of Regents, ABET criteria, or SACS criteria, all of the Institute's undergraduate programs and graduate degrees now have outcomes assessment programs. Much of what was learned in responding to the Board of Regents mandate in 1992 was used to prepare for EC 2000 in 1997. Similarly, a great deal of the lessons learned through EC 2000 were subsequently applied in preparation for the SACS re-accreditation in 1998. As a result of these experiences, seven suggestions are offered for those developing outcomes-based assessment programs.
One of the first questions faculty frequently ask, especially if the assessment program is being developed in response to external constituents, is: "What do they want?" "They" could be a governing board, accreditation body, or other external constituent. The focus should be on what the college wants and needs. Assuring that the assessment programs developed align with other external requirements should be a secondary issue, albeit an important one. Getting assistance and buy-in from faculty, administrators, and others, is easier if the effort is focused on providing useful information for the college. Fundamentally, most external constituents, such as ABET, require the development of educational objectives and program outcomes consistent with the mission of the college, an assessment process that demonstrates they are being achieved, and a system of continuous evaluation and improvement [1]. There is ample room within that structure to both fulfill institutional needs and meet external requirements.
Most, if not all, engineering colleges have been doing continuous improvement and outcomes assessment for a long time. They may not have thought of it in those terms, but thoughtful evaluation and diligent monitoring of the effectiveness of the curriculum have always been major activities of most any engineering college. Georgia Tech had been conducting alumni surveys, holding external advisory board meetings, benchmarking against peer institutions, etc, for quite some time. It may not have been as systematic as it could have been, and the quality of the effort in some areas may have needed some attention, but the quantity of effort was considerable. Indeed, there seems to be a predisposition among some faculty to assume that because external constituents are now proposing requirements that appear to be a dramatically new and different, such as EC 2000, then they must propose in response dramatically new and different approaches of evaluation. Such is not the case. Georgia Tech made considerable strides in developing its assessment programs by focusing on improving existing practices. It also helped in getting support from the faculty to develop more systematic assessment programs by working on improving existing activities that were already viewed as useful and helpful to their operations.
Raising the quality of assessment activities improves significantly when faculty share their efforts with other academic units. Further, for those units that have not yet given outcomes assessment much thought, collaboration greatly facilitates their movement up the learning curve by having the opportunity to benefit from the lessons learned from those units that have given assessment some thought. Capitalizing on each other's best ideas saves time and improves the results. For example, alumni surveys in the past had been done by individual schools and the results were seldom shared with other schools. Much of what constitutes an engineering education is often common among the different disciplines. By developing a College-wide survey, the breadth and quality of the alumni surveys was improved, the disciplinary depth was still maintained, the support of the alumni office was used more efficiently, and a more thorough analysis was performed involving a larger data set of responses.
Mission, vision, objectives, goals, strategies, tactics, outcomes, methods, processes, programs, etc, etc. The field of assessment involves a significant amount of terminology. It is helpful to reach an agreement early in the development process as to what is meant by the terms that are used. It is important not to get bogged down in debates such as whether "goals" are the means to achieve "objectives" or vice versa. It is more important to simply agree on what is meant by whatever terms are used. It is also helpful to agree early in the development process on what will be the key elements required of each assessment plan. A common framework of key elements helps to identify opportunities for college-level or university-level collaboration, assures that important components of outcomes-based assessment are addressed in every degree program, and reduces faculty frustrations in working in this relatively new arena by keeping the program development focused. Within the overall framework, individual units have ample opportunity to develop assessment plans that reflect their disciplinary and/or institutional uniqueness.
Without knowing one's academic competition, it is difficult to know if the college is improving, holding steady, or losing ground. External calibration with respect to institutions with which a college aspires to be more competitive can be very enlightening. Using national averages can be helpful, but it probably will not provide as sharp a picture. Similarly, regular feedback from the key stakeholders helps identify the most important opportunities for improvement. Obtaining data is a major part of assessment, and there is a lot of data that could be gathered. Having defined benchmark institutions and identified key stakeholders facilitates keeping the data gathering activities focused and meaningful.
Most any piece of data has its limitations. Indeed, some data may only remotely measure what the college seeks to evaluate. Many aspects of education are not easily measured. Neither of these issues, however, should become roadblocks to gathering data. Many academic processes involve measures that are limited in their scope or are attempting to evaluate very subjective issues. Nonetheless, evaluating lots of data, especially over time, and from a variety of sources, eventually provides an informative picture.
This is the one area that represents the greatest challenge for most academic institutions, and certainly for U.S. colleges of engineering implementing EC 2000. For administrative-type actions, this tends not to be a major issue because of the "paper trail" nature of most administrative processes. For curriculum-type actions, however, the tendency is to simply make the change and to not necessarily document the deliberations and inputs leading to the decision to make the change. As a consequence of the SACS re-accreditation visit, the Institute assessment plans are being modified to have a section focused on the record keeping system used to document the use of results, in addition to the records of evidence of the use of results.
Georgia Tech has learned many lessons in developing its outcomes assessment programs [2][3]. Its most recent lesson learned is the need to refine and consolidate its efforts to assure that outcomes assessment is sustainable within the Institute's limited resources while also fulfilling the Institute's internal and external needs. The challenge is considerable. In a recent report to SACS, Georgia Tech estimated, conservatively, that it will spend nearly $750,000 next year on outcomes assessment activities.
The success of outcomes assessment on most campuses will depend in part on the degree to which institutions are able to satisfy the needs of multiple constituents without the need for multiple assessment programs. For the most part, engineering education has long been practicing outcomes assessment. It may not have thought of it in those terms, and it may not have documented well its efforts, but self-evaluation and program improvement have been major activities in most colleges of engineering. What has changed is the need to share those efforts and results with a diverse array of external constituents. Within the College of Engineering at Georgia Tech, the principal strategy has been to develop outcomes assessment plans around those elements that are most useful internally and also common to external requirements, and to implement them in a sustainable process that provides adequate documentation.
Jack R. Lohmann is Associate Dean for Academic Affairs, College of Engineering, and Professor of Industrial and Systems Engineering, at the Georgia Institute of Technology. He earned his BSME from Oklahoma State University and his MS and PhD in Industrial Engineering from Stanford University. Dr. Lohmann has served as a program director at the National Science Foundation in Washington, D.C. and he has also held visiting positions at the University of Southern California and Ecole Centrale Paris. He has received numerous outstanding teaching awards and has been recognized for his professional and research accomplishments by the White House Office of Science and Technology Policy, Michigan Society of Professional Engineers, Society of Manufacturing Engineers, American Society for Engineering Education, and the National Science Foundation. He is a Fellow of the Institute of Industrial Engineers.