ABSTRACT
Under the sponsorship of the National Science Foundation's Combined Research-Curriculum Development (CRCD) program a new curriculum in high performance computing is being introduced at Penn State. Also, a new graduate minor in high performance computing is being established. The minor will be available to students in all colleges including Engineering, Science, Earth and Mineral Sciences, as well as Arts and Architecture and Business. Several new courses have been developed at both the senior undergraduate and graduate levels. This curriculum development activity has also had an impact on other courses in engineering and science. In this paper we describe some of the innovative features of the new courses including the development of web-based materials, the use of high-technology classrooms, and the use of computational laboratory instruction in the coursework. Two different configurations of Unix-based classrooms have been set up and their system administration, hardware and software are described. Both faculty and research staff have been involved in the development of coursework and facilities. The importance and benefits of such collaborations are discussed.
INTRODUCTION
Product development and advances in basic science have become heavily dependent on simulations performed on high performance computers. Such simulations are at the leading edge of scientific computing and many problems remain beyond the capabilities of current computing capabilities. This need for high performance computers and communications has resulted in the High Performance Computing and Communications (HPCC) program: a multi-agency government effort to advance the scientific computing capabilities of the United States by a factor of 1000 by the end of the present decade. The achievement of "scaled teraflop performance" must rely on parallel computer architectures. The conversion of supercomputing from sequential/vector to parallel is crucial as that is where the United States still leads and it represents the future of high performance computing. However, this lead is being jeopardized by our failure to get parallel machines and parallel software technology into the hands of users. Thus, the education of our students in the capabilities and uses of advanced parallel computers is an essential component in the maintenance of the United States' competitive industrial edge as well as our continued leadership in fundamental scientific research. In recognition of this need, faculty in the Colleges of Engineering, Science, and Earth and Mineral Sciences combined in 1995 to propose to the National Science Foundation to develop a curriculum in high performance computing that emphasizes the application of advanced computing to the solution of problems of practical importance in science and engineering. This proposal was funded by the National Science Foundation under its Combined Research Curriculum Development Program. Some of the initial activities, stimulated by the CRCD award were described by Morris et al.1. These activities included the establishment of an interdisciplinary institute to promote the use of the fastest and most advanced computers in education and research. Called the "Institute for High Performance Computing Applications" (IHPCA), the organization was established in the College of Engineering in consultation and cooperation with the Colleges of Science and Earth and Mineral Sciences, the Center for Academic Computing and the Applied Research Laboratory at Penn State. The Institute has both a research and instructional mission. More information may be accessed the Institute home page on the World Wide Web at http://cac.psu.edu/~lnl/ihpca/. As part of its instructional mission, new courses have been developed and a new graduate minor in high performance computing is being instituted. The new course development was described in ref. 1. The evaluation of these courses is presently underway and the techniques used in this evaluation and the related results will be described in a subsequent report. In the next section some details are provided about the graduate minor. This is followed by a description of web-based materials developed for one of the new courses. The development of two new computer classrooms/laboratories is then discussed. A new seminar series, taught by faculty and staff is then described. Finally, some summary comments are made.
GRADUATE MINOR IN HIGH PERFORMANCE COMPUTING
The objectives of the new minor are to educate graduate students in scientific and high performance computing with an emphasis on the capabilities and uses of parallel computers. Three new courses have been introduced and these form the core of the graduate minor. They are listed as:
AERSP(Aerospace Engineering) 597: Numerical Methods on Parallel Computers
CSE (Computer Science & Engineering) 597: Parallel Architectures and Computation
NucE (Nuclear Engineering) 597: Scientific Algorithms for Parallel Computers
From the names of the offering departments, some of the multidisciplinery nature of the program is clear. In addition, courses in the Colleges of Science, Earth and Mineral Sciences and Engineering have been designated as "related courses." Students in any major may obtain the graduate minor. For masters candidates they must complete six credits from the core courses plus an additional three credits (for a total of nine credits). Doctoral candidates must also complete six credits of core courses with an additional nine credits from the related courses. The executive committee of the IHPCA makes decisions concerning which courses may be included in the related courses list. In addition, students are encouraged to register for the one credit seminar series offered every semester. The material covered in the seminars is described below.
There are several special features of the new graduate minor. It has involved the development of a new core of courses. It includes instruction by faculty and staff in several colleges. Also, it is available to students in all majors. Many of the courses involve laboratory activity and are making use of new Unix-based computer classrooms. These are described in more detail below. However, some of these features have resulted in a slow approval process that, the authors are sure, are not unique to their institution. A graduate minor must be approved at both the college and graduate school level. This approval involves committee deliberations. Knowing that several departments and colleges would need to support the minor, supporting letters were obtained from key departments in three colleges. However, at the first review level it became clear that many more departments were felt to have an interest and activity in high performance computing. So additional letters of approval from these departments were requested and obtained. In spite of this overwhelming support, and all review levels expressed their unqualified approval of the concept of a graduate minor in high performance computing, a key concern at the university level was the fact that the core courses had only been offered as "experimental courses." The executive committee of the IHPCA is presently working with the appropriate departments to develop a course offering schedule that will insure that the core courses are available on a regular basis and that they are approved quickly as permanent courses. In addition, it is planned to identify these core courses with an additional designation, such as AERSP/HPC, to denote their relationship to the minor. It is expected that the minor will be sent to the Board of Trustees for approval during this fall semester. The message here is that the traditional approval process does not lend itself well to non-traditional activities which, by their very nature, evolve very rapidly. However, in the long run, the authors believe that the addition of a graduate minor in high performance computing, available to students in any department, is an attractive way to expose students to and reward students for high performance computing activities.
WEB-BASED CLASS MATERIALS
With increased access for students to the World Wide Web (WWW) course materials may be made available via the Internet. There are several advantages to the preparation of class materials through this medium. Not of the least of these is the multiplicative nature of links within the main document. This means that the student has access to the equivalent of many text books, sets of class notes, and additional materials. These may include example problems, exercises, animations of solutions, technical references, and news items.
Two of the courses initiated under the CRCD program have made extensive use of web-based class materials. One is the seminar in computer hardware and software described below. The second is a course entitled "Numerical Methods on Parallel Computers," developed by Professor Lyle N. Long. The course materials are available at http://cac.psu.edu/~lnl/497/497.html. To provide some indication of the potential for this form of class material some samples from this site are included here. The class home page begins with a Table of Contents:
TABLE OF CONTENTS
Each of these underlined items contains a link to additional information. For example, the link to "Parallel Computing On-line Books" leads to
Parallel Computing On-line Books
This represents six text books and an important report on the future of scientific computing in the United States. Other books, reports and tutorials are contained within other parts of the compiled course materials. For example, a trip to the "Languages" link gives references for Fortran 90 (F90), High Performance Fortran (HPF), Message Passing Interface (MPI), and Java. The "Fortran 90" section contains the following links:
Where do these links take us? The "F90 Introduction" is a viewgraph presentation introducing the language. "F90 from ORNL CSEP" is an electronic book developed by the Computational Science Education Project at the Oak Ridge National Laboratory. "Metcalf's Tutorial" leads to tutorial exercises based on ref. 2. "F90 Code" leads to the Fortran Resources link from Lahey Computer Systems Inc. home page. It gives various samples of downloadable source code. Finally, "F90/HPF at MHPCC" leads to a HPF/F90 Parallel Programming Workshop produced by the Maui High Performance Computing Center.
A clear idea of the potential resources can only be obtained by a personal tour of the web site. However, these resources only serve to supplement the course content provided by the course instructor who provides lectures, discussions, exercises, tutorials and examinations. This frees the instructor from the personal development of educational materials that are clearly widely available. The primary investment of time is in the initial establishment of the class web site. This is now made easier through the use of applications, such as the editor in Netscape which makes a detailed knowledge of HTML unnecessary in HTML document preparation.
HIGH PERFORMANCE/TECHNICAL COMPUTING CLASSROOM
In a partnership with several departments from the Colleges of Engineering and Science and the Center for Academic computing at Penn State, undertook an initiative to plan, fund and implement two classrooms for the specific use of teaching high performance and/or technical computing. External support was obtained from IBM under their Selected University Research (SUR) program. The basic intent was to provide a computer to each student in each of two classrooms, one in the College of Engineering and one in the College of Science, providing full access to a rich suite of applications, compilers and tools. Additionally, each classroom was to have an identical system in a podium with screen projection capability. In this section, a description is given of the hardware and software used to make these two classrooms work.
Implementation Details
When the planning process for these laboratories started we became intrigued by not only the challenges associated with the rapid deployment of hardware and software before the start of classes in the fall of 1995, but by the opportunity to study the relative characteristics of two architectures for the two classrooms. Specifically, the classroom in the College of Engineering is implemented with workstations that are running a full instance of the operating system at each seat, and in the College of Science, the classroom has X-terminals at each seat with large Unix servers. Additionally, this implementation was seen as an opportunity to exploit and test some very new and exciting technologies, namely ATM for high-speed networking.
Hardware
As already mentioned, there are two different architectures implemented for these two classrooms. The one for the College of Engineering has a full instance of the operating system running at each student seat and in the podium. More specifically, we have RS/6000 model 40P's for each of 30 seats plus the podium. At the time of the deployment, the 40P was the most inexpensive model RS/6000 workstation available. Each system consists of a 601 PowerPC cpu, 32MB of memory, a 1GB disk, a 17in. 1280X1024 RGB monitor, keyboard and mouse, i.e. a full workstation running AIX 4.1 (IBM's version of System 5 Unix). Each system is networked via a 10baseT switch based ethernet. There are, generally, seven workstations per hub and each hub is connected to a switch port. This was felt to be adequate to minimize cost while minimizing the ethernet collision potential, given the fact that we were not generally using NFS or remote X-windows, both of which can be quite "chatty" and network taxing.
One of the special aspects of this implementation is the fact that the network that these systems live on is on one side of a router that has an OC3(155Mb/s) ATM as its primary interface to the outside world. This is, in turn, connected to an ATM switch that is an OC12 (622Mb/s) ATM connected across campus to the computer building. This makes for a very low latency, high bandwidth path to our file service and other general network services, including routing to Penn State's FDDI backbone. An additional detail relative to the hardware in the College of Engineering classroom is that the podium RS/6000 is connected to a ceiling-mounted high resolution projector. The intent here is to make it so that everyone in the classroom can see what the instructor sees.
In the College of Science classroom, there is an X-terminal at each of the 30 students seats and one in the podium. These X-terminals are networked via a 16Mb token ring to two RS/6000 model G40's each with 2 604 PowerPC cpu's, 128MB of RAM, and 4GB's of disk. One of the G40's has an OC3 ATM fiber connection which then acts as the router for the token ring network in the classroom. For projection of the instructor's screen to the classroom a 1024X768 color active matrix LCD panel with overhead projector is used. This is not a great solution, but the ceiling in the room that was allocated for this classroom wasn't tall enough to accommodate a ceiling mounted high resolution projection system properly, and money was not budgeted for major room renovations.
Software
Unix is a true multi-user, multitasking mainframe-class operating system that deserves our respect, or, at least, when it comes to its administration, it deserves our attention. Unix has the potential to support many users doing many different things concurrently. Thus, to provide maximum availability and usability, one must pay attention to the details. Additionally, the complexity of managing/administering this, and other operating systems, is compounded proportionally with the number of instances of the operating system that exist. Without going into all of the details, we needed to establish very precise, robust and automatic methods for administering, installing, managing and updating these systems with a rather small work force that had many other duties.
The core of our administrative model for running all these systems is a combination of AFS (with package), Kerberos, IBM's Network Installation Manager, Perl scripts, and brute force. The Unix administrators supporting the laboratories are an experienced group and know the much more traditional methods of Unix systems administration, namely the use of NFS and NIS for clustered workstation and user management. But, they are also well aware of the performance and security issues of NFS and NIS in a subnetted enterprise-wide environment. Also, they already had experience with the Andrew File System (AFS), first developed by Carnegie-Mellon University and later provided by the Transarc Corporation, and felt strongly that it would provide for the scalability and security that we needed.
Base system software installation was facilitated using IBM's Network Installation Manager (NIM). NIM is an object oriented, network based installation server and distributed system management system that is included with all AIX 4.x systems. Though we found the concepts a little difficult at first, we found that we really liked its power and flexibility. The basic approach here was to install and configure all the pieces of the system that we felt were needed, i.e. AIX, the compilers, tools and AFS, on one system. Then, to create a system image backup (mksysb) of that system over the network to our NIM server, and then to use NIM to install all the other systems. The full procedure for doing this is provided in the Network Installation Guide provided with the software from IBM.
User administration is accomplished via Perl scripts that are used by our accounts group. Specifically, they receive class account requests forms from instructors that stipulate the class number and section that requires the use of the classroom systems. A class list of students is generated from a course information database and, with this as an input file for the script, all the necessary configuration and administration steps are taken to make it so the students can log in and use the systems. AFS volumes are created for each user for their respective home directories and entries are added into master password and group files for later propagation to the individual systems. Kerberos entries are not a concern because everyone at Penn State, faculty and staff included, have entries added into the database once their affiliation is established.
Experiences
The different architectures of the two classrooms have taught us some things, though none of them were entirely unexpected. Consider the Engineering classroom, with a full workstation at each seat. In the plus column for this architecture is clearly the redundancy. This gives a lack of dependency on any one system or instance of the operating system. If a single system crashes, or losses its ethernet configuration, which is most often the case, most of the rest of the classroom is still perfectly usable. For the College of Science classroom, however, there is definitely and obviously a strong dependency on the two instances of the operating system. Lose one system, you lose half of your compute capability for the classroom. Worse yet, if you lose the one system that does the routing for the token ring network in that room, you lose the whole classroom. This exposure could have been obviated by putting an ATM switch in the room and having each system be ATM connected; but, this is expensive.
The advantage of the X-terminal/server architecture has been clear, and this appears to continue. The amount of attention that has to be paid to College of Science classroom as a whole is far less than to the College of Engineering classroom. The engineering classroom requires more monitoring and intervention on a weekly basis then does the science classroom. There are a lot of reasons for this, but basically there is a lot more hardware and instances of software involved that provide for a statistically higher chance of failure. Additionally, the installation and setup time of the College of Science classroom was significantly less. There are other details that can be pointed to that justifies and/or supports one architecture over the other but, we prefer the "mainframe" architecture, that is, the X-terminals with servers. X-terminals, and the more recent "Network Computers", are the modern day "dumb" terminals and the servers are not much more then small mainframes. For computer professionals, concerned with the support of tens of thousands of concurrent users and the many instances of different operating systems, we need to minimize variables to be successful. This architecture can provide for this while still providing a rich and modern end-user interface with access to the applications that people need and want.
Despite our preference to minimize parameters in our network, we still have to live in a very heterogeneous and distributed environment. Therefore, we plan to move to DCE from our current Kerberos and AFS deployment for the subject classrooms and most of the rest of the systems that we deal with. That is, NT, Unix, and even our mainframe MVS systems. DCE being the Distributed Computing Environment provided by several vendors has the promise of a more robust file system, better security, and more robust administration. Additional futures plans involve working more with NT, since more and more of the applications are on it first and/or exclusively. Also, we plan to deploy many more network computers with a few large servers to provide for the level of scale with manageability that we require.
Overall, the experience of the system administrators in running these systems has been a source of pride for their group. The fact that this implementation has been a positive experience may be credited to the technical capabilities and programming skills of the group. Given an average user count of about 900 people per semester and the surprising small number of problems, the implementation may be considered to have been a success.
SEMINAR SERIES
Penn State's graduate student population, like most large universities, comes from all fifty states and many other countries. It is often difficult for these students to learn the computer and networking skills required for them to complete their graduate degrees successfully. Even students whose research is not related directly to high performance computing, need significant training in order to perform tasks which are assumed essential today (email, disk storage, networking, software packages, graphics, etc.). This training often varies significantly from department to department, and college to college. And when the training is not adequate, it can put a strain on the central computing facilities staff. Training these students one-on-one is very time consuming and inefficient.
In order to ease the student's transition into computing at Penn State, our Center for Academic Computing (CAC) offers numerous seminars ( http://www.cac.psu.edu/computing/learn.html) throughout the year. These are very effective and well attended. In fact, enrollment often has to be limited and enrollment is on a first come, first serve basis. These seminars cover topics such as web publishing tools, Unix, visualization, programming, software packages, etc.
While the CAC seminar series is very valuable, the seminars cannot always be offered at convenient times in the graduate student's career. Also, the students do not receive course credit for these seminars. We have two approaches that are helping to minimize these problems. First, we now offer a one-credit graduate course every fall semester that is designed to teach incoming graduate students the key tools they will need to succeed. With one lecture per week, we cannot go into great depth, but usually just making them aware that the tools are there is enough to get them started. In addition, we have web pages ( http://www.psu.edu/ihpca/fall597e.html) for all the lectures that they can refer to later, as they need the material for their studies. These web pages are also heavily used by students that do not enroll in the seminar series.
There are several goals intended for this seminar series. First, it provides a primer for graduate students interested in the graduate minor in high performance computing. Second, it provides a collection point for the volume of material which had already been developed in support of high performance computing. Finally, it provides an opportunity for skilled research staff to bring their knowledge into the classroom.
The topics covered are as varied as the backgrounds of the speakers. The class meets once a week for fifteen weeks. Students registered in the seminar series earn one graduate credit. Papers describing two seminars in detail are required of the registered students, in addition to attendance. In fact, the average attendance at the seminars was over fifty each week.
The topics covered in the classes are listed below. Further details, including the lecture notes and additional links are posted at http://cac.psu.edu/~lnl/ihpca/fall597e.html
| 1. Introduction
History of high performance computing Current state of high performance computing |
9. Using the IBM PSU SP2
Architectural overview How to submit jobs, etc. |
| 2. Introduction to computing resources
and basic skills
Overview of resources for class (Computer labs, Web pages, etc.) Overview of editors, compilers, Unix utilities, HTML, Java |
10. Fortran 90
|
| 3. Scientific Publishing
TeX, LaTeX |
11. High Performance Fortran and
Fortran 95
Overview of HPF Overview of Fortran options at PSU |
| 4. Introduction to graphics
Overview of public domain graphics utilities |
12. Message Passing Interface, Part I
Virtual Workshop |
| 5. Graphics (continued)
Overview of locally popular vendor software |
13. Message Passing Interface, Part II
Complete Section One of the MPI Virtual Workshop, a elf-paced web-based workshop experience developed at Cornell Theory Center, delivered locally at PSU |
| 6. Graphics (continued)
Overview of locally popular vendor software |
14. Machine Room Tour |
| 7. Software Libraries
Review of Web collections IMSL, ESSL, Recipes, BLAS, LAPACK, etc. |
15. High Speed Networking
Mass Storage Technology PSU SP High Speed Networking/Technology Futures DCE Internet II VBNS |
| 8. Symbolic Math Packages
Mathematica Matlab |
The lecturers for the seminars were Dr. Lyle Long, Professor of Aerospace Engineering, and the
following members of the Advanced Information Technologies Group in the Center for
Academic Computing:
Jim Leous (Unix Systems Group),
Jeff Nucciarone (Numerically Intensive Computing Group),
Kevin Morooney (Numerically Intensive Computing Group),
John Kalbach (Advanced Projects Group),
Ray Masters (Visualization and Graphics Group),
Vijay Agarwala (Numerically Intensive Computing Group), and
Jeffery Almoney (Advanced Projects Group).
There are many benefits from this unique effort. Traditionally, many of these topics had only been offered as a series of computing seminars delivered out of the central computing facility. By institutionalizing the knowledge with course credit, attendance has been excellent. Also, students have been introduced to most of the skill sets they need to have a successful computational experience at any organization (national laboratory, private sector, university) and they are introduced to all of the people they would encounter locally as they investigate the lecture topics they more thoroughly. As the students progress through their curriculum elsewhere, faculty are able to focus their curriculum on the science or engineering topic, knowing that the students know how to get their work done computationally.
In addition to this fall semester offering, which emphasizes practical hardware and software issues related to high performance computing, a spring semester seminar series has been organized for two years. The material in this series emphasizes the application of high performance computing to the solution of practical problems in engineering, science, and the arts. The speakers and topics covered in spring 1997 are available at http://cac.psu.edu/~lnl/ihpca/spring597e97.html.
A second project underway in the training of students in the use of the computer facilities at Penn State is called our 'Video Server' project. We have a large 95-processor IBM SP2 at Penn State and, in a project with AT&T and IBM, we have dedicated one of these processors to act as our video-on-demand server. The ultimate goal here is to replace the standard audio visual department (with video tape libraries) by having all the video tapes stored on disk, with the ability of instructors to play them in classrooms.
DISCUSSION
The various projects and activities described in this paper are designed to make students more effective in the use of computer resources. Computer skills, from word processing to high performance parallel computing, are needed in various degrees by all students. This paper has talked about curriculum development, computer facilities, the use of the World Wide Web for class materials, and a high performance computing minor that spans all colleges. These activities are aimed specifically at high-end computer users. For these activities to be successful and to continue to expand it is necessary for faculty in different colleges and in different disciplines to work together with research support staff. Our experience at Penn State has been that such collaborations are productive, mutually beneficial, and collegial. Faculty, staff and students all benefit from these cooperative activities. They are well worth making the effort.
REFERENCES
1. P. J. Morris, L. N. Long, A. Haghighat and M. L. Brady, "Curriculum Development in Advanced Computation," ASEE Annual Conference, Washington, DC, June 23-26, 1996.
2. M. Metcalf and J. Reid, "Fortran 90/95 Explained," Oxford University Press, 1996.
Back to Table of
Contents