Assessment Grading Systems and Regulations in Learning Outcome Models

DODRIDGE, Melvyn

School Of Engineering, University Of Derby, Kedleston Road Derby, m.dodridge@derby.ac.uk

Abstract: Learning outcomes and their assessment has given rise to considerable debate as to whether outcomes should be graded or just pass/fail. Assessment models employing learning outcomes have been used for some time in the UK in further education programmes for example National Vocational Qualifications (NVQs) and General National Vocational Qualification (GNVQs), however, they have only recently been introduced into Higher Education study programmes for example Higher National Certificates and Diplomas (HNCs/HNDs). Learning outcome assessment models have not found favour in degree level programmes in the university sector but this situation is likely to change as a result of the publication by the Quality Assurance Agency (QAA) for Higher Education in October 1998 entitled ‘The Way Ahead’. The new framework they have proposed will now be developed and implemented over the next few years and one of the key four areas is the programme specification which will be outcome based.

Keywords: assessment, learning outcomes, grading, regulations

Traditional assessment methods and compensation

Traditional methods of assessment in modular based programmes normally involves the marking and subsequent grading of coursework and examinations from which the overall module grade is calculated in accordance with the weighting of each element of assessment and assessment regulations. Assessment models have employed learning objectives for many years but the introduction of learning outcome or competence based models to the lower levels of post school education nationally has now filtered into higher level programmes.

A number of universities and colleges offering degree and BTEC higher level programmes are now employing learning outcome models. In fact for new BTEC programmes this is now a mandatory requirement. In many cases however all that has happened is that learning objectives have been renamed learning outcomes and the true purpose of employing such a model has been avoided. There is in many cases a mismatch between assessment regulations and the principles which underlie learning outcomes.

Students are less clear on what they need to achieve in models employing learning objectives. There may be some overall indication of the pass mark for a module and the weighting of individual pieces of assessed work and to whether any compensation is allowed in the programme regulations. Assessment, in many cases, uses the traditional time constrained summative examination in which there is broad coverage of the module general objectives. The paper normally offers a choice of questions typically four from six in two hours or five from eight in three hours. Compensation is normal in that poorly answered questions can be compensated by ones which are well answered. The student passes the module if the pass mark, usually around 40% is met.

More recently there has been a move away from programmes where all assessment is by summative examinations to formative assessment, in many cases using wholly coursework, or mixed mode assessment in which there is a combination of coursework and summative examination. Compensation is normally applicable whereby in the former case, coursework marks are averaged to produce the module mark and in the latter case a minimum criteria is set for passing both the coursework and the examination but a good performance, normally in the coursework, compensates for in this case a poor examination performance. In many cases the weighting of the examination is equal to or greater than 50% of the module assessment. Compensation increases the module pass rate but can be justified on the grounds that examination performance is usually graded lower than coursework. Standards are safeguarded by setting a minimum criteria.

There are many possible reasons for the student performance in the examination being normally lower than that obtained in coursework in practice. For instance, examination technique, errors, misunderstandings, structure of questions, illness, revision too little and too late, nervous problems, etc.

The table in Figure 1 shows the required coursework performance with the minimum criteria set at 30% and a module pass mark of 40% for various coursework/ examination assessment weightings. As can be seen a good performance of 63% in coursework is required in the case of 70E/30C model.

E/C - Examination/Coursework Weighting

Fig. 1. Compensation between examination and coursework components

In a true learning outcomes model it is expected that all the module outcomes must be achieved before credit is awarded and it is therefore difficult to see how the traditional summative examinations are compatible with this type of model. A learning outcomes model therefore prevents students from performing badly in items of assessed work or missing out altogether sections of the curriculum content whereas in a learning objectives model employing compensation they may well have passed.

Traditional Grading Systems, Grade Reporting and Assessment Boards

There are numerous grading systems in use some using numerical grades and others using alphabetical grades [1], however where assessment board reports are generated by computer controlled record systems a numerical base is used for calculating module grades in respect of the weighting of each assessed element. Many universities allow compensation between the overall coursework grade and the end of module examination grade. Record systems can accommodate compensation but this can prove problematic when introducing the requirement to achieve learning outcomes.

In practice many of the present assessment regulations were formulated prior to the introduction of the learning outcomes assessment model and no substantial changes have been made since. There is the need for change as the present regulations are not compatible with the assessment of learning outcomes for a number of reasons. For example, the marks recorded on the Student Record System and reported on assessment board reports are traditional in nature and include an overall coursework mark, an examination mark and the overall module mark from which the module grade is obtained.

Traditional grading systems take no account of the achievement or failure of learning outcomes. Failure is normally restricted to flagging up whether the failure is in coursework, examination or both. It is normal for the marking tutor to hold information on the exact nature of any failure as in the main data bases used in assessment record systems are inadequate. The coursework element is likely to comprise of several learning outcomes and even the examination may include more than one. The result for each assessed element is not reported even if it can be recorded on the Student Record System where marks are inputted at assessment level, and therefore there is no mechanism to record success or failure at learning outcome level. A typical module assessment board report is shown in Fig. 2.

Fig. 2. Typical module assessment board report

The minimum pass mark in this grading system is 9.50 but compensation is allowed in coursework or examination down to 6.00, provided the overall mark is 9.50 or greater. Now, student A passes both elements and receives the grade C+ for the module on their transcript, however student B also passes with the lowest pass grade despite failing to meet the overall coursework requirements because compensation is valid. Lets suppose that the coursework is divided into three elements equally weighted and that student B fails to submit anything for the second of the three coursework assessments. As stated the student passes on the overall performance but in a learning outcomes model would have failed if a learning outcome(s) is attached to each assessment. In effect compensation can be applied twice in the traditional model. i.e between coursework elements and/or between the coursework and examination components.

Modular based programmes usually employ two assessment boards although these may be run concurrently. Module boards make decisions in respect of module grades and consider each module and the students taking the module. Decisions are normally pass with appropriate grade, referred in coursework, referred in examination, referred in both coursework and examination or outright fail. The board however should be making decisions on achievement or non-achievement of learning outcomes where learning outcome models are employed. The obsession with numerical marks often leads, in the case of poor performance, to the raising of the mark resulting in a pass grade being awarded for the module. Little is mentioned on the status of learning outcomes. This effective compensation is processed with complete disregard to learning outcome failure and to the students overall profile which is only seen at the programme board.

Programme boards make decisions in respect of progression and awards. Each students overall profile is considered. The Module board may have made decisions to condone a low mark which is effectively compensation, however further decisions on compensation can be made at the programme board. Decisions at the first board may be reversed, for example, a referral in one module out of the academic year’s work may be condoned as a pass in order to allow progression or facilitate an award. This in fact is another mechanism to allow compensation. The processes carried out at both of these two boards seem unnecessarily complex.

There are considerable differences in philosophy with respect to grading. Competence based programmes such as National Vocational Qualifications (NVQs) are in the main skills based and grade outcomes as pass or fail. With a skill there must be a threshold level which merits a pass and clearly there are many examples where any further grading would serve no real purpose. The National Open College Network who offer foundation programmes which can lead into degree programmes also grade in the same manner. In General National Vocational Qualifications (GNVQs) a pass grade is automatically awarded where there is evidence of all of the requirements being met. An overall merit or distinction however, can be awarded if a third or more of the evidence meets the grading criteria for merit or distinction.

Although university degree programmes and BTEC Higher National Certificates and Diplomas may have elements which are skills based they also include knowledge, understanding, intellectual skills and transferable skills. Given that degree programmes are awarded with a classification and that BTEC modules are graded pass, merit or distinction then there is every reason to consider a threshold level in order to pass a learning outcome and to reward performance in excess of this by an appropriate grade.

Three common approaches to learning have been identified in several studies. Biggs [2] and others classified students into three learning categories:

1. Surface Learners - These show little interest in ideas and deeper meanings and they prefer rote learning and highly structured forms of teaching and assessment.
2. Deep Learners - These are interested in the subject matter, they have less interest in facts and are liable to be problem solves. They prefer flexibility in teaching and assessment.
3. Achieving Learners - These are highly organised, have an interest in attaining high results and relate their learning to the importance of the materials in examinations. They adopt surface or deep learning to suit their desired aims.

Traditional examinations suit surface learners who have good memories and knowledge of a subject. Students in this category could also be expected to problem solve at a basic level. Coursework however suits deep learners particularly where assignments are open ended. Achieving learners would find a combination of examination and coursework more appropriate to obtaining the highest grade. Surface learners may well be satisfied with just a pass grade however deep and achieving learners would benefit from assessment opportunities which are graded.

Learning Outcomes Reporting

The present method of recording and reporting assessment data was quite suitable in the pre-learning outcome days for regulatory frameworks with or without mixed mode compensation. Assessment board decisions in respect of referral were only valid in the pre-learning outcomes model. When the learning outcomes model was introduced at the University of Derby the regulatory framework remained in force and only in light of operational experience were shortcomings noted. The main problem was with mixed mode compensation and under this arrangement it was possible to record a module pass grade in cases where there was a learning outcome(s) failure. The author of this paper provided a solution in which one of two numerical values could be recorded in the case of a learning outcome(s) failure. The values chosen were set just below the two threshold levels for a pass as they would be unlikely to occur in practice. The values of 5.99 (threshold level 6.00) or 9.49 (threshold level 9.50) were used to record a learning outcome failure in mixed mode and single mode assessment respectively. However, this is only used where a student would have otherwise received a pass mark. The recording of these values is the responsibility of the module tutor and if forgotten may result in a pass grade being awarded. The actual learning outcomes failed are not recorded in this case and neither in the case of where a referral grade is awarded in other circumstances.

Unlike the present regulatory system which treats overall coursework with the same status as the traditional end of module examination, a true learning outcomes model would attach the same status to each outcome, and subsequent recording and reporting should reflect this. The method of assessment which we put so much emphasis on in our board reports i.e. ‘coursework’ and ‘examination’ is irrelevant in a learning outcomes model. What is required is very simple and involves recording for each assessment, irrespective of the assessment mode, the grade where the learning outcome has been achieved and failure where it has not been achieved. Compensation cannot reasonably be considered at this level but may be appropriate at the stage or programme level. The requirements of a module assessment board report are shown in Figure 3. It is considered however that it would be more meaningful for all assessment data to be reported in the alphabetical scale as this is more meaningful to the assessor and external examiner.

Fig. 3. Learning Outcomes module assessment board report

If an assessment included more than one learning outcome and more than one is failed then more digits will be required e.g. learning outcome 2, 3 (LO 2,3). Where a learning outcome(s) is failed the module grade is not calculated, the result is simply fail (F). The requirement for a module board is therefore questioned since at this level i.e. module level the student either achieves or fails to achieve the learning outcomes. Modules are either pass with appropriate grade or fail. There is one situation where module board papers usefully present data in respect of each module so that exceptionally good or poor performance as opposed to a normal distribution can be easily spotted. However module data and statistics can be presented at the Programme Board. I therefore cannot see any reason why it would be more appropriate to ratify decisions applying to learning outcomes at a module board. In a learning outcome model, utilising module reports as shown, the information will be clearly reflected when the students profile is examined in the programme reports. The employment of one board in addition offers a significant cost advantage.

Is Compensation valid in a Learning Outcomes Model?

It has already been argued that compensation at module level is inappropriate. However, at Stage or Programme level, it may well be justified, when considering a student’s overall profile. EDEXCEL regulations state for BTEC programmes that a pass is awarded for the achievement of all outcomes against the specified assessment criteria. Outcomes typically number 2 to 4 with 3 to 5 assessment criteria for each outcome. A Higher National Diploma programme comprises 16 modules and therefore typically a total of 192 criteria have to be met during the duration of the programme. There are two issues, firstly, are there too many outcomes, and secondly is it necessary to achieve all of them? There is some confusion as to what the term ‘assessment criteria’ means, and there appears to be no common answer when examining a number of learning outcome models. Taking an example from the BTEC model with an outcome which has 4 criteria as follows:

Outcome: Investigate circuits with feedback

Criteria:

• Deduce an expression for the closed loop gain of a feedback system.
• Determine the conditions for stability and the effects on circuit performance when feedback is applied.
• Design a circuit employing negative feedback.
• Investigate the effects of applying feedback to single and multiple stage circuits.

Despite the outcome itself being very general, it should be of fundamental or of intrinsic importance to the module. ‘Design a circuit employing negative feedback’ is more specific outcome but covers most of the requirements in the topic of ‘negative feedback’. The corresponding assessment criteria would be the evidence to ensure the design has been accomplished. Totalling the outcomes on this basis now gives 48 - a more reasonable number. Regarding the second point of compensation, the need to achieve all 48 outcomes in the previous example is questioned. To quote from the American Association for Higher Education [3]: ”We should not articulate a vision for the ideal graduate as a mandate for every graduate of every engineering programme to demonstrate competence and proficiency in every attribute. Each goal is a yardstick against which to measure programme success, not an item on a checklist to inventory failure.” It would therefore be reasonable to say, compensate for one or two learning outcome failures on the basis of good performance in others. It is important in modular programmes to ensure that the specific identity of the programme is reflected in the award, and it would therefore seem sensible that all learning outcomes in core modules should be achieved and any failure confined to optional modules. Clearly it would not be possible to compensate in some cases and where these occur the failed learning outcome(s) needs to be recorded for each module concerned. This is much simpler and a more precise method than used at present, where in particular reporting referred in coursework is open to interpretation. As in the present regulations there would be the need for trailing the achievement of learning outcomes into the next semester or year and in order to maintain quality control the requirement to terminate study where a high number of learning outcomes have been failed.

Grading & Performance Indicators

The need for twelve pass grades and five fail grades in the University of Derby grading system [1] is questioned in a learning outcomes assessment model. There are only four honours degree classifications which is a good argument to reduce the number of grades to for example just A, B, C and D. One fail grade is also sufficient as the degree of under performance can be indicated by the assessor on the Student Assessment Record Form. Many competence based models use only pass and fail grades and this is clearly inadequate, for in particular, the ‘achieving’ student. EDEXCEL BTEC programmes use three grades which may well prove sufficient in any learning outcomes model. Grade descriptors give an indication of the level of performance and clearly the number would reduce in line with fewer grades. The present descriptors in the Derby grading system are confusing in any case since there are too many grades. The wording of descriptors are therefore quite similar in grades above and below any particular grade.

EDEXCEL BTEC programmes have recently introduced a learning outcomes model and have revised their descriptors for pass, merit and distinction performance. [4] However, within the last year they have withdrawn the pass descriptor. This would appear logical since pass performance is simply achieving the learning outcome. In the Derby grading system this would be equivalent to removing the D- descriptor. The latest version of the BTEC merit and distinction descriptors are stated as:-

Merit Grade Descriptor

In order to achieve a merit the student must:

• use of a range of methods and techniques to collect, analyse and process information/data
• apply and analyse detailed knowledge and skills, using relevant theories and techniques
• coherently present and communicate work using technical language accurately.

Distinction Grade Descriptor

In order to achieve a distinction the student must:

• check validity when collecting, analysing and processing complex information/data
• evaluate and synthesise relevant theories and techniques to generate and justify valid conclusions
• show an individual approach in presenting and communicating work coherently, using technical language fluently.

To quote from the EDEXCEL guidelines:The grade descriptors listed are for grading the total evidence produced for each unit (module). In order to do this, the descriptor should be contextualised and incorporated into the design of assessment activities. Some units may not give the opportunity for the use of all the features of the Merit or Distinction grade descriptors. Consequently, centres should select the appropriate features of the grade descriptors for those units. Students’ work should then be assessed using the assessment criteria and contextualised grade descriptors.

Some of the Pitfalls in Learning Outcome Models and Grade Recording

Experience has shown that there is a tendency for module designers to specify too many learning outcomes for example as many as eight in a module with around 100/120 hours of notional learning time. Students may not achieve all of the outcomes and this presents problems for assessment boards as to whether to offer a referral opportunity or to condone any failed outcomes. In many cases some of the learning outcomes may not be of intrinsic value to the module and three to four learning outcomes for the module size stated is now considered a manageable number for students to achieve thus avoiding potential over assessment. EDEXCEL BTEC programmes now include in their module specification learning outcomes as illustrated in their new guidelines [4]. It should be noted that 3 to 4 learning outcomes is the norm.

One question which always causes debate is whether learning outcomes should be tested more than once prior to formal referral. For example where a module is assessed by both coursework and summative examination, learning outcomes may be tested twice, once in coursework and again in the examination. This however can prove problematic as discussed in one of the following examples. Many learning outcome models can be devised but careful thought is required to avoid pitfalls. Problems can result for the tutor, the student and in the subsequent recording of grades. Some examples are shown in Fig. 4.

*Summative examination in example (b)

Fig. 4. Example Learning Outcome Assessment Models

In example (a) Learning Outcome(s) are tested twice (all coursework). Whilst there should be adequate opportunities for achieving learning outcomes, if more than one attempt is allowed prior to formal referral, this can lead to difficulties in recording success or failure. In this example each assessment tests two learning outcomes and each outcome is tested twice. There are two problems when it comes to recording grades. For example a student could pass in Assessment 1 say Learning Outcome 1 but fail Learning Outcome 2 which is retrieved in Assessment 2 and Learning Outcome 1 is then failed in Assessment 4. Firstly if Learning Outcome 1 is passed in Assessment 1 there is no need to repeat it in Assessment 4 and therefore it may not be possible to undertake Assessment 4 if Learning Outcome 4 only is required. The second point is that it is not possible to grade Assessment 1 or Assessment 4 and a learning outcome failure should be recorded even though one of the learning outcomes has been achieved in each. A pass grade cannot be recorded even if overall the work is considered to be in that category.

Example (b) is similar to example (a) except that it contains an end of module examination. Similar difficulties arise, for example, it can be argued that if the student has passed Assessments 1 to 3 there is no need to take the examination as the outcomes have already been achieved. On the other hand if an outcome needs to be achieved in the examination how can the tutor ensure there is an opportunity where there is a choice of questions?

Principles in the application of Learning Outcomes

The learning outcomes working group at the University of Derby debated the previously mentioned issues at length and formulated a number of principles in the application of learning outcomes which would avoid the pitfalls outlined. These are:

• Learning Outcomes are learning achievements which are fundamental to the overall aims of a module.
• Each designated learning outcome should be distinct from other learning outcomes and should not be compound in structure
• A pass grade is only awarded for an assessed piece of work if the learning outcome(s) have been satisfied.
• Each learning outcome should be formally assessed once prior to referral or retake,
• There should only be a small number of learning outcomes for each module (2 to 4 are suggested for a module with a notional learning time of 120 hrs)

Two examples are shown in Fig 5

Fig 5. Suitable Learning Outcome Assessment Models

Example (a) is very simple to operate but some tutors would prefer to have fewer but longer assessments as in example (b). Where more than one learning outcome is assigned to an assessment the record system must be capable of recording and reporting data on any failed outcomes.

Referral Opportunities

Assessment regulations whether traditional or where a learning outcome model is employed in the assessment process will have safeguards for underperformance usually in the form of referral or retake opportunities. However such regulations differ considerably and in some cases are influenced by external bodies such as accreditors. In some accredited programmes there is for example very little in the way of referral opportunity in the first stage, the degree classification being determined by the first time attempt. There has however been notable changes in recent years particularly in competence based programmes such as NVQ’s and GNVQ’s

Issues for consideration should include:

• Should we provide a much earlier formal opportunity for redeeming failure.
• Is there a case for informal referral?
• Should the module grade be capped at the lowest pass grade following success in referral
• How many referral opportunities should there be?
• Should retaking a module be replaced by multiple referral opportunities?

Assessment by portfolio has been traditionally used in art and design programmes but less so in engineering. It is increasingly being used at the lower levels e.g. NVQ’s and GNVQ’s where students keep their assessed work, in a binder for example, thus enabling tutors, assessors, verifiers, prospective employers etc see their achievements. Final grading is summative and when all the evidence meets the requirements of all modules. Formative feedback is given however on the students progress towards meeting the grading criteria throughout the programmes so they can concentrate on areas of their work which need improvement. The advantage is that as students gain experience and skills with progress it is likely that later assessment activities will provide more evidence towards the grading criteria than earlier work.

With traditional assessment methods referral work is normally different, particularly in examinations, to the original work set and therefore in a learning outcomes model it can be more difficult to set another piece of work addressing the same outcome. The question is raised as to the work set for a referral in cases where the existing work addressed several outcomes but where only say one outcome has been failed. Should the student retake the work in its entirety or should work only address the failed outcome? It may by quite difficult to set a separate piece of work addressing the one outcome but on the other hand if possible it can be used to advantage in reducing marking time.

The other issues mentioned are still being debated but in some cases may be difficult to resolve whilst there is a mismatch between current assessment regulations and the principles which underlie learning outcomes.

The Quality Assurance Agency

The Quality Assurance Agency (QAA) for Higher Education set out a new quality assurance framework [5] in October 1998 which will be developed and implemented in the near future.

To support the new approach, work will be completed on:-

• the Qualifications Framework
• Programme Specifications
• Subject Benchmark Standards
• Codes of Practice

Assessment of student learning is considered under the heading of Programme Specifications and the QAA have published a document entitled. ”Guidelines for Preparing Programme Specifications”. [6]

The QAA suggests the primary purpose of the Programme Specification to be:

• to provide clear and concise information for intending students employers etc. on the intended outcomes of programmes of study.
• to help teaching teams set out clearly and concisely.
• the learning outcomes of their programmes of study.
• the teaching and learning methods that enable learners to achieve these.
• the assessment process that demonstrates their achievement.

The QAA will be making and reporting judgements at the ”subject/programme” level. The review cycle will include ”the appropriateness of student assessment methods as instruments for measuring the intended outcomes”. Learning outcome models and assessment regulations will therefore be debated at length as we enter the next millennium in light of current experience

References

1. Dodridge, M. J., Assessment Grading Systems in Undergraduate Programmes of Study, 2^nd International Conference on Engineering Education (ICEE 98), August 1998.
2. Biggs, J.B., Approaches to the Enhancement of Tertiary Teaching, Higher research and Development, Vol 8, No 1, 1989.
3. Angelo, T. A., American Association for Higher Education Bulletin, November 1995, pp7-9.
4. EDEXCEL - BTEC: Guidelines for Higher National Qualifications in Engineering, Foundation, 1997.Quality Assurance Agency for Higher Education: Higher Quality - bulletin No.4, October 1998
5. Quality Assurance Agency for Higher Education: Guidelines for Preparing Programme Specifications, October 1998.