Whitcher.qxp 1/7/09 3:21 pm Page 69
Role of Simulators in Appraising Competence
Multiple-choice and open questions are a reasonable way to assess the
Figure 1: Rule and Test Relationships
lower levels of underpinning knowledge. Questions of the open type
are arguably better at demonstrating some of the middle levels of
Test: loading operation
learning such as compression, application and analysis. However, even
those who attain the highest level in Bloom’s taxonomy may not
necessarily be considered competent since this is within the cognitive
Test 1: HD compressor Test 2: De-ballasting Test 3 Test 4
domain and does not include any task completion.
Rule 1 Rule 3
Competence
Competence is a construct and as such is difficult to measure. As can
be seen from the definitions above, competence is more than just the
Rule 2 Rule 4
completion of a task. While it starts with knowledge, it incorporates
other things and other constructs such as mental models and Figure 2: An Estimate of the Probability of Pass or Fail
situational awareness. As with most constructs there is much debate
Fail patterns
about definitions and even more about ways to measure them.
Knowledge and understanding are not divorced from competence; Fail Uncertainty Pass
rather, they contribute to it and may be inferred from observable
behaviour. In the UK there has been a lot of development of
competence-based vocational training and qualifications in recent
Pass patterns
100
years. In these cases the general policy has been to measure
performance, ideally in the workplace, as a measure of competence. Severity 0~100%
0
On LNG and LPG vessels, measuring performance in the workplace
is not easy to achieve for a variety of reasons. Liquid cargo handling • inserts spool piece from inert gas (IG) line to liquid line;
simulators provide the opportunity for candidates to demonstrate • opens valve from IG line to liquid line;
their competence in conducting operations; however, measuring • opens tank-isolating valves;
their performance to determine whether they are competent is not • opens tank-filling valves; and
straightforward. There needs to be a clear and consistent standard • lines up at least three tanks.
to measure against and a mechanism to measure with. Without this,
an assessor who simply watches candidate performance is only The list of performance criteria is converted into a checklist that is
providing a subjective opinion, which will vary between assessors. ticked off by the assessors as they observe the candidate’s actions. This
Vocational qualifications comprise units that are made up of approach provides a clear and repeatable standard against which all
elements. Each element is then broken down into performance candidates are measured. By using this methodology, the assessment
criteria. Range statements may then be applied to individual becomes objective and standardised, since all assessors must follow the
elements, as well as performance criteria to clarify the conditions same standard.
under which they must be conducted.
The main limitation of this method of assessment is that it requires a
In order to produce the units, elements, performance criteria and range one-to-one assessment. It is not possible for a single assessor to
statements, a functional analysis of the activity is required. In the observe every action of more than one candidate. An automated
context of LNG and LPG ship operations, the Society of International assessment tool that could observe more than one candidate at a time
Gas Tanker and Terminal Operators (SIGTTO) LNG or LPG Suggested and record the completion of all of the performance criteria in the same
Competency Standards provide a good starting point. Although way as a human assessor but for multiple candidates is clearly desirable.
written as a competence-based standard, it does not provide the same
level of detail as a vocational qualification. Automated Assessment Methods
Automated assessments have been used in simulation for a number of
At Warsash Maritime Academy, assessment of LNG and floating years; however, the methodologies used to measure and, where
production, storage and offloading (FPSO) cargo control room applicable, determine competence are sometimes simplistic. The ideal
operators has been conducted successfully for a number of years. The automated assessment tool would be one with all of the knowledge
first stage is to develop a consistent standard to assess against using a and experience of a human assessor that can identify both correct and
functional analysis similar to the vocational standards. incorrect actions. It would not be subject to the distraction that human
assessors might experience, which could influence the judgement they
A unit might be primary inerting of cargo tanks, which could be broken derive from observed performance.
down into elements and performance criteria as follows:
An expert system is capable of identifying correct and incorrect actions
Unit 1: Primary inerting of cargo tanks in a similar way to a human assessor and, importantly, is able to
Element 1: Line up delivery of inert gas to tanks determine context in a similar way to a person, so it can determine
Performance criteria are: whether a particular action is correct or not in a given context. For
HYDROCARBON WORLD VOLUME 4 ISSUE 1
69
Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32  |  Page 33  |  Page 34  |  Page 35  |  Page 36  |  Page 37  |  Page 38  |  Page 39  |  Page 40  |  Page 41  |  Page 42  |  Page 43  |  Page 44  |  Page 45  |  Page 46  |  Page 47  |  Page 48  |  Page 49  |  Page 50  |  Page 51  |  Page 52  |  Page 53  |  Page 54  |  Page 55  |  Page 56  |  Page 57  |  Page 58  |  Page 59  |  Page 60  |  Page 61  |  Page 62  |  Page 63  |  Page 64  |  Page 65  |  Page 66  |  Page 67  |  Page 68  |  Page 69  |  Page 70  |  Page 71  |  Page 72  |  Page 73  |  Page 74  |  Page 75  |  Page 76  |  Page 77  |  Page 78  |  Page 79  |  Page 80  |  Page 81  |  Page 82  |  Page 83  |  Page 84  |  Page 85  |  Page 86  |  Page 87  |  Page 88  |  Page 89  |  Page 90  |  Page 91  |  Page 92  |  Page 93  |  Page 94  |  Page 95  |  Page 96  |  Page 97  |  Page 98  |  Page 99  |  Page 100  |  Page 101  |  Page 102  |  Page 103  |  Page 104  |  Page 105  |  Page 106  |  Page 107  |  Page 108