“Implementing
EBT”
Author: Norman
MacLeod
2023-05-23
Table of
Contents:
2 Introduction Pilot Training and Pilot Learning
2.1 Piloting as Goal-directed Action
2.2 Pilot ‘Knowledge’ is Retrospective
2.4 Error as Learning Feedback
3 The Development of Training in Aviation
3.1 How Humans (and Animals) Learn
3.6 Competencies in Civil Aviation
4 Implementing Instructional Systems Design
4.2 Developing the Job Task Analysis
4.3 Normal v Non-normal/Emergency
4.4 The Training Needs Analysis (TNA)
4.5 Describing the Output Standard
5 Developing Competence Frameworks and Markers.
5.2 Developing a Competence Model
5.6 Validating a Marker Framework.
6 Some Thoughts on the idea of ‘Knowledge’ as Competence
7.2 Testing of Declarative Knowledge
7.4 Managing the Output from Tests.
8.3.2 Designing Event Sets to Create Surprise.
8.3.4 Training for Uncertainty
9.2 Reasons for Grading Performance
9.4 Constructing a Grade Scale
10.3 Observation of Performance
10.4 Assigning a Score to a Performance – Sources of
Assessor Unreliability in Evaluation
11 Instructor and Assessor Training, Qualification and
Standardisation
11.2 The Training of Instructors
11.4 The Importance of Debriefing
11.5 Classical Debriefing Structures
11.6 ‘Safety II’ meets Elite Team Sports
11.7 Diagnosis, Debriefing and Facilitation
11.8 Instructor Concordance Assurance
11.9 Calibrating the Grading System (AMC1/GM2
ORO.FC.231(d)(2))
12 System Safety and Evaluation
12.2 An Overview of Training Evaluation
12.3 Data Gathering and the SC
12.4 The Data-gathering Structure
12.10 Flight Data Monitoring (FDM) and Analysis
13.2 The Problem of Compliance
13.3 An Approach to CRM Training
14.4 Phase 3 - Programme Launch
15 The Safety Case - Managing Hazards and Risk in the
Training System
15.2 The Structure of the SC..
15.3 Constructing the Top-level Goals
15.4 Collecting the Best Evidence
AQP Advanced
Qualification Programme
ATQP Alternative
Training and Qualification Programme
CBT Competency-based
Training
CBTA Competency-based
Training and Assessment
CF Competency
Framework
EBT Evidence-based
Training
EVAL Evaluation
Phase
FOI Flight
Operations Inspector
IP Implementation
Plan
ISD Instructional
Systems Design
ITQI IATA
Training and Qualification Initiative
LC Line
Check
LOE Line
Operational Evaluation
LOQE Line
Operational Quality Evaluation (see LOSA)
LOSA Line
Operational Safety Audit
LPC License
Proficiency Check
MT Manoeuvres
Training
MBT Manoeuvres-based
Training
NAA National
Aviation Authority
NGAP Next
Generation Aviation Professional
OFDM Operational
Flight Data Monitoring
OJT On-the-Job
Training
OPC Operator’s
Proficiency Check
OPS Operational
Performance Standard
SAT Systems
Approach to Training
SBT Scenario-based
Training Phase
SC Safety
Case
SME Subject
Matter Expert
TA Task
Analysis
TNA Training
Needs Analysis
TPS Training
Performance Standard
Of the
billions of photons that strike the retina in the eye, only 40 per second are
processed by the brain. It seems
that the brain feeds forward an expectation of what the eye ‘should’ be seeing,
which is then compared with actual data received by the eye and the brain then
attempts to resolve any discrepancies.
By implication, the version of the world we hold in our heads is
probabilistic, not a truth.
Furthermore, no two people can possibly hold the same version of the
world although their individual versions usually correlate sufficiently for
them each to think that they are looking at the same scene. But this is only a part of the problem.
Imagine
that you are at the controls of an aircraft. In your head you hold a version of the
status of the aircraft and, also, a model of how it will respond to any inputs
you make via the controls or through the automation. You have acquired this model through
training and experience. You now
make an input, the aircraft responds and becomes established in a new, stable
state. If the hypothesis outlined
above is correct, your interpretation of the new status of the aircraft is
equally probabilistic, not a truth.
Furthermore, the final status of the aircraft is just one of many
possible end states that could have been achieved. The cause-and-effect relationship
between your input and the outcome is no more than a hypothesis about how the
world will respond. The robustness
of your model of the world will influence the probability of achieving the
desired outcome but it cannot guarantee it for several reasons.
First,
aviation takes place in a dynamic environment and, as such, exhibits
non-ergodicity. In simple terms this means that there is an inherent volatility
in the world that guarantees that nothing ever happens the same way twice. Second, the world is complex. Again, in simple terms, complexity means
that aviation involves multiple agents but with no single controlling
authority. The component parts,
therefore, have a habit of acting in unexpected ways. Finally, the world exhibits radical
uncertainty, which is to say that things go wrong in ways we could never
anticipate.
When we
operate an aircraft, we follow a trajectory from flight initiation to aircraft
shut down. That trajectory
comprises a sequence of goals, each of which has a specific configuration that
allows the task to be achieved within the constraints of the laws of
aerodynamics. The pilot’s job is to
configure the device in accordance with the requirements of the specific target
goal, to manage transitions between goals and, occasionally, to adapt to
unanticipated circumstances that might require goals to be modified or new
goals created. This all takes place
in a space defined by legal, commercial and aerodynamic constraints. To illustrate the point let us look at
one small segment of a flight, the final approach. In very simple terms the task can described
as:
Each of
these goal states has a specific set of criteria that must be met for the goal
to be achieved. In addition, there
are specific processes that must be applied to achieve each goal and to
transition between goals. Outputs
from the aircraft’s Digital Flight Data Recorder (DFDR) allows us to explore
the way pilots manage this notional trajectory.
This
next graphic shows data from 301 Airbus pilots attempting to flare the
aircraft. The DFDR output for the
Pitch Angle parameter has been processed by an algorithm that looks at the
statistical relationship between data points. The central dark blue band shows the
most closely related 50% of data while the light blue bands show the outer 20% of
the distribution (some data is lost because it fails the test of statistical
significance). The bands show
the distribution of data from 300 pilots who flew a normal approach. The red line is the trace of the 301st
pilot whose performance is a statistical aberration. The trace of data from this flight
differs significantly from the cohort of 300 peers.
4 3 2 1
With
data, we can render the goal state model tangible. We can follow the aircraft’s status on
the final descent path (1), the transition to the flare (2), the aircraft
established in the flare (3) and, finally, the transition to the ‘landed’ goal
(4).
But
this visualisation of normal data shows something more. First, we can see what happens
next. The Pilot 301 is initially
struggling to maintain the aircraft within the normal distribution but, at
Point A on the display, the aircraft’s path diverges. For whatever reason, Pilot 301 was
unable to maintain the aircraft within normal bounds. But that does not mean that the cohort
of 300 were perfect. The picture
shows us what happened next for the cohort, but it also shows us what didn’t
happen. They didn’t exceed the bounds of the normal distribution. Why not?
The
graphic is not simply an overlay of individual traces. It is a density plot of specific data
points, each of which is a function of several other factors but captured as a
value for a single parameter, in this case Pitch Angle. It can also be seen as a smart
graphic. We can interrogate the
array, and, for a specific point, we can trace several probable outcomes. For example, if we look at the area at
Point B, the circle encloses several related data points which represent the
Pitch angle of a cluster of aircraft at that moment in time. We can now trace those aircraft forward
to Point C and see the distribution of probable outcomes that relate to the
aircraft’s status at Point B.
Controlling for other variables, such as wind vectors, turbulence and
control inputs (all of which are captured in data and can be displayed), we can
start to understand why Pilot 301 followed an erratic path while the cohort of
300 did not. From a pilot training
perspective, we can start to understand how pilots can increase the probability
of achieving the desired aircraft status.
By
describing a flight as a sequence of goal states we can begin to examine the
knowledge base pilots draw on to constrain the range of probable outcomes. The criteria that apply to each goal
comprises what is known as declarative knowledge while the rules we apply to
manoeuvre between goals is known as process knowledge. The role of the aviation training system
is to provide sufficient declarative and process knowledge to allow a pilot to
operate an aircraft unsupervised.
But once a pilot enters productive service, we need to build upon that
foundation of retrospective knowledge and equip her with the skills needed to
cope with a world that is governed by the laws of probability. If aviation is to maintain its enviable
reputation for safety and airlines are to operate at maximum efficiency, the
system we use to train pilots must equip them to function in the world I have
just described. This means that the
training system must transition from one that is retrospective to one that is
prospective.
Retrospective
learning deals with the past. It
describes a set of known relationships that hold under the conditions
applicable at the time of sampling.
For example, if I was to ask what the capital of Germany is you would
say Berlin. But the accuracy of
that fact depends upon historical circumstances: Berlin has not always been the
capital of Germany. Equally, the
corpus of knowledge described by the EASA ATPL ground training curriculum
represents a set of decisions about historic artefacts and relationships. There is no empirical evidence to suggest
that the domain content prescribed by the syllabus possess any fundamental
worth. Retrospective learning
clearly has some value in that it provides what can be called underpinning
knowledge. But such knowledge is
only of use if it meets 2 criteria.
First, it must be capable of being generalised. This means that the specific information
taught must be capable of being recast as general principles that can be applied
to novel situations. Second, it
must be generative. The information
presented must be capable of supporting the creation of new knowledge. Retrospective learning can only go so
far in preparing pilots to cope with future challenges. Prospective learning, on the other hand,
supports adaptive behaviour capable of coping with the unknown.
The
concept of prospective learning is rooted in attempts to formulate models of
evolutionary development: how does learning contribute to an organism’s chances
of survival and, therefore, its opportunity to pass on its genes. It also has roots in Artificial
Intelligence. So, how must Machine Learning (ML) algorithms be written if
devices are to be truly ‘smart’ rather than simply being better than humans at
a limited range of tasks. The
concept has many overlaps with existing models of learning but does offer some
useful insights.
There
are 4 aspects of prospective learning that we need to consider. First, an entity must demonstrate continual
learning, which is remembering those aspects of the past that are relevant
to the future. In ML, new code
over-writes old code and so ‘forgetting’ is absolute, even of the old code had
some advantages. Equally, the very
first use of the term ‘proactive’ was to describe how prior learning in humans
interfered with new learning. The
implication is that pilot training systems must be designed so that we encode
declarative and process knowledge in such a way that it supports future
action. Much aviation ground
training seems to be little more than baggage. Unless academic knowledge can be
complied (generalisable and generative) in such a way that it informs action it
is of little value and will be quickly forgotten.
This
idea leads on to the second requirement of prospective learning, which is causal
estimation. Recognising that
outcomes are probabilistic, not deterministic, causal estimation requires us to
learn the structure of relations that support decisions that maximise the
probability of the most desired outcome.
We gain an understanding cause and effect, hopefully, because of
training. But, as we gain
experience, we elaborate our repertoire of goal state criteria and action
rules. This, in turn, builds better
causal estimation. Training systems
need to draw attention to the cues in the environment that suggest flaws in our
causal estimation, often resulting in situations that overwhelm the pilot’s
sense-making abilities (think ‘Air France’ and ‘startle’).
Because
of the complexity of normal life, we need ways to improve the efficiency of our
search for relevant information.
Known as constraints, these are things like heuristics, biases
and our assumed knowledge of prior probability distributions (‘priors’) that we
use to constrain the search space.
Of course, heuristics and biases will be flawed. Equally, a prior, in this context, is
simply a belief about what normally happens. The use of constraints need a critical
thinking control loop that gives feedback on the efficacy of our search
strategy.
Finally,
and interestingly, prospective learning includes curiosity, which is
action that informs future decisions, including future unmet situations. The EASA CBT Pilot Competence framework
describes ‘Knowledge’ as a competence.
While both clumsy and untenable (see Chapter 4), the spirit of the
concept comes close to the idea of curiosity. From a prospective learning perspective,
investment in curiosity requires effort that will offer no short-term reward
but could result in a pay off at some future time. Time spent refreshing procedural and
technical knowledge might have little subjective utility when set against any
alternative uses of that time, but the curiosity concept suggests that an
investment will support better coping strategies when faced with novel
situations. Curiosity, importantly,
also describes investing in learning around a topic, going beyond the defined
curriculum, doing more than the minimum.
Curiosity captures the concept of intrinsic motivation in learning
theory. Students with intrinsic
motivation - that is, they are learning a topic because they have an interest
in it - tend to out perform students with extrinsic motivation. Extrinsic motivation describes
students following a topic because they have to: they need to tick the box.
At this
point it might be worth saying something about error in learning. In simple terms, error reflects the
degree of fit between task demands and the action taken to satisfy the goal
requirements. Because there is some
buffering in the system, rarely is there a perfect fit between inputs and
outcomes. The system is constantly
adjusting to variations and perturbations.
Where action exceeds the system’s buffering capacity, the discrepancy is
noted as an ‘error’. To illustrate
the role of error in learning I want to use another analogy from Machine
Learning. ML algorithms work on
datasets that have been divided into 2 parts. One part is used to train the algorithm
and the other part is then used to test if the algorithm works. Unfortunately,
while the algorithm can deal with problems that are found within the
distribution of the data used for training (In-distribution Learning), it will
struggle or fail when presented with a problem that is
Out-of-Distribution. Humans, on the
other hand, can cope with Out-of-Distribution learning.
Consider
these 2 approaches. The first pilot (top) is encountering a crosswind component
that falls within the normal distribution of values experienced by the cohort
of 300 other pilots while the second pilot (bottom) encountered an
‘out-of-distribution’ crosswind component (note that the scales have been
adjusted on the Drift presentation to accommodate the data):
Both
pilots have made an ‘error’ but their responses are interesting. Whereas the first pilot seems to have
over-corrected for drift on touchdown, the response of the second pilot appears
to have been inadequate under the circumstances. Error, in this sense, is a feedback
signal. Whereas the first pilot is possibly fine tuning an existing model, the
latter will be elaborating on her model through exposure to a possibly novel
situation. In a prospective
learning context, error allows us to fine tune our causal explanation mechanism
and support continual learning through elaboration of the mapping between
stored knowledge and goal-directed action.
Out-of-distribution learning, which is a characteristic of human, rather
than machine, learning, is nonetheless brittle or fallible and, sometimes, the
price to be paid for the learning opportunity is catastrophic. Effective pilot training needs to allow
for out-of-distribution learning but in a safe context.
In this
Introduction we have set out a challenge for the pilot training system. Drawing on some concepts from other
domains, we have identified some criteria that a model of pilot learning must
satisfy. At the core of the model
is the need for pilots to cope with novelty. This has implications for the current
trend towards competency-based training.
Although
this Handbook has been developed to support Regulators and Operators wanting to
introduce EBT/CBTA to their training system, It is hoped that this Introduction
has offered some insights that will add value to the transition from legacy
training to the revised models.
New-born
infants to not emerge into the world completely unprepared to cope with what is
coming their way. We are a product
of evolution and certain behaviours seem to be transmitted genetically. For example, infants can orientate
towards faces while still in the womb and new-borns respond to voices ahead of
other sounds. Importantly,
new-borns appear to have the basic building blocks of associative learning.
Action
– behaviour - involves 2 processing
systems: bottom-up and top-down.
The bottom-up system responds to stimuli in a fast, automatic manner
while the top-down system is slow and deliberative. Importantly, the top-down system
modulates the bottom up system based on information stored in memory. Hommel, in his Theory of Event Coding,
proposed that perceived and produced actions - what we see and what we do - are the same in that the
processing of inputs and the initiation of action flows from the same neural
paths. Watching others fires the
same neurones as if I was doing that work myself. This is the essence of mimicry. The candidate mechanism for the system
is the mirror neurone, first discovered in macaque monkeys and reported in the
mid-1990s.
This
fundamental framework underpins mimicry, the main way humans learn. Of course, as the human matures it
acquires experiences stored in memory.
When I act
with others, pre-reflective (that is, prior to conscious evaluation), bottom up
processing feeds forward signals.
These are largely derived from information stored in memory about the
task we are engaged in, who is supposed to be doing what etc. They create the world that I am expecting to see. Because we are social animals, other
people are a part of that world I am looking at. Because of the way the mirror neurones
work, their actions trigger the same responses in me as if I was doing what
they are doing. It is through this
process that can learn by watching.
Mimicry underpins the way medieval guilds inducted apprentices into
their crafts. Unfortunately, it is
an inefficient model of learning. The roots of more structured approaches to
the development of training can be traced back to the American psychologist
Skinner. Working within the behaviourist tradition, Skinner elaborated the
concept of operant conditioning, which claims that learning can be influenced
by manipulating the learner’s environment.
Frame-based programmed learning, the model that still underpins most
computer-based training packages today, was the product of Skinner’s work.
The basic template for military
pilot training was established in 1917 with the formulation of what was called
the ‘Gosport System’, which relied heavily on mimicry. In the intervening 100 years the
industry has experienced a series of catastrophic shocks, the solution to which
has typically been additional technology.
Mid-air collisions gave rise to TCAS; flying into the ground was cured
with GPWS; ROPS is intended to stop aircraft going off the end of runways. What has not really changed is how we
train pilots and yet the role of the pilot has been transformed from that of
controlling a device to managing a flight path.
In the 1950s, Benjamin Bloom published his Taxonomy of Intellectual
Behaviours (1956), producing the first hierarchical model of different types of
learning. Bloom identified 3 learning domains: Cognitive, Affective and
Psychomotor. The cognitive domain referred to the processes associated with
mental skills, the affective domain refers to attitudes and the psychomotor
domain encompasses physical skills. The legacy of Bloom’s work is the
tripartite Knowledge, Skill, Attitude (K/S/A) classification scheme still used
in training design today. Two former students of Bloom, Anderson and Krathwohl,
later developed the taxonomy by matching types of knowledge to types of
activities.
In 1962 Robert Mager established the concept of Learning Objectives as
the key building blocks of training design. Mager proposed that training should
be based on a clear statement of observable behaviour. It is important to
remember that the behaviourist tradition worked in terms of observed outputs
from mental activity, the mental aspect remains hidden from view. So, a
behavioural objective is a statement of what a student should be able to do as
a result of some mental process being accurately executed. Mager refined the
concept by adding the degree of accuracy required to be certain that the
performance was reliable. He also proposed that the conditions under which the
performance was to be enacted should be made clear. Mager’s work underpins the
‘Performance, Standard and Condition’ structure of training objectives.
The contributions to the development of structured training have so far
concentrated on the identification and definition of training goals. In 1965
Robert Gagne published his ‘Stages of Instruction’, laying down the framework
for the delivery of training. Gagne identified a set of conditions to be met
and some activities to be conducted that, combined, would lead to effective
learning. Gagne’s work shaped the way lessons are delivered in classrooms
today. Gagne was also one of the first to propose the application of systems
concepts to education.
This era was the time of huge investment in complex technological
projects such as nuclear power and manned spaceflight. In order to successfully
accomplish these projects, man and technology had to be enabled to work
effectively together. Many of the tools of modern management, such as project
management and structured decision-making, were stimulated by the demands of
these complex projects.
The first coherent model of structured training was probably that of
Robert Glaser, published in 1962 but it was the USAF ‘5 Step Approach’,
published that same year, that brought the components of modern instructional
system design together for the first time. The 5 steps are:
• Analyse system
requirements;
• Define education
and training requirements;
• Develop objectives
and tests;
• Plan, develop and
validate training;
• Conduct and
evaluate training.
There have been various iterations of the basic 5 Step model and the
work of Florida State University, which published its ADDIE model in 1975, is
representative of the final stage in structured training systems development.
ADDIE stands for analysis, design, development, implementation and evaluation.
Whereas the 5 Step model was essentially linear in its conceptualisation, the
ADDIE model reflects a cyclical approach to training design in that the output
from training is constantly evaluated against operational need and changes made
as required.
Labelled the ‘Systems Approach to Training’ (SAT), the model was widely
adopted by the US military and by many NATO countries. The Systems Approach to
Flying Training (SAFT) was used to reconfigure ab-inito pilot training in the
UK RAF in the early 1970s.
On 4 October 1957 the Soviet Union launched Sputnik, the first
artificial satellite to orbit the earth.
In one creation myth, this humiliation for the United States resulted in
the ‘competence’ movement.
Recognising that simple course graduation was no guarantee of
proficiency, instead it was decised that there needed to be a framework for
demonstrating employability. In
1982, Richard Boyzatis published ‘The Competent Manager: A model for effective
performance’ which is also credited with starting the competence movement Reflecting changes in the workplace and
society, with increased job insecurity and worker mobility, the competence
theorists attempted to identify core skills, or competences, that were suitably
generic and transferable between workplaces. For example, a steelworker might
possess a set of competences that would allow that person to find work in a
different sector of industry but only require minimal retraining. Competences
were reflected in vocational training courses in schools and higher education.
In the UK, competence frameworks were developed for commercial pilots and cabin
crew by the industry Lead Body although the associated National Vocational Qualification
for pilots lapsed because of a lack of uptake.
A competence framework comprises descriptions of desired workplace
behaviour arranged in clusters. Communications skills, people management and
team skills are the 3 most frequent competence clusters according to a 2007
survey. The behaviours are usually tagged with specific underpinning knowledge
required of the individual to support the demonstration of the desired
behaviour.
The recognition that the existing framework for training and checking
mandated by the FAA in the USA might not be guaranteeing the competence of
commercial pilots gave rise to the introduction of the Advanced Qualification
Program (AQP) in 1990. Pilot proficiency is typically assessed in terms of
manoeuvres repeated at prescribed time intervals. So, a set repertoire of
manoeuvres must be flown to pre-determined levels of accuracy and must be
demonstrated at set intervals. However, this ‘one size fits all’ approach to
maintaining a competent workforce was increasingly being considered
inefficient. For a start, pilots acquire proficiency at different rates and
skills decay at different rates. Airlines operate into very different
environments with very different equipment and yet all have to meet the same training
requirements. The guiding principles of AQP are that each individual operator
must determine the skill set required of its pilots and that training and
checking must be based on the needs of individual pilots within the operational
context. The AQP regulations allow for the voluntary adoption of the program;
operators can continue to follow the manoeuvre/ interval-based, or ‘legacy’,
model. The AQP concept has been broadened to include cabin crew and dispatcher
training.
Aware of developments in the US, the first draft of JAR OPS 1 included
line entries referring to AQP. The
promulgation of JAR OPS 1.978 - the Alternative Training and Qualification
Programme - in 2006 provided a framework for JAA – later EASA - carriers to
adopt a training and checking regime based on line operations. The regulation
is built on the experience of AQP but incorporates developments in flight data
capture and analysis, safety management and auditing that have occurred in the
intervening period since AQP was first introduced.
We have just briefly sketched out the origins of structured models of
training analysis and design. From this it can be seen that ATQP is simply the
application of Instructional Systems Design (ISD) to commercial pilot training
and testing. In order to understand the significance of ATQP is necessary to,
first, review the existing framework for training and testing. In broad terms,
commercial pilot training comprises 4 phases: initial license training; type
conversion training; operator’s conversion training; recurrent training. In
addition, it is possible to distinguish 2 discrete aspects of the system. The
first is to train to a set standard. The second is to test competence, both at
the end of initial training and, again, at set intervals during employment.
Traditionally, the broad structure and content of the training course and the
criteria for success have been contained in regulations promulgated by national
authorities. The role of the airline training department is to configure
training in such a way that it demonstrates compliance with regulatory
requirements.
This model of training is pragmatic in the sense that it is rooted in
generations of operational experience and is successful in that aviation
remains a highly reliable, yet hazardous, industry. However, in a competitive
marketplace, training departments compete for resources with the rest of the
airline. As such, there is often little spare capacity to accommodate changes
in the operational world, such as the seasonal characteristics of operations or
changes in technology. The ‘compliance’ model of training delivers a product
that meets regulatory demands but is not necessarily mapped onto the needs of
the specific airline.
Recognising that the aviation industry faced a potential recruitment
shortfall across all sectors, IATA, through the ITQI, and ICAO, through NGAP,
have both been looking at introducing structured training models based on a
competence approach. Both are looking at a broader audience that just pilots
but the key difference, initially, was that ICAO were concerned that any
competence framework should cover initial selection and training as well as
in-service development and advancement.
The IATA project was the
first to bear fruit. In order to
break away from the historical ‘set manoeuvre’ model, a large scale analysis of
various data sources was undertaken, resulting in the EBT Data Report. This provided ‘evidence’ of what
training topics were more appropriate for modern generation aircraft. The goal of training was recast: no
longer did pilots have to demonstrate accomplishment in manoeuvres, they had to
demonstrate ‘competence’ in the control of the aircraft, including the
management of the flight path under normal and non-normal circumstance.
One of the first
challenges was to define a ‘competence’? Was it generic or specific? One school of thought suggests
that the manipulation of numbers and words and the ability to learn were
fundamental ‘competences’ and all performance flows from these basic abilities. Another line of thought, discussed
earlier, took the view that competencies were arbitrary clusters of skill and
knowledge that were applicable to a specific work context. In effect, they are whatever you want
them to be as long as they make your workforce effective.
Another problem is the
interpretation of ‘evidence’ in.
The current usage of the term ‘evidence-based’ can be traced to a 1972 paper by
Archie Cochrane that questioned the effectiveness of medical treatments. Given the increasing costs of delivering
healthcare and the range of treatments available, how do you decide what works
best? The combinations of patient,
condition and treatment should be evaluated using Randomised Controlled Trials
(RCT) as the gold standard of evidence.
So, the ‘evidence’ was what could be proven to work best. Other fields, such as social policy,
have adopted the concept but the underlying idea remains the same. In aviation, a direct analogy would be
an investigation of pilot experience level, skill to be trained and training
device employed. The closest we
have come to true ‘EBT’ in aviation
are attempts to assess training transfer in flight simulators. Prof Inez de Florio-Hensen, at
Kassel University, argues that, in education, EBT is, in any case, an
unattainable goal. The range of
variables – student, teacher, subject matter, training situation – is simply
too great to make RCTs meaningful.
CBTA seems to be the application of ISD to initial training in all
specialisations while EBT refers to airline recurrent training.
We can use the
trajectory of a pilot’s career (or any other employee) as an organising
framework to bring some of these concepts together. In the diagram above, the
trajectory starts with initial training for the award of a license. The input standard is usually a novice
with no skill or knowledge and the output standard is someone deemed fit to
hold a license. But a license
simply allows an individual to operate an aircraft appropriate to their
qualification. It does not
guarantee any level of expertise beyond a baseline level of safety.
As that
individual gains in experience they may want to start looking for a job. First, they need an aircraft type
rating. This involves applying
their prior learning to the specific instance of the new aircraft type. But they also need to convince an
employer that they are a good fit for the company. This is where a ‘competence’ model comes
in handy. The employer knows that
the applicant is legal if they possess a license and the required ratings. The competence model describes the additional
attributes needed of the pilot to be successfully employed by that
airline. At the recruitment stage
the employer is simply looking for evidence of behaviour that maps onto the
competence model. In short, what
does the pilot bring with them that can be exploited and developed in their new
role?
As the pilot
progresses, 2 things need to happen.
From a legal perpective the airline needs to show that the pilot has
maintained the level of proficiency required to hold their license. From the airline’s perspective, the
pilot needs to show that they are capable of coping with the operational
demands likely to be encountered.
The airline needs to have a competence model that captures those demands
and sampling tool to evaulate the individual.
The pilot
lifecycle approach shows how we need different tools at different stages.
Aviation Authorities
around the world have relied on the periodic accomplishment of a set of
manoeuvres as proof that a pilot is competent. Supporters of EBT argue that the event
set has failed to keep track with changes in technology and so a new way of
assessing is needed. It shoud be remembered that the State
has a legal duty to ensure the safety of its aviation system. The airline wants to be sure that its
pilots can do the job. The State
and the Airline have different goals but, historically, have used the same
performance measure.
EBT enthusiasts confuse
product and process. Product is the
observable output while process is how that output is achieved. The EBT argument is with the ‘product’ –
an anachronistic set of manoeuvres – but ignores process. If a pilot can cope with the historic
manoeuvres then she can cope with any in-flight problem. At this point we need to go full circle
back to the beginnings of structured training design and ask what do we really
mean by skills and knowledge.
Stellan Ohlsson, in his book ‘Deep Learning’, proposes that
expertise relies on an individual being able to establish a set of constraints
that must be satisfied for a particular condition to be considered true. Through training and experience we
develop an increasingly fine-grained set of constraints for an increasingly
varied repertoire of situations.
Process knowledge comprises the actions and rules sets we deploy to
satisfy the situational constraints.
Interestingly, he proposes that errors are, in fact, feedback signals
that tell us that a constraint has not been satisfied. In effect, either we have applied
inadequate constraints (faulty knowledge) or implemented an incorrect action.
The implications of
Ohlsson’s ideas are significant and
far-reaching. For a start, it is
not enough for a pilot to demonstrate the control of the aircraft to be
considered competent: they might have just been lucky on that day. We need to explore how pilots think
about control.
Competence, then, is
thinking made manifest. In the rest
of this manual we will look at how to achieve that goal.
We
can see 3 broad trends that have emerged since that first formalized flight
training system. Training is
expensive and so it has come under increasing pressure to be cost
effective. In order to achieve this
goal, systematic approaches to analysis and design have been implemented. Finally, steps have been taken to map
training onto operational need.
These last 2 themes form the structure of the rest of this book.
The
starting point for both philosophies (AQP/ATQP and EBT) is a framework that
describes the range of performance we expect of pilots under all normal and
non-normal situations. AQP/ATQP follows the conventional ISD model and
starts with a Task Analysis (TA).
EBT, on the other hand, starts with a Competence Model (CM). The processes we use to develop both are
similar. The real difference is how
we describe performance, As we will
see later the eventual output from a TA is a syllabus of instruction. A CM is more akin to a job specification
or a set of Terms of Reference. In
both cases the final output needs to be a comprehensive description of the
performance expected such that we can be confident that we are able to verify
that pilots are competent.
The
main difference between the ISD-derived models and the competence approach is
that the former has a well-established process with well-understood methods
that can be applied to the task of develoing training. The competence approach, on the
otherhand, lacks any rigorous processes, in part because it was never intended
to be a design methodology.
This
chapter will deal with the ISD methodology while, in the next chapter we will
look at developing competence models.
2.2
The ADDIE Process Explained
The
ISD model, of which ADDIE is just one version, is a closed loop that starts
with analysis of requirements and comes back to the starting point via a number
intermediate of stages. The concept
is illustrated below. Because ADDIE
is refered to in the official documentation, we will discuss it within the
broader ISD context.
Step
1 is the Analysis phase. Here, we
look at the task required to be mastered by the trainee. We look a the
characteristics of the trainees, constraints on delivery, projet timescales
etc. Step 2 is the Design
phase. Here we define the output
standard. We do this now because
this will shape how much time is needed to train and what methods will be
needed to train and test. An output
from Step 2 is the syllabus. The
syllabus is usually framed as a set of learning goals or objectives that must
be achieved by the trainee. We will
look at objectives in more detail later. We also look at testing methods unde
the Design umbrella.
Having
described the syllabus, the next step is to do a Training Needs Analysis
(TNA). The TNA identifies the gap
between the Output Standard and skills mix of the entry level students. For example, an ab initio pilot just
graduating from a flight school will have a bigger gap between his current
status and that required at the end of a type conversion. However, an experienced pilot converting
from a different aircraft type will have a narrower gap between entry level and
output standard. The TNA will
inform the next step, which is curriculum design.
Analysis
Evaluate Implement Develop Design
Although
useage does vary, I will use the syllabus to describe the course training
objectives and the curriculum to describe the allocation of objectives to
training events. Curriculum Development -Step 3- is where decisions are made about method
of delivery, training media to be used, time to be allocated, sequencing of
events and so on. It is where we do
the heavy lifting of making a course.
Then we have the Implementation phase. Implementation covers initial roll out,
prototyping, fine-tuning and bedding down the production version of the course.
It also includes trainer training and standardization.
Once
the course is up and running, we need to think about Evaluation. We look at this in more detail later in
the book but. Essentially, evaluation asks ‘does the course work?’ If not, then we need a process for Modification.
The
purpose of a JTA is to establish the baseline criteria for each job. The JTA
can be likened to a product specification for the conduct of duties and, as
such, has some similarity with a competence framework. The difference being that the TA is a
more fine-grained description of the actions associated with completing the
task. The TA will ultimately determine the instructional goals and objectives,
specify the type of knowledge required for the job, assist in determining
instructional activities and aid in constructing performance assessments. It
will also serve as the basis for auditing the company’s training
The
JTA, then, is exactly what the name implies; it is a list of actions associated
with the task for which an individual is responsible, stated in observable
objective statements. That is, each task should begin with a verb that
describes the nature of the activity associated with that task. For example, simply listing items to be
checked by a pilot is not a JTA. Instead, responsibilities should be listed in
terms of the observable action associated with that responsibility. For
example:
Poor task list:
1. engine oil
2. engine temperature
Acceptable task list:
1. check engine oil quantity
2. observe engine temperature
The
first step in developing the JTA is to compile a task list. There are several
ways to create a task list. One way is direct observation of the task. The
analyst observes a representative sample of the workforce and notes down job
behaviours as they occur. While this method works well because it takes place in
a naturalistic environment, it often does not allow the observer to catch all
aspects of the job. There is a view that the observer should be familiar with
the job, while others feel that a novice is best. The familiar observer may be
able to label tasks correctly and more accurately, but may bring their own bias
to the task. While a novice observer may not know the reasons for a particular
task, they are clear of procedural bias and assumptions.
Another
method for creating a task list is interviewing a Subject Matter Expert (SME).
It is best to include more than one SME for the interview process in order to
cover all situations and perspectives. The interviewer will typically ask the
SME to talk through the job out loud. The interviewer will want to ask questions
such as:
• What specific
duties must an employee perform?
• What units of work
must be completed?
• What handbooks must
be consulted?
Once
the task listing is complete, the next stage is to review each task to see if
there is a need for further decomposition. The task decomposition should not be
overly detailed such that the listing becomes cumbersome. Equally, it should
not be so vague such that it does not provide an adequate description of the
company’s requirements. There is a view that cut-and-pasting a JTA saves time. After all, an A-320 is an A-320 no
matter what the logo on the tail says.
In fact a JTA is specific to each company: no 2 airlines fly the same.
The
JTA is rooted in behaviourist psychology and, therefore, typically examines just
the observable behaviours needed to perform a job. However, some tasks require
non-observable behaviours, such as evaluative thought processes associated with
process control and decision-making skills. These types of behaviours can still
be represented in the task analysis but require an additional cognitive task
analysis. Table 2.1 briefly lists some types of task analysis and when to use
them. A common mistake is trying to force fit a job into a task analysis for
which it is ill suited.
Job/Performance
Used
for procedural skills
Learning/Needs Only
identifies what must be taught; secondary analysis
Cognitive Examines
how people think about situations
Content/Subject Matter Breaking
down large amounts of information into teachable
units
Table
2.1. Types of Task Analysis
There
is no ‘ideal’ template for laying out a JTA. To a large degree it depends upon
the depth of analysis and intended use of the JTA within an airline. In an
integrated Safety, Quality and Training system, the content of the JTA will be
referenced to training records, crew scheduling, auditing and event reporting.
Therefore, it makes sense for the JTA to be built in a database product. Here
is one example of a JTA:
Takeoff Operations:
Normal Takeoff Procedure
Release Brakes
Align airplane on runway centreline
Transfer control of airplane to First Officer, if required
Call out": "YOU HAVE THE AIRCRAFT," if required
Call out": "I HAVE THE AIRCRAFT," if required
Maintain directional control with Rudder Pedal Steering and Rudder
GUARD Nose Wheel Steering until both engines stabilised and symmetrical
Advance Thrust Levers to approximately 50% N1
Ensure engines stabilised and symmetrical
Advance Thrust levers to FLEX or TOGA detent as required
Apply slight or full-forward side stick as required
Call out: "FLEX" or "TOGA" as required
Verify "FLEX" or "TOGA", SRS, and RWY (if
applicable) on FMA
Compare LP Rotor Speed (N1) to N1 rating limit on E/WD
Call out: "FLEX SET," or "TOGA SET" prior to 80
knots
Assume/Maintain Control of Thrust Levers
Call out: "80 KNOTS"
Acknowledge 80 knot call out: "CHECKED"
Remove forward side stick pressure at 80 knots to be neutral by 100
knots
Maintain wings level attitude with side stick
Monitor engine instruments
Call out: "V1" at V1 -5 knots
Remove hand from thrust levers
Call out: "ROTATE" at VR
At Vr, Rotate smoothly to SRS commanded attitude (or 110 degrees if no
SRS)
Call deviations from normal flight instrument indications
Call out: "POSITIVE RATE" (when a positive rate of climb is
indicated)
Ensure positive Rate of Climb
Call out: "GEAR UP"
Here is another example from a different airline for the same aircraft
at the same stage of flight:
1. Demonstrate the ability to perform a normal takeoff and initial climb
to flap retraction altitude in accordance with AOM and the company FOM.
2. Apply the appropriate CRM skills when performing a takeoff.
The JTA underpins the training development process and is fundamental to
the continued safe delivery of airline training. The JTA supports the Safety
Case (SC) (see Chapter 11) and also drives curriculum development. It is the
most time-consuming component of the ISD process. However, unless it is done
properly, it can also be the Achilles heel of the Training Department. Time
spent getting the TA correct will show a payback later in the implementation
phase
We
can develop a JTA, then, by inspecting documents, by observing performance and
by interviewing line pilots. We
could also look at safety reports and LOSA for evidence of poor performance
that can then be used to elaborate on the original analysis.
Table
2.1 shows a JTA that was developed for an Airbus operator. First, a number of management pilots who
were also trainers decided on a meaningful structure to describe the
performance of a pilot (Units of Work).
Next, the company Operations Manual, the aircraft FCOMs and the company
Flight Crew Training Manual were cross- referenced to the task structure. Rather than describe the job, references
were used for the sake of efficiency.
Each reference relates to a piece of documentation that describes the
task. Finally, experienced training
captains were asked to review the document and identify any gaps. The process looked at normal operations.
Unit of Work |
Baseline (SOP, FCTM) |
Probable Contingencies |
Pre-Duty |
OMA 5.2.1-5.2.3 (recency req.), 6.1 (medical fitness), 6.2 (medical precautions), 7 (FTLs), 8.1.12 (documents to be carried), 14.1.1 (documents to be carried by crew) , 14.1.2 (uniform) , 14.5 (crew bags) |
|
Dispatch |
FCOM: PRO-NOR-SOP-02 P1/6 |
|
Aircraft Pre-flight |
FCTM NO-020 P4/12-11/12 e-Library –
Loadsheet ACARS PERF setup FCOM: PRO-NOR-SOP-03 P1/2 (safety exterior inspection), PRO-NOR-SOP-04 (power-up & before walkaround), PRO-NOR-SOP-05 (exterior inspection), PRO-NOR-SOP-06 (cockpit preparation), PRO-NOR-SRP-01-10 (cockpit preparation) |
|
Start/Pushback/towing |
FCTM NO-030 (eng
start) FCOM PRO-NOR-SOP 01 P9/20 (pushback & towing), PRO-NOR-SOP-07 (before pushpack or start), PRO-NOR-SOP-08 (engine start), PRO-NOR-SOP-09 (after start), PRO-NOR-SRP-01-10 (before pushback or start), OMA 8.3.20 (pre-taxi) |
|
Taxi |
FCTM NO-040 PRO-NOR-SOP-10 (Taxi), PRO-NOR-SRP-01-20 (taxi), OMA 8.3.21 (taxi) |
|
Take off/Rotation |
FCTM NO-050
P1-8/14 FCOM: PRO-NOR-SOP-11 (entering the runway), PRO-NOR-SOP-12 (takeoff), PRO-NOR-SRP-01-30 (takeoff), OMA 8.3.22 (takeoff) |
|
Initial Climb (to CLB thrust) |
FCTM NO-050 P8-13/14, FCOM PRO-NOR-SOP-13 (after takeoff), OMA 8.3.22.1 (climb graph) |
|
Departure (SID) |
OMA 8.3.23 (Departure and climb) |
|
Climb to cruise level |
FCTM NO-060 FCOM PRO-NOR-SOP-14 (climb), PRO-NOR-SRP-01-40, PRO-NOR-SRP-01-50 |
|
Cruise |
FCTM NO-070 FCOM PRO-NOR-SOP-15 (cruise) |
|
Descent preparation |
FCTM NO-080 FCOM: PRO-NOR-SOP-01 P15/20 (landing perf), PRO-NOR-SOP-16 (decent preparation), PRO-NOR-SRP-01-50 |
|
Descent |
FCTM NO-090 FCOM: PRO-NOR-SOP-01 P15/20 (descent profile), PRO-NOR-SOP-17 (descent initiation/monitoring/adjustment), PRO-NOR-SRP-01-60, OMA 8.3.25 (descent) |
|
Approach (STAR/Holding) |
FCTM: NO-100 (Holding), NO-110 P1-4/10 (Initial App), PRO-NOR-SOP-18, PRO-NOR-SRP-01-70 P1-3/32, OMA 8.3.25.3 (holding speed), 8.3.26 (approach) |
|
Final Approach |
FCTM: NO-110 P4-9/10
(final App), NO-120 (ILS), NO-130
(Non precision app) FCOM: PRO-NOR-SOP-01 P 15/20 (stabilized approach), PRO-NOR-SRP-01-70 P3-10/32, OMA 8.3.26.1 (stabilized approach) 8.3.26.2 (approach ban), 8.3.26.3 (ILS) |
FCTM NO-160 (LVO app), FCOM PRO-NOR-SRP-01-70 P11-23/32 |
Flare and Landing |
FCTM NO-170 e-Library –
landing tips & Final Approach and Landing Technique FCOM PRO-NOR-SOP-19, OMA 8.3.27 (landing) |
|
Go Around/Rejected LDG |
FCTM NO-180 e-Library –
Go-Around FCOM: PRO-NOR-SOP-01 P 16/20 (mandatory missed approach), PRO-NOR-SOP-20, PRO-NOR-SRP-01-80, OMA 8.3.28 (go around) |
|
Rollout |
FCOM: PRO-NOR-SOP-01 P17/20 (touchdown and rollout), PRO-NOR-SOP-21 (after landing) |
|
Taxi in and clean up |
FCTM NO-190 |
|
Shutdown |
FCOM PRO-NOR-SOP-22 (parking) |
|
Securing |
FCOM PRO-NOR-SOP-23 (securing the aircraft) |
|
Table
2.1 Airbus JTA
For
each element of competence, Subject Matter Experts (SMEs) were asked to
identify the range of contingencies that might apply in order to verify that
coverage was complete. We also
looked at differences between roles (PM/PF) and advancement (Command).
Dealing
with non-normal or emergency procedures requires a different approach. Whereas normal operations follows a
distinctive, repetitious pattern (generally speaking), non-normal/emergency
situations tend to require a safety template to be overlain on the situation
which is then used to select an appropriate action. Competence in this sense might be
described in generic terms:
• Establish/sustain
control
• Evaluate
systems/flight path status
• Identify problem
• Identify
appropriate checklist(s)
• Execute checklists
• Validate system
response
• Choose next course
of action
• Monitor status
The
TA is a job specification. It describes the actions required of an operator if
a task is to be completed successfully. The goal of training is to develop the
skills of an individual so that they can complete the tasks in the TA without
supervision and to an acceptable standard. The first stage in developing a
course is to scrutinise the TA in order to identify those aspects of
performance that will need training.
Training
Need will be driven by the entry level of the trainees. An airline Initial Type Conversion
designed for ab inito cadets recently graduated from flight school will require
greater depth and content that an Operator’s Conversion course designed for
previously qualified pilots recruited from another airline. In the context of airline recurrent
training, it is unlikely that we will be developing modules with content tat is
completely novel. In most cases,
‘training’ will be updating, adapting or linking to existing knowledge. Decisions about the depth of knowledge
and the time allocated to training will be influenced by this analysis of the
entry level.
For
each task we need to identify the skilled performance associated with the task
as well as any underpinning knowledge essential for successful task completion.
Underpinning knowledge will include an explanation of why the task is
important, any theoretical knowledge associated with completing the task,
probably risks attached to the task and any alternative strategies for task
completion.
Having
clarified the goals of the course, we now need to create the syllabus by
writing the Training Objectives (TOs). A TO typically comprises 3 parts:
A statement of performance.
A statement of the conditions under which the performance is to be
demonstrated.
A statement of the standard to be achieved for the performance to be
considered acceptable.
The
performance statement is worded in terms of observable actions using verbs. We
want to be able to witness the performance in order to assess the level of
achievement. Therefore, objectives describe the external manifestation of
competence. Because of the variability encountered during normal line
operations, any specific skill might be performed under a range of conditions.
The condition statement describes the range of contingencies under which a
trainee will be expected to perform in training so that we can be assured that
they will be able to cope with line operations. The standards statement defines
any bounds of acceptable performance we want to attach to each objective. A
standard might be a tolerance within which the skill is to be performed or it
might be a procedural limitation. An example of the TO might be:
Performance:
Land the aircraft
Conditions:
Within a range of crosswinds, at night, within a range of runway surface
conditions
Standards:
Within touchdown zone, within speed and ROD constraints.
These items drawn from the EASA HP&L syllabus illustrate weaknesses
in objective formulation:
a)
List the
factors determining pulse rate.
b)
Question the
established expression ’safety first’ in a commercial entity
c)
Describe the
personality, attitude and behaviours of an ideal crew member
Item a) is a valid objective.
Item b) starts with a verb but the rest of the performance statement
makes no sense. Is the student
supposed to question a safety policy in order to to elaborate on the entity’s
SMS? Is the aim to question the veracity of the statement in the first
place? Item c) collapses 3 possible
objectives into one and is a good illustration of ‘signposting’. Rather than require students to declare any
knowledge in relation to the 2 key concepts - personality and attitude - it
suggests that there is a desired ‘correct answer’ which, in any case, could
only be an opinion given the uncertain status of personality traits in relation
to pilot performance.
Writing TOs is as much an art as a science. Drafting acceptable TOs does require
skill so it might be worth looking at some examples to illustrate the
challenge. Appendix to Annex I to
ED Decision 2018/001/R offers this set of TOs in relation Mental Maths:
100 09 00 00 |
MENTAL MATHS |
Show, in non-calculator
tests and/or exercises, the ability in a time-efficient manner to make
correct mental calculation approximations: |
|
(1) |
To convert between
volumes and masses of fuel using range of units. |
(2) |
For applied questions
relating to time, distance and speed. |
(3) |
For applied questions
relating to rate of climb or rate of descent, distance and time. |
(4) |
To add or subtract time,
distance, and fuel mass in practical situations. |
(5) |
To calculate fuel burn
given time and fuel flow in practical situations. |
(6) |
To calculate time
available (for decision-making) given extra fuel. |
(7) |
To determine top of
descent using a given simple method. |
(8) |
To determine values that
vary by a percentage, e.g. dry-to-wet landing distance and fuel burn. |
(9) |
To estimate heights at
distances on a 3-degree glideslope. |
(10) |
To estimate headings
using the 1-in-60 rule. |
(11) |
To estimate headwind and
crosswind components given wind speed and direction and runway in use |
This example is clumsy and can be reframed thus:
Performance |
Condition (Common to all LOs: In an examination comprising x questions. Without the use of aids to calculation) |
1. Apply the 4 Rules of Number |
Using Whole Numbers, Decimals, Percentages. |
2. Convert between units of measurement |
Mass, Volume, Distance, Time. Given conversion factors |
3. Apply Rules of Thumb |
1 in 60 rule, Rule of Thirds (headwind and crosswind components). Lateral navigation (track, heading) Vertical navigation (height) |
We saw that the purpose of the TNA is to establish
what needs to be taught given the entry level of the students. Although
students will be expected to demonstrate the performance described in LO 1, we
can assume that they are already numerate and so no formal training will be
provided. Equally, we can assume
that our students understand the various terms such as ‘mass’ volume, ‘decimal’,
‘percentage’ and so on so we do not need to offer training.
For LO3, however, some of the Rules of Thumb might
not be known to the class. In this
case we need to elaborate. So, in
this case we can consider ‘Apply Rules of Thumb’ to be the Terminal Objective
(TO) and we would create some Enabling Objectives (EO) that allow the student
to achieve the TO.
Performance |
Condition |
3.1 State the 1 in 60 Rule |
in relation to: Lateral navigation Vertical Navigation |
3.2 State the Rule of Thirds |
In relation to: Heading Groundspeed |
In this example, the 3 TOs all describe what would be called a ‘skill’,
in this case the mental manipulation of values. TOs are traditionally divided into 3
categories: skills, knowledge and attitudes. Skills are what you ‘do’ while
‘knowledge’ is what you know. There
are no ‘attitude’ objectives contained in the table. We might decide that an attitude
objective is appropriate in this case.
So, we might consider these as candidates:
a)
State the
reasons why a ‘gross error check’ on outputs is needed when entering data into
aircraft systems
b)
List the
reasons for conducting a ‘dead
reckoning’ cross check on aircraft performance
Attitudinal objectives are recognised as difficult to define and almost
impossible to test.
In this chapter we have looked at ISD as a model of training design and
have differentiated between ISD as a process for designing inputs to bring
about behaviour change and the use of ‘competencies’ to describe workplace
performance. The remaining steps in
ISD wil be covered in the following chapters.
In
Chapter 1 we look at various models of training design. A key difference between classical
approaches to training design and the competence approach is that the former
addresses a specific job or task whereas the latter supposedly is designed to
develop a ‘generic’ set of behaviours that can be transferred between different
jobs. For example, there might be a
range of different workplaces that all require an ability to make ‘decisions’. If I have a fundamental
‘decision-making’ toolkit then it doesn’t matter if I am an office clerk or an
astronaut, I can still have a go at making a decision. The complexity of the decision to be
made and the consequences of failure may differ but the process remains the
same.
Whereas ISD, then, looks at the interventions needed to bring about a
change in performance, the competence concept really looks at workplace
performance: can someone do a job?
To a degree, ‘competence’ is blind to prior training. It isn’t interested in how a candiate
got here, just can that person do the job.
There are some conventional ISD concepts that can help clarify the
differences beteen the approach. In
any training system there are constraints on what can be achieved in the time
and resources available and the scope of the training system in terms of
workplace performance. The output
from training is usually described as the Training Performance Standard (TPS)
and recognises that there is a gap between that and the Operational Performance
Standard (OPS). The OPS, in fact,
equates to the level of competence expected of a person in productive
employment. The gap between TPS and
OPS can be bridged by formal programmes of On the Job Training (OJT), mentoring
or simply informal development through exposure to the real world. The TPS is usually specified in ISD – it
is the graduation standard – but the OPS is often left undefined.
To illustrate the problem, consider this OB from the Communication
competence:
OB 2.8 Uses and interprets non-verbal
communication in a manner appropriate to the organisational and social
culture (my emphasis)
In the 100KSA (2018) Communication requirement this has been elaborated
as:
09) Show the ability to correctly interpret
non-verbal communication.
10) Show the ability to use appropriate eye
contact, body movement and gestures that are consistent with and support verbal
messages.
The OB relates to non-verbal communication in a very specific context:
in relation to the organisational and social culture. The 100KSA elaborations establish an
expectation - correctly, appropriate, consistent, supporting - without making
clear what training inputs might be required nor how a trainee can meet these
expectations. Nor does the 100KSA formulation address issues of organisational
and social culture.
In theory it ought to be possible to trace a line from the initial
ground training requirement, through the practical application to achieving the
final operational assessment. The
piecemeal approach to developing commercial pilot training is still some way
off that goal. The TPS, then, should identify a set of generic
performance elements that broadly map onto the OPS. The TPS should describe
both activities and underpinning knowledge that supports the activity described
in the OPS.
One
problem we have is that we also need a mechanism for assessing
performance. It is important to
understand that a competence model and an assessment framework are not the same
things. The ‘problem’ is that they
may overlap and share terminology.
In this chapter we explore some of these issues.
Although there are well-defined activities associated with the ISD
process, developing competencies is less well supported. The paradox of EBT is that it claims to
be moving away from ‘task-based’ assessment but being ‘competent’ is
fundamentally based on doing tasks.
However, implicit in the competence approach is that performance is
abstract – generalisable across different work contexts – and aimed at
maintaining control of tasks, especially in uncertainty. Competencies try to guarantee control.
The UK Chartered Institute for Personnel and Development makes the
following points about competencies:
They ‘focus on someone’s personal attributes
or inputs. They can be defined as the behaviours (and technical attributes
where appropriate) that individuals must have, or must acquire, to perform
effectively at work.
[…[ are broader concepts that cover
demonstrable performance outputs as well as behavioural inputs. They may relate
to a system or set of minimum standards needed to perform effectively at work.
A ‘competency framework’ is a structure that
sets out and defines each individual competency (such as problem-solving or
people management) required by individuals working in an organisation’.
‘In designing a competency framework, care
should be taken to include only measurable components. It's important to
restrict the number and complexity of competencies, typically aiming for no
more than 12 for any particular role (preferably fewer), and arranging them
into clusters to make the framework more accessible for users. The framework
should contain definitions and/or examples of each competency, particularly
where it deals with different levels of performance for each of the expected
behaviours. It should also outline the negative indicators for that competency
competency – the behaviours deemed unacceptable’.
Importantly, there is no single, universal solution to this idea of a
competence model, nor is there is a single way to develop them. Organisations need to develop a model
that supports their commercial and operational goals. Competence models are useful because
they make clear to employees what is expected of them in any particular job or
role. This is why they should be
framed in the language of observable behaviour. A competence model also directs attention
to training requirements. A person
cannot be expected to do a job if they have not been properly trained. However, if you look again at the
extracts above, it does talk about behaviours that individuals must have or
acquire. This last point is
significant. The ‘must have’ items
can be dealt with by recruiting people who have done the job before or through
pre-employment training. Acquiring
competence, as we saw erlier, can be done through structured workplace
development. Which brings us to
EBT. If you scratch the surface of
the EBT concept, what we are really talking about is a process of structure
workplace mentoring aimed at sustaining and developing a pilot’s ability to
cope with operational demands. It
is not really training at all.
The rise of CBT/EBT coincided with the discovery of ‘Black Swans’ -
catastrophic but unpredictable events that we still need to be able to cope
with. Although, by definition, we
cannot train to deal with ‘Black Swans’, we can still use the concept as a
jumping off point. There are 2
other, more common, properties of the world we need to consider: non-ergodicity
and radical uncertainty. The first
describes how things never happen the same way twice and the second relates to
how things have a tendency to fail in ways we never anticipate. So, a competent pilot must be able to
cope with a constant level of perturbation in the workplace (think ‘threats, if
it helps) and, should something happen, then be able to restore an acceptable
level of control as quickly as possible.
In terms of ‘competence’, we can illustrate the situation like this:
A problem we face with
developing competencies is that, often, behaviour is based on deeper,
underlying processes that occur internally. Behaviour is just the manifestation of
these processes. It could be argued
that true ‘competence’ is really these behavioural precursors. The table below proposes a number of
target precursors.
Competence |
Supporting Activities |
Analysing |
Causal analysis; risk
appraisal; establish the gap between observed and expected; establish
abnormal cues based on mental model; compare assumptions about cause and
effect relations among cues |
Planning |
Identify options;
establish operational constraints; clarify remaining
capability/functionality; planning for contingencies |
Organising |
Identify actions required;
establish resources required; implementing contingency plan |
Validating |
Referencing observed
behaviours to expectations; establish deviations from normal state; use
critical thinking |
Deciding |
Validate rule set; identify information requirements;
validate efficacy of option; establish time reuirements |
Communicating |
Use proper phraseology :
pay attention to completeness of standard reports; seek
information/clarification/ check understanding; exchange information and comprehensions
to establish a shared understanding of the problem; formulate and communicate
hypothesis about cause and effect relationships |
Collaborating |
Monitor, support others,
provide guidance and suggestions; states appropriate priorities; update situation
periodically; resolve opposing interpretations based on team conflict
resolution |
Coping |
Create space and time;
control stress |
Assessment of performance
is highly problematic. The tools we
use – marker frameworks – must meet 2 criteria if they are to be considered
useful. First, a category must meet
the requirement of validity.
Validity is the degree to which the tool measures the target
attribute. Second, the tool must be
reliable, which is the extent to which it is dependable across time. So,if I assess a candidate at time 1
then, assuming no change in performance, the score from an assessemtn at time 2
should be the same.
The competencies listed
above are precursors to performance.
They act in combinations to generate the workplace behaviours that are
accessible to observation and, therefore, assessment. This has implications for validity. How can I be sure that my observational
category is directly linked to the underpinning precursor. The more direct the relationship the
better the validity. It is for this
reason that ‘Situational Awareness’ is unlikely to have verifiable validity as
a marker. The relationship between
competencies and outputs is suggested in the diagram below
Competence Marker
Analysing
Planning Application
of Procedures
Organising
Validating Management
of Systems
Deciding
Communicating
Task
Management
Collaborating
Coping
Fig.
3.1 Relationship between a Competence and a Marker
A
competence framework is an attempt to describe all the skills and underpinning
knowledge required of an individual filling a role in an organisation. The idea is that the role is larger than
the specific ‘job’. Someone can be
accomplished in their ‘job’ but can still be lacking in overall
effectiveness. Historically,
specific job-related requirements would be described by a task analysis. Role-related requirements are covered,
in part, by the Job Description or Terms of Reference. However, Job Descriptions etc typically
only covered a minimum sub-set of what was required to be fully effective in
the role. The competence framework
attempts to bridge the gap by, first, more fully describing the role and then
by elaborating on the performance required in the role. A ‘behavioural marker’ is a description
of an element of competence that can be observed in the workplace. The relationship between the 2 concepts
is illustrated in Fig. 4.1
It was said earlier
that a competence model is not the same as an assessment framework. Assessment under EBT (and also the
earlier CRM requirement) is based on using observable behaviour as the evidence
on which to judge ‘competence’.
Thus, whereas a competence model is a broad description of expectations,
an assessment framework is a subset of competence that can be routinely
observed in the workplace. Below is
an example of an assessment framework:
The NOTECHS Behavioural Markers
Categories Elements
Example
Behaviours
Co-opERATION Team
building and Establishes
atmosphere for open
maintaining communication
and participation
Considering others Takes
condition of other crew
members into account
Supporting others Helps
other crew members in
demanding situation
Conflict solving Concentrates
on what is right
rather than who is right
LEADERSHIP AND Use
of authority and Takes
initiative to ensure
MANAGERIAL SKILLS assertiveness involvement
and task completion
Maintaining standards Intervenes
if task completion
deviates from standards
Planning and coordinating
Clearly
states intentions and goals
Workload management Allocates
enough time to
complete tasks
SITUATION System
awareness Monitors
and reports changes in
AWARENESS system’s
states
Environmental Collects
information about the
awareness environment
Anticipation Identifies
possible future problems
DECISION MAKING Problem
definition / Reviews
causal factors with other
diagnosis crew
members
Option generation States
alternative courses of
action
Asks other crew member
for
options
Risk assessment / Considers
and shares risks of
Option choice alternative
courses of action
Outcome
review Checks
outcome against plan
There are 3 common
methods used to construct assessment frameworks. In aviation, probably the earliest
framework was the NASA/University of Texas Crew Effectiveness Marker
system. This was developed by
looking at a range of fatal aircraft accidents and identifying what behaviours
contributed to crew failure. This
method could be called the ‘historical’ approach. The NOTECHS framework illustrated above
was developed by a committee of SMEs. The EASA framework is,
similarly, the output from a committee.
A third approach is to interview line pilots to get their views. By using structured interview techniques
and ‘card sort’ techniques it is possible to develop an ecologically valid
assessment framework. An example of
such an approach is given here:
1. COMMUNICATION
This dimension
relates to the way in which an individual communicates. It includes the extent to which the
speaker is clear, easy to understand and unambiguous.
Positive indicators
include:
The sharing
information and prior experience, actively seeking opinions, giving input not
just when requested but also proactively. Positive responses to inputs
(acknowledgement, repeating messages).
Negative indicators
include:
The failure to
listen or ignoring information.
Failure to explain decisions, actions, intentions. An unwillingness to
communicate (needs constant prompting or repeated requests). Failure to check misunderstood
communication (demonstrates hesitancy or uncertainty).
2. TASK MANAGEMENT
This dimension
relates to the conduct of the task. It includes the consistent and appropriate
use of checklists and procedures.
Making effective use of time. The avoidance of distraction and
maintaining the bigger picture of things happening around the aircraft.
Positive indicators
include:
A consistent, but
flexible, use of SOPs. Monitoring the use of checklists during
busy periods and the positive verification that tasks have been completed.
Maintaining an even tempo of work (no unnecessary haste or urgency). Recognising when to minimise
non-essential conversation.
Maintaining awareness of other aircraft, objects etc around the aircraft
both in the air and on the ground.
Actively developing mental pictures of what to expect during the next stage of
flight (e.g. through verbalisation of expected landmarks, events, system
changes etc). Anticipation and thinking ahead. Being aware of time
available/remaining, being aware of things around the aircraft (in the air and
on the ground), verifying geographical position.
Negative indicators
include:
Too strict an
adherence to or rigid application of SOPs.
Spending too much time out-of-the-loop on admin tasks, failure to update
on events when off-frequency. Rushing or delaying actions unnecessarily
3. TEAM BUILDING
This dimension
describes the extent to which effective working relationships are established
and maintained within the crew. It includes behaviour which binds the team and
which establishes a task focus.
Positive indicators
include:
Setting the
tone. Clarifying expectations and
standards of performance. The recognition that others have a part to play in
the crew process. Clear allocation
of tasks and responsibilities.
Briefing any excursions from SOPs. Fostering a sense of comfort and
inclusiveness in the group.
Negative indicators
include:
Avoiding
responsibility for actions, preventing full expression of views, intolerance,
failure to allow individuals to fulfil their role, interference in the work of
others.
4. PLANNING AND DECISION-MAKING
This dimension
relates to the way crews go about making decisions and agree upon appropriate
courses of action.
Positive indicators
include:
Sharing problems
and concerns, clarifying plans, identifying and discussing options and
alternatives. Evaluating risks,
pointing out errors of thinking, explaining decisions, seeking agreement on
courses of action.
Negative indicators
include:
Hasty reaction to
events, failure to consider alternatives, failure to discuss solutions,
over-reliance on other agencies.
5. INTERACTION STYLE
This dimension
relates to the way crew members interact with one another. It includes an individuals’ personal
style, their way of dealing with others and their approach to the task.
Positive indicators
include:
An optimistic,
positive approach to the job, friendly and approachable. Personable and easy to get on with. Patient with others, sensitive to their
needs and open to feedback.
Conscientious and dependable (can be relied upon to do the job).
Negative indicators
include:
Overbearing,
confrontational, aggressive. Prone
to getting upset when things go wrong.
Sometimes lacking in confidence, timid or given to inappropriate
behaviour (e.g. poor use of humour).
Lacking in skills and unstructured in their approach to the job. Too relaxed or too rigid application of
the rules. Inflexible.
It is important to
remember that assessment must be appropriate to an airline’s needs. The competences required of a business
jet crew, compared to a cargo crew or a wide body ULH passenger crew will
differ. Markers are abstract
constructs that attempt to capture an aspect of behaviour that is deemed
important to the operation.
The 2 examples of
marker frameworks illustrated above contain a broad statement of a behaviour - ‘Cooperation’,
for example - and then some elaboration in the form of example behaviours or
positive indicators. The
elaboration is an attempt to help assessors to better understand the scope of
the marker. The better the
assessors understand the boundaries of performance, the more standardised
assessments will be. However, the
natural tendency is for assessors to look for the elaborating examples
specifically rather than use them to guide their judgement. The trainee is then assessed based on
how many of the example behaviours are observed. This is actually codified in the EASA
VENN. This approach is wrong. The broad sweep of normal behaviour
makes it impossible to describe every way a specific competence element might
be demonstrated by an individual.
Assessors must use their expertise and judgement.
Assessors will look
at performance and extract behaviour elements. These can be physical actions, gestures
and other non-verbal signals or speech acts. These elements represent the evidence
upon which an assessor will evaluate performance. The marker framework must be capable of
capturing those element deemed most significant in terms of performance
outcomes but must do so in a way such that multiple assessors will make the
same categorisation of observed acts: as far as possible, assessors should
place the same event in the same category.
Therefore, marker schemes must be validated as part of the initial
development phase of the EBT. To do
this, first, collect some segments of crew performance on video. Next, SMEs who are fully conversant with
the marker examine the video and identify the significant behaviour
elements. These are then
categorised using the markers. Only
those elements that are unanimously agreed upon by the project team are retained
for phase 2. Next, small groups of
potential assessors observe the videos and, independently, identify behaviour
elements and assign to markers. The
results are then compared with the SME benchmark. Elements assigned to the same category
by both SMEs and trial subjects can be ignored. Where elements are assigned to different
categories then consideration must be given to redesigning the category, either
through changing the definition of the marker or by better elaboration through
examples, including specifying what is NOT included under the marker.
EASA has not
published any evidence to suggest that the 9 competencies have been validated.
If an airline wants
to develop its own assessment marker scheme, the following process will help:
Step 1. From the SMS, develop a model of current
and predicted operational hazards
Step 2. Construct a ‘look up table’ of crew
competence (Fig 4.1 as an example)
Step 3. Cross reference ‘look up table’ to
hazard model and verify coverage
Step 4. Identify critical skills to cope with
hazard model
Step 5. Identify elements of critical skill set
that are routinely observed during normal operations
Step 6. Construct
marker framework (category and descriptors)
Step 7. Cross
reference markers to ‘look up table’
A competence
framework is a broad description of a set of behaviours and underpinning
knowledge associated with successful performance. A behavioural marker is a subset of a
competence that can be observed and assessed in the workplace. Markers generally comprise a top-level
label and definition supported by example behaviours. The examples are intended to clarify and
better communicate the intent of the marker. Markers must be validated before use.
The introduction of a set of ‘competencies’ against which to assess
pilot performance has involved some debate around the issue of a specific
‘Knowledge’ competence. Although
not adopted by ICAO, it is included in the EASA framework for EBT. Its description and associated
‘observable behaviours’ are listed in the table below.
Application of knowledge
(KNO) |
|
Description: |
Demonstrates knowledge
and understanding of relevant information, operating instructions, aircraft
systems and the operating environment |
OB 0.1 |
Demonstrates practical
and applicable knowledge of limitations and systems and their interaction |
OB 0.2 |
Demonstrates the required
knowledge of published operating instructions |
OB 0.3 |
Demonstrates knowledge of
the physical environment, the air traffic environment and the operational
infrastructure (including air traffic routings, weather, airports) |
OB 0.4 |
Demonstrates appropriate
knowledge of applicable legislation. |
OB 0.5 |
Knows where to source
required information |
OB 0.6 |
Demonstrates a positive
interest in acquiring knowledge |
OB 0.7 |
Is able to apply
knowledge effectively |
The list of OBs is supposed to represent statements of observable
performance against which an individual’s competence can be assessed. OBs 0.1, 0.2 and 0.4 relate to the
simple recall of information: limitations, systems functioning, interactions
between systems, operating instructions and legislation. OB 0.3 relates to recall of information
particular to a destination. The
remaining 3 OBs do not flow from the top-level description and appear to be
after-thoughts. OB 0.5 points to a
need to have efficient search methods to find information while OB 0.6 reflects
an attitude towards study or maintaining currency.
The competence description positions ’knowledge’ as little more than
information contained in textual artefacts. However, OB 0.7 suggests that, to
be competent, you must be able to ‘apply’ knowledge ‘effectively’. But what does that mean?
It is a convention to classify knowledge as either declarative or
procedural. In essence, the former
describes what we can say and the latter describes what we can do. The declarative/procedural dichotomy is
not new and the first 4 OBs listed in the table are examples of what we would
consider declarative knowledge.
Although ‘apply knowledge’ nods towards the procedural side of things,
it is too vague a statement to be of any real use. What we are really interested in is how
do we ‘apply’ knowledge? To what
do we ‘apply’ it?
Ohlsson, in his book ‘Deep Learning’, prefers the term ‘process’ to
procedural. In his view,
declarative knowledge is more than whatever can be recalled from memory and
recited. Rather, it comprises
arrays of constraints that must be satisfied for any action or intervention in
the world to be considered legitimate or successful. His ‘process’ knowledge describes the
rule sets needed to control action.
This formulation starts to get closer to a useful description of
‘knowledge’ that could inform an approach to training and performance
measurement. Knowledge supports
action. Declarative knowledge is
used to establish the legitimacy of the current status of the task in relation
to our operational goal while process knowledge allows us to achieve congruence
between the actual and the desired states of the world. From this perspective, the ‘Knowledge’
competence is an inadequate formulation.
Advances in neuroscience, and in study of the visual system in
particular, have resulted in significant changes in our understanding of how
the brain works. Historically, the
study of cognition was predicated on information flowing from the outside - the
surrounding world - to the brain, being processed and then out again through
action driven by routines stored in memory. It now seems that this might not be the
case.
For a moment I want you to close your eyes. I want you to recall the scene in front
of you at the point at which you closed your eyes. Picture in your mind
everything that was in your field of view.
Take a few moments to recreate the scene. Then open your eyes. What do you see? In all probability your answer will be
‘I see what was there when I closed my eyes’. If there was a window in your field of
view you might notice that something has changed. The drift of clouds across the sky might
have changed the lighting conditions.
Essentially, though, the world is still how it was when you closed your
eyes. Or maybe not.
In your mind you constructed a view of the outside world and when you
opened your eyes you projected your internal. mental view onto the scene in
front of you. You then cross
checked to see if what you perceived matched your expectations. Neuroscience is increasing revealing
that, in terms of cognition, the flow is from the inside out and not the other
way around. Cognition is not simply
interrogating the sensory world and interpreting cues. Rather, it is a process of validating
expectations based on stored data and reconciling differences. So what does this mean for the idea of
‘knowledge’.
The physicist, Carlo Rovelli, explores the nature of reality from the
perspective of quantum physics in his book ‘Helgoland’. He makes the point that ‘knowledge’
describes more than just a ‘library’ of stored concepts, facts and rules. it is the very the process of
interacting with the world. In this
view, ‘knowledge’ is a dynamic process of detecting discrepancies between the
projected and the encountered worlds and the actions taken to reconcile
differences. In this view the world
is not a static ‘out there’, it is something that is created as part of achieving
our goals. Returning to Ohlsson, ‘declarative’ knowledge can now be seen as a
repertoire of conditions, acquired through training and experience, that allow
us to detect differences between our projected expectations and our actual
encounters. In effect, declarative
knowledge is error detection.
Process knowledge describes the ways we reconfigure the world to achieve
our goals.
There are 2 significant implications that flow from this discussion for
the idea of ‘competence’ as formulated in the ICAO/EASA model. First, the OBs that require simple
recall have nothing to do with ‘knowledge’. They relate to unstable artefacts that
describe arbitrary constraints. To
be considered ‘competent, I must, of course, perform within those
constraints. But I called them
unstable simply because technology and policies are not static. LOSA observations are littered with
supposed ‘errors’ that merely reflect the fact that the pilot being observed
was working with an out of date framework of policy and procedures. Pilots can still fly aircraft but they
cannot necessarily recall the latest rule changes. Whilst rules, procedures and limitations
are important, making them the focus of performance assessment places an undue
emphasis on the easily-captured but, probably, less important aspects of
performance.
The second implication is that ‘the ability to apply knowledge
effectively’ (OB 0.7) must be rendered meaningful. Technical, systems information is of use
not just because it allows a pilot to diagnose what has happened but
more because it supports the construction of expectations: it allows me to know
what I will see and, therefore, be able to tell if what is
happening is what is required. We
need to develop training that addresses how pilots create expectations during a
flight, how they detect ‘errors’ between the expected and actual status, how
they diagnose the causes of any discrepancy and then, finally, how to intervene
to restore equilibrium. This is
‘knowledge as action’. This is true
‘competence’.
Finally, if knowledge really is action then it suggests that any meaningful attempt to assess performance should concentrate more on the utility of outcomes in relation to operational goals. An ability to recite chapter and verse is evidence only of a reliable memory, not an indication of competence. Without a doubt, outcomes must be validated against prevailing constraints - policies and rules - but that is the final stage of performance, not its underpinning driver. This approach poses a serious challenge to concepts such as ‘situational awareness’ and ‘error’. It seems that what will call SA is more likely to be a reflection of the efficacy of our interventions in the world to restore equilibrium. Errors are not outcomes to be managed but, rather, are simply feedback signals. It is the status of the current task that must be managed in order to remove