Of the billions of photons that strike the retina in the eye, only 40 per second are processed by the brain. It seems that the brain feeds forward an expectation of what the eye ‘should’ be seeing, which is then compared with actual data received by the eye and the brain then attempts to resolve any discrepancies. By implication, the version of the world we hold in our heads is probabilistic, not a truth. Furthermore, no two people can possibly hold the same version of the world although their individual versions usually correlate sufficiently for them each to think that they are looking at the same scene. But this is only a part of the problem.

Imagine that you are at the controls of an aircraft. In your head you hold a version of the status of the aircraft and, also, a model of how it will respond to any inputs you make via the controls or through the automation. You have acquired this model through training and experience. You now make an input, the aircraft responds and becomes established in a new, stable state. If the hypothesis outlined above is correct, your interpretation of the new status of the aircraft is equally probabilistic, not a truth. Furthermore, the final status of the aircraft is just one of many possible end states that could have been achieved. The cause-and-effect relationship between your input and the outcome is no more than a hypothesis about how the world will respond. The robustness of your model of the world will influence the probability of achieving the desired outcome but it cannot guarantee it for several reasons.

First, aviation takes place in a dynamic environment and, as such, exhibits non-ergodicity. In simple terms this means that there is an inherent volatility in the world that guarantees that nothing ever happens the same way twice. Second, the world is complex. Again, in simple terms, complexity means that aviation involves multiple agents but with no single controlling authority. The component parts, therefore, have a habit of acting in unexpected ways. Finally, the world exhibits radical uncertainty, which is to say that things go wrong in ways we could never anticipate.

2.1 Piloting as Goal-directed Action

When we operate an aircraft, we follow a trajectory from flight initiation to aircraft shut down. That trajectory comprises a sequence of goals, each of which has a specific configuration that allows the task to be achieved within the constraints of the laws of aerodynamics. The pilot’s job is to configure the device in accordance with the requirements of the specific target goal, to manage transitions between goals and, occasionally, to adapt to unanticipated circumstances that might require goals to be modified or new goals created. This all takes place in a space defined by legal, commercial and aerodynamic constraints. To illustrate the point let us look at one small segment of a flight, the final approach. In very simple terms the task can described as:

A white circle with black text

Description automatically generated

Each of these goal states has a specific set of criteria that must be met for the goal to be achieved. In addition, there are specific processes that must be applied to achieve each goal and to transition between goals. Outputs from the aircraft’s Digital Flight Data Recorder (DFDR) allows us to explore the way pilots manage this notional trajectory.

This next graphic shows data from 301 Airbus pilots attempting to flare the aircraft. The DFDR output for the Pitch Angle parameter has been processed by an algorithm that looks at the statistical relationship between data points. The central dark blue band shows the most closely related 50% of data while the light blue bands show the outer 20% of the distribution (some data is lost because it fails the test of statistical significance). The bands show the distribution of data from 300 pilots who flew a normal approach. The red line is the trace of the 301st pilot whose performance is a statistical aberration. The trace of data from this flight differs significantly from the cohort of 300 peers.

4

3

2

1

A graph with a line graph

Description automatically generated with medium confidence

With data, we can render the goal state model tangible. We can follow the aircraft’s status on the final descent path (1), the transition to the flare (2), the aircraft established in the flare (3) and, finally, the transition to the ‘landed’ goal (4).

But this visualisation of normal data shows something more. First, we can see what happens next. The Pilot 301 is initially struggling to maintain the aircraft within the normal distribution but, at Point A on the display, the aircraft’s path diverges. For whatever reason, Pilot 301 was unable to maintain the aircraft within normal bounds. But that does not mean that the cohort of 300 were perfect. The picture shows us what happened next for the cohort, but it also shows us what didn’t happen. They didn’t exceed the bounds of the normal distribution. Why not?

The graphic is not simply an overlay of individual traces. It is a density plot of specific data points, each of which is a function of several other factors but captured as a value for a single parameter, in this case Pitch Angle. It can also be seen as a smart graphic. We can interrogate the array, and, for a specific point, we can trace several probable outcomes. For example, if we look at the area at Point B, the circle encloses several related data points which represent the Pitch angle of a cluster of aircraft at that moment in time. We can now trace those aircraft forward to Point C and see the distribution of probable outcomes that relate to the aircraft’s status at Point B. Controlling for other variables, such as wind vectors, turbulence and control inputs (all of which are captured in data and can be displayed), we can start to understand why Pilot 301 followed an erratic path while the cohort of 300 did not. From a pilot training perspective, we can start to understand how pilots can increase the probability of achieving the desired aircraft status.

2.2 Pilot ‘Knowledge’ is Retrospective

By describing a flight as a sequence of goal states we can begin to examine the knowledge base pilots draw on to constrain the range of probable outcomes. The criteria that apply to each goal comprises what is known as declarative knowledge while the rules we apply to manoeuvre between goals is known as process knowledge. The role of the aviation training system is to provide sufficient declarative and process knowledge to allow a pilot to operate an aircraft unsupervised. But once a pilot enters productive service, we need to build upon that foundation of retrospective knowledge and equip her with the skills needed to cope with a world that is governed by the laws of probability. If aviation is to maintain its enviable reputation for safety and airlines are to operate at maximum efficiency, the system we use to train pilots must equip them to function in the world I have just described. This means that the training system must transition from one that is retrospective to one that is prospective.

Retrospective learning deals with the past. It describes a set of known relationships that hold under the conditions applicable at the time of sampling. For example, if I was to ask what the capital of Germany is you would say Berlin. But the accuracy of that fact depends upon historical circumstances: Berlin has not always been the capital of Germany. Equally, the corpus of knowledge described by the EASA ATPL ground training curriculum represents a set of decisions about historic artefacts and relationships. There is no empirical evidence to suggest that the domain content prescribed by the syllabus possess any fundamental worth. Retrospective learning clearly has some value in that it provides what can be called underpinning knowledge. But such knowledge is only of use if it meets 2 criteria. First, it must be capable of being generalised. This means that the specific information taught must be capable of being recast as general principles that can be applied to novel situations. Second, it must be generative. The information presented must be capable of supporting the creation of new knowledge. Retrospective learning can only go so far in preparing pilots to cope with future challenges. Prospective learning, on the other hand, supports adaptive behaviour capable of coping with the unknown.

2.3 Controlling the Future

The concept of prospective learning is rooted in attempts to formulate models of evolutionary development: how does learning contribute to an organism’s chances of survival and, therefore, its opportunity to pass on its genes. It also has roots in Artificial Intelligence. So, how must Machine Learning (ML) algorithms be written if devices are to be truly ‘smart’ rather than simply being better than humans at a limited range of tasks. The concept has many overlaps with existing models of learning but does offer some useful insights.

There are 4 aspects of prospective learning that we need to consider. First, an entity must demonstrate continual learning, which is remembering those aspects of the past that are relevant to the future. In ML, new code over-writes old code and so ‘forgetting’ is absolute, even of the old code had some advantages. Equally, the very first use of the term ‘proactive’ was to describe how prior learning in humans interfered with new learning. The implication is that pilot training systems must be designed so that we encode declarative and process knowledge in such a way that it supports future action. Much aviation ground training seems to be little more than baggage. Unless academic knowledge can be complied (generalisable and generative) in such a way that it informs action it is of little value and will be quickly forgotten.

This idea leads on to the second requirement of prospective learning, which is causal estimation. Recognising that outcomes are probabilistic, not deterministic, causal estimation requires us to learn the structure of relations that support decisions that maximise the probability of the most desired outcome. We gain an understanding cause and effect, hopefully, because of training. But, as we gain experience, we elaborate our repertoire of goal state criteria and action rules. This, in turn, builds better causal estimation. Training systems need to draw attention to the cues in the environment that suggest flaws in our causal estimation, often resulting in situations that overwhelm the pilot’s sense-making abilities (think ‘Air France’ and ‘startle’).

Because of the complexity of normal life, we need ways to improve the efficiency of our search for relevant information. Known as constraints, these are things like heuristics, biases and our assumed knowledge of prior probability distributions (‘priors’) that we use to constrain the search space. Of course, heuristics and biases will be flawed. Equally, a prior, in this context, is simply a belief about what normally happens. The use of constraints need a critical thinking control loop that gives feedback on the efficacy of our search strategy.

Finally, and interestingly, prospective learning includes curiosity, which is action that informs future decisions, including future unmet situations. The EASA CBT Pilot Competence framework describes ‘Knowledge’ as a competence. While both clumsy and untenable (see Chapter 4), the spirit of the concept comes close to the idea of curiosity. From a prospective learning perspective, investment in curiosity requires effort that will offer no short-term reward but could result in a pay off at some future time. Time spent refreshing procedural and technical knowledge might have little subjective utility when set against any alternative uses of that time, but the curiosity concept suggests that an investment will support better coping strategies when faced with novel situations. Curiosity, importantly, also describes investing in learning around a topic, going beyond the defined curriculum, doing more than the minimum. Curiosity captures the concept of intrinsic motivation in learning theory. Students with intrinsic motivation - that is, they are learning a topic because they have an interest in it - tend to out perform students with extrinsic motivation. Extrinsic motivation describes students following a topic because they have to: they need to tick the box.

2.4 Error as Learning Feedback

At this point it might be worth saying something about error in learning. In simple terms, error reflects the degree of fit between task demands and the action taken to satisfy the goal requirements. Because there is some buffering in the system, rarely is there a perfect fit between inputs and outcomes. The system is constantly adjusting to variations and perturbations. Where action exceeds the system’s buffering capacity, the discrepancy is noted as an ‘error’. To illustrate the role of error in learning I want to use another analogy from Machine Learning. ML algorithms work on datasets that have been divided into 2 parts. One part is used to train the algorithm and the other part is then used to test if the algorithm works. Unfortunately, while the algorithm can deal with problems that are found within the distribution of the data used for training (In-distribution Learning), it will struggle or fail when presented with a problem that is Out-of-Distribution. Humans, on the other hand, can cope with Out-of-Distribution learning.

A graph of height above ground

Description automatically generated

Consider these 2 approaches. The first pilot (top) is encountering a crosswind component that falls within the normal distribution of values experienced by the cohort of 300 other pilots while the second pilot (bottom) encountered an ‘out-of-distribution’ crosswind component (note that the scales have been adjusted on the Drift presentation to accommodate the data):

Both pilots have made an ‘error’ but their responses are interesting. Whereas the first pilot seems to have over-corrected for drift on touchdown, the response of the second pilot appears to have been inadequate under the circumstances. Error, in this sense, is a feedback signal. Whereas the first pilot is possibly fine tuning an existing model, the latter will be elaborating on her model through exposure to a possibly novel situation. In a prospective learning context, error allows us to fine tune our causal explanation mechanism and support continual learning through elaboration of the mapping between stored knowledge and goal-directed action. Out-of-distribution learning, which is a characteristic of human, rather than machine, learning, is nonetheless brittle or fallible and, sometimes, the price to be paid for the learning opportunity is catastrophic. Effective pilot training needs to allow for out-of-distribution learning but in a safe context.

2.5 Conclusion

In this Introduction we have set out a challenge for the pilot training system. Drawing on some concepts from other domains, we have identified some criteria that a model of pilot learning must satisfy. At the core of the model is the need for pilots to cope with novelty. This has implications for the current trend towards competency-based training.

Although this Handbook has been developed to support Regulators and Operators wanting to introduce EBT/CBTA to their training system, It is hoped that this Introduction has offered some insights that will add value to the transition from legacy training to the revised models.

3 The Development of Training in Aviation

3.1 How Humans (and Animals) Learn

New-born infants to not emerge into the world completely unprepared to cope with what is coming their way. We are a product of evolution and certain behaviours seem to be transmitted genetically. For example, infants can orientate towards faces while still in the womb and new-borns respond to voices ahead of other sounds. Importantly, new-borns appear to have the basic building blocks of associative learning.

Action – behaviour - involves 2 processing systems: bottom-up and top-down. The bottom-up system responds to stimuli in a fast, automatic manner while the top-down system is slow and deliberative. Importantly, the top-down system modulates the bottom up system based on information stored in memory. Hommel, in his Theory of Event Coding, proposed that perceived and produced actions - what we see and what we do - are the same in that the processing of inputs and the initiation of action flows from the same neural paths. Watching others fires the same neurones as if I was doing that work myself. This is the essence of mimicry. The candidate mechanism for the system is the mirror neurone, first discovered in macaque monkeys and reported in the mid-1990s.

This fundamental framework underpins mimicry, the main way humans learn. Of course, as the human matures it acquires experiences stored in memory. When I act with others, pre-reflective (that is, prior to conscious evaluation), bottom up processing feeds forward signals. These are largely derived from information stored in memory about the task we are engaged in, who is supposed to be doing what etc. They create the world that I am expecting to see. Because we are social animals, other people are a part of that world I am looking at. Because of the way the mirror neurones work, their actions trigger the same responses in me as if I was doing what they are doing. It is through this process that can learn by watching.

3.2 Training Design

Mimicry underpins the way medieval guilds inducted apprentices into their crafts. Unfortunately, it is an inefficient model of learning. The roots of more structured approaches to the development of training can be traced back to the American psychologist Skinner. Working within the behaviourist tradition, Skinner elaborated the concept of operant conditioning, which claims that learning can be influenced by manipulating the learner’s environment. Frame-based programmed learning, the model that still underpins most computer-based training packages today, was the product of Skinner’s work.

The basic template for military pilot training was established in 1917 with the formulation of what was called the ‘Gosport System’, which relied heavily on mimicry. In the intervening 100 years the industry has experienced a series of catastrophic shocks, the solution to which has typically been additional technology. Mid-air collisions gave rise to TCAS; flying into the ground was cured with GPWS; ROPS is intended to stop aircraft going off the end of runways. What has not really changed is how we train pilots and yet the role of the pilot has been transformed from that of controlling a device to managing a flight path.

In the 1950s, Benjamin Bloom published his Taxonomy of Intellectual Behaviours (1956), producing the first hierarchical model of different types of learning. Bloom identified 3 learning domains: Cognitive, Affective and Psychomotor. The cognitive domain referred to the processes associated with mental skills, the affective domain refers to attitudes and the psychomotor domain encompasses physical skills. The legacy of Bloom’s work is the tripartite Knowledge, Skill, Attitude (K/S/A) classification scheme still used in training design today. Two former students of Bloom, Anderson and Krathwohl, later developed the taxonomy by matching types of knowledge to types of activities.

In 1962 Robert Mager established the concept of Learning Objectives as the key building blocks of training design. Mager proposed that training should be based on a clear statement of observable behaviour. It is important to remember that the behaviourist tradition worked in terms of observed outputs from mental activity, the mental aspect remains hidden from view. So, a behavioural objective is a statement of what a student should be able to do as a result of some mental process being accurately executed. Mager refined the concept by adding the degree of accuracy required to be certain that the performance was reliable. He also proposed that the conditions under which the performance was to be enacted should be made clear. Mager’s work underpins the ‘Performance, Standard and Condition’ structure of training objectives.

The contributions to the development of structured training have so far concentrated on the identification and definition of training goals. In 1965 Robert Gagne published his ‘Stages of Instruction’, laying down the framework for the delivery of training. Gagne identified a set of conditions to be met and some activities to be conducted that, combined, would lead to effective learning. Gagne’s work shaped the way lessons are delivered in classrooms today. Gagne was also one of the first to propose the application of systems concepts to education.

This era was the time of huge investment in complex technological projects such as nuclear power and manned spaceflight. In order to successfully accomplish these projects, man and technology had to be enabled to work effectively together. Many of the tools of modern management, such as project management and structured decision-making, were stimulated by the demands of these complex projects.

The first coherent model of structured training was probably that of Robert Glaser, published in 1962 but it was the USAF ‘5 Step Approach’, published that same year, that brought the components of modern instructional system design together for the first time. The 5 steps are:

• Analyse system requirements;

• Define education and training requirements;

• Develop objectives and tests;

• Plan, develop and validate training;

• Conduct and evaluate training.

3.3 ADDIE

There have been various iterations of the basic 5 Step model and the work of Florida State University, which published its ADDIE model in 1975, is representative of the final stage in structured training systems development. ADDIE stands for analysis, design, development, implementation and evaluation. Whereas the 5 Step model was essentially linear in its conceptualisation, the ADDIE model reflects a cyclical approach to training design in that the output from training is constantly evaluated against operational need and changes made as required.

Labelled the ‘Systems Approach to Training’ (SAT), the model was widely adopted by the US military and by many NATO countries. The Systems Approach to Flying Training (SAFT) was used to reconfigure ab-inito pilot training in the UK RAF in the early 1970s.

3.4 The Competence Concept

On 4 October 1957 the Soviet Union launched Sputnik, the first artificial satellite to orbit the earth. In one creation myth, this humiliation for the United States resulted in the ‘competence’ movement. Recognising that simple course graduation was no guarantee of proficiency, instead it was decised that there needed to be a framework for demonstrating employability. In 1982, Richard Boyzatis published ‘The Competent Manager: A model for effective performance’ which is also credited with starting the competence movement Reflecting changes in the workplace and society, with increased job insecurity and worker mobility, the competence theorists attempted to identify core skills, or competences, that were suitably generic and transferable between workplaces. For example, a steelworker might possess a set of competences that would allow that person to find work in a different sector of industry but only require minimal retraining. Competences were reflected in vocational training courses in schools and higher education. In the UK, competence frameworks were developed for commercial pilots and cabin crew by the industry Lead Body although the associated National Vocational Qualification for pilots lapsed because of a lack of uptake.

A competence framework comprises descriptions of desired workplace behaviour arranged in clusters. Communications skills, people management and team skills are the 3 most frequent competence clusters according to a 2007 survey. The behaviours are usually tagged with specific underpinning knowledge required of the individual to support the demonstration of the desired behaviour.

3.5 ISD in Civil Aviation

The recognition that the existing framework for training and checking mandated by the FAA in the USA might not be guaranteeing the competence of commercial pilots gave rise to the introduction of the Advanced Qualification Program (AQP) in 1990. Pilot proficiency is typically assessed in terms of manoeuvres repeated at prescribed time intervals. So, a set repertoire of manoeuvres must be flown to pre-determined levels of accuracy and must be demonstrated at set intervals. However, this ‘one size fits all’ approach to maintaining a competent workforce was increasingly being considered inefficient. For a start, pilots acquire proficiency at different rates and skills decay at different rates. Airlines operate into very different environments with very different equipment and yet all have to meet the same training requirements. The guiding principles of AQP are that each individual operator must determine the skill set required of its pilots and that training and checking must be based on the needs of individual pilots within the operational context. The AQP regulations allow for the voluntary adoption of the program; operators can continue to follow the manoeuvre/ interval-based, or ‘legacy’, model. The AQP concept has been broadened to include cabin crew and dispatcher training.

Aware of developments in the US, the first draft of JAR OPS 1 included line entries referring to AQP. The promulgation of JAR OPS 1.978 - the Alternative Training and Qualification Programme - in 2006 provided a framework for JAA – later EASA - carriers to adopt a training and checking regime based on line operations. The regulation is built on the experience of AQP but incorporates developments in flight data capture and analysis, safety management and auditing that have occurred in the intervening period since AQP was first introduced.

We have just briefly sketched out the origins of structured models of training analysis and design. From this it can be seen that ATQP is simply the application of Instructional Systems Design (ISD) to commercial pilot training and testing. In order to understand the significance of ATQP is necessary to, first, review the existing framework for training and testing. In broad terms, commercial pilot training comprises 4 phases: initial license training; type conversion training; operator’s conversion training; recurrent training. In addition, it is possible to distinguish 2 discrete aspects of the system. The first is to train to a set standard. The second is to test competence, both at the end of initial training and, again, at set intervals during employment. Traditionally, the broad structure and content of the training course and the criteria for success have been contained in regulations promulgated by national authorities. The role of the airline training department is to configure training in such a way that it demonstrates compliance with regulatory requirements.

This model of training is pragmatic in the sense that it is rooted in generations of operational experience and is successful in that aviation remains a highly reliable, yet hazardous, industry. However, in a competitive marketplace, training departments compete for resources with the rest of the airline. As such, there is often little spare capacity to accommodate changes in the operational world, such as the seasonal characteristics of operations or changes in technology. The ‘compliance’ model of training delivers a product that meets regulatory demands but is not necessarily mapped onto the needs of the specific airline.

3.6 Competencies in Civil Aviation

Recognising that the aviation industry faced a potential recruitment shortfall across all sectors, IATA, through the ITQI, and ICAO, through NGAP, have both been looking at introducing structured training models based on a competence approach. Both are looking at a broader audience that just pilots but the key difference, initially, was that ICAO were concerned that any competence framework should cover initial selection and training as well as in-service development and advancement.

The IATA project was the first to bear fruit. In order to break away from the historical ‘set manoeuvre’ model, a large scale analysis of various data sources was undertaken, resulting in the EBT Data Report. This provided ‘evidence’ of what training topics were more appropriate for modern generation aircraft. The goal of training was recast: no longer did pilots have to demonstrate accomplishment in manoeuvres, they had to demonstrate ‘competence’ in the control of the aircraft, including the management of the flight path under normal and non-normal circumstance.

One of the first challenges was to define a ‘competence’? Was it generic or specific? One school of thought suggests that the manipulation of numbers and words and the ability to learn were fundamental ‘competences’ and all performance flows from these basic abilities. Another line of thought, discussed earlier, took the view that competencies were arbitrary clusters of skill and knowledge that were applicable to a specific work context. In effect, they are whatever you want them to be as long as they make your workforce effective.

Another problem is the interpretation of ‘evidence’ in. The current usage of the term ‘evidence-based’ can be traced to a 1972 paper by Archie Cochrane that questioned the effectiveness of medical treatments. Given the increasing costs of delivering healthcare and the range of treatments available, how do you decide what works best? The combinations of patient, condition and treatment should be evaluated using Randomised Controlled Trials (RCT) as the gold standard of evidence. So, the ‘evidence’ was what could be proven to work best. Other fields, such as social policy, have adopted the concept but the underlying idea remains the same. In aviation, a direct analogy would be an investigation of pilot experience level, skill to be trained and training device employed. The closest we have come to true ‘EBT’ in aviation are attempts to assess training transfer in flight simulators. Prof Inez de Florio-Hensen, at Kassel University, argues that, in education, EBT is, in any case, an unattainable goal. The range of variables – student, teacher, subject matter, training situation – is simply too great to make RCTs meaningful.

CBTA seems to be the application of ISD to initial training in all specialisations while EBT refers to airline recurrent training.

A black screen with white text

Description automatically generated

3.7 A Career in Aviation

We can use the trajectory of a pilot’s career (or any other employee) as an organising framework to bring some of these concepts together. In the diagram above, the trajectory starts with initial training for the award of a license. The input standard is usually a novice with no skill or knowledge and the output standard is someone deemed fit to hold a license. But a license simply allows an individual to operate an aircraft appropriate to their qualification. It does not guarantee any level of expertise beyond a baseline level of safety.

As that individual gains in experience they may want to start looking for a job. First, they need an aircraft type rating. This involves applying their prior learning to the specific instance of the new aircraft type. But they also need to convince an employer that they are a good fit for the company. This is where a ‘competence’ model comes in handy. The employer knows that the applicant is legal if they possess a license and the required ratings. The competence model describes the additional attributes needed of the pilot to be successfully employed by that airline. At the recruitment stage the employer is simply looking for evidence of behaviour that maps onto the competence model. In short, what does the pilot bring with them that can be exploited and developed in their new role?

As the pilot progresses, 2 things need to happen. From a legal perpective the airline needs to show that the pilot has maintained the level of proficiency required to hold their license. From the airline’s perspective, the pilot needs to show that they are capable of coping with the operational demands likely to be encountered. The airline needs to have a competence model that captures those demands and sampling tool to evaulate the individual.

The pilot lifecycle approach shows how we need different tools at different stages.

3.8 Is EBT a Flawed Concept?

Aviation Authorities around the world have relied on the periodic accomplishment of a set of manoeuvres as proof that a pilot is competent. Supporters of EBT argue that the event set has failed to keep track with changes in technology and so a new way of assessing is needed. It shoud be remembered that the State has a legal duty to ensure the safety of its aviation system. The airline wants to be sure that its pilots can do the job. The State and the Airline have different goals but, historically, have used the same performance measure.

EBT enthusiasts confuse product and process. Product is the observable output while process is how that output is achieved. The EBT argument is with the ‘product’ – an anachronistic set of manoeuvres – but ignores process. If a pilot can cope with the historic manoeuvres then she can cope with any in-flight problem. At this point we need to go full circle back to the beginnings of structured training design and ask what do we really mean by skills and knowledge. Stellan Ohlsson, in his book ‘Deep Learning’, proposes that expertise relies on an individual being able to establish a set of constraints that must be satisfied for a particular condition to be considered true. Through training and experience we develop an increasingly fine-grained set of constraints for an increasingly varied repertoire of situations. Process knowledge comprises the actions and rules sets we deploy to satisfy the situational constraints. Interestingly, he proposes that errors are, in fact, feedback signals that tell us that a constraint has not been satisfied. In effect, either we have applied inadequate constraints (faulty knowledge) or implemented an incorrect action.

The implications of Ohlsson’s ideas are significant and far-reaching. For a start, it is not enough for a pilot to demonstrate the control of the aircraft to be considered competent: they might have just been lucky on that day. We need to explore how pilots think about control.

Competence, then, is thinking made manifest. In the rest of this manual we will look at how to achieve that goal.

3.9 Conclusion

We can see 3 broad trends that have emerged since that first formalized flight training system. Training is expensive and so it has come under increasing pressure to be cost effective. In order to achieve this goal, systematic approaches to analysis and design have been implemented. Finally, steps have been taken to map training onto operational need. These last 2 themes form the structure of the rest of this book.

4 Implementing Instructional Systems Design

4.1 Introduction

The starting point for both philosophies (AQP/ATQP and EBT) is a framework that describes the range of performance we expect of pilots under all normal and non-normal situations. AQP/ATQP follows the conventional ISD model and starts with a Task Analysis (TA). EBT, on the other hand, starts with a Competence Model (CM). The processes we use to develop both are similar. The real difference is how we describe performance, As we will see later the eventual output from a TA is a syllabus of instruction. A CM is more akin to a job specification or a set of Terms of Reference. In both cases the final output needs to be a comprehensive description of the performance expected such that we can be confident that we are able to verify that pilots are competent.

The main difference between the ISD-derived models and the competence approach is that the former has a well-established process with well-understood methods that can be applied to the task of develoing training. The competence approach, on the otherhand, lacks any rigorous processes, in part because it was never intended to be a design methodology.

This chapter will deal with the ISD methodology while, in the next chapter we will look at developing competence models.

2.2 The ADDIE Process Explained

The ISD model, of which ADDIE is just one version, is a closed loop that starts with analysis of requirements and comes back to the starting point via a number intermediate of stages. The concept is illustrated below. Because ADDIE is refered to in the official documentation, we will discuss it within the broader ISD context.

Step 1 is the Analysis phase. Here, we look at the task required to be mastered by the trainee. We look a the characteristics of the trainees, constraints on delivery, projet timescales etc. Step 2 is the Design phase. Here we define the output standard. We do this now because this will shape how much time is needed to train and what methods will be needed to train and test. An output from Step 2 is the syllabus. The syllabus is usually framed as a set of learning goals or objectives that must be achieved by the trainee. We will look at objectives in more detail later. We also look at testing methods unde the Design umbrella.

Having described the syllabus, the next step is to do a Training Needs Analysis (TNA). The TNA identifies the gap between the Output Standard and skills mix of the entry level students. For example, an ab initio pilot just graduating from a flight school will have a bigger gap between his current status and that required at the end of a type conversion. However, an experienced pilot converting from a different aircraft type will have a narrower gap between entry level and output standard. The TNA will inform the next step, which is curriculum design.

Analysis

Evaluate

Implement

Develop

Design

A diagram of a process

Description automatically generated

Although useage does vary, I will use the syllabus to describe the course training objectives and the curriculum to describe the allocation of objectives to training events. Curriculum Development -Step 3- is where decisions are made about method of delivery, training media to be used, time to be allocated, sequencing of events and so on. It is where we do the heavy lifting of making a course. Then we have the Implementation phase. Implementation covers initial roll out, prototyping, fine-tuning and bedding down the production version of the course. It also includes trainer training and standardization.

Once the course is up and running, we need to think about Evaluation. We look at this in more detail later in the book but. Essentially, evaluation asks ‘does the course work?’ If not, then we need a process for Modification.

4.2 Developing the Job Task Analysis

The purpose of a JTA is to establish the baseline criteria for each job. The JTA can be likened to a product specification for the conduct of duties and, as such, has some similarity with a competence framework. The difference being that the TA is a more fine-grained description of the actions associated with completing the task. The TA will ultimately determine the instructional goals and objectives, specify the type of knowledge required for the job, assist in determining instructional activities and aid in constructing performance assessments. It will also serve as the basis for auditing the company’s training

The JTA, then, is exactly what the name implies; it is a list of actions associated with the task for which an individual is responsible, stated in observable objective statements. That is, each task should begin with a verb that describes the nature of the activity associated with that task. For example, simply listing items to be checked by a pilot is not a JTA. Instead, responsibilities should be listed in terms of the observable action associated with that responsibility. For example:

Poor task list:

1. engine oil

2. engine temperature

Acceptable task list:

1. check engine oil quantity

2. observe engine temperature

The first step in developing the JTA is to compile a task list. There are several ways to create a task list. One way is direct observation of the task. The analyst observes a representative sample of the workforce and notes down job behaviours as they occur. While this method works well because it takes place in a naturalistic environment, it often does not allow the observer to catch all aspects of the job. There is a view that the observer should be familiar with the job, while others feel that a novice is best. The familiar observer may be able to label tasks correctly and more accurately, but may bring their own bias to the task. While a novice observer may not know the reasons for a particular task, they are clear of procedural bias and assumptions.

Another method for creating a task list is interviewing a Subject Matter Expert (SME). It is best to include more than one SME for the interview process in order to cover all situations and perspectives. The interviewer will typically ask the SME to talk through the job out loud. The interviewer will want to ask questions such as:

• What specific duties must an employee perform?

• What units of work must be completed?

• What handbooks must be consulted?

Once the task listing is complete, the next stage is to review each task to see if there is a need for further decomposition. The task decomposition should not be overly detailed such that the listing becomes cumbersome. Equally, it should not be so vague such that it does not provide an adequate description of the company’s requirements. There is a view that cut-and-pasting a JTA saves time. After all, an A-320 is an A-320 no matter what the logo on the tail says. In fact a JTA is specific to each company: no 2 airlines fly the same.

The JTA is rooted in behaviourist psychology and, therefore, typically examines just the observable behaviours needed to perform a job. However, some tasks require non-observable behaviours, such as evaluative thought processes associated with process control and decision-making skills. These types of behaviours can still be represented in the task analysis but require an additional cognitive task analysis. Table 2.1 briefly lists some types of task analysis and when to use them. A common mistake is trying to force fit a job into a task analysis for which it is ill suited.

Job/Performance Used for procedural skills

Learning/Needs Only identifies what must be taught; secondary analysis

Cognitive Examines how people think about situations

Content/Subject Matter Breaking down large amounts of information into teachable units

Table 2.1. Types of Task Analysis

There is no ‘ideal’ template for laying out a JTA. To a large degree it depends upon the depth of analysis and intended use of the JTA within an airline. In an integrated Safety, Quality and Training system, the content of the JTA will be referenced to training records, crew scheduling, auditing and event reporting. Therefore, it makes sense for the JTA to be built in a database product. Here is one example of a JTA:

Takeoff Operations:

Normal Takeoff Procedure

Release Brakes

Align airplane on runway centreline

Transfer control of airplane to First Officer, if required

Call out": "YOU HAVE THE AIRCRAFT," if required

Call out": "I HAVE THE AIRCRAFT," if required

Maintain directional control with Rudder Pedal Steering and Rudder

GUARD Nose Wheel Steering until both engines stabilised and symmetrical

Advance Thrust Levers to approximately 50% N1

Ensure engines stabilised and symmetrical

Advance Thrust levers to FLEX or TOGA detent as required

Apply slight or full-forward side stick as required

Call out: "FLEX" or "TOGA" as required

Verify "FLEX" or "TOGA", SRS, and RWY (if applicable) on FMA

Compare LP Rotor Speed (N1) to N1 rating limit on E/WD

Call out: "FLEX SET," or "TOGA SET" prior to 80 knots

Assume/Maintain Control of Thrust Levers

Call out: "80 KNOTS"

Acknowledge 80 knot call out: "CHECKED"

Remove forward side stick pressure at 80 knots to be neutral by 100 knots

Maintain wings level attitude with side stick

Monitor engine instruments

Call out: "V1" at V1 -5 knots

Remove hand from thrust levers

Call out: "ROTATE" at VR

At Vr, Rotate smoothly to SRS commanded attitude (or 110 degrees if no SRS)

Call deviations from normal flight instrument indications

Call out: "POSITIVE RATE" (when a positive rate of climb is indicated)

Ensure positive Rate of Climb

Call out: "GEAR UP"

Here is another example from a different airline for the same aircraft at the same stage of flight:

1. Demonstrate the ability to perform a normal takeoff and initial climb to flap retraction altitude in accordance with AOM and the company FOM.

2. Apply the appropriate CRM skills when performing a takeoff.

The JTA underpins the training development process and is fundamental to the continued safe delivery of airline training. The JTA supports the Safety Case (SC) (see Chapter 11) and also drives curriculum development. It is the most time-consuming component of the ISD process. However, unless it is done properly, it can also be the Achilles heel of the Training Department. Time spent getting the TA correct will show a payback later in the implementation phase

We can develop a JTA, then, by inspecting documents, by observing performance and by interviewing line pilots. We could also look at safety reports and LOSA for evidence of poor performance that can then be used to elaborate on the original analysis.

Table 2.1 shows a JTA that was developed for an Airbus operator. First, a number of management pilots who were also trainers decided on a meaningful structure to describe the performance of a pilot (Units of Work). Next, the company Operations Manual, the aircraft FCOMs and the company Flight Crew Training Manual were cross- referenced to the task structure. Rather than describe the job, references were used for the sake of efficiency. Each reference relates to a piece of documentation that describes the task. Finally, experienced training captains were asked to review the document and identify any gaps. The process looked at normal operations.

Unit of Work	Baseline (SOP, FCTM)	Probable Contingencies
Pre-Duty	OMA 5.2.1-5.2.3 (recency req.), 6.1 (medical fitness), 6.2 (medical precautions), 7 (FTLs), 8.1.12 (documents to be carried), 14.1.1 (documents to be carried by crew) , 14.1.2 (uniform) , 14.5 (crew bags)
Dispatch	FCOM: PRO-NOR-SOP-02 P1/6
Aircraft Pre-flight	FCTM NO-020 P4/12-11/12 e-Library – Loadsheet ACARS PERF setup FCOM: PRO-NOR-SOP-03 P1/2 (safety exterior inspection), PRO-NOR-SOP-04 (power-up & before walkaround), PRO-NOR-SOP-05 (exterior inspection), PRO-NOR-SOP-06 (cockpit preparation), PRO-NOR-SRP-01-10 (cockpit preparation)
Start/Pushback/towing	FCTM NO-030 (eng start) FCOM PRO-NOR-SOP 01 P9/20 (pushback & towing), PRO-NOR-SOP-07 (before pushpack or start), PRO-NOR-SOP-08 (engine start), PRO-NOR-SOP-09 (after start), PRO-NOR-SRP-01-10 (before pushback or start), OMA 8.3.20 (pre-taxi)
Taxi	FCTM NO-040 PRO-NOR-SOP-10 (Taxi), PRO-NOR-SRP-01-20 (taxi), OMA 8.3.21 (taxi)
Take off/Rotation	FCTM NO-050 P1-8/14 FCOM: PRO-NOR-SOP-11 (entering the runway), PRO-NOR-SOP-12 (takeoff), PRO-NOR-SRP-01-30 (takeoff), OMA 8.3.22 (takeoff)
Initial Climb (to CLB thrust)	FCTM NO-050 P8-13/14, FCOM PRO-NOR-SOP-13 (after takeoff), OMA 8.3.22.1 (climb graph)
Departure (SID)	OMA 8.3.23 (Departure and climb)
Climb to cruise level	FCTM NO-060 FCOM PRO-NOR-SOP-14 (climb), PRO-NOR-SRP-01-40, PRO-NOR-SRP-01-50
Cruise	FCTM NO-070 FCOM PRO-NOR-SOP-15 (cruise)
Descent preparation	FCTM NO-080 FCOM: PRO-NOR-SOP-01 P15/20 (landing perf), PRO-NOR-SOP-16 (decent preparation), PRO-NOR-SRP-01-50
Descent	FCTM NO-090 FCOM: PRO-NOR-SOP-01 P15/20 (descent profile), PRO-NOR-SOP-17 (descent initiation/monitoring/adjustment), PRO-NOR-SRP-01-60, OMA 8.3.25 (descent)
Approach (STAR/Holding)	FCTM: NO-100 (Holding), NO-110 P1-4/10 (Initial App), PRO-NOR-SOP-18, PRO-NOR-SRP-01-70 P1-3/32, OMA 8.3.25.3 (holding speed), 8.3.26 (approach)
Final Approach	FCTM: NO-110 P4-9/10 (final App), NO-120 (ILS), NO-130 (Non precision app) FCOM: PRO-NOR-SOP-01 P 15/20 (stabilized approach), PRO-NOR-SRP-01-70 P3-10/32, OMA 8.3.26.1 (stabilized approach) 8.3.26.2 (approach ban), 8.3.26.3 (ILS)	FCTM NO-160 (LVO app), FCOM PRO-NOR-SRP-01-70 P11-23/32
Flare and Landing	FCTM NO-170 e-Library – landing tips & Final Approach and Landing Technique FCOM PRO-NOR-SOP-19, OMA 8.3.27 (landing)
Go Around/Rejected LDG	FCTM NO-180 e-Library – Go-Around FCOM: PRO-NOR-SOP-01 P 16/20 (mandatory missed approach), PRO-NOR-SOP-20, PRO-NOR-SRP-01-80, OMA 8.3.28 (go around)
Rollout	FCOM: PRO-NOR-SOP-01 P17/20 (touchdown and rollout), PRO-NOR-SOP-21 (after landing)
Taxi in and clean up	FCTM NO-190
Shutdown	FCOM PRO-NOR-SOP-22 (parking)
Securing	FCOM PRO-NOR-SOP-23 (securing the aircraft)

Table 2.1 Airbus JTA

For each element of competence, Subject Matter Experts (SMEs) were asked to identify the range of contingencies that might apply in order to verify that coverage was complete. We also looked at differences between roles (PM/PF) and advancement (Command).

4.3 Normal v Non-normal/Emergency

Dealing with non-normal or emergency procedures requires a different approach. Whereas normal operations follows a distinctive, repetitious pattern (generally speaking), non-normal/emergency situations tend to require a safety template to be overlain on the situation which is then used to select an appropriate action. Competence in this sense might be described in generic terms:

• Establish/sustain control

• Evaluate systems/flight path status

• Identify problem

• Identify appropriate checklist(s)

• Execute checklists

• Validate system response

• Choose next course of action

• Monitor status

4.4 The Training Needs Analysis (TNA)

The TA is a job specification. It describes the actions required of an operator if a task is to be completed successfully. The goal of training is to develop the skills of an individual so that they can complete the tasks in the TA without supervision and to an acceptable standard. The first stage in developing a course is to scrutinise the TA in order to identify those aspects of performance that will need training.

Training Need will be driven by the entry level of the trainees. An airline Initial Type Conversion designed for ab inito cadets recently graduated from flight school will require greater depth and content that an Operator’s Conversion course designed for previously qualified pilots recruited from another airline. In the context of airline recurrent training, it is unlikely that we will be developing modules with content tat is completely novel. In most cases, ‘training’ will be updating, adapting or linking to existing knowledge. Decisions about the depth of knowledge and the time allocated to training will be influenced by this analysis of the entry level.

For each task we need to identify the skilled performance associated with the task as well as any underpinning knowledge essential for successful task completion. Underpinning knowledge will include an explanation of why the task is important, any theoretical knowledge associated with completing the task, probably risks attached to the task and any alternative strategies for task completion.

4.5 Describing the Output Standard

Having clarified the goals of the course, we now need to create the syllabus by writing the Training Objectives (TOs). A TO typically comprises 3 parts:

A statement of performance.

A statement of the conditions under which the performance is to be demonstrated.

A statement of the standard to be achieved for the performance to be considered acceptable.

The performance statement is worded in terms of observable actions using verbs. We want to be able to witness the performance in order to assess the level of achievement. Therefore, objectives describe the external manifestation of competence. Because of the variability encountered during normal line operations, any specific skill might be performed under a range of conditions. The condition statement describes the range of contingencies under which a trainee will be expected to perform in training so that we can be assured that they will be able to cope with line operations. The standards statement defines any bounds of acceptable performance we want to attach to each objective. A standard might be a tolerance within which the skill is to be performed or it might be a procedural limitation. An example of the TO might be:

Performance: Land the aircraft

Conditions: Within a range of crosswinds, at night, within a range of runway surface conditions

Standards: Within touchdown zone, within speed and ROD constraints.

These items drawn from the EASA HP&L syllabus illustrate weaknesses in objective formulation:

a) List the factors determining pulse rate.

b) Question the established expression ’safety first’ in a commercial entity

c) Describe the personality, attitude and behaviours of an ideal crew member

Item a) is a valid objective. Item b) starts with a verb but the rest of the performance statement makes no sense. Is the student supposed to question a safety policy in order to to elaborate on the entity’s SMS? Is the aim to question the veracity of the statement in the first place? Item c) collapses 3 possible objectives into one and is a good illustration of ‘signposting’. Rather than require students to declare any knowledge in relation to the 2 key concepts - personality and attitude - it suggests that there is a desired ‘correct answer’ which, in any case, could only be an opinion given the uncertain status of personality traits in relation to pilot performance.

Writing TOs is as much an art as a science. Drafting acceptable TOs does require skill so it might be worth looking at some examples to illustrate the challenge. Appendix to Annex I to ED Decision 2018/001/R offers this set of TOs in relation Mental Maths:

100 09 00 00	MENTAL MATHS
Show, in non-calculator tests and/or exercises, the ability in a time-efficient manner to make correct mental calculation approximations:
(1)	To convert between volumes and masses of fuel using range of units.
(2)	For applied questions relating to time, distance and speed.
(3)	For applied questions relating to rate of climb or rate of descent, distance and time.
(4)	To add or subtract time, distance, and fuel mass in practical situations.
(5)	To calculate fuel burn given time and fuel flow in practical situations.
(6)	To calculate time available (for decision-making) given extra fuel.
(7)	To determine top of descent using a given simple method.
(8)	To determine values that vary by a percentage, e.g. dry-to-wet landing distance and fuel burn.
(9)	To estimate heights at distances on a 3-degree glideslope.
(10)	To estimate headings using the 1-in-60 rule.
(11)	To estimate headwind and crosswind components given wind speed and direction and runway in use

This example is clumsy and can be reframed thus:

Performance	Condition (Common to all LOs: In an examination comprising x questions. Without the use of aids to calculation)
1. Apply the 4 Rules of Number	Using Whole Numbers, Decimals, Percentages.
2. Convert between units of measurement	Mass, Volume, Distance, Time. Given conversion factors
3. Apply Rules of Thumb	1 in 60 rule, Rule of Thirds (headwind and crosswind components). Lateral navigation (track, heading) Vertical navigation (height)

We saw that the purpose of the TNA is to establish what needs to be taught given the entry level of the students. Although students will be expected to demonstrate the performance described in LO 1, we can assume that they are already numerate and so no formal training will be provided. Equally, we can assume that our students understand the various terms such as ‘mass’ volume, ‘decimal’, ‘percentage’ and so on so we do not need to offer training.

For LO3, however, some of the Rules of Thumb might not be known to the class. In this case we need to elaborate. So, in this case we can consider ‘Apply Rules of Thumb’ to be the Terminal Objective (TO) and we would create some Enabling Objectives (EO) that allow the student to achieve the TO.

Performance

Condition

3.1 State the 1 in 60 Rule

in relation to:

Lateral navigation

Vertical Navigation

3.2 State the Rule of Thirds

In relation to:

Heading

Groundspeed

In this example, the 3 TOs all describe what would be called a ‘skill’, in this case the mental manipulation of values. TOs are traditionally divided into 3 categories: skills, knowledge and attitudes. Skills are what you ‘do’ while ‘knowledge’ is what you know. There are no ‘attitude’ objectives contained in the table. We might decide that an attitude objective is appropriate in this case. So, we might consider these as candidates:

a) State the reasons why a ‘gross error check’ on outputs is needed when entering data into aircraft systems

b) List the reasons for conducting a ‘dead reckoning’ cross check on aircraft performance

Attitudinal objectives are recognised as difficult to define and almost impossible to test.

4.6 Conclusion

In this chapter we have looked at ISD as a model of training design and have differentiated between ISD as a process for designing inputs to bring about behaviour change and the use of ‘competencies’ to describe workplace performance. The remaining steps in ISD wil be covered in the following chapters.

5 Developing Competence Frameworks and Markers.

5.1 Introduction

In Chapter 1 we look at various models of training design. A key difference between classical approaches to training design and the competence approach is that the former addresses a specific job or task whereas the latter supposedly is designed to develop a ‘generic’ set of behaviours that can be transferred between different jobs. For example, there might be a range of different workplaces that all require an ability to make ‘decisions’. If I have a fundamental ‘decision-making’ toolkit then it doesn’t matter if I am an office clerk or an astronaut, I can still have a go at making a decision. The complexity of the decision to be made and the consequences of failure may differ but the process remains the same.

Whereas ISD, then, looks at the interventions needed to bring about a change in performance, the competence concept really looks at workplace performance: can someone do a job? To a degree, ‘competence’ is blind to prior training. It isn’t interested in how a candiate got here, just can that person do the job. There are some conventional ISD concepts that can help clarify the differences beteen the approach. In any training system there are constraints on what can be achieved in the time and resources available and the scope of the training system in terms of workplace performance. The output from training is usually described as the Training Performance Standard (TPS) and recognises that there is a gap between that and the Operational Performance Standard (OPS). The OPS, in fact, equates to the level of competence expected of a person in productive employment. The gap between TPS and OPS can be bridged by formal programmes of On the Job Training (OJT), mentoring or simply informal development through exposure to the real world. The TPS is usually specified in ISD – it is the graduation standard – but the OPS is often left undefined.

To illustrate the problem, consider this OB from the Communication competence:

OB 2.8 Uses and interprets non-verbal communication in a manner appropriate to the organisational and social culture (my emphasis)

In the 100KSA (2018) Communication requirement this has been elaborated as:

09) Show the ability to correctly interpret non-verbal communication.

10) Show the ability to use appropriate eye contact, body movement and gestures that are consistent with and support verbal messages.

The OB relates to non-verbal communication in a very specific context: in relation to the organisational and social culture. The 100KSA elaborations establish an expectation - correctly, appropriate, consistent, supporting - without making clear what training inputs might be required nor how a trainee can meet these expectations. Nor does the 100KSA formulation address issues of organisational and social culture.

In theory it ought to be possible to trace a line from the initial ground training requirement, through the practical application to achieving the final operational assessment. The piecemeal approach to developing commercial pilot training is still some way off that goal. The TPS, then, should identify a set of generic performance elements that broadly map onto the OPS. The TPS should describe both activities and underpinning knowledge that supports the activity described in the OPS.

One problem we have is that we also need a mechanism for assessing performance. It is important to understand that a competence model and an assessment framework are not the same things. The ‘problem’ is that they may overlap and share terminology. In this chapter we explore some of these issues.

5.2 Developing a Competence Model

Although there are well-defined activities associated with the ISD process, developing competencies is less well supported. The paradox of EBT is that it claims to be moving away from ‘task-based’ assessment but being ‘competent’ is fundamentally based on doing tasks. However, implicit in the competence approach is that performance is abstract – generalisable across different work contexts – and aimed at maintaining control of tasks, especially in uncertainty. Competencies try to guarantee control.

The UK Chartered Institute for Personnel and Development makes the following points about competencies:

They ‘focus on someone’s personal attributes or inputs. They can be defined as the behaviours (and technical attributes where appropriate) that individuals must have, or must acquire, to perform effectively at work.

[…[ are broader concepts that cover demonstrable performance outputs as well as behavioural inputs. They may relate to a system or set of minimum standards needed to perform effectively at work.

A ‘competency framework’ is a structure that sets out and defines each individual competency (such as problem-solving or people management) required by individuals working in an organisation’.

‘In designing a competency framework, care should be taken to include only measurable components. It's important to restrict the number and complexity of competencies, typically aiming for no more than 12 for any particular role (preferably fewer), and arranging them into clusters to make the framework more accessible for users. The framework should contain definitions and/or examples of each competency, particularly where it deals with different levels of performance for each of the expected behaviours. It should also outline the negative indicators for that competency competency – the behaviours deemed unacceptable’.

Importantly, there is no single, universal solution to this idea of a competence model, nor is there is a single way to develop them. Organisations need to develop a model that supports their commercial and operational goals. Competence models are useful because they make clear to employees what is expected of them in any particular job or role. This is why they should be framed in the language of observable behaviour. A competence model also directs attention to training requirements. A person cannot be expected to do a job if they have not been properly trained. However, if you look again at the extracts above, it does talk about behaviours that individuals must have or acquire. This last point is significant. The ‘must have’ items can be dealt with by recruiting people who have done the job before or through pre-employment training. Acquiring competence, as we saw erlier, can be done through structured workplace development. Which brings us to EBT. If you scratch the surface of the EBT concept, what we are really talking about is a process of structure workplace mentoring aimed at sustaining and developing a pilot’s ability to cope with operational demands. It is not really training at all.

The rise of CBT/EBT coincided with the discovery of ‘Black Swans’ - catastrophic but unpredictable events that we still need to be able to cope with. Although, by definition, we cannot train to deal with ‘Black Swans’, we can still use the concept as a jumping off point. There are 2 other, more common, properties of the world we need to consider: non-ergodicity and radical uncertainty. The first describes how things never happen the same way twice and the second relates to how things have a tendency to fail in ways we never anticipate. So, a competent pilot must be able to cope with a constant level of perturbation in the workplace (think ‘threats, if it helps) and, should something happen, then be able to restore an acceptable level of control as quickly as possible. In terms of ‘competence’, we can illustrate the situation like this:

A diagram of a company

Description automatically generated

A problem we face with developing competencies is that, often, behaviour is based on deeper, underlying processes that occur internally. Behaviour is just the manifestation of these processes. It could be argued that true ‘competence’ is really these behavioural precursors. The table below proposes a number of target precursors.

Competence	Supporting Activities
Analysing	Causal analysis; risk appraisal; establish the gap between observed and expected; establish abnormal cues based on mental model; compare assumptions about cause and effect relations among cues
Planning	Identify options; establish operational constraints; clarify remaining capability/functionality; planning for contingencies
Organising	Identify actions required; establish resources required; implementing contingency plan
Validating	Referencing observed behaviours to expectations; establish deviations from normal state; use critical thinking
Deciding	Validate rule set; identify information requirements; validate efficacy of option; establish time reuirements
Communicating	Use proper phraseology : pay attention to completeness of standard reports; seek information/clarification/ check understanding; exchange information and comprehensions to establish a shared understanding of the problem; formulate and communicate hypothesis about cause and effect relationships
Collaborating	Monitor, support others, provide guidance and suggestions; states appropriate priorities; update situation periodically; resolve opposing interpretations based on team conflict resolution
Coping	Create space and time; control stress

5.3 Assessing Competence

Assessment of performance is highly problematic. The tools we use – marker frameworks – must meet 2 criteria if they are to be considered useful. First, a category must meet the requirement of validity. Validity is the degree to which the tool measures the target attribute. Second, the tool must be reliable, which is the extent to which it is dependable across time. So,if I assess a candidate at time 1 then, assuming no change in performance, the score from an assessemtn at time 2 should be the same.

The competencies listed above are precursors to performance. They act in combinations to generate the workplace behaviours that are accessible to observation and, therefore, assessment. This has implications for validity. How can I be sure that my observational category is directly linked to the underpinning precursor. The more direct the relationship the better the validity. It is for this reason that ‘Situational Awareness’ is unlikely to have verifiable validity as a marker. The relationship between competencies and outputs is suggested in the diagram below

Competence Marker

Analysing

Planning Application of Procedures

Organising

Validating Management of Systems

Deciding

Communicating

Task Management

Collaborating

Coping

Word Pictures,Grade Scale,Group

Fig. 3.1 Relationship between a Competence and a Marker

5.4 Competencies v Markers

A competence framework is an attempt to describe all the skills and underpinning knowledge required of an individual filling a role in an organisation. The idea is that the role is larger than the specific ‘job’. Someone can be accomplished in their ‘job’ but can still be lacking in overall effectiveness. Historically, specific job-related requirements would be described by a task analysis. Role-related requirements are covered, in part, by the Job Description or Terms of Reference. However, Job Descriptions etc typically only covered a minimum sub-set of what was required to be fully effective in the role. The competence framework attempts to bridge the gap by, first, more fully describing the role and then by elaborating on the performance required in the role. A ‘behavioural marker’ is a description of an element of competence that can be observed in the workplace. The relationship between the 2 concepts is illustrated in Fig. 4.1

5.5 Designing Markers

It was said earlier that a competence model is not the same as an assessment framework. Assessment under EBT (and also the earlier CRM requirement) is based on using observable behaviour as the evidence on which to judge ‘competence’. Thus, whereas a competence model is a broad description of expectations, an assessment framework is a subset of competence that can be routinely observed in the workplace. Below is an example of an assessment framework:

The NOTECHS Behavioural Markers

Categories Elements Example Behaviours

Co-opERATION Team building and Establishes atmosphere for open

maintaining communication and participation

Considering others Takes condition of other crew

members into account

Supporting others Helps other crew members in

demanding situation

Conflict solving Concentrates on what is right

rather than who is right

LEADERSHIP AND Use of authority and Takes initiative to ensure

MANAGERIAL SKILLS assertiveness involvement and task completion

Maintaining standards Intervenes if task completion

deviates from standards

Planning and coordinating Clearly states intentions and goals

Workload management Allocates enough time to

complete tasks

SITUATION System awareness Monitors and reports changes in

AWARENESS system’s states

Environmental Collects information about the

awareness environment

Anticipation Identifies possible future problems

DECISION MAKING Problem definition / Reviews causal factors with other

diagnosis crew members

Option generation States alternative courses of

action

Asks other crew member for

options

Risk assessment / Considers and shares risks of

Option choice alternative courses of action

Outcome review Checks outcome against plan

There are 3 common methods used to construct assessment frameworks. In aviation, probably the earliest framework was the NASA/University of Texas Crew Effectiveness Marker system. This was developed by looking at a range of fatal aircraft accidents and identifying what behaviours contributed to crew failure. This method could be called the ‘historical’ approach. The NOTECHS framework illustrated above was developed by a committee of SMEs. The EASA framework is, similarly, the output from a committee. A third approach is to interview line pilots to get their views. By using structured interview techniques and ‘card sort’ techniques it is possible to develop an ecologically valid assessment framework. An example of such an approach is given here:

1. COMMUNICATION

This dimension relates to the way in which an individual communicates. It includes the extent to which the speaker is clear, easy to understand and unambiguous.

Positive indicators include:

The sharing information and prior experience, actively seeking opinions, giving input not just when requested but also proactively. Positive responses to inputs (acknowledgement, repeating messages).

Negative indicators include:

The failure to listen or ignoring information. Failure to explain decisions, actions, intentions. An unwillingness to communicate (needs constant prompting or repeated requests). Failure to check misunderstood communication (demonstrates hesitancy or uncertainty).

2. TASK MANAGEMENT

This dimension relates to the conduct of the task. It includes the consistent and appropriate use of checklists and procedures. Making effective use of time. The avoidance of distraction and maintaining the bigger picture of things happening around the aircraft.

Positive indicators include:

A consistent, but flexible, use of SOPs. Monitoring the use of checklists during busy periods and the positive verification that tasks have been completed. Maintaining an even tempo of work (no unnecessary haste or urgency). Recognising when to minimise non-essential conversation. Maintaining awareness of other aircraft, objects etc around the aircraft both in the air and on the ground. Actively developing mental pictures of what to expect during the next stage of flight (e.g. through verbalisation of expected landmarks, events, system changes etc). Anticipation and thinking ahead. Being aware of time available/remaining, being aware of things around the aircraft (in the air and on the ground), verifying geographical position.

Negative indicators include:

Too strict an adherence to or rigid application of SOPs. Spending too much time out-of-the-loop on admin tasks, failure to update on events when off-frequency. Rushing or delaying actions unnecessarily

3. TEAM BUILDING

This dimension describes the extent to which effective working relationships are established and maintained within the crew. It includes behaviour which binds the team and which establishes a task focus.

Positive indicators include:

Setting the tone. Clarifying expectations and standards of performance. The recognition that others have a part to play in the crew process. Clear allocation of tasks and responsibilities. Briefing any excursions from SOPs. Fostering a sense of comfort and inclusiveness in the group.

Negative indicators include:

Avoiding responsibility for actions, preventing full expression of views, intolerance, failure to allow individuals to fulfil their role, interference in the work of others.

4. PLANNING AND DECISION-MAKING

This dimension relates to the way crews go about making decisions and agree upon appropriate courses of action.

Positive indicators include:

Sharing problems and concerns, clarifying plans, identifying and discussing options and alternatives. Evaluating risks, pointing out errors of thinking, explaining decisions, seeking agreement on courses of action.

Negative indicators include:

Hasty reaction to events, failure to consider alternatives, failure to discuss solutions, over-reliance on other agencies.

5. INTERACTION STYLE

This dimension relates to the way crew members interact with one another. It includes an individuals’ personal style, their way of dealing with others and their approach to the task.

Positive indicators include:

An optimistic, positive approach to the job, friendly and approachable. Personable and easy to get on with. Patient with others, sensitive to their needs and open to feedback. Conscientious and dependable (can be relied upon to do the job).

Negative indicators include:

Overbearing, confrontational, aggressive. Prone to getting upset when things go wrong. Sometimes lacking in confidence, timid or given to inappropriate behaviour (e.g. poor use of humour). Lacking in skills and unstructured in their approach to the job. Too relaxed or too rigid application of the rules. Inflexible.

It is important to remember that assessment must be appropriate to an airline’s needs. The competences required of a business jet crew, compared to a cargo crew or a wide body ULH passenger crew will differ. Markers are abstract constructs that attempt to capture an aspect of behaviour that is deemed important to the operation.

5.6 Validating a Marker Framework.

The 2 examples of marker frameworks illustrated above contain a broad statement of a behaviour - ‘Cooperation’, for example - and then some elaboration in the form of example behaviours or positive indicators. The elaboration is an attempt to help assessors to better understand the scope of the marker. The better the assessors understand the boundaries of performance, the more standardised assessments will be. However, the natural tendency is for assessors to look for the elaborating examples specifically rather than use them to guide their judgement. The trainee is then assessed based on how many of the example behaviours are observed. This is actually codified in the EASA VENN. This approach is wrong. The broad sweep of normal behaviour makes it impossible to describe every way a specific competence element might be demonstrated by an individual. Assessors must use their expertise and judgement.

Assessors will look at performance and extract behaviour elements. These can be physical actions, gestures and other non-verbal signals or speech acts. These elements represent the evidence upon which an assessor will evaluate performance. The marker framework must be capable of capturing those element deemed most significant in terms of performance outcomes but must do so in a way such that multiple assessors will make the same categorisation of observed acts: as far as possible, assessors should place the same event in the same category. Therefore, marker schemes must be validated as part of the initial development phase of the EBT. To do this, first, collect some segments of crew performance on video. Next, SMEs who are fully conversant with the marker examine the video and identify the significant behaviour elements. These are then categorised using the markers. Only those elements that are unanimously agreed upon by the project team are retained for phase 2. Next, small groups of potential assessors observe the videos and, independently, identify behaviour elements and assign to markers. The results are then compared with the SME benchmark. Elements assigned to the same category by both SMEs and trial subjects can be ignored. Where elements are assigned to different categories then consideration must be given to redesigning the category, either through changing the definition of the marker or by better elaboration through examples, including specifying what is NOT included under the marker.

EASA has not published any evidence to suggest that the 9 competencies have been validated.

5.7 A Proposed Solution

If an airline wants to develop its own assessment marker scheme, the following process will help:

Step 1. From the SMS, develop a model of current and predicted operational hazards

Step 2. Construct a ‘look up table’ of crew competence (Fig 4.1 as an example)

Step 3. Cross reference ‘look up table’ to hazard model and verify coverage

Step 4. Identify critical skills to cope with hazard model

Step 5. Identify elements of critical skill set that are routinely observed during normal operations

Step 6. Construct marker framework (category and descriptors)

Step 7. Cross reference markers to ‘look up table’

5.8 Conclusion

A competence framework is a broad description of a set of behaviours and underpinning knowledge associated with successful performance. A behavioural marker is a subset of a competence that can be observed and assessed in the workplace. Markers generally comprise a top-level label and definition supported by example behaviours. The examples are intended to clarify and better communicate the intent of the marker. Markers must be validated before use.

6 Some Thoughts on the idea of ‘Knowledge’ as Competence

The introduction of a set of ‘competencies’ against which to assess pilot performance has involved some debate around the issue of a specific ‘Knowledge’ competence. Although not adopted by ICAO, it is included in the EASA framework for EBT. Its description and associated ‘observable behaviours’ are listed in the table below.

Application of knowledge (KNO)
Description:	Demonstrates knowledge and understanding of relevant information, operating instructions, aircraft systems and the operating environment
OB 0.1	Demonstrates practical and applicable knowledge of limitations and systems and their interaction
OB 0.2	Demonstrates the required knowledge of published operating instructions
OB 0.3	Demonstrates knowledge of the physical environment, the air traffic environment and the operational infrastructure (including air traffic routings, weather, airports)
OB 0.4	Demonstrates appropriate knowledge of applicable legislation.
OB 0.5	Knows where to source required information
OB 0.6	Demonstrates a positive interest in acquiring knowledge
OB 0.7	Is able to apply knowledge effectively

The list of OBs is supposed to represent statements of observable performance against which an individual’s competence can be assessed. OBs 0.1, 0.2 and 0.4 relate to the simple recall of information: limitations, systems functioning, interactions between systems, operating instructions and legislation. OB 0.3 relates to recall of information particular to a destination. The remaining 3 OBs do not flow from the top-level description and appear to be after-thoughts. OB 0.5 points to a need to have efficient search methods to find information while OB 0.6 reflects an attitude towards study or maintaining currency.

The competence description positions ’knowledge’ as little more than information contained in textual artefacts. However, OB 0.7 suggests that, to be competent, you must be able to ‘apply’ knowledge ‘effectively’. But what does that mean?

It is a convention to classify knowledge as either declarative or procedural. In essence, the former describes what we can say and the latter describes what we can do. The declarative/procedural dichotomy is not new and the first 4 OBs listed in the table are examples of what we would consider declarative knowledge. Although ‘apply knowledge’ nods towards the procedural side of things, it is too vague a statement to be of any real use. What we are really interested in is how do we ‘apply’ knowledge? To what do we ‘apply’ it?

Ohlsson, in his book ‘Deep Learning’, prefers the term ‘process’ to procedural. In his view, declarative knowledge is more than whatever can be recalled from memory and recited. Rather, it comprises arrays of constraints that must be satisfied for any action or intervention in the world to be considered legitimate or successful. His ‘process’ knowledge describes the rule sets needed to control action. This formulation starts to get closer to a useful description of ‘knowledge’ that could inform an approach to training and performance measurement. Knowledge supports action. Declarative knowledge is used to establish the legitimacy of the current status of the task in relation to our operational goal while process knowledge allows us to achieve congruence between the actual and the desired states of the world. From this perspective, the ‘Knowledge’ competence is an inadequate formulation.

Advances in neuroscience, and in study of the visual system in particular, have resulted in significant changes in our understanding of how the brain works. Historically, the study of cognition was predicated on information flowing from the outside - the surrounding world - to the brain, being processed and then out again through action driven by routines stored in memory. It now seems that this might not be the case.

For a moment I want you to close your eyes. I want you to recall the scene in front of you at the point at which you closed your eyes. Picture in your mind everything that was in your field of view. Take a few moments to recreate the scene. Then open your eyes. What do you see? In all probability your answer will be ‘I see what was there when I closed my eyes’. If there was a window in your field of view you might notice that something has changed. The drift of clouds across the sky might have changed the lighting conditions. Essentially, though, the world is still how it was when you closed your eyes. Or maybe not.

In your mind you constructed a view of the outside world and when you opened your eyes you projected your internal. mental view onto the scene in front of you. You then cross checked to see if what you perceived matched your expectations. Neuroscience is increasing revealing that, in terms of cognition, the flow is from the inside out and not the other way around. Cognition is not simply interrogating the sensory world and interpreting cues. Rather, it is a process of validating expectations based on stored data and reconciling differences. So what does this mean for the idea of ‘knowledge’.

The physicist, Carlo Rovelli, explores the nature of reality from the perspective of quantum physics in his book ‘Helgoland’. He makes the point that ‘knowledge’ describes more than just a ‘library’ of stored concepts, facts and rules. it is the very the process of interacting with the world. In this view, ‘knowledge’ is a dynamic process of detecting discrepancies between the projected and the encountered worlds and the actions taken to reconcile differences. In this view the world is not a static ‘out there’, it is something that is created as part of achieving our goals. Returning to Ohlsson, ‘declarative’ knowledge can now be seen as a repertoire of conditions, acquired through training and experience, that allow us to detect differences between our projected expectations and our actual encounters. In effect, declarative knowledge is error detection. Process knowledge describes the ways we reconfigure the world to achieve our goals.

There are 2 significant implications that flow from this discussion for the idea of ‘competence’ as formulated in the ICAO/EASA model. First, the OBs that require simple recall have nothing to do with ‘knowledge’. They relate to unstable artefacts that describe arbitrary constraints. To be considered ‘competent, I must, of course, perform within those constraints. But I called them unstable simply because technology and policies are not static. LOSA observations are littered with supposed ‘errors’ that merely reflect the fact that the pilot being observed was working with an out of date framework of policy and procedures. Pilots can still fly aircraft but they cannot necessarily recall the latest rule changes. Whilst rules, procedures and limitations are important, making them the focus of performance assessment places an undue emphasis on the easily-captured but, probably, less important aspects of performance.

The second implication is that ‘the ability to apply knowledge effectively’ (OB 0.7) must be rendered meaningful. Technical, systems information is of use not just because it allows a pilot to diagnose what has happened but more because it supports the construction of expectations: it allows me to know what I will see and, therefore, be able to tell if what is happening is what is required. We need to develop training that addresses how pilots create expectations during a flight, how they detect ‘errors’ between the expected and actual status, how they diagnose the causes of any discrepancy and then, finally, how to intervene to restore equilibrium. This is ‘knowledge as action’. This is true ‘competence’.

Finally, if knowledge really is action then it suggests that any meaningful attempt to assess performance should concentrate more on the utility of outcomes in relation to operational goals. An ability to recite chapter and verse is evidence only of a reliable memory, not an indication of competence. Without a doubt, outcomes must be validated against prevailing constraints - policies and rules - but that is the final stage of performance, not its underpinning driver. This approach poses a serious challenge to concepts such as ‘situational awareness’ and ‘error’. It seems that what will call SA is more likely to be a reflection of the efficacy of our interventions in the world to restore equilibrium. Errors are not outcomes to be managed but, rather, are simply feedback signals. It is the status of the current task that must be managed in order to remove