NERL_Major_Incident_Investigation_Preliminary_Report_pdf local copy
What weird software. There are many such cases. Not many (any?) 5-letter ones but loads of shorter ones.
They don’t say which waypoint it was, do they?
Not trivial to fix though. They ought to do a quick sanity check on each FP to ensure no waypoint is specified more than once. But that will fail if some jet files via SAM (EGHI) and is going to SAM (LGSM) 
The usual way to deal with this is to check for a “reasonable” distance. Jepp Flitestar does it wrong quite often, and FF does/did also.
Peter wrote:
Not trivial to fix though.
In general it would be by simply ditching the plan onto an operator’s queue to be reviewed rather than stopping the whole system because one plan misfires. Pretty much that is what the validation system of IFPS does all day long and with gusto.
The other bit is, that the problem appears to have been that there were no dedicated waypoints for crossing the FIR boundary and therefore no dedicated entry and exit point the system could identify. Fine, I can understand that. What is difficult to understand however is, why waypoints several 1000 NM away would be relevant? Let’s say the system goes through the route to check for the entry and exit points and finds none, it should then take the next point before or after the last known UK waypoint to cut out the UK route segment rather than 2 waypoints each being at the very end and beginning of the plan?
I guess 20/20 is a great hindsight and that they will work in that direction to resolve it.
Pretty much that is what the validation system of IFPS does all day long
Maybe, but the member countries have worked so hard to limit how much data is sent to IFPS (to protect national ATC staff workload / to retain control over domestic policies) that IFPS is not validating a whole load of stuff which it could be.
there were no dedicated waypoints for crossing the FIR boundary
That is a “requirement” for VFR only, and is country-dependent. I doubt IFPS check for that. But an FIR boundary is not the same as an ATC sector?
I think this is a “software classic”; what is curious is that it has taken this long to surface.
I also bet they are not disclosing the whole story, even if they (obviously) have fixed this particular one.
There really are two things going on there.
This one is a “shit happens” issue, an exotic edge case not considered in design and not caught in testing. Given that 15 million flight plans were processed and this is the first where it occurred, from a simple software engineering point of view not a massive issue; a flaw in the logic trying to guess the correct waypoint by proximity. I unerstand that NATS does not publish the exact conditions nor the waypoint names to prevent some idiot filing another one “to see what happens” or maliciously.
This is utter stupidity. Whoever made that design decision should never be let anywhere near systems design, ever. For penance, I suggest he has to live in a house where any electrical fault (let’s say, a failed light bulb) shuts down all electricity and prevents turning it back on until the light bulb is identified and replaced. In the dark.
You really have to admire the chutzpah of calling this a “fail safe” design; in that sense if I have a heart attack my heart has “failed safe”; it has stopped so will not suffer another heart attack. Being dead is hardly a desirable state to be in; safe as it may be.
I unerstand that NATS does not publish the exact conditions nor the waypoint names to prevent some idiot filing another one “to see what happens” or maliciously.
You don’t have to be a genius to come up with something. In France there are still lots of two-letter waypoint names, corresponding to NDBs that may or may not still be in operation. For example CM is overhead Avignon LFMV, it’s used in all the arrivals and approaches there and for en route stuff too. I’m sure there must be a CM somewhere else in the world.
johnh wrote:
You don’t have to be a genius to come up with something. In France there are still lots of two-letter waypoint names, corresponding to NDBs that may or may not still be in operation. For example CM is overhead Avignon LFMV, it’s used in all the arrivals and approaches there and for en route stuff too. I’m sure there must be a CM somewhere else in the world.
I find it quite common when entering waypoint names and in particular beacon ids in a GPS navigator to be asked to choose among two or even more.
The failure to process a flight plan with (a) two distinct waypoints with the same name in the route and (b) one of them being before and one after UK exit (or transit, not clear from the report)
Would my example of 2 × SAM cause this too?
It meets the second condition.
Example: EGHH SAM …. SAM LGSM
Shouldn’t you try?
Cobalt wrote:
This is utter stupidity. Whoever made that design decision should never be let anywhere near systems design, ever.
It may not be as simple as that.
As I understand it (and my understanding may be limited) it was not a straightforward ‘invalid entry’ failure. It was a failure that the code had never come up against before and didn’t know what it was. As it turns out, it was caused by a problem with an entry (a flight plan). But it might not have been caused by an entry, it might have been a problem with the waypoint database – a potentially dangerous one – and in that case ‘all stop’ perhaps is the desired response. You don’t know until afterwards that just binning that one flight plan would have sorted it.
Why it took so long to establish the problem and get it back up is anyone’s guess, but I’d say the most likely is human in nature – a case of getting the right person(s) onto the job.
Does anyone really understand what happened Technically?
The report says this
Having correctly identified the entry point, the software moved on to search for the exit point
from UK airspace in the waypoint data.
Having completed those steps, FPRSA-R then searches the ICAO4444 section of the ADEXP file.
It initially searches from the beginning of that data, to find the identified UK airspace entry point.
This was successfully found. Next, it searches backwards, from the end of that section, to find
the UK airspace exit point. This did not appear in that section of the flight plan so the search was
unsuccessful. As there is no requirement for a flight plan to contain an exit waypoint from a
Flight Information Region (FIR) or a country’s airspace, the software is designed to cope with this
scenario.
Therefore, where there is no UK exit point explicitly included, the software logic utilises the
waypoints as detailed in the ADEXP file to search for the next nearest point beyond the UK exit
point. This was also not present. The software therefore moved on to the next waypoint. This
search was successful as a duplicate identifier appeared in the flight plan.
Having found an entry and exit point, with the latter being the duplicate and therefore
geographically incorrect, the software could not extract a valid UK portion of flight plan between
these two points. This is the root cause of the incident. We can therefore rule out any cyber
related contribution to this incident.
So
1. the software found a point of entry into UK airspace which was on the FIR boundary.
2. no point of exit was found on the boundary (starting the search at the end of the flight plan and working back towards the beginning of the plan).
The software logic […] to search for the next nearest point beyond the UK exit
point.
That I don’t get. If it doesn’t have a UK exit point, how can it search for the nearest point to it?
Presumably they mean it’s trying to find the first point outside UK airspace so that it can interpolate the exit location, but that’s not what it’s saying.
Do they mean that it takes the entry point, and checks later points in the plan to find the first one outside of UK airspace? That would make sense expect for it’s hard to understand how that wasn’t found.
This was also not present. The software therefore moved on to the next waypoint.
Next point searching in which direction? From the end back? Or from the entry point forward. The text seems to indicate it’s search from the exit point that wasn’t found, which obviously doesn’t make any sense.
This search was successful as a duplicate identifier appeared in the flight plan.
Having found an entry and exit point, with the latter being the duplicate and therefore
geographically incorrect, the software could not extract a valid UK portion of flight plan between
these two points.
So,
1. We now have a valid entry point on the FIR boundary
2. We have a exit point (still not sure how it was arrived at) but the exit point is one with a duplicate name.
3. So the software then looks up the co-ordinates for both points and picks the incorrect co-ordinates for the exit point (instead it picks the coordinates for the point 4000nm away from the UK) and things the thing between the entry point and this point must be the UK portion, which makes no sense to it.
Is that right?
How it arrives at the UK exit point isn’t clear to me. Lots of possibilities for how it could have been done, but it’s not clear what the logic employed actually was, at least to me. The report reads to me like someone who didn’t understand the logic wrote the report based on an explanation written by someone who did understand it!
