From what I’ve gathered, it does sound like an input sanitisation issue. The problem is the developers probably thought ‘well all the input only comes from official trusted sources so we don’t need to bother’, which is a really bad take because even if the official trusted sources that are providing input to you are never malicious, they can still make errors.
Little Bobby Tables is as always alive and well.
Yes, I know Peter is being facetious about Java, but input validation is an issue regardless of the language used, and all external systems (even your own) should always be considered untrusted. As a former full time Java developer I’ve seen lots of horrific Bobby Tables code. Fortunately (at least with RDBMS based stuff) most people eventually got a clue and started using the standard toolkit-provided query parameterisation which helped a lot, but you’ve still got to make sure you only accept valid input and reject everything else.
And yes, if you have a problem like this, you can have it repeated (e.g. it may have been something like: remote system sends flight plan, crashes your system, watchdog restarts your system, remote system hasn’t got a response so it retries after a timeout period, and crashes your system again. Then, because of the masses of change control paperwork, there’s no way to hotfix a system like this, so you have to ask the owners of the remote system to manually remove the problem flight plan – but then they have to go through their paperwork to allow someone to manually root through the system and find it, and remove it, so for several hours the remote system is just crashing yours every time it comes up. Now someone in management realises you need input validation, so with the change request, the business case, the planning, etc. etc. maybe in 5 years time your system will be robust enough for this to not happen again – if, and only if the change gets approved which is no means certain and possibly not that likely – in the meantime you just hope that no one sends you a flight plan like that again, and the Russians don’t find out how to send you a flight plan like that. Or you have to beg your upstreams not to do it, which is effectively broadcasting “Send us a flight plan like this and you can take us down for a day” to the Russians…
I had to google on bobby tables. You can do same with PHP or actually anything, AFAIK. But there is something peculiar about the Java gravy train, coupled with the NATS corporate culture (which also drives the gravy train), which attracts the very types that write crap code. Various discussions I read basically say that their “just culture” (hahaha – remember the UK “bust them all” infringements policy) is a “no blame culture” so everybody keeps their job no matter how bad. I suspect all ATC organisations will be similar; it is the job that drives the management personality and shapes the whole company.
AFAIK the IFPS connection is a VPN, like most of their stuff, including the validation gateway the AR uses. So, obviously, ahem, the data must be valid 
How many Russians do you think work at NATS? On the basis of the EuroGA poster ratio, plenty enough! They are often good programmers; I’ve used a fair few “Ukrainians” on freelancer.com…
Just seen this from PPRUNE;
Having just seen the CEO of NATS say a single piece of faulty data caused the meltdown I dont believe him
i know from Airline sources that the problem was known at 0800, 4hrs before NATS admitted to the problem. Thats why flights were initially posted as being delayed before being canx around Midday Ie NATS covered up the problem.
Eventually circa 4hrs later around 1500 NATS announced they had fixed the problem. They hadnt the bug had washed out because the system holds about 4hrs of data
So my analysis is that every time there Is a faulty input of data the system resets itself to protect itself. Fine but surely an airline when discovering its Flight Plan had been rejected will check and input its data correctly . So I believe a DOS input by a third party was re introduced causing the NATS system to go down again. As so many Flight Plans were being inputted it was very difficult to detect, only when NATS admitted there was a problem did the malevolent data ceased being inputted and it eventually washed out
So CEO please explain the delay
That doesn’t ring true because
Fine but surely an airline when discovering its Flight Plan had been rejected will check and input its data correctly
is done by IFPS, whereas it was NATS code that bombed after it got the validated FP from IFPS.
Martin Rolfe said that Nats’ automatic flight planning had been designed to stop gor safety reasons to ensure that no incorrect information could impact the rest of the air traffic system.
He ruled out a cyber attack and declined to confirm or deny claims that the wrongly submitted information came from a French airline.
Well there you go its Froggies fault again.
Watch out Russia and any other country that opposes us. We can bring your whole air traffic system crashing down without affecting our own.
Now if only we could figure out how we did it.🤣
gallois wrote:
Now if only we could figure out how we did it
I think that is exactly the reason they keep very quiet right now. And rightly so.
Peter wrote:
is done by IFPS, whereas it was NATS code that bombed after it got the validated FP from IFPS.
There are two systems that the IFPS communicates with at NATS and that’s UK Flow Management Position (FMP) and Flight Data Operations (FDO) both established at Swanwick. This will run the FPL via the SRD. Although; the IFPS is not obliged to comply with the SRD, and when processing messages, the FP shall only validate messages in accordance with current and relevant Route Availability Document (RAD) restrictions and route availability requirements.
So I suspect, the issue will lie with the FMP/FDO.
I don’t think a FP is rejected due to SRD noncompliance. IFPS has done the ACK before then and that is all you hear. Probably the way it works is that the route is amended per SRD before distribution to enroute units (which is not an IFPS job). It is mostly tactically operated using radar anyway.
But I agree that is probably where it fell over.
The ADEXP waypoints plan included two waypoints along its route that were geographically
distinct but which have the same designator.
Although there has been work by ICAO and other bodies to eradicate non-unique waypoint names
there are duplicates around the world. In order to avoid confusion latest standards state that
such identical designators should be geographically widely spaced. In this specific event, both of
the waypoints were located outside of the UK, one towards the beginning of the route and one
towards the end; approximately 4000 nautical miles apart.
Having found an entry and exit point, with the latter being the duplicate and therefore
geographically incorrect, the software could not extract a valid UK portion of flight plan between
these two points. This is the root cause of the incident. We can therefore rule out any cyber
related contribution to this incident.
Therefore, on taking over the duties of the primary server, the backup system applied the same
logic to the flight plan with the same result.
So the problem was that a flight plan contained two identically named waypoints 4000 NM apart, which caused the system trying to extract the UK portion of the flight plan to fail and crash the whole system. When the backup server did the same thing, it also failed.
And the priority was that EVERY flight plan in the system gets processed and if one can’t, the whole system shuts down. I guess that logic will be revised.
At least it is now clear that it was not a cyber attack.
