State of the World

After several months of behind-the-scenes work, we’ve moved the last of our services to our new infrastructure. You will hopefully have seen no impact at all, although a couple of people reported problems on Thursday when we moved our front-end servers.

The new infrastructure gives us much more resilience than before, and also allows us to scale up and down according to demand. When we’ve done this in the past, it’s been a manual process.

Now that everything has been successfully migrated, we’re going to start on rolling out the new version of OpenTrainTimes. This will happen slowly and gradually, and there won’t be any ‘big bang’ change. More about that in weeks to come.

We’ve almost caught up with the backlog of map updates – here are some things we’ve done recently:

There are a lot of support tickets with fixes to implement, which we’ll be doing over the next few weeks and making more frequent releases.

That’s your lot for now – we’ll be back soon with more details of the new version and how we’re going to roll it out. Stay tuned!

Service degradation over the past weeks

The last week hasn’t been great in terms of availability of our data feeds. We’ve had numerous outages that are outside our control:

  • On 3rd June, we had an outage of all our data feeds from 0320 to around 0333. This was followed by a period of high latency (the delay between a message being generated by a train describer and it being received by our systems – sometimes known as lag) between 0446 and 0512, then from 0518 to 0524. From 0631 to around 0735, data was being received by our systems between 15 seconds a 2 minutes late.
  • On 4th June, we had an outage of all our data feeds from 2342 (on 3rd June) to around 0912. This resulted in our live track diagrams freezing for the duration of the outage, and for some hours afterwards, maps showing out-of-date data and the location screens not reporting any train movements.
  • On 5th June, we had an outage of all our data feeds from 0941 to 1103 – again resulting in our maps showing out-of-date data for some time after the feeds returned, due to the nature of the feeds.
  • On 6th June, we had a partial outage on our train movement data feed from 1740 until around 2115 on 7th June. Our monitoring systems should have picked up this issue but didn’t – we are investigating why. On the same day, from 1747 to around 2130, we had an outage of all our our data feeds – both feeds returning to normal afterwards, but again resulting in our site showing incomplete and out-of-date data.
  • On 7th June, we had an outage of all our data feeds from 0846 to 0851, although the maps will have updated quickly afterwards due to the volume of trains running in the end of the morning peak period
  • On 8th June, we had an outage of our train describer feed between 1155 and 1201, from 1215 to 1311, from 1314 to 1328, from 1343 to 1350, from 1358 to 1407, from 1415 to 1430, and from 1438 to 1440. During this time, the train movement feed was operating normally, but this issue caused our maps to freeze repeatedly and intermittently for nearly two and a half hours.
  • On 9th June, we had an outage of all our feeds between 1320 and 1350, a partial outage between 1428 and 1440, then a full outage from 1440 to 1453, followed by a further outage between 1601 and 1620.

Quite rightly, many of you have voiced your concern over our inability to provide you with a useful service over the past week. We have no control over the Open Data feeds, which are available to anyone at https://datafeeds.networkrail.co.uk/, and we are not happy at the huge number of outages and the combined total downtime over the past week. These issues have been happening for months, if not years, with seemingly no resolution in sight.

We are facing a difficult decision: do we continue to use the public feeds to operate the OpenTrainTimes public website, or do we move to more reliable feeds which aren’t available to, or suitable for, many other developers and websites to use – since they’re not quick to set up, and not free to use when you consider the additional hardware and technical work required. The answer to this isn’t simple, and we continue to hope that the Open Data feeds will be fixed so that everyone can benefit from them.