The last week hasn’t been great in terms of availability of our data feeds. We’ve had numerous outages that are outside our control:
- On 3rd June, we had an outage of all our data feeds from 0320 to around 0333. This was followed by a period of high latency (the delay between a message being generated by a train describer and it being received by our systems – sometimes known as lag) between 0446 and 0512, then from 0518 to 0524. From 0631 to around 0735, data was being received by our systems between 15 seconds a 2 minutes late.
- On 4th June, we had an outage of all our data feeds from 2342 (on 3rd June) to around 0912. This resulted in our live track diagrams freezing for the duration of the outage, and for some hours afterwards, maps showing out-of-date data and the location screens not reporting any train movements.
- On 5th June, we had an outage of all our data feeds from 0941 to 1103 – again resulting in our maps showing out-of-date data for some time after the feeds returned, due to the nature of the feeds.
- On 6th June, we had a partial outage on our train movement data feed from 1740 until around 2115 on 7th June. Our monitoring systems should have picked up this issue but didn’t – we are investigating why. On the same day, from 1747 to around 2130, we had an outage of all our our data feeds – both feeds returning to normal afterwards, but again resulting in our site showing incomplete and out-of-date data.
- On 7th June, we had an outage of all our data feeds from 0846 to 0851, although the maps will have updated quickly afterwards due to the volume of trains running in the end of the morning peak period
- On 8th June, we had an outage of our train describer feed between 1155 and 1201, from 1215 to 1311, from 1314 to 1328, from 1343 to 1350, from 1358 to 1407, from 1415 to 1430, and from 1438 to 1440. During this time, the train movement feed was operating normally, but this issue caused our maps to freeze repeatedly and intermittently for nearly two and a half hours.
- On 9th June, we had an outage of all our feeds between 1320 and 1350, a partial outage between 1428 and 1440, then a full outage from 1440 to 1453, followed by a further outage between 1601 and 1620.
Quite rightly, many of you have voiced your concern over our inability to provide you with a useful service over the past week. We have no control over the Open Data feeds, which are available to anyone at https://datafeeds.networkrail.co.uk/, and we are not happy at the huge number of outages and the combined total downtime over the past week. These issues have been happening for months, if not years, with seemingly no resolution in sight.
We are facing a difficult decision: do we continue to use the public feeds to operate the OpenTrainTimes public website, or do we move to more reliable feeds which aren’t available to, or suitable for, many other developers and websites to use – since they’re not quick to set up, and not free to use when you consider the additional hardware and technical work required. The answer to this isn’t simple, and we continue to hope that the Open Data feeds will be fixed so that everyone can benefit from them.
We are back to a sorry state of affairs, similar to the problems we had back in December 2018, where the train describer feed from the Network Rail Data Feeds platform starts sending message through later and later, then finally drops lots of messages.
This isn’t a problem affecting just us – sites such as RailCam and Traksy are also affected. There are alternative feeds that we can use which don’t have this problem of late and missing messages, but using them at the moment wouldn’t be in the interests of the whole Open Data community. For this reason, we’ve decided to let the quality of information on our site degrade so we can get the right level of focus on the problem.
Happy New Year from all of us at OpenTrainTimes!
We’ve just released a slightly delayed set of updates to the site. These were due to go live about a week ago, but due to illness we didn’t have time to test everything to our satisfaction.
In this release:
- A new map of Mossley Hill to Runcorn, covering the new workstations at Manchester ROC, as well as Ditton signal box. Garston Intermodal Freight Terminal is also drawn
- Improved functionality from the new control system at York ROC means we can show, and we’ve added, routes and signal aspects for the whole of the York area, covering York station and parts of the surrounding routes
- The Hull area map now contains route indications
Minor fixes include adding a missing signal at Cogload Junction, L394 signal at Seven Kings, F456 signal at Feltham, W367 signal at Southfields, 1999 signal at Gilberdyke, and new platforms berths at Skipton, Ilkley and Bradford Forster Square.
Unfortunately, at least three maps – the Exeter, Liverpool Street and Airedale maps have become corrupted during deployment and rather than rolling back the whole release, we’ll fix them as soon as we can.
The last few months have been hectic. We’re continuing to work on the next major release of the site, which is a total rewrite – and to make sure the public-facing site remains free, we’ve been taking on more consultancy and software development work for the rail industry. Some of the projects that have had our input are very interesting, but not in the public domain – but the important thing is they’ll make sure there is always a public, free version of the site.
Many of you will have noticed that the Network Rail Data Feeds platform has been substantially less reliable than we need it to be over the past weeks and months. Outages started happening several times a day, meaning we weren’t able to bring you accurate information. This didn’t just affect us – all other sites using the Open Data platform were affected. The sheer scale of the problem and the number of people affected has been brought to Network Rail’s attention, and their supplier and maintainer of the Data Feeds platform is working on making it more stable.
Separately, we’ve invested a great deal of time and a lot of money in getting resilient connectivity set up to Network Rail for our commercial projects. We are seriously considering moving the public-facing website over to this feed – but there’s work to do so we obfuscate (hide) freight and engineering trains in the same way as the Open Data feed does. This will mean we’ll be immune to any future outages that affect the Open Data platform, which is obviously good – but is this the right thing for us to do? Having such a large and visible site to highlight problems with the data feeds is useful – it means there’s one more big name that’s affected when problems occur, and that helps get the problems fixed for everyone, not just us. Maybe we should continue with the public site using public feeds. Comment on this post with your thoughts.
We’ve set up a status site to show you what’s going on. Any outages will be reported here once our systems detect them, and also tweeted to a new account (details to come). You can subscribe via email to alerts too.
This status page plugs directly in to our monitoring systems, which we’ve moved over to Amazon EC2 so we can run the public site alongside our commercial products. Over Christmas, we’re going to be moving the last bits of the website over. You shouldn’t see anything different.
Finally, we made a few updates to the existing site:
- The Filton 4-Tracking project has completed, and our Bristol Parkway map has been updated. We’ve had to move the Avonmouth route on to a new map, because there’s simply no way we could manage to fit it on to the existing map!
- The line between Saltmarshe, Howden and Ferriby has been recontrolled to York ROC, and is now available on our Hull area map
- The Thameslink route through Wimbledon was moved to new signalling system at Three Bridges ROC some months ago and, a little late, we’ve updated our map to include route and signal indications
Until the next time, have a peaceful festive period, and see you in 2019!
Our new site infrastructure is progressing really well. The mapping engine is under test at the moment, and we have a big configuration exercise about to start to get it all working. More on that in a few weeks.
Despite all the testing, we’ve released a map of the Horsham area, and fixed a couple of issues on the Derby map.
Earlier this week, we launched our updated Derby map and promised that we’d be adding route indications in due course. Today, we released the update, along with several other issues fixed:
- A crossover from Royal Oak Sidings at Paddington was commissioned at Christmas and is now present on the map
- We’ve included two berths at Glenrothes with Thornton which will more accurately show trains in the station
- There’s a new pair of berths between Shiplake and Henley which will stop trains disappearing on the branch
- Signal 2490 at Oxford now shows descriptions when trains pass through
- Two berths near Proof House Junction outside Birmingham New Street were showing incorrect descriptions
We’re still working hard on our new Scottish map – more on that in a subsequent update.
This week, we’ve been working on extending existing maps to continue to cover more of the railway network, and so the following maps have extra track and signalling:
Due to popular request, we’re now displaying the last train that entered a platform at London Kings Cross. Let us know what you think of it – positive or negative!
We’ve also fixed bugs at Woodborough Sidings, Liverpool Lime Street, between Poynton and Adington, and signal 2215 at Swindon – all of which were reported as bugs to firstname.lastname@example.org. Every email you send to the support email address is logged and used to help us prioritise what to fix and work on next.
We’ve been busy again this week with another new map of Oxenholme to Carlisle, which almost completes our coverage of the West Coast Main Line up to the border with Scotland. Expect an extension through Motherwell when the area is resignalled.
Amongst other things, we’ve also fixed a few problems and added platform numbers at Bristol Temple Meads, added a berth at Barnt Green where trains seemed to disappear, fixed the platforms at Liverpool Lime Street and missing berths near Kemsley on the Chatham map.
Let’s see if we can keep up the pace and get another brand new map out for you in the next week!
We’re pleased to announce that the Oxford area map has been updated as result of the major resignalling between Didcot and Oxford over the past few weeks. The map went live on Sunday evening, and we’re still fixing a few bugs with it (such as limit of shunt boards appearing as TVM430 markers) – but this evening we’ve added signal aspects to the map. There are still some issues on the Didcot map to fix too, and we’ll update that map in the coming days.
Also, by popular demand, we’ve drawn a map of Bristol Temple Meads which covers down to Bridgwater and joins up with our Exeter map at Cogload Junction.
Other issues we’ve worked on include dead berths between Leeds and Wakefield Westgate, incorrect routes from signal 176 at Shortlands, some missing GPLs at Swanley, new signals between Liskeard and Taunton, a missing crossover at Northallerton, routes on the East Coastway map, a problem with the fringe between Kings Cross and Peterborough PSB, route issues at Bradley Junction near Huddersfield – along with HU752’s signal berth not updating, updates as a result of the Ashton Moss resignalling, and a couple of minor updates to the North Lincolnshire map.
Finally, in a change to how we normally do things – the next map is probably going to be Liverpool Lime Street.
Earlier today, the OpenTrainTimes website was down. The root cause was that one of the disks was full, which we’ve fixed and put our recovery process in to action. Given the time it takes to replay a backlog of messages – a problem we’ve had before – it took until about 3pm for us to catch up on the missed messages.
Although we have plenty of monitoring in place to alert us of problems such as this, we’ve identified – in this case – that the monitoring was set up for the wrong server. Consequently, nothing alerted us to the imminently-filling disk.
We’ll be fixing this over the coming days.
We’re sorry for the extended degradation in service today. We don’t take outages lightly, and we’ll be looking at ways to stop this happening in the future.