What’s new – 25th June 2017

Monday’s problems on the website has caused us to scale our release plans back a bit to make sure the site’s stable. Some of the work we’ve done since Monday has involved adding in extra monitoring to check that our fixes have been successful and, so far, they have been.

So, what have we added?

The biggest thing is the new Gloucester map, which links up with the Bromsgrove and Swindon maps, covering Gloucester to near Chepstow, Stroud and Stonehouse to Standish Junction, Gloucester Yard and all the way up to just outside Bristol Parkway. We’re planning to extend our coverage westwards too.

We’ve also fixed numerous little things, from platform numbers and junction links on the Warrington map, to missing signals between Crewe and Winsford, some TVM430 marker boards which should have been limit-of-shunt markers at Staines, and missing signals at Worting Junction. We’ve also taken off the schedule count on the Schedule Search page, which was causing inefficient database lookups and contributing to Monday’s issue.

Thanks to everyone who’s been in touch over the past fortnight – it’s time to sit in the remains of the sunshine with a glass of wine and chill out for a bit.

Post-Incident Review

We had some problems with OpenTrainTimes earlier today. Although the public site is not operated for profit, we take uptime seriously and we’ve produced this review of what happened.

If you use OpenTrainTimes as part of your job and you’re interested in a commercially supported version of the site, including freight data and integrations with your stock and crew systems, please drop us a mail at hello@opentraintimes.com.

What happened?

Earlier this evening, we had multiple users reporting that maps on OpenTrainTimes were lagging.

Upon investigation, we found an unusually large number of users on the site for the time of day combined with a 45 minute backlog on our train describer feed.

We temporarily disabled the train movement feed and turned off logging for the real-time maps in order to process this backlog. Once the backlog had cleared, we turned the train movement feed back on and monitored the service whilst the backlog of TRUST messages cleared.

The site returned to normal operation by about 2045.

More detail

OpenTrainTimes is a very popular site, and several hundred users are usually viewing multiple maps at the same time. This figure grows steadily and gradually over time, and we review our capacity every few months to make sure we’re not caught out. Each time we release a new map, the base load on our servers increases as we have anything up to 500 new pieces of signalling data to process – and then there are the extra users that the maps attract.

But that wasn’t the issue – but not by the number of users, but by the type of users!

Briefly, when a user’s web browser connects to our map server, it either uses a long-lived connection over which map data is sent, or it requests map data every few seconds. Several things influence which is chosen, but it’s usually down to whether the device is behind a proxy server – not all proxy servers allow, or support, long-lived connections over websockets.

This evening, we noticed a larger than normal number of polling users. Since a large percentage of OpenTrainTimes users are coming from a mobile device, we think this may be because a change was made at one of the mobile network providers which meant our websocket implementation couldn’t be used by clients.

Normally, this is OK – but the gradual and continual increase in users each week, coupled with a gradual surge in the number of connections that our server was logging, meant there was insufficient CPU time available to process all of the data coming in to us from Network Rail.

The first thing we did was to turn off logging – we don’t really need it day-to-day, and it bought us some time. We then switched off processing TRUST messages, allowing them to queue whilst we allocated the rest of the server’s capacity to processing the backlog of train describer (TD) messages. It took about 20 minutes to process the TD messages, after which we turned TRUST messages back on. Processing the backlog of those messages, plus the remaining TD messages took about another hour.

What we’re going to do about it

First of all, we’re sorry that we missed a trick and took too long to respond to the initial reports of a problem.

We’re going to add some new health checks to our monitoring system, one of which will enable us to monitor the size of any message backlog.

We’re also going to look at scaling out our servers to cope with the extra demand and leave more breathing room – but this means our costs will double, so we’ll need to make sure this is sustainable.

And finally, we’re going to press forward with the new version of OpenTrainTimes which builds on the six years experience we’ve had working with railway data, and will be quicker and better than the current version.

So, sorry for the problems this evening.

Peter Hicks
Director, OpenTrainTimes Ltd.

What’s new – 11th June 2017

The last fortnight has been hard work, but we’re very pleased to bring you more maps!

  • The Crewe (exc.) to Leyland map fills in the gap between the Crewe and Preston maps, covering Warrington Bank Quay and Wigan North Western stations
  • The Southampton area map now has route indications in a number of places, and more to come
  • The Letchworth to Waterbeach map has now been extended up to Kings Lynn, linking up at Peterborough (although with a massive gap in train describer coverage) and to the Ely to Norwich map
  • The Thameslink Core map now has route indications throughout, especially useful south of Blackfriars

As always, a number of smaller issues have been fixed, and we’re grateful to everyone who’s emailed or contacted us to report small inaccuracies in our maps.

And finally, we’re currently working through the backlog of support requests – there are about 30 waiting to be answered, and a mixture of quick and not-so-quick tasks. If we haven’t gotten around to replying, we will do soon.

What’s new – 28th May 2017

Although it feels like we’ve been slow to churn out new maps, you’d not think it if you looked back at progress over the last fortnight.

We’ve just made this week’s release live. In it, you’ll find the new Southampton map covering Worting Junction (now on the slightly extended Esher to Basingstoke map) to Hinton Admiral and filling in our coverage on the Hinton Admiral to Weymouth map. The West of England map has been extended too, and covers Grateley, Andover and Overton up to Worting Junction, plus Dean, Mottisfont & Dunbridge and Romsey to both Eastleigh and Redbridge.

The keener eyed amongst you will note that the Southampton map is drawn with London at the right and Bournemouth on the left. We’re in the process of revising how we draw maps of the routes out of Waterloo – our original plan several years ago is proving difficult for trains out of Waterloo, which is the only terminus in London that we consider to be in the lower-left quadrant of a fictitious circle around London. We are planning to switch the direction of the other maps, but we need to write some software to correctly realign all the elements. So, for now, please accept our apologies!

In other news, the Exeter and Westbury maps have the missing signals between the maps added.

We’ve also fixed route indications on the Darlington map, which should now show correctly for all trains entering and leaving the south side of the station.

Everything else is minor and not worth calling out individually. If you’ve reported a problem to support@opentraintimes.com, you should have had an email when the problem has been investigated and/or fixed. If you need to report an issue, it’s best by email – we can keep track of it, rather than trying to pick up potential issues from Facebook, Twitter, email and the blog!

Until next time, enjoy the Bank Holiday!

What’s new – 14th May 2017

Another Sunday is upon us, and having looked back at the last fortnight, it doesn’t seem like there’s much going on from a casual glance. But we’re hard at work on some other projects with train operators to bring an industry-strength version of OpenTrainTimes to operational staff. If you work in the industry and want to see how we can help you bring greater situational awareness to your operations, please get in touch!

We’ve had many requests to join up the Westbury and Exeter maps, covering Tiverton Parkway and Taunton, so we’ve extended the Exeter map for you! We’ve also launched our 100th map – this time of the Sunderland area, controlled by one of the Tyneside IECCs. This joins up with the Newcastle map and covers the Tyne & Wear Metro to South Hylton, and down to Hartlepool.

We’ve also fixed a very subtle bug which caused train schedules not to be properly updated from 25th April to 9th May. The problem was due to corrupt data we received, and we’ve done a thorough investigation in to what happened and how to fix it.

The next update will be in a fortnight’s time – hopefully with our Warrington area map finished, plus some of the smaller, more detailed maps for suburban parts of the railway.

What’s new – 1st May 2017

There isn’t a big update this time unfortunately – it’s hard to justify sitting in a cottage in North Yorkshire doing work when you could be out having fun – which is exactly what’s been happening! However, we have some significant things that have just gone live on the site.

  • You can now search for locations and find the right map instantly! No more having to remember which map has Long Buckby on it – just click on the Maps link, enter a station name in the "Jump to a location" box, and click the "Show map" link to go straight there. We’re still working out how to make it such that you can click on the location and it’ll jump to the map – but we wanted to release this feature early since it’s been requested many times
  • The last ten maps you’ve viewed on a device are now shown in the "Recently used maps" part of the maps page. The last map you visited is at the top, with the nine previous ones beneath it. Useful when you need to refer to several maps during the day
  • The route-based listing of maps is now tidied and the maps in each route are now hidden. Don’t panic – clicking on "East Midlands" or any of the other routes on the maps page will bring up the familiar map listing
  • And finally, the Romford to Ingatestone and Billericay map now reflects the changed track layout at Shenfield, which will be in passenger use soon. Further signals on this map will also now show a signal aspect

Minor things, worthy of a mention but not a fanfare, include Ilkeston station being added to the East Midlands maps, platform numbers fixed at Preston, route indications and Lea Bridge station added to the Bethnal Green to Chingford, Enfield Town and Cheshunt map and some signals on the East Coastway map being put in the right place.

We’re still hard at work in the lab, working on new maps, enhancements to existing maps and the next big thing. All our maps are drawn by hand during non-working hours, and we think they’re the best darned looking maps out there! We always want to bring you more, but we have to go to bed sometimes…

Until next time, stay safe!

What’s new – 17th April 2017

Over the last month, we’ve been busily working away on some other projects to keep the bills paid and the lights on. Don’t worry – we’re not going away and we’re not going to ask for donations!

This time, we have our 99th map – of Broxbourne to Shelford, neatly linking up with the Letchworth to Waterbeach map and completing our coverage of both routes up to Cambridge. We’ve actually had that map ready for a couple of weeks now, but it’s not a good idea to release new code and maps when you’re about to jump on a long-haul flight.

We’ve also updated our very popular London Bridge map has been updated for the Easter weekend works, as well as the usual bug-fixes and enhancements. Amongst the fixes are platform number and spelling mistake fixes, missing berths added, additional routes and stations added (such as Cambridge North), and a few less visible issues.

Until next time, and the 100th map, keep safe and remember: never use OpenTrainTimes for any safety-critical activity!

What’s new – 12th March 2017

We’d planned to get more done over the last fortnight, but keeping the site stable and the day-job takes precedence. But rather than delay this week’s release so we can get more done, we’re going to make the new maps live today and carry forward the other work scheduled.

We’ve also fixed routes from W1090 signal at Barnes, added a route indication to WH212 signal at Kentish Town, tidied up the Harrowgate and North London Lines maps a bit, added some more indications on, and fixed the title of, the Leeds West map and lit up 501 signal at Tonbridge.

Phew. Back to work!

Planned Maintenance – Saturday 4th March 2017

This morning, we had a problem with one of the servers that OpenTrainTimes runs on. The outage lasted for around 20 minutes in total, and our monitoring systems and internal processes worked perfectly during this time.

We need to perform some further maintenance on the site, which we’ve planned for 2200 (10pm) on Saturday 4th March. We’re allowing an hour, but anticipate the work to be much quicker.

During this time, you’ll not be able to access the site or any of the real-time track diagrams. When the site is available again, it’ll spend a few minutes catching up with the backlog of messages, and then things will be back to normal.

Storm Doris and the latest running trains

Thursday 23rd February was the busiest day ever on the site – we had over twice the number of visitors to the site and at one point, had well over a thousand users viewing our live maps. Our continual monitoring and service improvement meant we only had a single alert from our monitoring system – and that was a warning rather than an error.

Looking back over what happened that day, we’ve dug out the trains which arrived at their destination the latest. Virgin Trains’ 0840 Glasgow Central to London Euston arrived 351 minutes late at 1901 vice 1310, and on the freight side of things, the 2022 Cardiff Tidal to Mossend arrived 488 minutes late.