Post-Incident Review

We had some problems with OpenTrainTimes earlier today. Although the public site is not operated for profit, we take uptime seriously and we’ve produced this review of what happened.

If you use OpenTrainTimes as part of your job and you’re interested in a commercially supported version of the site, including freight data and integrations with your stock and crew systems, please drop us a mail at hello@opentraintimes.com.

What happened?

Earlier this evening, we had multiple users reporting that maps on OpenTrainTimes were lagging.

Upon investigation, we found an unusually large number of users on the site for the time of day combined with a 45 minute backlog on our train describer feed.

We temporarily disabled the train movement feed and turned off logging for the real-time maps in order to process this backlog. Once the backlog had cleared, we turned the train movement feed back on and monitored the service whilst the backlog of TRUST messages cleared.

The site returned to normal operation by about 2045.

More detail

OpenTrainTimes is a very popular site, and several hundred users are usually viewing multiple maps at the same time. This figure grows steadily and gradually over time, and we review our capacity every few months to make sure we’re not caught out. Each time we release a new map, the base load on our servers increases as we have anything up to 500 new pieces of signalling data to process – and then there are the extra users that the maps attract.

But that wasn’t the issue – but not by the number of users, but by the type of users!

Briefly, when a user’s web browser connects to our map server, it either uses a long-lived connection over which map data is sent, or it requests map data every few seconds. Several things influence which is chosen, but it’s usually down to whether the device is behind a proxy server – not all proxy servers allow, or support, long-lived connections over websockets.

This evening, we noticed a larger than normal number of polling users. Since a large percentage of OpenTrainTimes users are coming from a mobile device, we think this may be because a change was made at one of the mobile network providers which meant our websocket implementation couldn’t be used by clients.

Normally, this is OK – but the gradual and continual increase in users each week, coupled with a gradual surge in the number of connections that our server was logging, meant there was insufficient CPU time available to process all of the data coming in to us from Network Rail.

The first thing we did was to turn off logging – we don’t really need it day-to-day, and it bought us some time. We then switched off processing TRUST messages, allowing them to queue whilst we allocated the rest of the server’s capacity to processing the backlog of train describer (TD) messages. It took about 20 minutes to process the TD messages, after which we turned TRUST messages back on. Processing the backlog of those messages, plus the remaining TD messages took about another hour.

What we’re going to do about it

First of all, we’re sorry that we missed a trick and took too long to respond to the initial reports of a problem.

We’re going to add some new health checks to our monitoring system, one of which will enable us to monitor the size of any message backlog.

We’re also going to look at scaling out our servers to cope with the extra demand and leave more breathing room – but this means our costs will double, so we’ll need to make sure this is sustainable.

And finally, we’re going to press forward with the new version of OpenTrainTimes which builds on the six years experience we’ve had working with railway data, and will be quicker and better than the current version.

So, sorry for the problems this evening.

Peter Hicks
Director, OpenTrainTimes Ltd.

What's new – Sunday 25th October 2015

Summer’s officially over – the mornings aren’t as light, and the evenings are darker. To make up for this, there are four new or updated maps!

  • The York map has been extended to Thirsk
  • The Darlington map map covers Thirsk, Northallerton, Darlington and Durham, plus the Northallerton to Eaglescliffe and Darlington to Eaglescliffe routes
  • The Newcastle map covers Durham to Newcastle, plus Dunston, Manors and Heworth
  • The Sileby to Langley Mill map has been redrawn, covering Sileby to Sheet Stores Junction, some of the route to Stenson Junction and Derby, plus Toton and Beeston

As always, I’ve been hard at work fixing the smaller bugs that you’ve been reporting – the highlights are:

  • The Crewe map includes ‘train ready to start’ indications for the platforms at Crewe
  • The Exeter map includes has some fixes which caused trains to disappear around Dawlish (no, not in to the sea!)
  • Sometimes, map elements would appear with large yellow sections due to a code problem – that’s been fixed

I have a well-earned holiday coming up shortly, so expect some non-map related features in the next release, probably in a fortnight’s time!

Solstice Release

Hello again, after a two week break. It feels like I’ve been flat out over the last couple of months, churning out maps like there’s no tomorrow.

I’ve managed to fix a handful of small problems with the maps:

There are also some other bugs that I know about and am trying to fix:

  • On the Charing Cross, Cannon Street and London Bridge to Forest Hill map, trains never step out of the berth for signal L156, and are (seemingly) manually interposed in to the berth for signal L148. This could be a fault with the train describer itself, which I’ll have to work around
  • There are occasional problems with some trains being linked to schedules which are not the right ones
  • Some signals and TRTS indications always show on when they’re not – I believe this is linked to the problem above

I’m working hard to try and get them fixed, so please accept my apologies for the fact it’s taking a while.

That’s it for this week – the next maps are a surprise (read that as "I haven’t decided which ones to tackle next").

// Peter

Happy Birthday, OpenTrainTimes

On Monday 10th January 2012, I launched OpenTrainTimes.  I believe it marked a turning point in opening up Great Britain’s railway data – leading the way and showing that it can be done, and that the outcome would be positive.

We’ve come a long way in those three years.  Network Rail opened up detailed real-time data through their Data Feeds platform and have opened up their timetable, fares and associated data through their Rail Industry Data portal.

Most recently, National Rail Enquiries opened up their Live Departure Boards web service and loosened their terms and conditions so that nearly anyone can work with the data.

In the coming months, the pièce de résistance will be unveiled – open and scaleable access to Darwin – one of the most important systems that produces a ‘single source of truth’, whose information is distributed to websites, mobile phones, station departure boards and numerous other technology platforms.

If you’re familiar with OpenTrainTimes, you’ll realise that it only uses two or three of the available data sources.  There are two very good reasons for that.  First, it’s a more compelling argument when you can appraise somebody else’s work as a talking point and reason to open up data than when you’re presenting your own.  Secondly, OpenTrainTimes started off as an experiment and I never expected it to be as popular as it is.  The architecture has run in to a number of scalability issues which can only be fixed with a lot of behind-the-scenes work.

I’ve been putting in many hours of work each week, outside the time I spent working for Rockshore to improve the railway’s real-time systems, to re-build the entire site and make it even more successful than it is.

I am not quite at the end yet – the majority of the heavy lifting’s already been done and the scaffolding’s starting to be taken down.  There are a few more weeks of testing I need to do to iron out bugs and make sure the new site performs much better than the other did.  Please get in touch if you’re interested in helping test the new site.

When I launch the new site, probably in a month or so’s time, it’ll include real-time data from more feeds at Network Rail and, once the Darwin feed from National Rail Enquiries launches, it’ll include forward-looking predictions that mirror what you see on other systems powered by Darwin.  No “Your site says X, but the National Rail site says Y, how do I don’t know who’s right?” – consistency trumps accuracy in predicting when a train will turn up.

Thank you to everyone who’s helped me out – especially to the numerous industry people who have kept my enthusiasm up and made helpful suggestions on where to go next.

Watch this space – a new OpenTrainTimes is around the corner!

Scheduled downtime – Saturday 21st June 2014

OpenTrainTimes is a popular site – so much so that it’s necessary to upgrade one of the servers that runs the site.

To do this, there will be an outage from approximately 0900 to 1300 on the morning of Saturday 21st June, and the site will be offline during this time.

After this upgrade is complete, there will be plenty of spare capacity on the site to allow the next set of exciting features to be developed… watch this space!

Happy Easter

A very Happy Easter to you all!

A brief update on some changes on the site, based on feedback I’ve had over the past few days:

There are also some bugs which have been reported, which I’m still working on:

  • Maps do not display fully on Internet Explorer – and it’s only a problem on Internet Explorer. I’m currently investigating.
  • Maps are very slow on some browsers – which appears to be a problem with the Javascript code I’m using. Again, I’m currently investigating.

Finally, thanks to everyone for your feedback – it’s great to be back and developing OpenTrainTimes!

The return of OpenTrainTimes

I am very pleased to announce that Rockshore, have stepped forward and agreed to sponsor OpenTrainTimes in the mid-term. As a result, I’m delighted to say that OpenTrainTimes is now back!

I have had a steady stream of wholly supportive and positive emails over the last month from people who have used the site. Thank you to everyone who took the time to get in touch – there are more of you than I thought!

So, the site is back and running on new code which I’d been working on for some months before the site was shut down. There are a few important changes, however:

  • The beta server is gone – it was too labour-intensive to run two versions of the site. Instead, many of the features and improvements from the beta site are now on the main site
  • The layout is different – this is OpenTrainTimes version 3, considerably changed under the bonnet from previous versions, based on everything I’ve learned over the last three years
  • Some features are missing – simply down to the fact this is my hobby. I will be bringing the PPM statistics back soon, don’t worry
  • There are more maps – see below for a list, and there will be more coming
  • There are bugs – this code is fairly new, so please report bugs to feedback@opentraintimes.com. I already know about the dodgy handling of trains over midnight

Many people absolutely loved the maps, and I’m pleased to say that the following brand-new maps are now included on the site:

The previous maps (West Coast, East Coast, Thameslink/Midland Mainline) maps will be ported over in due course.

Once the new site is stable and bedded in, I’ll be working on more maps and functionality.

Covering costs

I’ve never wanted to run the site at a profit – but I need to cover costs. Some of the things I’m considering to keep the site up and running in the long term are:

  • A one-off donation
  • A regular donation
  • Subscriptions, which will give access to additional features on the site

Any excess money will be set aside to keep the site running for longer, and I’ll continue to donate my spare time and energy for the benefit of everyone.

Autumn Update

It’s been a couple of months since the last blog post and quite a long time since the site’s been updated. Several people have emailed me to ask what’s going on, and this blog post will explan everything.

This year has not been an easy year for me on a personal level. I’ve had several massive changes, including starting a new career in January and the sudden death of my father in June. On top of this, I’ve been extremely busy with my day-job and the last thing I want to do some days is come home and spend another three hours writing code! I pump a great deal of energy and passion in to my job, which is paying off, but I don’t want to burn out.

I made the unconscious decision to work on some other things for a while, setting aside working on the beta site and working on other things. About a month ago, I returned to working on the site, reviewing the code I’ve written over the past three years, and came to the conclusion that I need to make the site better by replacing it, rather than continually painting and extending it. I’m doing that job right now – which takes time, but I am in no rush.

There are also several other problems I’m tackling – one of which being exactly how far you can go with the Network Rail Data Feeds. The fact of the matter is that it’s impossible to predict much of the future from these data feeds, because there’s a lot of other data which isn’t public. Unfortunately, that situation is very unlikely to change – I’ve been trying quite vocally for the past few years, and I know far too much about all the issues at play.

In summary, I’m still very much behind the site and Open Data in general. I want OpenTrainTimes to continue to be free and useful, but to go to the next level, I need to take some time to rebuild the foundations.

OpenTrainTimes 2.2 goes in to public beta

It’s been a long time coming, but I’m pleased to say that OpenTrainTimes 2.2 is now in public beta. I’d intended to launch this a couple of months ago, but various things – including ill health and on my part and shortly afterwards, the unexpected death of my father – delayed things somewhat.

Before I get in to what’s new and changed, I’d like to point out that I run OpenTrainTimes out of my own pocket and I’ve never accepted any donations nor put advertising on the site. If you like the site and you want to give some money, please send whatever you think is appropriate to Marie Curie Cancer Care – through the OpenTrainTimes Development page on JustGiving. You can also text “OPEN67” with either £1, £2, £3, £4, £5 or £10 to 70070 to donate.

Boring bit over, here’s the exciting stuff :)

The new version is in public beta because I’ve not been able to test it as thoroughly as previous versions, and there still may be bugs. Please go check it out at http://beta.opentraintimes.com/ and report bugs to feedback@opentraintimes.com.

There are several extra maps:

…and some of the old maps have been upgraded:

…and two notable new features:

  • There’s more control over searching for schedules. You can hide passenger trains, empty passenger trains and non-passenger trains, as well as filter by schedule type and whether the train passes
  • The schedule route map is back – click “Functions” and “Show route map” on a schedule page

So, please go and test the new site and feed back any problems. If all goes well, I’ll make this live in a couple of weeks, or whenever the majority of bugs are squashed.

Real-time train movements

It’s been a month since I last updated you all on the plans for the future… so where are we?

First, good news – I have real-time train movements working in the development environment. This has taken several weeks of solid effort, writing test plans, writing code, fixing code and committing it. There’s reporting for the vast majority of trains, and cancellations, reinstatements and short-tripped trains are all handled well.

The bad news is that, when testing the system in an environment similar to production, it runs too slowly. There’s around a 4-5 minute lag through most of the day, which is unacceptable and I can’t release code in this state. There are two options – either upgrade the production server (which is expensive), or rewrite the code to be more efficient. I’ve opted for the latter.

Rest assured that real-time movements are still very much on the cards, along with a bunch of other features that I’ll write about next week.