The return of OpenTrainTimes

I am very pleased to announce that Rockshore, have stepped forward and agreed to sponsor OpenTrainTimes in the mid-term. As a result, I’m delighted to say that OpenTrainTimes is now back!

I have had a steady stream of wholly supportive and positive emails over the last month from people who have used the site. Thank you to everyone who took the time to get in touch – there are more of you than I thought!

So, the site is back and running on new code which I’d been working on for some months before the site was shut down. There are a few important changes, however:

  • The beta server is gone – it was too labour-intensive to run two versions of the site. Instead, many of the features and improvements from the beta site are now on the main site
  • The layout is different – this is OpenTrainTimes version 3, considerably changed under the bonnet from previous versions, based on everything I’ve learned over the last three years
  • Some features are missing – simply down to the fact this is my hobby. I will be bringing the PPM statistics back soon, don’t worry
  • There are more maps – see below for a list, and there will be more coming
  • There are bugs – this code is fairly new, so please report bugs to feedback@opentraintimes.com. I already know about the dodgy handling of trains over midnight

Many people absolutely loved the maps, and I’m pleased to say that the following brand-new maps are now included on the site:

The previous maps (West Coast, East Coast, Thameslink/Midland Mainline) maps will be ported over in due course.

Once the new site is stable and bedded in, I’ll be working on more maps and functionality.

Covering costs

I’ve never wanted to run the site at a profit – but I need to cover costs. Some of the things I’m considering to keep the site up and running in the long term are:

  • A one-off donation
  • A regular donation
  • Subscriptions, which will give access to additional features on the site

Any excess money will be set aside to keep the site running for longer, and I’ll continue to donate my spare time and energy for the benefit of everyone.

Autumn Update

It’s been a couple of months since the last blog post and quite a long time since the site’s been updated. Several people have emailed me to ask what’s going on, and this blog post will explan everything.

This year has not been an easy year for me on a personal level. I’ve had several massive changes, including starting a new career in January and the sudden death of my father in June. On top of this, I’ve been extremely busy with my day-job and the last thing I want to do some days is come home and spend another three hours writing code! I pump a great deal of energy and passion in to my job, which is paying off, but I don’t want to burn out.

I made the unconscious decision to work on some other things for a while, setting aside working on the beta site and working on other things. About a month ago, I returned to working on the site, reviewing the code I’ve written over the past three years, and came to the conclusion that I need to make the site better by replacing it, rather than continually painting and extending it. I’m doing that job right now – which takes time, but I am in no rush.

There are also several other problems I’m tackling – one of which being exactly how far you can go with the Network Rail Data Feeds. The fact of the matter is that it’s impossible to predict much of the future from these data feeds, because there’s a lot of other data which isn’t public. Unfortunately, that situation is very unlikely to change – I’ve been trying quite vocally for the past few years, and I know far too much about all the issues at play.

In summary, I’m still very much behind the site and Open Data in general. I want OpenTrainTimes to continue to be free and useful, but to go to the next level, I need to take some time to rebuild the foundations.

OpenTrainTimes 2.2 goes in to public beta

It’s been a long time coming, but I’m pleased to say that OpenTrainTimes 2.2 is now in public beta. I’d intended to launch this a couple of months ago, but various things – including ill health and on my part and shortly afterwards, the unexpected death of my father – delayed things somewhat.

Before I get in to what’s new and changed, I’d like to point out that I run OpenTrainTimes out of my own pocket and I’ve never accepted any donations nor put advertising on the site. If you like the site and you want to give some money, please send whatever you think is appropriate to Marie Curie Cancer Care – through the OpenTrainTimes Development page on JustGiving. You can also text “OPEN67” with either £1, £2, £3, £4, £5 or £10 to 70070 to donate.

Boring bit over, here’s the exciting stuff 🙂

The new version is in public beta because I’ve not been able to test it as thoroughly as previous versions, and there still may be bugs. Please go check it out at http://beta.opentraintimes.com/ and report bugs to feedback@opentraintimes.com.

There are several extra maps:

…and some of the old maps have been upgraded:

…and two notable new features:

  • There’s more control over searching for schedules. You can hide passenger trains, empty passenger trains and non-passenger trains, as well as filter by schedule type and whether the train passes
  • The schedule route map is back – click “Functions” and “Show route map” on a schedule page

So, please go and test the new site and feed back any problems. If all goes well, I’ll make this live in a couple of weeks, or whenever the majority of bugs are squashed.

Real-time train movements

It’s been a month since I last updated you all on the plans for the future… so where are we?

First, good news – I have real-time train movements working in the development environment. This has taken several weeks of solid effort, writing test plans, writing code, fixing code and committing it. There’s reporting for the vast majority of trains, and cancellations, reinstatements and short-tripped trains are all handled well.

The bad news is that, when testing the system in an environment similar to production, it runs too slowly. There’s around a 4-5 minute lag through most of the day, which is unacceptable and I can’t release code in this state. There are two options – either upgrade the production server (which is expensive), or rewrite the code to be more efficient. I’ve opted for the latter.

Rest assured that real-time movements are still very much on the cards, along with a bunch of other features that I’ll write about next week.

Springtime Review

Thank you to the vast number of you who have taken the “How Do You Use OpenTrainTimes?” Survey. It’s open for the next three weeks, so if you haven’t responded yet, please do so. The results are interesting, useful and in some cases, quite surprising – and they’re shaping plans for the future.

One of the most requested features is real-time train movement data. I’ve been working in the background for the past couple of months on getting this up and running, and I’ve made excellent progress in the past two weeks. Assuming I continue at this pace, I’m estimating releasing OpenTrainTimes 2.2 with real-time train movements in three weeks.

Something else I’ve been working on is producing timetable data for Great Britain in GTFS format. After chatting to a few people at the FutureEverything Innovation Challenge this weekend, I realised that there are 587 transit agencies providing GTFS data, and seemingly nobody in Great Britain.

What do I normally do when there’s a gaping hole in a domain that I understand and care about? I fix it, and I’ve created data.opentraintimes.com which will have a daily-updated GTFS-format snapshot of GB timetable data. There are just two more bugs to fix relating to trains which run overnight, but I’m anticipating fixing these early this week.

On to this week’s updates – as I’ve been so busy doing other things, there are only a few minor bugs which I’ve fixed:

  • The London Waterloo – Earlsfield/Chiswick/North Sheen map had platform numbers at Barnes and Putney labelled incorrectly. Note that due to caching, you might still see the old map for a couple of hours)
  • When using Internet Explorer 9, the Feedback link was rotated incorrectly
  • Trains at Stratford Parkway (STY) weren’t showing
  • Times on the homepage were an hour behind reality

Expect a period of relative quiet over the next couple of weeks whilst I get real-time data up and running – but please, as always, keep your feedback coming!

// Peter

OpenTrainTimes 2.1 is here

It’s been a long time coming, but OpenTrainTimes 2.1 is here.

The biggest change, and the one that’s taken weeks of effort, is that the timetable database is faster, and I’ve been able to introduce some new features surrounding it:

  • You can search for arrivals, departures or all trains at a location
  • The Functions link on a location page will allow selection of a 24-hour time window
  • Train schedule pages have been reworked in Working Timetable (WTT) mode

Whilst it’s not quite as fast as I’d like it to be, I am working on making it even faster.

There are a few things that are currently missing, as I haven’t been able to re-implement them in time – namely the route maps for trains and the facilities and technical tabs – but I am working on these now and hope to have them back in a few days, along with several more features (including real-time data) in the coming weeks.

As always (and I say this every time) – thank you to everyone for your feedback and suggestions. Please try out the new site and let me know your comments and suggestions.

Where we are, and where we're going

Freight trains are now on the site, albeit with the headcode masked out in schedules, and with a scrambled placeholder on the real-time maps.

As I predicted, this resulted in a massive spike in traffic to the site which has all but disappeared as most people are using other sites such as www.realtimetrains.co.uk. Although it doesn’t have all the polish of OpenTrainTimes, it real-time information and is fast.

So why isn’t OpenTrainTimes as fast as RealTimeTrains?

There are two reasons.

First, the database format I use is robust and mature, but isn’t as fast as I want it to be. I’m in the process of re-engineering the database to be much quicker – something I hoped to launch today, but as I have a full-time consultancy job and an awesome lady in my life, I don’t have much time during the week to sit down and write code. I also underestimated the amount of time it would take to convert the existing database, so I’m running a bit behind schedule.

The new database format is denormalised – something that makes it quick to read, but slightly more difficult to update unless you’ve architected your system properly. Whilst testing, I found I could bring up all the trains at Clapham Junction in a single day in around 2 seconds. It takes more than 30 seconds to do this in the current system. Good news!

I’m hoping to launch the restructured, quicker database, some time this week (possibly next Sunday) once I’ve made sure everything works better than it currently does.

The second reason is that the amount of money I spend on hosting the site is small. The reason www.realtimetrains.co.uk is lightning fast is due, in part, to the fact it’s running on a system that is many times more powerful than I have at my disposal.

I could throw money at the problem, but I have rent, bills and savings. I could introduce adverts on to the site, but given the user base, few people are likely to click them and they’d make the site look ugly – something I’ve tried to avoid. Or I could ask people to donate, or subscribe – but this changes the game entirely and means that I’m obliged to provide a certain level of service.

What’re you going to do?

I’m pinning my hopes on the restructured database making things faster – which means I can work on putting more real-time data on the site. However, if it doesn’t speed things up, then I’m considering throwing in the towel and just showing statistics and real-time track maps. There’s not much point in having the largest and most visibile part of the site being the slowest and most resource intensive!

What else are you planning

After the new database is in place, and if it’s fast enough, I’ll be able to launch a bunch of new features such as email and instant message alerting, tracking trains in real-time and being able to click on the signalling maps to bring up schedule information. Just about everything I want to do relies on having a fast timetable database underneath the site, so it’s a critical thing to get right.

I’ve also got some other ideas up my sleeve which aren’t ready for the public eye yet, as they involve data which isn’t yet open and available to the public.

So – fingers crossed and let’s everything goes to plan!

New site release

It’s a week since I relaunched the site, and there’s been a steady stream of messages coming in reporting bugs, offering thanks or suggesting new features. Please, keep them coming!

I have just deployed new code to the server which should fix these bugs:

  • The signalling map sometimes didn’t load berth data, resulting in seeing a screen with four numbers in boxes
  • Bedford South Junction was missing from the Bedford – Luton map
  • The ‘favourite’ icon wasn’t appearing in browsers
  • The feedback button sometimes overlapped, invisibly, some schedules making it impossible to click the ‘i’ button
  • Allowances were shown in hours and minutes, and are now shown in minutes

There’s a bunch more work to do this week. Amongst other things, I’m working on speeding the site up – there are times when it’s impossibly slow for seemingly no reason. I’m also half way there with a map of London Euston – Hatch End (Wembley Mainline) and I hope this will be in next week’s release.

Thank you to everyone for the continued support and lovely emails!

On Freight

Here are possibly my final words on opening up freight schedules.

Freight schedules are commercially sensitive. Real-time data on freight trains is commercially sensitive. Freight operating companies (FOCs) don’t want the whole world to know about their operations because they’re in competition with other modes of transport – primarily road haulage. This makes the data quite different from passenger TOCs because there’s no outside competition.

I’ve tried to come up with a compelling justification for opening up freight schedules and real-time data, but I’m afraid I simply cannot find one. A few people have been in touch with me with legitimate reasons to analyse freight data – for example, in transport planning to construct bus timetables such that they don’t use level crossings around the time freight is due. However, the vast majority of people are enthusiasts, some of whom think everything should be free and don’t understand that commercial organisations are there to make money, not to service the enthusiast community.

You can get ‘gen’ about freight trains if you look in the right place. Yahoo! Groups has several groups just for freight workings, including consists (the loco and wagon numbers). Most of it appears to be supplied by insiders with access to the right IT systems and then filtered out gradually through a network of others. It’s is a process that’s been going on for years. I won’t comment on whether I agree with it or not – it’s not relevant – but I know this process will continue.

And now, the final words on whether freight will ever appear back in OpenTrainTimes – it’s very unlikely unless the Department for Transport, Network Rail and freight operators change the situation from above. I’m not going to pursue the issue any more – it was never on my agenda in the first place, and isn’t something I have time to fight for any more. However, if a freight company is willing for their schedules to appear on OpenTrainTimes, please contact me via email and I can make it happen.

Intermittent outage

Sorry if you’ve been unable to access the site over the last 48 hours – there was a problem with one of the DNS servers that handles opentraintimes.com which was directing people at an old server for the site. It’s fixed now.

In case you want the details – the primary nameserver for opentraintimes.com has been upgraded and reinstalled, but the BIND configuration I put back was a copy from several months ago before I migrated from Amazon to Rackspace. The serial number on the zone was lower than that on the secondary servers for the domain, so none of the secondaries picked it up. It’s a case of luck as to which server you get sent to, so there was roughly a 1-in-5 chance of picking up the server with rogue information.

Backups are good, but restoring the right one is even better!