Earlier today, the OpenTrainTimes website was down. The root cause was that one of the disks was full, which we’ve fixed and put our recovery process in to action. Given the time it takes to replay a backlog of messages – a problem we’ve had before – it took until about 3pm for us to catch up on the missed messages.
Although we have plenty of monitoring in place to alert us of problems such as this, we’ve identified – in this case – that the monitoring was set up for the wrong server. Consequently, nothing alerted us to the imminently-filling disk.
We’ll be fixing this over the coming days.
We’re sorry for the extended degradation in service today. We don’t take outages lightly, and we’ll be looking at ways to stop this happening in the future.
I’ve seen a couple of reports of missing services from OTT that appear on other sites. Were any of the queued updates from vstp dropped?
The site doesn’t process VSTP messages at the moment – it’s an architectural problem which we’re fixing with the next major release
The data is hours out when on maps at Newcastle station. It’s 1100 now and showing data from around 0500. It happened since the outage the other day.
This is due to a separate, unrelated outage outside our control and affecting all Open Data sites
Ah ok thanks.. it seems to be catching up now.
seems to be out of sync again this lunchtime.
Rgds
Saying the above it has cleared back to normal at the moment again
Peter,
Once you are seeing stable current time data perhaps you could archive the backlog of data, repopulate the diagrams with current data say from the last 20 minutes and once the displays are up to date populate the train listings from the archived data?
Sorry I am not a programmer just a bug detector, who occasionally made the correct necessary fix suggestion. The programmers where I used to work saw me as necessary evil if it helped make the product work reliably.
What has happened to the map at yate, the signals are working wrong, it seems the tvsc centre has gone funny, have sinallers noticed.