Big Data Visualisation
Find out what happened when we combined our BIG data expertise with open data from Network Rail
Last year, Network Rail provided public access to all UK train movement data for the first time. This enabled data scientists to analyse the train data in many different ways. Here at RDF Group we were particular interested to review train performance data at a station level of granularity. We decided to present the data of train lateness visually as heat maps. We have been looking at Big Data technology and ways of offering the analysis capabilities it enables to our customers. We felt the storing, processing and visualisation of train movement data would be a good test of our Big Data infrastructure, technology stacks and demonstrate our technical capabilities. Also it would be a fun, interesting and rewarding project to do.
Over a three month period, starting in late 2013, four RDF Group developers set up an Agile team to investigate and trial various Big Data technologies. As a result of the project, several technologies were adopted. The types of Big Data technology that we looked at included products such as Hadoop, Cloudera, HBase, Stomp, Map Reduce, Flume and Oozie.
- Data Collection and storage using Gozirra, Flume and HBase, by Shaun Dugan.
- Storing and querying geospatial and temporal data, by Mark Waldron.
- Extracting data from HBase, by Andy Palmer.
- Raildata Visualisation, by Dave Hampton.