big data and analytics

There is more data available to transportation professionals today than any given time in the history of the subject. This is both a challenge and an opportunity. The challenges that the volume of data is so vast that it would be easy to be overwhelmed by a tidal wave of data. The opportunity lies and efficiently transforming the data to information, extracting insight, and understanding from the information and developing new response strategies and plans based on new insight and understanding. We have the possibility of developing transportation strategies and service delivery models that are much more accurate in matching the needs of the traveler. The challenge with data management from a transportation perspective, is essentially a bridge building problem. 

The various centers of transportation service provision that currently existed within a city must be networked into a patchwork world of data sources that can be shared across the city and from which analytics can be effectively conducted. Recent advances in data science assist with this immensely. Big data management systems and are available for us to handle blocks of data that in the past would have had to be split into smaller chunks another to be manageable. This new ability to handle extremely large volumes of data was initially driven by the needs of the Internet and webpages, but now provides us with new abilities to take an enterprisewide view of data. 

Because we are no longer segmenting the data, it is possible to get a clear view across all data. Our ability to turn the data into information and apply advanced analytics to the data has also progressed in leaps and bounds. We have new possibilities for understanding trends and patterns and understanding causal mechanisms. This can all contribute to better transportation service delivery across the spectrum from transportation planning to design, to project delivery, to operations, and to maintenance. To achieve all this, we must take a smart approach to data management making use of the most advanced technologies and making organizational improvements required to support the data sharing. 

Traditionally, data in transportation agencies could be described as stove piped. Held in individual silos with little connectivity between them and little understanding of what data is available. In some cases, the situation has improved, and data cockpits have emerged. They still do not enable enterprisewide data management but are configured as collections of data and tools to support a specific job or a group of staff. This improves efficiency but does not allow the ultimate efficiency of being able to use data as the utility across all organizational activities. We used to talk about the objective of database management is ensuring that data goes in once and is used many times. 

We have still not reached that goal and in many cases, data goes in many times and may not even be used once. We have experience with several agencies who have admitted that they do not use all the data at their disposal because it is perceived as too expensive and too inaccessible use. That is why smart data management is so important. It is also important in paving the way for the application of emerging data science such as Artificial Intelligence and Machine Learning to transportation. Both will need easy, structured access to big data sources.