The current increase in monitoring systems, supported by robust and high-throughput communication links, has led to the creation of massive point based high temporal-fidelity datasets. However, in many cases the sensor observations are streamed through multiple fractured data platforms. The individual platforms are aimed at various end-users and are not uniformly compatible with various scientific applications, posing formidable technical and logistical challenges. For example, one of the largest publicly available monitoring systems is the expanding set of surface weather observations. The weather data from the Greater Chicago area is available in several forms including a standard enterprise level upstream provider network (LDM), a series of XML HTTP request, and standard ASCII streams. These data forms, their disseminating platforms and communication mechanisms pose challenges due to the wide heterogeneity. We have realized a framework for aggregating weather observations from the Chicago metropolitan area. The framework aggregates weather
data from multiple sources and makes the aggregated data available in NetCDF format through an OpenDAP portal. The current system outputs one weather data point every 10 seconds for every available station. The format of the NetCDF file was selected to follow accepted conventions and allow additional processing using standard CF-1.X compliant analysis software. The complete platform has been designed around open standards and libraries.

In this talk I will describe the architecture of this platform and the decision making process behind the major technical components. I will briefly discuss the sources of data acquisition, the quality control process and the OpenDAP portal that provides a unified assess method to all of the resulting data products. A couple of example weather products generated using the platform will be presented. The talk will end with a brief discussion on the limitations of the platform and proposed methods of improvement.

Bio:

James Bittner has a Bachelors of Science in Civil Engineering from Michigan Technological University in 2008, and a Masters of Science in Civil Engineering from University of Illinois at Chicago in 2012. He will be joining the PhD program in Civil Engineering at University of Illinois in Urbana/Champaign in the Fall of 2013. He has also worked in industry on problems involving the estimation of land reclamation contracts and researched superior methods of hydraulic slurry transport engineering.