{"id":228,"date":"2019-03-14T09:54:17","date_gmt":"2019-03-14T08:54:17","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/data-gathering-and-knowledge-management-for-smart-cities-in-snap4city-platform\/"},"modified":"2020-01-31T11:26:58","modified_gmt":"2020-01-31T10:26:58","slug":"data-gathering-and-knowledge-management-for-smart-cities-in-snap4city-platform","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/data-gathering-and-knowledge-management-for-smart-cities-in-snap4city-platform\/","title":{"rendered":"Data Gathering and Knowledge Management for Smart Cities in Snap4City Platform"},"content":{"rendered":"

Imagine you are in the monitoring and control room of your city, where all relevant data<\/span> coming from sensors, mobile devices<\/span>, social media streams, IoT devices, vehicles and so on are displayed on multifunctional dashboards. Imagine you can read on those dashboards predictions and user behaviour analysis, which can support your decisions and address your strategies<\/span>. As IT professionals, we know this is the work of separate, interlaced technologies, not magic, and we like to move from the initial sense of wonder to understanding and confidence.<\/p>\n

Snap4City, an open-source platform that allows us to create solutions for smart cities<\/span>, is the best testbed to learn how this magic happens. Snap4City is developed by DISIT Lab (Distributed Systems and Internet Technologies Lab) of University of Florence and is currently in use to aggregate open data<\/span> and private data<\/span> from and to some local administrations in Italy<\/span>.<\/p>\n

Being a framework that must produce and provide some kind of value from data exploitation, at Snap4City’s core is the ability to aggregate and integrate data<\/span> from different providers, in different protocols.<\/p>\n

The 5 Vs of Smart Cities Data<\/h2>\n

The aims, problems and challenges for Snap4City are to turn data<\/span> from disparate sources into actionable information. In fact, Snap4City provides a platform able to ingest and take advantage of large amounts of spread data<\/span>, exploiting data integration and reasoning to deliver new services and application to citizens and administrators.<\/p>\n

\"\"<\/p>\n

Img 1: Private and public data, from static and real-time sources<\/em><\/p>\n

This data can be provided in many different protocols and formats, and from many different sources. Moreover, the data<\/span> is not aligned, for example, the same street names, dates or tags may differ when provided by different sources.<\/p>\n

Snap4City must be able to manage a diversity of data types – variety<\/strong> – that are being created with rapidly increasing speed by technological advances – velocity<\/strong>. This huge amount of information – volume<\/strong> – can change its exact flow from time to time – variability<\/strong> – and the infrastructure required to collect and interpret data<\/span> must produce insights – value<\/strong>.<\/p>\n

Those are the five relevant characteristics of a big data problem\/solution, a.k.a. \u201cthe 5 Vs of big data<\/span><\/strong>\u201d.<\/p>\n

Modalities and Strategies for Data Ingestion<\/h2>\n

Snap4City platform gathers information from several sources. Data<\/span> can be provided in any format and via any protocol. This means that data<\/span> can be both structured and non-structured, as well as flowing as static data<\/span> or as real time data, with or without metadata descriptors.<\/p>\n

Static data<\/span> can be imported in Snap4City via data<\/span> driven, stream, sporadic and\/or periodic processes. Static data<\/span> is typically ingested with DataGate for automated ingestion, while both static and real-time data<\/span> can be ingested through Node-RED, Apache NiFi, and custom procedures in the form of ETL processes (Extract, Transform, Load). Several different formats and structures can be addressed by creating specific ETL processes for each data source family.<\/p>\n

\"\"<\/p>\n

Img 2: Schematic of semantic data aggregation in Snap4City\/Km4City<\/em><\/p>\n

The mined\/acquired data is subsequently stored into noSQL database<\/span>. From here, information can be exploited for data analytics, dashboards, etc.<\/p>\n

ETL Processes for Data Ingestion<\/h2>\n

ETL processes are used for data gathering, collecting files from HTTP\/FTP protocols. Some ETL processes that have been developed and are now in place to manage data ingestion for the Smart City<\/span> of Florence and Tuscany are accessible on DISIT Lab Github project page<\/a>.<\/p>\n

These include management<\/span> for data<\/span> from traffic sensors, parking lots, weather forecast, cost of fuels, environmental data<\/span>, etc. Examples of sources and data ingestion processes by ETL and Snap4City applications are crawling public web pages for collecting hospital triage statuses, periodically reading from web server<\/span> data in GTFS format about public transportation schedule, stops, paths, etc., integrating civic number location from OpenStreetMap to Snap4City Knowledge Base.<\/p>\n

\"\"<\/p>\n

Img 3:  Example of batch processing in Snap4City integrated ETL development environment for dynamic data ingestions<\/em><\/p>\n

Node-RED Blocks for Data Ingestion<\/h2>\n

To manage and ingest real time data and for IoT applications, Snap4City provides several Node-RED blocks for accessing data<\/span> in a specific format (XML, CSV, HTML, YALM, JSON) and translates them into the JavaScript Object representation. Once the data<\/span> is represented in this target format, it can be easily and uniformly processed and transformed for the application needs.<\/p>\n

The same Node-RED blocks can be exploited for exporting a value in the JavaScript Object representation in the corresponding format. Once the data<\/span> has been processed, it can be visualised on a dashboard or stored in a Big Data Storage and\/or Knowledge Base by using dedicated Node-RED blocks.<\/p>\n

\"\"<\/p>\n

Img 04: Example of Node-RED flow in Snap4City integrated development environment<\/span> for event driven, real time data ingestion<\/em><\/p>\n

Detailed information about how to create ETL process and Node-Red flows in Snap4City can be found on the project website<\/a>.<\/p>\n

Semantic Modeling and City Knowledge Base<\/h2>\n

Being strictly related to data<\/span> moving to and from sensors and actuators placed on territory, Snap4City must deal with complex and interrelated geospatial info. For example, Snap4City must be able to manage restricted traffic zone gates, or environmental sensors for air quality, pollution, rain or public light pillars, must know public buses routes or POIs inside a defined area and so on.<\/p>\n

Any entity managed by Snap4City is mapped into a smart-city dedicated ontology<\/a>. This ontology and the related knowledge model enables the description of smart cities<\/span>. It is not small and can be viewed as consisting of various macro classes or macro categories:<\/p>\n