User Tools

Site Tools


wikistudholland:utwente_qijun

Data Acquisition and integration with google map

Qijun Jiang, WUR

INTRODUCTION

General framework of the analysis

  1. Background: In order to integrate sensor networks with GIS and remote sensing to a big web. we need to know how they communicate and how the data flow works. Here just try to explore it using a small example which get metadata from http://www.knmi.nl/klimatologie/metadata/ then integrated the stations data with google map to interactively get important information.
  2. keywords: Data Acquisition, Integration, Google map

Project objectives

Data Acquisition from website, Data processing and integration with google map.

METADATA

Vector data

  • google map api

Raster data

  • google map api

METHOD

Extract data from website. Transfer data to needed format. Integrated data with google map using google map api.

Computational implementation

- bash
#/bin/bash
 
# grep keywords N.B. to find the data needed.
user@ubuntu:~$ curl http://www.knmi.nl/klimatologie/metadata/046_alkmaar.html | grep N.B.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0              
  <td width="79%">ca. 5239' N.B. 04� 45' O.L.</td>
100 12917  100 12917    0     0   107k      0 --:--:-- --:--:-- --:--:--  128k
 
user@ubuntu:~$ curl http://www.knmi.nl/klimatologie/metadata/045_delft.html | grep N.B. | awk ' {if (NR==1) print $3 $4 $6 $7 } '
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13240  100 13240    0     0   135k      0 --:--:-- --:--:-- --:--:--  172k
52�00'04�22'
 
#save data into temp.txt
user@ubuntu:~$ curl http://www.knmi.nl/klimatologie/metadata/046_alkmaar.html | grep N.B. | awk ' {if (NR==1) print $3 $4 $6 $7 } ' >temp.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12916  100 12916    0     0   105k      0 --:--:-- --:--:-- --:--:--  134k
 
#get rid of "°" & "'"
user@ubuntu:~$ egrep -o '[0-9][0-9]' temp.txt
52
39
04
45
 
#save and transfer column to row
user@ubuntu:~$ egrep -o '[0-9][0-9]' temp.txt >temp1.txt
user@ubuntu:~$ tr '\n' ' ' <temp1.txt
52 39 04 45 user@ubuntu:~$
 
#prosess each html file using for loop combined with $() and save the date to LocationDDMM.TXT
 
for d in 045_delft.html 046_alkmaar.html 055_middelburg_eo.html 056_057_breda.html 049_delft_eo.html 048_044_haarlem.html 047_bergen.html; do echo >>Location.TXT &(tr '\n' ' ' <temp1.txt >>Location.TXT $(egrep -o '[0-9][0-9]' temp.txt >temp1.txt $(curl http://www.knmi.nl/klimatologie/metadata/$d | grep N.B. | awk ' {if (NR==1) print $3 $4 $6 $7 } ' >temp.txt))) ; done 
 
#change DDMM format to DD format
user@ubuntu:~$ awk  '{ print $1+$2/60,$3+$4/60 }' LocationDDMM.TXT >LocationDD.TXT
user@ubuntu:~$ cat LocationDD.TXT
52.65 4.75
52 4.36667
51.5 3.61667
51.5833 4.78333
52.3833 4.65
52.6667 4.71667
52 4.35
0 0

Use location parameters in LocationDD.txt gained from bash to build web application.

Here I use "//" instead of "#".

- JavaScript + HTML
# my JavaScript + HTML code here
<!DOCTYPE html> 
<html> 
<head> 
  <title>Distribution of Meta data stations</title> 
  <script src="http://maps.google.com/maps/api/js?key={API_KEY}sensor=false" 
          type="text/javascript"></script>
  <style type="text/css">
			html { height: 100%; width: 100%; }
			body { height: 100%; width: 100%; margin: 0; padding: 0; }
			#map {height: 100%; width: 100%; }
  </style>
</head> 
<body>
  <div id="map" ></div>
  <script type="text/javascript">
//Define locations variable 
    var locations = [
      ['Delft, (Leiden) en Rijnsburg', 52, 4.35,],
      ['Breda', 51.58, 4.78,],
      ['Haarlem', 52.38, 4.65,],
      ['Bergen', 52.67, 4.71,],
	  ['Delft', 52.00, 4.37,],
	  ['Middelburg en Oostkapelle', 51.50, 3.62,],
      ['Alkmaar', 52.65, 4.75,]
    ];
//Define a map, zoom levels, center and default Map type
    var map = new google.maps.Map(document.getElementById('map'), {
      zoom: 8,
      center: new google.maps.LatLng(52.01, 5.01),
      mapTypeId: google.maps.MapTypeId.SATELLITE
    });
//Define a infowindow to displays content of marker
    var infowindow = new google.maps.InfoWindow();
 
    var marker, i;
//add markers through a loop
    for (i = 0; i < locations.length; i++) {  
      marker = new google.maps.Marker({
        position: new google.maps.LatLng(locations[i][1], locations[i][2]),
        map: map
      });
//through Handling Events to display the name of the markers when click happened
      google.maps.event.addListener(marker, 'click', (function(marker, i) {
        return function() {
          infowindow.setContent(locations[i][0]);
          infowindow.open(map, marker);
        }
      })(marker, i));
    }
  </script>
</body>
</html>

RESULTS

  • Distribution of Meta data stations
    Distribution of Meta data stations
wikistudholland/utwente_qijun.txt · Last modified: 2021/01/20 20:36 (external edit)