{ "cells": [ { "cell_type": "markdown", "id": "c70eeba2", "metadata": {}, "source": [ "# Google Earth Engine use via Python, containers and other mythical beasts\n", "\n", "The easiest way to run any _Python_ stuff on a SE VM (actually an OSgeolive 15 image with some local stuff) is running **Docker** [here](https://docs.docker.com/get-docker/). Which is also a modern way of running stuff in _isolated environments_ aka _containers_.\n", "\n", "Specifically, it allows to run a Jupytarlab instance, using an official image taken from **Docker Hub**.\n", "\n", "All information are available [here](https://github.com/jupyter/docker-stacks) and the full documentation is https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html.\n", "\n", "Different flavors of Jupyter _docker stacks_ can be installed and run under docker, they are illustrated here: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html\n", "\n", "So one can add an official image to SE VM with:\n", "\n", "\tdocker pull jupyter/scipy-notebook \n", "\n", "which is one of most complete Docker image and includes an **Anacoda** installation. After that one can use:\n", "\n", "\tdocker run -p 10000:8888 -v ${PWD}:/home/jovyan/work jupyter/scipy-notebook\n", " \n", "note that the container run as non privileged user (ID=1001 in this case) all time, so you should take care of the ownership of the files." ] }, { "cell_type": "markdown", "id": "d1108b8b", "metadata": {}, "source": [ "First of all, note that the default `jovyan` user is not included in sudoers and has a disabled password. So, in order to run privileged commands one has to manage a few chages by using a root interactive session on the running container:\n", "\n", "\tdocker ps\n", "\tdocker exec -it -u 0 /bin/bash\n", "\t[ ... make changes ... ]\n", "\tdocker commit \n", "\n", "Within this session it is possible to add a few useful stuff to the base image, e.g.\n", "\n", " apt install gdal-bin gdal-dev\n", " python3 -m pip install rasterio\n" ] }, { "cell_type": "markdown", "id": "03b45c82", "metadata": {}, "source": [ "Let's try to use Google Earth Engine directly in the SE VM, it _could_ conflict with other stuff, but we could eventually revert to use **Docker** in case of problems." ] }, { "cell_type": "markdown", "id": "77a7affa", "metadata": {}, "source": [ "The basic use of Python API is very similar to the Javascript one." ] }, { "cell_type": "code", "execution_count": 1, "id": "511df530", "metadata": {}, "outputs": [], "source": [ "#!python3 -m pip install earthengine-api" ] }, { "cell_type": "code", "execution_count": 2, "id": "b1949eee", "metadata": {}, "outputs": [], "source": [ "#!python3 -m pip install earthengine-api --upgrade" ] }, { "cell_type": "code", "execution_count": 3, "id": "0282a0f7", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

To authorize access needed by Earth Engine, open the following\n", " URL in a web browser and follow the instructions:

\n", "

https://code.earthengine.google.com/client-auth?scopes=https%3A//www.googleapis.com/auth/earthengine%20https%3A//www.googleapis.com/auth/devstorage.full_control&request_id=tYrbiziekwJR5KGf8BSkdFz7V1VPUdU_YQn23qKzNgY&tc=jtvL0uK4Ko5K50yD7mYdSYSjC-R7v76VnmCvlnBmhx0&cc=POqtZ-KRzgCTNgvrBUTxf-DVnUmy1nGid2GD16c2izg

\n", "

The authorization workflow will generate a code, which you should paste in the box below.

\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Enter verification code: 4/1AbUR2VMAhSyJisx3XRgzXUIu5RHLkY1xnwsIWJOr-Agl2jmeK1J_neggo40\n", "\n", "Successfully saved authorization token.\n" ] } ], "source": [ "import ee\n", "ee.Authenticate()\n", "ee.Initialize()" ] }, { "cell_type": "code", "execution_count": 4, "id": "a516686a", "metadata": {}, "outputs": [], "source": [ "import folium" ] }, { "cell_type": "code", "execution_count": 5, "id": "6ab09296", "metadata": {}, "outputs": [], "source": [ "lat, lon = 45.77, 4.855" ] }, { "cell_type": "code", "execution_count": 6, "id": "214ee81b", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mymap = folium.Map(location=[lat, lon], zoom_start=10)\n", "mymap" ] }, { "cell_type": "markdown", "id": "c0f061d3", "metadata": {}, "source": [ "An easier way of integrating graphics to the regular `ee` class methods is using the [Geemap](https://geemap.org) third-party package. It can use both `folium` or `leaflet` Javascript clients. The choice of the client depends on the status of their support." ] }, { "cell_type": "code", "execution_count": 7, "id": "b57f9acb", "metadata": {}, "outputs": [], "source": [ "#!python3 -m pip install geemap" ] }, { "cell_type": "code", "execution_count": 8, "id": "8ee1ddfd", "metadata": {}, "outputs": [], "source": [ "#!python3 -m pip install jupyter_contrib_nbextensions" ] }, { "cell_type": "code", "execution_count": 7, "id": "2d8c8275", "metadata": {}, "outputs": [], "source": [ "import geemap" ] }, { "cell_type": "code", "execution_count": 8, "id": "e03aecbf", "metadata": {}, "outputs": [], "source": [ "Map = geemap.Map(center=[40,-100], zoom=5)" ] }, { "cell_type": "markdown", "id": "2ea6bf31", "metadata": {}, "source": [ "\n", "You need to autheticate (eventually with `2FA`) every time you use GEE, even via Python. Note that a one time token is typically used, but it is required to authorize the Jupyter client for use of GEE.\n", "\n", "If the `ipyleaflet` widget does not appear, typically you need to restart the IPython kernel." ] }, { "cell_type": "code", "execution_count": 9, "id": "56636a40", "metadata": {}, "outputs": [], "source": [ "Map.add_basemap('Esri.OceanBasemap')" ] }, { "cell_type": "code", "execution_count": 10, "id": "8d29b477", "metadata": {}, "outputs": [], "source": [ "Map.add_basemap('Esri.NatGeoWorldMap')" ] }, { "cell_type": "code", "execution_count": 11, "id": "2ebcf709", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4ed466772d47476382d31c0f4da631e1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[40, -100], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(children=(T…" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Map" ] }, { "cell_type": "code", "execution_count": 12, "id": "695e6168", "metadata": {}, "outputs": [], "source": [ "Map2 = geemap.Map(center=[40,-100], zoom=4)" ] }, { "cell_type": "code", "execution_count": 13, "id": "e5da8301", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1362e7e6ef99481f9e1f51c09615b46a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[40, -100], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(children=(T…" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Map2" ] }, { "cell_type": "markdown", "id": "5c55e430", "metadata": {}, "source": [ "This is the common way to use `folium` instead of `leaflet` in the notebook. Currently seems the most stable way of using Geemap." ] }, { "cell_type": "code", "execution_count": 14, "id": "9c6690af", "metadata": {}, "outputs": [], "source": [ "import geemap.foliumap as geemap\n", "Map = geemap.Map(center=[40,-100], zoom=4)" ] }, { "cell_type": "code", "execution_count": 15, "id": "cc651577", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Map" ] }, { "cell_type": "code", "execution_count": 16, "id": "8068b4cc", "metadata": {}, "outputs": [], "source": [ "Map2 = geemap.Map(center=[40,-100], zoom=5)\n", "Map2.add_basemap('Esri.OceanBasemap')\n", "Map2.add_basemap('Esri.NatGeoWorldMap')" ] }, { "cell_type": "code", "execution_count": 17, "id": "c9723938", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Map2" ] }, { "cell_type": "markdown", "id": "3d58f186", "metadata": {}, "source": [ "## Display an NDWI image" ] }, { "cell_type": "code", "execution_count": 18, "id": "ec648e6e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import folium\n", "\n", "image = ee.Image('LANDSAT/LC08/C02/T1_TOA/LC08_044034_20140318')\n", "\n", "# Create an NDWI image, define visualization parameters and display.\n", "ndwi = image.normalizedDifference(['B3', 'B5'])\n", "ndwi_viz = {'min': 0.5, 'max': 1, 'palette': ['00FFFF', '0000FF']}\n", "\n", "# Define a map centered on San Francisco Bay.\n", "map_ndwi = folium.Map(location=[37.5010, -122.1899], zoom_start=10)\n", "\n", "def add_ee_layer(self, ee_image_object, vis_params, name):\n", " map_id_dict = ee.Image(ee_image_object).getMapId(vis_params)\n", " folium.raster_layers.TileLayer(\n", " tiles=map_id_dict['tile_fetcher'].url_format,\n", " attr='Map Data © Google Earth Engine',\n", " name=name,\n", " overlay=True,\n", " control=True\n", " ).add_to(self)\n", "\n", "folium.Map.add_ee_layer = add_ee_layer\n", "\n", "\n", "# Add the image layer to the map and display it.\n", "map_ndwi.add_ee_layer(ndwi, ndwi_viz, 'NDWI')\n", "display(map_ndwi)" ] }, { "cell_type": "markdown", "id": "afe25799", "metadata": {}, "source": [ "## Zonal statistics" ] }, { "cell_type": "code", "execution_count": null, "id": "9a74cf98", "metadata": {}, "outputs": [], "source": [ "import ee\n", "import geemap\n", "import os" ] }, { "cell_type": "code", "execution_count": null, "id": "91197894", "metadata": {}, "outputs": [], "source": [ "Ma = geemap.Map()\n", "Map" ] }, { "cell_type": "code", "execution_count": null, "id": "61b044d0", "metadata": {}, "outputs": [], "source": [ "dem = ee.Image('USGS/SRTMGL1_003')" ] }, { "cell_type": "code", "execution_count": null, "id": "40af8b81", "metadata": {}, "outputs": [], "source": [ "# Add Earth Engine dataset\n", "dem = ee.Image('USGS/SRTMGL1_003')\n", "\n", "# Set visualization parameters.\n", "dem_vis = {\n", " 'min': 0,\n", " 'max': 4000,\n", " 'palette': ['006633', 'E5FFCC', '662A00', 'D8D8D8', 'F5F5F5'],\n", "}\n", "\n", "# Add Earth Engine DEM to map\n", "Map.addLayer(dem, dem_vis, 'SRTM DEM')\n", "\n", "# Add Landsat data to map\n", "landsat = ee.Image('LANDSAT/LE7_TOA_5YEAR/1999_2003')\n", "\n", "landsat_vis = {'bands': ['B4', 'B3', 'B2'], 'gamma': 1.4}\n", "Map.addLayer(landsat, landsat_vis, \"LE7_TOA_5YEAR/1999_2003\")\n", "\n", "states = ee.FeatureCollection(\"TIGER/2018/States\")\n", "Map.addLayer(states, {}, 'US States')" ] }, { "cell_type": "code", "execution_count": null, "id": "aed321c2", "metadata": {}, "outputs": [], "source": [ "out_dir = os.path.join(os.path.expanduser('~'), 'Downloads')\n", "out_dem_stats = os.path.join(out_dir, 'dem_stats.csv')\n", "\n", "if not os.path.exists(out_dir):\n", " os.makedirs(out_dir)\n", "\n", "# Allowed output formats: csv, shp, json, kml, kmz\n", "# Allowed statistics type: MEAN, MAXIMUM, MINIMUM, MEDIAN, STD, MIN_MAX, VARIANCE, SUM\n", "geemap.zonal_statistics(dem, states, out_dem_stats, statistics_type='MEAN', scale=1000)" ] }, { "cell_type": "code", "execution_count": null, "id": "c71137ed", "metadata": {}, "outputs": [], "source": [ "out_landsat_stats = os.path.join(out_dir, 'landsat_stats.csv')\n", "geemap.zonal_statistics(\n", " landsat, states, out_landsat_stats, statistics_type='SUM', scale=1000\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "056a92e9", "metadata": {}, "outputs": [], "source": [ "geemap.create_download_link(out_dem_stats)" ] }, { "cell_type": "code", "execution_count": null, "id": "14e99b1a", "metadata": {}, "outputs": [], "source": [ "geemap.create_download_link(out_landsat_stats)" ] }, { "cell_type": "markdown", "id": "58b85953", "metadata": {}, "source": [ "## Convert JS to Python" ] }, { "cell_type": "code", "execution_count": null, "id": "b7096ca0", "metadata": {}, "outputs": [], "source": [ "import ee\n", "import geemap" ] }, { "cell_type": "code", "execution_count": null, "id": "e437b489", "metadata": {}, "outputs": [], "source": [ "js_snippet = \"\"\"\n", "// Load an image.\n", "var image = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_044034_20140318');\n", "\n", "// Define the visualization parameters.\n", "var vizParams = {\n", " bands: ['B5', 'B4', 'B3'],\n", " min: 0,\n", " max: 0.5,\n", " gamma: [0.95, 1.1, 1]\n", "};\n", "\n", "// Center the map and display the image.\n", "Map.setCenter(-122.1899, 37.5010, 10); // San Francisco Bay\n", "Map.addLayer(image, vizParams, 'false color composite');\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": null, "id": "9c4d8468", "metadata": {}, "outputs": [], "source": [ "geemap.js_snippet_to_py(\n", " js_snippet, add_new_cell=True, import_ee=True, import_geemap=True, show_map=True\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "42a599e4", "metadata": {}, "outputs": [], "source": [ "import ee\n", "import geemap\n", "\n", "Map = geemap.Map()\n", "import geemap\n", "\n", "Map = geemap.Map()\n", "\n", "# Load an image.\n", "image = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_044034_20140318')\n", "\n", "# Define the visualization parameters.\n", "vizParams = {\n", " 'bands': ['B5', 'B4', 'B3'],\n", " 'min': 0,\n", " 'max': 0.5,\n", " 'gamma': [0.95, 1.1, 1]\n", "}\n", "\n", "# Center the map and display the image.\n", "Map.setCenter(-122.1899, 37.5010, 10); # San Francisco Bay\n", "Map.addLayer(image, vizParams, 'False color composite')\n", "\n", "Map" ] }, { "cell_type": "code", "execution_count": null, "id": "731fd83c", "metadata": {}, "outputs": [], "source": [ "js_snippet = \"\"\"\n", "\n", "// Load an image.\n", "var image = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_044034_20140318');\n", "\n", "// Create an NDWI image, define visualization parameters and display.\n", "var ndwi = image.normalizedDifference(['B3', 'B5']);\n", "var ndwiViz = {min: 0.5, max: 1, palette: ['00FFFF', '0000FF']};\n", "Map.addLayer(ndwi, ndwiViz, 'NDWI', false);\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": null, "id": "70a66ead", "metadata": {}, "outputs": [], "source": [ "geemap.js_snippet_to_py(js_snippet)" ] }, { "cell_type": "code", "execution_count": null, "id": "b58fff7b", "metadata": {}, "outputs": [], "source": [ "import ee\n", "\n", "# Load an image.\n", "image = ee.Image('LANDSAT/LC08/C01/T1_TOA/LC08_044034_20140318')\n", "\n", "# Create an NDWI image, define visualization parameters and display.\n", "ndwi = image.normalizedDifference(['B3', 'B5'])\n", "ndwiViz = {'min': 0.5, 'max': 1, 'palette': ['00FFFF', '0000FF']}\n", "Map.addLayer(ndwi, ndwiViz, 'NDWI', False)\n", "\n", "Map" ] }, { "cell_type": "code", "execution_count": null, "id": "46dac764", "metadata": {}, "outputs": [], "source": [ "Map" ] }, { "cell_type": "markdown", "id": "corresponding-luxembourg", "metadata": {}, "source": [ "## Importing stuff" ] }, { "cell_type": "code", "execution_count": null, "id": "average-poultry", "metadata": {}, "outputs": [], "source": [ "import ee\n", "import geemap" ] }, { "cell_type": "code", "execution_count": null, "id": "african-vintage", "metadata": {}, "outputs": [], "source": [ "geemap.ee_search()" ] }, { "cell_type": "code", "execution_count": null, "id": "solid-efficiency", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.2" } }, "nbformat": 4, "nbformat_minor": 5 }