Raster data in GeoServer and GeoTools: Achievements, issues and future developments

0 %
100 %
Information about Raster data in GeoServer and GeoTools: Achievements, issues and future...
Technology

Published on September 24, 2014

Author: geosolutions

Source: slideshare.net

Description

The purpose of this presentation is, on a side, to dissect the developments performed during last year as far as raster data support in GeoTools and GeoServer is concerned, while on the other side to introduce and discuss the future development directions.
Advancements and improvements for the management of raster mosaic and pyramids will be introduced and analyzed, as well as the latest developments for the exploitation of GDAL raster sources.
Extensive details will be provided on the latest updates for the management of multidimensional raster data used in the Remote Sensing and MetOc fields.
The presentation will also introduce and provide updates on the JAITools and ImageIO-Ext projects. JAITools provides a number of new raster data analysis operators, including powerful and fast raster algebra support. ImageIO-Ext bridges the gap across the Java world and native raster data access libraries providing high performance access to GDAL, Kakadu and other libraries.
The presentation will wrap up providing an overview of unresolved issues and challenges that still need to be addressed, suggesting tips and workarounds allowing to leverage the full potential of the systems.

Raster data in GeoServer and GeoTools: Achievements, issues and future developments Ing. Simone Giannecchini, GeoSolutions Ing. Andrea Aime, GeoSolutions FOSS4G 2014, Portland 8th-13th September 2014

Outline  Who we are  The building blocks  Recent Achievements  What’s cooking FOSS4G 2014, Portland 8th-13th September 2014

GeoSolutions  Founded in Italy in late 2006  Expertise • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization  Supporting/Developing FOSS4G projects  MapStore, GeoServer  GeoBatch, GeoNetwork  Clients  Public Agencies  Private Companies  http://www.geo-solutions.it FOSS4G 2014, Portland 8th-13th September 2014

The technology stack FOSS4G 2014, Portland 8th-13th September 2014

GeoServer building blocks WMS WFS WPS REST Google GeoTools GWC ImageIO(-Ext) JTS GDAL WCS JAI(-Ext) Kakadu JAITools Vector Style Raster FOSS4G 2014, Portland 8th-13th September 2014

JAI  Java Advanced Imaging Library  SUN then Oracle Image Processing library  Huge number of basic building operations  Tile-based, deferred execution model  Integrate Tile Caching (In Memory)  Easily Extensible  Natively Accelerated Code available for many operations  No source code for native ops  Development more or less stopped! FOSS4G 2014, Portland 8th-13th September 2014

JAI-Ext  Oracle JAI Drop-In Replacement  Open Source (Apache 2.0)  High Performance  Pure Java  Support for No-Data  Support for Processing ROI  Several fixes compared to JAI  Support for Band Masks FOSS4G 2014, Portland 8th-13th September 2014

JAI-Ext  Operations  Mosaic  Scale  Affine  Translate  Lookup  Algebra  BandMerge  BandSelect  Border  Buffer  Crop  High scalability support  Op. registry  Tile cache  Tile scheduler  SquareRoot  Stats  Warp  ZonalStats FOSS4G 2014, Portland 8th-13th September 2014

JAITools/Jiffle  High performance raster processing  Extends, complements, replaces Oracle Java Advanced Imaging (JAI) Library  Open Source, BSD license  Low Level Operations, e.g. RangeLookup  Higher Level Operations  Vectorize, Contour  Raster algebra (Jiffle)  Works mostly in pure raster space (Image Processing)  Stable release is 1.3.0 FOSS4G 2014, Portland 8th-13th September 2014

ImageIO  SUN then Oracle Image I/O library  Partly ships with JDK partly as extension  Tile-based, deferred & immediate execution model  Easily Extensible  Natively Accelerated Code available for many formats  No source code for native (de)coders  Development more or less stopped! FOSS4G 2014, Portland 8th-13th September 2014

ImageIO-Ext  High Performance raster I/O  Extends, complements, replaces Oracle JAI ImageIO Library  Open Source, LGPL and BSD licenses  Plugins:  BigTiff*, MatFile 5, TurboJPEG  JP2000 (Kakadu)  GDAL integration via ImageIO-ext  JPEG2000, MrSID, ECW  BigTIFF, ERDAS Image, HDF4  Complete list here FOSS4G 2014, Portland 8th-13th September 2014

GeoTools  The Open Source Java GIS Toolkit  LGPL  Management of Geospatial data  CRUD Operations  Raster and vector  Rendering  GDAL Support  Based on best-of-breed Open Source projects  Java Advanced Imaging (JAI), ImageIO, ImageIO-Ext, JAITools (soon JAI-Ext too)  Java Topology Suite (JTS) FOSS4G 2014, Portland 8th-13th September 2014

GeoServer  GeoSpatial enterprise gateway  Java Enterprise  Management of raster and vector data  Based on GeoTools  Standards compliant  OGC WCS 1.0, 1.1.1 (RI), 2.0.1  OGC WFS 1.0, 1.1 (RI), 2.0  OGC WMS 1.1.1, 1.3.0  OGC WPS 1.0.0  OGC CSW 2.0.2  Google Earth/Maps support FOSS4G 2014, Portland 8th-13th September 2014

Recent Achievements FOSS4G 2014, Portland 8th-13th September 2014

GS – Faster PNG encoding  High Performance/Pure Java PNG output format  More Control on bugs/enhancements FOSS4G 2014, Portland 8th-13th September 2014  Based on the open source PNGJ library  Optimized pixel extraction to PNG data buffer code

GS – raster projection handling  Better polar projection support FOSS4G 2014, Portland 8th-13th September 2014

GS – raster projection handling  Rasters crossing dateline FOSS4G 2014, Portland 8th-13th September 2014

GT – image mosaic masks  Support for vector masks as shp/wkb/wkt FOSS4G 2014, Portland 8th-13th September 2014

GT – image mosaic masks  Also useful for clipping over a certain area  E.g., clip over administrative boundaries FOSS4G 2014, Portland 8th-13th September 2014

GS – WCS 2.0  WCS 2.0  The basics  Core service  KVP binding  XML binding  GetCoverage extensions  CRS  Scaling  Interpolation  Range subsetting  GeoTiff  GML  NetCDF  Output format extensions  GeoTIFF  GML Grid  NetCDF  Ensure CITE tests compliancy FOSS4G 2014, Portland 8th-13th September 2014

GS – WCS 2.0 Processing Chain Crop RangeSubset Scale & Interpolate Reproject & Interpolate Encode Core Extension Extension Extension Extension FOSS4G 2014, Portland 8th-13th September 2014

GS – WCS-EO  WCS Earth Observation profile  Build on top of a working WCS 2.0 with full extensions  WCS 2.0  CRS extension  Range subsetting extension  Interpolation extension  Scaling extension  GeoTiff extension  NetCDF extension  Add support for the WCS-EO extras  Listing coverage datasets in the capabilities documents (based on image mosaic contents, which will have to be marked as “exposed” so that we show their inner structure for EO)  Support describe coverage dataset  Support returning results for an entire dataset in GetCoverage FOSS4G 2014, Portland 8th-13th September 2014

GS – Coverage Views  Virtual layers created merging bands from different sources FOSS4G 2014, Portland 8th-13th September 2014

GS – Coverage Views FOSS4G 2014, Portland 8th-13th September 2014

GT – NetCDF and Grib  NetCDF/Grib official Support  Based on Unidata NetCDF JAVA library  Exposes current interfaces (GridCoverage2DReader,…) too for GeoServer compatibility before changes  1 coverage for each Dataset’s variable (multiple coverages).  Internal data exposed as set of 2D slices  Custom dimensions support  Ancillary XML file:  filter on NetCDF variables/coverages  define low level index schema (for dimensions) FOSS4G 2014, Portland 8th-13th September 2014

GT – NetCDF and GRIB  ImageMosaic NetCDF/GRIB integration  Allow the ImageMosaic to handle NetCDF/GRIB files  Expose NetCDF/GRIB internal structure (times, elevations)  Make ImageMosaic handle slices of the source files as granules Mosaic NetCDF 1 NetCDF 2 NetCDF … N … FOSS4G 2014, Portland 8th-13th September 2014

GT – NetCDF and GRIB  NetCDF Polyphemus Sample Dataset FOSS4G 2014, Portland 8th-13th September 2014

GT – Multicoverage Multidimentional interfaces  StructuredGridCoverage2DReader  exposes the underlying granule structure and dimensions descriptors of a specific coverage  getGranules (return a GranuleSource)  getDimensionDescriptors  allows to create/remove coverages and add new sources  createCoverage  removeCoverage  Harvest  GetGranules FOSS4G 2014, Portland 8th-13th September 2014

GS –multiple coverages per store  Multiple Coverages from Mosaic, NetCDF, GRIB FOSS4G 2014, Portland 8th-13th September 2014

GT – access mosaic contents  GranuleSource/GranuleStore  GranuleSource:  ReadOnly access to Granules (with Query support)  getGranules(Query)  SimpleFeatureCollection  getSchema  SimpleFeatureType  getCount(Query), getBounds(Query)  E.g.: get all granules having acquisitionYear > 2012 and Country=Italy  GranuleStore:  add/modify/delete capabilities using Transactions  addGranules (SimpleFeatureCollection)  removeGranules (Filter)  updateGranules  E.g.: remove all granules having acquisitionYear < 1990 FOSS4G 2014, Portland 8th-13th September 2014

GS – REST over mosaic contents  Granule Index CRUD Operations via REST  CREATE curl -u admin:Geos -XPUT -H "Content-type:application/zip" --data-binary @http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/t emperature/file.imagemosaic  READ index schema curl -v -u admin:Geos -XGET "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/polyphemus /coverages/NO2/index.xml"  READ AND FILTER WFS like with CQL filtering and paging curl -v -u admin:Geos -XGET "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/p olyphemus/coverages/NO2/index/granules.xml?limit=1&filter=time='2013-03- 03T00:00:00Z'" FOSS4G 2014, Portland 8th-13th September 2014

GS – REST over mosaic contents  Granule Index CRUD Operations via REST  UPDATE curl -v -u admin:Geos -XPOST -H "Content-type: text/plain" -d "/polyphemus_20130303.nc" "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/pol yphemus/external.imagemosaic"  DELETE WFS like with CQL filtering and paging or by ID curl -v -u admin:geoserver -XDELETE "http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/ coverages/NO2/index/granules.xml?filter=location='polyphemus_20130301.nc'" curl -v -u admin:geoserver -XGET "http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/ coverages/NO2/index/granules/NO2.2689.xml" FOSS4G 2014, Portland 8th-13th September 2014

GS – WCS NetCDF out  NetCDF Output Format  GetCoverage request parameter: &Format=NetCDF  Based on Unidata NetCDF JAVA library  Create a NetCDF output file on top of a GranuleStack:  A GridCoverage2D List:  a GridCoverage2D for each combination of requested Dimension values  GridCoverages should come from a StructuredGridCoverage2DReader  Supports WCS 2.0.1  Trimming: specify values range for a dimension  Slicing: specify single value for a dimension  Supports timeRanges/numberRanges on output FOSS4G 2014, Portland 8th-13th September 2014

GS – WCS NetCDF out  Sample WCS request http://localhost:8080/geoserver/wcs?request=GetCoverage& service=WCS&version=2.0.1&coverageId=geosolutions__N O2 &Format=NetCDF &subset=http://www.opengis.net/def/axis/OGC/0/Long(5,20) &subset=http://www.opengis.net/def/axis/OGC/0/Lat(40,50) &subset=http://www.opengis.net/def/axis/OGC/0/elevation (300,1250) &subset=http://www.opengis.net/def/axis/OGC/0/time ("2013-03-01T10:00:00.000Z","2013-03-01T22:00:00.000Z") FOSS4G 2014, Portland 8th-13th September 2014

GS – NetCDF out in ToolsUI FOSS4G 2013, Nottingham 21th September 2013

GS – WMS EO Wizard to configure EO layer groups Extending LayerGroup concept Support same style on both raster and vector data Support custom dimensions Alter map on the fly to support band combination  Earth observation profile  GSIP 84 FOSS4G 2014, Portland 8th-13th September 2014

GS – WMS-EO  WMS-EO Quirks  Root layer must respond with a specific layer rather than the composition of the children  Kind of a default visualization for a certain Dataset  E.g. when I send a GetMap for the root layer I could get back the browse image default layer  Same styles for both Raster as well as Vector data  E.g. yellow can be use for both flags as well as outlines  Peculiar behavior for Band layer ( raw data )  E.g. multiple bands at different wavelengths  Can request either 1 (grayscale image) or 3 (RGB image)  Different combinations are prohibited  Peculiar Behavior for GetFeatureInfo FOSS4G 2014, Portland 8th-13th September 2014

GS – dynamic color map QGIS palettes GDAL pam stats Input raster Dynamic ColorMap FOSS4G 2014, Portland 8th-13th September 2014 Map!

What’s cooking Plus what’s on our wish list FOSS4G 2014, Portland 8th-13th September 2014

NODATA support  Natively handle NODATA with JAI-EXT  Support in all aspects  Scaling  Reprojection  Interpolation FOSS4G 2014, Portland 8th-13th September 2014

GS – Raster algebra WPS  Raster Algebra as WPS  Gets an OGC Filter containing logical operations between raster layers and produces a binary (0/1) layer FOSS4G 2014, Portland 8th-13th September 2014

GS – Raster algebra in Coverage views  JIFFLE FOSS4G 2014, Portland 8th-13th September 2014

GS – Raster algebra in Coverage Views  Define new bands using a Jiffle formula  Generate fully computed layers  Still using overviews and other native abilities FOSS4G 2014, Portland 8th-13th September 2014

The End Questions? andrea.aime@geo-solutions.it simone.giannecchini@geo-solutions.it FOSS4G 2014, Portland 8th-13th September 2014

Old news  FOSS4G 2014, Portland 8th-13th September 2014

ImageIO-Ext  LibJPEG-Turbo  derivative library of libjpeg based on SIMD instructions (MMX, SSE2, NEON)  accelerate baseline JPEG compression and decompression on x86, x86-64, and ARM systems  available on many platforms  Open Source, LGPL  ImageIO-Ext-TurboJPEG  A JPEG ImageIO plugin built on top of the JAVA wrapper for LibJPEG-Turbo (TurboJPEG)  20%-30% SpeedUp* on JPEG encoding  10%-20% SpeedUp* on JPEG decoding  More info here  Integrates with TIFF reader and writer FOSS4G 2014, Portland 8th-13th September 2014

ImageIO-Ext  NITF encoding Support (ImageIO-Ext-NITF)  Based on NITRO  Support for Tagged Record Extensions (TREs)  Support ImageSegments JPEG2000 compression through Kakadu ImageIO-Ext plugin  (Numerically Lossless/VisuallyLossless/Lossy)  NPJE, EPJE BIIF (Basic Image Interchange Format) profiles  J2KLRA TRE support  Support for multiple TextSegments  Support for multiple ImageSegments  Support for ShapeFile Data Extension Segment (CSSHPA) FOSS4G 2014, Portland 8th-13th September 2014

GeoTools Band Combine + Symbology On-the-fly Contouring Raster to point vector On-the-fly Polygonalization  Improved internal JAI Concurrency  Introduced and Improved JAI Tile Recycling  Reduced depth of JAI Processing Chains  More aggressive factory caching  Added a few specific high perf JAI operations  e.g. crop  Created Specific high perf ROI class  Improving Tile Caching  Work ongoing  Transparent File vs Memory Caching when writing raster FOSS4G 2014, Portland 8th-13th September 2014

GeoTools  Optimized Rendering/Processing for raster data  Optimized Crop Chains  Coalescing multiple Crops into a single one  Optimize Affine Transformation Chains  Coalescing multiple Affine Transformations into a single one NEAREST FOSS4G 2014, Portland 8th-13th September 2014

GeoTools BICUBIC FOSS4G 2014, Portland 8th-13th September 2014

GeoTools  ImageMosaic  Custom Dimensions  More than simple time/elevation support  E.g.: wavelength, cloudLevel, …  Dimensions with ranges: (startTime, endTime), (minElevation,maxElevation), …  Dynamic read parameters  Filtering  Supports for OGC Filters through the “filter” param  E.g.: Return only ImageMosaic granules where attribute Country = Italy.  filter=“Country=‘Italy’ ” FOSS4G 2014, Portland 8th-13th September 2014

ImageMosaic  Custom/Additional dimensions FOSS4G 2014, Portland 8th-13th September 2014

GeoTools  ImageMosaic  Ordering  SORT_BY clause support on DBMS based mosaics  sorting=“time A, elevation D”  Stacking VS Merging  Merging: different granules  merged together in the final image as default  Stacking: different granules  stacked on different bands  E.g. MeteoSat data: 12 channels (1 granule for each Channel) 1 = VIS0.6, 2 = VIS0.8, …, 11 = IR13.4, 12 = HRV.  Want return a RGB image made of channels 1,3,7  Stack them FOSS4G 2014, Portland 8th-13th September 2014

GeoTools  New API (unsupported/coverage-experiment)  Driver: Get a connection to a coverage store.  CoverageAccess: a physical storage of coverage data. Get access to a source of coverages.  CoverageSource: provides data access to a coverage.  Updated GridCoverage core API  Old GridCoverageReader:  inefficient and thread unsafe way to access multiple coverages  read GridCoverages from the input stream in a sequential order  New GridCoverage2DReader:  Coverages and properties may be accessed by coverage name:  get Envelope(name), get GridRange(name), get CRS(name), read(name), get dynamicParameters (name), …  dynamicParameters support (for custom dimensions) FOSS4G 2014, Portland 8th-13th September 2014

GeoServer  LibJPEG-Turbo Output format  Encoding WMS output JPEG images through LibJPEG Turbo library  Based on the TurboJpeg ImageIO-Ext plugin built on top of the JAVA interface for LibJPEG-Turbo (TurboJPEG)  Provided as a GeoServer extension  Up to 80% SpeedUp measured on JPEG encoding  Can be disabled at startup:  -Ddisable.turbojpeg=true  More Info here FOSS4G 2014, Portland 8th-13th September 2014

GeoServer  More Color Quantizers OCTTREE Speed loss is around 10% MEDIANCUT FOSS4G 2014, Portland 8th-13th September 2014

GeoServer  WMS Custom Dimensions  Work on GeoTools raster Additional Domains  NetCDF  ImageMosaic  Turn additional querying dimensions into WCS/WMS Dimensions  ImageMosaic Domain Category  WMS 1.1.1 DIM_CATEGORY  WMS 1.3.0 DIM_CATEGORY  Perfect for large multidimensional mosaics!  E.g. meteo (forecast time, elevation, band, base time, model name) FOSS4G 2014, Portland 8th-13th September 2014

GeoServer  WMS TIME, ELEVATION & More http://localhost:8080/geoserver/geosolutions/wms?...&time=2013-03- 01T00:00:00.000Z&elevation=35.0&DIM_FILEDATE=2013-03- 01T00:00:00.000Z&DIM_UPDATED=2013-04-08T08:18:41.597Z FOSS4G 2014, Portland 8th-13th September 2014

Add a comment

Related presentations

Related pages

Raster data in GeoServer and GeoTools: Achievements ...

Raster data in GeoServer and GeoTools: Achievements, issues and future developments ... GeoTools, GeoServer
Read more

Raster Data In GeoServer And GeoTools: Achievements ...

... to dissect the developments ... GeoServer And GeoTools: Achievements, Issues ... as raster data support in GeoTools and GeoServer ...
Read more

Raster data in GeoServer and GeoTools: Achievements ...

×Close Share Raster data in GeoServer and GeoTools: Achievements, issues and future developments
Read more

Raster Data In GeoServer And GeoTools: Achievements ...

... GeoServer And GeoTools: Achievements, Issues ... the developments performed during last year as far as raster data support in GeoTools and GeoServer ...
Read more

Raster data in GeoServer and GeoTools: Achievements ...

... Achievements, issues and future developments. ... developments performed during last year as far as raster data support in GeoTools and GeoServer ...
Read more

Slide 1 - MECI

Raster data. in . GeoServer. and . GeoTools: achievements, issues . and . future developments. Ing. Andrea . Aime. GeoSolutions
Read more

GeoTools - YouTube

Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Developments - Duration: 30:26. Andrea Ross 1,329 views. 30:26
Read more

Raster Archivi - GeoSolutions

... for raster data which ... GeoSolutions Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Developments from ...
Read more