Redmine

Jupyter is a popular project that makes interactive development fun and easy. Jupyter notebooks are used by CSR Geomatics for data processing and analysis, graphic visualizations, research and development, etc.

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

Moreover, Gisaf contains tools that makes easy to use data and models in Jupyter.

It is now possible to access our installation of Jupyter Lab online, with the notebooks that we develop: https://gis.auroville.org.in/notebooks/ (password required: please contact us).

Process management¶

So far, data acquisition from online devices (weather stations, sensors, everything under the "IoT" umbrella) had to be done aside from Gisaf: e.g. by setting up cron jobs or pub/sub services at the OS level.

Gisaf has integrated these capabilities, and a "scheduler" for different "jobs" has been added. It makes the supervision of these jobs easy to supervise (in the admin area). Beside, and probably more importantly, that integration open doors for more applications in the domains of real time monitoring and management.

Under the hoods, redis and mqtt are used for inter-process communications, which keeps the Gisaf architecture easily scalable.

Major bug fix¶

A bug in the import function of the raw survey files, in the new admin's basket, was found (#10830) and corrected.

A lot of time and care were put in fixing the existing data in AV CSR Geomatics.

Publication of data through new generation APIs¶

Gisaf moves towards the OGC API specifications (aka WFS3) for better integration and easing data access. This is most useful for people who need access from a GIS desktop: instead of downloading layers and working on local copies for making maps, GIS desktop users can establish a direct connection with the "/ogcapi" URL prefix (#10948).

Note that the specifications (https://github.com/opengeospatial/ogcapi-features) and tools (QGis, https://github.com/geopython/pygeoapi, etc) will probably replace the old WFS specifications as industry standards, but it will probably take some time for these to mature enough for heavy load production usage.

Improvements¶

Labels on the map¶

A simple toggle button was added to display labels of features, directly on the map (#10920).

Other¶

Many minor improvements, updates and bug fixes, as always

Few major updates, worth mentioning: Angular updates to version 10.1, Graphql to 3.0.

Import baskets¶

• Misc improvements and fixes
  • Many bug fixes for specific cases, as it's now part of the regular workflow of AV CSR Geomatics
  • Auto import (upload and import the data to the database in a single step)
  • Detailed user feedback about the imports
  • Baskets accept the newer geopackage format, along with the old style shapefiles

• The "old admin" basket was deactivated. The "old admin" interface is now left only for convenience, for occasional minor updates.

Admin¶

• Improvements and misc fixes for reconciliations (autocomplete when searching layer to reconcile to, etc)
• Add a function (button) to change the status of features

Measures¶

The dropdown menu for selecting the feature to visualize is now filtered, with only the features that have something to actually display.

Tags¶

The location of tags is stored in the database. This allows a much needed feature: during the import of geographical data from the baskets, tags can be "migrated" to overlapping new features. This is mostly useful for tags that store record ids in another service/API.

Libraries updates¶

Little by little, Gisaf moves to use more vector-based processing (GeoPandas/Pandas/Numpy/PyGoes/rtree, etc) as the technology now enables easier integration with these more efficient tools for manipulation of lare amount of data.

Update of the plugin interfaces (tags, actions, etc)¶

• Plugins are now better structured
• New plugin types (downloader, which deals with the download of automatically generated data, eg. CSV files for readings, reports...)
• This results to cleaner GUI (map's info panel => tools)

... And, as always: the performance of various operations was improved, many small bugs fixed, and minor improvements could be added :)

Integrity checks¶

In Gisaf, the structure of the database and the processes (eg. basket imports) ensure that basics of the quality of the data is met. However, the quality of data can be improved further: data integrity.

Each integrity check is defined as a "business rule", like a condition that filters database records. Data matching these rules are tagged, and these tags can eventually be used for correction actions (automatic, semi-automatic, manual), reports, etc.

In Gisaf, this new feature is included in the admin. The first integrity check is for determining the map features which miss some information (surveyor). The integrity check tags the features, which are then easy to spot on the online map.

Vacuum¶

An other useful feature was added: a button to "vacuum" the database. The experience shows that under some circumstances (eg. uncontrolled shutdown), Postgres internals can be affected. The symptom is very visible in Gisaf's map: the tree of layers looks almost empty, as if all the data had vanished.

No panic! A simple click on a button in the web admin interface should fix that (Manage => Integrity and maintenance => Vacuum DB).

The recent releases of Gisaf include online admin/management tools, that were previously dealt with tweaks on a Flask admin sibling site.

As always, the web interface is relatively minimalist and task centered, it should be more intuitive and faster for daily routine work.

The most important of these features are:

• Baskets: with lots of refactoring and major updates under the hood, adopting more GeoPandas. Note that points based models are now imported automatically, straight from the basket to the database.

• Survey categories: the edition of the survey categories (addition, updates, deletions) do not require system admin intervention as previously, as Gisaf now creates, updates and drop the tables in the database when categories are changed from the new interface.

• User, roles, access (ACLs)

With these updates, it is not required anymore to have an instance of Flask-admin running specific code (with annoying duplication of the definition of models, for example) for administration tasks.

Gisaf can now be used to generate widgets to be embedded in other web sites. For now, there's one for rainfall.

It's very easy: just add the following in the web site (adjust the size as needed):

<iframe src='https://gis.auroville.org.in/embed/rainfall' width="300" height="150" frameborder='0'></iframe>

It's easy to generate these widgets, typically in notebooks.

Another round of speed ups were achieved by using a combination of several techniques: use of compression and cache.

Loading the application takes now around 1.5MB of network, compared to about 5MB.

Loading layers was also reduced drastically. Typically, compression reduces the network traffic by 4 times. In other words, it's 4 times faster.

Even better, the browser's cache is used optimally: loading the same layer again is now almost instantaneous... unless it was updated in between.

Permalinks¶

Any view of the map can be shared with an URL (often called a permalink). That link will open the view (lat, lon, zoom, etc), along with the layers and the base map.

Live layers¶

The live layers now behave almost the same as any regular (database) layer: one can export using the same menu, view information.
Beside, they also can have a description (displayed on mouse over the live layer in the layer tree), an icon, etc.

Background layer opacity¶

The styling of vector background wasn't pretty due to the change of opacity. Changing opacity works now as expected.

User editable base map definitions¶

Any user with the relative role (permission) can now edit base map layer: add new ones, change or delete existing ones. See the Create a base map with the selected layers option of the More... menu in the left panel.

Admin¶

The icon for categories is now displayed correctly (with the GisafSymbol font, rather than a obscure Unicode value)

Icons¶

All known point layers known to date have a proper icon. Beside, the layer tree (left panel) now displays icons according to the geometry type (Point, Line or Polygon) rather than an _?_.

Under the wood improvements¶

- Huge performance/loading speed improvements for category (survey) and live layers, lesser improvement for the custom (Misc) ones

- Improved the styling of icons in the WebUI (no more css definitions necessary)

- Made the home page in the dashboard configurable

Bug fixes¶

Many small bugs were fixed in the map view:

- correct resize of the map when the info panel is closed

- layers disappearing on some occasions, eg. on changing the background layer, when network requests were interrupted, etc.

- reprojection issues, causing a wrong calculation of area, length displayed in the info panel

A whole new version of the site is available online: a new look, and a deep revamp under the scene.

Few new features:

• Dashboards, defined in Jupyter notebooks
• Search (filter) the element displayed on the map
• A button on the map which allows to zoom on the selected layers (i.e. eventually, only the filtered items)
• A new admin, well integrated with the main site, but still work-in-progress: it can currently only display the content of the tables.

Note that the "old" admin, which is still required for much of the work like imports/baskets, is available at another URL: /old-admin

We have included the survey done so far for the Residential Zone. It includes more than 50000 survey points, and the associated line work.