GIS

h1. %{color:BLUE}  Wiki%

h1. [[CSR Geomatics Team]]

h1. [[Collaborations]]

h1. [[Data Analysis]]

h1. [[CSR Geomatics Server Setup]]

h1. [[Database Backup and Restoration]]

h1. [[CAD to GIS Workflow]] %{color:RED}  - TO BE UPDATED%

h1. [[GIS Conventions]]

h1. [[Working with Gisaf]]

h1. [[Gisaf Workflow]]

h1. [[GIS to CAD Workflow]] %{color:RED}  - TO BE UPDATED%

h1. [[Rain gauges]]

h1. [[Weather Station]]

h1. [[GDAL - Useful Tips]]

h1. [[Gisaf - Auxiliary Data (non-geographical data)]]

h1. [[QGis - Useful Tips]]

h1. [[AutoCAD Civil 3D - Useful Tips]]

h1. [[Wells]]

h1. [[Gisaf Data Modifications]]

h1. [[Other Useful Softwares]]

h1. [[Portable Ultrasonic Flow Meter]]

h1. Documentation- DST- Vegetation Indexing

h2. Steps for Dzetsaka Classification tool for Vegetation indexing in QGIS

 1. Install the plugin Dzetsaka classfication tool.

 2. Open the Raster from the Survey.

 3. Create a polygon shapefile for index sampling. Mark polygons and give the ID's (1,2,3 for Tree, grass, bare land etc) Cover the variations in samples as much as possible. 

     More the samples, better the indexing.

 4. Apply Dzetsaka Classification tool, Select the base raster and the sample- index polygon shapefile created in step 3.

 5. The result is a Raster with DN numbers specified in the Shapefile in step 3.

 6. Apply the Sieve raster command (Raster> Analysis> Sieve)- Try different threshold numbers and view the results till the noise is removed from the Raster.

 7. Polygonise the Raster to Vector (From processing)

 8. Run the v.generalise tool on the shapefile. This tool removes the pixelated boundaries of the polygons in the Vector.

h1. Documentation- DST- Interpolation (Processing toolbox)

The following 4 tools have been mostly explored and the results were compared. The ones used for quick analysis are 1. Cubic Spline and 2. V.surf.spline . The rest of the tools are for further exploration and used depending on the need of the project. In some tools, the elevation values of points should be stored in the attribute table (Using field calculator and giving command - *Z($Geometry)* )

# Interpolate (Cubic spline) - SAGA

# V.surf.bspline - GRASS. Parameters to set - cell size. Set this parameter above 0.001 ( 0.00001, 0.000001 etc) and check the results.

# V.surf.rst - GRASS

# Krigging - SAGA

[[Documentation- DST- Survey- Office workflow]]  - *%{color:RED} to be written%*

h1. From CAD to GIS by Giulio

*FEATURES CREATION*

1.	Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)

2.	Create features in CAD (Points, lines, polygons)

3.	Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)

*FEATURES IMPORT INTO DB FIRST TIME*

4.	Create zip file of the shapefile

5.	Upload into the GISAF Shapefiles Basket

6.	Import the shapefile into DB

7.	Save the shapefile on Local Machine

*FEATURES IMPORT INTO DB EVERYTIME*

8.	Combine the new features to corresponding last shape files (Insert the process here).

9.	Follow step 4-8 again

*FEATURES EDITING IN QGIS*

10.	Open the table in QGis

11.	Save as a shapefile (b) in TM AUSPOS CRS

12.	In CAD, open a new drawing and assign AUSPOS CRS

13.	Import the shapefile (b) (MapImport) with Object Data (Data tab > Create Object Data > OK), tick “Import polygons as closed polylines”, then press OK

14.	Edit features

15.	Change workspace into “Planning and analysis”

16.	Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)

*FEATURES IMPORT INTO DB*

17.	Create zip file of the shapefile

18.	Upload into the GISAF Shapefiles Basket

19.	Import the shapefile into DB

20.	Delete the shapefile from Local Machine

h1. QGIS- Miscellaneous

*QGIS Introduction:*

https://www.birdseyeviewgis.com/blog/2018/2/22/my-favorite-features-of-qgis-30to-date

3D visualization of raster DEM- https://www.youtube.com/watch?v=2KrCsbP0kUs

*Spatial query:* 

Spatial Query is selection of features that satisfies a certain condition which relates to other features in a space.

Using plugin- Spatial query

http://www.geodose.com/2018/03/spatial-query-in-qgis-3.html

Labelling:

Labelling with more than one field names and in different lines

|| '\n' ||

*Hierarchy of extensions*

File levels and their uses.

https://nathanw.net/2014/03/22/all-the-q-files/

+The Project file (.qgs)+

It contains: Layer source pointer + Style information + Composers + a whole heap of other stuff

+The Layer Definition file (.qlr)+

It contains: Layer source pointer + Style information

+The QML file (.qml)+

It contains: Style information

h2. Giulio's documentation

h1. Documentation - Reconciliation of points using Gisaf

Reconciliation of points is a procedure used when a point is stored in a wrong table, because its category was either wrongly recorded in the field by the surveyor, or it has ben reviewed later by the surveyor or the data validator and found to be wrong.

h2. Definition: Raw points are all points coming from the field survey. Raw points can be points referring to a Point feature (e.g. trees, or elevation points, or floor level), or points measured in the field to draw a line (e.g. the vertices of a fence) or a polygon (e.g. the corners of a building outline).

In the overall workflow, *+reconciliation affects only raw points+*. It takes place after the field textfile is uploaded into the basket and its points imported (raw points stored in the raw survey tables). Here you can visualize the Workflow diagram: [[Survey_data]].

If a raw point refers to a point feature, reconciling it means moving it to another category/table meant for point features, not for lines/polygons. So, *+only raw points of point features can be reconciled+*. Raw points pertaining to line features and/or polygon features cannot be reconciled, so these raw points will remain in their original wrong table.

PLEASE NOTE: 

The attributes of a line/polygon, which are derived from their defining raw points, will not be modified by any reconciliation, because *+only raw points of point features can be reconciled+*.

h2. How to perform Reconciliation

To perform reconciliation of points: Login to Gisaf -> click on the G icon on the upper-left corner of the website page -> Manage -> Reconciliation by orig.ID

On the right end side of the screen, clicking on the field "Destination" a list of all categories will appear: these categories refer not to the Raw survey tables, but to the V_ tables of the database (points, lines, polygons).

Under it, in the field "Original ID", the original point number of the point to be reconciled is to be entered.

Clicking on the "Search points" button, the result shows the database unique id of the point, its survey category, its survey date, its geometry type (point, line, polygon), and the Project the point belongs to. In case of multiple points with the same original id (in case of different Projects, the field number of points might be not unique if the numbering of points in the field has restarted) all points having that original id are displayed: thanks to their date or Project or type, it is easy to identify the correct point to be reconciled.

Once the point to be reconciled is identified, clicking on the button "Reconcile" will run the reconciliation, and a message will appear stating that it has been done successfully. An error message can appear if a reconciliation of a raw point of a line/polygon feature has been attempted: this type of points cannot in fact be reconciled.

Once a raw point has been reconciled, *+it cannot be reconciled again+*. In case a raw point has been wrongly reconciled, it cannot be reconciled again through the above procedure, but it has to be reconciled manually through QGis or pgadmin software.

h1. Documentation - Status and Status Changes

Status have been created to keep track og changes in surveyed features.

It is an additional (though provided for in AIA standards), single digit value, at the end of the Gisaf Category/Cad layer name.

Each Status need to have a corresponding CAD layer/Gisaf Category, with a short code associatedto it, so that field entries can be done easily.

Status have been defined as follows:

N - New Work

E - Existing o remain

D - Existing to demolish, Demolished or Changed

F- Future work, Proposed feature

T - Temporary work

M - Item to be moved

X - Not in contract

By default, Status is defined as E (Existing). Status can anyhow be changed later using gAdmin, or QGis (through the PostGis connection).

It needs to be done manually, one feature (point, line, polygon) at a time.

In the future Status changes might be incorporated in the Admin panel.

h1. Documentation - Tags retained after re-import of same geometry

Documentation - Tags retained after re-import of same geometry

The linework for infrastructure survey carried out by Eric Chacra in May 2020 was originally imported with a problem of ambiguity in the "Accuracy" table and in the "Accuracy" table.

The result was that lines did not inherit the attributes survey date, accuracy, equipment, surveyor.

Nevertheless lines were displayed on the Gisaf map, without these attributes, and tags were given to some of these lines.

The values for the two tables ("Accuracy", "Accuracy") have been corrected, ambiguity resolved.

The lines in the layers V-ELEC-UGND------E and V-COMM-CABL------E have been then reimported, the attributes have been properly assigned, and the tags have been retained.

4 August 2020

h1. Access to data

h2. Connection to server directly from CSR

To connect to the server directly without going through Aurinoco server, the correct url is

http://gis.csr.av

h2. Connection to Gisaf via QGis through WFS / OGC API

This works only on QGis from version 3.14.15 onward

In the browser, click on WFS/OGC API, then right-click to create a new connection

Give a name (e.g. OGC API Qgis Gisaf)

Give the url https://gis.auroville.org.in/ogcapi

Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine

Click on OK

The list of layers will appear in the Browser under WFS/OGC API.

h1. How to create a new projection in QGis

To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".

A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.

Click on the green "+" sign on the right top part of the window to create a new projection.

In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS -  Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).

In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"

In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".

Finally, click on OK.

In a more explicit  way, the parameters mean the following:

Map Projection:        TransverseMercator (TM)

False Easting:          370455.6300

False Northing:        1328608.9940

Latitude of Origin:    12°00'57.79560" (DMS)   12.01605433 (DD)

Central Meridian:     79°48'35.96164"  (DMS)   79.80998934 (DD)

Scale Factor:             1.00000000

Zone Width:              6.0°

h1. Elimination of Duplicate points – General criteria

It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.

It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.

It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.

If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.

How to identify duplicate points?

The following criteria can be used:

1.	The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)

2.	The orig_id (serial number) of the points are not in series

3.	The survey date is not the same

4.	In case of trees, the species of trees is the same

5.	5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)

6.	The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.

h1. Linework for the Survey Area

h2. 1.	Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)

Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.

Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.

The Initial Linework for the Survey Area is also stored temporarily in 

+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+

h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS

h2. 2.	Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)

Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.

As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:

1.	*If features are lines:*

•	Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).

h2. The CRS for the export shall be EPSG:4326 - WGS 84

2.	*If features are polygons:*

•	Lines shall be converted into polygons:

to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:

!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!

The new window for “Lines to Polygons” conversion will appear:

!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!

•	Always cross check the input layer, to make sure that the input layer is the active one

•	Save the output in a temporary layer

•	The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)

h2. The CRS for the export shall be EPSG:4326 - WGS 84

Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.

h2. Linework for the whole Survey Zone

h2. 1.	Merging Shapefiles / Geopackages - (Selvarani, System 1)

A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.

Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:

•	To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:

!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!

The new window for “Union” will appear:

!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!

•	To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).

•	(The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)

h2. 2.	Storing Shapefiles / Geopackages - (Selvarani, System 1)

Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)

Date in the name of Final Shapefile / Geopackage needs to be updated.

Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.

h2. 3.	Topology check of merged shapefiles

The topology checker is applied again on the merged shapefiles / geopackages.

The “id_field” shall be removed from the attribute table.

h2. 4.	Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)

Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.

*Then delete the Merged Shapefile / Geopackage folder from System 1.

*

h2. 5.	Note about Shapefiles and Geopackages

All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).

The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be noted, as Gisaf database does not accept the Multipolygon and Multiline geometry types.

A different way to create "Polygons" is to use the command Vector -> Geometry Tools -> Multipart to Single Parts and apply it to the layer: from "Multipolygon" it will become "Polygon" (check in layer Properties).

As on 13 March 2021, Gisaf can accept Multipolygon layers, because the command "Multipart to Single parts" has been integrated into the importing command (see Redmine ticket #11691)

h1. Creating 3D Shapefile/Geopackage

While creating a Shapefile/Geopackage, the File name/Database-table name, Geometry type and CRS have to be entered.

In order to create a 3D Shapefile/Geopackage, the additional dimensions *"Z(+M values)"/ "Include Z dimension"* has to be ticked: this way the 3D Shapefile/Geopackage is accepted by Gisaf without errors, otherwise the Shapefile/Geopackage can't be imported in Gisaf because the Z dimension is missing (the geometries in the database are all 3D).

h1. Exporting from QGis (shapefiles and/or geopackages) to CAD dxf format

An algorithm has been created by Selvarani, to see the whole process click here:

https://redmine.auroville.org.in/projects/gis/wiki/Automatic_export_from_QGis_to_dxf