Project

General

Profile

Wiki » History » Version 90

Selvarani C, 26/02/2021 10:09

1 1 Philippe May
h1. Wiki
2 2 Philippe May
3 48 Philippe May
h2. About the Auroville CSR Geomatics Studio
4 48 Philippe May
5 48 Philippe May
We are a team working on geographical and related information with an engineering and scientific approach. The concept of *geomatics* is explained here: https://en.wikipedia.org/wiki/Geomatics.
6 48 Philippe May
7 48 Philippe May
We develop, maintain and publish data on this web site: https://gis.auroville.org.in.
8 48 Philippe May
9 48 Philippe May
h3. Team
10 48 Philippe May
11 48 Philippe May
Currently, the team consists of:
12 48 Philippe May
13 48 Philippe May
* Bala
14 48 Philippe May
15 48 Philippe May
* Giulio
16 48 Philippe May
17 48 Philippe May
* Philippe
18 48 Philippe May
19 48 Philippe May
* Raj
20 48 Philippe May
21 48 Philippe May
* Ram
22 48 Philippe May
23 48 Philippe May
* Selvarani
24 48 Philippe May
25 48 Philippe May
h3. Collaborations
26 1 Philippe May
27 47 Philippe May
Quick report of the collaborations the Geomatics Team the areas of work: see [[collaborations]].
28 45 Philippe May
29 89 Selvarani C
30 88 Selvarani C
h1. Collaborations
31 88 Selvarani C
32 88 Selvarani C
Collaborations/coordination with other other groups, etc
33 88 Selvarani C
 
34 88 Selvarani C
|_.CSR Geomatics Description of Activity / Research / Project|_.AA|_.AVF|_.LB|_.FAMC|_.WG|_.CSR|_.Talam|_.DST|
35 88 Selvarani C
|Topographic Survey of Auroville City Area in coordination with L’avenir d’Auroville: Matrimandir entire compound and Residential Zone Sector 1 completed, Sector 2 half completed|yes|||||||yes|
36 88 Selvarani C
|Topographic Survey of specific projects area: Vibrance, Wasteless, Cultural Zone along the Crown Road, Baraka, Gardens of Unexpected |  |  |  |  |  |  |  |  |
37 88 Selvarani C
| Collective wastewater treatment systems health check-up survey: 68 plants evaluated | yes |  |  |  | yes | yes |  | yes |
38 88 Selvarani C
| Manual weekly monitoring of water level in selected wells on Auroville land on approximately 50 wells (number fluctuates depending on local conditions) | yes |  |  |  | yes |  |  | yes |
39 88 Selvarani C
| Collection of rainfall data through manual raingauges distributed to Aurovilians: data received regularly from at least 7 raingauges |  |  |  |  | yes |  |  | yes |
40 88 Selvarani C
| Collection of weather data through automatic weather station installed at CSR: data collected every minute, stored in the database, and published online in real time |  |  |  |  | yes | yes |  | yes |
41 88 Selvarani C
| Collaboration with Land Board for survey of identified land boundary stones: collection of coordinates of Government boundary stones for georefering of cadastral maps |  | yes | yes |  |  |  |  |  |
42 88 Selvarani C
| Collaboration with AV Foundation for compilation of land ownership map: geographic land records as provided by AV Foundation, protected by login access |  | yes |  |  |  |  |  |  |
43 88 Selvarani C
| Collaboration with L’avenir d’Auroville for data sharing and coordinated system set-up: organisation of geographic data for unique online platform to enhance planning processes (initial stage) | yes |  |  |  |  |  |  |  |
44 88 Selvarani C
| Collaboration with L’avenir d’Auroville and FAMC for database links: structural work on procedures for linking databases already existing with various agencies | yes |  |  | yes |  |  |  |  |
45 88 Selvarani C
| Study of application of drone topographic survey in Auroville context: pilot test conducted on Residential Zone with outside drone survey agency | yes |  |  |  |  | yes |  | yes |
46 88 Selvarani C
| Study of open source software for drone topographic survey: testing of softwares like OpenDroneMap, MeshLab, CloudeCompare to evaluate work flow, processes and limitations |  |  |  |  |  | yes |  | yes |
47 88 Selvarani C
| Surface water modelling (ongoing): research on evaluation of surface water collection potential through automatic processing of field data |  |  |  |  |  | yes |  | yes |
48 88 Selvarani C
| Collaboration with TALAM on research on radio-transmission for automatic wells level monitoring (ongoing): evaluation of application of radio-transmitted signals for water-related monitoring and metering (selected borewells, flow meters) |  |  |  |  |  |  | yes | yes |
49 88 Selvarani C
| Organisation of data collected in Auroville in the past (ongoing): standardisation of data in geospatial database |  |  |  |  |  | yes |  |  |
50 88 Selvarani C
| Provision for basic statistical analyses of geographic-related data (indicators/dashboard) |  |  |  |  |  | yes |  | yes |
51 88 Selvarani C
| Publication of collected data in standardised format on web portal gis.auroville.org.in |  |  |  |  |  | yes |  | yes |
52 88 Selvarani C
53 88 Selvarani C
* AA : L'avenir d'Auroville
54 88 Selvarani C
* AVF : Auroville Foundation
55 88 Selvarani C
* LB : Auroville Land Board
56 88 Selvarani C
* FAMC : Auroville Funds and Assets Management Committee
57 88 Selvarani C
* WG : Auroville Water Group
58 88 Selvarani C
* CSR : Auroville Centre for Scientific Research
59 88 Selvarani C
* TALAM : a project under CSR
60 88 Selvarani C
* DST : Department of Science and Technology, Ministry of Science and Technology, New Delhi
61 88 Selvarani C
62 5 Philippe May
h2. Our workflow
63 2 Philippe May
64 5 Philippe May
See [[Workflow]].
65 2 Philippe May
66 3 Philippe May
67 3 Philippe May
h2. Working with Gisaf
68 3 Philippe May
69 6 Philippe May
h3. Survey data
70 6 Philippe May
71 12 Philippe May
Raw survey data are contained in CSV files, typically downloaded from surveying instruments.
72 12 Philippe May
73 25 Philippe May
See more information of the process for the survey data (including a flow diagram): [[Survey data]]
74 12 Philippe May
75 12 Philippe May
h3. Categories
76 12 Philippe May
77 12 Philippe May
The categories define the types of the geographical features.
78 12 Philippe May
79 12 Philippe May
See [[Categories]]
80 6 Philippe May
81 6 Philippe May
82 1 Philippe May
h3. QGis: work on shapefiles
83 1 Philippe May
84 1 Philippe May
Go to [[shapefiles]].
85 5 Philippe May
86 5 Philippe May
87 18 Philippe May
h2. Data analysis
88 18 Philippe May
89 18 Philippe May
See [[Data analysis]]
90 18 Philippe May
91 18 Philippe May
92 5 Philippe May
93 5 Philippe May
h2. Hosting
94 5 Philippe May
95 5 Philippe May
The team is located in the CSR of Auroville.
96 5 Philippe May
97 5 Philippe May
We have setup a server for hosting the software and database. See [[CSR_server]] for technical information about the setup.
98 7 Philippe May
99 7 Philippe May
100 11 Philippe May
h3. Backup and restoration of the database
101 1 Philippe May
102 1 Philippe May
See [[db-support]]
103 17 Philippe May
104 17 Philippe May
105 17 Philippe May
h2. Gear
106 17 Philippe May
107 17 Philippe May
h3. Survey equipment
108 17 Philippe May
109 17 Philippe May
See [[survey equipment]]
110 17 Philippe May
111 17 Philippe May
h3. Weather station
112 17 Philippe May
113 17 Philippe May
See [[Ambient_Weather_weather_station]]
114 11 Philippe May
115 11 Philippe May
116 7 Philippe May
h2. Plan for future
117 7 Philippe May
118 8 Philippe May
Beside living well, see [[plan]].
119 8 Philippe May
120 49 Philippe May
h2. Other
121 1 Philippe May
122 50 Philippe May
* [[GDAL (OGR) tools]]
123 49 Philippe May
124 8 Philippe May
h2. Links and references
125 8 Philippe May
126 10 Philippe May
[[links]]
127 16 Philippe May
128 16 Philippe May
129 16 Philippe May
h2. Old docs
130 16 Philippe May
131 15 Philippe May
[[Shapefiles]]
132 15 Philippe May
[[Data (measurements, auxiliary tables)]]
133 16 Philippe May
134 13 Philippe May
h2. Pavneet's docs (imported from gisaf's wiki)
135 1 Philippe May
136 15 Philippe May
137 14 Philippe May
[[Rules of Map making - What all Maps should have!]]
138 14 Philippe May
[[Survey Database]]
139 14 Philippe May
[[Field to finish]]
140 14 Philippe May
[[Survey Data Post-Processing]]
141 14 Philippe May
[[Wells Documentation]]
142 14 Philippe May
[[Civil 3D useful commands]]
143 14 Philippe May
[[Online references for Civil 3D]]
144 1 Philippe May
[[connections in QGIS- using browser panel and Add postGIS]]
145 33 Pavneet Kaur
[[Reconcilation of Raw survey data using pgAdmin]]
146 22 Pavneet Kaur
[[importing RAW data to GISAF]]
147 26 Pavneet Kaur
[[Editing Z value of features in Shapefiles in QGIS]]
148 34 Pavneet Kaur
[[Miscellaneous- Civil 3D]]
149 36 Pavneet Kaur
[[Documentation- Rain Gauge]]
150 40 Pavneet Kaur
[[Documentation- Wells Monitoring (Piezometer by Bala)]]
151 40 Pavneet Kaur
[[Documentation- Flow meter, by Bala]]
152 42 Pavneet Kaur
[[Documentation- DST- Vegetation Indexing]]
153 44 Pavneet Kaur
[[Documentation- DST- Interpolation]]
154 43 Pavneet Kaur
[[Documentation- DST- Survey- Office workflow]]
155 40 Pavneet Kaur
[[From CAD to GIS, by Giulio]]
156 41 Pavneet Kaur
[[QGIS- Miscellaneous]]
157 52 Giulio Di Anastasio
158 1 Philippe May
h2. Giulio's documentation
159 59 Giulio Di Anastasio
160 52 Giulio Di Anastasio
[[Documentation - Reconciliation of points using Gisaf]]
161 65 Giulio Di Anastasio
[[Documentation - Status and Status Changes]]
162 66 Giulio Di Anastasio
[[Documentation - Tags retained after re-import of same geometry]]
163 67 Giulio Di Anastasio
164 67 Giulio Di Anastasio
165 67 Giulio Di Anastasio
h1. Access to data
166 67 Giulio Di Anastasio
167 67 Giulio Di Anastasio
h2. Connection to server directly from CSR
168 67 Giulio Di Anastasio
169 67 Giulio Di Anastasio
To connect to the server directly without going through Aurinoco server, the correct url is
170 67 Giulio Di Anastasio
http://gis.csr.av
171 67 Giulio Di Anastasio
172 67 Giulio Di Anastasio
h2. Connection to Gisaf via QGis through WFS / OGC API
173 67 Giulio Di Anastasio
174 67 Giulio Di Anastasio
This works only on QGis from version 3.14.15 onward
175 67 Giulio Di Anastasio
176 67 Giulio Di Anastasio
In the browser, click on WFS/OGC API, then right-click to create a new connection
177 67 Giulio Di Anastasio
Give a name (e.g. OGC API Qgis Gisaf)
178 67 Giulio Di Anastasio
Give the url https://gis.auroville.org.in/ogcapi
179 67 Giulio Di Anastasio
180 67 Giulio Di Anastasio
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
181 67 Giulio Di Anastasio
Click on OK
182 67 Giulio Di Anastasio
The list of layers will appear in the Browser under WFS/OGC API.
183 68 Giulio Di Anastasio
184 68 Giulio Di Anastasio
185 68 Giulio Di Anastasio
h1. How to create a new projection in QGis
186 68 Giulio Di Anastasio
187 68 Giulio Di Anastasio
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
188 68 Giulio Di Anastasio
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
189 68 Giulio Di Anastasio
Click on the green "+" sign on the right top part of the window to create a new projection.
190 68 Giulio Di Anastasio
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS -  Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
191 68 Giulio Di Anastasio
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
192 68 Giulio Di Anastasio
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
193 68 Giulio Di Anastasio
194 68 Giulio Di Anastasio
Finally, click on OK.
195 68 Giulio Di Anastasio
196 68 Giulio Di Anastasio
In a more explicit  way, the parameters mean the following:
197 68 Giulio Di Anastasio
Map Projection:        TransverseMercator (TM)
198 68 Giulio Di Anastasio
False Easting:          370455.6300
199 68 Giulio Di Anastasio
False Northing:        1328608.9940
200 68 Giulio Di Anastasio
Latitude of Origin:    12°00'57.79560" (DMS)   12.01605433 (DD)
201 68 Giulio Di Anastasio
Central Meridian:     79°48'35.96164"  (DMS)   79.80998934 (DD)
202 68 Giulio Di Anastasio
Scale Factor:             1.00000000
203 68 Giulio Di Anastasio
Zone Width:              6.0°
204 69 Giulio Di Anastasio
205 69 Giulio Di Anastasio
h1. Elimination of Duplicate points – General criteria
206 69 Giulio Di Anastasio
207 69 Giulio Di Anastasio
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
208 69 Giulio Di Anastasio
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
209 69 Giulio Di Anastasio
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
210 69 Giulio Di Anastasio
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
211 69 Giulio Di Anastasio
How to identify duplicate points?
212 69 Giulio Di Anastasio
The following criteria can be used:
213 69 Giulio Di Anastasio
1.	The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
214 69 Giulio Di Anastasio
2.	The orig_id (serial number) of the points are not in series
215 69 Giulio Di Anastasio
3.	The survey date is not the same
216 69 Giulio Di Anastasio
4.	In case of trees, the species of trees is the same
217 69 Giulio Di Anastasio
5.	5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
218 69 Giulio Di Anastasio
6.	The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
219 70 Giulio Di Anastasio
220 70 Giulio Di Anastasio
221 70 Giulio Di Anastasio
h1. Linework for the Survey Area
222 70 Giulio Di Anastasio
223 70 Giulio Di Anastasio
h2. 1.	Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
224 70 Giulio Di Anastasio
225 70 Giulio Di Anastasio
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
226 74 Selvarani C
Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.
227 70 Giulio Di Anastasio
The Initial Linework for the Survey Area is also stored temporarily in 
228 70 Giulio Di Anastasio
229 74 Selvarani C
+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+
230 70 Giulio Di Anastasio
231 70 Giulio Di Anastasio
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
232 70 Giulio Di Anastasio
233 70 Giulio Di Anastasio
h2. 2.	Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
234 1 Philippe May
235 1 Philippe May
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
236 70 Giulio Di Anastasio
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
237 74 Selvarani C
1.	*If features are lines:*
238 70 Giulio Di Anastasio
•	Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
239 70 Giulio Di Anastasio
240 70 Giulio Di Anastasio
h2. The CRS for the export shall be EPSG:4326 - WGS 84
241 70 Giulio Di Anastasio
242 74 Selvarani C
2.	*If features are polygons:*
243 1 Philippe May
•	Lines shall be converted into polygons:
244 74 Selvarani C
245 1 Philippe May
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
246 87 Selvarani C
247 84 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!
248 70 Giulio Di Anastasio
249 86 Selvarani C
250 1 Philippe May
The new window for “Lines to Polygons” conversion will appear:
251 86 Selvarani C
252 86 Selvarani C
253 83 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!
254 1 Philippe May
255 71 Giulio Di Anastasio
•	Always cross check the input layer, to make sure that the input layer is the active one
256 71 Giulio Di Anastasio
•	Save the output in a temporary layer
257 74 Selvarani C
•	The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)
258 72 Giulio Di Anastasio
259 74 Selvarani C
h2. The CRS for the export shall be EPSG:4326 - WGS 84
260 74 Selvarani C
261 72 Giulio Di Anastasio
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
262 1 Philippe May
263 74 Selvarani C
h2. Linework for the whole Survey Zone
264 74 Selvarani C
265 74 Selvarani C
h2. 1.	Merging Shapefiles / Geopackages - (Selvarani, System 1)
266 74 Selvarani C
267 72 Giulio Di Anastasio
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
268 1 Philippe May
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
269 72 Giulio Di Anastasio
•	To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
270 72 Giulio Di Anastasio
271 85 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!
272 1 Philippe May
273 1 Philippe May
274 85 Selvarani C
The new window for “Union” will appear:
275 1 Philippe May
276 86 Selvarani C
277 86 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!
278 73 Giulio Di Anastasio
279 74 Selvarani C
•	To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).
280 73 Giulio Di Anastasio
•	(The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)
281 73 Giulio Di Anastasio
282 74 Selvarani C
h2. 2.	Storing Shapefiles / Geopackages - (Selvarani, System 1)
283 74 Selvarani C
284 74 Selvarani C
Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)
285 73 Giulio Di Anastasio
Date in the name of Final Shapefile / Geopackage needs to be updated.
286 73 Giulio Di Anastasio
Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.
287 73 Giulio Di Anastasio
288 74 Selvarani C
h2. 3.	Topology check of merged shapefiles
289 74 Selvarani C
290 73 Giulio Di Anastasio
The topology checker is applied again on the merged shapefiles / geopackages.
291 73 Giulio Di Anastasio
The “id_field” shall be removed from the attribute table.
292 73 Giulio Di Anastasio
293 74 Selvarani C
h2. 4.	Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)
294 74 Selvarani C
295 73 Giulio Di Anastasio
Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.
296 74 Selvarani C
*Then delete the Merged Shapefile / Geopackage folder from System 1.
297 74 Selvarani C
*
298 73 Giulio Di Anastasio
299 74 Selvarani C
h2. 5.	Note about Shapefiles and Geopackages
300 73 Giulio Di Anastasio
301 73 Giulio Di Anastasio
All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).
302 77 Selvarani C
The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be notes, as Gisaf database does not accept the Multipolygon and Multiline geometry types.